Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI: Improved customizability and scriptability of verdi storage maintain #5936

Merged
merged 15 commits into from
Mar 30, 2023

Conversation

Crivella
Copy link
Contributor

@Crivella Crivella commented Mar 17, 2023

Introduced changes

This pull requests, adds 2 option to verdi storage maintain:

  • --no-repack: disables the do_repack option when running a full maintenance with the DiskObjectStore backend.
  • --yes or -y: skip the confirmation prompt to use command in scripts (eg periodic maintenance with CRON)

Not using the flags leave the behavior of the command unaltered for back-compatibility

Reasoning

When performing a storage maintenance operation through the verdi CLI, the only two available options right now is to do either a live or non live operation.

Using the DiskObjectStore backend this translates to doing one of the 2:

  • pack_all_loose only, which store all loose files inside packs, but does not remove the loose files itself.
  • pack_all_loose, do_repack, clean_storage, vacuum

I think it would be useful to have a more granular control of the live maintain operation.
Especially the do_repack option can be undesired, when performing maintenance on a huge repository that is being backed up via rsync. Instead, performing the clean_storage periodically is almost mandatory for having efficient backups. Without it we end-up in the same situation as the old storage of having a huge amount of IO nodes to rsync, plus the redundancy of having both the loose and packed version of a file if verdi storage maintain is being run periodically.

@Crivella Crivella changed the title Improved customizability and scriptability of verdi storage mantain CLI: Improved customizability and scriptability of verdi storage mantain Mar 17, 2023
@Crivella Crivella changed the title CLI: Improved customizability and scriptability of verdi storage mantain CLI: Improved customizability and scriptability of verdi storage maintain Mar 22, 2023
@ltalirz
Copy link
Member

ltalirz commented Mar 26, 2023

hey @Crivella, just checking: have you talked to anyone from the AiiDA team about this already who might be able to review this PR?

@Crivella
Copy link
Contributor Author

Hi @ltalirz, I got in touch with @chrisjsewell on slack last week, he said he should give it a look this week.

@chrisjsewell chrisjsewell self-requested a review March 30, 2023 08:01
Copy link
Member

@chrisjsewell chrisjsewell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great cheers!

@chrisjsewell chrisjsewell merged commit f581309 into aiidateam:main Mar 30, 2023
@Crivella Crivella deleted the feature-no_repack branch March 30, 2023 08:30
@sphuber
Copy link
Contributor

sphuber commented Mar 30, 2023

Thanks @Crivella , useful addition. I was just wondering if we could have better used the --force option instead of --yes. The --force option is used consistently across verdi to skip prompts. Introducing a new option (--yes is not used anywhere as far as I know) reduces consistency. Do you agree @chrisjsewell and @Crivella ? If so, I can quickly open a PR to address that.

@Crivella
Copy link
Contributor Author

Hi @sphuber I am not against it. Sorry, didn't notice that --force was already used when i grepped since it was hidden in the OverridableOption

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants