Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add @maintenance decorator to methods that cannot be executed concurrently #6

Closed
giovannipizzi opened this issue Apr 13, 2020 · 3 comments
Milestone

Comments

@giovannipizzi
Copy link
Member

Add a guard to avoid maintenance methods (repack, delete, add directly to packs) are performed in parallel.

Add decorator that opens a lock file in exclusive mode, and stops if someone else is already performing maintenance operations.

Possibly check (both this, and the one that locks a pack) that they have a finally block that releases the lock.

@giovannipizzi
Copy link
Member Author

Also:

  • check if there is a second level of locking, e.g. when deleting?
  • until Problem during concurrent tests #4 is fixed, check if we need to prevent also normal usage of the repository by any process while in maintenance? (E.g. with a lock file, containing the PID, and in case of failure of the packing process, we explain how to check if it's still running, and how to delete the lock).

@giovannipizzi
Copy link
Member Author

In AiiDA, we will try to put all safeguards to avoid problems. Putting a decorator properly is a bit tricky, so I am moving this to a 'nice to have'. This will need to be properly documented, though...

@giovannipizzi giovannipizzi mentioned this issue Aug 26, 2020
6 tasks
@giovannipizzi
Copy link
Member Author

After careful thinking: while one could implement guards (e.g. a decorator @maintenance for the relevant methods) to prevent concurrent access (see e.g. issue #6), this is complex to
implement (mostly because one needs to record any time a process starts accessing the repository - so a maintenance
operation can refuse to start - as well as check if any maintenance operation is running at every first access to a
container and refuse to start using it).
While not impossible, this is not easy (also to properly clean up if the computer reboots unexpectedly when a
maintenance operation is running, etc) and also might have performance implications as a check has to be performed
for every new operation on a Container.

We note that however such logic was implemented in AiiDA
(that uses disk-objectstore as a backend). Therefore guards are in place there, and if one needs to do the same
from a different code, inspiration can be taken from the implementation in AiiDA.

I will therefore close this, and the docs will be added as part of #3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant