feat(io/default): Try to clean up during idle updates #182
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds an attempt to clean up a DefaultIONode during an idle update by: * looking for `.placeholder` files and deleting them * attempting to remove acq directories Because this routine runs when the node is idle (i.e. only when there's no other I/O occurring), no placeholders should be on the node. Any which are found are clearly spurious due to prior crashes. I've also implemented a check to reduce how often it runs. It will always run at start-up (when I suspect most uncleanliness would be found), and then once every 100 times the node transitions from not-idle to idle. While implementing this, I discovered that the code that was deleting acq dirs wasn't stopping at the StorageNode.root, meaning there was a potential to delete the node directory itself (plus anything above that)! In practice, on DefaultIO nodes, this couldn't happen because all such nodes have a `ALPENHORN_NODE` file at the top level, but that's not necessarily true for other IO classes which still use the DefaultIO's delete function (for example, the LustreHSM I/O class). I've fixed this bug while moving the directory deletion code from the delete_async into its own function in `ioutil` because the cleanup task is now also using it. Also, removed submitting an uncessary job which was deleting zero file copies.
ljgray
approved these changes
Jun 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds an attempt to clean up a DefaultIONode during an idle update by:
.placeholderfiles and deleting themBecause this routine runs when the node is idle (i.e. only when there's no other I/O occurring), no placeholders should be on the node. Any which are found are clearly spurious due to prior crashes.
I've also implemented a check to reduce how often it runs. It will always run at start-up (when I suspect most uncleanliness would be found), and then once every 100 times the node transitions from not-idle to idle. Not really sure how often is appropriate. It might even be sufficient to only run it on start-up.
While implementing this, I discovered that the code that was deleting acq dirs wasn't stopping at the StorageNode.root, meaning there was a potential to delete the node directory itself (plus anything above that)!
In practice, on DefaultIO nodes, this couldn't happen because all such nodes have a
ALPENHORN_NODEfile at the top level, but that's not necessarily true for other IO classes which still use the DefaultIO's delete function (for example, the LustreHSM I/O class).I've fixed this bug while moving the directory deletion code from the delete_async into its own function in
ioutilbecause the cleanup task is now also using it.Also, removed submitting an uncessary job which was deleting zero file copies.