GDCC/8604 Improve archiver error handling #8612
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also increased wait to 20K seconds (and use new constant) to help handle larger data
GDCC/8604-improve_archiver_error_handling
…chiver_error_handling
…chiver_error_handling
…chiver_error_handling
…chiver_error_handling
sekmiller
approved these changes
Jul 13, 2022
|
As far as I can see it is up-to-date w.r.t. develop. |
…chiver_error_handling
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: This PR fixes a thread management issue that allowed partial bags to be generated in cases where, for example, datasets were missing physical files. As described in the issue, the new code uses thread.join to assure the main thread gets an accurate success/fail report from the bag generation thread.
The PR also includes changes to the DuraCloud archiver class that changes the folder structure created in the DuraCloud service to add datasets within the same collection to the same DuraCloud 'space'. While transparent to Dataverse, this change makes it easier for an admin to manage batch archiving (see #8610) with Duracloud by reducing the number of manual operations required on that side. The DuraCloud archiver also gets the update from the Google archiver to wait for zipped datafile bytes to be ready before starting the transfer (see large comment in source).
Finally, as part of syncing the DUraCloud and Google ASrchivers and preparing for an S3 archiver, this PR includes refactoring to put the basic setup of the bag generation thread into the abstract base class.
Which issue(s) this PR closes:
Closes #8604
Special notes for your reviewer: These changes have been tested at TDL and QDR (DuraCloud and Google respectively) (albeit in combination with other PRs/with the QDR fork).
Suggestions on how to test this: Not sure there is much that can be tested without setting up Google/DuraCloud. The changes can be demonstrated by TDL/QDR. Another PR(s) are planned as part of DataCommons to apply the same threading technique to storage in S3/Harvard DRS where it should be easier to confirm that the new code works, e.g. in cases where datasets are missing physical files.
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?: see #8611
Additional documentation: