Added preupload function to fix issues on failed uploads #52
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a preupload function to fix issues on failed uploads. The idea is that, in case an upload fails, there might be "orphan" files (files successfully copied to the target RSE by Rucio, with Rucio that has been able to checksum them and create metadata for them, however not attached to the dataset rule, and the rule of the dataset often not even created).
In that case, the preupload function needs to be run before retrying again the upload, because the preupload does the following actions:
As a conclusion, after this preupload, the did shows up in Rucio, but containing only the orphan files. Then the upload function needs to be executed again so that all remaining files can be uploaded and the did will be complete.