Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added preupload function to fix issues on failed uploads #52

Merged
merged 2 commits into from
Feb 14, 2024

Conversation

lucascottolavina
Copy link
Contributor

Added a preupload function to fix issues on failed uploads. The idea is that, in case an upload fails, there might be "orphan" files (files successfully copied to the target RSE by Rucio, with Rucio that has been able to checksum them and create metadata for them, however not attached to the dataset rule, and the rule of the dataset often not even created).

In that case, the preupload function needs to be run before retrying again the upload, because the preupload does the following actions:

  • It adds the dataset associated to the did we wanted to upload
  • It loops over all local files to be uploaded, so to know their number and their names. For each file, it searches in Rucio catalogue if such a filename is already present. If so, it attaches it to the dataset
  • finally, it creates a replication rule on the RSE (the RSE is an input parameter of the preupload function, however, it's important that the RSE must be the same chosen by the previous upload attempt). After this latest operation, the did will show up in Rucio

As a conclusion, after this preupload, the did shows up in Rucio, but containing only the orphan files. Then the upload function needs to be executed again so that all remaining files can be uploaded and the did will be complete.

Copy link

@FaroutYLq FaroutYLq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this fix. Can we add some docstr to it to explain the motivation and what preupload does. Basically the PR description can be copy-pasted here I think.

Also it might also worth emphasizing that we have to assume the dataset rule has been removed to make this work. If the dataset containing an orphan file has dataset rule existed, the upload will fail as before. I think we are fine now and I am happy, but in future we can have another surgery in admix.upload itself to avoid this scenario.

@FaroutYLq FaroutYLq merged commit 44f1630 into stable Feb 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants