forked from rucio/rucio
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Transfers: native multi-hop between transfertools. Closes rucio#5403
Adapt the transfertool argument of the submitter to also accept a comma-separated list of transfertool names. Most of the needed functionality to accept such a list was already implemented. I perform a small refactoring to put the "external_name" of each transfertool class as a class attribute. This allows for a more generic code in some parts. Also adapt the logic which prepares the path submission to a transfertool. Until now, the submitter was forced to submit the full path to a single transfertool. This commit changes the behavior. Now a full path can be splitted into sub-paths, each of them handled by a different transfertool. In one submitter iteration, only the first sub-path will be submitted, the rest will be left in a "queued" state, but protected from being picked by another submitter thanks to the new "transfer_hops" table. When the first sub-path will be completed, submission will resume naturally. To ensure intermediate hops left in the queued state are picked by the correct submitter, we now set the "transfertool" field on intermediate requests. Obviously, the value of this field can be different from that of the initial request. Because of the experimental nature of the globus transfertool, I add a hard-coded protection which ensures that submission is only done to globus if it's the first element in the submitters "transfertool" argument.
- Loading branch information
Showing
9 changed files
with
157 additions
and
74 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.