implement cross-transfertool multihop #5403
Comments
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Mar 28, 2022
Adapt the transfertool argument of the submitter to also accept a comma-separated list of transfertool names. Most of the needed functionality to accept such a list was already implemented. I perform a small refactoring to put the "external_name" of each transfertool class as a class attribute. This allows for a more generic code in some parts. Also adapt the logic which prepares the path submission to a transfertool. Until now, the submitter was forced to submit the full path to a single transfertool. This commit changes the behavior. Now a full path can be splitted into sub-paths, each of them handled by a different transfertool. In one submitter iteration, only the first sub-path will be submitted, the rest will be left in a "queued" state, but protected from being picked by another submitter thanks to the new "transfer_hops" table. When the first sub-path will be completed, submission will resume naturally. To ensure intermediate hops left in the queued state are picked by the correct submitter, we now set the "transfertool" field on intermediate requests. Obviously, the value of this field can be different from that of the initial request. Because of the experimental nature of the globus transfertool, I add a hard-coded protection which ensures that submission is only done to globus if it's the first element in the submitters "transfertool" argument.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Mar 28, 2022
Adapt the transfertool argument of the submitter to also accept a comma-separated list of transfertool names. Most of the needed functionality to accept such a list was already implemented. I perform a small refactoring to put the "external_name" of each transfertool class as a class attribute. This allows for a more generic code in some parts. Also adapt the logic which prepares the path submission to a transfertool. Until now, the submitter was forced to submit the full path to a single transfertool. This commit changes the behavior. Now a full path can be splitted into sub-paths, each of them handled by a different transfertool. In one submitter iteration, only the first sub-path will be submitted, the rest will be left in a "queued" state, but protected from being picked by another submitter thanks to the new "transfer_hops" table. When the first sub-path will be completed, submission will resume naturally. To ensure intermediate hops left in the queued state are picked by the correct submitter, we now set the "transfertool" field on intermediate requests. Obviously, the value of this field can be different from that of the initial request. Because of the experimental nature of the globus transfertool, I add a hard-coded protection which ensures that submission is only done to globus if it's the first element in the submitters "transfertool" argument.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Mar 30, 2022
Adapt the transfertool argument of the submitter to also accept a comma-separated list of transfertool names. Most of the needed functionality to accept such a list was already implemented. I perform a small refactoring to put the "external_name" of each transfertool class as a class attribute. This allows for a more generic code in some parts. Also adapt the logic which prepares the path submission to a transfertool. Until now, the submitter was forced to submit the full path to a single transfertool. This commit changes the behavior. Now a full path can be splitted into sub-paths, each of them handled by a different transfertool. In one submitter iteration, only the first sub-path will be submitted, the rest will be left in a "queued" state, but protected from being picked by another submitter thanks to the new "transfer_hops" table. When the first sub-path will be completed, submission will resume naturally. To ensure intermediate hops left in the queued state are picked by the correct submitter, we now set the "transfertool" field on intermediate requests. Obviously, the value of this field can be different from that of the initial request. Because of the experimental nature of the globus transfertool, I add a hard-coded protection which ensures that submission is only done to globus if it's the first element in the submitters "transfertool" argument.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Mar 30, 2022
Adapt the transfertool argument of the submitter to also accept a comma-separated list of transfertool names. Most of the needed functionality to accept such a list was already implemented. I perform a small refactoring to put the "external_name" of each transfertool class as a class attribute. This allows for a more generic code in some parts. Also adapt the logic which prepares the path submission to a transfertool. Until now, the submitter was forced to submit the full path to a single transfertool. This commit changes the behavior. Now a full path can be splitted into sub-paths, each of them handled by a different transfertool. In one submitter iteration, only the first sub-path will be submitted, the rest will be left in a "queued" state, but protected from being picked by another submitter thanks to the new "transfer_hops" table. When the first sub-path will be completed, submission will resume naturally. To ensure intermediate hops left in the queued state are picked by the correct submitter, we now set the "transfertool" field on intermediate requests. Obviously, the value of this field can be different from that of the initial request. Because of the experimental nature of the globus transfertool, I add a hard-coded protection which ensures that submission is only done to globus if it's the first element in the submitters "transfertool" argument.
bari12
added a commit
that referenced
this issue
Apr 4, 2022
Transfers: native multi-hop between transfertools. Closes #5403
bari12
pushed a commit
that referenced
this issue
Apr 4, 2022
Adapt the transfertool argument of the submitter to also accept a comma-separated list of transfertool names. Most of the needed functionality to accept such a list was already implemented. I perform a small refactoring to put the "external_name" of each transfertool class as a class attribute. This allows for a more generic code in some parts. Also adapt the logic which prepares the path submission to a transfertool. Until now, the submitter was forced to submit the full path to a single transfertool. This commit changes the behavior. Now a full path can be splitted into sub-paths, each of them handled by a different transfertool. In one submitter iteration, only the first sub-path will be submitted, the rest will be left in a "queued" state, but protected from being picked by another submitter thanks to the new "transfer_hops" table. When the first sub-path will be completed, submission will resume naturally. To ensure intermediate hops left in the queued state are picked by the correct submitter, we now set the "transfertool" field on intermediate requests. Obviously, the value of this field can be different from that of the initial request. Because of the experimental nature of the globus transfertool, I add a hard-coded protection which ensures that submission is only done to globus if it's the first element in the submitters "transfertool" argument.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Motivation
Right now, rucio can only perform a multihop transfer if its entire path is submitter at once to the same FTS instance. Until recently, rucio didn't even know which transfers were multi-hop after submitting them.
Since 1.28, a new transfer_hops table was introduced, which modelizes the dependency between transfers, effectively storing a "path" in the database. It is now possible to "pause" a transfer from being handled in because it waits for a previous hop to be executed. This opens the door for cross-transfertool multi-hops, i.e. when part of the transfer will be executed by FTS, and another by globus.
The text was updated successfully, but these errors were encountered: