implement support for transfers via BitTorrent #6479
Comments
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Jan 22, 2024
Additional metadata is added to each file did: the merkle root and piece layers used by the bittorrent v2 format. This way, we can reconstruct the .torrent files from this data, allowing us to transfer files directly between RSEs using bittorrent clients running on each of the RSEs. This, initial implementation, relies on the qBittorrent client, but other clients which support bittorrent v2 could be added later. A new RSE protocol was needed for the task. The protocol must be configured with the hostname+port of the bittorrent's data channel and the 'magnet' scheme. We use custom extensions to the magnet format to store the name/scope/path of a replica, so the link is not currently importable into existing torrent clients, but this leaves the door open for the future. It would be possible to generate magnet links which actually work with such clients directly in list-replicas.
voetberg
pushed a commit
to voetberg/rucio
that referenced
this issue
Mar 21, 2024
Additional metadata is added to each file did: the merkle root and piece layers used by the bittorrent v2 format. This way, we can reconstruct the .torrent files from this data, allowing us to transfer files directly between RSEs using bittorrent clients running on each of the RSEs. This, initial implementation, relies on the qBittorrent client, but other clients which support bittorrent v2 could be added later. A new RSE protocol was needed for the task. The protocol must be configured with the hostname+port of the bittorrent's data channel and the 'magnet' scheme. We use custom extensions to the magnet format to store the name/scope/path of a replica, so the link is not currently importable into existing torrent clients, but this leaves the door open for the future. It would be possible to generate magnet links which actually work with such clients directly in list-replicas.
voetberg
pushed a commit
to voetberg/rucio
that referenced
this issue
Apr 15, 2024
Additional metadata is added to each file did: the merkle root and piece layers used by the bittorrent v2 format. This way, we can reconstruct the .torrent files from this data, allowing us to transfer files directly between RSEs using bittorrent clients running on each of the RSEs. This, initial implementation, relies on the qBittorrent client, but other clients which support bittorrent v2 could be added later. A new RSE protocol was needed for the task. The protocol must be configured with the hostname+port of the bittorrent's data channel and the 'magnet' scheme. We use custom extensions to the magnet format to store the name/scope/path of a replica, so the link is not currently importable into existing torrent clients, but this leaves the door open for the future. It would be possible to generate magnet links which actually work with such clients directly in list-replicas.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Rucio already supports two 3rd party copy drivers: globus and FTS. However, both of these strongly rely on non-standart/wlcg-specific extensions to execute transfers between source and destination RSEs.
What if we could just install an of-the-shelf bittorrent client on each RSE and use its administrative interface to move the desired data. In theory, it would even be possible to perform "true" multi-source transfers this way.
The text was updated successfully, but these errors were encountered: