Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Globus transfer concurrency #171

Closed
lukaszlacinski opened this issue Dec 13, 2021 · 1 comment
Closed

Improve Globus transfer concurrency #171

lukaszlacinski opened this issue Dec 13, 2021 · 1 comment
Assignees
Labels
Globus Globus priority: high High priority task

Comments

@lukaszlacinski
Copy link
Contributor

With --non-blocking option, zstash submits a Globus transfer and immediately creates a subsequent tarball. Zstash does not wait until the transfer completes to start creating a subsequent tarball. On machines where zstash tarballs are created faster than they are transferred to a remote endpoint, zstash will create multiple Globus transfers that are put in a queue by the Globus service. Zstash should submit transfers to transfer as many zstash tarballs as have been created at the moment. On machines where it takes more time to create a tarball than transfer it, each Globus transfer will have one file. On machines where it takes less time to create a tarball than transfer it, the first transfer will have one file, but the number of tarballs in subsequent transfers will grow finding dynamically the most optimal number of tarballs per transfer.

@lukaszlacinski lukaszlacinski self-assigned this Dec 13, 2021
@forsyth2 forsyth2 added the Globus Globus label Dec 13, 2021
@forsyth2 forsyth2 added the priority: high High priority task label Aug 29, 2022
@forsyth2
Copy link
Collaborator

forsyth2 commented Nov 8, 2022

Resolved by #214.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Globus Globus priority: high High priority task
Projects
None yet
Development

No branches or pull requests

2 participants