Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Support for Globus transfer? #71
It would be great to have support for Globus transfer within Parsl. We are interested in, e.g., transferring 10s-1000sGB from a server at NPCF to an XSEDE resource such as Comet or Bridges for (re)processing.
We aren't currently using Parsl but the abstraction of queue managers is an attractive feature. Globus transfer would be another particularly useful feature, that we will need to implement somehow. The docs say Globus support will appear "in the near future" and I see it listed in the roadmap. However, I can't find an open issue so I am opening one.
My questions are:
We have started working on Globus integration but don't yet have a timeline for when it will be released.
The basic model is built around a Globus file abstraction that will allow scripts to be written with references to remote (Globus accessible data). Parsl will handle the staging of data to the endpoint/worker node based on the site configuration. As part of this work we're adding Globus Auth support to authenticate through the Parsl script.
Do you have any thoughts or requirements for how you'd like Globus to be integrated? We could also put together a proof of concept by using Parsl + the Globus SDK to get a better idea of how this could/should work.
We have a meeting Feb 2 with your group that @danielskatz set up. We can discuss then, but I wanted to get the discussion started so that we can make the most of that time. For requirements, I'll defer to the expert, @max-zilla, but here is a start:
referenced this issue
Feb 14, 2018
added a commit
Mar 24, 2018
@dlebauer, We have globus support going into the upcoming release. Please note that this feature is still in an experimental state, and we'll be pushing refinements to the File model quickly in minor releases.