Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revisit deferred URL output transfer #222

Open
JoshKarpel opened this issue May 21, 2020 · 0 comments
Open

Revisit deferred URL output transfer #222

JoshKarpel opened this issue May 21, 2020 · 0 comments
Labels
enhancement New feature or request file-transfer

Comments

@JoshKarpel
Copy link
Contributor

8d91538 enables a worse form of URL output files, where users must specify their remaps ahead-of-time via MapOptions.

This PR will also contain a turned-off implementation of late-binding URL output transfer via a custom plugin. It's not turned on for two reasons:

  • It doesn't work with s3 file transfer, because s3 file transfer is done by pre-signed URLs generated by the shadow. We have no easy way to do this in the plugin right now. We could use HTChirp to fetch the user's s3 credentials, but that connection may not be secure, so sending tokens over it is bad.
  • It requires Python 3 and the Python 3 HTCondor bindings to be installed on bare metal (the plugin cannot run inside the container). This is easy to guarantee in testing, but hard/impossible to guarantee on, say, the OSG.

I think its day will still come, but that day is not today...

Originally posted by @JoshKarpel in #216 (comment)

@JoshKarpel JoshKarpel added enhancement New feature or request file-transfer labels May 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request file-transfer
Projects
None yet
Development

No branches or pull requests

1 participant