Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pluggable URI downloaders #281

Closed
mlin opened this issue Nov 18, 2019 · 3 comments
Closed

pluggable URI downloaders #281

mlin opened this issue Nov 18, 2019 · 3 comments
Labels
enhancement New feature or request

Comments

@mlin
Copy link
Collaborator

mlin commented Nov 18, 2019

The runtime's URI downloading code is meant to be extensible, by providing custom WDL tasks to download based on the URI scheme (s3://, gs://, etc.). The internal foundation is there (#260 #279), but we need to define and document the plug-in mechanism.

@mlin mlin added the enhancement New feature or request label Nov 18, 2019
@mlin
Copy link
Collaborator Author

mlin commented Dec 8, 2019

@mlin
Copy link
Collaborator Author

mlin commented Dec 11, 2019

We've put the heavy lifting for URI downloading into protocol-specific WDL tasks, which lets them benefit from all task runtime's built-in functionality (fetching docker images for binary dependencies, running them asynchronously, parallel scheduling & resource isolation, logging, error/signal handling, retry logic, ...).

Three further considerations for the downloader plug-in mechanism:

  1. How are plugins installed & discovered by the runtime
  2. How to provide any security credentials the downloader may need (e.g. AWS key id & secret)
  3. Answers should generalize to other kinds of plugins we're going to have eventually

setuptools entry points are a strong candidate for how plugins should be installed and registered, then discovered with pkg_resources.

Other helpful:

https://amir.rachum.com/blog/2017/07/28/python-entry-points/ (blog)
https://entrypoints.readthedocs.io/en/latest/ (streamline discovery)
conda/conda#404 (conda considerations)
jupyter/notebook#2894 (discussion of numerous practical issues in the analogous problem for Jupyter)

@kislyuk

@mlin
Copy link
Collaborator Author

mlin commented Dec 13, 2019

mlin added a commit that referenced this issue Dec 16, 2019
Discover plugins with the setuptools entry point group "miniwdl.plugin.file_download", named as the supported URI scheme (e.g. "s3" or "gs"). The entry point of each plugin is a context manager yielding WDL task source code and inputs to accomplish the download.

#281
@mlin mlin closed this as completed Mar 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant