-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pluggable URI downloaders #281
Comments
We've put the heavy lifting for URI downloading into protocol-specific WDL tasks, which lets them benefit from all task runtime's built-in functionality (fetching docker images for binary dependencies, running them asynchronously, parallel scheduling & resource isolation, logging, error/signal handling, retry logic, ...). Three further considerations for the downloader plug-in mechanism:
setuptools entry points are a strong candidate for how plugins should be installed and registered, then discovered with pkg_resources. Other helpful: https://amir.rachum.com/blog/2017/07/28/python-entry-points/ (blog) |
Discover plugins with the setuptools entry point group "miniwdl.plugin.file_download", named as the supported URI scheme (e.g. "s3" or "gs"). The entry point of each plugin is a context manager yielding WDL task source code and inputs to accomplish the download. #281
The runtime's URI downloading code is meant to be extensible, by providing custom WDL tasks to download based on the URI scheme (s3://, gs://, etc.). The internal foundation is there (#260 #279), but we need to define and document the plug-in mechanism.
The text was updated successfully, but these errors were encountered: