-
Notifications
You must be signed in to change notification settings - Fork 765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use boto3 instead of awscli python module to copy packages #387
Comments
Thanks for the issue! Yes indeed, |
True about the bootstrapping. I like the minimal dependencies. With boto3 I tested replacing https://github.com/Netflix/metaflow/blob/master/metaflow/environment.py#L90 with: def get_python_download_s3_file(self, file_url, local_path):
return (
"%s -c \"import boto3; " % self._python()
+ "exec('try:\\n from urlparse import urlparse\\nexcept:\\n from urllib.parse import urlparse');"
+ "parsed = urlparse('%s'); " % file_url
+ "boto3.client('s3').download_file(parsed.netloc, parsed.path.lstrip('/'), '%s')\"" % local_path
) |
@savingoyal I've reopened a PR for this because we are seeing issues where if someone has both awscli v1 and v2 installed, the python dependencies are incompatible / break in weird ways. So instead of relying on this implicit dependency on awscli v1, we should just use boto3 to bootstrap. |
this is now released in version |
awscli
is superseded by awscliv2 which doesn't support a Python package.Solution:
aws s3 cp
python module/file in Metaflow that uses uses already declared and installedboto3
dependency instead of deprecated (and undeclared dependency)python -m awscli
to perform the following action:https://github.com/Netflix/metaflow/blob/master/metaflow/environment.py#L90
The text was updated successfully, but these errors were encountered: