Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conda support #108

Open
wants to merge 14 commits into
base: master
from

Conversation

Projects
None yet
@mathieu1
Copy link
Contributor

mathieu1 commented May 24, 2016

This is a reboot of #94 targetting the dev branch. Still WIP, discussion can start after some documentation is provided.

@mathieu1 mathieu1 referenced this pull request May 24, 2016

Closed

Support conda environments #94

0 of 7 tasks complete
@coveralls

This comment has been minimized.

Copy link

coveralls commented May 24, 2016

Coverage Status

Coverage decreased (-2.2%) to 74.816% when pulling ea78346 on mathieu1:conda_support into 1202044 on Miserlou:dev.

@mathieu1 mathieu1 force-pushed the mathieu1:conda_support branch from ea78346 to 618aa8e May 26, 2016

@coveralls

This comment has been minimized.

Copy link

coveralls commented May 26, 2016

Coverage Status

Coverage decreased (-2.2%) to 74.847% when pulling 618aa8e on mathieu1:conda_support into 1202044 on Miserlou:dev.

@coveralls

This comment has been minimized.

Copy link

coveralls commented May 27, 2016

Coverage Status

Coverage decreased (-2.3%) to 74.724% when pulling 2916e9e on mathieu1:conda_support into 1202044 on Miserlou:dev.

@mathieu1

This comment has been minimized.

Copy link
Contributor Author

mathieu1 commented May 27, 2016

Ok, the discussion can start. Here's the current status:

  • Implement condasupport, including full content of conda environment (not only python packages).
  • Make a conda package for zappa and for each of its dependencies (uploaded to my channel on anaconda.org). The current version of this branch is packaged under version 0.17.7.dev.
  • Create a repository (see mathieu1/zappa_conda) which serves both to test the current branch and can serve as an initial documentation for now, to help decide on the next steps.
  • Document how to use zappa with conda on other platforms than linux-64 (and discuss limitations, e.g. Windows doesn't have symlinks)
  • Work further on minifying conda packages. On this topic, available conda packages of numpy depend on large numerical optimization libraries (MKL, >100MB! or openblas, >10MB). This contributes to making it hard to fit other packages that depend on numpy such as pandas, scipy, scikit-learn, (although each of them is large on its own). A "noblas" version of numpy is underway, see conda-forge/numpy-feedstock#2

To start the discussion, here are a few questions/issues:

  • Given the advantages listed in #94 and the fact that it does not change any behavior for non-conda users, does this PR look viable for inclusion in Zappa? Should I explain/motivate further?
  • I am still unsure how to proceed with testing. The best I have in that direction is the above-mentioned repository, but it's hard to automate this testing from within zappa.
@coveralls

This comment has been minimized.

Copy link

coveralls commented May 27, 2016

Coverage Status

Coverage decreased (-2.6%) to 74.45% when pulling 75b98f6 on mathieu1:conda_support into 1202044 on Miserlou:dev.

@Miserlou

This comment has been minimized.

Copy link
Owner

Miserlou commented May 31, 2016

This is really cool progress! Sorry I haven't been very responsive on this thread.

exclude=self.zappa_settings[self.api_stage].get('exclude', [])
exclude=self.zappa_settings[self.api_stage].get('exclude', []),
exclude_conda_packages = self.zappa_settings[self.api_stage].get('exclude_conda_packages',
['pip','python','readline','sqlite','wheel', 'boto3', 'botocore'])

This comment has been minimized.

@mathom

mathom Jul 6, 2016

Collaborator

I believe openssl and tk (among other things) are included with miniconda as well. You probably won't want them in your lambda function either.

@mathom

This comment has been minimized.

Copy link
Collaborator

mathom commented Jul 6, 2016

As far as testing goes I'd recommend mocking and building out units vs trying to figure out how to make some AWS deployment for it.

Edit: testing packaging/unpackaging locally would probably be a good idea too.

@mathom

This comment has been minimized.

Copy link
Collaborator

mathom commented Jul 6, 2016

It might be useful to warn the end user if there are large packages in their conda environment, too. The limits can be found here: http://docs.aws.amazon.com/lambda/latest/dg/limits.html . Note that there's a compressed as well as an uncompressed limitation.

@Erstwild

This comment has been minimized.

Copy link

Erstwild commented Jul 11, 2016

Tried out the dev branch and did some tinkering. Could not get conda support to work for me. I'm going to stick with elastic beanstalk for now. Definitely, be back in the future when I have more time.

@sdementen

This comment has been minimized.

Copy link
Contributor

sdementen commented Sep 11, 2016

hello @mathieu1,
At the target, would the approach you are exploring/developing allow to develop within a conda environement on whatever Conda supported system (linux/windows/MacOS, 32/64 bit, ...) and then to package in a zip file the equivalent environment for the AWS image running behind AWS lambda ?
With the ability to choose "light" version to fit within AWS limits ?

@brendancol

This comment has been minimized.

Copy link

brendancol commented Nov 26, 2016

@mathieu1 @Miserlou this is super exciting.

@mathieu1 mathieu1 force-pushed the mathieu1:conda_support branch 2 times, most recently from 4047089 to 2707025 Nov 27, 2016

@coveralls

This comment has been minimized.

Copy link

coveralls commented Nov 27, 2016

Coverage Status

Coverage decreased (-1.1%) to 77.094% when pulling 35618b0 on mathieu1:conda_support into bee2601 on Miserlou:dev.

@kalefranz

This comment has been minimized.

Copy link

kalefranz commented Nov 28, 2016

Hey everyone. I'm the lead developer on the conda project right now. Just wanted to chime in and say I think this is awesome. I'll try to spend more time with this in the coming days. If there are any specific questions or things you are stuck on, please ask! Tagging @msarahan here too, who's the tech lead of the conda-build project.

@Miserlou

This comment has been minimized.

Copy link
Owner

Miserlou commented Nov 28, 2016

Cool! Hey @kalefranz and @msarahan ! Thanks for stopping by, I would love to have your support on this awesome but neglected PR.

Please also feel free to stop by the Slack channel for discussion: https://slack.zappa.io/

@viksit

This comment has been minimized.

Copy link

viksit commented Dec 5, 2016

+1 here. I think the conda support for Zappa is going to be critical for those of us who want to deploy existing applications which rely on scientific computing libraries (numpy, tensorflow et al) which are already running in these environments. It's impossible to move them to a virtuaenv format given conda ships with platform compliant versions of the lower level C libraries that speed up their python wrappers.

@Miserlou

This comment has been minimized.

Copy link
Owner

Miserlou commented Dec 5, 2016

Zappa does support hundreds of those libraries already, however conda support would certainly improve that number. See more here: https://blog.zappa.io/posts/zappa-adds-support-for-manylinux-wheels

@yoavram

This comment has been minimized.

Copy link

yoavram commented Dec 19, 2016

+1

@hello-di

This comment has been minimized.

Copy link

hello-di commented Dec 24, 2016

+1 for conda support!

quit()

if conda_mode:

This comment has been minimized.

@postelrich

postelrich Dec 28, 2016

Except for the precompiled_packages warning, why does this matter if its conda_mode or not?

This comment has been minimized.

@mathieu1

mathieu1 Jan 8, 2017

Author Contributor

It changes the way packaging works. In the standard virtualenv mode, zappa takes the site-packages folder of the virtualenv and possibly replaces some binaries using lambda-packages. In conda_mode zappa does a few things:

  1. Create a temporary copy of the conda environment for the following manipulations
  2. Remove excluded conda packages (like python itself)
  3. Move the site-packages folder to the root of the conda environment
  4. Zip up this moved site-packages along with all other remaining binaries in the conda environment.
if 'VIRTUAL_ENV' in os.environ:
venv = os.environ['VIRTUAL_ENV']
elif 'CONDA_ENV_PATH' in os.environ:

This comment has been minimized.

@postelrich

postelrich Dec 28, 2016

envvar is now CONDA_PREFIX

This comment has been minimized.

@bearnshaw

bearnshaw Jan 4, 2017

may want to check both CONDA_ENV_PATH and CONDA_PREFIX while people have various versions of conda installed.

This comment has been minimized.

@mathieu1

mathieu1 Jan 8, 2017

Author Contributor

Thanks, this is fixed.

@bearnshaw

This comment has been minimized.

Copy link

bearnshaw commented Jan 4, 2017

+1 from me too. @Miserlou What needs to be done here to merge? Get the coverage to 100%?

@mathieu1 mathieu1 force-pushed the mathieu1:conda_support branch from 35618b0 to fd85cca Jan 7, 2017

@coveralls

This comment has been minimized.

Copy link

coveralls commented Jan 7, 2017

Coverage Status

Coverage increased (+1.6%) to 79.786% when pulling fd85cca on mathieu1:conda_support into bee2601 on Miserlou:dev.

@coveralls

This comment has been minimized.

Copy link

coveralls commented Jan 7, 2017

Coverage Status

Coverage increased (+1.6%) to 79.786% when pulling 03e8d95 on mathieu1:conda_support into bee2601 on Miserlou:dev.

@mattalhonte

This comment has been minimized.

Copy link

mattalhonte commented Dec 4, 2017

Sorry if it's annoying, but the mainpage includes the line:

Conda users should comment here.

Maybe there should be links to the stuff you're describing instead?

@cscanlin

This comment has been minimized.

Copy link

cscanlin commented Dec 4, 2017

Fair enough, but the stuff I'm describing is here. The branch this PR references is functional for basic conda environments, and you can use it as you see fit. It can be a little buggy though, so it's just not ready to be merged.

I guess more specifically my suggestion would be for someone to try out this branch if they're interested, and then report back with error messages if it doesn't work. This way we can start moving forward on fixing the remaining bugs so this can actually be merged.

That said, this feature doesn't really have a "sponsor" at the moment to help drive this forward. So anybody who really needs it is encouraged to step up!

@mattalhonte

This comment has been minimized.

Copy link

mattalhonte commented Dec 4, 2017

Oh, neat! Assumed it would take a deep knowledge of Lambda and/or conda in order to help out here. If just giving it a go and posting error messages would be useful, that's a horse of a different color! :)

@cscanlin

This comment has been minimized.

Copy link

cscanlin commented Dec 4, 2017

👍 Check out this repo linked above for a minimum example of a working deploy too: https://github.com/mathieu1/zappa_conda

@devinbostIL

This comment has been minimized.

Copy link

devinbostIL commented Dec 6, 2017

This is awesome! My team is very interested in implementing this. What still needs to be done to get Zappa working completely with anaconda?

@mattalhonte

This comment has been minimized.

Copy link

mattalhonte commented Dec 7, 2017

Here's what I got after trying to do the "Hello World" example:

Creating zappa-conda-dev-ZappaLambdaExecutionRole IAM Role..
Creating zappa-permissions policy on zappa-conda-dev-ZappaLambdaExecutionRole IAM Role.
Packaging project as zip..
Traceback (most recent call last):
  File "/home/matt/anaconda2/envs/zappa_conda_env/bin/zappa", line 6, in <module>
    sys.exit(zappa.cli.handle())
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/cli.py", line 2197, in handle
    cli.on_exit()
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/cli.py", line 1945, in on_exit
    self.remove_uploaded_zip()
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/cli.py", line 1936, in remove_uploaded_zip
    self.zappa.remove_from_s3(self.handler_path, self.s3_bucket_name)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/zappa.py", line 775, in remove_from_s3
    self.s3_client.delete_object(Bucket=bucket_name, Key=file_name)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/client.py", line 517, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/client.py", line 572, in _convert_to_request_dict
    api_params, operation_model)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/validate.py", line 270, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Key, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

@solalatus

This comment has been minimized.

Copy link

solalatus commented Dec 19, 2017

+1 for Conda, deploying some machine learning models would be easier

@joaovcarvalho

This comment has been minimized.

Copy link

joaovcarvalho commented Jan 19, 2018

+1 for conda support

@gwindesPP

This comment has been minimized.

Copy link

gwindesPP commented Feb 27, 2018

@Miserlou and @mathieu1 any updates on readiness of conda support? Thread seems to have gone quite.

@andrewbolster

This comment has been minimized.

Copy link

andrewbolster commented Apr 9, 2018

bump

@aneesha

This comment has been minimized.

Copy link

aneesha commented May 11, 2018

  • 1 for support
@kalefranz

This comment has been minimized.

Copy link

kalefranz commented May 11, 2018

This is something I've wanted to work on for a long time, but I just don't have the bandwidth given other priorities. However, I'm at PyCON now and through the full sprint week next week, and if anyone has questions about conda or wants pointers, I'm happy to chat.

@jkarimi91

This comment has been minimized.

Copy link

jkarimi91 commented May 29, 2018

bump

@lsorber

This comment has been minimized.

Copy link

lsorber commented Jun 12, 2018

Comparing this PR's implementation for conda support with the current packaging implementation in zappa/core.py, it looks like a workaround could be:

  1. Activate your conda environment.
  2. Run export VIRTUAL_ENV=$CONDA_PREFIX.
  3. Try to zappa update dev.

I imagine you would need to run these commands on Linux or a lambda-compatible Docker container for it to work though. Also, you might want to disable use_precompiled_packages.

Anyone care to give the above a try?

@oldani

This comment has been minimized.

Copy link

oldani commented Jun 20, 2018

bump

@lsorber

This comment has been minimized.

Copy link

lsorber commented Jun 20, 2018

I can confirm that the above workaround works fine for conda environments.

The only remaining issue is distributing shared libraries under $VIRTUAL_ENV/lib. That's an issue that applies to both conda and virtualenv's however, and a workaround for that is described in #1260.

@orangeSi

This comment has been minimized.

Copy link

orangeSi commented Aug 22, 2018

@lsorber I tried as you said, like this:

cd /some/zappa/demo1
source activate zappa
export VIRTUAL_ENV=/somepath/conda/anaconda3/envs/zappa
zappa deploy dev

it seem worked first, but finally got these error :

Downloading and installing dependencies..
 - sqlite==python36: Using precompiled lambda package
Packaging project as zip.
Traceback (most recent call last):
  File "/some/zappa/demo1/handler_venv/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/ssl_.py", line 267, in ssl_wrap_socket
FileNotFoundError: [Errno 2] No such file or directory

update1:
I try again, it said Connection timed out, maybe the problem of network, I will try later.

update2:
now I got this error:

Deploying API Gateway..
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

thanks for help~

@orangeSi

This comment has been minimized.

Copy link

orangeSi commented Aug 23, 2018

update:
I do like as this https://developer.amazon.com/zh/blogs/alexa/post/8e8ad73a-99e9-4c0f-a7b3-60f92287b0bf/new-alexa-tutorial-deploy-flask-ask-skills-to-aws-lambda-with-zappa ,it works!
and then I replace virtualenv with conda, it works also!
try it just~

@leonxchen

This comment has been minimized.

Copy link

leonxchen commented Sep 5, 2018

@mathieu1 is this thread died or it is still WIP?

@mihow

This comment has been minimized.

Copy link
Contributor

mihow commented Oct 22, 2018

@kalefranz I love the approach of just including an environment.yml file and letting Zappa install the packages from that. (You proposed this way back in #108 (comment)).

The current Zappa approach of using the packages in your active environment causes unpredictable problems for us. Some devs end up with different packages in there working env that are deployed by Zappa, but aren't actually needed by the app. Also both our continuous integration server and our containerized version of the app need to install a virtualenv and activate it before running zappa update. It would be slick if it was just enough to have a environment.yml file (or Pipfile.lock, requirements.txt, etc. for that matter)

@agilebean

This comment has been minimized.

Copy link

agilebean commented Dec 5, 2018

are people still working on this? would be great as - like many others - work with many conda environments...

@EKami

This comment has been minimized.

Copy link

EKami commented Dec 31, 2018

Any update on this?

@Adblu

This comment has been minimized.

Copy link

Adblu commented Feb 19, 2019

conda support, conda support, conda support... hmm.... WHERE IS CONDA SUPPORT !!!!!!!!!!!!!!

@Adblu

This comment has been minimized.

Copy link

Adblu commented Feb 19, 2019

conda support, conda support, conda support... hmm.... WHERE IS CONDA SUPPORT !!!!!!!!!!!!!!

and yes. Im stuck with windows.

@jhtravis

This comment has been minimized.

Copy link

jhtravis commented Feb 25, 2019

would benefit from this integration as well...

@clashofphish

This comment has been minimized.

Copy link

clashofphish commented Feb 27, 2019

Would benefit from this support. Already in Conda and have everything set up on my machine. Is a shame that I have to re-setup to use Zappa.

@jneves jneves added the needs-info label Mar 2, 2019

@jneves

This comment has been minimized.

Copy link
Collaborator

jneves commented Mar 2, 2019

Sorry, need to understand if this is still under development. If not, and I apologise for the lack of response, I'd prefer to close it. @mathieu1 would you be willing to update it? Or do you think @lsorber's workaround at #108 (comment) is a reasonable alternative?

@jneves jneves added the needs-merge label Mar 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.