Package Zappa app dependencies for an alternate virtualenv

[nickovs]

Context

Currently Zappa packages up pretty much everything in your current virtualenv when you package, deploy or update your application. This often results in the inclusion of a large amount of content that is only used for development and local testing which will never be needed when the code is running on AWS, for instance copies of pylint or black, or libraries that are being used in other projects. While it is possible to exclude a list of individual packages, this is cumbersome when the excluded packages have large or deep dependency trees.

In order to deploy only the necessary package, an alternate virtual environment can be created. Unfortunately this is currently very cumbersome, since it involves the user having to disable their current venv, activate the alternate one, issue the deploy/update command, deactivate the alternate venv and reactivate their development environment.

In the light of this it would be helpful to be able to specify an alternate virtualenv to be packed for use on Lambda, that is distinct from the one used for development and testing in which the Zappa command is executed.

Expected Behavior

The user should be able to provide the path of an alternate virtualenv as a command line argument.

Actual Behavior

Currently to use an alternate virtualenv the user must deactivate their dev venv, activate their production venv, run the deploy/update process, deactivate their production venv and reactivate their dev venv.

Possible Fix

The core Zappa class's create_lambda_zip() method already supports having an alternate venv path being passed to it. Exposing this method's argument via a command line option would allow the user to select their production venv while deploying from their development environment.

[tommie-lie]

I don't think having a second virtualenv (which has to be maintained separately) is very developer-friendly: poetry won't create and maintain multiple venvs and updating multiple venvs manually is always error-prone (do you remember to update package X in your production environment when you just added it to your dev environment?)

But I found pip-check-reqs which seems like it can alleviate most of the problems described here. black and pylint should not appear in any imports and you can exclude your tests directory, so that you can even exclude packages that are only required for testing (like some pytest packages, mocking libraries, etc.).
Including such a mechanism to auto-detect packages that are really required from the current environment to the lambda bundle would be really nice.

[nickovs]

@tommie-lie If Zappa can add a reliable way to auto-detect which packages are really needed and only package those then that would certainly be better than having to maintain a separate environment.

Another alternative would be to add an option to Zappa to supply a requirements.txt file and have it only package the listed requirements. While this would still require manual maintenance, maintaining requirements.txt files is well understood by most developers (and can be automated with tools like pip-check-reqs) and going that route would avoid adding another complex dependency to Zappa.

[dickermoshe]

Completely agree.
The auto-detecting of needed packages is a feature no-one asked for.
It would be developer friendly to have a requirements.txt andrequirements-dev.txt file.

However being that using Zappa within a Docker container is recommended anyway, you can create a separate virualenv in that Container.
Also see zappadock for easily using a Docker Container for Development with Zappa

[monkut]

I've experienced this as well, where I've built out a project with testing and development packages which resulted in a package too big for lambda. Resulting in me re-creating the environment without test/dev tools and re-packaging.

Unfortunately the python package management landscape is vast and there is not a single tool used, making a feature like this difficult to implement fully.

In addition, it would be ideal to package in the lambda environment to reduce issues where the wrong package/wheel is used.

So in my opinion it would be best to move to a docker-based packaging approach.
You would still have the issue of how to define the dev vs prodction environments though.

[github-actions]

Hi there! Unfortunately, this Issue has not seen any activity for at least 90 days. If the Issue is still relevant to the latest version of Zappa, please comment within the next 10 days if you wish to keep it open. Otherwise, it will be automatically closed.

[github-actions]

Hi there! Unfortunately, this Issue was automatically closed as it had not seen any activity in at least 100 days. If the Issue is still relevant to the latest version of Zappa, please open a new Issue.