Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: pipenv bundle command #986

Closed
nicholasbishop opened this issue Oct 27, 2017 · 11 comments
Closed

Feature request: pipenv bundle command #986

nicholasbishop opened this issue Oct 27, 2017 · 11 comments

Comments

@nicholasbishop
Copy link

@nicholasbishop nicholasbishop commented Oct 27, 2017

(Hope it's OK to file a feature request here, didn't see any explicit documentation regarding that.)

tl;dr: I'd like to use pipenv to generate a bundle suitable for AWS Lambda

For reference, here is the AWS Lambda documentation for how to create a bundle with pip:
http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#deployment-pkg-for-virtualenv

There are a couple existing pipenv issues that touch on this use case:

But neither of those directly requests this feature, so I figured it might be helpful to do so.

@erinxocon
Copy link
Contributor

@erinxocon erinxocon commented Oct 29, 2017

Hi @nicholasbishop! Feature requests are certainly welcome. I don't think this is in the scope of pipenv. Those directions essentially just tell you to copy your site packages folder along with your code. Pip does nothing but install the code to the site packages folder. You can get the location of your virutalenv by doing (on unix) open (pipenv --venv) and then following those directions regarding copying the site package folder.

@Zebradil
Copy link

@Zebradil Zebradil commented Jan 8, 2018

I would vote for this feature. It is strange to have «check style» functionality and not to have some of the useful pip's functionality. I tried to migrate one of my AWS Lambda project to using pipenv, but it turned out that it's easier to use original venv functionality + pip + requirements.txt. Just because I can't specify exact directory of package installation with pipenv.

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Jan 8, 2018

@Zebradil If you mean pipenv check --style, that functionality is planned to be removed.

@joestump
Copy link

@joestump joestump commented May 7, 2018

I found this example on GitHub that looks sensible for packaging Lambdas with pipenv. You could move those to a bundle "script" in the Pipfile and run pipenv run bundle.

@ashemedai
Copy link

@ashemedai ashemedai commented Jun 5, 2018

@joestump Unfortunately that's not sensible (unless you work in an environment where there's a large homogeneity of systems). I've been through similar hoops in the last few days. You might encounter both lib/python3.6/site-packages and lib64/python3.6/site-packages that you need to add to the zip command. Then you also need to start adding checks to see whether or not the directories exist.

In the end the easiest remains, as @Zebradil pointed out:

build:
	mkdir -p $(BUILDDIR)
	$(pipenv) lock -r > $(BUILDDIR)/requirements.txt
	cp -R $(SRCDIR)/* $(BUILDDIR)
	$(pipenv) run pip install --isolated --disable-pip-version-check -r $(BUILDDIR)/requirements.txt -t $(BUILDDIR) -U

That way you can just point the CodeUri to build and use aws cloudformation package as it should.

This makes it rather platform-agnostic (had to get this working on a Linux, macOS, and Windows environment). Having to step out of pipenv with a run pip install feels weird. And pipenv --venv as @erinxocon mentioned does not help in any of these scenarios.

@ashemedai
Copy link

@ashemedai ashemedai commented Jun 5, 2018

In addition, I do need to check that script section of Pipfile. That might actually allow me to keep some stuff local to Pipfile without having to bloat the Makefile with Python-specific things and allows us, where I work, to keep using our more standardised Lambda build pipeline (which supports C#, Go, Node.js, and Python).

@Zebradil
Copy link

@Zebradil Zebradil commented Jun 5, 2018

At the end I got to the following command:

$ pipenv run pip install -r <(pipenv lock -r) --target _build/

Works with bash, doesn't work with pure shell.
But I see some sense in preserving requirements.txt in the bundle. Just as metadata of the package.

@ashemedai
Copy link

@ashemedai ashemedai commented Jun 8, 2018

The problem was that if I want to try the redirection soltion in a Makefile the result became unreadable fast. And like you said, I figured that having the requirements.txt part of the zip in S3 would be an added benefit since, currently, we do not really do any release management on these functions. We always install the latest ones from the repository.

@ghost
Copy link

@ghost ghost commented Sep 20, 2018

The problem was that if I want to try the redirection soltion in a Makefile the result became unreadable fast. And like you said, I figured that having the requirements.txt part of the zip in S3 would be an added benefit since, currently, we do not really do any release management on these functions. We always install the latest ones from the repository.

The easiest way around this is stuffing the requirements into a temp file. See below

$(eval TEMPFILE = $(shell mktemp))
pipenv lock -r > ${TEMPFILE}
pipenv run pip install -r ${TEMPFILE} --target _build/
rm -f ${TEMPFILE}
@pecigonzalo
Copy link

@pecigonzalo pecigonzalo commented Oct 9, 2019

@erinxocon could we at least make pipenv --site-packages or something return the path?

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Oct 18, 2019

That sounds like a reasonable idea. Would you mind open an issue (and maybe create a PEEP proposal) for it? --site-packages is probably not enough; but something like --venv-path=[key] to return sysconfig.get_path(key) would have everything you need for packaging (and maybe support :all-json: and :all-list: etc. to return get_paths() with a certain format.

Meanwhile one of the following should return what you need:

pipenv run python -c "import json, sysconfig; print(json.dumps(sysconfig.get_path('purelib')))"
pipenv run python -c "import json, sysconfig; print(json.dumps(sysconfig.get_path('platlib')))"

Refer to sysconfig documentation on more keys to use (and the difference between purelib and platlib, although I believe these two should be identical in most virtual environment contexts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.