produce a virtualenv zipapp #1287

asottile · 2019-01-20T06:25:36Z

this "works" -- I would like some feedback on the approach however:

$ python3 tasks/make_zipapp.py 
zipapp created at /tmp/x/virtualenv/virtualenv.pyz
$ python virtualenv.pyz venv
New python executable in /tmp/x/virtualenv/venv/bin/python
Installing setuptools, pip, wheel...
done.
$ python3 virtualenv.pyz venv3
Using base prefix '/usr'
New python executable in /tmp/x/virtualenv/venv3/bin/python3
Also creating executable in /tmp/x/virtualenv/venv3/bin/python
Installing setuptools, pip, wheel...
done.
$ du -hs virtualenv.pyz 
2.0M	virtualenv.pyz

pfmoore · 2019-01-20T11:45:50Z

I like this approach a lot. Distributing virtualenv as a standalone pyz file that can be run with any Python interpreter sounds like a much better approach. From what I can tell, the pyz file will still support the -p option, even though it's not particularly needed for a standalone pyz.

There are a lot of projects that depend on virtualenv that likely expect it to be installed (tox, pew, pipenv, ...) - otherwise, I'd be in favour of making this the official distribution method for virtualenv. It may be worth reaching out to those projects to see how hard it would be for them to switch from needing an installed virtualenv, to simply needing a virtualenv command available.

gaborbernat · 2019-01-20T12:43:05Z

The main impediment Paul for tox e.g. is that while one can guarantee virtualenv exist via install requires, one cannot do the same for pyz. This method would only work if downstream tools themselves would be pyz, however one downside of doing so is that no direct command to the tool is available, but rather someone needs to pass the pyz to the interpreter. Therefore I would say not viable. asottile please add some tests for this so we can validate it works for all flavours of python.

pfmoore · 2019-01-20T13:09:42Z

@gaborbernat Thanks for the clarification on tox's position. I thought something like that would be the issue. I've done some work bundling various tools, including tox and virtualenv, as zipapps using shiv. It needs a little fiddling to ensure that the unbundled zipapp is on sys.path for subprocess calls (by setting PYTHONPATH, which causes tox to issue a warning). I'd wondered whether there was a way (probably involving changes to tox and other such tools) to work around that, but I've not really had the time to look into it properly. It's very much a niche use case, but it's a pretty useful one (having Python dev tools available standalone as pyz apps is really convenient for certain types of workflow).

I'd still like to see the pyz form as a supported approach, even if only as an additional option.

gaborbernat · 2019-01-20T13:23:00Z

Yeah, 😁 agreed. later on tox could look into pyz/shiva/pex as well. Some people requested and we accepted to distribute it e.g. via docker too.

asottile · 2019-01-20T17:40:26Z

I've written some tests, re-enabled install_wheel to run with search_dirs=None, and fixed -p -- the diff is probably best viewed with ?w=1

gaborbernat

I like the approach. The tests need some tweaking though, documentation extended, and I feel like we should avoid unwrapping the support files at every invocation and instead have a cache dir.

tasks/make_zipapp.py

docs/changelog/1092.feature.rst

tasks/make_zipapp.py

gaborbernat · 2019-01-21T11:02:44Z

virtualenv.py

-        # Probably some boot script; just in case virtualenv is installed...
+@contextlib.contextmanager
+def virtualenv_support_dirs():
+    """Context manager yielding either [virtualenv_support_dir] or []"""


maybe more accurate to say: locate virtualenv support files directory

virtualenv.py

asottile · 2019-01-21T18:47:40Z

coincidentally, this branch also fixes:

PYTHONPATH=virtualenv-16.2.0-py2.py3-none-any.whl python -m virtualenv venv

tests/test_zipapp.py

tox.ini

gaborbernat · 2019-01-21T19:18:26Z

virtualenv.py

-        from urllib.parse import urljoin
-        from urllib.request import pathname2url
+    with search_dirs_context() as search_dirs:
+        wheels = find_wheels(["setuptools", "pip"], search_dirs)


this generates a long diff because of the indent, what if we would just extract the indented part into its own function?

This is why I suggested viewing the diff with ?w=1 in this comment

Granted that works as a bypass, though would we move this into a sub-function we could avoid changing the lines altogether, not?

they'd still change because I'd have to move them -- plus it would increase the api surface of virtualenv (as far as I can tell this function isn't a documented interface but I've found a bunch of calls to it on github >.<)

Not if you move it to a private function, right after this function. The private function call would be an insert and everything below would be all is same.

I'd rather not add lines for no reason, what's the problem with the indent?

gaborbernat

let's keep this open then until I get to polish it myself

gaborbernat

Now only if we can fix #1291 👍

gaborbernat · 2019-01-25T15:42:41Z

Released as 16.3.0.

asottile changed the title ~~WIP: produce a virtualenv zipapp~~ produce a virtualenv zipapp Jan 20, 2019

gaborbernat requested changes Jan 21, 2019

View reviewed changes

gaborbernat reviewed Jan 21, 2019

View reviewed changes

tests/test_zipapp.py Show resolved Hide resolved

gaborbernat reviewed Jan 21, 2019

View reviewed changes

tests/test_zipapp.py Outdated Show resolved Hide resolved

gaborbernat reviewed Jan 21, 2019

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

gaborbernat reviewed Jan 21, 2019

View reviewed changes

gaborbernat requested changes Jan 21, 2019

View reviewed changes

gaborbernat approved these changes Jan 25, 2019

View reviewed changes

gaborbernat mentioned this pull request Jan 25, 2019

Use importlib over imp in python3.x #1289

Merged

asottile and others added 4 commits January 25, 2019 13:48

produce a virtualenv zipapp

f53ada3

avoid extra indentation for wheel install

fc79cae

assert exe different

32781c2

windows fix

1de9da3

gaborbernat merged commit cfd9c81 into pypa:master Jan 25, 2019

asottile deleted the zipapp branch January 29, 2019 03:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

produce a virtualenv zipapp #1287

produce a virtualenv zipapp #1287

asottile commented Jan 20, 2019

pfmoore commented Jan 20, 2019

gaborbernat commented Jan 20, 2019

pfmoore commented Jan 20, 2019

gaborbernat commented Jan 20, 2019

asottile commented Jan 20, 2019

gaborbernat left a comment

gaborbernat Jan 21, 2019

asottile commented Jan 21, 2019

gaborbernat Jan 21, 2019

asottile Jan 21, 2019

gaborbernat Jan 21, 2019

asottile Jan 21, 2019

gaborbernat Jan 21, 2019

asottile Jan 21, 2019

gaborbernat left a comment

gaborbernat left a comment

gaborbernat commented Jan 25, 2019

produce a virtualenv zipapp #1287

produce a virtualenv zipapp #1287

Conversation

asottile commented Jan 20, 2019

pfmoore commented Jan 20, 2019

gaborbernat commented Jan 20, 2019

pfmoore commented Jan 20, 2019

gaborbernat commented Jan 20, 2019

asottile commented Jan 20, 2019

gaborbernat left a comment

Choose a reason for hiding this comment

gaborbernat Jan 21, 2019

Choose a reason for hiding this comment

asottile commented Jan 21, 2019

gaborbernat Jan 21, 2019

Choose a reason for hiding this comment

asottile Jan 21, 2019

Choose a reason for hiding this comment

gaborbernat Jan 21, 2019

Choose a reason for hiding this comment

asottile Jan 21, 2019

Choose a reason for hiding this comment

gaborbernat Jan 21, 2019

Choose a reason for hiding this comment

asottile Jan 21, 2019

Choose a reason for hiding this comment

gaborbernat left a comment

Choose a reason for hiding this comment

gaborbernat left a comment

Choose a reason for hiding this comment

gaborbernat commented Jan 25, 2019