virtualenv cannot handle pythonXY.zip? (entire stdlib in zipfile)

[anthonyrisinger]

our base python install zips the entire stdlib (except config, site-packages, and lib-dynload) for space and performance (84MiB => 28MiB, similar load speed gains)...

...however, virtualenv doesn't seem to like this at all:

# virtualenv --distribute test
Traceback (most recent call last):
  File "/home/arisinger/.local/bin/virtualenv", line 9, in <module>
    load_entry_point('virtualenv==1.9.1', 'console_scripts', 'virtualenv')()
  File "/home/arisinger/.local/lib/python2.7/site-packages/virtualenv.py", line 979, in main
    no_pip=options.no_pip)
  File "/home/arisinger/.local/lib/python2.7/site-packages/virtualenv.py", line 1081, in create_environment
    site_packages=site_packages, clear=clear))
  File "/home/arisinger/.local/lib/python2.7/site-packages/virtualenv.py", line 1289, in install_python
    writefile(site_filename_dst, SITE_PY)
  File "/home/arisinger/.local/lib/python2.7/site-packages/virtualenv.py", line 445, in writefile
    f = open(dest, 'wb')
IOError: [Errno 2] No such file or directory: 'test/lib/python27.zip/site.py'

next, i tried unzipping only site., which didn't work, because python27.zip is _PREFERRED. so, i removed site._ from python27.zip and tried again:

# virtualenv --distribute test
New python executable in test/bin/python2.7
Also creating executable in test/bin/python
Traceback (most recent call last):
  File "/home/arisinger/test/lib/python2.7/site.py", line 67, in <module>
    import os
  File "/home/arisinger/test/lib/python2.7/os.py", line 49, in <module>
    import posixpath as path
ImportError: No module named posixpath
ERROR: The executable test/bin/python2.7 is not functioning
ERROR: It thinks sys.prefix is u'/home/arisinger' (should be u'/home/arisinger/test')
ERROR: virtualenv is not compatible with this system or executable

...no dice? i tried with[out] --system-site-packages and same result. zipping the stdlib is probably not common, but it's an explicitly known and expected use case... how can i make this work?

[pfmoore]

My first thought is that I can't imagine this ever working (or being supported) for virtualenv. The codebase is built entirely around copying files from the base install, and short using of a "virtual filesystem" style of library, ewe're not going to be able to write the code so that it transparently handles extracting files from zipfiles the same way.

The best I can think of is that we should put a note in the documentation saying that we explicitly don't support this configuration. I know that's no help to you, sorry :-(

[anthonyrisinger]

no worries... i actually think i got it working rather "painlessly"... BEHOLD! :)

TL;DR

[me@you zipvenv]$ tree -FL 4 --charset ascii
.
|-- bin/
|   |-- activate
|   |-- easy_install*
|   |-- easy_install-2.7*
|   |-- pip*
|   |-- pip-2.7*
|   |-- python -> python2.7*
|   `-- python2.7*
|-- include/
|   `-- python2.7 -> /home/anthony/.local/include/python2.7/
`-- lib/
    |-- python2.7/
    |   |-- config -> /home/anthony/.local/lib/python2.7/config/
    |   |-- distutils/
    |   |   |-- distutils.cfg
    |   |   |-- __init__.py
    |   |   `-- __init__.pyc
    |   |-- lib-dynload -> /home/anthony/.local/lib/python2.7/lib-dynload/
    |   |-- no-global-site-packages.txt
    |   |-- orig-prefix.txt
    |   |-- os.py -> /home/anthony/.local/lib/python2.7/os.py
    |   |-- site-packages/
    |   |   |-- distribute-0.6.36-py2.7.egg/
    |   |   |-- easy-install.pth
    |   |   |-- pip/
    |   |   |-- pip-1.3.1-py2.7.egg-info/
    |   |   |-- setuptools-0.6c11-py2.7.egg-info
    |   |   `-- setuptools.pth
    |   |-- site.py
    |   `-- site.pyc
    `-- python27.zip -> /home/anthony/.local/lib/python27.zip

SPECIFICS

...from within sys.prefix/python2.7:

unzip ../python27.zip 'site.*' 'opcode.*' 'distutils**'
zip -d ../python27.zip 'site.*' 'opcode.*' 'distutils**'

...which extracts, then removes, specific modules from the zipfile that virtualenv uses to get it's bearings... unfortunately there does not appear to be a way around removing stuff from the zipfile because it holds higher precedence than the loose files (for good reason)... but sys.meta_path or something could fix this (on that note, it seems like everything virtualenv does could be implemented via sys.meta_path (patching distutils, preemptive-whatever)???)

....that's pretty much it, but to be pedantic, i also symlinked config/lib-dynload/os.py/python27.zip/include, copied python2.7, copied activate(changed path)/site.py/distutils from another venv, created orig-prefix.txt and no-global-site-packages.txt, and finally... installed distribute + pip (distribute_setup.py + get-pip.py) after sourcing the activate script.

...at this point sys.*prefix properly referenced the venv, and as further proof, i successfully pip install lxml into the correct place! weee!

not sure if you guys think it's too much, but tbh, i started with a working venv, and tweaked only a small number of things to make it work properly. is there anything else needed to confirm proper function? from within the venv, i get this:

[me@you zipvenv]$ python <<EOF
import sys
from pprint import pprint as pp
pp(sys.prefix)
pp(sys.exec_prefix)
pp(sys.executable)
pp(sys.path)
EOF

'/home/anthony/zipvenv'
'/home/anthony/zipvenv'
'/home/anthony/zipvenv/bin/python'
['',
 '/home/anthony/zipvenv/lib/python2.7/site-packages/distribute-0.6.36-py2.7.egg',
 '/home/anthony/zipvenv/lib/python27.zip',
 '/home/anthony/zipvenv/lib/python2.7',
 '/home/anthony/zipvenv/lib/python2.7/plat-linux2',
 '/home/anthony/zipvenv/lib/python2.7/lib-tk',
 '/home/anthony/zipvenv/lib/python2.7/lib-old',
 '/home/anthony/zipvenv/lib/python2.7/lib-dynload',
 '/home/anthony/.local/lib/python2.7',
 '/home/anthony/zipvenv/lib/python2.7/site-packages',
 '/home/anthony/zipvenv/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info']

NOTES

• python ITSELF looks for a loose os.py as a "marker" signaling the presence of the stdlib... IMO this is a bug in python, since it just complains, then immediately finds it in python27.zip...
• ... virtualenv also uses os.__file__ for some reason in site.py instead of just __file__, but it's only to set builtins.license?
• main issue is that the zip file is preferred to loose files... if that can be fixed by virtualenv site.py, python27.zip could be used unmodifed

[pfmoore]

Thanks for the analysis. I don't know if we can fix the zip vs loose file priority issue, but I'll see if I can get some time to look into this (no promises, I'm afraid, but of course "patches welcome" :-))

This might be good to write up as a cookbook-style entry for the virtualenv docs. Again, I'll try to get to that, but it'd need to cover Python 3 as well, to be complete, which means I'll need to do some testing.

Leaving the report open as a reminder to myself (or anyone else who's interested!) that this could do with writing up. Many thanks for working this through and reporting back.

[anthonyrisinger]

cool, np... in related news, i just noticed a STANDARD python install DOES NOT favor the zipfile (which, i guess, makes sense... flexibility at some [probably negligible] import-time expense):

#... `/usr/bin/python2.7 -m site` is not working for me...?
[me@you zipvenv]$ /usr/bin/python2.7 /usr/lib/python2.7/site.py                              
sys.path = [
    '/usr/lib/python2.7',
    '/usr/lib/python27.zip',
    '/usr/lib/python2.7/plat-linux2',
    '/usr/lib/python2.7/lib-tk',
    '/usr/lib/python2.7/lib-old',
    '/usr/lib/python2.7/lib-dynload',
    '/usr/lib/python2.7/site-packages',
    '/usr/lib/python2.7/site-packages/gst-0.10',
    '/usr/lib/python2.7/site-packages/gtk-2.0',
    '/usr/lib/python2.7/site-packages/setuptools-0.6c11.egg-info',
    '/usr/lib/python2.7/site-packages/wx-2.8-gtk2-unicode',
]
USER_BASE: '/home/anthony/.local' (doesn't exist)
USER_SITE: '/home/anthony/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

...thus is must be virtualenv's site.py mucking that part up? haven't checked it out, but seems likely.

[gst]

FYI: I'm also facing the situation with python embedded within softwares like maya, nuke, etc..

they effectively bundle a python where python27 directory is zipped the way as @anthonyrisinger reports.

it doesn't seem there had progress on this issue since then ?

may I eventually help ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

[blurredart]

This issue still exists. Please help

[gaborbernat]

This requires a new implementation, which may be possible after the rewrite.