New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

virtualenv goes wild when python install is symlinked #490

Closed
rofl0r opened this Issue Oct 30, 2013 · 6 comments

Comments

Projects
None yet
2 participants
@rofl0r
Copy link
Contributor

rofl0r commented Oct 30, 2013

in my setup, python is in /opt/python/{include, lib}
and symlinked into /
like this: /bin/python -> ../opt/python/bin/python

Symlinking Python bootstrap modules
  Symlinking /lib/python2.7/config -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/config
  Symlinking /lib/python2.7/lib-dynload -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/lib-dynload
  Symlinking /lib/python2.7/os.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking /lib/python2.7/posixpath.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking /lib/python2.7/ntpath.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/ntpath.py
  Symlinking /lib/python2.7/genericpath.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/genericpath.py
  Symlinking /lib/python2.7/fnmatch.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/fnmatch.py
  Symlinking /lib/python2.7/locale.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/locale.py
  Symlinking /lib/python2.7/encodings -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/encodings
  Symlinking /lib/python2.7/codecs.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/codecs.py
  Symlinking /lib/python2.7/stat.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/stat.py
  Symlinking /lib/python2.7/UserDict.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/UserDict.py
  Symlinking /lib/python2.7/copy_reg.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/copy_reg.py
  Symlinking /lib/python2.7/types.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/types.py
  Symlinking /lib/python2.7/re.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/re.py
  Symlinking /lib/python2.7/sre.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/sre.py
  Symlinking /lib/python2.7/sre_parse.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/sre_parse.py
  Symlinking /lib/python2.7/sre_constants.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/sre_constants.py
  Symlinking /lib/python2.7/sre_compile.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/sre_compile.py
  Symlinking /lib/python2.7/warnings.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/warnings.py
  Symlinking /lib/python2.7/linecache.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/linecache.py
  Symlinking /lib/python2.7/_abcoll.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/_abcoll.py
  Symlinking /lib/python2.7/abc.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/abc.py
  Symlinking /lib/python2.7/_weakrefset.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/_weakrefset.py
Creating /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/site-packages
Writing /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/site.py
Writing /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/orig-prefix.txt
Creating parent directories for /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/include
Symlinking /include/python2.7 -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/include/python2.7
Symlinking /bin/..//opt/python/lib/python2.7/mailbox.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/mailbox.py
Symlinking /bin/..//opt/python/lib/python2.7/repr.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/repr.pyo
Symlinking /bin/..//opt/python/lib/python2.7/symbol.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/symbol.py
Symlinking /bin/..//opt/python/lib/python2.7/mimify.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/mimify.pyo
Symlinking /bin/..//opt/python/lib/python2.7/ihooks.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/ihooks.pyo
Symlinking /bin/..//opt/python/lib/python2.7/macpath.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/macpath.pyo
Symlinking /bin/..//opt/python/lib/python2.7/mimetools.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/mimetools.pyo
Symlinking /bin/..//opt/python/lib/python2.7/smtplib.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/smtplib.pyo
Symlinking /bin/..//opt/python/lib/python2.7/sched.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/sched.py
Symlinking /bin/..//opt/python/lib/python2.7/zipfile.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/zipfile.pyo
Symlinking /bin/..//opt/python/lib/python2.7/user.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/user.pyo
Symlinking /bin/..//opt/python/lib/python2.7/_threading_local.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/_threading_local.pyo
Symlinking /bin/..//opt/python/lib/python2.7/xml -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/xml
Symlinking /bin/..//opt/python/lib/python2.7/SocketServer.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/SocketServer.py
Symlinking /bin/..//opt/python/lib/python2.7/robotparser.pyo -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/robotparser.pyo
Symlinking /bin/..//opt/python/lib/python2.7/re.py -> /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/re.py
Symlinking failed, copying to /src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/re.py
copy: src:/bin/..//opt/python/lib/python2.7/re.py dest:/src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/re.py

Traceback (most recent call last):
  File "/src/build/firefox/mozilla-release/python/virtualenv/virtualenv.py", line 2573, in <module>
    main()
  File "/src/build/firefox/mozilla-release/python/virtualenv/virtualenv.py", line 974, in main
    never_download=options.never_download)
  File "/src/build/firefox/mozilla-release/python/virtualenv/virtualenv.py", line 1075, in create_environment
    site_packages=site_packages, clear=clear))
  File "/src/build/firefox/mozilla-release/python/virtualenv/virtualenv.py", line 1319, in install_python
    copyfile(join(exec_dir, fn), join(lib_dir, fn))
  File "/src/build/firefox/mozilla-release/python/virtualenv/virtualenv.py", line 447, in copyfile
    copyfileordir(src, dest)
  File "/src/build/firefox/mozilla-release/python/virtualenv/virtualenv.py", line 424, in copyfileordir
    shutil.copy2(src, dest)
  File "/lib/python2.7/shutil.py", line 128, in copy2
    copyfile(src, dst)
  File "/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: '/src/build/firefox/mozilla-release/firefox-build-dir/_virtualenv/lib/python2.7/re.py'
Traceback (most recent call last):
  File "/src/build/firefox/mozilla-release/build/virtualenv/populate_virtualenv.py", line 384, in <module>
    manager.ensure()
  File "/src/build/firefox/mozilla-release/build/virtualenv/populate_virtualenv.py", line 103, in ensure
    return self.build()
  File "/src/build/firefox/mozilla-release/build/virtualenv/populate_virtualenv.py", line 315, in build
    self.create()
  File "/src/build/firefox/mozilla-release/build/virtualenv/populate_virtualenv.py", line 122, in create
    raise Exception('Error creating virtualenv.')
Exception: Error creating virtualenv.


@rofl0r

This comment has been minimized.

Copy link
Contributor Author

rofl0r commented Oct 30, 2013

/src/build/firefox/mozilla-release # la firefox-build-dir/_virtualenv/lib/
total 12
drwxr-xr-x    3 root     root          4096 Oct 30 04:01 .
drwxr-xr-x    4 root     root          4096 Oct 30 04:01 ..
drwxr-xr-x    3 root     root          4096 Oct 30 04:01 python2.7
/src/build/firefox/mozilla-release # la firefox-build-dir/_virtualenv/lib/python2.7/
total 44
drwxr-xr-x    3 root     root          4096 Oct 30 04:01 .
drwxr-xr-x    3 root     root          4096 Oct 30 04:01 ..
lrwxrwxrwx    1 root     root            41 Oct 30 04:01 SocketServer.py -> /opt/python/lib/python2.7/SocketServer.py
lrwxrwxrwx    1 root     root            43 Oct 30 04:01 UserDict.py -> ../..//opt/python/lib/python2.7/UserDict.py
lrwxrwxrwx    1 root     root            42 Oct 30 04:01 _abcoll.py -> ../..//opt/python/lib/python2.7/_abcoll.py
lrwxrwxrwx    1 root     root            46 Oct 30 04:01 _threading_local.pyo -> /opt/python/lib/python2.7/_threading_local.pyo
lrwxrwxrwx    1 root     root            46 Oct 30 04:01 _weakrefset.py -> ../..//opt/python/lib/python2.7/_weakrefset.py
lrwxrwxrwx    1 root     root            38 Oct 30 04:01 abc.py -> ../..//opt/python/lib/python2.7/abc.py
lrwxrwxrwx    1 root     root            41 Oct 30 04:01 codecs.py -> ../..//opt/python/lib/python2.7/codecs.py
lrwxrwxrwx    1 root     root            21 Oct 30 04:01 config -> /lib/python2.7/config
lrwxrwxrwx    1 root     root            43 Oct 30 04:01 copy_reg.py -> ../..//opt/python/lib/python2.7/copy_reg.py
lrwxrwxrwx    1 root     root            24 Oct 30 04:01 encodings -> /lib/python2.7/encodings
lrwxrwxrwx    1 root     root            42 Oct 30 04:01 fnmatch.py -> ../..//opt/python/lib/python2.7/fnmatch.py
lrwxrwxrwx    1 root     root            46 Oct 30 04:01 genericpath.py -> ../..//opt/python/lib/python2.7/genericpath.py
lrwxrwxrwx    1 root     root            36 Oct 30 04:01 ihooks.pyo -> /opt/python/lib/python2.7/ihooks.pyo
lrwxrwxrwx    1 root     root            26 Oct 30 04:01 lib-dynload -> /lib/python2.7/lib-dynload
lrwxrwxrwx    1 root     root            44 Oct 30 04:01 linecache.py -> ../..//opt/python/lib/python2.7/linecache.py
lrwxrwxrwx    1 root     root            41 Oct 30 04:01 locale.py -> ../..//opt/python/lib/python2.7/locale.py
lrwxrwxrwx    1 root     root            37 Oct 30 04:01 macpath.pyo -> /opt/python/lib/python2.7/macpath.pyo
lrwxrwxrwx    1 root     root            36 Oct 30 04:01 mailbox.py -> /opt/python/lib/python2.7/mailbox.py
lrwxrwxrwx    1 root     root            39 Oct 30 04:01 mimetools.pyo -> /opt/python/lib/python2.7/mimetools.pyo
lrwxrwxrwx    1 root     root            36 Oct 30 04:01 mimify.pyo -> /opt/python/lib/python2.7/mimify.pyo
lrwxrwxrwx    1 root     root            41 Oct 30 04:01 ntpath.py -> ../..//opt/python/lib/python2.7/ntpath.py
-rw-r--r--    1 root     root             1 Oct 30 04:01 orig-prefix.txt
lrwxrwxrwx    1 root     root            37 Oct 30 04:01 os.py -> ../..//opt/python/lib/python2.7/os.py
lrwxrwxrwx    1 root     root            44 Oct 30 04:01 posixpath.py -> ../..//opt/python/lib/python2.7/posixpath.py
lrwxrwxrwx    1 root     root            37 Oct 30 04:01 re.py -> ../..//opt/python/lib/python2.7/re.py
lrwxrwxrwx    1 root     root            34 Oct 30 04:01 repr.pyo -> /opt/python/lib/python2.7/repr.pyo
lrwxrwxrwx    1 root     root            41 Oct 30 04:01 robotparser.pyo -> /opt/python/lib/python2.7/robotparser.pyo
lrwxrwxrwx    1 root     root            34 Oct 30 04:01 sched.py -> /opt/python/lib/python2.7/sched.py
drwxr-xr-x    2 root     root          4096 Oct 30 04:01 site-packages
-rw-r--r--    1 root     root         27321 Oct 30 04:01 site.py
lrwxrwxrwx    1 root     root            37 Oct 30 04:01 smtplib.pyo -> /opt/python/lib/python2.7/smtplib.pyo
lrwxrwxrwx    1 root     root            38 Oct 30 04:01 sre.py -> ../..//opt/python/lib/python2.7/sre.py
lrwxrwxrwx    1 root     root            46 Oct 30 04:01 sre_compile.py -> ../..//opt/python/lib/python2.7/sre_compile

etc.
all files which are a symlink to ../../opt.... are messed up. the symlink target does not exist.

@rofl0r

This comment has been minimized.

Copy link
Contributor Author

rofl0r commented Oct 30, 2013

this fixes it:

--- org.py      2013-10-30 04:29:36.049913046 +0000
+++ python/virtualenv/virtualenv.py     2013-10-30 04:33:10.797897289 +0000
@@ -438,6 +438,9 @@
         srcpath = os.path.abspath(src)
     else:
         srcpath = os.readlink(src)
+        if not srcpath[0] == '/':
+               # relative symlink, needs to be prefixed with absolute source path
+               srcpath = os.path.join(os.path.dirname(src), srcpath)
     if symlink and hasattr(os, 'symlink') and not is_win:
         logger.info('Symlinking %s :: %s -> %s', src, srcpath,  dest)
         try:
@rofl0r

This comment has been minimized.

Copy link
Contributor Author

rofl0r commented Nov 2, 2013

full patch is available here https://github.com/sabotage-linux/sabotage/blob/master/KEEP/firefox-virtualenv.patch , it fixes several other issues with virtual-env symlink handling

for starters it's a bug to symlink subdirs. this caused the distutils dir to be symlinked, pointing to the installation directory, so overwriting __init__.py with your own version caused the real one to be overwritten.

@stale

This comment has been minimized.

Copy link

stale bot commented Jan 15, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

@stale stale bot added the wontfix label Jan 15, 2019

@rofl0r

This comment has been minimized.

Copy link
Contributor Author

rofl0r commented Jan 15, 2019

it's only stale because nobody here even bothered to look at it

@gaborbernat

This comment has been minimized.

Copy link
Contributor

gaborbernat commented Jan 15, 2019

Care to create a PR to fix it?

@stale stale bot removed the wontfix label Jan 15, 2019

rofl0r added a commit to rofl0r/virtualenv that referenced this issue Feb 6, 2019

rofl0r added a commit to rofl0r/virtualenv that referenced this issue Feb 7, 2019

fix handling of relative symlinks
this fixes the scenario where the python base install is symlinked
with relative symlinks.

e.g. /bin/python -> ../opt/python2/bin/python

closes pypa#490.

clrpackages pushed a commit to clearlinux-pkgs/virtualenv that referenced this issue Feb 15, 2019

virtualenv: Autospec creation for update from version 16.3.0 to versi…
…on 16.4.0

Anthony Sottile (2):
      Use importlib in python3 (#1293)
      virtualenv does not *always* have a runtime dependency on setuptools (#1300)

Bernat Gabor (2):
      release 16.4.0
      release 16.4.0

Bernát Gábor (2):
      Update virtualenv.py
      upgrade setuptools and pip (#1312)

daa (2):
      Do not check pip config if such command is not available (#1303)
      Link or copy PyPy headers instead of include directory as a whole (#1302)

rofl0r (1):
      fix handling of relative symlinks (#1309)

v16.4.0 (2019-02-09)
--------------------

Bugfixes
^^^^^^^^

- fixes the scenario where the python base install is symlinked with relative symlinks (`#490 <https://github.com/pypa/virtualenv/issues/490>`_)
- Use ``importlib`` over ``imp`` in ``virtualenv.py`` for ``python >= 3.4`` - by Anthony Sottile (`#1293 <https://github.com/pypa/virtualenv/issues/1293>`_)
- Copy or link PyPy header files instead of include directory itself (`#1302 <https://github.com/pypa/virtualenv/issues/1302>`_)
- Allow virtualenv creation with older pip not having ``config`` command
  correspondingly disabling configuration related features (such as pip cert
  setting) in this case. (`#1303 <https://github.com/pypa/virtualenv/issues/1303>`_)

Features

(NEWS truncated at 15 lines)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment