running a symlink results in an inaccurate sys.path and module imports fail

[nickjj]

Hi,

I'm running Ubuntu 18.04.3 with Python 3.6.9 (the default version on this distro).

Here's as much info as I have at the moment:

1. I created a virtualenv using 20.0.3
2. I ran pip install docker inside of this venv which is located at /usr/local/lib/docker/virtualenv
3. I created a symlink lrwxrwxrwx 1 root root 43 Feb 12 18:25 python-docker -> /usr/local/lib/docker/virtualenv/bin/python

The symlink was created with absolute paths.

If I run python-docker, I get thrown into a Python prompt and running import docker fails.

If I run /usr/local/lib/docker/virtualenv/bin/python, I get thrown into a Python prompt and running import docker is successful. The above path is the symlink destination, I literally copy / pasted it.

Here's the sys.path output which is different between them:

# This is where `import docker` fails using the symlink:
root@ubuntu-s-1vcpu-1gb-nyc3-01:/usr/local/lib/docker/virtualenv/bin# python-docker
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']


# This is where `import docker` works using the absolute path:
root@ubuntu-s-1vcpu-1gb-nyc3-01:/usr/local/lib/docker/virtualenv/bin# /usr/local/lib/docker/virtualenv/bin/python
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/docker/virtualenv/lib/python3.6/site-packages']

In the working case, we can see that it references the virtualenv where as the symlink does not.

If you perform the same exact work flow using virtualenv 16.3.0 then both the symlink and absolute path works and they have an identical sys.path of ['', '/usr/local/lib/docker/virtualenv/lib/python36.zip', '/usr/local/lib/docker/virtualenv/lib/python3.6', '/usr/local/lib/docker/virtualenv/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/usr/local/lib/docker/virtualenv/lib/python3.6/site-packages'].

So it sounds like maybe something between 16.3.0 and 20.0.3 is busted when it comes to setting up the system path between symlinks and absolute path binaries.

Thanks to @altendky on IRC for thinking to look at virtualenv's version.

[altendky]

Looks to be introduced at v20.

 ~/1599   master ✚  venv/bin/pip install 'virtualenv<20'; rm -rf testenv p; venv/bin/virtualenv -p python3 testenv; ln -s testenv/bin/python p; testenv/bin/python -c 'import sys; print(sys.path)'; ./p -c 'import sys; print(sys.path)';
Collecting virtualenv<20
  Using cached virtualenv-16.7.9-py2.py3-none-any.whl (3.4 MB)
Installing collected packages: virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 20.0.3
    Uninstalling virtualenv-20.0.3:
      Successfully uninstalled virtualenv-20.0.3
Successfully installed virtualenv-16.7.9
Running virtualenv with interpreter /home/altendky/.pyenv/shims/python3
Already using interpreter /home/altendky/.pyenv/versions/3.8.1/bin/python3
Using base prefix '/home/altendky/.pyenv/versions/3.8.1'
New python executable in /home/altendky/1599/testenv/bin/python3
Also creating executable in /home/altendky/1599/testenv/bin/python
Installing setuptools, pip, wheel...
done.
['', '/home/altendky/1599/testenv/lib/python38.zip', '/home/altendky/1599/testenv/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
['', '/home/altendky/1599/testenv/lib/python38.zip', '/home/altendky/1599/testenv/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
 ~/1599   master ✚  venv/bin/pip install 'virtualenv==20'; rm -rf testenv p; venv/bin/virtualenv -p python3 testenv; ln -s testenv/bin/python p; testenv/bin/python -c 'import sys; print(sys.path)'; ./p -c 'import sys; print(sys.path)';
Collecting virtualenv==20
  Downloading virtualenv-20.0.0-py2.py3-none-any.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 1.2 MB/s 
Requirement already satisfied: six<2,>=1.12.0 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (1.14.0)
Requirement already satisfied: filelock<4,>=3.0.0 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (3.0.12)
Requirement already satisfied: appdirs<2,>=1.4.3 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (1.4.3)
Installing collected packages: virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 16.7.9
    Uninstalling virtualenv-16.7.9:
      Successfully uninstalled virtualenv-16.7.9
Successfully installed virtualenv-20.0.0
['', '/home/altendky/.pyenv/versions/3.8.1/lib/python38.zip', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
['', '/home/altendky/.pyenv/versions/3.8.1/lib/python38.zip', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/site-packages']

[altendky]

For at least a little more completeness on my check...

 ~   master ✚  python3 --version --version
Python 3.8.1 (default, Dec 27 2019, 18:45:57) 
[GCC 9.2.1 20191008]
 ~   master ✚  type python3
python3 is /home/altendky/.pyenv/shims/python3

[gaborbernat]

This is an interesting issue. Note though from my test this behaviour is the same if you would be using venv instead of virtualenv; so the issue is probably upstream for CPython and as such I'd recommend opening an issue under https://bugs.python.org/. With virtualenv 20 our virtual environment creation now matches venv's method, hence why this now is discovered.

[gaborbernat]

So reading through https://www.python.org/dev/peps/pep-0405/#specification and thinking this through this kind of behaviour is no longer supported by design.

The python executable is no longer a copy of the host python (as was with the old virtualenv), but instead we just symlink it and we use the pyenv.cfg as marker for the virtual environment prefix. If you symlink a symlink with python 3+ you effectively:

This PEP proposes to add a new first step to this search. If a pyvenv.cfg file is found either adjacent to the Python executable or one directory above it (if the executable is a symlink, it is not dereferenced), this file is scanned for lines of the form key = value. If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment.

• if the pyenv.cfg is not discovered you resolve back to the system python (this is what happens in your case here),
• if the pyenv.cfg would be found (e.g. you would symlink it alongside the python executable symlink), the home prefix would be sent where the symlink lives; so you would need to also symlink the lib/include folder to pull in the site-package for that.

TLDR. Symlinking virtual environment python executable is not supported, no longer works on Python 3.4 and later. If this is something you would like to do, consider instead creating a proxy script; e.g. with bash:

printf '#!/bin/bash\n/usr/local/lib/docker/virtualenv "$@"' 1>python-docker
chmod u+x python-docker
./python-docker -m site # this now will work

[nickjj]

TLDR. Symlinking virtual environment python executable is not supported, no longer works on Python 3.4 and later.

On my host, I'm using 3.6 so it seems at the Python level it's still supported.

[gaborbernat]

@nickjj not sure I follow, what do you mean at python level?

[nickjj]

In your TL;DR it sounded like you were saying Python itself doesn't support symlinking virtualenvs as of Python 3.4+, but as long as you stick to an older version of virtualenv / venv, symlinking virtualenvs works with Python 3.6.x and 3.8.x. So I was wondering if this is really a Python 3.4 issue, or a virtualenv / venv issue.

[gaborbernat]

Let me rephrase:

• venv (part of Python3.4 or later) never supported symlinking the python path,
• virtualenv 20 or later follow the venv standard, and does not support symlinking (and cannot),
• virtualenv <20 used a different mechanism to generate virtual environments that as a side effect supported symlinking; note virtualenv<20 is no longer maintained, but your use case still work with it.

I don't think it's a virtualenv/venv issue per se, it's a design decision that does not allows/supports such use case, and cannot by design. I don't think this use case was also officially supported with virtualenv <20, but worked as a side-effect of how the virtual environment creation happened.

PS. If you really want the symlink you can pin your virtualenv to <20; but I'd instead suggest using the shim script above instead.

[nickjj]

Thanks.

What's interesting is Ubuntu 20.04 (coming out in a few months) and the next stable release of Buster that's still years away both continue to have virtualenv 15.1.x in their repos. So it sounds like as long as you apt install virtualenv, you'll be good to go for a few years at least.

[gaborbernat]

That might be true as long as you stick to apt-get things... but as soon as you start using pip you might be back here. I assume to run into this issue you've also pip installed virtualenv 20; so 🤷‍♂ not sure how apt-get using old virtualenv relates to this issue.