-
-
Notifications
You must be signed in to change notification settings - Fork 1k
-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
running a symlink results in an inaccurate sys.path and module imports fail #1599
Comments
Looks to be introduced at v20. ~/1599 master ✚ venv/bin/pip install 'virtualenv<20'; rm -rf testenv p; venv/bin/virtualenv -p python3 testenv; ln -s testenv/bin/python p; testenv/bin/python -c 'import sys; print(sys.path)'; ./p -c 'import sys; print(sys.path)';
Collecting virtualenv<20
Using cached virtualenv-16.7.9-py2.py3-none-any.whl (3.4 MB)
Installing collected packages: virtualenv
Attempting uninstall: virtualenv
Found existing installation: virtualenv 20.0.3
Uninstalling virtualenv-20.0.3:
Successfully uninstalled virtualenv-20.0.3
Successfully installed virtualenv-16.7.9
Running virtualenv with interpreter /home/altendky/.pyenv/shims/python3
Already using interpreter /home/altendky/.pyenv/versions/3.8.1/bin/python3
Using base prefix '/home/altendky/.pyenv/versions/3.8.1'
New python executable in /home/altendky/1599/testenv/bin/python3
Also creating executable in /home/altendky/1599/testenv/bin/python
Installing setuptools, pip, wheel...
done.
['', '/home/altendky/1599/testenv/lib/python38.zip', '/home/altendky/1599/testenv/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
['', '/home/altendky/1599/testenv/lib/python38.zip', '/home/altendky/1599/testenv/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
~/1599 master ✚ venv/bin/pip install 'virtualenv==20'; rm -rf testenv p; venv/bin/virtualenv -p python3 testenv; ln -s testenv/bin/python p; testenv/bin/python -c 'import sys; print(sys.path)'; ./p -c 'import sys; print(sys.path)';
Collecting virtualenv==20
Downloading virtualenv-20.0.0-py2.py3-none-any.whl (4.6 MB)
|████████████████████████████████| 4.6 MB 1.2 MB/s
Requirement already satisfied: six<2,>=1.12.0 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (1.14.0)
Requirement already satisfied: filelock<4,>=3.0.0 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (3.0.12)
Requirement already satisfied: appdirs<2,>=1.4.3 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (1.4.3)
Installing collected packages: virtualenv
Attempting uninstall: virtualenv
Found existing installation: virtualenv 16.7.9
Uninstalling virtualenv-16.7.9:
Successfully uninstalled virtualenv-16.7.9
Successfully installed virtualenv-20.0.0
['', '/home/altendky/.pyenv/versions/3.8.1/lib/python38.zip', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
['', '/home/altendky/.pyenv/versions/3.8.1/lib/python38.zip', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/site-packages'] |
For at least a little more completeness on my check... ~ master ✚ python3 --version --version
Python 3.8.1 (default, Dec 27 2019, 18:45:57)
[GCC 9.2.1 20191008]
~ master ✚ type python3
python3 is /home/altendky/.pyenv/shims/python3 |
This is an interesting issue. Note though from my test this behaviour is the same if you would be using |
So reading through https://www.python.org/dev/peps/pep-0405/#specification and thinking this through this kind of behaviour is no longer supported by design. The python executable is no longer a copy of the host python (as was with the old virtualenv), but instead we just symlink it and we use the
TLDR. Symlinking virtual environment python executable is not supported, no longer works on Python 3.4 and later. If this is something you would like to do, consider instead creating a proxy script; e.g. with bash: printf '#!/bin/bash\n/usr/local/lib/docker/virtualenv "$@"' 1>python-docker
chmod u+x python-docker
./python-docker -m site # this now will work |
On my host, I'm using 3.6 so it seems at the Python level it's still supported. |
@nickjj not sure I follow, what do you mean at python level? |
In your TL;DR it sounded like you were saying Python itself doesn't support symlinking virtualenvs as of Python 3.4+, but as long as you stick to an older version of virtualenv / venv, symlinking virtualenvs works with Python 3.6.x and 3.8.x. So I was wondering if this is really a Python 3.4 issue, or a virtualenv / venv issue. |
Let me rephrase:
I don't think it's a virtualenv/venv issue per se, it's a design decision that does not allows/supports such use case, and cannot by design. I don't think this use case was also officially supported with virtualenv <20, but worked as a side-effect of how the virtual environment creation happened. PS. If you really want the symlink you can pin your virtualenv to <20; but I'd instead suggest using the shim script above instead. |
Thanks. What's interesting is Ubuntu 20.04 (coming out in a few months) and the next stable release of Buster that's still years away both continue to have virtualenv 15.1.x in their repos. So it sounds like as long as you apt install virtualenv, you'll be good to go for a few years at least. |
That might be true as long as you stick to |
Virtualenv 20+ no longer supports using symlinks to a Python interpreter due to venv no longer supporting this. So now we'll create our own proxy script which functionality wise serves the same purpose as a symlink. More details are at: pypa/virtualenv#1599
Symlinks to binaries in the virtualenv bin dir will stop working once we upgrade virtualenv due to changes in how it sets up the virtual environments (using some symlinks itself). We had to revert the last upgrade attempt because we missed some of these, so trying to update them first this time. This PR: * Changes all use of such symlinks to use the link's target directly instead * Stops creating these symlinks, since they'll stop working soon anyway If we can get this deployed without incident, we should be clear to re-attempt the reverted https://github.com/edx/configuration/pull/6083 (after applying the configparser and zipp pins from https://github.com/edx/configuration/pull/6115 and https://github.com/edx/configuration/pull/6116). More details on why such symlinks stopped working in virtualenv 20+: * pypa/virtualenv#1599 * pypa/virtualenv#1773
Hi,
I'm running Ubuntu 18.04.3 with Python 3.6.9 (the default version on this distro).
Here's as much info as I have at the moment:
pip install docker
inside of this venv which is located at/usr/local/lib/docker/virtualenv
lrwxrwxrwx 1 root root 43 Feb 12 18:25 python-docker -> /usr/local/lib/docker/virtualenv/bin/python
The symlink was created with absolute paths.
If I run
python-docker
, I get thrown into a Python prompt and runningimport docker
fails.If I run
/usr/local/lib/docker/virtualenv/bin/python
, I get thrown into a Python prompt and runningimport docker
is successful. The above path is the symlink destination, I literally copy / pasted it.Here's the
sys.path
output which is different between them:In the working case, we can see that it references the virtualenv where as the symlink does not.
If you perform the same exact work flow using virtualenv 16.3.0 then both the symlink and absolute path works and they have an identical
sys.path
of['', '/usr/local/lib/docker/virtualenv/lib/python36.zip', '/usr/local/lib/docker/virtualenv/lib/python3.6', '/usr/local/lib/docker/virtualenv/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/usr/local/lib/docker/virtualenv/lib/python3.6/site-packages']
.So it sounds like maybe something between 16.3.0 and 20.0.3 is busted when it comes to setting up the system path between symlinks and absolute path binaries.
Thanks to @altendky on IRC for thinking to look at virtualenv's version.
The text was updated successfully, but these errors were encountered: