Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

running a symlink results in an inaccurate sys.path and module imports fail #1599

Closed
nickjj opened this issue Feb 12, 2020 · 10 comments
Closed

running a symlink results in an inaccurate sys.path and module imports fail #1599

nickjj opened this issue Feb 12, 2020 · 10 comments
Labels

Comments

@nickjj
Copy link

@nickjj nickjj commented Feb 12, 2020

Hi,

I'm running Ubuntu 18.04.3 with Python 3.6.9 (the default version on this distro).

Here's as much info as I have at the moment:

  1. I created a virtualenv using 20.0.3
  2. I ran pip install docker inside of this venv which is located at /usr/local/lib/docker/virtualenv
  3. I created a symlink lrwxrwxrwx 1 root root 43 Feb 12 18:25 python-docker -> /usr/local/lib/docker/virtualenv/bin/python

The symlink was created with absolute paths.

If I run python-docker, I get thrown into a Python prompt and running import docker fails.

If I run /usr/local/lib/docker/virtualenv/bin/python, I get thrown into a Python prompt and running import docker is successful. The above path is the symlink destination, I literally copy / pasted it.

Here's the sys.path output which is different between them:

# This is where `import docker` fails using the symlink:
root@ubuntu-s-1vcpu-1gb-nyc3-01:/usr/local/lib/docker/virtualenv/bin# python-docker
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']


# This is where `import docker` works using the absolute path:
root@ubuntu-s-1vcpu-1gb-nyc3-01:/usr/local/lib/docker/virtualenv/bin# /usr/local/lib/docker/virtualenv/bin/python
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/docker/virtualenv/lib/python3.6/site-packages']

In the working case, we can see that it references the virtualenv where as the symlink does not.

If you perform the same exact work flow using virtualenv 16.3.0 then both the symlink and absolute path works and they have an identical sys.path of ['', '/usr/local/lib/docker/virtualenv/lib/python36.zip', '/usr/local/lib/docker/virtualenv/lib/python3.6', '/usr/local/lib/docker/virtualenv/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/usr/local/lib/docker/virtualenv/lib/python3.6/site-packages'].

So it sounds like maybe something between 16.3.0 and 20.0.3 is busted when it comes to setting up the system path between symlinks and absolute path binaries.

Thanks to @altendky on IRC for thinking to look at virtualenv's version.

@altendky

This comment has been minimized.

Copy link
Contributor

@altendky altendky commented Feb 12, 2020

Looks to be introduced at v20.

 ~/1599   master ✚  venv/bin/pip install 'virtualenv<20'; rm -rf testenv p; venv/bin/virtualenv -p python3 testenv; ln -s testenv/bin/python p; testenv/bin/python -c 'import sys; print(sys.path)'; ./p -c 'import sys; print(sys.path)';
Collecting virtualenv<20
  Using cached virtualenv-16.7.9-py2.py3-none-any.whl (3.4 MB)
Installing collected packages: virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 20.0.3
    Uninstalling virtualenv-20.0.3:
      Successfully uninstalled virtualenv-20.0.3
Successfully installed virtualenv-16.7.9
Running virtualenv with interpreter /home/altendky/.pyenv/shims/python3
Already using interpreter /home/altendky/.pyenv/versions/3.8.1/bin/python3
Using base prefix '/home/altendky/.pyenv/versions/3.8.1'
New python executable in /home/altendky/1599/testenv/bin/python3
Also creating executable in /home/altendky/1599/testenv/bin/python
Installing setuptools, pip, wheel...
done.
['', '/home/altendky/1599/testenv/lib/python38.zip', '/home/altendky/1599/testenv/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
['', '/home/altendky/1599/testenv/lib/python38.zip', '/home/altendky/1599/testenv/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
 ~/1599   master ✚  venv/bin/pip install 'virtualenv==20'; rm -rf testenv p; venv/bin/virtualenv -p python3 testenv; ln -s testenv/bin/python p; testenv/bin/python -c 'import sys; print(sys.path)'; ./p -c 'import sys; print(sys.path)';
Collecting virtualenv==20
  Downloading virtualenv-20.0.0-py2.py3-none-any.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 1.2 MB/s 
Requirement already satisfied: six<2,>=1.12.0 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (1.14.0)
Requirement already satisfied: filelock<4,>=3.0.0 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (3.0.12)
Requirement already satisfied: appdirs<2,>=1.4.3 in ./venv/lib/python3.8/site-packages (from virtualenv==20) (1.4.3)
Installing collected packages: virtualenv
  Attempting uninstall: virtualenv
    Found existing installation: virtualenv 16.7.9
    Uninstalling virtualenv-16.7.9:
      Successfully uninstalled virtualenv-16.7.9
Successfully installed virtualenv-20.0.0
['', '/home/altendky/.pyenv/versions/3.8.1/lib/python38.zip', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/home/altendky/1599/testenv/lib/python3.8/site-packages']
['', '/home/altendky/.pyenv/versions/3.8.1/lib/python38.zip', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/lib-dynload', '/home/altendky/.pyenv/versions/3.8.1/lib/python3.8/site-packages']
@altendky

This comment has been minimized.

Copy link
Contributor

@altendky altendky commented Feb 12, 2020

For at least a little more completeness on my check...

 ~   master ✚  python3 --version --version
Python 3.8.1 (default, Dec 27 2019, 18:45:57) 
[GCC 9.2.1 20191008]
 ~   master ✚  type python3
python3 is /home/altendky/.pyenv/shims/python3
@gaborbernat gaborbernat changed the title Somewhere between virtualenv 16.3.0 and 20.0.3, running a symlink results in an inaccurate sys.path and module imports fail running a symlink results in an inaccurate sys.path and module imports fail Feb 13, 2020
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 24, 2020

This is an interesting issue. Note though from my test this behaviour is the same if you would be using venv instead of virtualenv; so the issue is probably upstream for CPython and as such I'd recommend opening an issue under https://bugs.python.org/. With virtualenv 20 our virtual environment creation now matches venv's method, hence why this now is discovered.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 24, 2020

So reading through https://www.python.org/dev/peps/pep-0405/#specification and thinking this through this kind of behaviour is no longer supported by design.

The python executable is no longer a copy of the host python (as was with the old virtualenv), but instead we just symlink it and we use the pyenv.cfg as marker for the virtual environment prefix. If you symlink a symlink with python 3+ you effectively:

This PEP proposes to add a new first step to this search. If a pyvenv.cfg file is found either adjacent to the Python executable or one directory above it (if the executable is a symlink, it is not dereferenced), this file is scanned for lines of the form key = value. If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment.

  • if the pyenv.cfg is not discovered you resolve back to the system python (this is what happens in your case here),
  • if the pyenv.cfg would be found (e.g. you would symlink it alongside the python executable symlink), the home prefix would be sent where the symlink lives; so you would need to also symlink the lib/include folder to pull in the site-package for that.

TLDR. Symlinking virtual environment python executable is not supported, no longer works on Python 3.4 and later. If this is something you would like to do, consider instead creating a proxy script; e.g. with bash:

printf '#!/bin/bash\n/usr/local/lib/docker/virtualenv "$@"' 1>python-docker
chmod u+x python-docker
./python-docker -m site # this now will work
@nickjj

This comment has been minimized.

Copy link
Author

@nickjj nickjj commented Feb 24, 2020

TLDR. Symlinking virtual environment python executable is not supported, no longer works on Python 3.4 and later.

On my host, I'm using 3.6 so it seems at the Python level it's still supported.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 24, 2020

@nickjj not sure I follow, what do you mean at python level?

@nickjj

This comment has been minimized.

Copy link
Author

@nickjj nickjj commented Feb 24, 2020

In your TL;DR it sounded like you were saying Python itself doesn't support symlinking virtualenvs as of Python 3.4+, but as long as you stick to an older version of virtualenv / venv, symlinking virtualenvs works with Python 3.6.x and 3.8.x. So I was wondering if this is really a Python 3.4 issue, or a virtualenv / venv issue.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 24, 2020

Let me rephrase:

  • venv (part of Python3.4 or later) never supported symlinking the python path,
  • virtualenv 20 or later follow the venv standard, and does not support symlinking (and cannot),
  • virtualenv <20 used a different mechanism to generate virtual environments that as a side effect supported symlinking; note virtualenv<20 is no longer maintained, but your use case still work with it.

I don't think it's a virtualenv/venv issue per se, it's a design decision that does not allows/supports such use case, and cannot by design. I don't think this use case was also officially supported with virtualenv <20, but worked as a side-effect of how the virtual environment creation happened.

PS. If you really want the symlink you can pin your virtualenv to <20; but I'd instead suggest using the shim script above instead.

@nickjj

This comment has been minimized.

Copy link
Author

@nickjj nickjj commented Feb 24, 2020

Thanks.

What's interesting is Ubuntu 20.04 (coming out in a few months) and the next stable release of Buster that's still years away both continue to have virtualenv 15.1.x in their repos. So it sounds like as long as you apt install virtualenv, you'll be good to go for a few years at least.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Feb 24, 2020

That might be true as long as you stick to apt-get things... but as soon as you start using pip you might be back here. I assume to run into this issue you've also pip installed virtualenv 20; so 🤷‍♂ not sure how apt-get using old virtualenv relates to this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.