-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
3.11.0a3: under tox, sys._base_executable is wrong #90186
Comments
Under tox, sys._base_executable is not an actual file for 3.11.0a3. It was fine in 3.11.0a2. To reproduce: --- 8< -------------------- [testenv] [testenv:py311a2] Create a new Python 3.8 virtualenv, and install latest tox (3.24.4 for me). Then run "tox". I see: -------------------------------------------------------------------------------- This came to my attention because the coverage.py test suite uses "python -m venv" to create environments. They worked under 3.11.0a2, but failed under a3. I tracked it down to a difference in sys._base_executable. I couldn't see a difference in those values without tox, but I'm not sure why it changes the results. |
Steve, could this be related to the changes in getpath? |
Ned, are you able to bisect this or provide a simpler reproducer that doesn't involve tox? |
Commit 9f2f7e4 has correct _base_executable $ venv/bin/python
Python 3.11.0a2+ (heads/bpo-45847-simple-115-g9f2f7e42269:9f2f7e42269, Dec 10 2021, 10:09:54) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/home/heimes/dev/python/cpython/venv/bin/python' _base_executable in commit 99fcf15 is wrong $ venv/bin/python
Python 3.11.0a2+ (heads/bpo-45847-simple-116-g99fcf150521:99fcf150521, Dec 10 2021, 10:12:35) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/home/heimes/dev/python/cpython/python' |
git bisect also identifies that commit as the first bad: 99fcf15 is the first bad commit
|
Indeed, seems my original hunch is correct. Steve, could you take a look when you have some time? |
I'm going to need a decent amount of time to learn all of these components, because I never use this OS, Tox, nor virtualenv :) I'll try and get to it, but don't hold your breath. Luckily, Modules/getpath.py is much easier to follow and modify than the old systems. If there's a pyvenv.cfg involved, base_executable should be calculated based on the "home" key in it. Previously, I don't think we calculated it at all on Linux - it was just sys.executable before site.py changes anything. On Windows, it was always intended to be "the executable that new venvs should be created with" so that venvs created from venvs would be based off the same install, rather than trying to chain. I have no idea what the correct path for that is in this context, so could do with some help. |
Tox isn't needed, just venv from the stdlib: $ python3.11.0a2 -m venv venv_a2
$ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/private/tmp/venv_a2/bin/python
True
$ python3.11.0a3 -m venv venv_a3
$ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/usr/local/bin/python
False |
What's the contents of the pyvenv.cfg in these cases? It looks like the first case is definitely wrong, because the base |
The two venvs seem analogous: $ cat venv_a2/pyvenv.cfg
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0
$ ls -al venv_a2/bin
total 72
drwxr-xr-x 13 nedbatchelder wheel 416 Dec 11 10:43 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 11 10:43 ../
-rw-r--r-- 1 nedbatchelder wheel 9033 Dec 11 10:43 Activate.ps1
-rw-r--r-- 1 nedbatchelder wheel 1993 Dec 11 10:43 activate
-rw-r--r-- 1 nedbatchelder wheel 919 Dec 11 10:43 activate.csh
-rw-r--r-- 1 nedbatchelder wheel 2061 Dec 11 10:43 activate.fish
-rwxr-xr-x 1 nedbatchelder wheel 244 Dec 11 10:43 pip*
-rwxr-xr-x 1 nedbatchelder wheel 244 Dec 11 10:43 pip3*
-rwxr-xr-x 1 nedbatchelder wheel 244 Dec 11 10:43 pip3.11*
lrwxr-xr-x 1 nedbatchelder wheel 14 Dec 11 10:43 python@ -> python3.11.0a2
lrwxr-xr-x 1 nedbatchelder wheel 14 Dec 11 10:43 python3@ -> python3.11.0a2
lrwxr-xr-x 1 nedbatchelder wheel 14 Dec 11 10:43 python3.11@ -> python3.11.0a2
lrwxr-xr-x 1 nedbatchelder wheel 29 Dec 11 10:43 python3.11.0a2@ -> /usr/local/bin/python3.11.0a2
$ cat venv_a3/pyvenv.cfg
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0
$ ls -al venv_a3/bin
total 72
drwxr-xr-x 13 nedbatchelder wheel 416 Dec 11 10:43 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 11 10:43 ../
-rw-r--r-- 1 nedbatchelder wheel 9033 Dec 11 10:43 Activate.ps1
-rw-r--r-- 1 nedbatchelder wheel 1993 Dec 11 10:43 activate
-rw-r--r-- 1 nedbatchelder wheel 919 Dec 11 10:43 activate.csh
-rw-r--r-- 1 nedbatchelder wheel 2061 Dec 11 10:43 activate.fish
-rwxr-xr-x 1 nedbatchelder wheel 244 Dec 11 10:43 pip*
-rwxr-xr-x 1 nedbatchelder wheel 244 Dec 11 10:43 pip3*
-rwxr-xr-x 1 nedbatchelder wheel 244 Dec 11 10:43 pip3.11*
lrwxr-xr-x 1 nedbatchelder wheel 14 Dec 11 10:43 python@ -> python3.11.0a3
lrwxr-xr-x 1 nedbatchelder wheel 14 Dec 11 10:43 python3@ -> python3.11.0a3
lrwxr-xr-x 1 nedbatchelder wheel 14 Dec 11 10:43 python3.11@ -> python3.11.0a3
lrwxr-xr-x 1 nedbatchelder wheel 29 Dec 11 10:43 python3.11.0a3@ -> /usr/local/bin/python3.11.0a3 |
I tried the same stuff as nedbat on WSL2, and I see similar change in the path of sys._base_executable (though I get a different "base" path on a3, so the path exists even there). $ ~/.pyenv/versions/3.11.0a2/bin/python -m venv venv_a2
$ ~/.pyenv/versions/3.11.0a3/bin/python -m venv venv_a3
$ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/home/ss/venv_a2/bin/python
True
$ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/home/ss/.pyenv/versions/3.11.0a3/bin/python
True
$ cat venv_a2/pyvenv.cfg
home = /home/ss/.pyenv/versions/3.11.0a2/bin
include-system-site-packages = false
version = 3.11.0
$ cat venv_a3/pyvenv.cfg
home = /home/ss/.pyenv/versions/3.11.0a3/bin
include-system-site-packages = false
version = 3.11.0 |
But on windows with the python.org installer, the behavior is same for both both a2 and a3. # With a3 installed
# With a2 installed
|
This is the intended behaviour, and yes it's changed from previous versions (but not on Windows, where it was always correct). The previous value was incorrect, hence it was marked as an internal field. If your venv doesn't have a "python" binary in the directory set as "home" in pyvenv.cfg (which by definition, according to Lib/venv/init.py, is dirname(sys.executable) ), then there's some other issue with venv on your platform. |
I'm downgrading this from release blocker. If Vinay thinks there's a venv-related release blocker here he's welcome to raise the priority again, but I only see an intentional change to an internal value. Tools relying on internal fields will have to adapt for 3.11. |
But the value as it's calculated now seems to give a file that doesn't exist - how can that be correct? |
This started because a set of tests in the coverage.py test suite fail with 3.11.0a3. They attempt to create a venv in the test, and the test is already running inside a virtualenv. The venv creation fails, with a reference to a non-existent file. I wrote the bug about sys._base_executable because that seemed to be the central difference that was causing the problem. I was not relying on sys._base_executable, venv was. I will find steps to reproduce that don't reference sys._base_executable if you need them. |
Is sys._base_executable correct without a venv? It should be the same as sys.executable in that case. venv calculates 'home' here: Lib/venv/init.py#L117 executable = sys._base_executable
dirname, exename = os.path.split(os.path.abspath(executable))
context.executable = executable # not relevant to this bug
context.python_dir = dirname # written as "home = ..." If the path doesn't exist later on, it's because it didn't exist in the first place. *That* could be the real bug. |
Because we never actually use the executable referenced by the 'home' path in a pyvenv.cfg. It's only used as a starting point to locate a couple of entries in sys.path. So if the executable isn't there, it's no big deal. It shouldn't even matter in the nested-creation case, because we still don't execute it then. It's just how we end up with the same value for 'home' in any venvs created within a venv. |
This is actually not true on Windows or (I believe) some situations on macOS, where we need to use a redirecting launcher to actually launch the binary specified in this value. But on POSIX, because all the references are hardcoded in the binary, we can just copy the original around and it never loses track of its real home and never has to launch its original copy. |
Demonstration of a problem with only stdlib, and no undocumented features. This is on a Mac: This works: $ python3.10 -V
Python 3.10.0
$ python3.10 -m venv v310
$ v310/bin/python -m venv v310-nested
$ v310-nested/bin/python -V
Python 3.10.0 This does not work: $ python3.11 -V
Python 3.11.0a3
$ python3.11 -m venv v311
$ v311/bin/python -m venv 311-nested
Error: [Errno 2] No such file or directory: '/private/tmp/bpo-46028/311-nested/bin/python' |
I assume /private/tmp/bpo-46028/311-nested/bin/python3.11 exists though? You also have v311/bin/python3.11, right? If you use that one, does it |
Or possibly that error is coming from the attempt to copy it? And since This could be corrected in getpath.py with a platform-specific quirk The actual binary could also be added to pyvenv.cfg as another value - |
Yes, that file exists, but it's a symlink to a non-existent file: $ ls -al 311-nested/bin
total 0
drwxr-xr-x 5 nedbatchelder wheel 160 Dec 13 18:04 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 13 18:04 ../
lrwxr-xr-x 1 nedbatchelder wheel 21 Dec 13 18:04 python@ -> /usr/local/bin/python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 13 18:04 python3@ -> python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 13 18:04 python3.11@ -> python
$ ls -al /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory |
Does the first venv's 'python' link to python3[.11]? If so, maybe _base_executable should be based on real_executable's filename rather than executable (that is, *after* resolving symlinks rather than before). I don't *think* that will cause any issues, and it shouldn't be any more expensive to compute. Only has to change for when a pyvenv.cfg is detected I think. |
Here's the experiment again with 3.10.1 and 3.11.0a3, and more ls's: $ python3.10 -V
Python 3.10.1
$ python3.10 -m venv v310
$ ls -al v310/bin
total 72
drwxr-xr-x 12 nedbatchelder wheel 384 Dec 16 06:42 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 16 06:42 ../
-rw-r--r-- 1 nedbatchelder wheel 9033 Dec 16 06:42 Activate.ps1
-rw-r--r-- 1 nedbatchelder wheel 1993 Dec 16 06:42 activate
-rw-r--r-- 1 nedbatchelder wheel 919 Dec 16 06:42 activate.csh
-rw-r--r-- 1 nedbatchelder wheel 2061 Dec 16 06:42 activate.fish
-rwxr-xr-x 1 nedbatchelder wheel 246 Dec 16 06:42 pip*
-rwxr-xr-x 1 nedbatchelder wheel 246 Dec 16 06:42 pip3*
-rwxr-xr-x 1 nedbatchelder wheel 246 Dec 16 06:42 pip3.10*
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 06:42 python@ -> python3.10
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 06:42 python3@ -> python3.10
lrwxr-xr-x 1 nedbatchelder wheel 25 Dec 16 06:42 python3.10@ -> /usr/local/bin/python3.10
$ ls -al /usr/local/bin/python3.10
lrwxr-xr-x 1 nedbatchelder admin 53 Dec 16 06:38 /usr/local/bin/python3.10@ -> /usr/local/pyenv/pyenv/versions/3.10.1/bin/python3.10
$ v310/bin/python -m venv v310-nested
$ v310-nested/bin/python -V
Python 3.10.1
$ ls -al v310-nested/bin
total 72
drwxr-xr-x 12 nedbatchelder wheel 384 Dec 16 06:43 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 16 06:43 ../
-rw-r--r-- 1 nedbatchelder wheel 9033 Dec 16 06:43 Activate.ps1
-rw-r--r-- 1 nedbatchelder wheel 2014 Dec 16 06:43 activate
-rw-r--r-- 1 nedbatchelder wheel 940 Dec 16 06:43 activate.csh
-rw-r--r-- 1 nedbatchelder wheel 2082 Dec 16 06:43 activate.fish
-rwxr-xr-x 1 nedbatchelder wheel 249 Dec 16 06:43 pip*
-rwxr-xr-x 1 nedbatchelder wheel 249 Dec 16 06:43 pip3*
-rwxr-xr-x 1 nedbatchelder wheel 249 Dec 16 06:43 pip3.10*
lrwxr-xr-x 1 nedbatchelder wheel 37 Dec 16 06:43 python@ -> /private/tmp/bpo46028/v310/bin/python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 16 06:43 python3@ -> python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 16 06:43 python3.10@ -> python
$ ls -al /private/tmp/bpo46028/v310/bin/python
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 06:42 /private/tmp/bpo46028/v310/bin/python@ -> python3.10
$ python3.11 -V
Python 3.11.0a3
$ python3.11 -m venv v311
$ ls -al v311/bin
total 72
drwxr-xr-x 12 nedbatchelder wheel 384 Dec 16 06:45 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 16 06:45 ../
-rw-r--r-- 1 nedbatchelder wheel 9033 Dec 16 06:45 Activate.ps1
-rw-r--r-- 1 nedbatchelder wheel 1993 Dec 16 06:45 activate
-rw-r--r-- 1 nedbatchelder wheel 919 Dec 16 06:45 activate.csh
-rw-r--r-- 1 nedbatchelder wheel 2061 Dec 16 06:45 activate.fish
-rwxr-xr-x 1 nedbatchelder wheel 246 Dec 16 06:45 pip*
-rwxr-xr-x 1 nedbatchelder wheel 246 Dec 16 06:45 pip3*
-rwxr-xr-x 1 nedbatchelder wheel 246 Dec 16 06:45 pip3.11*
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 06:45 python@ -> python3.11
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 06:45 python3@ -> python3.11
lrwxr-xr-x 1 nedbatchelder wheel 25 Dec 16 06:45 python3.11@ -> /usr/local/bin/python3.11
$ ls -al /usr/local/bin/python3.11
lrwxr-xr-x 1 nedbatchelder admin 55 Dec 9 12:23 /usr/local/bin/python3.11@ -> /usr/local/pyenv/pyenv/versions/3.11.0a3/bin/python3.11
$ v311/bin/python -m venv v311-nested
Error: [Errno 2] No such file or directory: '/private/tmp/bpo46028/v311-nested/bin/python'
$ ls -al v311-nested/bin
total 0
drwxr-xr-x 5 nedbatchelder wheel 160 Dec 16 06:45 ./
drwxr-xr-x 6 nedbatchelder wheel 192 Dec 16 06:45 ../
lrwxr-xr-x 1 nedbatchelder wheel 21 Dec 16 06:45 python@ -> /usr/local/bin/python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 16 06:45 python3@ -> python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 16 06:45 python3.11@ -> python
$ ls -al /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory |
This PR *might* be a fix, but I think it's only partial. It isn't going to work if you've made a venv and copied "python3"->"python", for example. It still might be best solved by writing base_executable into pyvenv.cfg, and then simply reading it out. |
Thanks, your change looks good. The exact symlinks in the nested venv's are different for 3.10.1 and your 3.11, but they both work: $ python3.10 -c "import sys; print(sys.version)"
3.10.1 (main, Dec 14 2021, 08:30:13) [Clang 12.0.0 (clang-1200.0.32.29)]
$ python3.10 -m venv v310
$ ls -al v310/bin/py*
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 18:48 v310/bin/python@ -> python3.10
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 18:48 v310/bin/python3@ -> python3.10
lrwxr-xr-x 1 nedbatchelder wheel 25 Dec 16 18:48 v310/bin/python3.10@ -> /usr/local/bin/python3.10
$ v310/bin/python -m venv v310-nested
$ ls -al v310-nested/bin/py*
lrwxr-xr-x 1 nedbatchelder wheel 37 Dec 16 18:48 v310-nested/bin/python@ -> /private/tmp/bpo46028/v310/bin/python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 16 18:48 v310-nested/bin/python3@ -> python
lrwxr-xr-x 1 nedbatchelder wheel 6 Dec 16 18:48 v310-nested/bin/python3.10@ -> python $ $ python3.11 -c "import sys; print(sys.version)"
3.11.0a3+ (heads/pr/30144:8b260a8606, Dec 16 2021, 14:31:40) [Clang 12.0.0 (clang-1200.0.32.29)]
$ python3.11 -m venv v311
$ ls -al v311/bin/py*
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 18:50 v311/bin/python@ -> python3.11
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 18:50 v311/bin/python3@ -> python3.11
lrwxr-xr-x 1 nedbatchelder wheel 25 Dec 16 18:50 v311/bin/python3.11@ -> /usr/local/bin/python3.11
$ v311/bin/python -m venv v311-nested
$ ls -al v311-nested/bin/py*
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 18:50 v311-nested/bin/python@ -> python3.11
lrwxr-xr-x 1 nedbatchelder wheel 10 Dec 16 18:50 v311-nested/bin/python3@ -> python3.11
lrwxr-xr-x 1 nedbatchelder wheel 33 Dec 16 18:50 v311-nested/bin/python3.11@ -> /usr/local/cpython/bin/python3.11 |
See also: https://discuss.python.org/t/virtual-environments-vs-nix-python-upgrades/12588 "Virtual environments vs. *nix Python upgrades". |
Merged my PR, but I want to leave this open in commit review for now - I'm not sure it deals with all the issues here, and probably not everything from the Discourse thread linked by Victor (though it might come close). |
I'm going to assume that since we made it to RC without further comment, the change I made fixes it. If not, I guess 3.11 is going to be the dividing line where the behaviour changed... |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: