sys.prefix is set incorrectly on Mac OS X

[mppf]

BPO	42312
Nosy	@ronaldoussoren, @ned-deily, @mppf

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-11.19:34:53.073>
created_at = <Date 2020-11-10.15:20:27.318>
labels = ['OS-mac', 'type-bug', '3.9']
title = 'sys.prefix is set incorrectly on Mac OS X'
updated_at = <Date 2020-11-12.15:15:01.605>
user = 'https://github.com/mppf'

bugs.python.org fields:

activity = <Date 2020-11-12.15:15:01.605>
actor = 'ronaldoussoren'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-11.19:34:53.073>
closer = 'ronaldoussoren'
components = ['macOS']
creation = <Date 2020-11-10.15:20:27.318>
creator = 'mppf'
dependencies = []
files = []
hgrepos = []
issue_num = 42312
keywords = []
message_count = 12.0
messages = ['380677', '380679', '380680', '380692', '380695', '380697', '380702', '380710', '380714', '380791', '380825', '380827']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'mppf']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42312'
versions = ['Python 3.9']

[mppf]

I have been trying to create a wrapper script for python3 in a venv that behaves similarly to a symbolic link. I am able to use exec -a in bash to run python3 with argv[0] set to the wrapper script. This allows it to function similarly to the symbolic link in a venv. However, this approach does not work on Mac OS X with a homebrew installation. I think this is a bug.

Here are the simple steps to reproduce (assuming bash shell):

cd /tmp
python3 -m venv test-venv
(exec -a test-venv/python3 python3 -c 'import sys; print(sys.executable); print (sys.prefix);')

### Good output (Ubuntu 20.04)
/tmp/test-venv/python-wrapper
/tmp

### Bad output (Homebrew on Mac OS X)
/usr/local/opt/python@3.9/bin/python3.9
/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9

Here are some things that might be related:

• the Mac OS X framework launcher and how it uses realpath (and bpo-22490)
• site.py code in def venv and the conditional on __PYVENV_LAUNCHER__. The if branch is not being run in this configuration.
• setting the environment variable PYTHONEXECUTABLE (e.g. export PYTHONEXECUTABLE=test-venv/python3 before the other commands) causes the if branch in the conditional on __PYVENV_LAUNCHER__ in site.py def venv to be run. This allows sys.executable to be set as expected but sys.prefix is still wrong.

If you are interested in something closer to the use case, the below explains how to get a more user-facing reproducer:

$ python3 -m venv test-venv

-- put this into test-venv/python-wrapper --
#!/usr/bin/env bash

# remove path component to prevent infinite loop
export PATH=${PATH//test-venv/missing}

# Now run the real python3 interpreter but tell it that it
# is being launched at the current path, so it can
# correctly find dependencies in the venv
exec -a "$0" python3 "$@"

$ chmod a+x test-venv/python-wrapper

$ ./test-venv/python-wrapper -c 'import sys; print(sys.executable); print (sys.prefix);'

(and with this script the problematic behavior is exactly the same as the exec commands above)

[mppf]

In the above I meant to include the bin path in the examples, but it does not matter for the behavior

(exec -a test-venv/bin/python3 python3 -c 'import sys; print(sys.executable); print (sys.prefix);')

[ned-deily]

I'm not sure I understand exactly what you are trying to accomplish but one potential issue strikes me: you may need to ensure you are execing the right python binary by including a more complete path:

$ (exec -a test-venv/bin/python3 test-venv/bin/python3 -c 'import sys; print(sys.executable); print (sys.prefix);')
/private/tmp/test-venv/bin/python3
/private/tmp/test-venv

[mppf]

I'm not sure I understand exactly what you are trying to accomplish but one potential issue strikes me: you may need to ensure you are execing the right python binary by including a more complete path:

That does not help with the original problem I was trying to solve, because I was trying to create a wrapper script that used whichever python3 is available according to the PATH variable (other than potentially the one for this venv).

Whether or not you think that is a reasonable thing to do, the examples I showed have a difference in behavior between Mac OS X and linux that is probably undesirable.

[ned-deily]

Sorry, I didn't intend to criticize what you are trying to do, I just not sure I understand it so I could give a more helpful response. It seems to me that an explanation for the difference in behavior you are seeing between your Ubuntu and macOS setups is in exactly what the value of PATH is when the exec is performed on each and exactly which python3 is found first. For example, when I run the test exec on my macOS system, it is clear that the python3 being invoked is not the venv one but a different python3 altogether that shows up earlier on PATH. Could such differences explain what you are seeing?

[ronaldoussoren]

The difference in behaviour is because the executable does not look at argv[0] to find its location on macOS. Your trick therefore doesn't work.

I prefer the current behaviour.

What I don't understand is *why* you want to do this. If I read your report correctly you're trying to run an Python 3 executable outside of the virtual environment as if it is in the virtual environment. Why not just use the binary inside the virtual environment?

[ronaldoussoren]

Note that the same is true for python3 outside of a venv, it does not use argv[0] to locate says.prefix (for framework builds). That’s intentional.

[mppf]

For example, when I run the test exec on my macOS system, it is clear that the python3 being invoked is not the venv one but a different python3 altogether that shows up earlier on PATH.

In the test case I am interested in, PATH is not set to the venv at all. You're supposed to be able to run a script in a venv without activating. In my case, I am trying to write a wrapper script that behaves similarly to the symbolic link in test-venv/bin/python3 but that works if the path to the python3 interpreter changes in the future. (For example, one might install and then uninstall pyenv).

[ronaldoussoren]

The way sys.prefix is calculated on macOS ensures that the correct sys.prefix is calculated even if you copy the binary to a different location. That's functionality I don't want to drop.