venv creation on macOS Catalina is failing

[dend]

BPO	38705
Nosy	@ronaldoussoren, @ned-deily, @dend

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-11-06.07:35:21.894>
created_at = <Date 2019-11-05.20:58:06.043>
labels = ['OS-mac', 'type-bug', '3.7']
title = 'venv creation on macOS Catalina is failing'
updated_at = <Date 2020-10-27.03:42:44.786>
user = 'https://github.com/dend'

bugs.python.org fields:

activity = <Date 2020-10-27.03:42:44.786>
actor = 'benesch'
assignee = 'none'
closed = True
closed_date = <Date 2019-11-06.07:35:21.894>
closer = 'ned.deily'
components = ['macOS']
creation = <Date 2019-11-05.20:58:06.043>
creator = 'dend'
dependencies = []
files = []
hgrepos = []
issue_num = 38705
keywords = []
message_count = 7.0
messages = ['356061', '356064', '356066', '356086', '379724', '379726', '379730']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'benesch', 'dend']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38705'
versions = ['Python 3.7']

[dend]

I have a project, where I am working inside a virtual environment (on macOS Catalina). Inside said project, I am attempting to create a new virtual environment with:

venv.create(virtual_environment_directory, with_pip=True)

This, however, errors out:

File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/user1/Documents/repos/adg/src/adg/main.py", line 35, in <module>
CommandProcessor.validate(args, current_os)
File "/Users/user1/Documents/repos/adg/src/adg/helpers/commandprocessor.py", line 21, in validate
LibraryProcessor.process_libraries(command.library, command.platform, command.out)
File "/Users/user1/Documents/repos/adg/src/adg/helpers/coreutil.py", line 79, in process_libraries
LibraryInstaller.create_environment()
File "/Users/user1/Documents/repos/adg/src/adg/helpers/coreutil.py", line 40, in create_environment
venv.create(virtual_environment_directory, with_pip=True)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/venv/init.py", line 363, in create
builder.create(env_dir)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/venv/init.py", line 68, in create
self._setup_pip(context)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/venv/init.py", line 261, in _setup_pip
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/user1/Documents//repos/adg/src/dtemp/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGABRT: 6>.

Seems like attempting to run this both from inside and outside the virtual environment results in the error.

[dend]

Should also mention that this is working as expected in Python 3.8.0

[ned-deily]

Thanks for the report but without a supplying a reproducible test case it's really difficult to say what issue you might be seeing. FWIW, I was not able to reproduce a failure but I was just guessing what your project code really does. Also be aware that you appear to be using the Apple-supple Python 3.7.3 that is included with Apple's Xcode 11 Command Line Tools for 10.15 Catalina and is not part of a standard macOS installation. I believe that Python3 is there only to support other parts of the Command Line Tools and it is not recommended that you use it for your own projects.

[dend]

@ned.deily actually you bring up a very good point, I did not notice that it was using the default Catalina Python 3 install. I've chatted with @brett.cannon about this earlier and it seemed like the issue was in that the Python version I was using is no longer supported, with the fix likely being in one of the later branches. I've updated my installation from python.org and everything works as expected now. Let me know if you want me to close this issue, and thank you for your insights!

[benesch]

This issue continues to exist with the system Python on macOS. Here is how to reproduce on macOS Catalina 10.15.7 and Xcode 12.1.

$ /usr/bin/python3 --version
Python 3.8.2

$ /usr/bin/python3 -c 'import venv; venv.create("venv");'

$ venv/bin/python3
dyld: Library not loaded: @executable_path/../Python3
  Referenced from: /Users/benesch/venv/bin/python3
  Reason: image not found
Abort trap: 6

Weird, isn't it? The thing that gets copied into the venv is just a totally different file than /usr/bin/python3:

$ ls -lah venv/bin/python3
-rwxr-xr-x 1 benesch staff 148K Oct 26 23:16 venv/bin/python3

$ ls -lah /usr/bin/python3
-rwxr-xr-x 1 root wheel 31K Sep 21 20:29 /usr/bin/python3

I assume this is Apple's bug, but I'll be damned if I have to file another report into the blackhole that is radar. If someone needs a quick workaround, just create your venvs with symlinks, which works just fine:

$ /usr/bin/python3 -c 'import venv; venv.create("venv2");'
$ venv2/bin/python3
Python 3.8.2 (default, Sep 24 2020, 19:37:08) 
[Clang 12.0.0 (clang-1200.0.32.21)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Incidentally this is why /usr/bin/python3 -m venv works just fine, since it defaults to symlinks=True on macOS.

[benesch]

Oops, sorry, I forgot to actually include the correct command in the workaround. Here it is:

$ /usr/bin/python3 -c 'import venv; venv.create("venv2", symlinks=True);'

[benesch]

So it looks like /usr/bin/python3 is just a shim that immediately execs the real Python binary:

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3

This binary is identical to the one that ends up in the venv.

Problematically, this binary is dynamically linked, and links the "Python3" shared library using a relative path:

$ otool -L /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3:
	@executable_path/../Python3 (compatibility version 3.8.0, current version 3.8.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0)

So obviously copying that binary to a different path breaks the @executable_path-relative link. Ah, well. Definitely seems like Apple's bug. Not sure there is anything that even could be done on the CPython side.