Virtual environments break on Homebrew upgrades due to using a Cellar link

[cjolowicz]

On Homebrew, virtual environments created by uv venv reference the Python installation under Cellar in their interpreter symlink and pyvenv.cfg, which has the full downstream version in its path. These virtual environments break when Homebrew upgrades the respective Python package to the next maintenance release. In recent versions of venv and virtualenv, this issue was resolved by using the stable link under $(brew --prefix)/opt/python@3.x/ instead. For example, on Python 3.11 macOS aarch64 this would be the interpreter in /opt/homebrew/opt/python@3.11/bin.

This shell session demonstrates the problem:

❯ uv venv
❯ readlink .venv/bin/python
/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin/python3.11
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin

When Homebrew upgrades its python@3.11 package and cleans up the old installation under Cellar, those references in the virtual environment start to dangle.

For comparison, here's what I get with venv from Homebrew's python@3.11 installation:

❯ python3.11 -m venv .venv
❯ readlink .venv/bin/python
python3.11
❯ readlink .venv/bin/python3.11
/opt/homebrew/opt/python@3.11/bin/python3.11
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.11/bin

And with virtualenv:

❯ virtualenv --version
virtualenv 20.25.0 from ~/.local/pipx/venvs/virtualenv/lib/python3.12/site-packages/virtualenv/__init__.py
❯ virtualenv .venv
❯ readlink .venv/bin/python
/opt/homebrew/opt/python@3.12/bin/python3.12
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.12/bin

Affected platforms: Linux and macOS with Homebrew Python

❯ uv --version
uv 0.1.4
❯ gcc -dumpmachine
arm64-apple-darwin23.2.0

[charliermarsh]

Do you by any chance have a link to the relevant PRs or issues in venv and/or virtualenv?

[cjolowicz]

I don't, but FWIW venv derives the home key from sys._base_executable, which is the stable path.

[charliermarsh]

Hmm yeah, we're using sys._base_executable as of an open PR, but even that gives me:

❯ python3.8
Python 3.8.18 (default, Aug 24 2023, 19:48:18)
[Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/opt/homebrew/bin/python3.8'

[charliermarsh]

The problem is that we're calling canonicalize on the sys.executable.

[charliermarsh]

@konstin -- We may want to revisit this change (#965). I think we should only do this (canonicalize) if we're in a virtual environment?

[charliermarsh]

We may want something like...

• If we're not in a venv, use sys.executable.
• If we are in a venv, resolve symlinks once (not recursively) on sys.executable.

That's closer to virtualenv:

# if we're not in a virtual environment, this is already a system python, so return the original executable
# note we must choose the original and not the pure executable as shim scripts might throw us off
return self.original_executable

[ofek]

I don't know if it's quite relevant but they just fixed a bug by storing the resolved absolute path in the metadata file that is generated for virtual environments: pypa/virtualenv#2682

Also, I know all of you are basically Rust experts, but in case I am reading the source correctly please never ever use canonicalize directly as it is literally (I'm not joking) broken on Windows and will cause all of us bugs and unexpected behavior. Please don't use it. Instead, it's common to use the dunce crate or the normpath crate when you don't want to resolve symlinks. See even Armin talking about it.

[charliermarsh]

Haha. We do use fs::canonicalize but we then strip UNC paths everywhere. (I guess we could use dunce's canonicalize to simplify that process.)

[charliermarsh]

I'd actually expect that virtualenv change to break this case, if I'm reading it correctly:

❯ realpath /opt/homebrew/opt/python@3.8/bin/python3.8
/opt/homebrew/Cellar/python@3.8/3.8.18_2/Frameworks/Python.framework/Versions/3.8/bin/python3.8

So now virtualenv would also use the Cellar path, IIUC?

[ofek]

I don't own a macOS to test that for you unfortunately but that sounds right.

[charliermarsh]

On main, virtualenv uses home = /opt/homebrew/Cellar/python@3.8/3.8.18_2/bin. So, they might have the same problem @cjolowicz?

[charliermarsh]

\cc @gaborbernat -- it seems like virtualenv on main will resolve symlinks for system Pythons, which seems desirable in some cases (hence the issue in virtualenv) but not in others (hence this issue). I can't tell which is "correct" though.

[gaborbernat]

I do not have a good answer here. 😱

[konstin]

We may want to revisit this change (#965). I think we should only do this (canonicalize) if we're in a virtual environment?

Agreed, i think this is better, i'm also thinking about the use case where somebody might have intentional redirects for their python setup.

For the Cellar issue, note that technically different patch versions aren't compatible from a packaging perspective, technically a project could require python_full_version != "3.12.2". In practice projects i've only seen lower bounds for patch version (e.g. >3.8.1), upper or exact bounds would be against how python patch versions are maintained and we're building a more reliable tool by intentionally ignoring this detail.