Python binary now dynamically links lib with relative path @loader_path/../Python

[blubars]

brew gist-logs <formula> link OR brew config AND brew doctor output

$ brew config
HOMEBREW_VERSION: 4.1.3
ORIGIN: https://github.com/Homebrew/brew
HEAD: 128df14e0afd0525564051a7665708c2973a3cab
Last commit: 12 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 9d9bf1cc622f1eddb836c01075866351fe9d0e4a
Core tap last commit: 3 hours ago
Core tap branch: master
Core tap JSON: 31 Jul 23:58 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.3 build 1403
Git: 2.41.0 => /opt/homebrew/bin/git
Curl: 7.87.0 => /usr/bin/curl
macOS: 13.3.1-arm64
CLT: 14.3.0.0.1.1679647830
Xcode: N/A
Rosetta 2: false

$ brew doctor
Your system is ready to brew.

Verification

• My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.
• I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

After upgrading python via homebrew, poetry fails to install when following poetry's official install docs. We determined that the reason is that:

• the poetry install script copies the binary (rather than making a symlink)
• the python3 binary now dynamically links to a shared library using a relative path with @loader_path/../Python. Previously, it used an absolute path.

We are curious if this change to the python lib path is intended?

What happened (include all command output)?

Failed poetry install.

$ curl -sSL https://install.python-poetry.org | POETRY_HOME=/tmp/poetry-install-test python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/tmp/poetry-install-test/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.5.1): An error occurred. Removing partial environment.
Poetry installation failed.
See /Users/blubars/poetry-installer-error-kkcak6me.log for error logs.
$ cat /Users/blubars/poetry-installer-error-kkcak6me.log
dyld[81073]: Library not loaded: @loader_path/../Python
  Referenced from: <3F2D0F69-E25A-3340-8B52-46400654C50E> /private/tmp/poetry-install-test/venv/bin/python3.11
  Reason: tried: '/private/tmp/poetry-install-test/venv/bin/../Python' (no such file), '/usr/local/lib/Python' (no such file), '/usr/lib/Python' (no such file, not in dyld cache)

Traceback:

  File "<stdin>", line 923, in main
  File "<stdin>", line 562, in run

Before, on an older version of homebrew-core (not sure which, sorry!):

$ otool -L `which python3`
/opt/homebrew/bin/python3:
	/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/Python (compatibility version 3.11.0, current version 3.11.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

After:

$ otool -L `which python3`                   
/opt/homebrew/bin/python3:
	@loader_path/../Python (compatibility version 3.11.0, current version 3.11.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

What did you expect to happen?

The poetry install copies the python binary. Why it does this is maybe a good question, but there is apparently at least one commonly-used thing out there that does this and expects the binary to continue working. Wanted to raise this to ask if this change was intended or if it should be changed back in light of this consequence?

Step-by-step reproduction instructions (by running brew commands)

brew upgrade python (or some other install/upgrade that causes python to be upgraded)
curl -sSL https://install.python-poetry.org | python3 -

[carlocab]

This change was intentional -- it is meant to improve the relocatability of our bottles, and to improve bottle pour times (because they will need less relocation when relative paths are used).

Copying binaries to new locations on macOS is, in general, not a safe operation, and the Poetry installer should not assume it can do this without accounting for how these binaries link to their dependencies. Note that their install script will result in a broken Poetry installation when the python@3.11 formula version or revision is bumped -- the relative install name just surfaces that problem earlier now.

This should probably be fixed in the Poetry installer. Doing this will also fix the breakage from version/revision bumps that I mention above.

As a workaround, something like this might work:

install_name_tool \
  -change @loader_path/../Python "$(brew --prefix python@3.11)/Frameworks/Python.framework/Python" \
  "$POETRY_HOME/venv/bin/python3.11"

You may need to codesign the binary at "$POETRY_HOME/venv/bin/python3.11" after doing this.

[blubars]

Got it, thanks @carlocab. Appreciate the thoughtful and detailed response! Closing this.

[jfly]

@carlocab, just for my own learning, what changed in brew or homebrew-core to change this linking behavior?

[carlocab]

See Homebrew/brew#15571.