`bootstrap_impl=script` results in corrupted (?) symlinks to interpreter on `py_binary` re-builds

[BryceBeagle]

🐞 bug report

Affected Rule

py_binary / py_executable

Is this a regression?

I don't think so. This issue appears on 1.4.x, 1.5.x, 1.6.3, and HEAD of main

Description

We recently enabled @rules_python//python/config_settings:bootstrap_impl=script in our monorepo and we're seeing some issues running py_binary targets where it looks like the cache is getting corrupted somehow.

❯ bazel run //domains/dbm/apps/apis/dbm_investigator

INFO: Invocation ID: 09217114-4ad4-4d02-9561-2cba2355f56c
INFO: Analyzed target //domains/dbm/apps/apis/dbm_investigator:dbm_investigator (2444 packages loaded, 54196 targets configured).
INFO: Found 1 target...
Target //domains/dbm/apps/apis/dbm_investigator:dbm_investigator up-to-date:
  bazel-bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator
INFO: Elapsed time: 67.231s, Critical Path: 8.00s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator

ERROR: Python interpreter not found: /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator.runfiles/dd_source/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3

lrwxr-xr-x@ 1 bryce.beagle  wheel  199 Oct  3 16:20 /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator.runfiles/dd_source/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3 -> /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3

But, after bazel clean --expunge this issue goes away and the interpreter is found again. The target works perfectly with a clean cache and the interpreter path is resolved as-expected. Execution also works again fine if we remove just the location the venv sits in using rm -rf $(bazel info execution_root)/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/ and let that rebuild.

I've been able to reproduce this simply by adjusting the srcs to one one of the py_binary's py_library deps. This causes something to rebuild and rewrite the interpreter symlink. (This probably also occurs changing other inputs to the target, but I haven't tested)

Analyzing this symlink when the target works, we can see that it's a relative symlink straight to the interpreter in the execroot.

❯ ls -hal $(bazel info execution_root)/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator.runfiles/dd_source/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3

lrwxr-xr-x@ 1 bryce.beagle  wheel    68B Oct  3 14:53
  /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator.runfiles/dd_source/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3@ 
  -> ../../../../../../../../python_3_10_aarch64-apple-darwin/bin/python3

but after adjusting the srcs and causing the target to rebuild, this symlink has changed:

❯ ls -hal $(bazel info execution_root)/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator.runfiles/dd_source/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3

lrwxr-xr-x@ 1 bryce.beagle  wheel   199B Oct  3 14:59 
  /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/dbm_investigator.runfiles/dd_source/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3@
  -> /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3

If we then look at that target, it has the exact same content as the old symlink:

❯ ls -hal /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3

lrwxr-xr-x@ 1 bryce.beagle  wheel    68B Oct  3 14:53
  /private/var/tmp/_bazel_bryce.beagle/27d027b106a956ea021b722a5bac172d/execroot/dd_source/bazel-out/darwin_arm64-fastbuild/bin/domains/dbm/apps/apis/dbm_investigator/_dbm_investigator.venv/bin/python3@
  -> ../../../../../../../../python_3_10_aarch64-apple-darwin/bin/python3

but because the symlink origin is in a different location (no longer in the runfiles), the relative path fails to resolve.

I'm really confused on how or why this would be occurring.

One thing to note is that --@rules_python//python/config_settings:venvs_use_declare_symlink=no makes this problem go away entirely. We can't rely on this, though because we need to exec the interpreter for signal propagation and do not want to set RULES_PYTHON_EXTRACT_ROOT

I'm wondering if this is an issue because we're still using bazel 7? I know that there's some issues with bootstrap_impl=script until bazel 8?

🔬 Minimal Reproduction

TODO

🌍 Your Environment

Operating System:

MacOS

Output of bazel version:

7.4.1

Rules_python version:

1.4.x, 1.5.x, 1.6.3, and HEAD of main