bazel_tools import issues with python_rule version 1.7.0-rc4

[dabanki]

🐞 bug report

Affected Rule

The issue is caused by the rule: py_binary and py_test, specifically how the execution environment, runfiles tree, and sys.path are constructed for external dependencies like @bazel_tools

Is this a regression?

Yes, this appears to be a regression. The behavior was not present before the rules_python version was upgraded in bazelbuild/bazel#27498 to 1.7.0-rc4 from 1.6.0.

Description

After updating rules_python to the version in bazelbuild/bazel#27498, imports of the deprecated runfiles library using from bazel_tools.tools.python.runfiles import runfiles fail with ModuleNotFoundError: No module named 'bazel_tools'.

This occurs in integration tests, such as //src/test/py/bazel:py_test.

The root cause appears to be a change in how the runfiles tree is structured and how sys.path is populated. Debugging shows that sys.path includes:

• .../xxx.runfiles/_main
• .../xxx.runfiles/bazel_tools

The import fails because Python looks for a top-level bazel_tools package/module within these paths. While .../xxx.runfiles/bazel_tools is the root of the @bazel_tools repository, it does not contain a package named bazel_tools. The required module is at .../xxx.runfiles/bazel_tools/tools/python/runfiles.py.

🔬 Minimal Reproduction

fetch and checkout PR branch: bazelbuild/bazel#27498

Run the failing test:
bash bazel-bin/src/bazel test //src/test/py/bazel:py_test \ --test_filter="PyRunfilesLibraryTest.testPyRunfilesLibraryCurrentRepository" \ --test_output=streamed

🔥 Exception or Error

python
Traceback (most recent call last):
  File "/.../execroot/_main/bazel-out/....runfiles/_main/pkg/binary.py", line XX, in 
    from bazel_tools.tools.python.runfiles import runfiles
ModuleNotFoundError: No module named 'bazel_tools'

🌍 Your Environment

Operating System:

  
ProductName:            macOS
ProductVersion:         15.7.2
BuildVersion:           24G325
  

Output of bazel version:

  
Build label: 8.4.2-homebrew
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Tue Jan 01 00:00:00 1980 (315532800)
Build timestamp: 315532800
Build timestamp as int: 315532800
  

Rules_python version:

  
1.7.0-rc4
  

Anything else relevant?
Debug output showing sys path information from the test: https://pastebin.com/mR9ZMiX2

[rickeylev]

I have PR #3423 out to fix.

Right now, I think its fine to call this a regression. When I refactored that system_python bootstrap code, I didn't intend for this change in behavior.

However, this behavior of adding the runfiles root to sys.path is undesirable and causes issues. It's been removed in our newer bootstrap implementation.

Right now, bazel_tools.tools.python.runfiles is just a shim to import the canonical libraries from rules_python. That's fine, however, if it's going to continue to be a compatibility shim, it needs two changes:

1. explicit __init__.py files need to be added up its directory tree. Right now it's relying on the legacy behavior of auto-creating init.py files.
2. It isn't setting the imports attribute to tell what path should be on sys.path.