Runfiles `Rlocation()` call without `source_repo` param fails in custom python package

[benspector42]

🐞 bug report

Affected Rule

python.runfiles Rlocation()

Is this a regression?

No

Description

When we call Rlocation() inside of a custom python package without passing in the source_repo= parameter and import our package for use elsewhere, if sys.path includes a path to our package's code that is outside the runfiles directory, then the Rlocation() call will fail.

Specifically, this is caused by the code here, which attempts to look up runfiles using the repository mapping of the caller of the Rlocation() method. If sys.path has found and run the copy of our python package that resides outside of the runfiles directory, then the Rlocation() call is legitimately coming from outside the runfiles directory, causing the underlying call to self.CurrentRepository() to fail.

A custom package which calls Rlocation() shouldn't be required to provide the source_repo= parameter or else run the risk of error due to being found outside of the runfiles directory due to the configuration of sys.path.

🔬 Minimal Reproduction

MODULE.bazel

module(name = "reproducer")

bazel_dep(
    name = "rules_python",
    version = "1.9.0"
)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.14",
)

BUILD.bazel

load("@rules_python//python:py_binary.bzl", "py_binary")
load("@rules_python//python:py_library.bzl", "py_library")

py_binary(
    name = "binary",
    srcs = ["binary.py"],
    deps = [":library"],
    visibility = ["//visibility:public"],
)

py_library(
    name = "library",
    srcs = ["library/__init__.py"],
    deps = ["@rules_python//python/runfiles"],
    visibility = ["//visibility:public"],
)

binary.py

from library import rlocation_wrapper

if __name__ == "__main__":
    rlocation_wrapper()

library/__init__.py

from python.runfiles import Runfiles


def rlocation_wrapper() -> None:
    runfiles_obj = Runfiles.Create()
    print(runfiles_obj.Rlocation("library/__init__"))

In order to have sys.path find the package outside the runfiles directory, we can set the PYTHONPATH environment variable to the outer folder containing the package.
To reproduce the failure, run:

export PYTHONPATH=/home/<username>/<reproduction_folder>
bazelisk run //:binary

🔥 Exception or Error

ValueError: /home/<username>/<reproduction_folder>/library/__init__.py does not lie under the runfiles root /home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles

Full traceback:

Traceback (most recent call last):
  File "/home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles/_main/_binary_stage2_bootstrap.py", line 537, in <module>
    main()
    ~~~~^^
  File "/home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles/_main/_binary_stage2_bootstrap.py", line 531, in main
    _run_py_path(main_filename, args=sys.argv[1:])
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles/_main/_binary_stage2_bootstrap.py", line 319, in _run_py_path
    runpy.run_path(main_filename, run_name="__main__")
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles/_main/binary.py", line 4, in <module>
    rlocation_wrapper()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/<username>/<reproduction_folder>/library/__init__.py", line 6, in rlocation_wrapper
    print(runfiles_obj.Rlocation("library/__init__"))
          ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles/rules_python+/python/runfiles/runfiles.py", line 306, in Rlocation
    source_repo = self.CurrentRepository(frame=2)
  File "/home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles/rules_python+/python/runfiles/runfiles.py", line 404, in CurrentRepository
    raise ValueError(
    ...<3 lines>...
    )
ValueError: /home/<username>/<reproduction_folder>/library/__init__.py does not lie under the runfiles root /home/<username>/.cache/bazel/_bazel_<username>/0123456789abcdef0123456789abcdef/execroot/_main/bazel-out/k8-fastbuild/bin/binary.runfiles

🌍 Your Environment

Operating System:

Ubuntu 24.04.4 LTS

Output of bazelisk version:

bazel 9.1.0

Rules_python version:

1.9.0

[rickeylev]

Looking at runfiles.py, it looks like there's some code to handle this case. However, it's special cased to Windows and Python 3.9 and earlier. The case it is handling is due to some issue where sys.path has the underlying paths instead of the runfiles paths, so __file__ doesn't look right.

On Slack, you said:

is hard coding source_repo="" the correct way to work around the issue?

Maybe? It depends on which repo's runfiles you want to look up. Empty string means the main repo (i.e. the repo that the "@" in @//foo:bar refers to). If you're only interested in files from the main repo, then yes, it's fine.

Stepping back -- why is a runfiles lookup being done by a library not in a runfiles tree? I'm just trying to figure out if always falling back to the main repo makes sense instead of raising an error.

[benspector42]

To my comment in Slack, in my particular use case I am trying to look up the runfiles of the main repo, so passing in the empty string is providing me with the behavior I'm looking for. I just wasn't sure if this was a broader issue worthy of further discussion.

To answer your other question - the library performing runfiles lookup is present in the runfiles tree (typically as a symlink). The issue I'm encountering is due to sys.path finding and running the copy of that library on my machine (the target of that symlink) which does not lie under the .runfiles directory, which results in the error.

[rickeylev]

Ahh, gotcha. So it's another special case that the existing logic isn't handling.

For my edification, what Python version and OS?

I'm thinking we should just remove that special casing logic and have it always fallback to the main repo by default

[benspector42]

I encountered this on Ubuntu 24.04.4 LTS with python 3.14.