[Bug]: The interpreter cannot do repo-relative imports

[mvukov]

What happened?

It looks like the generated venv isn't complete. I cannot import repo-local packages with repo-relative import mechanism.

Version

Development (host) and target OS/architectures: Ubuntu 20.04

Output of bazel --version: 5.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

rules_py: c89d948

Language(s) and/or frameworks involved:

Python and Bazel

How to reproduce

I created a simple workspace in a fork: https://github.com/oqton/rules_py/tree/feature/import_path_bug. To reproduce just do:

cd examples/import_path_bug
bazelisk run //libs/bar

On my machine I get:

bazelisk run //libs/bar
INFO: Analyzed target //libs/bar:bar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //libs/bar:bar up-to-date:
  bazel-bin/libs/bar/bar
INFO: Elapsed time: 0.443s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
[
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python38.zip",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8/lib-dynload",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/bar.venv/lib/python3.8/site-packages",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/foo",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar"
]
Traceback (most recent call last):
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar/bar_app.py", line 6, in <module>
    from libs.foo import foo_lib
ModuleNotFoundError: No module named 'libs'

With regular Python rules running the target works as expected.

[mvukov]

I think the following entry is missing in the path:

"/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug",

[mvukov]

The issue is in https://github.com/aspect-build/rules_py/blob/main/py/private/venv/venv.bzl#L58. If I extend the imports_depset as:

    imports_depset = depset(
        direct = [ctx.workspace_name],
        transitive = [imports_depset],
    )

, then the provided example works as expected:

bazelisk run //libs/bar
INFO: Analyzed target //libs/bar:bar (2 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //libs/bar:bar up-to-date:
  bazel-bin/libs/bar/bar
INFO: Elapsed time: 2.668s, Critical Path: 2.28s
INFO: 3 processes: 2 internal, 1 linux-sandbox.
INFO: Build completed successfully, 3 total actions
INFO: Build completed successfully, 3 total actions
[
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python38.zip",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8/lib-dynload",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/bar.venv/lib/python3.8/site-packages",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/foo",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug"
]
print_foo

@mattem How would you like to proceed?

[mattem]

I'm not sure we'd want to manipulate the PYTHONPATH here, as we'd be back in the same set of issues that users experience under rules_python, this would also not work for IDEs.
Perhaps we can do something similar to pips editable install and link the module into the venv?

[mvukov]

So, the current situation is that (as explained above) we have these entries in the Python path

...
.../bar.runfiles/import_path_bug/libs/foo
.../bar.runfiles/import_path_bug/libs/bar

but the repo root is missing .../bar.runfiles/import_path_bug -- that one I added in my fix.

Going into a bit more details, before my fix:

cd examples/import_path_bug/
bazelisk run //libs/bar  # Fails.
cat bazel-bin/libs/bar/bar.venv.pth
../../../..
../../../../import_path_bug/libs/foo
../../../../import_path_bug/libs/bar

with my fix:

bazelisk run //libs/bar  # Works.
cat bazel-bin/libs/bar/bar.venv.pth
../../../..
../../../../import_path_bug/libs/foo
../../../../import_path_bug/libs/bar
../../../../import_path_bug

So, I think the paths are wired correctly via virtual env.

@mattem WDYT?