Skip to content

fix: add the runfiles root as a python package #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 6, 2022
Merged

fix: add the runfiles root as a python package #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions py/private/venv/venv.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,9 @@ def _make_venv(ctx, name = None, main = None, strip_pth_workspace_root = None):
# each segment from site-packages in the venv to the root of the runfiles tree.
# Four .. will get us back to the root of the venv:
# {name}.runfiles/.{name}.venv/lib/python{version}/site-packages/first_party.pth
escape = ([".."] * 4)
escape = "/".join(([".."] * 4))
pth_add_all_kwargs = dict({
"format_each": "/".join(escape) + "/%s",
"format_each": escape + "/%s",
})

# If we are creating a venv for an IDE we likely don't have a workspace folder at with everything inside, so strip
Expand All @@ -82,6 +82,13 @@ def _make_venv(ctx, name = None, main = None, strip_pth_workspace_root = None):
"map_each": _pth_import_line_map,
})

# A few imports rely on being able to reference the root of the runfiles tree as a Python module,
# the common case here being the @rules_python//python/runfiles target that adds the runfiles helper,
# which ends up in bazel_tools/tools/python/runfiles/runfiles.py, but there are no imports attrs that hint we
# should be adding the root to the PYTHONPATH
# Maybe in the future we can opt out of this?
Copy link
Member

@alexeagle alexeagle Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it does feel smelly, like you should use the runfiles helper library rather than import these directly? maybe it should be optional, to avoid top-level dirs colliding with things on the path

Copy link
Collaborator Author

@mattem mattem Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow the suggestion? How would you use the runfiles helper to import the runfiles helper?

Copy link
Member

@alexeagle alexeagle Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I see. In rules_nodejs we just exposed an environment variable to all actions to help them find the runfiles helper.

If the python runfiles library were published to pypi, and fetched as a pip dep, could it naturally work that way?

Copy link
Collaborator Author

@mattem mattem Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'd still need the root of the runfiles tree as a package, as the import path that's documented (and therefore likely in everyones code) starts with bazel_tools, unless rules_py special cased that dependency and nested it somehow.

Yeah, if this were a regular pypi package it would fall though the normal paths.

Copy link
Collaborator Author

@mattem mattem Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternativy, rules_py vendors the runfiles helper and exposes it somewhere else at a nested package or something. Would mean a breaking change for users.
Also I wonder how many things rely on this behaviour.

Copy link
Member

@alexeagle alexeagle Jun 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you could publish a bazel_tools package to pypi and users imports would continue to work, right? seems like a more principled answer, but I don't mean to block a merge here

pth_lines.add(escape)

pth_lines.add_all(
imports_depset,
**pth_add_all_kwargs
Expand Down
10 changes: 9 additions & 1 deletion py/tests/external-deps/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
load("//py:defs.bzl", "py_binary", "py_library")
load("//py:defs.bzl", "py_binary", "py_library", "py_test")
load("@python_toolchain//:defs.bzl", "host_platform")
load("@aspect_bazel_lib//lib:write_source_files.bzl", "write_source_files")

Expand Down Expand Up @@ -54,3 +54,11 @@ write_source_files(
"expected_click": "click_out",
},
)

py_test(
name = "test_can_import_runfiles_helper",
srcs = ["test_can_import_runfiles_helper.py"],
deps = [
"@bazel_tools//tools/python/runfiles",
],
)
1 change: 1 addition & 0 deletions py/tests/external-deps/expected_pathing
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ sys path:
(py_toolchain)/lib/python3.9
(py_toolchain)/lib/python3.9/lib-dynload
(pwd)/bazel-out/host/bin/py/tests/external-deps/pathing.runfiles/pathing.venv/lib/python3.9/site-packages
(pwd)/bazel-out/host/bin/py/tests/external-deps/pathing.runfiles
(pwd)/bazel-out/host/bin/py/tests/external-deps/pathing.runfiles/aspect_rules_py/py/tests/external-deps

Entrypoint Path: (pwd)/bazel-out/host/bin/py/tests/external-deps/pathing.runfiles/aspect_rules_py/py/tests/external-deps/pathing.py
Expand Down
4 changes: 4 additions & 0 deletions py/tests/external-deps/test_can_import_runfiles_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
try:
from bazel_tools.tools.python.runfiles import runfiles
except ModuleNotFoundError as err:
assert False, "Expected to be able to import runfiles helper"
10 changes: 10 additions & 0 deletions py/tests/import-pathing/tests.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@ def _can_resolve_path_in_workspace_test_impl(ctx):
imports = py_library.make_imports_depset(fake_ctx).to_list()
asserts.equals(env, "aspect_rules_py/foo/bar", imports[0])

# Empty string import is semantically equal to "."
fake_ctx = _ctx_with_imports([""])
imports = py_library.make_imports_depset(fake_ctx).to_list()
asserts.equals(env, "aspect_rules_py/foo/bar", imports[0])

# Empty imports array results in no imports
fake_ctx = _ctx_with_imports([])
imports = py_library.make_imports_depset(fake_ctx).to_list()
asserts.equals(env, 0, len(imports))

# The import path is the parent package
fake_ctx = _ctx_with_imports([".."])
imports = py_library.make_imports_depset(fake_ctx).to_list()
Expand Down