-
Notifications
You must be signed in to change notification settings - Fork 532
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: multi-toolchain support #846
Conversation
c6eef27
to
70fa529
Compare
This adds support for multiple Python versions on the same Bazel workspace. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
A py_test using 3.10 runs a py_binary using 3.9. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
See comment in code for the reasons. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
re: specifying the python version using the rule name vs attribute: you don't necessarily need to rename, or follow what I did; see below: rules_python/examples/multi_python_versions/tests/BUILD.bazel Lines 1 to 4 in 1053ab0
I can think of a few cases on a monorepo for doing the renaming. The most obvious would be a monorepo where a library is shared between multiple projects, and those projects end up having different python version requirements. The maintainers of the shared library will want to assert the tests pass on all python versions, so a rename like this would be required. Otherwise, it makes more sense to just change the import statement on a BUILD file and it's like magic to move to another version. E.g. when adopting this new feature: -load("@rules_python//python:defs.bzl", "py_binary", "py_test")
+load("@python//3.8:defs.bzl", "py_binary", "py_test") re: transitioning platforms: I pushed 1053ab0 with this. Indeed much simpler. Thanks! |
This avoids confusion with the global `rule` https://bazel.build/rules/lib/globals#rule. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
…support Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly looks good! Ok to merge after addressing comments.
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
c60f069
to
3316928
Compare
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
The previous behaviour was that location expansion on native rules would accept binary targets as inputs and resolve to a single location. This was not an option on custom rules. This patch aligns the behaviour of Starlark's ctx.expand_location to the native rules. E.g. a py_binary can be expanded using location/execpath/rootpath instead of previously only locations/execpaths/rootpaths. For more context, see bazelbuild/rules_python#846. To be able to add the multi-toolchain feature to rules_python, I had to trick `ctx.expand_location` using the `tools` attribute while this is not fixed. Closes #16381. PiperOrigin-RevId: 502466606 Change-Id: I3977f2dd479a55308bdda982588697fb04abcedf
#846 introduced this dependency, and solved the broken examples by installing the dependency only for our own code. However, this was an inintentional breaking change for users, who may not have had bazel_skylib installed in their workspace. This effectively reverts #370. At that time we didn't want any dependencies, because managing them under Bazel's WORKSPACE semantics is so difficult for users. Now that bzlmod has reached General Availability in Bazel 6, such dependencies can be managed more easily. This also allows us to introduce bzl_library calls in our BUILD files, making it less brittle to ensure that users can generate docs for their rules/macros which load from rules_python.
The previous behaviour was that location expansion on native rules would accept binary targets as inputs and resolve to a single location. This was not an option on custom rules. This patch aligns the behaviour of Starlark's ctx.expand_location to the native rules. E.g. a py_binary can be expanded using location/execpath/rootpath instead of previously only locations/execpaths/rootpaths. For more context, see bazelbuild/rules_python#846. To be able to add the multi-toolchain feature to rules_python, I had to trick `ctx.expand_location` using the `tools` attribute while this is not fixed. Closes #16381. PiperOrigin-RevId: 502466606 Change-Id: I3977f2dd479a55308bdda982588697fb04abcedf
* feat: multi-toolchain support This adds support for multiple Python versions on the same Bazel workspace. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * feat: cross-version testing A py_test using 3.10 runs a py_binary using 3.9. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: error message Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * doc: add link to bazelbuild/bazel PR fixing expand_location Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: set environment variables for py_binary too Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * test: extra case for default version taking another version Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: fail if args attribute is set Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: remove confusing output with same target name Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: buildifier Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * revert: use testing.TestEnvironment See comment in code for the reasons. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: linting issues Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: black linter Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: move tests to a sub-dir Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * feat: add multi_pip_parse Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: add missing aliases Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: use requirement function in example Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: deleted packages Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: update generated docs Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: version checking of the rule is already done by other tests Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: add python_interpreter_target to multi_pip_parse Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: windows Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: unify py_test and py_binary transition impls Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: test compatible with all platforms Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * rebase: adjust multi_python_versions on ci Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: use usr flags instead of platforms in transition Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: rename rule -> rule_impl This avoids confusion with the global `rule` https://bazel.build/rules/lib/globals#rule. Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: reduce repetition of args Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: missing test and binary-specific attributes Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: add srcs and deps attrs for path expansion Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: missing bazel_skylib on integration tests Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * refactor: use ctx.target_platform_has_constraint over select Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * doc: why symlink <name>.zip under Windows Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: apply suggestions from code review Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: use incoming edge transitions Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: use RunEnvironmentInfo when available Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> * fix: cfg should be target not exec Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com> Signed-off-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
PR Checklist
PR Type
What kind of change does this PR introduce?
What is the current behavior?
Most mono repos using Bazel have the problem of needing support for multiple Python versions. Currently, there's no simple way to accomplish this.
What is the new behavior?
Add support for registering multiple Python toolchains with different versions as an opt-in feature so that users can gradually migrate portions of their mono repos to this new feature.
Using load statements to override
py_binary
andpy_test
, this feature implementation also plays well with the Gazelle extension by using Gazelle's# gazelle:map_kind from_kind to_kind to_kind_load
feature. Also, pure Python libraries can take advantage of this by testing against multiple Python versions in the same BUILD file (see examples).This feature is achieved by using platform transitions behind the scenes.
Does this PR introduce a breaking change?
Other information