Port Python rules to Starlark

[rickeylev]

This is the public tracking issue for the effort to port the Python rules from Java to Starlark.

I'll provide status updates here upon request or when there are particularly notable developments.

Internally, there's been a lot of progress on this. The work has stayed internal so far simply because, internally, Google has quite a few modifications to the rules and I am not/wasn't entirely sure how feasible things would be in practice, and I simply don't have time to, or experience in, managing the Google->Github side of things. Things have gone pretty good overall, though; we're very close to enabling py_binary and py_test within Google.

The rough plan is to re-implement the rules using the "builtins" mechanism of Bazel. This basically allows Bazel to bundle Starlark code with itself and provides mechanisms for Java<->Starlark interop, in particular ways for Starlark to call private APIs the Bazel team isn't sure should be generally public or not. While this means you would still have to wait for Bazel release to get changes, modifications should be much easier and accessible.

When and how to transition out of Starlark builtins to e.g. rules_python, is still TBD. That's probably going to depend on behavioral differences and the extent of reliance on the Java<->Starlark builtins interop.

Status (March 2023)

• All the rules are using the Starlark implementation
• Porting providers is in-progress

Within Google, we have the Starlark implementation enabled. So rewriting in Starlark is possible and works; just a matter of tracking down edge cases now.

Blockers

• None at the moment

Incompatibilities (or other observable changes in behavior)

• No python 2 support: as part of Bazel 7, support for Python 2 is being removed.
  This applies to the Java-implemented rules, too.
• --experimental_build_transitive_python_runfiles=true not supported: This is
  being dropped as part of Bazel 7 and applies to the Java-implemented rules, too.
  This feature enables an obscure behavior where building one target materializes
  the runfiles of dependent binaries as if they were directly built.
• stamp attribute can no longer be a configurable (select-based) boolean value.
  Integers (select-based or not) are still supported (and preferred).
• legacy_create_init attribute can no longer be a configurable (select-based boolean
  value. Integers (select-based or not) are still supported (and preferred).
• In cquery commands, the provider name as returned by the providers() function
  has changed. These "new" names are considered internal and should not be relied upon;
  they will change again when the rules are removed from Bazel and moved to the
  rule_python repository.
  • PyInfo -> @_builtins//:common/python/providers.bzl%PyInfo
  • PyRuntimeInfo -> @_builtins//:common/python/providers.bzl%PyRuntimeInfo
• For py_test to add the requires-darwin execution requirement, the Apple
  OS constraints (@platforms//os:{ios,macos,tvos,watchos}) must match. Setting
  --cpu flag (e.g --cpu=darwin_x86_64) isn't sufficient. Note that this only
  affects remotely running a py_test built for Apple platforms.

TODO

• Implement imports attribute
• PyInfo provider
• PyRuntimeInfo provider
• PyCcLinkParamsProvider provider
• Port flags to Starlark
• py_binary / py_test
• py_library
• py_runtime
• Re-enable allowing using the Starlark impl for tests where it was disabled for Google-internal reasons (e.g. imports attribute)
• Bazel regression tests passing
• Python 2 disabled
• experimental_build_transitive_python_runfiles disabled
• Enable the rules
• Enable the providers
• Fix downstream project failures

cc: @comius @brandjon

[UebelAndre]

Thanks for filing this! Is there also a tracking issue for the cc rules port (and I suppose all the other built in rules)?

[rickeylev]

Per request, I've put together a public doc that talks about hurdles for moving the implementation over to Starlark. There's still more in the internal doc I need to move over, but it currently covers the major parts.

https://docs.google.com/document/d/1-dC4QAxX67r7qK77D5nrnpykBgZEN-I9KBGTBVZba4M/edit?resourcekey=0-jYhsNDu8EKga8kTuMX4u9Q#

edit: corporate security policies prevent me from making the doc actually public. Press the "request access" button and I will grant access.

[GregBowyer]

@rickeylev does this cover hermetic toolchains ala bazelbuild/rules_python#691 ?

[rickeylev]

The toolchain rule and providers will also move into Starlark, yes.

[GregBowyer]

Sorry I will be more clear, with the rule and providers does this go towards working through your concerns and considerations as per bazelbuild/rules_python#691 (comment)

I ask because right now, we run a very hermetic world using bazel and NixOS, but this leads us to hacks like ticketmaster/rules_python@299cb89.

In a twist of fate we would be totally hermetic and deterministic with NixOS, but bazel actions tend to (rightfully) eat the $PATH which makes things unable to perform /usr/bin/env python3

These hacks work; however, there are limitations with caches, remoting and deterministic builds that would be good to start resolving.

[rickeylev]

Ah ok. The rewrite into Starlark doesn't directly address those problems. It does make it much easier to address those problems, though, because (1) you can use Starlark instead of Java, and (2) changes won't be coupled to Bazel releases