feat(toolchains): support runtime registration from manifest#3802
feat(toolchains): support runtime registration from manifest#3802rickeylev wants to merge 3 commits into
Conversation
Currently, all supported Python runtime versions and their platform-specific metadata (URLs, SHA256s, strip_prefix) must be hardcoded in `python/versions.bzl`. This makes it slow and difficult to adopt new Python versions or custom builds without updating `rules_python` itself. This PR introduces the ability to dynamically fetch and register Python runtimes from a remote python-build-standalone (PBS) manifest file (e.g., `SHA256SUMS`). This is supported via two new attributes in `python.override`: - `add_runtime_manifest_urls`: A list of URLs pointing to manifest files to parse and register. - `runtime_manifest_sha`: The SHA256 hash of the manifest file.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for dynamically fetching and registering Python runtimes from a python-build-standalone manifest file using python.override(add_runtime_manifest_urls = ..., runtime_manifest_sha = ...). It includes a new Starlark parser for the manifest files, integration into the toolchain configuration logic, and comprehensive unit and integration tests. Critical feedback has been provided regarding the use of unsupported string methods (removesuffix/removeprefix) in Starlark, a runtime error in handling mctx.download results without allow_fail = True, and a locale-dependent date parsing issue in the integration tests.
This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.
This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.
This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.
rickeylev
force-pushed
the
feat.toolchain.manifest
branch
3 times, most recently
from
57479a5 to
85e7778
Compare
…ce mode Workspace builds running under older Bazel versions do not support Bzlmod module extensions. This commit introduces Starlark helper macros to conditionally gate and register Bzlmod-specific unit tests, resolving test suite loading crashes in workspace CI jobs.
Removes the external test_helpers.bzl under tests/python/ and inlines the register_python_tests macro directly inside python_tests.bzl. This simplifies the test suite layout while keeping legacy workspace gating completely intact.
aignas
left a comment
aignas
left a comment
There was a problem hiding this comment.
Thank you. This was on my mind for some time, so thanks for taking a stab at it.
What do you think about doing the following:
- Registering the manifests we use in our
MODULE.bazelfile - Changing the rule that prints the contents that we should paste to
python/versions.bzlto instead update the file with what is defined in theMODULE.bazelfile. This way we know the manifest files are interpreted in exactly the same way as we would normally do it.
| add_runtime_manifest_urls = [ | ||
| "https://github.com/astral-sh/python-build-standalone/releases/download/20260414/SHA256SUMS", | ||
| ], | ||
| runtime_manifest_sha = "ce18fdfd47c66830a40ea9b9e314a14b1636bbfd684501bc5ca1fc6d55a7933f", |
There was a problem hiding this comment.
This is really good. Maybe we could dog-food the API and update the python/versions.bzl.
| add_runtime_manifest_urls = [ | ||
| "https://github.com/astral-sh/python-build-standalone/releases/download/20260414/SHA256SUMS", | ||
| ], |
There was a problem hiding this comment.
I would love to support labels or files that users define themselves. That way they can be in the repo and we don't need to have a custom dictionary in the python/versions.bzl anymore, we could just have a list of files.
| if plus: | ||
| python_version = left | ||
| build_version, sep, rest = tail.partition("-") | ||
| if not sep: | ||
| return None | ||
| else: | ||
| python_version, sep, rest = left.partition("-") | ||
| if not sep: | ||
| return None | ||
| build_version = "" |
There was a problem hiding this comment.
What about using our own PEP440 compliant parser?
The format 3.10.20+20260414 is compliant.
We could the use the version comparison function to do the ordering.
| if expects_full != is_full: | ||
| continue |
There was a problem hiding this comment.
If there is install_only and full, which one will we prefer? I think we should prefer install_only due to size.
| if is_full: | ||
| v_dict.setdefault("strip_prefix", {})[matched_platform] = "python/install" | ||
| else: | ||
| v_dict.setdefault("strip_prefix", {})[matched_platform] = "python" |
There was a problem hiding this comment.
We should do elif here because this is install_only path. other types should be ignored.
This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.
This mock module was extracted from bazel-contrib#3802. It provides a helper for defining a mock module for the python bzlmod extension, which is standalone and generally useful for testing rules_python's bzlmod extension.
2 participants
Currently, all supported Python runtime versions and their
platform-specific metadata (URLs, SHA256s, strip_prefix) must be
hardcoded in
python/versions.bzl. This makes it slow and difficultto adopt new Python versions or custom builds without updating
rules_pythonitself.This PR introduces the ability to dynamically fetch and register
Python runtimes from a remote python-build-standalone (PBS) manifest
file (e.g.,
SHA256SUMS).This is supported via two new attributes in
python.override:add_runtime_manifest_urls: A list of URLs pointing to manifestfiles to parse and register.
runtime_manifest_sha: The SHA256 hash of the manifest file.The manifest file format is the python-build-standalone SHA256SUMS
format (
SHA FILENAME), extended to allow arbitrary URLs for thefilename (to allow more arbitrary locations).