Support bzlmod

[kormide]

PR Checklist

Please check if your PR fulfills the following requirements:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature (please, look at the "Scope of the project" section in the README.md file)
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• Other... Please describe:

What is the current behavior?

The requirements file is parsed in a python script that generates requirements.bzl and passes information from the file into whl_library called by install_deps within the generated file. In order to declare the wheel repositories in a bzlmod extension, we need to be able to parse that same information in Starlark.

What is the new behavior?

Can parse a requirements file in Starlark which will be uses in a pip_parse bzlmod extension. This in in a similar vein to how rules_js parses a pnpm lockfile in starklark for the npm_translate_lock extension.

Does this PR introduce a breaking change?

• Yes
• No

Other information

This isn't currently used, but it's a nice reviewable chunk for the bzlmod work.

[kormide]

fyi: @alexeagle, @Wyverald

[alexeagle]

My understanding is that something was added after you wrote the yaml parser, which permits one module extension to see a workspace that was created by another, and so perhaps that means it's possible to fetch some tool (like yq) and then execute it from a module extension.

Perhaps that means we could use an existing requirements.txt parser like from pip or poetry that's likely to handle all the edge cases correctly. I doubt it, but might be worth a bit of poking.

[alexeagle]

Maybe @hrfuller can help with reviews as well, he has a good memory and wrote the pip_parse stuff.

[alexeagle]

do you only need the name for the module extension implementation, and not the hashes? I thought you'd need all the data required to produce a requirements.bzl file, like the hashes for downloading the packages.

OH I remember - the pip_parse rule doesn't actually know how to download anything, it will still end up calling pip install on individual packages which are required by the build...

[kormide]

Yeah this is the same requirement info that gets passed to whl_library.

[kormide]

My understanding is that something was added after you wrote the yaml parser, which permits one module extension to see a workspace that was created by another, and so perhaps that means it's possible to fetch some tool (like yq) and then execute it from a module extension.

Perhaps that means we could use an existing requirements.txt parser like from pip or poetry that's likely to handle all the edge cases correctly. I doubt it, but might be worth a bit of poking.

Yeah originally I wanted to call pip_parse in the module extension but have it output an extra JSON file containing the result of parsing requirements that I could easily load in starlark. But where I got stuck was finding some way to load that file from the pip_parse repo within the extension. I'm not sure that's possible. But maybe I could call module_ctx.execute to execute a python script that uses the same parsing library? Can a module extension create and own a file like that?

[Wyverald]

Yeah originally I wanted to call pip_parse in the module extension but have it output an extra JSON file containing the result of parsing requirements that I could easily load in starlark.

That's not directly possible (you can't declare a repo in an extension and immediately read from that repo), but you can use the trick mentioned by Alex: have an extension declare a pip_parse repo, use_repo that repo in your module, and then load from it in another extension.

But maybe I could call module_ctx.execute to execute a python script that uses the same parsing library? Can a module extension create and own a file like that?

Alternatively, this works too. A module extension can create files, but these files are not addressable after the extension impl function finishes.

[kormide]

Yeah originally I wanted to call pip_parse in the module extension but have it output an extra JSON file containing the result of parsing requirements that I could easily load in starlark.

That's not directly possible (you can't declare a repo in an extension and immediately read from that repo), but you can use the trick mentioned by Alex: have an extension declare a pip_parse repo, use_repo that repo in your module, and then load from it in another extension.

But maybe I could call module_ctx.execute to execute a python script that uses the same parsing library? Can a module extension create and own a file like that?

Alternatively, this works too. A module extension can create files, but these files are not addressable after the extension impl function finishes.

Thanks for the clarification. That's interesting that a module context can create temporary files. I think one extension is a better developer experience than two, so I'll look into that approach and see where it leads me.

[hrfuller]

Cool use of a state machine, thanks for working on this.

[groodt]

Looks good. I've got a different proposal to consider and wondered what you might think.

1. Update rules_python to consume a pip "installation report" instead of a transitively locked requirements.txt https://pip.pypa.io/en/stable/reference/installation-report/
2. Parse the locked json report with https://bazel.build/rules/lib/json#decode

The benefits of the json installation report has a few benefits:

• Easier to parse
• It has the direct urls
• It captures the environment (sys info) of the locking platform
• While it isn't presently an agreed "pip lockfile", it is easier to parse and has a clearer spec than a requirements.txt file.

The direct url info in particular will be helpful in future, because it would enable us to parse the format into .bzl http_file or http_archive for retrieval in repository rules using the bazel machinery without having to invoke pip.

[Wyverald]

The direct url info in particular will be helpful in future, because it would enable us to parse the format into .bzl http_file or http_archive for retrieval in repository rules using the bazel machinery without having to invoke pip.

This might be built into Bzlmod in the future, without you guys having to do anything. With the planned lockfiles feature, Bzlmod will store the results of module extension resolution in the lockfile; said result is actually just the names and definitions of the repos that the extension generated.

[groodt]

The direct url info in particular will be helpful in future, because it would enable us to parse the format into .bzl http_file or http_archive for retrieval in repository rules using the bazel machinery without having to invoke pip.

This might be built into Bzlmod in the future, without you guys having to do anything. With the planned lockfiles feature, Bzlmod will store the results of module extension resolution in the lockfile; said result is actually just the names and definitions of the repos that the extension generated.

That would be something different to what I mention above. That would be for locking the bazel rules dependency graph, not the dependency graph(s) of resolved python packages from external sources. There is no mechanism for bzlmod to do that presently because we aren’t capturing it presently. Hence my line of questioning.

[alexeagle]

I don't think it's ever a good idea for Bazel to manage the lockfile when a language-idiomatic package manager manages the dependencies and constraints. Then you get a different result outside of Bazel and the language tooling gets confused.

Anyhow this PR is just to make a bzlmod module extension for the existing repository rules, not to redesign them.

[kormide]

I'm going to hold off on this for now and see if I can just execute a python script in the module extension to do the parsing (using the same library that pip_parse uses).

[fmeum]

Thanks for the clarification. That's interesting that a module context can create temporary files. I think one extension is a better developer experience than two, so I'll look into that approach and see where it leads me.

@kormide Note that even though rules_python would have to define two module extensions, end users would not need to interact with the one that provides the tools (such as yq) for the other.

[kormide]

Thanks for the clarification. That's interesting that a module context can create temporary files. I think one extension is a better developer experience than two, so I'll look into that approach and see where it leads me.

@kormide Note that even though rules_python would have to define two module extensions, end users would not need to interact with the one that provides the tools (such as yq) for the other.

That's good to know. I had ruled that out because I thought that users would have to declare both extensions.

I'm going to put up a solution with a single module extension first because I'm close to getting that working, and if it feels like there's too much duplication I can go back to the two extension approach.

[kormide]

I repurposed this PR to include the full bzlmod solution.

Commit 1: requirements starlark parser (added support for parsing pip options)
Commit 2: bzlmod extension
Commit 3: bzlmod release support (workspace snippet, BCR app config)

I ended up going with the starlark parsing of requirements as an initial solution here. The alternative is to add an additional module extension that parses the requirements file and outputs something like JSON that starlark can read. In the end, (I think) we need to have the whl_library declarations inside of the pip_parse extension, hence the need for either a parser or another extension to load the wheel arguments.

[kormide]

This is a template for an entry that will appear the Bazel Central Registry (example).

I wasn't sure who the main maintainers of rules_python are. @rickeylev @groodt are you okay with being listed here?

[rickeylev]

SGTM

[f0rmiga]

@kormide you can also add me.

[kormide]

This points calls to xyz_requirement(...) to the aliases inside the pip_parse repo, so that we don't need to declare a bunch of use_repos in MODULE.bazel.

[kormide]

Just realized that I still need to support registering the python toolchain via bzlmod. The tests are failing on CI because the bzlmod example doesn't use a consistent python version on all platforms (I think it's using whatever the executor has installed), and it's outputting the python version to the vendored requirements file, causing a diff_test to fail.

[kormide]

Added a module extension for registering a python toolchain.

[fmeum]

Does every root module have to manually register a toolchain or is the host Python used if nothing else is specified?

[kormide]

It's optional and will use the host if not specified. I added a comment indicating that in the release workspace_snippet.

[kormide]

In the case of this example, I had to use it to fix the python version because the generated requirements_lock.txt file stamps the python version in a comment, and on CI the different platforms had different versions installed, so the diff_test was failing.

[kormide]

This PR needs another review since adding the full bzlmod solution, but it's not clear to me who I should ping for that.

[f0rmiga]

@alexeagle is probably the best person to do the final review.

[alexeagle]

nice! sorry it took me a while to get to this

[alexeagle]

for later: it's gross how this repository rule depends on packages from pypi, it should be self-contained and require only the python interpreter like what we did in rules_js