feat(py_wheel): deps can provide requirement specifiers for Requires-Dist

[pcj]

This feature adds two new providers (PyRequirementInfo and PyRequirementsInfo) to py_wheel.bzl.

Downstream users can use these Providers to supply additional requirement specifiers to the generated wheel metadata.

The use case is that we'd like wheels to contain files limited to transitive dependencies not including those which are available from pypi directly. If we use py_package out of the box, we get "fat wheels" that contain too many files (all of protobuf, for example). Rather, we'd like to be able to have our own "thinner" implementation of py_package that filters protobuf out, but also include Requires-Dist: protobuf>=4.25.2 in the wheel metadata.

This is similar but more limited in scope than #1182.

[pcj]

Example py_project rule that demonstrates usage:

"Implementation of py_package rule"

load("@rules_python//python:packaging.bzl", "PyRequrirementInfo", "PyRequrirementsInfo")

PYPI_LOCK_PREFIX = "../pypi/_lock/"

def _py_package_impl(ctx):
    inputs = depset(
        transitive = [dep[DefaultInfo].data_runfiles.files for dep in ctx.attr.deps] +
                     [dep[DefaultInfo].default_runfiles.files for dep in ctx.attr.deps],
    )

    files = []
    requirements = []

    for input_file in inputs.to_list():
        filename = input_file.short_path
        if filename.startswith(PYPI_LOCK_PREFIX):
            filename = filename[len(PYPI_LOCK_PREFIX):]
            name, version = filename.split("@")
            if name and version:
                requirements.append(PyRequrirementInfo(
                    name = name,
                    version = version,
                    specifier = "%s>=%s" % (name, version),
                ))
        else:
            files.append(input_file)

    return [
        DefaultInfo(
            files = depset(direct = files),
        ),
        PyRequrirementsInfo(
            label = ctx.label,
            requirements = requirements,
        ),
    ]

py_package = rule(
    doc = """\
A rule to select all files in transitive dependencies of deps which
belong to given set of Python packages.

This rule is intended to be used as data dependency to py_wheel rule.
""",
    implementation = _py_package_impl,
    attrs = {
        "deps": attr.label_list(
            doc = "",
        ),
    },
)

The _lock/ prefix is specific to https://github.com/jvolkman/rules_pycross

[aignas]

I like the idea but was wondering if the py_project rule should be part of the rules_python, at least as an example if we are not sure that we want to increase the API surface we are maintaining. What do you think, @pcj?

[pcj]

I'm happy to include it, perhaps under a directory like examples/pycross_project/ since this particular implementation is specific to rules_pycross?

cc @jvolkman

[jvolkman]

Aren't specifier and version redundant? Do you need both?

[pcj]

Yes it is redundant. I left version in because I thought someone might want to know the concrete version rather than having to parse the specifier, but since I don't have a specific use for it myself, should probably be removed for the sake of simplicity (could be added later)

[jvolkman]

Does it make sense to also include markers?

[pcj]

Probably? I'm somewhat of an imposter when it comes to python, but after doing more reading it would make sense to include them (List[String]?)

[jvolkman]

I'm happy to include it, perhaps under a directory like examples/pycross_project/ since this particular implementation is specific to rules_pycross?

cc @jvolkman

Not sure it makes sense to have this example in rules_python as written, with dependencies on _lock (a sort-of-implementation-detail, although it's described in a comment) and pypi (a user-specified repo name).

Could the example be retooled to rely only on stuff in rules_python?

[pcj]

Added examples/custom_py_project/

[aignas]

This will deplete the number of runners we have available for running builds. I remember @rickeylev did some work to optimize this so maybe relying on the bazel integration test framework would be better here?

[aignas]

nit: please rebase and put the bullet point into an appropriate section.

[aignas]

Any new example code should be compatible with bzlmod, IMHO so that it is easier to to maintain the examples in the future. Are there reasons to not do this in this case?

[pcj]

I just copied over the files from examples/pip_parse_resolved, but perhaps a different example would be better as a base. Alternatively, I can just remove the WORKSPACE bits.

[aignas]

This may break if we merge #1766, I wonder if there is a better way to do this.

[github-actions]

This Pull Request has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions]

This PR was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"