Generate entry_points.txt for python_tests that require entry points from a python_distribution

[cognifloyd]

This adds a new entry_point_dependencies field to python_tests and python_test targets.

This allows tests to depend on a subset (or all) of the entry_points defined on python_distribution targets (only on the entry_points field, not in the provides field).

For example:

python_tests(
    name="tests",
    entry_point_dependencies={
        "//address/to:python_distro_tgt_1": ["*"],  # all entry points
        # only one group of entry points
        "//address/to:python_distro_tgt_2": ["console_scripts"],
        "//address/to:python_distro_tgt_4": ["st2common.runners.runner"],
        # or multiple groups of entry points
        "//address/to:python_distro_tgt_5": ["console_scripts", "st2common.runners.runner"],
        # or 1+ individual entry points
        "//address/to:python_distro_tgt_6": ["console_scripts/foo-bar"],
        "//address/to:python_distro_tgt_7": ["console_scripts/foo-bar", "console_scripts/baz"],
        "//address/to:python_distro_tgt_8": ["st2common.runners.runner/action-chain", "console_scripts/foo-bar"],
    },
)

A dependency defined in entry_point_dependencies emulates an editable install of those python_distribution targets. Instead of including all of the python_distribution's sources, only the specified entry points are made available. The entry_points metadata is also installed in the pytest sandbox so that tests (or the code under test) can load that metadata via pkg_resources or importlib.metadata.entry_points or similar.

Misc notes:

• I added this to the pants.backend.experimental.python backend and also registered the rules I extracted from pants.backend.experimental.python.framework.stevedore with the stevedore backend since it needs those rules.
• Unlike all other dependencies fields, entry_point_dependencies is a dict[str, list[str]]. The .keys() of the dict are logically similar to standard dependencies fields. Using a dict provides more granular control over the dependencies.
• Also unlike other dependencies fields, this field does NOT result in a dependency on the python_distribution target. Instead, that target provides the entry_points metadata to infer dependencies on the actual code for those entry points.
• I extracted most of the logic for this from the experimental stevedore plugin. Enabling the stevedore plugin is not required to use this feature.

Closes #11386
Closes #15481

[kaos]

lgtm. 👍🏽

[kaos]

How does this manifest to the user? (i.e. is this possible with BUILD file (mis)configuration? or is it merely a sign of an internal pants bug?)

What I'd prefer is to have the error message convey if this is in any way fixable when encountered, or if it's an internal bug and they need to turn to the pants project for assistance/report a bug.

[cognifloyd]

This serves to narrow the types for mypy. Everywhere that could call this rule bails out before calling it if the field is empty (See infer_entry_point_dependencies and generate_entry_points_txt_from_entry_point_dependencies). So, I expect this to only ever happen during development, not a released version.

Maybe I could raise an exception like InvalidField. Would that be better?

[cognifloyd]

I turned this into a more verbose ValueError. Thanks for the suggestion.

[kaos]

Ah, right. In that case, I think an assert is just fine, as we don't expect to ever hit it, and it's presence is only to satisfy mypy. I've done the same, but leaving a hint as to why it's there (and thus, no assert message, as it's not supposed to ever hit):

pants/src/python/pants/engine/internals/parametrize.py

Line 235 in e6b377d

assert parametrize_group is not None # Type narrowing to satisfy mypy.