Using Pants-built python_distribution wheel without resolve parameterization

[HeroCC]

Hi Pants peeps! I have a question on the interaction between python_requirement and python_distribution.

We’re trying to model an internal Python library as a Pants-built wheel, then consume that wheel from another project in the same repo without first uploading it to an external package registry.

The important constraint: the consuming project should own its dependency versions through its own resolve. The library wheel should behave like a normal pip package artifact whose metadata participates in dependency resolution, rather than forcing the consumer to use the library project’s resolve.

Extra Details

Simplified tree:

  repo/
    libs/
      dd_internal_pyspark/
        src/python/
          PANTSBUILD
          pyproject.toml
          dd_internal_pyspark/
            PANTSBUILD
            __init__.py
            dd_pyspark.py
            dd_spark_session.py

    subprojects/
      pyspark_smoke_testing/
        PANTSBUILD
        3rdparty/python/
          PANTSBUILD # <---- this is where we include dd-internal-pyspark
          lockfile.json
        src/python/pyspark_smoke_testing/app/pyspark_job/
          PANTSBUILD
          pyspark_smoke_testing_job_writer.py
      some_other_pyspark_task/
        PANTSBUILD
...

We have separate resolves:

[python.resolves]
dd_internal_pyspark = "libs/dd_internal_pyspark/3rdparty/python/lockfile.json"
pyspark_smoke_testing = "subprojects/pyspark_smoke_testing/3rdparty/python/lockfile.json"
some_other_pyspark_task = "subprojects/..."

The relevant macro is basically:

def wheel(distribution_name: str, version: str, **kwargs):
    kwargs["sdist"] = False
    kwargs["wheel"] = True
    kwargs["name"] = distribution_name
    kwargs["generate_setup"] = kwargs.get("generate_setup", True)

    resources(name="wheel_resources", sources=["pyproject.toml", "*.md"])
    kwargs["dependencies"] = kwargs.get("dependencies", []) + [":wheel_resources"]

    python_distribution(
        output_path=f"wheels/{distribution_name}/{version}",
        provides=python_artifact(
            name=distribution_name,
            version=version,
        ),
        **kwargs,
    )

The consumer currently declares the library as a normal third-party requirement:

python_requirement(
    name="dd-internal-pyspark",
	# we package the library in one PR, merge it, then update this version in a second PR
    requirements=["dd-internal-pyspark~=2.0.0"], 
)

python_requirement(
    name="pyspark",
    requirements=["pyspark"],
)

python_requirement(
    name="numpy",
    requirements=["numpy>=2.3.0"],
)

# other libraries...

But our goal is to consume the locally built libs/dd_internal_pyspark/src/python:dd-internal-pyspark wheel instead of going back and forth between a registry. Something like this

python_requirement(
    name="dd-internal-pyspark",
    requirements=[
        "dd-internal-pyspark @ libs/dd_internal_pyspark/src/python:dd-internal-pyspark",
    ],
)

We also tried including it directly as a dependency on the python_sources, but this does not give us the behavior we want. Pants treats the distribution’s source/dependency closure as part of the target graph, so all targets need to share the same resolve. That makes the library’s own resolve leak into the consumer.

We do not want to solve this by parametrizing the library over every consumer resolve. We want the consumer resolve to remain authoritative, e.g. pyspark_smoke_testing should decide its numpy, pyspark, etc. versions, and the local wheel should participate like a regular package artifact.

TLDR

Is there a Pants-native way to say: build this python_distribution target, then use the resulting wheel as the artifact for a python_requirement in another resolve?

We’re trying to understand what model Pants expects here, especially for monorepos with internal Python packages where consumers should retain independent resolves.

[HeroCC]

I made this a discussion since I'm not sure if this is a feature request or if I'm simply using it wrong. Looked around and haven't seen any similar discussions before. Happy to make this into a proper issue & feature request if you think it trends more towards that.

[benjyw]

I can't think of a way to do this at present. This has been floated before (see #18254). See that issue and the PR it links to for more context. Pants will build the wheel and put it in the sys path for test, run and repl, but it doesn't do anything special for packaging, and it doesn't attempt to figure out conflicting resolves.

But you might be able to work around this using the !! operator to subtract the deps that are causing the conflicts? That might be a long and annoying list to maintain though.

Otherwise, I think we'll have to tackle this more substantially, somehow.