Proposal: Make GitHub Releases our primary (and only) distribution of Pants wheels

[thejcannon]

As a solution for #11614 we've talked about getting off PyPI by getting on GitHub Releases.

Technical Bits

(Precursor) Automating Releases

As a precursor to this, in order to ensure releases exist to have assets be uploaded, we need to automate the release being created from the tag.

Implementation: Roughly gh releases create ...

PR: #19444

(We'd need to propagate this change backwards)

Uploading wheels on new releases

We already build wheels as part of the release, they just need uploading.

Implementation: Roughly gh releases upload ...

PR: #19444

Uploading wheels on prior releases

Although not strictly necessary, it'd be nice to upload the wheels to older releases.

Implementation: A script that downloads them from PyPI and uploads them to the release.

PR : None, but done!

pants_requirements target (and plugin support)

The pants_requirements target will now generate a python_requirement target that points directly to the GitHub Release assets for the wheels. How this is done is an implementation detail.

This would change the behavior of pinning loosely to pinning the exact version of Pants being run.

PR: #19432

Scie-pants package resolution

scie-pants will be changed such that if the version being requested is > the version is is added in, it will use the wheel from the relevant GitHub Release. (E.g. pip install pantsbuild.pants --find-links https://github.com/pantsbuild/pants/releases/expanded_assets/release_<version>).

We could have this unconditionally use the GitHub release (instead of looking at the version) if we backport. That is up for discussion.

Further changes could have us uploading a PEX and using that. That will happen in a separate discussion.

EDIT: Jumping the gun, and uploading the per-platform PEX to be used by scie-pants in upload PR

PR For using local PEX: pantsbuild/scie-pants#226
PR for backporting to using --find-links for v2 wheels: (Not yet)

Scie-pants latest version resolution

Scie-pants will be changed to use the GitHub Release with the "latest" tag to determine the latest release.

PR: pantsbuild/scie-pants#225

The breaking changes

Sunsetting PyPI

As a result of this effort, we will stop uploading packages to PyPI. No proposal is made to yank/remove existing releases.

Sunsetting binaries.pantsbuild.org for wheels/index.html

As a result of this effort, we will stop uploading wheels to PyPI as well as generating/uploading index.html (for --find-links support).

Sunsetting PANTS_SHA support

Additionally to facilitate this change, we will deprecate usage of PANTS_SHA for any Pants non-stable release version. Since the release process is getting streamlined and automated, if a user needs a wheel build we''ll just produce a stable release build for them.

Potential Future Changes

Adding release notes

GitHub supports release notes, which we likely should be uploading. We can shift the changelog process from our current PyPI-focused one to put the notes in the release. This process will be automated as part of release creation.

It will likely come after the wheel uploading change, but before true PyPI sunsetting.

Generating and hosting a simple cheeseshop HTML index

To facilitate easier installing of wheels (both in-release by avoiding the need for platform markers, and across-releases) we could generate a simple HTML document with links to each release. Then users could point to it from pip's --find-links option.

[tgolsson]

Great write-up @thejcannon! As we've discussed before this has some impact on matrix testing for plugins that do not release in-tandem with Pants, and potentially want to support multiple versions. For example; I test for previous, current, and next in my CI flows to support usage no matter where you are in an upgrade cycle. This is ironically something I do because Pants isn't stable from a plugin perspective (yet) - if it was stable; I'd care much less about that kind of testing.

With that in mind; I think this ties into my previous discussion here. Where does plugin development happen in the future? How long until we get to that point? How stable do we want the API to be then? I think the solution of using [python-repos].indexes is an OK temporary solution; but that would be included in all resolves which is a bit unnecessary and wasteful (so we'd want [python-repos].resolve-indexes -- something that'd be nice either way).

On the other hand, having to manually expand the pants_requirements with hardcoded paths seems onerous, as in the proposed new implementation. However; I'm wondering if the pants_requirements target could somehow be made generic over a range -- so you can ask for a minor version; and potentially "next-with-exact".

In the case of how I test; that'd look something like

pants_requirements(version="15", resolve="pants-215")
pants_requirements(version="16", resolve="pants-216")
pants_requirements(prerelease="17", resolve="pants-217")

Exact syntax/field names obviously up for discussion, but from my perspective being able to lock to an rc version is much more important than pinning to a specific patch release for a stable version. This is a bit counter to your thought (loose-to-tight range); but I think for plugins the goal should be to be as general as possible, and really not tied to whatever tooling you use to develop it.

[thejcannon]

Forgive me, but could that not be done in a macro on top of python_requirement today? The yuckiness is obfuscated, but the end result is the same?

I'm guessing for released versions, the pin isn't really controversial. But for the bleeding-edge Pants, the pin is... wait the bleeding edge could just use the actual pants_requirements target.

[tgolsson]

Forgive me, but could that not be done in a macro on top of python_requirement today? The yuckiness is obfuscated, but the end result is the same?

I mean; yes. Everything can be done in macro. I think my point is if our delivery method will be hard-links; then we should ensure that is smooth by making pants_requirements smarter. :-)

I'm guessing for released versions, the pin isn't really controversial. But for the bleeding-edge Pants, the pin is... wait the bleeding edge could just use the actual pants_requirements target.

Not sure which part you're talking about, but it seems controversial to me -- I think even if my development repo is on 2.16.0 I should be testing against 2.16.1 when that is out, unless I explicitly make it narrower. Bleeding edge, yes, agreed. But I'm not envisioning running my repo on 2.17a3 if that's what I need to test against.

[tgolsson]

Just to summarize: my ideal solution would look something like this:

pants_requirements(version="15", resolve="pants-215")
pants_requirements(version="16", resolve="pants-216")
pants_requirements(prerelease="17", resolve="pants-217")

Where the current pants_requirements target gets a bit smarter. What that does under the hood doesn't really matter - direct links, find-links, something else entirely... it's all a detail.

As a stopgap solution; I think requiring extra resolves on-top of python_requirements is OK;

# pants.toml
[python-repos]
find-links = ["https://something-something"]
# or
find-links-for-resolve = {
    pants = "https://something-something"
}

for resolve, spec_range in { ... }:
    python_requirements(name=resolve, resolve=resolve, requirements=[f"pantsbuild.testutil{spec_range}"])

And the absolute worst would be needing something like

PY_VER="x"
for resolve, (pants_semver, plat_tag, machine) in { ... }:
    # not enough to make it work, already a mess if it has to change.
    python_requirements(name=resolve, resolve=resolve, requirements=[
        f'pantsbuild.pants @ https://github.com/pantsbuild/pants/releases/download/release_{PANTS_SEMVER}/pantsbuild.pants-{PANTS_SEMVER}-cp{PY_VER}-cp{PY_VER}-{plat_tag}_{machine}.whl ; sys_platform == "linux" and platform_machine == "x86_64"',
        f'pantsbuild.pants.testutil @ https://github.com/pantsbuild/pants/releases/download/release_{PANTS_SEMVER}/pantsbuild.pants.testutil-{PANTS_SEMVER}-cp{PY_VER}-cp{PY_VER}-{plat_tag}_{machine}.whl ; sys_platform == "linux" and platform_machine == "x86_64"',
        f'pantsbuild.pants @ https://github.com/pantsbuild/pants/releases/download/release_{PANTS_SEMVER}/pantsbuild.pants-{PANTS_SEMVER}-cp{PY_VER}-cp{PY_VER}-{plat_tag}_{machine}.whl ; sys_platform == "darwin" and platform_machine == "arm64" and platform_release == "11.0"'
        f'pantsbuild.pants.testutil @ https://github.com/pantsbuild/pants/releases/download/release_{PANTS_SEMVER}/pantsbuild.pants.testutil-{PANTS_SEMVER}-cp{PY_VER}-cp{PY_VER}-{plat_tag}_{machine}.whl ; sys_platform == "darwin" and platform_machine == "arm64" and platform_release == "11.0"'
    ])

It just seems like a very fiddly mechanic, and makes a lot of assumptions about formats, naming, and cannot do wide ranges.

[thejcannon]

I see, so since Pants should have it's own historical knowledge I think we could asjust pants_requirements to also allow for supporting prior releases and provide the right markers.

I can incorporate that into #19432.

Not sure which part you're talking about, but it seems controversial to me -- I think even if my development repo is on 2.16.0 I should be testing against 2.16.1 when that is out, unless I explicitly make it narrower. Bleeding edge, yes, agreed. But I'm not envisioning running my repo on 2.17a3 if that's what I need to test against.

As it stands, I think the new target will allow you two choices:

• Give a version, like 2.16.0. That's precisely what you get. As for whether you get 2.16.0 or 2.16.1, I think you'll have to be the one to bump that. Is there a reason you'd prefer one over the other, other than "one is later"? I would imagine the plugin API doens't break across update versions like that
• Don't give a version, you get the exact version of whatever Pants is running as.

The crux of the issue with trying to match a range is that there is no single index of all the assets to easily match against, so for simplicity direct pins is all we get.

[thejcannon]

This is officially complete (albeit still requiring a release of scie-pants)