Should we stop building wheels for every commit in main?

[benjyw]

Currently we build Pants wheels for every (non-docs) commit in main. This is so people can use PANTS_SHA to easily run Pants at any such commit.

However this seems to be used only rarely in practice, and is very costly in CI resources, especially on self-hosted runners.

The wheel build shards take almost an hour, on each of 5 target platforms (in large part because we have no effective Rust caching). So we're burning a lot of trees to build and store wheels that will almost certainly never be used. And meanwhile PR CI runs can be delayed waiting for resources to free up.

I propose that we only build wheels when we need them for a release. If someone needs to run at a SHA, we give clear guidance on how to build those wheels as needed.

[kaos]

Concur. Rarely use PANTS_SHA from main to run stuff with. If that would be required, manually triggering a workflow to build them would be perfectly fine.

[sureshjoshi]

What's the use case for "heavy use"? Just to always grab bleeding edge, or something more particular?

[sureshjoshi]

I didn't mention before, but I'm a long-time fan of "nightly" builds

[danxmoran]

What's the use case for "heavy use"? Just to always grab bleeding edge, or something more particular?

More particular - we really don't want to ride the bleeding edge, but over the past year have had multiple instances where we need a bugfix / perf improvement / new feature in order to realistically continue our adoption of Pants. "Official" releases drop infrequently/inconsistently enough that we get antsy waiting around for them.

Nightly builds would definitely satisfy our needs - we are not that impatient 😅

[jsirois]

IIUC all such needs though are coordinated with a maintainer; so that maintainer always? knows the fix is needed by Color for testing and so the maintainer could run a manual workflow for wheels for that fix only. I just point this out because nightly is still wasteful, just less so; and manual jobs are easy to set up. Pex uses a hybrid manual form / tag push trigger for its releases for its release workflow. Manually running that workflow just means filling in a tag name. For Pants we could be filling in a sha.

[kaos]

For Pants we could be filling in a sha.

Or just the branch name..

[jsirois]

I also concur. I think it would be good, as always, to dig on why our wisdom has changed though. I find these things disturbing, and, afaict, there has been no relevant change since we deemed this wise, except maybe our sensitivity to waste / vs convenience.

[stuhood]

At least once a week, users use PANTS_SHA releases (even just according to comments in Slack, let alone unreported usage), so I think that the need is still there. But I do think that it's likely that that need can be satisfied via nightlies or manually kicked off SHA builds, or both.

[stuhood]

I'm not sure there was much collective wisdom applied, just someone had the idea to do it, and did it? Probably without realizing the cost, or this was back when the cost was a lot lower.

CI has always tested the building of wheels, IIRC: so the difference between testing the building of wheels and actually uploading them was small.

[jsirois]

Well, there's an unqualified plural there too. Has it tested our current crop of 10 wheel permutations or just one or a subset. Our current 10 is a bit out of control.

[stuhood]

Right.

Having said that, scie-pants does allow us to start limiting our need for wheel permutations (across Python versions at least) fairly soon. Leaning in there would also be a way to reduce CI costs.

[illicitonion]

FWIW one of the reasons I recall us building wheels (and pexes) on PRs is because years back, the process to build these artifacts would break and we wouldn't find out about it until release time, and it was pretty frustrating as a releaser to have to bisect through a big pile of changes, and maybe have to do multiple reverts or fix-ups if there were dependent changes.

Maybe we haven't seen this kind of breakage getting caught in PRs, and it's not worth the resources any more?

Alternatively, maybe we could make wheel-building nightly or something to keep some of the safety while removing the delay? But I suspect the exotic self-hosted runners are probably the most valuable ones to actually be testing.

[stuhood]

If someone needs to run at a SHA, we give clear guidance on how to build those wheels as needed.

What would that guidance be? It's fairly challenging to build wheels for Pants across multiple platforms. You need:

1. a CI job that looks a lot like ours
2. a place to host the wheels
3. pants script modifications to fetch wheels from that location

[jsirois]

You use our CI job, just add a form for an input SHA. ~like so: https://github.com/pantsbuild/pex/blob/e8844ec52da53671d5b8b0b87e0ba2b683e8d9a1/.github/workflows/release.yml#L6

[jsirois]

My surmise above re Color's use was that ~every instance already involves a maintainer and a fix commit. If about correct this just means the maintainer fills out a GitHub form when needed.

[stuhood]

Yea, I could be fine with that as long as it remains shared infrastructure. It just isn't something that each consuming org should build on their own.