Limit Request: `pantsbuild.pants` - 40 GB

[stuhood]

Project URL

https://pypi.org/project/pantsbuild.pants/

Does this project already exist?

• Yes

Which limit do you want increased?

total project size

New limit

40 GB

Which indexes

PyPI

Reasons for the request

Hey folks!

The https://www.pantsbuild.org/ project has been releasing to PyPI for 7 years, initially with weekly stable releases, briefly with quarterly stable releases, and for the last few years with monthly stable releases. We've accumulated a lot of releases as a result.

We have used the pypi-cleanup tool to delete as many of our dev and rc releases as possible, but we're getting into the danger zone of considering whether some releases are too recent to delete, and we're probably a few months away from needing to delete stable releases. See pantsbuild/pants#11614 for more backstory.

Our wheel(s) contain 30% native code (which makes up most of the total volume on disk) and 70% python. Unfortunately, we are not able to publish a universal wheel (because APIs are not stable enough across 3.x minor versions), so we end up needing to publish across multiple Python versions.

We're asking for our limit to be increased from 20GB to 40GB (doubled) in order to avoid needing to bug you all again in the next few years (fingers crossed).

Thank you for your consideration!

Code of Conduct

• I agree to follow the PSF Code of Conduct

[pradyunsg]

Collecting the relevant numbers to help other moderators/admins make a decision here.

Looking at https://pypi.org/project/pantsbuild.pants/#history, the project get a new release roughly every 7-10 days and each release seems to be ~200 MB (two ~60 MB manylinux wheel + four ~9MB MacOS wheels + an sdist).

With an additional 20 GB, that'll give this project ~20000 MB more, which is ~100 releases which would last... two-three years at their current release cadence and size, without deletions.

The project seems to regularly trim their older developement and pre-releases though! Between 1 April 2021 to 15 Oct 2021, there have been 55 releases. Of this, 49 are pre-releases (17 dev, 32 rc). The six stable releases are 2.7.0, 2.6.1, 2.6.0, 2.5.2, 2.5.1 and 2.5.0. From 1 April 2020 to 15 Oct 2020, there are currently only six releases (1.30.1, 1.30.0, 1.29.0, 1.28.0, 1.27.0, 1.26.0).

I think it is reasonable to expect that they'll continue to trim older pre-releases like this, with which they'll end up with 12 releases every year, which would last >8 years.

[pradyunsg]

Question for @stuhood: What's the story with the 60 MB manylinux wheel vs the 9 MB MacOS? What makes it so much bigger than the MacOS wheels?

[pradyunsg]

My thoughts on this one: This is a reasonable request overall. The project is well established and is generally mindful of their PyPI usage (they have a step of deleting stale releases in their release process -- pantsbuild/pants#13234).

This is a big enough request though, that I'd like to wait on at least one more moderator / admin to comment before actioning on this. :)

[stuhood]

Thank you for the very thoughtful analysis!

Question for @stuhood: What's the story with the 60 MB manylinux wheel vs the 9 MB MacOS? What makes it so much bigger than the MacOS wheels?

I think that the situation there is that on Linux many system APIs end up embedded in the binary (which improves compatibility), whereas macOS makes that effectively impossible, and requires that you dynamically link in more cases.

But: we also ship with debug symbols in order to improve debuggability (which recently helped us here, for example). That doesn't explain the Linux/macOS difference, but disabling/stripping debug symbols would likely shrink these significantly.

If the PyPI community would prefer that we disable those moving forward, we would be willing to: we would still need an increased limit, but possibly not quite as much.

[pradyunsg]

I think it's worthwhile to investigate how much of a difference it'd make.

Right now, the Linux wheels are ~7x the size of MacOS wheels.

If they come down to around the same size as the MacOS wheels, that would make your entire releases be 9x7 = 63 MB, which is about as big as a single Linux wheel right now. With such reduction, the requested limit increase would be sufficient for >300 releases, which is >20 years.

[pradyunsg]

This is a big enough request though, that I'd like to wait on at least one more moderator / admin to comment before actioning on this. :)

This is basically what we're waiting on now. I've pinged the relevant people, who might be able to take a closer look when they have time.

[ewdurbin]

Project total size for pantsbuild.pants increased to 40GiB.

[stuhood]

Thank you!