Project Limit Request: ray-cpp - 100 GB

[aslonnie]

Project URL

https://pypi.org/project/ray-cpp/

Does this project already exist?

• Yes

New limit

100GB

Update issue title

• I have updated the title.

Which indexes

PyPI, TestPyPI

About the project

Ray is a unified framework for scaling AI and Python applications.

https://github.com/ray-project/ray

How large is each release?

Around 500MiB in total for a full release, including wheels for Linux amd64/arm64, MacOS Intel/apple silicon, and Windows amd64. We do full release for all platforms around every 6 weeks, and to Linux and MacOS every week.

How frequently do you make a release?

Weekly for Linux and MacOS (when we can); every 6 weeks for Windows.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[aslonnie]

any action I am still required?

[cmaureir]

Hey @aslonnie 👋
Usually when a request does such a significant jump, it's difficult to simple approve it.
The project is currently sitting on 20GB and a jump to 100GB is a lot.

Additionally, weekly releases does increase the project size significantly.
As a comparison, you can always check the general stats https://pypi.org/stats/ and see that only a few projects have been granted a large space, mainly due to backs and forth regarding mechanism to reduce the sizes, etc.

In the case of ray for example, you are providing 1-wheel-per-python version. Have you explored the Python Limited API https://docs.python.org/3/c-api/stable.html in order to provide 1-wheel-per-os-arch ?

Are there some common components for all the wheels that could be distributed as a dependency to avoid having them in each wheel?

[aslonnie]

The project is currently sitting on 20GB and a jump to 100GB is a lot.

how about 40GB or 50GB? can we pay for larger space?

In the case of ray for example, you are providing 1-wheel-per-python version. Have you explored the Python Limited API https://docs.python.org/3/c-api/stable.html in order to provide 1-wheel-per-os-arch ?

we can have a look, though I imagine it won't be a trivial effort to pull it off.

---

fwiw, we have been deleting past / old rc releases to make space for new releases.

[aslonnie]

looking at https://pypi.org/stats/ , actually ray is at 60+GB already..

[can-anyscale]

@cmaureir We understand your suggestion for a better mechanism to reduce the size and are actively working on a more sustainable solution. However, we urgently need assistance as we have already deleted all the RC versions on PyPI. Our only remaining option is to delete older versions, which will impact some users. Can we increase the size to 50GB while we work on a better solution on our end, so that we can unblock the next Ray release? We would greatly appreciate your help and understanding.

[cmaureir]

Hello @aslonnie @can-anyscale 👋
I've set the project limit for ray-cpp to 60 GB on PyPI and TestPyPI. I will be closing this issue at the moment, even though it's not the size you asked for, but feel free to re-open if your research to reduce the sizes doesn't work.
Have a nice week 🚀

[aslonnie]

many thanks @cmaureir !

[can-anyscale]

thanks @cmaureir, really appreciate it