File Limit Request: bitsandbytes - 300 MiB

[Titus-von-Koeller]

Project URL

https://pypi.org/project/bitsandbytes

Does this project already exist?

• Yes

New Limit

300 MiB

Update issue title

• I have updated the title.

Which indexes

PyPI, TestPyPI

About the project

Hey, thanks so much for all the great work with PyPi!

Project description

bitsandbytes was started 1.5 years ago to make the research code of deep learning researcher Tim Dettmers accessible for other researchers and the broad public. By now it has become immensely popular and helps greatly in the effort to democratize AI. The library makes large language model inference and training more accessible by dramatically reducing memory consumption with 8-bit optimizers and k-bit quantization.

• 8-bit optimizers uses block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.
• LLM.Int() or 8-bit quantization enables large language model inference with only half the required memory and without any performance degradation. This method is based on vector-wise quantization to quantize most features to 8-bits and separately treating outliers with 16-bit matrix multiplication.
• QLoRA or 4-bit quantization enables large language model training with several memory-saving techniques that don't compromise performance. This method quantizes a model to 4-bits and inserts a small set of trainable low-rank adaptation (LoRA) weights to allow training.

Efforts to reduce binary size

We're currently maxing out the 100MB in every release. The issue is that we support lot's of CUDA versions, each with "fat binaries" that support many compute capabilities.

• We couldn't merge this PR to add sm_90a to enable use of accelerated wgmma and setmaxnreg instructions because of concerns about binary size.
• We're evaluating a solution to only support CUDA major versions in this PR.
• We've evaluated how to cut down binary size and reduced the CC and CUDA versions supported relative to prior releases, with this build matrix being the result.
• There are ongoing discussions on how to keep the build small.

Reasons for the request

Overall, we're very conscious of the size of our package, but we're always scraping at the 100MB. We'll keep focusing on keeping the binaries as small as technically possible, but it would be great to have some headroom at our disposal if we need, so we don't get blocked by this in the wrong moment.

cc @matthewdouglas @younesbelkada

Code of Conduct

• I agree to follow the PSF Code of Conduct

[cmaureir]

Hey @Titus-von-Koeller 👋
I've set the upload limit for bitsandbytes to 300 MB on PyPI and TestPyPI. Please be mindful of the frequency of releases at that size.
Have a nice week 🚀

[Titus-von-Koeller]

@cmaureir Yes, will do, thank you!

Can you see how close we are to reaching the maximum total data volume for all releases combined? We're a bit afraid that we're already close to that limit based on our estimates, but somehow couldn't find that info in our PyPi view.

Could you increase that as well and let us know where we're currently at, please?