File Limit Request: bitsandbytes - 300 MiB #3753
Description
Project URL
https://pypi.org/project/bitsandbytes
Does this project already exist?
- Yes
New Limit
300 MiB
Update issue title
- I have updated the title.
Which indexes
PyPI, TestPyPI
About the project
Hey, thanks so much for all the great work with PyPi!
Project description
bitsandbytes was started 1.5 years ago to make the research code of deep learning researcher Tim Dettmers accessible for other researchers and the broad public. By now it has become immensely popular and helps greatly in the effort to democratize AI. The library makes large language model inference and training more accessible by dramatically reducing memory consumption with 8-bit optimizers and k-bit quantization.
- 8-bit optimizers uses block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost.
- LLM.Int() or 8-bit quantization enables large language model inference with only half the required memory and without any performance degradation. This method is based on vector-wise quantization to quantize most features to 8-bits and separately treating outliers with 16-bit matrix multiplication.
- QLoRA or 4-bit quantization enables large language model training with several memory-saving techniques that don't compromise performance. This method quantizes a model to 4-bits and inserts a small set of trainable low-rank adaptation (LoRA) weights to allow training.
Efforts to reduce binary size
We're currently maxing out the 100MB in every release. The issue is that we support lot's of CUDA versions, each with "fat binaries" that support many compute capabilities.
- We couldn't merge this PR to add sm_90a to enable use of accelerated wgmma and setmaxnreg instructions because of concerns about binary size.
- We're evaluating a solution to only support CUDA major versions in this PR.
- We've evaluated how to cut down binary size and reduced the CC and CUDA versions supported relative to prior releases, with this build matrix being the result.
- There are ongoing discussions on how to keep the build small.
Reasons for the request
Overall, we're very conscious of the size of our package, but we're always scraping at the 100MB. We'll keep focusing on keeping the binaries as small as technically possible, but it would be great to have some headroom at our disposal if we need, so we don't get blocked by this in the wrong moment.
cc @matthewdouglas @younesbelkada
Code of Conduct
- I agree to follow the PSF Code of Conduct