OptiBits

OptiBits is a lightweight, high-performance framework that automates LLM inference optimization through hyperparameter search across diverse numerical formats, including floating point and integer types—delivering faster, leaner, and more efficient models with minimal effort. 🚀

Key features:

• Automates hyperparameter tuning for optimal performance.
• Implements custom quantization, including 2-bit quantization.
• A custom endpoint to quickly visualize
• Plug & Play: Directly specify models from Hugging Face.

For example, this is the results of running various quantization parameters GPT2. You can replace this with any model from Hugging Face.

🔗 Get Started

📌 GitHub: github.com/Gongsta/optibits

📌 Install:

pip install optibits

📌 Run:

optibits --model mistralai/Mistral-7B

This defaults to searching over all quantization strategies.

🚀 Optimize your LLMs today!