Skip to content

Pingala-8 one-click runtime (Windows and Linux)

Choose a tag to compare

View all tags
@harshit-genrius harshit-genrius released this 21 Oct 10:54
· 16 commits to main since this release
v2.4-oneclick
bf15772

Ready-to-run Pingala-8 runtime: patched llama.cpp (commit 7ac59a6) that runs the B4 2-bit models. Needs an NVIDIA GPU (RTX 30, 40 or 50 series) and a recent driver.

Windows: download pingala8-windows-cuda12.zip, unzip, double-click run_qwen3_4b.bat (or run_llama31_8b.bat). The first run downloads the model from Hugging Face; then a chat page opens at http://127.0.0.1:8080.

Linux: download pingala8-linux-cuda13.tar.gz, extract, run ./run.sh qwen (or ./run.sh llama), open http://127.0.0.1:8080.

Same file size as llama.cpp IQ2_XXS. WikiText-2 perplexity on Qwen3-4B: 16.64 (IQ2_XXS 28.61, our best IQ2_KT 18.15). On Llama-3.1-8B: 8.44 (IQ2_KT 9.36). It loses to IQ2_KT on Qwen3-14B (9.57 against 9.40). Prompt processing is slower than IQ2_XXS (0.74x).

Paper: https://doi.org/10.5281/zenodo.23481757
Models: https://huggingface.co/Genrius-AI/Qwen3-4B-Pingala8-2bit and https://huggingface.co/Genrius-AI/Llama-3.1-8B-Pingala8-2bit (Built with Llama)

Assets 6
Loading