Compressed LLMs by 85% that scores higher for llama.cpp's own 2-bit at the same size. Free one-click build.

[harshit-genrius]

We made a new 2-bit format for local models. It makes Qwen3-4B 85% smaller (8 GB -> 1.24 GB) and Llama 3.1 8B 85% smaller (16 GB -> 2.4 GB), and at the same file size it scores clearly better than llama.cpp's IQ2_XXS and ik_llama.cpp's IQ2_KT.

Try it in 2 minutes (Windows, NVIDIA RTX 30/40/50): download the zip from the release page, unzip, double-click run_qwen3_4b.bat. It downloads the model once and opens a chat page in your browser. Linux: ./run.sh qwen.

How good is it? (Qwen3-4B, all files the same 1.24 GB size)

Test (plain words)	Original model	llama.cpp IQ2_XXS	IQ2_KT	This
Quiz score: average of 5 standard tests (HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, MMLU)	56.3	41.9	46.5	50.8
Share of the original's quiz score kept	100%	74%	83%	90%
Quality lost vs the original model (lower is better)	0%	130%	46%	34%
Picks the same next word as the original model	-	-	63.7%	71.3%

On Llama 3.1 8B (2.4 GB, 85% smaller) the same pattern holds: IQ2_XXS's error score is 35% higher than this one's, IQ2_KT's 10% higher.

(For the technical crowd: "quality lost" is WikiText-2 perplexity relative to the original, 16.64 vs 12.46; IQ2_XXS 28.61, IQ2_KT 18.15; Llama 9.14 vs 10.04 vs 12.35. Context 4096, same imatrix for every file. "Same next word" is top-1 agreement with Q8_0 over 72 chunks. "Original" quiz column is the 8-bit Q8_0 file. Held-out C4 text keeps the same order.)

Speed (one RTX 5070 Ti Laptop, compared with llama.cpp's IQ2_XXS at the same size):

• Chatting (how fast it writes the reply): about the same, 1.10x
• Reading a long prompt or document before replying: 26% slower (0.74x)
• Serving several chats at once: 3-5x slower (0.18-0.34x)
• Running on CPU only: about 40% slower (0.56-0.62x)

Where it loses (please read):

• Qwen3-14B: it loses slightly to IQ2_KT (about 2% worse; perplexity 9.57 vs 9.40, tie on C4 text). The prediction was written down before the run, so it's a real loss.
• The Qwen3-4B number is from my last of many builds, so treat it as tuned. C4, Llama and 14B were held out or pre-registered.
• Needs a patched llama.cpp (CUDA). Stock llama.cpp, Ollama and LM Studio can't load it yet. No Metal/Vulkan.

How: an 8-state trellis code (Ungerboeck) over groups of 4 weights, after a Hadamard rotation and QuIP-style LDLQ rounding. Builds on QTIP / QuIP# and ikawrakow's IQ2_KT.

Links:

• One-click build: https://github.com/Genrius-AI/pingala8-llm/releases/tag/v2.4-oneclick
• Code, patch, all logs: https://github.com/Genrius-AI/pingala8-llm
• Models: https://huggingface.co/Genrius-AI/Qwen3-4B-Pingala8-2bit and https://huggingface.co/Genrius-AI/Llama-3.1-8B-Pingala8-2bit
• Paper: https://doi.org/10.5281/zenodo.23481757

Edit: After a comment from @ssvb we also tested Ternary Bonsai and GSQ on the same test - results are in the new section "How it compares to other small models" below. Short version: a retrained model (Bonsai) is better; ours is the best of the formats we tested for compressing a model you already have, at this size, without retraining.

How it compares to other small models (same test: WikiText-2 perplexity, ctx 4096, lower is better):

Format	Size	Perplexity	What it is
Ternary Bonsai 4B (prism-ml)	1.07 GB	11.59	Retrained Qwen3-4B (needs a training run and their llama.cpp fork)
GSQ Q2_K (ISTA-DASLab)	1.67 GB	14.37	Compression, but 34% bigger
Mine	1.24 GB	16.64	Compression of the original weights, no retraining
IQ2_KT (ik_llama.cpp)	1.25 GB	18.15	Compression, same size
IQ2_XXS (llama.cpp)	1.25 GB	28.61	Compression, same size

Bonsai even beats the original Qwen3-4B on this test (12.46), so it is really a new, retrained model. If you only want the best small Qwen3-4B, use Bonsai. Ours is for compressing models nobody has retrained for you.

[ssvb]

Is "best rival" show in your table actually Ternary Bonsai from https://huggingface.co/prism-ml/Ternary-Bonsai-4B-gguf? BTW, now a better Ternary Bonsai 2 is also available, but it's only used for newer models, such as https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf (a quantization of Qwen3.8-27B).

Or is it GSQ from https://huggingface.co/ISTA-DASLab/Qwen3-4B-GGUF-GSQ? BTW, now GSQ-RCO is also available, but it's only used for the newer models, such as https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

[harshit-genrius]

Good question, and fair point - I should've been clearer.

"Best rival" in my table meant IQ2_KT (ik_llama.cpp), the strongest format I compared at the same file size. I only compared formats that compress the original weights without retraining. Bad wording on my part.

After you asked, I ran the other two on the same test (WikiText-2 perplexity, ctx 4096, lower is better):

Format	Size	Perplexity
Ternary Bonsai 4B	1.07 GB	11.59
GSQ Q2_K	1.67 GB	14.37
Mine	1.24 GB	16.64
IQ2_KT	1.25 GB	18.15
IQ2_XXS	1.25 GB	28.61

So, honestly:

• Bonsai wins clearly, and it's smaller. It's a retrained model though - they trained Qwen3-4B into ternary weights, so you only get it for models they choose to retrain, and it needs their own llama.cpp fork. It even beats the original Qwen3-4B on this test (12.46), so it's really a different model.
• GSQ wins too, but its smallest file is 34% bigger.
• Ours is for compressing any model you already have, in a few hours on one GPU, to exactly IQ2_XXS size. In that lane it beats IQ2_XXS by a lot and IQ2_KT by a bit.

If anyone knows of a GSQ file around 1.24 GB, point me to it and we'll test it - that would be the fair comparison.