Add an easy install support of Ternary-Bonsai-2-27B really good model for everyone

[orionp13]

Hi,

It would be great to see Bonsai 2 (prism-ml/Ternary-Bonsai-2-27B-GGUF) added to the official model list on llama.app/models.

I’ve been testing it locally, and I’m really impressed. It runs surprisingly well on hardware with only 8 GB of VRAM and limited RAM, while delivering excellent performance.

So far, it’s one of the most capable local models I’ve ever tried for this level of hardware.

I think it would be a great addition to the official model catalog.

Have a nice day,
Alexander

[159753a52]

For anyone checking the 8 GB claim before trying it, here are the numbers for the PrismML files (sizes from the HF API; totals are file + KV cache + ~1 GiB of buffers, whole model on the GPU, no mmproj):

• PTQ1_0: 5,946,648,928 bytes (5.54 GiB). At 32K context it comes to 7.10 GiB with a q4_0 KV cache, 7.60 GiB with q8_0. A reported RTX 2060 Super 8GB run at 32K used 6.2–6.5 GB at 16.8 tok/s.
• PQ2_0: 7,206,168,928 bytes (6.71 GiB), already tight at 8K (~7.85 GiB).
• Long context is cheap because only 16 of the 64 layers keep a KV cache: PTQ1_0 reaches 256K in ~11.0 GiB with q4_0 KV (an RTX 3060 12GB reported 11.7 GB at 262K).

One thing that matters for the llama.app/models request: PTQ1_0 and PQ2_0 are new tensor types that only PrismML's llama.cpp fork implements, so mainline can't load these files today; the types would have to be upstreamed first. The closest file mainline does load is ISTA-DASLab's GSQ-RCO GGUF of the same Qwen3.8 27B, but its smallest one (IQ2_XS, 7.84 GiB) already needs ~9 GiB at 8K, so that's a 12 GB-card option, not an 8 GB one.

I put all the Bonsai files side by side here (I built it): [Link removed by moderator]/ternary-bonsai-2-27b-vram/

[alexandre-leng]

It work very well on my 4060 8GB ram for asking question like a simple chatGPT but powerful :)