SYCL on Arc B580: ternary weights and q4_0 KV attention on XMX (Bonsai 2 27B, 128K context in 12 GB)

[Torchit1]

I've been working on the SYCL backend for Intel Arc B580 (Xe2), mostly for PrismML's ternary Bonsai 2 27B, and wanted to share the results and a couple of findings in case they're useful to other SYCL work.

Branch: https://github.com/Torchit1/llama.cpp/tree/arc-b580 (guide: docs/bonsai-arc-b580.md)

What's in it:

• Ternary PTQ1_0 weights repacked to 2 bits at load and run on the int8 x int2 DPAS path (Intel's TernSYCL kernels), with round-to-nearest activation quantisation (KL divergence 0.00022 against reference logits).
• Decode and draft-verify attention (1-32 query tokens) straight from the q4_0 KV cache on XMX. Xe2's IGC provides intel_sub_group_i8_i4_matrix_mad_k32, which isn't in the public cl_intel_subgroup_matrix_multiply_accumulate list; the raw q4_0 bytes XOR 0x88888888 work as its B operand. It's exact on the B580.
• Full 1024-token prompt batches through oneDNN's int8 GEMM (about 10% faster long prompts).
• Memory changes so MTP drafting fits 128K context in 12 GB (q4_0 KV for the draft context, a smaller draft batch).
• The same attention kernel generalises to Gemma 4 12B (head 512, GQA 16), plus a q4_0 small-batch GEMM on XMX: Gemma 4 12B QAT with Google's assistant drafter goes from about 40 to 100 t/s on new code.

Results on a B580 at 128K context: about 90 t/s writing new code, 250-370 t/s on code edits, 42 t/s plain, 44 t/s with 115K tokens of history. Same card, model and flags against the SYCL support in PrismML's fork: prompt reading 207 vs 895 t/s, plain generation 21 vs 42, generation at 32K context 8 vs 36.

All the XMX paths are gated to Xe2 and fall back to the existing kernels elsewhere. One user report from an A770 (Alchemist): it runs on the fallback kernels but needed -b 256 -ub 256 to avoid a Level Zero ERROR_UNKNOWN, which may be worth a look on the mainline side too.

I'd be glad to help upstream any of the general pieces (the q4_0 KV attention in particular) if there's interest.

[159753a52]

The memory side of "128K in 12 GB with MTP", for anyone trying this on another 12 GB card (estimates; the file sizes are exact):

• Weights: PTQ1_0 is 5.54 GiB (5,946,648,928 B). The community file with the MTP layer grafted back is 6.53 GiB (7,012,820,512 B).
• Main KV at 128K, 16 of 64 layers: 8.0 GiB f16, 4.25 GiB q8_0, 2.25 GiB q4_0.
• Draft context (one attention layer) at 128K: 0.5 GiB f16 vs 0.14 GiB q4_0.
• Compute buffers: ~1 GiB.

That's ~9.9 GiB with MTP and everything in q4_0. Real runs sit higher at long context: sudoingX's RTX 4070 12GB sweep measured 9,918 MiB at 131K filled context with a q4_0 cache and no MTP, about 1.1 GiB above that estimate. So with MTP you're around 11 GiB, and the 0.36 GiB from moving the draft KV to q4_0, plus a smaller draft batch, is roughly the difference between fitting and not. That matches what you found.

Your long-context number is also close to the bandwidth limit. At 115K of history, one token reads 5.95 GB of weights + ~2.2 GB of q4_0 KV. At the B580's 456 GB/s that caps plain decode at ~56 t/s, so 44 t/s is ~78% of the ceiling, which is about as close as decode usually gets.

Per-card context limits for each Bonsai file and KV type are in a page I built: [Link removed by moderator]/ternary-bonsai-2-27b-vram/

[Torchit1]

Thanks, that breakdown matches what I found. Moving the draft KV to q4_0 and shrinking the draft batch was exactly what got 128K to fit.

One thing on the bandwidth part: the 44 t/s at ~115K is with MTP plus n-gram drafts, not plain decode. Each verify pass can accept more than one token, so the plain ceiling doesn't bound it. I measured plain decode on the same 114.6K-token prompt: 26.4 t/s, so about 47% of your ~56 t/s ceiling (41 t/s at short context, ~54% of ~77). MTP takes it to 43 t/s at 115K with identical output. So there's still a fair bit of headroom in the plain decode path.

[arthw]

@Torchit1
It's good job!

Is it possible to contribute this feature to llama.cpp SYCL backend?
We has noticed the PTQ1_0 feature and some PRs for CPU are on review in llama.cpp.

Thank you!

[Torchit1]

Thanks Neo! Yes, happy to. I'd suggest doing it in pieces, starting with the parts that don't depend on PTQ1_0: the q4_0 KV-cache decode attention on XMX (it helps any model with a q4_0 cache on Arc), then the gated delta-net kernels for the Qwen3.5-style hybrids. The PTQ1_0 XMX path can follow once the type is merged upstream; it uses Intel's TernSYCL kernels, so we'd also need to agree on how to vendor those. I'll open the first PR when I get a chance.

[Torchit1]

Thanks Neo, I've emailed you separately about this.

[arthw]

@Torchit1
Yes, I see your email and reply!

Thank you!