Repository navigation
arc-b580-v0.10.1
Pre-release
Pre-release
Fewer round trips and faster long-context attention on the B580: up to 6.5% faster than v0.10, and 12-15% faster than v0.8. Greedy outputs are byte-identical to v0.10.
- Against v0.10 on the B580 (same public models and flags): MTP with a 32K trimmed draft head 80.4 -> 84.7 t/s (+5.4%); MTP on mtp-lean 78.3 -> 80.5 (+2.8%). Live settings (MTP + n-gram drafts, 128K context, same head): short prompts 75.9 -> 78.6 (+3.6%), 48K 78.7 -> 82.4 (+4.7%), 115K 83.5 -> 88.9 (+6.5%).
- One read-back for the draft chain: the one-graph draft chain now returns all its draft tokens in one device-to-host copy instead of one per token. Each small copy costs about 0.2 ms on this SYCL stack, so this alone is worth about 2% with a draft head. On by default (LLAMA_MTP_CHAIN_TIGHT is now 31; bit 16 is this change).
- 256-register file for the q4_0 decode attention at 3+ query tokens (the MTP and n-gram verify batches): no register spills, same occupancy, and the kernel is 6-17% faster at 128K. GGML_SYCL_FA_DEC_GRF=0 turns it off.
- Three general improvements from PrismML's llama.cpp fork (github.com/PrismML-Eng/llama.cpp), with their authorship kept: KV seq_rm limited to the used cell range (ggml-org#311), reuse of evicted prompt-checkpoint buffers in the server (ggml-org#312), and top-k taken straight from the logits when it starts the sampler chain (ggml-org#313).
- Opt-in: GGML_SYCL_FA_DEC_B2D=1 uses 2D block loads for the decode attention's K/V reads (within noise on the B580).
Windows x64 build of the arc-b580 branch for Intel Arc (SYCL). EXPERIMENTAL: built by CI and not yet tested on a
Windows machine. Needs a current Intel Arc driver (an old driver's ze_loader.dll crashes at start).
Unzip, put the model next to it, and run run-bonsai.bat. See README.txt and
docs/bonsai-arc-b580.md. Please report results in https://github.com/Torchit1/llama.cpp/discussions/1