Repository navigation
arc-b580-v0.8
Pre-release
Pre-release
·
34 commits
to arc-b580
since this release
Faster plain generation on the B580 from fused kernels around the ternary mat-vecs, plus faster decode attention at long context.
- Plain generation (no speculation), public mtp-lean model, B580: short prompt ~41 -> 46 t/s, 48K context ~36 -> 40, 115K ~31.5 -> 35.
- Each token's activation is Hadamard-rotated and int8-quantized once and reused by every mat-vec that reads it. The residual add + RMS norm, the FFN SwiGLU and the delta-net pre-norm are folded into that rotation. The delta-net conv chain and its gate block each run as one kernel. Mat-vecs that share an input (q/k/v, gate/up) run in one launch. Small verify batches (up to 32 rows) use a smaller TernSYCL tile.
- Decode attention prefetches the next tile's rows: with MTP, +4% at 48K and +7.5% at 115K.
- n-gram drafts start at 64 tokens, double after a full accept and halve after a rejection (LLAMA_ARG_SPEC_NGRAM_ADAPT=0 restores a fixed length).
- Vendored TernSYCL updated to 8b0ae42.
- Every new piece can be switched off with an environment variable for comparison (list in docs/bonsai-arc-b580.md, "Switches"). KL divergence is unchanged within noise. The fusions have exact tests: test-backend-ops -o ARCLAB_* (75/75 on the B580).
Windows x64 build of the arc-b580 branch for Intel Arc (SYCL). EXPERIMENTAL: built by CI and not yet tested on a
Windows machine. Needs a current Intel Arc driver (an old driver's ze_loader.dll crashes at start).
Unzip, put the model next to it, and run run-bonsai.bat. See README.txt and
docs/bonsai-arc-b580.md. Please report results in https://github.com/Torchit1/llama.cpp/discussions/1