Skip to content

GRIMOIRE v1.9.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@doopeworld doopeworld released this 09 Oct 12:44
v1.9.0
78274a5

Correctness warning: MTP with one sequence slot

The v1.9.0 binary can skip or repeat chunks of an answer when MTP runs with GRIMOIRE_SEQ_SLOTS=1, including at the default GRIMOIRE_MTP_K=4. Reproduced on Qwen3.8 and Ornith.

For this release, serve MTP with GRIMOIRE_SEQ_SLOTS set to at least 2; the README command and Unraid templates use 8. The source fix is dd286b3 on longctx-attn-wip. The v1.9.0 download does not include that fix.

Speculative decoding on Qwen3.8-27B: one user now decodes 66-70 tok/s on llama-benchy (v1.8.3: 54-55) and 113-126 tok/s on code.

Measured with llama-benchy 0.4.0 on one Arc Pro B70, Qwen3.8-27B-GPTQ-Int4-MTP-BF16, GRIMOIRE_MTP=1 GRIMOIRE_MTP_K=6 GRIMOIRE_MTP_DRAFT_VOCAB=65536 GRIMOIRE_SEQ_SLOTS=8. Totals are tok/s.

v1.8.3 (4 drafts) v1.9.0
decode, 1 user, after pp512 / pp4096 53.9 / 55.2 69.6 / 65.6
decode, 8 users, after pp512 / pp4096 187.4 / 149.0 197.9 / 155.3
prompt pp4096, 1 user 2,069 1,979

A coding task (write a Python function, thinking off, 2K / 8K prompt): 126 / 113 tok/s, every answer correct. Tool calling passes end to end.

What changed (97b56da .. 78274a5)

  • Verify attention on the matrix engine (head_dim 256 models): 5-17x faster per layer. Speculative decoding no longer slows down as the context grows.
  • Fused verify pass. The DeltaNet recurrence and conv run once per conversation instead of once per draft. A rejected draft is rolled back by replaying the accepted tokens, bit for bit the same result. This also frees ~2.4 GB of VRAM on Qwen3.8-27B.
  • Faster INT4 matrix math for 2-8 rows (verify, several users): 4-25% faster per projection. 4 users on group-64 INT4 checkpoints decode 9% faster.
  • GRIMOIRE_MTP_DRAFT_VOCAB=<n> now also works with INT4 / GPTQ output heads. It drafts from the first n tokens only; every emitted token is still checked against the full vocabulary.
  • Power governor: on by default only when several GPUs serve one model. Off on one GPU, where the card's firmware already holds its 275 W rating.
  • Multi-GPU (pipeline / tensor parallel): host-staged exchange, real-VRAM prompt chunks, paced weight upload, Flash-Next pipeline, three-stage pipelines (e.g. two B70 + a B580; build with GRIM_TARGETS). Also v1.8.3's FP8 decode kernels.

Recommended for Qwen3.8-27B GPTQ + MTP

GRIMOIRE_MTP=1 GRIMOIRE_MTP_K=6 GRIMOIRE_MTP_DRAFT_VOCAB=65536. The default is still 4 drafts. Prose decodes about as fast with 4-7 drafts; code is faster with 6-7.

A/B switches

GRIMOIRE_VERIFY_DPAS=0, GRIMOIRE_DN_VERIFY_FUSED=0, GRIMOIRE_I4_FASTDQ=0 restore the previous kernels. GRIMOIRE_POWER_LIMIT_W=<watts> sets a power governor.

Known

  • Ornith-1.5-35B-A3B MXFP4, 512-token prompt, 1 user: 6,529 tok/s, v1.8: 7,240. This dates from v1.8.1 and is being bisected. Decode is unchanged (193.7 / 461.3 at 1 / 8 users).
  • Two GPUs through a passive M.2-to-OCuLink adapter: the second card can drop off the PCIe bus when it starts computing. That's the link, not the load. Use proper slots or an adapter with a redriver.

Use

curl -LO https://github.com/doopeworld/GRIMOIRE/releases/download/v1.9.0/grimoire-b70-v1.9.0.tar.gz
docker load < grimoire-b70-v1.9.0.tar.gz          # -> grimoire-b70:latest
docker run -d --name grimoire --network host --init --device /dev/dri/renderD128 \
  -e GRIMOIRE_SEQ_SLOTS=8 -e GRIMOIRE_MTP=1 -e GRIMOIRE_MTP_K=6 -e GRIMOIRE_MTP_DRAFT_VOCAB=65536 \
  -v /path/to/models:/models grimoire-b70:latest \
  server --model /models/Qwen3.8-27B-GPTQ-Int4-MTP-BF16 --proj int4 --ctx 16384 --port 8000

Find your B70's render node with ls -l /dev/dri/by-path/. The numbers can change after a reboot. Never set SYCL_UR_USE_LEVEL_ZERO_V2=0.

sha256 a03de41fa414dfd27b2b9bd3a5bddd31ca9e07629f7cb3e9c5306b8be2019fd0

Assets 3
Loading