GRIMOIRE v1.10.0 - multi-user MTP on two GPUs, Arc Pro B60, oneDNN

GRIMOIRE v1.10.0 is built from main at 3fb9df7. Its main changes are multi-user MTP on two GPUs, Arc Pro B60 support, oneDNN, and the newest Intel toolchain.

curl -LO https://github.com/doopeworld/GRIMOIRE/releases/download/v1.10.0/grimoire-b70-v1.10.0.tar.gz
docker load < grimoire-b70-v1.10.0.tar.gz        # -> grimoire-b70:v1.10.0

sha256 6786c9c1595ea677878823346d8a3ef36fda6c4dd5489697f31b93dde7355efa

New since v1.9.0

• Two GPUs (Megatron tensor parallel) for models that do not fit one card. Measured on Qwen3.8-27B-FP8 with two Arc Pro B70s: prompt 3,530 tok/s at 4K, and 29.5 tok/s decode for one user without MTP. Qwen3.8-Flash-Next runs tensor plus expert parallel at 44.6 tok/s decode.
• MTP for several users at once on two GPUs. Before this release it served one request at a time. With Qwen3.8-27B-FP8 + MTP, 1 / 2 / 4 / 8 users get 56.5 / 62.9 / 95.0 / 161.9 tok/s in total (llama-benchy, pp4096 / tg128). On the coding test it decodes at 88 tok/s, and the answers pass their unit tests.
• Intel oneDNN for the prompt GEMMs, on by default. Prompts run 5-26% faster on two cards. GRIMOIRE_ONEDNN=0 turns it off.
• Arc Pro B60 / Arc B580 (Battlemage G21) support. The same image carries code for the B70 and for G21, so the B60 no longer needs its own rebuild. The kernels are tuned on the B70, and we have not measured a B60 ourselves.
• KV cache fitted to the card. GRIMOIRE now prints KV cache: N slots x --ctx M, X GiB on this card at start. When GRIMOIRE_SEQ_SLOTS conversations at --ctx do not fit the card, it holds fewer at once and says why; the other requests wait for a free slot. Before, the driver silently moved memory to system RAM and every step crawled; this was the 4 x 200K test that never finished. GRIMOIRE_SLOTS_FIT=0 keeps the requested slots.
• No more misleading start-up lines. v1.9.0 printed "... unavailable; using fallback" five times at every start. Those lines described optional outside libraries that GRIMOIRE does not ship; GRIMOIRE's own fast kernels were running all along. The lines now appear only with GRIMOIRE_BRIDGE_DEBUG=1.
• Newest Intel stack: oneAPI DPC++ 2026.2, GPU driver (compute-runtime) 26.35 with IGC 2.41.5, and oneDNN 3.14. We checked it against the previous stack: 43 of the 44 regression answers are identical, the 44th changes one word with the same meaning, and speed is the same on one and two cards.
• MTP with GRIMOIRE_SEQ_SLOTS=1: the v1.9.0 bug that could skip or repeat parts of an answer is fixed.

Test system

Beelink GTi15 Ultra (Core Ultra 9 285H, 96 GB) with two Arc Pro B70 32 GB cards: one on PCIe Gen4 x8, one on Gen4 x4 through an M.2-OCuLink redriver. The cards have no peer-to-peer link. Numbers come from llama-benchy 0.4.0 through the OpenAI-compatible server. See the README for launch commands, including the two-GPU multi TP mode.

This is still an experimental release (pre-release).

GRIMOIRE v1.9.0

Correctness warning: MTP with one sequence slot

The v1.9.0 binary can skip or repeat chunks of an answer when MTP runs with GRIMOIRE_SEQ_SLOTS=1, including at the default GRIMOIRE_MTP_K=4. Reproduced on Qwen3.8 and Ornith.

For this release, serve MTP with GRIMOIRE_SEQ_SLOTS set to at least 2; the README command and Unraid templates use 8. The source fix is dd286b3 on longctx-attn-wip. The v1.9.0 download does not include that fix.

Speculative decoding on Qwen3.8-27B: one user now decodes 66-70 tok/s on llama-benchy (v1.8.3: 54-55) and 113-126 tok/s on code.

Measured with llama-benchy 0.4.0 on one Arc Pro B70, Qwen3.8-27B-GPTQ-Int4-MTP-BF16, GRIMOIRE_MTP=1 GRIMOIRE_MTP_K=6 GRIMOIRE_MTP_DRAFT_VOCAB=65536 GRIMOIRE_SEQ_SLOTS=8. Totals are tok/s.

	v1.8.3 (4 drafts)	v1.9.0
decode, 1 user, after pp512 / pp4096	53.9 / 55.2	69.6 / 65.6
decode, 8 users, after pp512 / pp4096	187.4 / 149.0	197.9 / 155.3
prompt pp4096, 1 user	2,069	1,979

A coding task (write a Python function, thinking off, 2K / 8K prompt): 126 / 113 tok/s, every answer correct. Tool calling passes end to end.

What changed (97b56da .. 78274a5)

• Verify attention on the matrix engine (head_dim 256 models): 5-17x faster per layer. Speculative decoding no longer slows down as the context grows.
• Fused verify pass. The DeltaNet recurrence and conv run once per conversation instead of once per draft. A rejected draft is rolled back by replaying the accepted tokens, bit for bit the same result. This also frees ~2.4 GB of VRAM on Qwen3.8-27B.
• Faster INT4 matrix math for 2-8 rows (verify, several users): 4-25% faster per projection. 4 users on group-64 INT4 checkpoints decode 9% faster.
• GRIMOIRE_MTP_DRAFT_VOCAB=<n> now also works with INT4 / GPTQ output heads. It drafts from the first n tokens only; every emitted token is still checked against the full vocabulary.
• Power governor: on by default only when several GPUs serve one model. Off on one GPU, where the card's firmware already holds its 275 W rating.
• Multi-GPU (pipeline / tensor parallel): host-staged exchange, real-VRAM prompt chunks, paced weight upload, Flash-Next pipeline, three-stage pipelines (e.g. two B70 + a B580; build with GRIM_TARGETS). Also v1.8.3's FP8 decode kernels.

Recommended for Qwen3.8-27B GPTQ + MTP

GRIMOIRE_MTP=1 GRIMOIRE_MTP_K=6 GRIMOIRE_MTP_DRAFT_VOCAB=65536. The default is still 4 drafts. Prose decodes about as fast with 4-7 drafts; code is faster with 6-7.

A/B switches

GRIMOIRE_VERIFY_DPAS=0, GRIMOIRE_DN_VERIFY_FUSED=0, GRIMOIRE_I4_FASTDQ=0 restore the previous kernels. GRIMOIRE_POWER_LIMIT_W=<watts> sets a power governor.

Known

• Ornith-1.5-35B-A3B MXFP4, 512-token prompt, 1 user: 6,529 tok/s, v1.8: 7,240. This dates from v1.8.1 and is being bisected. Decode is unchanged (193.7 / 461.3 at 1 / 8 users).
• Two GPUs through a passive M.2-to-OCuLink adapter: the second card can drop off the PCIe bus when it starts computing. That's the link, not the load. Use proper slots or an adapter with a redriver.

Use

curl -LO https://github.com/doopeworld/GRIMOIRE/releases/download/v1.9.0/grimoire-b70-v1.9.0.tar.gz
docker load < grimoire-b70-v1.9.0.tar.gz          # -> grimoire-b70:latest
docker run -d --name grimoire --network host --init --device /dev/dri/renderD128 \
  -e GRIMOIRE_SEQ_SLOTS=8 -e GRIMOIRE_MTP=1 -e GRIMOIRE_MTP_K=6 -e GRIMOIRE_MTP_DRAFT_VOCAB=65536 \
  -v /path/to/models:/models grimoire-b70:latest \
  server --model /models/Qwen3.8-27B-GPTQ-Int4-MTP-BF16 --proj int4 --ctx 16384 --port 8000

Find your B70's render node with ls -l /dev/dri/by-path/. The numbers can change after a reboot. Never set SYCL_UR_USE_LEVEL_ZERO_V2=0.

sha256 a03de41fa414dfd27b2b9bd3a5bddd31ca9e07629f7cb3e9c5306b8be2019fd0

GRIMOIRE v1.8.2

GRIMOIRE v1.8.2 -- tool calling, reasoning_content, no 2-second stalls when a long prompt arrives

docker load < grimoire-b70-v1.8.2.tar.gz      # -> grimoire-b70:latest

New

OpenAI tool calling. tools / tool_choice requests are supported on Qwen3.8-27B (and its
fine-tunes), Ornith-1.5-35B-A3B, Agnes-3.0-Flash and Qwen3.8-Flash-Next. The prompt is rendered
byte-for-byte as each checkpoint's own Jinja chat template. That was checked on 13 request cases
(tools, tool results, multi-turn, thinking on/off, reasoning effort) for each of these models. The
model's <tool_call> blocks come back as OpenAI tool_calls with finish_reason: "tool_calls",
streamed or not. Qwen3.8-27B GPTQ + MTP, asked "What is the weather in Paris?" with a
get_weather tool:

• it calls get_weather({"city": "Paris", "unit": "celsius"});
• given the tool's answer, it replies "The current weather in Paris is 18°C (64°F) with light
  rain ...".

Qwen3.6-35B-A3B's newer template is not supported yet, nor are the K2 and Muse templates. A request
with tools gets HTTP 400 on those models.

Reasoning in reasoning_content. For thinking models (Qwen3.8, Ornith, Agnes, Flash-Next, K2),
the answer is in content and the reasoning in reasoning_content, as OpenAI clients expect.
chat_template_kwargs: {"enable_thinking": false} turns thinking off.

A long prompt no longer freezes the other users. A prompt longer than 1,024 tokens that arrives
while other requests are decoding is processed in 1,024-token chunks, with the others' decoding in
between (GRIMOIRE_INTERLEAVE_CHUNK, 0 = off). Measured on Qwen3.8-27B GPTQ + MTP: user B sent
a 4K-token prompt while user A was streaming an answer.

	A's longest pause	B's first token
v1.8.1	2.03 s	1.99 s
v1.8.2	0.67 s	2.58 s

B's answer was identical either way.

Speed

The server now sends Qwen3.8's official chat template, the same prompt vLLM sends. It adds the
template's reasoning-effort system text, and the model's reasoning, and so how many of MTP's
drafts it accepts, shifts a little.

	1 user, tg128 after pp512 (llama-benchy, 4 runs)
v1.8.1	63.8 ± 5.1 tok/s
v1.8.2	58.3 ± 6.5 tok/s

vLLM 0.30.1 with the same checkpoint and flags measured 45.1. Release validation on the v1.8.2
image (8 slots, --ctx 16384, coherence test passed):

Test	1 user	8 users
pp512 / tg128	50.5	187.4
pp4096 / tg128	44.9	148.9
prompt pp4096	2,020

pp512 reads lower (1,305 vs ~1,420) only because llama-benchy does not count the template's
system text as prompt.

Image

grimoire-b70-v1.8.2.tar.gz, 640,927,385 bytes, sha256
b0a0e542b697719db53171d04f202bc5655f6d364e1a795c7532f3f2ee3cb1b5, built from commit 1ac79b4 (image id
83b3d39a3d01).

GRIMOIRE v1.8.1

GRIMOIRE v1.8.1 -- K2 answers stop where they should; two-GPU mode in the image

docker load < grimoire-b70-v1.8.1.tar.gz      # -> grimoire-b70:latest

Fixes

K2-Horizon-MoVA-36B-A4B kept writing after its answer. K2's tokenizer prefixes every
control token with ifm| (<|ifm|im_start|>, <|ifm|im_end|>, <|ifm|endoftext|>). GRIMOIRE
looked for the end-of-turn token only under ChatML's names, found none, and so never stopped a K2
answer. After "Paris" it went on to max_tokens with unrelated text. K2 now:

• stops on <|ifm|im_end|> and <|ifm|endoftext|> (the two ids in its generation_config.json);
• gets its own chat template. It is byte-identical to the checkpoint's Jinja template: BOS first,
  no newline between turns, and the assistant turn opens a <ifm|think> block.

llama-benchy on K2 (4 slots, --ctx 16384): pp512 919 tok/s, tg128 66.8 at 1 user, 83.4 total at
4 users, coherence test passed.

A second stop token for every model. It is whichever exists of <|eot|> (Harmony / Muse),
<|ifm|endoftext|> (K2) or <|endoftext|> (Qwen family, listed next to <|im_end|> in their
generation_config.json).

New: two GPUs in one container

multi runs one server rank per GPU (pipeline parallel by default, multi TP for tensor
parallel). This is for checkpoints that do not fit one B70. Example:
Ornith-1.5-35B-A3B-FP8 on two Arc Pro B70s:

docker run -d --init --stop-timeout 300 --ipc=host --shm-size=10g \
    --device /dev/dri/renderD128 --device /dev/dri/renderD131 \
    -v /dev/dri/by-path:/dev/dri/by-path:ro -v /path/to/models:/models -p 8000:8000 \
    -e ZE_AFFINITY_MASK=0,1 -e GRIMOIRE_MULTI_GPUS=2 -e GRIMOIRE_PP_SPLIT=20 -e GRIMOIRE_SEQ_SLOTS=8 \
    -e GRIMOIRE_DEFER_MOE_GATHER=1 -e GRIMOIRE_BF16_QKV=1 -e GRIMOIRE_BF16_DN_QKV=1 \
    grimoire-b70:latest multi --model /models/Ornith-1.5-35B-A3B-FP8 --proj fp8 --ctx 32000 --port 8000

Measured: ready in 63 s, 77 tok/s for one 128-token request. GRIMOIRE_PP_SPLIT is the number of
layers the first GPU keeps.

Other models, one 128-token request through the server (v1.8 / v1.8.1)

Model	--proj	Settings	Decode tok/s
Ornith-1.5-35B-A3B (MXFP4)	mxfp4	8 slots, --ctx 32000	201.8
Qwen3.6-35B-A3B GPTQ-Int4	int4	8 slots, --ctx 32000	128.7
K2-Horizon-MoVA-36B-A4B	mxfp4	4 slots, --ctx 16384	81-82
Muse-Glimmer-30B INT4-W4A16	int4	one request at a time, --ctx 16384	28.4
Agnes-3.0-Flash	mxfp4	8 slots, --ctx 16384	28.0
Qwen3.8-Flash-Next (tiered)	bf16	112 experts / layer in VRAM, --ctx 65536	22.3

Qwen3.8-27B GPTQ + MTP is unchanged from v1.8: llama-benchy tg128 62.9 at 1 user, 186.9 total at
8 users.

Image

grimoire-b70-v1.8.1.tar.gz, 640,834,579 bytes, sha256
794cee4159a03ab8a693a09ee6d85f6bfd3924a2dfbfdeb62995feeeb4b9d40e, built from commit d702908 (image id
0de8c1a47ecb).

GRIMOIRE v1.8

GRIMOIRE v1.8 -- Qwen3.8-27B GPTQ-Int4: MTP up to 72 tok/s, prompt 2,070 tok/s

docker load < grimoire-b70-v1.8.tar.gz        # -> grimoire-b70:latest

Model: SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16,
--proj int4 --ctx 16384 -e GRIMOIRE_MTP=1 -e GRIMOIRE_SEQ_SLOTS=8. One Arc Pro B70, llama-benchy
0.4.0, total tokens/s:

Test	Users	v1.7.1	v1.8
pp512 / tg128	1 / 2 / 4 / 8	49.0 / 67.4 / 84.2 / 139.2	53.9 / 87.9 / 121.7 / 187.4
pp4096 / tg128	1 / 2 / 4 / 8	51.1 / 38.4 / 36.2 / 39.7	55.2 / 78.9 / 107.9 / 149.0
prompt pp4096	1	1,731	2,069

Every llama-benchy run draws different text from its book, and how much MTP gains depends on the
text: four runs of v1.8 at 1 user gave 53.9 - 71.8 tok/s after pp512 and 55.2 - 63.5 after
pp4096 (the table is the release validation run, tools/bench/validate_image_v18.sh).

What changed

MTP drafts fixed. Two differences from how the checkpoint's MTP head is meant to be run:

• Each draft ran at the position of the token it was given instead of the position of the
  hidden state it came from. The head's own cache stores that pair at the hidden state's
  position, so every draft looked at its context shifted by one, with the newest entry stored
  twice.
• The head was fed the hidden state before the model's final norm; it expects the one after it
  (the vector the output layer reads).

MTP defaults. GRIMOIRE_MTP=1 is now enough:

• 4 drafts per step (GRIMOIRE_MTP_K, was 3). At 1 user, tg128 after pp512 / pp4096:
  3 drafts 61.9 / 56.0, 4 drafts 63.5 / 69.6, 5 drafts 66.2 / 65.0 tok/s (same session).
• The MTP head is stored in MXFP4. Drafting takes 5.1 instead of 9.2 ms per step. The head
  only proposes tokens: every token you get is still the model's own greedy choice.
  GRIMOIRE_MTP_HEAD_FMT=bf16 keeps it in BF16.
• Drafting stays on while up to 4 requests are active (GRIMOIRE_SPEC_MAX_SEQS, was 2).

INT4 verify faster. For 2 to 8 rows (the MTP check and batched decoding), the INT4
matrix kernel now runs with 128 registers per thread and splits the work over more threads:
42.2 -> 38.0 ms per MTP check. The output layer is also checked in one pass instead of row by row.

INT4 prompt processing. GPTQ / INT4 weights now go through the fast prompt path that
MXFP4 used, with the feed-forward gate fused in: 1,731 -> 2,069 tok/s at 4,096 tokens, same
output text (Qwen3.8-27B-W4A16, group 64: a 5,987-token prompt 4.72 -> 3.81 s).

Several long prompts at once. Two 4,096-token prompts do not fit one prefill call at
--ctx 16384 with 8 slots, so they are processed one after another. The first user's first token
used to go out before the second prompt was processed and then wait for it, which llama-benchy
counted as slow decoding. First tokens of a burst now go out together: pp4096/tg128 at 2 / 4 users
48.0 / 46.0 -> 78.9 / 107.9 tok/s total.

Comparison on the same card

vLLM 0.30.1 was run with the flags from the
B70 inference cookbook for
this checkpoint, in its own container on the same B70. That means MTP with 4 drafts, fp8 KV
cache and GPTQ, with no cookbook patches and the head in BF16. The same llama-benchy runs gave:

Test	Users	vLLM 0.30.1	GRIMOIRE v1.8
pp512 / tg128	1 / 2 / 4 / 8	45.1 / 85.2 / 130.4 / 122.3	53.9 - 71.8 / 87.9 / 121.7 / 187.4
pp4096 / tg128	1 / 2 / 4 / 8	40.1 / 57.8 / 64.3 / 46.0	55.2 - 63.5 / 78.9 / 107.9 / 149.0
prompt pp4096	1	1,948	2,069

vLLM's MTP accepted 37% of its drafts on llama-benchy's text. The cookbook's own 84-113 tok/s
come from prompts where about 95% of drafts are accepted.

Still open

• A long prompt that arrives while others are decoding still pauses them for its prefill.
  Mixing prefill chunks into the decoding steps is the next item.
• Tool calling in the server.

Image

grimoire-b70-v1.8.tar.gz, 640,860,913 bytes, sha256
2d8b5132c246dfb422326f48a079bc95a22c904c602b53c2c514c9435ac5c7a2, built from commit 05bc4ec
(image id afe77e3e7ec6).

GRIMOIRE v1.7.1

GRIMOIRE v1.7.1 -- several users on Qwen3.8-27B (with or without MTP), no VRAM overflow

docker load < grimoire-b70-v1.7.1.tar.gz      # -> grimoire-b70:latest

Fixes

Prefill no longer overflows VRAM on big dense models. A prompt's prefill scratch is about
1 MB per token on Qwen3.8-27B (its feed-forward is 34,816 wide). Eight concurrent 4K prompts
were admitted as one 32K-token batch, which asked for ~30 GB: the driver started moving memory
to system RAM, decode steps took ~10 s and the GPU could reset. v1.7.1 sizes every prefill
from the card's free VRAM (printed at startup as prefill chunk up to N tokens per call):
bursts are admitted in waves that fit, and a prompt longer than that is processed in chunks.
GRIMOIRE_PREFILL_CHUNK=<tokens> overrides it.

MTP together with GRIMOIRE_SEQ_SLOTS. With several requests in flight, MTP now drafts only
while at most 2 sequences are active (GRIMOIRE_SPEC_MAX_SEQS, default 2) and switches to plain
batched decoding above that; prompts are admitted in batches with MTP on too.

Qwen3.8-27B GPTQ-Int4 (SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16),
-e GRIMOIRE_MTP=1 -e GRIMOIRE_MTP_K=3 -e GRIMOIRE_SEQ_SLOTS=8 --ctx 16384, llama-benchy 0.4.0,
total generated tokens/s at 1 / 2 / 4 / 8 users:

Test	v1.7	v1.7.1
pp512 / tg128	53.8 / 58.5 / 61.7 / 76.2	49.0 / 67.4 / 84.2 / 139.2
pp4096 / tg128	50.7 / 38.8 / 29.5 / 32.5	51.1 / 38.4 / 36.2 / 39.7

Single-user MTP is the same within run-to-run noise (3-run A/B: 52.8 +- 2.7 vs 49.7 +- 0.7).
Ornith-1.5-35B-A3B (8 slots, --ctx 32000), pp4096/tg32: 171.2 / 213.6 / 299.4 / 393.5
(v1.6: 171 / 212 / 296 / 390).

Settings advice

• Context with 8 slots: the KV cache is reserved for every slot. On Qwen3.8-27B,
  --ctx 32000 x 8 slots takes 8.4 GB and leaves room for only ~1,500 prompt tokens per
  prefill call; --ctx 16384 x 8 slots leaves ~5,900. Use the smallest context you need.
• Long prompts with many users on Qwen3.8-27B are still admitted one after another
  (one 4K prompt fills a prefill call); faster INT4 prefill and smaller prefill scratch are next.

Image

grimoire-b70-v1.7.1.tar.gz, 640,771,417 bytes, sha256
d3061a2c820889ac31656a3c1cd295bb2b8900ff13f5e882f14f8747e0300df3, built from commit 9c403c8
(image id 22169e249ef9).

GRIMOIRE v1.7

GRIMOIRE v1.7 -- INT4 (GPTQ / AutoRound / W4A16) and speculative decoding on Qwen3.8-27B

Load the image:

docker load < grimoire-b70-v1.7.tar.gz      # -> grimoire-b70:latest

The README has a per-model launch table (arguments, how to turn on MTP / DFlash where it is
faster, measured speed).

What changed

INT4 checkpoints are fast now (GPTQ-Int4, AutoRound, W4A16). Until v1.6 every INT4 weight
ran on a generic SIMT GEMV. v1.7 has GRIMOIRE's own ESIMD INT4 kernels:

• single-user decode: an ESIMD GEMV that keeps the exact (q - z) * scale dequant;
• 2-16 rows (several users, and the speculative verify pass): a DPAS GEMM on the matrix
  engine with INT4 weights and bf16 activations.

Measured with llama-benchy 0.4.0 (512-token prompts, 128 generated tokens, total tokens/s,
GRIMOIRE_SEQ_SLOTS=8), one Arc Pro B70:

Checkpoint	v1.6: 1 / 2 / 4 / 8 users	v1.7: 1 / 2 / 4 / 8 users
Qwen3.8-27B GPTQ-Int4, group 128 (bjonor/Swift-1.5-Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16)	~20 at 1 user (same INT4 kernels as the row below)	34.8 / 48.1 / 86.4 / 145.3
Qwen3.8-27B W4A16, group 64 (an AutoRound export of Qwen/Qwen3.8-27B)	19.9 / 18.3 / 19.5 / 19.5	33.2 / 45.5 / 82.4 / 139.5

Single-user text is identical to the old kernels (checked over 160 greedy tokens on both
checkpoints).

MTP and DFlash are faster than plain decode on Qwen3.8-27B. The verify pass now runs on
the small-M DPAS GEMMs by default (it used a slower path before; GRIMOIRE_SMALLM_DPAS=0
restores it).

Qwen3.8-27B, one user, llama-benchy tg128 (plain decode for comparison):

Checkpoint	Plain	MTP (-e GRIMOIRE_MTP=1 -e GRIMOIRE_MTP_K=3)	DFlash2 (z-lab/Qwen3.8-27B-DFlash2, --dflash-model)
GPTQ-Int4 + BF16 MTP head (SergiioB/...-GPTQ-Int4-sym-G128-MTP-BF16, Swift-1.5)	34.8	51.3 - 54.5
FP8 (Qwen/Qwen3.8-27B-FP8, --proj mxfp4)	33.1	48.5 (v1.6: 17.6)	57.1

On v1.6 the same recipes ran at 30.1 (MTP) and 28.4 (DFlash2), below plain decode.

FP8 checkpoints: the MTP head loads with its block scales. The MTP loader looked its
tensors up without the FP8 weight_scale_inv tiles, so an FP8 checkpoint's head drafted
garbage (MTP ran at half the speed of plain decode). Fixed for the dense head and for MoE
heads' experts.

Muse-Glimmer-30B INT4: single-user decode 18.9 -> 26.9 tok/s (the INT4 kernel), and
with GRIMOIRE_SEQ_SLOTS=8 two and four users no longer collapse (1.6 / 3.0 -> 21.7 / 26.6
total tok/s) -- batching still does not add throughput on Muse, so run it single-user.

Notes

• GRIMOIRE_MTP_DRAFT_VOCAB (an old recipe setting, not used by this image) only limits the
  tokens the MTP draft can propose. Every emitted token is the target model's choice over
  the full vocabulary, so it can never cut tool-call or other high-id tokens from the output.
  The recipes in the README do not set it.
• The server does not support tool calling yet: requests with tools / tool_choice are
  rejected with HTTP 400.
• Known issue, fixed in the next release: the server's text decoder hides every added token,
  including Qwen's <think>, </think>, <tool_call> and </tool_call>, so clients that
  parse tool calls or the reasoning boundary out of the text cannot find them.

Image

grimoire-b70-v1.7.tar.gz, 640,750,822 bytes, sha256
a89ce0b7a7128f79554ada74102857fbf198355624c7ee4c19a70edbb8fb63bf, image id 0ea7472b81e4,
built from commit 191e05b.

GRIMOIRE v1.6

Long prompts under concurrency: 8 users × 4096-token prompts now decode at 390 tok/s total (v1.5: 74). Short-prompt concurrency and single-user speed are unchanged.

Measured with llama-benchy 0.4.0 on one Arc Pro B70, Ornith-1.5-35B-A3B-MXFP4-GRIMOIRE, GRIMOIRE_SEQ_SLOTS=8. Totals are tok/s.

4096-token prompts, 32 tokens generated (pp4096/tg32):

concurrent requests	v1.5 decode	v1.6 decode	v1.6 prompt processing
1	171.5	171.1	9,883
2	213.7	212.4	9,617
4	94.1	296.4	10,022
8	74.1	389.8	10,324

512-token prompts, 64 tokens generated (pp512/tg64), unchanged from v1.5:

concurrent requests	1	2	4	8
decode	193.4	237.9	355.2	466.2

Single request on the default server is unchanged: decode at tg256 is 195.5 tok/s at ~0 context, 183.5 at 2K and 172.3 at 4K. Prefill is ~9,900 tok/s at 4K.

What changed (bec7d2c)

v1.5 admitted at most 8,192 prompt tokens in one batched prefill. Only two 4096-token prompts fit, so the other six were admitted one after another. The first requests then waited ~2.5 s after their first token before decoding resumed.

• One prefill per burst. Simultaneous prompts now go through one prefill of up to 32,768 tokens. Eight 4096-token prompts take one 3.2-second prefill and then decode together.
• Overflow batches. Prompts beyond the cap go into further batches, not one at a time.
• Setting. Change the cap with GRIMOIRE_ADMIT_BATCH_TOKENS. The default is 32768; 0 admits one prompt at a time.

Use

curl -LO https://github.com/doopeworld/GRIMOIRE/releases/download/v1.6/grimoire-b70-v1.6.tar.gz
docker load < grimoire-b70-v1.6.tar.gz          # -> grimoire-b70:latest
docker run -d --name grimoire --network host --init --device /dev/dri/renderD128 \
  -e GRIMOIRE_SEQ_SLOTS=8 \
  -v /path/to/models:/models grimoire-b70:latest \
  server --model /models/<checkpoint> --proj mxfp4 --ctx 8192 --port 8000

Find your B70's render node with ls -l /dev/dri/by-path/. The numbers can change after a reboot. Never set SYCL_UR_USE_LEVEL_ZERO_V2=0.

sha256 699747d09d5fff9a431753b68b78de245162db275f5ed62fcbe561ef0d97928d

GRIMOIRE v1.5

Concurrent serving, round 3: 8 users at 464 tok/s total, 4 users at 355, 2 at 238. Results are now consistent from run to run. Single-user speed is unchanged.

Measured with llama-benchy 0.4.0 on one Arc Pro B70, Ornith-1.5-35B-A3B-MXFP4-GRIMOIRE, GRIMOIRE_SEQ_SLOTS=8, pp512/tg64. Totals are decode tok/s; per-request values are in parentheses.

concurrent requests	v1.3	v1.4	v1.5	v1.5 peak
1	193.4	193.5	193.6	197
2	145.8	215.0	237.8 (119)	242
4	198.3	264–293	354.8 (89)	360
8	255.2	348–392	464.2 (58)	469

Concurrent prompt processing (pp512, 8 at once): 9,547 tok/s total.

Single request on the default server (no GRIMOIRE_SEQ_SLOTS) is unchanged:

context	prefill	decode (tg256)
~0 tokens	~4,400 tok/s	195.7 tok/s
2K tokens	9,570–9,670 tok/s	183.4 tok/s
4K tokens	9,960–9,990 tok/s	172.2 tok/s

What changed since v1.4

• Burst admission (a97da18, 286ad6e). When several requests arrive together on an idle server, the scheduler waits up to 8 ms for the rest of the burst. The whole burst is then prefilled in one pass.
  • A lone request waits only if the previous busy period served several requests at once. A single-user chat never waits.
  • This is why c4 and c8 no longer swing between runs.
  • Change the wait with GRIMOIRE_ADMIT_HOLD_MS (default 8; 0 = never).
• Router top-k for every row in one ESIMD launch (8744f17). It now matches the single-user decode kernel bit for bit. The old batched kernel spent ~21 µs per layer at every batch size.
• Small-batch projections read the norm's bf16 rows directly (8744f17). The fp32 rows are no longer converted again for every projection.

GPU time per batched decode step on Ornith:

rows	v1.4	v1.5
2	8.7 ms	7.9 ms
4	11.5 ms	10.7 ms
8	17.0 ms	16.3 ms

Use

curl -LO https://github.com/doopeworld/GRIMOIRE/releases/download/v1.5/grimoire-b70-v1.5.tar.gz
docker load < grimoire-b70-v1.5.tar.gz          # -> grimoire-b70:latest
docker run -d --name grimoire --network host --init --device /dev/dri/renderD128 \
  -e GRIMOIRE_SEQ_SLOTS=8 \
  -v /path/to/models:/models grimoire-b70:latest \
  server --model /models/<checkpoint> --proj mxfp4 --ctx 8192 --port 8000

Find your B70's render node with ls -l /dev/dri/by-path/. The numbers can change after a reboot. Never set SYCL_UR_USE_LEVEL_ZERO_V2=0.

sha256 1aa2237d2ad2dccaeffa8e1176f322bc83c8332a42554722e0245a4039de140b

GRIMOIRE v1.4

Concurrent serving, round 2: c8 348–392 tok/s total (v1.3: 255). Every concurrency level now scales above single-user speed. Single-user speed is unchanged.

Measured with llama-benchy 0.4.0 on one Arc Pro B70, Ornith-1.5-35B-A3B-MXFP4-GRIMOIRE, GRIMOIRE_SEQ_SLOTS=8, pp512/tg64. Totals are decode tok/s; per-request values are in parentheses.

concurrent requests	v1.2	v1.3	v1.4	v1.4 peak
1	193	193.4	193.5	197
2	95	145.8	215.0 (107)	218
4	133	198.3	264–293 (77–80)	298
8	174	255.2	348–392 (51–53)	414–423

Concurrent prompt processing (pp512, 8 at once): 7,980–8,020 tok/s total. v1.3 managed 4,560.

Single request on the default server (no GRIMOIRE_SEQ_SLOTS) is unchanged:

context	prefill	decode (tg256)
~0 tokens	~4,100 tok/s	195.6 tok/s
2K tokens	9,530 tok/s	183.5 tok/s
4K tokens	9,910 tok/s	172.3 tok/s

The c4 and c8 totals vary between runs. They depend on how simultaneous requests group into admission batches.

What changed

• Batched admission (e449b7c). Prompts that arrive together are prefilled in one forward pass instead of one by one.
  • Per prompt: each runs its own conv, DeltaNet recurrence and causal attention.
  • Shared work: projections, MoE and norms read their weights once for all of them.
  • A single 512-token prompt touches every expert of the model, so one-at-a-time admission read all of the weights again for each prompt.
  • Turn it off with GRIMOIRE_ADMIT_BATCH_TOKENS=0. The default batch cap is 8192 tokens.
• Decode MoE over all rows (bd8b066). Batched decode runs the single-user decode step's MoE kernels, shared expert fused, over every row in two launches.
  • Each row's MoE output is bit-identical to a single-user decode step.
  • GPU time for an 8-row step: 19.3 → 17.0 ms. For a 2-row step: 11.2 → 8.7 ms.
  • Turn it off with GRIMOIRE_MOE_ROWS_MAX=0.

These paths serve batched decode and batched admission only. Single-request serving, speculative decoding and the prefix cache work exactly as in v1.3.

Use

curl -LO https://github.com/doopeworld/GRIMOIRE/releases/download/v1.4/grimoire-b70-v1.4.tar.gz
docker load < grimoire-b70-v1.4.tar.gz          # -> grimoire-b70:latest
docker run -d --name grimoire --network host --init --device /dev/dri/renderD128 \
  -e GRIMOIRE_SEQ_SLOTS=8 \
  -v /path/to/models:/models grimoire-b70:latest \
  server --model /models/<checkpoint> --proj mxfp4 --ctx 8192 --port 8000

GRIMOIRE_SEQ_SLOTS=8 serves up to 8 requests at once.

Find your B70's render node with ls -l /dev/dri/by-path/. The numbers can change after a reboot. Never set SYCL_UR_USE_LEVEL_ZERO_V2=0.

sha256 6ed70d2f6185d6c8ac7a6131e3d017c624fa3221237a66146a9df1a11887abfd