bloomery 0.3.13

Re-cut on 2026-11-08 under the same version. The Linux archives (sm86, sm120a, GB10) were rebuilt at
79893871104b (v0.3.13's first commit plus K3.1's sparse attention and prompt passes; the tag, 66b6a64b8, adds
docs only). Kimi K3.1 Pro now runs past
2,048 positions: past the lightning indexer's top-k the latent layers attend the rows the indexer picks, in the decode
step and in the prompt passes, and the seat runs a prompt as placed passes. The macOS archive is the first build's
(no Metal change). If you downloaded 0.3.13 before this, the sha256 of the Linux archives changed; re-download.

bloomery 0.3.13 serves Kimi K3.1 Pro (1T parameters) from a quantization our own tools wrote, runs DeepSeek V4.1-Flash
faster with a DGX Spark as the expert tier over plain TCP, and carries @juliankang4's tensor-parallel work on the
original V4.1 folder. Numbers below are each PR author's own measurement on the card named; the Kimi rows are ours,
measured as stated under their table.

At a glance

• Kimi K3.1 Pro (1T) served from Kimi-K3.1-Pro-IQ1_S, 580 GB written by bloomery-quantize, past 2,048 positions
  (its sparse attention); prompt 12.5 tok/s as passes, decode 2.94 (4.99 / 3.80 under --prefill steps) on RTX A6000
  • RTX 3090 with the set paged from two NVMe drives.
• Two NVMe drives, one sidecar file: decode +32 % (V4.1-Flash), +40 % (Mistral Large 4), +49 % (GLM-5.3-Flash) on the
  same binary.
• DeepSeek V4.1-Flash with a DGX Spark as the expert tier over plain TCP: decode 19.4–19.6 → 48.9 tok/s, prompt 29 → 210.5.
• Tensor-parallel V4.1 on the original folder, the Qwen3.8 MXFP8 autotune and the GDN precision fix (@juliankang4);
  Metal: the Qwen3.8 MTP sidecar and lookup decoding.

Kimi K3.1 Pro, quantized by bloomery

• Kimi-K3.1-Pro-IQ1_S (580 GB, 31 shards) was written by bloomery-quantize (crates/quantize): the routed
  experts at IQ1_S on the card, byte-identical to ggml's quantizer on the same input; the dense trunk at Q4_K to Q8_0
  on the host, byte-identical to llama-quantize's plan and writer. The checkpoint was read by our prepare
  (crates/model-hf) from Moonshot's MXFP4 safetensors. The importance matrix still comes from llama-imatrix, over
  a chat corpus K3.1 wrote (27 chunks of 8,192 tokens); the engine's own collector is the next release's.
• The set's chat template is the one from unsloth's Kimi-K3 GGUF (the K3.1 checkpoint ships none in its files; its
  encoder is Python). The reply's sections are split as the template spells them: the think span goes to
  reasoning_content and the response section's text to content, its markup in neither (reasoning_format
  and thinking behave as for the other seats).
• Quality against the released model (OpenRouter's Kimi K3.1 Pro, top-20 logprobs), 150 prompts over English, Korean
  and code, 35,226 positions, teacher forcing through generate_kimi_k3 --score at --place bp:

Set	Top-1 agreement	ΔNLL, nats	KL to the teacher's top-20
English	0.840 ± 0.018	0.289 ± 0.037	0.105 ± 0.017
Korean	0.877 ± 0.009	0.218 ± 0.022	0.082 ± 0.009
Code	0.913 ± 0.011	0.133 ± 0.018	0.052 ± 0.006
All	0.877 ± 0.009	0.213 ± 0.018	0.080 ± 0.007

• Serving: bloomery-serve --model kimik3 --place bp --ctx-size 2048 (and --ctx-size 8192 for the 4,096-id row) on RTX A6000 + RTX 3090 (the dense trunk and the
  hottest experts on the cards, the host's share of the experts in RAM, the rest paged from NVMe), one cold request of
  512 ids after a prompt of 512 ids (temperature 0, the rest of the sampling at its defaults), the release tarball's own
  bloomery-serve. Two things are set beside the defaults: BLOOMERY_HOST_ROOM=232G (the plan reads the live host
  room, and without the pin two loads of the same set are not one plan), and the set sits on the box's two NVMe drives,
  a byte-identical copy of the shards on the second drive named by a Kimi-K3.1-Pro-IQ1_S.shard-dirs sidecar beside the
  first (serving.md 5.4): the NVMe tier's reader then reads each expert's parts from the drive its id picks, so one
  layer's misses land on both drives at once. The set is larger than the host, so every request pages experts from the
  drives. The seat runs a prompt as placed passes by default; under --prefill steps it steps each id, as the first
  build of this tag did. The pass's plan keeps fewer experts in the NVMe arena (22.6 GB against 228.8 GB), so a step
  decodes slower: steps is the faster pick when the reply is longer than about 1.3 times the prompt.

Cards	--place	Drives	D	Decode tok/s	Prompt tok/s
RTX A6000 + RTX 3090	bp, passes (default)	two (sidecar)	512	2.94	12.48
RTX A6000 + RTX 3090	bp, --prefill steps	two (sidecar)	512	4.99	3.80
RTX A6000 + RTX 3090	bp, passes, --ctx-size 8192, 128 generated	two (sidecar)	4,096	2.33	12.12
RTX A6000 + RTX 3090	bp (the first build, stepped)	two (sidecar)	512	4.94	3.85

Against this release's own tree before the two host levers landed (the same request with 1,024 generated ids, one
drive, no sidecar) the decode rate was 3.80 and the prompt rate 2.57: the step load now prepares its union slabs once (the hits are computed
while the misses fill) and the reads split over two drives. With one drive and no sidecar the release binary reads the
shards where they are. Earlier in this release's tree (its pre-release binary, BLOOMERY_HOST_ROOM=252G, one drive,
two rounds, 128 generated ids) the same cards read 3.04 / 3.05 at D = 512 and 2.27 / 2.34 at D = 1,536, and with a
DGX Spark as the expert tier node over 10 GbE TCP in place of the RTX 3090 (BLOOMERY_TIER_NODES=<spark>:9000,
BLOOMERY_TIER_LINK=tcp, tier_server from the same tree on the Spark, BLOOMERY_CARD_BUDGET=100G) 4.00 / 4.05 at
D = 512 and 3.17 / 3.23 at D = 1,536; those rows are the tree's, not the tarball's, and are not re-measured here.

The two-drive pick is not K3.1's alone. The same binary, the same set with and without its sidecar, one cold request
of 512 ids then 128 generated, three rounds each with the order rotated, the host room pinned so the plan pages
through the tier's arena (docs/cards/mirror-ab-v41.card, -ml4, -glm), RTX A6000 + RTX 3090:

Model	Room	--place	Decode tok/s, one drive → two	Prompt tok/s, one drive → two
DeepSeek V4.1-Flash Q3_K_M	80G	bp	16.9 → 22.4 (+32 %)	26.1 → 40.5 (+55 %)
Mistral Large 4 UD-IQ1_S	80G	a	3.02 → 4.23 (+40 %)	13.5 → 21.5 (+59 %)
GLM-5.3-Flash UD-Q4_K_XL	48G	bp	5.51 → 8.22 (+49 %)	30.3 → 54.0 (+78 %)

One drive read at 4.2–4.7 GB/s over these requests, two at 5.9–7.2 GB/s: the second drive adds its rate, and the
first's queue is no longer the wall. The sidecar does nothing when the plan pages through the file mapping instead of
the arena (a room that moves under half the host leg's bytes prints arena=0: V4.1 at 128G, GLM at 80G), so these
rooms are the ones where the plan builds its default arena; a plan that pages little reads near 0 by construction.

• The K3.1 context runs past 2,048 positions a slot: past the lightning indexer's top-k (2,048) each latent layer
  attends the rows the indexer picks, in the step and in each pass row (every pass row the step's bits,
  gate-gpu-kimik3-dsa and gate-gpu-kimik3-place). --place bp plans up to 262,144 positions on these cards; contexts
  past 8,192 were planned, not timed. Kimi K3's UD-IQ1_S (unsloth, 594 GB) does not load at this tag: its dense trunk is 51.4 GB, past
  one 48 GB card; the seat and its gates hold the architecture on a fixture cut from that file.
• Tooling: bloomery-quantize --hf DIR reads an HF checkpoint folder directly, with no intermediate GGUF
  (gate-quantize-hf: header and sha256 per shard equal to the two-step path on a cut of the real folder);
  tools/ref/k31-quant.sh is the whole pipeline (prepare, imatrix, quantize, splice, upload);
  tools/ref/k31-eval.py the quality check above.

DeepSeek V4.1-Flash with a DGX Spark as the expert tier

A DGX Spark over a 10 GbE TCP link (BLOOMERY_TIER_NODES) holds part of the routed experts beside the box's cards and
host; ours, on the box (RTX A6000 stage card, 256 GB host) with one Spark, generate_ds41, prose prompt of 512 ids,
depth 512, n = 96 (#1173, #1175):

Host room	Tier	Decode tok/s	Prompt tok/s
128 GB	none	19.4–19.6	29
128 GB	one Spark, id order	34.7–35.0	104–108
128 GB	one Spark, BLOOMERY_EXPERT_RANK	48.9	210.5

• The node computes its own share of a prompt batch (#1173), so prefill with a node is 2× what it was.
• BLOOMERY_EXPERT_RANK=<router-rank file> (#1175, tools/ref/router-rank.py rank): the hottest experts of each
  layer on the card, the coldest on the node, the middle on the host. The rank above was learned on router sets that
  include the measured corpus; the cross-corpus number is still owed. Opt-in: BLOOMERY_PREFILL_CUT (#1173).
• Engram: V4.1-Flash GGUFs in upstream llama.cpp's concatenated table layout (unsloth, bartowski, …) open beside the
  per-site one; gguf-inventory --engram names a file's (#1174).

Tensor-parallel V4.1 on the original folder (@juliankang4)

• generate_ds41 --place tp runs one rank of a tensor-parallel greedy generation of the original-format DeepSeek
  V4.1 checkpoint folder, one process per rank over the fabric (#1160, #1170), checked at one, two and four ranks.
• The Qwen3.8 load picks the fastest way to run the MXFP8 gemv at four to eight columns on the card in use and keeps
  the pick in the cost store; --autotune off keeps the fixed rule (#1161, #1164).
• A routed MXFP4 down off the 256-value super-block reads the 32-value form (#1154); the GDN prompt delta keeps the
  one-token rule's precision (#1140, #1145); t0 runs under uutils coreutils (#1153).

Also in this release

• Apple Silicon: the Qwen3.8 MTP sidecar opens beside the model and generate_metal --draft mtp drafts with the
  window CUDA runs (#1144, #1158); two-row verify and lookup decoding (#1137); the MTP draft's policy moved to
  runtime behind a Host trait, one owner for CUDA and Metal (#1155).
• The lightning indexer and the input-form index key are common code (crates/gpu/src/{indexer,index_key}.rs).
• prepare (crates/model-hf) reads Kimi's tiktoken vocabulary and the Kimi K3 / K3.1 checkpoint map; --drop,
  --only, --arch, --follow and --room.

Known issues

• Kimi K3.1 Pro: the prompt pass's plan keeps fewer experts in the NVMe arena, so decode is 2.94 tok/s against 4.99
  under --prefill steps (D = 512, 512 generated); a plan that keeps the arena under the pass is the next work. A
  context past 8,192 is planned, not timed.
• Kimi K3.1 Pro's imatrix is llama-imatrix's; the engine's own collector (generate_kimi_k3 --imatrix) lands after
  this tag.
• The K3.1 set's chat template is unsloth's Kimi-K3 template, not one Moonshot ships (the checkpoint has none).
• The expert tier on another machine serves no prompt batch (the batch is the host's), a node card's room is its
  whole free reading unless BLOOMERY_CARD_BUDGET caps it, and the RAM arena (BLOOMERY_NVME_ARENA) refuses a plan
  with a rank file by name.
• (carried) GLM-5.3 at --place bp: a short request replayed after a long one can answer different text.
• (carried) V4.1 on one 24 GB card from a file with an r8 sidecar: refused at load by name; BLOOMERY_R8=off.

Headline rows

The release sitting on the re-cut's shipped bloomery-serve (sha256 cd27f240…, the sm86 tarball's member; the tarball 2d5e1100…), V4.1-Flash
DeepSeek-V4.1-Flash-1017-Q3_K_M with its r8 pair, at the serving defaults (--place unset → bp, --ctx-size unset →
1,048,576, no lever), one fresh server per request, RTX A6000 + RTX 3090 (the 3090 at its 250 W cap), two rounds, D = the
prompt's ids, 128 tokens decoded. Every row is a fresh server's first request ([cold]: 26–66 k major faults each).

Round	D	Decode tok/s	Prompt tok/s	Load s
1	512	40.80	104.85	85
2	512	38.15	96.62	56
1	4,096	24.29	281.24	63
2	4,096	33.29	323.25	55

The first build's sitting on its own binary (sha256 cf9fadae…) read 41.36 / 40.53 decode at 512 and 35.80 / 37.37 at
4,096. The re-cut changes no V4.1 code (its commits touch Kimi K3.1, the kimik3 seat and Raon1's shared latent file,
whose PTX is identical: just canon-check, 2,016 entries). Round 1's 4,096 row read 24.29 against round 2's 33.29; we
found no code change behind it, and the rows stand as measured.

Beside 0.3.12's sitting of the same file and cards (decode 40.83 / 40.94 at 512, 36.46 / 36.07 at 4,096; prompt 123.79 /
111.39 at 512, 330.85 / 340.17 at 4,096): the 512 rows and round 2's 4,096 row hold within one sitting's scatter;
round 1's 4,096 row does not. Before the first build, two sittings of this release's tree two commits earlier (29fd27d, before the sidecar's finishing commits) read 40.84 / 42.60 and 38.69 / 36.42 decode; a third,
just before them, read 41.58 / 35.26 but ended red on its replay check (the same request sent twice to one server answers the same ids):
the replay split at its second token with the two candidates' gap just over the near-tie band, while adaptive residency was moving
experts. The first build's sitting replayed identically; the re-cut's split at token 5 of 13 with the gap
inside the band (green). Off cell at this tag: --place a
on the RTX 3090 alone is refused by name on our box because the r8 pair sits beside the file (README known limits); a plain
Q3_K_M file serves at a. The re-cut's Kimi K3.1 rows above were measured in their own leases on the re-cut's sm86 tarball (bloomery-serve sha256
cd27f240…). The first build's stepped row was measured on the sm86 tarball built two commits before that build (29fd27d,
bloomery-serve sha256 7b89614b…). The kimik3 plan reads the live host room, so every K3.1 row carries
BLOOMERY_HOST_ROOM=232G.

Thanks

To @juliankang4 for the tensor-parallel ranks on the original V4.1 folder, the MXFP8 autotune and the GDN precision
work on the RTX 5090 and DGX Spark.