Release bloomery 0.3.14 · midagedev/bloomery

Re-cut 2026-11-14. The archives below replace the first 0.3.14 upload. They add K-EXAONE-2.0's MTP draft (BLOOMERY_DRAFT=mtp, opt-in), and every number in these notes comes from a new release sitting on the re-cut binary. If you installed 0.3.14 before the re-cut, reinstall to get the draft.

bloomery 0.3.14 serves LG AI Research's K-EXAONE-2.0-750B-A37B on one workstation, from a quantization our own tools
wrote with an importance matrix the engine collected itself. It also carries @yulmucoffee's prompt speed-up for Macs
with matrix units, and a K/V cache that starts small and grows. Numbers below are each PR author's own measurement on
the machine named; the EXAONE and V4.1 rows are ours, measured as stated under their table.

At a glance

K-EXAONE-2.0-750B-A37B (750B, 37B active) served from one RTX A6000 48 GB and 256 GB of host RAM:
IQ2_XXS, 280 GB, written by bloomery-quantize
straight from LG's FP8 checkpoint; decode 9.1–9.3 tok/s, prompt 44–55 tok/s (the release sitting, below). With the
model's own MTP layer drafting (BLOOMERY_DRAFT=mtp, opt-in) decode is 13.6–13.7 tok/s at a 4,096-token prompt, +39 %.
Macs whose GPU has matrix units (M5): prompt +39.8 % ± 3.5 on Qwen3.8-27B 4-bit at P = 512, M5 Pro (@yulmucoffee, #1187).
Qwen3.8 serving starts its K/V cache at 8,192 positions and grows it on demand; the card bytes past the floor hold
experts until the cache needs them.
A server's later requests keep their own prompt's expert pick: +10.4 % decode and +18 % prompt on Qwen3.8 at depth
4,096 (RTX A6000, one lease, 4 rounds; #1190).

K-EXAONE-2.0-750B-A37B

The file. midagedev/K-EXAONE-2.0-750B-A37B-GGUF,
29 shards, 280.15 GB: the routed experts' gate and up at IQ2_XXS (with the importance matrix), their down at Q4_K,
attention, the dense FFN, the shared experts, the embeddings, the head and the MTP layer at Q8_0. The card on the
Hub carries the weight errors per type and our benchmark runs against DeepSeek V4.1-Flash.
Made by bloomery. bloomery-quantize --hf read the FP8 safetensors folder and wrote the shards in one pass
(8,296 s, the checkpoint on a hard disk); the IQ2_XXS tensors were quantized on the card. The importance matrix came
from the engine's own collector (generate_exaone --imatrix) over a bootstrap Q4_K quant: 32 chunks of 2,048 tokens
of Wikipedia text (Korean, then English).
Serving.
```
bloomery-serve --model flint -m K-EXAONE-2.0-750B-A37B-IQ2XXS-00001-of-00029.gguf --ctx-size 8192
```
At --ctx-size 8192 the plan puts the dense trunk and 2,319 of the 19,456 routed experts on a 48 GB card (771 at
the default context, 262,144, whose K/V takes the rest of the card; decode 9.8 tok/s at ctx 8,192 against 9.1 at the default) and serves the rest from host
RAM; the load reads about 263 GB through the page cache, so 256 GB of RAM is the floor. One slot. The model thinks
unless the request sets "chat_template_kwargs": {"enable_thinking": false}.

Card	`--place`	Prompt length	Prompt tok/s (two rounds)	Decode tok/s (two rounds)
RTX A6000	`a` (unset)	512	44.39 / 50.60	9.13 / 9.06
RTX A6000	`a` (unset)	4,096	54.67 / 54.85	9.10 / 9.25

The release sitting (tools/release/sitting.sh) on this release's bloomery-serve from the sm86 tarball, under
the timing lease: one fresh server per row with every lever unset and --ctx-size unset (the plan picks 262,144),
128 tokens generated per request, temperature 0, on our workstation (RTX A6000 48 GB, Threadripper PRO 5975WX,
8-channel DDR4-3200, 251 GiB of RAM, the shards on a PCIe 4 NVMe). Three of the four rows are marked [cold] (3,845 to
127,863 major faults a request): the load reads the whole 280 GB through the page cache of a 251 GiB host, so part of
the host's experts is paged in again. The one warm row (50 faults) decodes at the cold rows' rate: the decode is
bound by the host's memory reads, not by the page cache. The first-request checks passed: a short
and a 4,950-token prompt, and a replay of the first request gave the same ids.

The MTP draft (opt-in). The file's next-token layer drafts one token a step and the trunk verifies it in a
two-row pass: BLOOMERY_DRAFT=mtp bloomery-serve --model flint -m … --ctx-size 8192. One slot. Its card bytes come
from the plan, so the card holds 51 fewer experts (2,268 against 2,319 at ctx 8,192). Same binary, same lease, three
rounds, ctx 8,192, 128 tokens at temperature 0, the off and mtp arms interleaved:

Prompt length	Decode tok/s, draft off	Decode tok/s, `BLOOMERY_DRAFT=mtp`	Paired difference
512	9.89 / 9.85 / 9.67	11.13 / 12.54 / 10.47	+12.5 / +27.3 / +8.3 %
4,096	9.83 / 9.84 / 9.78	13.66 / 13.64 / 13.67	+39.0 / +38.6 / +39.8 %

The draft is kept 78 to 84 % of the time at 4,096. At 512 it is kept more often and the gain still scatters; we have not
read why yet, so the draft stays off by default in this release. A drafted run on the same card plan as a plain run gives the
plain run's tokens, bit for bit; at the defaults the draft's reserve moves 51 experts to the host, which can move a
float sum and so, rarely, a token.

Not in this release. A 24 GB card and two cards were not tried.
With thinking off and a one-letter answer the model leans to the last option on multiple-choice questions it is
unsure of; llama.cpp on the same file leans the same way.

DeepSeek V4.1-Flash, the release sitting

The same sitting, Q3_K_M at --place bp on RTX A6000 + RTX 3090, ctx unset (the plan picks 1,048,576):

Prompt length	Prompt tok/s (two rounds)	Decode tok/s (two rounds)
512	168.69 / 227.80	44.05 / 45.66
4,096	428.43 / 427.61	39.82 / 38.85

The first round's 512 row was [cold] (75,841 page faults; its prompt rate is the low one); the others paid 12 to
19,847. 0.3.13's sitting rows on the same file and cards paid 27,000 to 87,000 (all [cold]), so the two releases'
rows were not read under one condition, and we do not claim a speed-up from them. The plan holds 2,599 card experts where 0.3.13's held 2,465.

Faster prompts on M5 Macs (@yulmucoffee, #1187)

The tensor-op GEMMs, the path a GPU with matrix units takes, now read the f16 operand's rows as half rows cast once a
pass, as the simdgroup GEMMs do. The output bits are unchanged. On an Apple M5 Pro (16-core GPU, macOS 26.5.1), their
measurement, five alternating rounds:

Model, P	main before (`004a23b`) tok/s	with #1187 tok/s	Difference
Qwen3.8-27B 4-bit prompt, 512	274.6	383.9	+39.8 % ± 3.5
Qwen3-4B 4-bit prompt, 512	1,738.9	2,462.9	+41.6 % ± 3.7
Qwen3-4B 4-bit prompt, 4,096	1,357.4	1,751.3	+29.0 % ± 2.2
decode (both models)			−0.3 % ± 0.9 to −0.0 % ± 0.4

@yulmucoffee also fixed the Metal gate's shader-validation reader for macOS 26.5's report form (#1180). Thank you.

A K/V cache that grows

Qwen3.8 serving starts its K/V at a floor of 8,192 positions and grows it as a sequence gets longer; the plan gives
the card bytes past the floor to experts (#1190). Past the floor, a sequence takes card bytes back from the coldest
experts (#1186), and BLOOMERY_KV_HOST=spill|all puts K/V pages in host memory behind the same pointer (#1188,
opt-in). BLOOMERY_KV_FLOOR=off reserves the whole context up front, as before. The floor, the growth and the host
pages have one common owner (runtime::stores, gpu::room::kv); V4.1 and K3.1 follow.
A fit past the card's memory is a named error (NoRoom: the store, the positions, the bytes asked, the free bytes),
not a raw driver OOM (#1185).
Every load now holds what it allocates to what it planned on its card: the pass heads, the captured graphs, the ring
floor and the prompt batch parts are planned terms (#1191). A two-slot or drafted load no longer fails its first
prompt past the card plan.

Fixes and other changes

bloomery-serve-ds41 steps the context down when the host, not the card, is short (the K/V spill or the NVMe tier's
floor), as it does for a card short, and names why it reads the source file when the r8 sidecar cannot be used (#1195
and the commits after it).
A server's later requests keep their own prompt's expert pick (#1190, above).
Under BLOOMERY_EXPERT_RANK the NVMe expert tier pages each layer's coldest host experts, not the highest ids
(#1176). An expert-tier node is skipped on every layer that routes none of its experts (#1177).
Kimi K3.1 Pro past 2,048 positions (#1183, #1184) was already in 0.3.13's re-cut archives.
Nemotron-H-MoE runs the hybrid trunk's walk in crates/chain (#1192, #1193; no behaviour change).
Verification: a build lane, narrower keys, a release blocking set and a nightly real-tier run (#1196).

Archive	For	sha256
`bloomery-0.3.14-linux-x86_64-cuda-sm86.tar.gz`	Linux x86-64, NVIDIA (Ampere and later; the default)	`367d5987…`
`bloomery-0.3.14-linux-x86_64-cuda-sm120a.tar.gz`	Linux x86-64, RTX 50 series and RTX PRO Blackwell (compute capability 12.0)	`a15450f7…`
`bloomery-0.3.14-linux-aarch64-cuda-sm121a.tar.gz`	DGX Spark and other GB10 machines	`e6885ddd…`
`bloomery-0.3.14-macos-arm64-metal.tar.gz`	Apple Silicon, macOS 26.2 or newer (Developer ID signed and notarized)	`4e71aca6…`

Navigation Menu

Repository navigation

bloomery 0.3.14

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found