bloomery 0.3.4

bloomery 0.3.4 adds Nemotron 3.5 Bolt 74B-A9B on one 24 GB card, embeddings beside the chat model, and lets
several DGX Sparks serve one model.

Nemotron 3.5 Bolt 74B-A9B (24GB GGUF), tok/s	RTX 3090 (250 W)	RTX A6000
Decode at depth 4096	86.69	87.42
Prompt 512 / 4096	1,773 / 2,047	2,143 / 2,772

One card per column, n=4 rounds in one timing lease per card, decode n=96 tokens.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2. Docker and Homebrew as in the README.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve -m PATH/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf --port 8080

New

• Nemotron 3.5 Bolt 74B-A9B (nemotron_h_moe): decode, a batched prompt path in passes of up to 4096 rows with a
  chunked Mamba-2 scan and an NVFP4 GEMM, and its chat template on the nemotron seat (#302, #303, #304, #309, #312, #314, #317, #320, #328, #400).
• Embeddings: bloomery-serve --embd-model holds EmbeddingGemma 2 or EmbeddingGemma v1 (300M) beside the chat
  model for /v1/embeddings (#299, #323, #390). EmbeddingGemma 2 also embeds sampled video frames and mono 16 kHz
  audio (#374, #375, #383, #397, #405).
• Several DGX Sparks serve one model (by @juliankang4): a tier node on another Spark owns routed experts by id
  range over RDMA and pages them from its own NVMe (#219, #289, #321, #330, #333, #358, #411). Qwen3.8 gives the same
  output as on one Spark; GLM-5.3-Flash and DeepSeek-V4.1-Flash open the same node tier (#420), and their greedy
  output can part from one Spark's when experts move between card and host, since the two quantize activations
  differently. No speed claim is made for multi-node.

Changed

• Nemotron 3 Nano 4B reads a prompt in batched passes instead of one decode step a token (#305).
• Nemotron-H prompt speed, each change bit-identical in its gates (#291, #296, #326, #329).
• Dense Qwen generation runs through the native Qwen seat (#339). EmbeddingGemma and Gemma4 image/video preparation
  got faster and share one cache budget (#412, #417, #418, #419).

Fixed

• Content after a think span starts where llama.cpp's parser starts it (#292).
• Named refusals instead of undefined output: tokenizer codepoints above U+10FFFF (#270, closes #256), placement sum
  overflows and odd host page sizes (#347, #359, #372, #408), trailing partial values and embedded NULs in reference
  readers (#274, closes #261; #334, closes #258).
• /props names the embedding model (#366); a symlinked model directory lists its draft sidecar once (#377);
  probe_nvread handles ext4 large folios (#356); Bolt's resident byte count (#277).
• Developer tooling and gate strictness (just ptx-compare, prediction cards, timing runners, gate and runner
  fixes): see the merged pull requests since v0.3.3.

Known limits

• 12 GB cards (RTX 3060): Bolt does not yet place routed experts in host memory; it needs a 24 GB card. Work in
  progress for the next release.
• Bolt runs on one card: --place bp is refused by name.
• From 0.3.3: Qwen3.6 has no DSpark draft; V4.1 DSpark serves one slot and refuses images; V4.1 DSpark under
  --place a can run out of card memory at load; Nemotron 3 Nano 4B serves one slot on one card with no prompt cache.

Checked before the tag

Release head 9723698f2 (the tag moved here from 44495f0ba on 2026-10-18 to carry #412, #417, #418, #419 and #420). Every needs-gpu
pull request without gpu-verified is covered by the release batch's gates or by its contributor's own card runs.
The tarball's bloomery-serve at the serving defaults (every lever unset, --ctx-size 16384), greedy, one load per
row, on an RTX A6000 + RTX 3090 (250 W) machine:

Model, place	Long prompt: tokens, prefill tok/s, first token	Decode after it	Short prompt decode
Nemotron 3.5 Bolt, unset (picks a)	5,680, 2,708, 2.11 s	87.4 tok/s	88.4 tok/s
Qwen3.8 UD-Q4_K_XL, bp	5,673, 1,710, 3.34 s	84.9 tok/s	97.3 tok/s
Qwen3.6-35B-A3B, unset (picks a)	—	—	147.0 tok/s

Bolt bp is refused by name (above). Tape of Bolt on the release head:
https://tape.midagedev.com/r/d34555nktajc3duqkmab

Thanks

• @juliankang4 for multi-node serving across DGX Sparks, the EmbeddingGemma encoders, video and audio, the Gemma4
  speedups, and many runner and gate fixes.
• @Burntgogi for the tokenizer and reference-reader refusals, the placement overflow checks, and gate strictness and
  gguf tooling fixes.