Skip to content

bloomery 0.3.8

Choose a tag to compare

View all tags
@midagedev midagedev released this 22 Oct 10:11
· 666 commits to main since this release
v0.3.8
06dbd85

bloomery 0.3.8 is a speed release. On one RTX 3090 at its 250 W cap, bloomery-serve at the serving defaults,
four rounds. Decode is tg tok/s at a 6-token prompt with 128 generated tokens. Each row's interval is in New.

before after
Qwen3.6-35B-A3B UD-IQ4_XS decode 91.94 135.62 tok/s
Qwen3.6-35B-A3B Q8_0 decode 68.73 89.60 tok/s
Qwen3.6-35B-A3B UD-Q6_K decode 81.59 104.96 tok/s
Qwen3-30B-A3B Q8_0 decode 49.62 59.08 tok/s
Qwen3-30B-A3B Q6_K decode 59.89 67.47 tok/s
Raon-1-mini pp512 312.4 2,507.5 tok/s
Raon-1-mini pp4096 298.8 1,880.4 tok/s

The Qwen rows are expert stacks that used to stay on the host and now run on the card. Raon's prompt runs in wide
units of up to 256 rows, which closes 0.3.7's known limit. Raon decode is unchanged within ± 0.5 %.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2. Docker and Homebrew as in the README.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve -m PATH/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --port 8080

New

All numbers below: one RTX 3090 at its 250 W cap, bloomery-serve at the serving defaults (--place and ctx unset),
tools/ref/depth-serve.sh, four rounds, base = the parent commit on main. Decode is tg tok/s at a 6-token prompt with
128 generated tokens.

  • Qwen3.6-35B-A3B on the card leg (#629, #645).

    • Before, the card leg refused Q8_0, Q6_K, IQ4_XS and IQ3_S expert stacks, so every layer of such a file stayed on
      the host.

    • #629 adds the Q8_0, Q6_K and IQ4_XS arms (a new Q6_K gate·up kernel). #645 adds an IQ3_S prompt GEMM
      (gemm_iq3s).

    • Measured, ctx 262,144:

      file before after
      Q8_0 68.73 89.60 (+30 %)
      UD-Q6_K 81.59 104.96 (+29 %)
      UD-IQ4_XS 91.94 135.62 (+47.5 % ± 7.6 %)

    • UD-IQ4_XS: the plan now puts all 10,240 experts on the card (0 before).

    • Fine-tunes on the same architecture (Sage-A1.5, Ornith) take the same path.

  • Qwen3-30B-A3B on the card leg (#640).

    • Qwen3-30B-A3B and its fine-tunes now read the same card-leg arms as Qwen3.6, from one place.

    • Measured, ctx 131,072:

      file card experts before after
      Q8_0 1,746 of 6,144 49.62 59.08 (+19.1 % ± 1.3 %)
      Q6_K 2,306 of 6,144 59.89 67.47 (+12.7 % ± 6.6 %)

    • Prefill at the same prompt also rose: Q8_0 81.4 → 108.2 tok/s, Q6_K 92.6 → 130.9 tok/s.

  • Raon-1-mini prompt processing (#627). It closes 0.3.7's known limit.
    • The prompt ran in chunks of 8 rows. It now runs in wide units of up to 256 rows, on the common GEMM, KDA and route
      owners.
    • Measured: pp512 312.4 → 2,507.5 tok/s, pp4096 298.8 → 1,880.4 tok/s. Decode is unchanged within ± 0.5 %.
    • A Q5_0 down stack now has a prompt route (gemm_q5_0p).
  • MiMo-V2.6 tool calls parse (#634): the compact Qwen3-Coder framing, streamed and not. A framing the server
    cannot read answers 501 by name; it does not guess.
  • Nemotron-3.5-Lightning (#646) on the existing nemotron seat, beside Nano and Bolt. No latent: the experts
    act on the hidden (2,688). The router keeps the top 6 of 128. The shared expert is Q8_0. No decode number in
    this note.
  • DSpark draft (#638) on the qwen3 seat. BLOOMERY_DRAFT=dspark serves Qwen3.6 with Red Hat's speculator and
    MiniCPM-V 4.7 with its own draft. Sage-A1.5 and Ornith-1.5 take that draft by the file's facts. A lookup that
    proposes nothing is refused. No decode number in this note.
  • macOS host chain (#611). The host page, memory, performance levels and peak RSS run on macOS. Free and inactive
    pages count as available, and a negative RSS peak is refused. This is on main after the v0.3.7 tag.

From our contributors

New in this release:

  • Qwen3.8 straight from a checkpoint folder (#635, by @juliankang4).
    • bloomery-serve -m <folder> and --hf <repo> serve NVIDIA's NVFP4 Qwen3.8 checkpoint as downloaded, with no
      prepare step. --hf also takes a repo that holds safetensors and no GGUF.
    • On one DGX Spark (GB10), with all 24,576 routed experts on the card: decode 30.26 tok/s after a 4,096-token prompt.
      The UD-Q4_K_XL GGUF of the same model reads 28.32 tok/s (#561; one request each, a record, not a verdict).
  • Tensor-parallel ranks read their parts from a GGUF file or a checkpoint folder (#618, by @juliankang4).
  • The start screen: the serve screen draws the wordmark and furnace over the placement panel (by @juliankang4).
  • Fixes by @juliankang4:
    • A draft window that no verify takes no longer leaves stale draft state (#631).
    • Recurrent stores are zeroed on the device at reset.
    • A split pick is a function of the prompt alone.
  • Fixes and design by @Burntgogi:
    • compressed-tensors targets resolve by the checkpoint's module class (#605);
    • the partial head-256 cache append is marked unsafe, with its contract (#612);
    • the census and lift design for layer composition (#621).

Shipped in earlier releases, with their numbers (each measured on the contributor's machine):

  • Two DGX Sparks for one model (#358, #411, by @juliankang4). A tier node on a second Spark binds its own RDMA end
    and pages its experts from its own NVMe. Qwen3.8 decode on GB10: one Spark 20.3–20.5 tok/s, two Sparks 22.2–22.3
    tok/s (64 tokens, one direct RoCE link).
  • --place tp for GLM-5.3-Flash's Hugging Face checkpoint (#519, by @juliankang4). With 2 ranks, each rank holds
    91.6 GB of weights, 50.5 % of what one rank holds alone (two DGX Sparks, a 200 Gb/s link).
  • The n-gram lookup draft on Qwen3.8 (#546, by @juliankang4). On one DGX Spark, code prompts: 29.78 → 35.83 tok/s
    (+20.3 %).
  • EmbeddingGemma 2 beside the chat model (#323, #324, #383, #397, #405, #414, by @juliankang4). /v1/embeddings
    takes text, images, video sampled at 1 FPS, and PCM audio; prompt_name picks the prefix.
    On one RTX 3090 with 0.3.8, beside Qwen3.6-35B-A3B Q8_0 at the serving defaults (--embd-model; client-timed,
    median of 7): one 14-token input in 6.1 ms; 32 short inputs (566 tokens) in 159 ms; 16 inputs of ~550 tokens
    (8,822 tokens) in 211 ms, 41,800 tokens/s.
  • MXFP8, FP8 and BF16 weights (#521, #562, #579, #592, #607, by @juliankang4). On GB10 the Q8_0 path's baseline in
    the same sitting: Qwen3.8 decode 178.74 / prefill 5,970.53 tok/s, GLM-5.3-Flash decode 86.40 / prefill 403.38 tok/s.
  • AMD: a native HIP forward of dense Qwen3.5 (#476, by @Burntgogi). All 24 layers and the output head run on HIP and
    match the reference.
  • Serve request guards (#590, by @Burntgogi): request heads are authorized and network image decoding is bounded.

Fixed

  • crates/placement: gemm_reads_blocks32 takes Q5_0 (main did not build after #627 and #629 met).
  • Clippy warnings on main back under the ratchet (25 → 3).
  • Developer tooling: see the merged pull requests since v0.3.7.

Known limits

  • Off cells of the user-facing pass. Each is a named refusal at load, not a crash:
    • Qwen3.6-35B-A3B and Qwen3-30B-A3B at --place bp: "1 expert tier cards, and the qwen3moe body serves 0 at most". These bodies serve one card with the host tier (--place a, the default).
    • Nemotron-3.5-Lightning at --place bp: a placed load runs one card with the host tier beside it.
    • Bolt at --place bp (from 0.3.4).
  • Nemotron-3.5-Lightning's UD-Q4_K_XL file is a placed load. The whole-card expert body is IQ2_XXS, which is Bolt's. On the A6000 the whole plan at 4,096 positions leaves 2,944 experts on the host, and the server opens placed. On the RTX A6000 at the serving defaults (--ctx-size 16384, 2,944 experts on the host): decode 89.2 tok/s after a 25-token prompt, 79.9 tok/s after a 5,866-token prompt (prefill 451 tok/s).
  • Real tier still open, carried from the release batch: gate-refset and gate-gpu-qwen4exp-nvfp4. Both read the Qwen3.8 NVFP4 checkpoint and its reference sets, which the maintainer's box does not hold (they are checked on the contributor's DGX Spark).
  • From 0.3.7: Raon's draft rarely pays at these prompts (#617). The prompt-processing limit from 0.3.7 is closed above.
  • From 0.3.6: gate-qdot's IQ4_XS clause is red by 5 ULP.
  • From 0.3.5: V4.1 on two cards (--place bp) is not faster than one card.
  • From 0.3.4: Bolt needs a 24 GB card and refuses --place bp.

Checked before the tag

  • Release batch on f02460d16, real tier, 3,674 s: 91 items, 5 red, 2 deferred. The reds were gate-gpu-clef-serve, gate-gpu-qwen3-serve, gate-refset, gate-gpu-qwen4exp-nvfp4, gate-gpu-nvtier. Deferred (needs-env): gate-prepare-corpus, gate-gpu-qwen38-nodes-remote.

  • The Qwen3.6 card-leg fixture variants (gate-gpu-qwen35moe-leg, including IQ4_XS, IQ3_S, Q8_0, Q6_K and the mixed down) were green at 6b0a82f91. Files touched after that commit do not include those leg kernels.

  • Closed after that batch, on later commits: clef serve (e0226df06); qwen3 serve on the 3090 at 578b62d09 (192 checks, 0 FAIL, 409 s) and its contrib-check t0. nvtier was not re-pinned: a warm quiet A6000 run on 578b62d09 was late 0, worst margin 1.60, 258 s; one idle plan was late 19.

  • DSpark on the A6000: gate-gpu-qwen3-dspark 77 s, exit 0, on the tree replayed as bf54f5912.

  • Lightning: fixture gate_nemotron_serve on the 3090, 7 s, exit 0. The real UD-Q4_K_XL file on the A6000, 15 s, exit 0: the server's 16 prose tokens and 16 chat tokens matched the placed session, and the held-split extension matched. That binary was built before 4b884ea3d. That commit changes bf16_gemv and the BF16 arm of site::gemv_planes only.

  • BF16 on the 3090 at 8a471012a: gate_bf16 16 s, exit 0, no FAIL. @juliankang4's run of the same gate on a DGX Spark: 13 s, exit 0. Spark tok/s are not in the tables above.

  • contrib-check t0 on 8a471012a succeeded (run 39752930321).

  • Router move 6e9c9bbc1..f53aa93d7: --no-jit canon-check, 3 bins, 1066 entries, 1066 identical, 0 differs.

  • User-facing pass of the candidate tarball (bloomery-serve 0.3.8 (commit 06dbd85f797c)), serving defaults (no --parallel, no lever, --ctx-size 16384), both cards visible: every a cell planned on the RTX A6000 (48 GB), the larger card; bp on RTX A6000 + RTX 3090. At this context the whole Qwen3.6 Q8_0, Qwen3-30B Q8_0 and Raon files fit the A6000, so these rows are not comparable with the 3090 tables above. Decode tok/s after a short prompt / after a ~5.9K-token prompt (prefill tok/s):

    | model | place | short | long (prefill) |
    |---|---|---|---|
    | Qwen3.6-35B-A3B Q8_0 | a | 134.4 | 111.5 (3,952) |
    | Qwen3-30B-A3B-Instruct-2507 Q8_0 | a | 150.0 | 105.1 (4,567) |
    | Raon-1-mini Q4_K_M | a | 152.7 | 139.6 (2,516; 4,061 tokens) |
    | Nemotron-3.5-Lightning UD-Q4_K_XL | a | 89.2 | 79.9 (451) |
    | Qwen3.8-Flash-Next UD-Q4_K_XL | a | 69.5 | 66.0 (1,848) |
    | Qwen3.8-Flash-Next UD-Q4_K_XL | bp | 84.3 | 80.9 (1,767) |

    One request each: a record, not a verdict. The off cells are listed under Known limits.

Thanks

  • @juliankang4 for the NVFP4 checkpoint serving, the tensor-parallel part reads, the start screen, and the draft and
    reset fixes.
  • @Burntgogi for the compressed-tensors resolution, the head-256 cache contract and the layer-composition design.

Contributors

juliankang4 and Burntgogi
Assets 4
Loading