Skip to content

bloomery 0.3.5

Latest
Latest

Choose a tag to compare

View all tags
@midagedev midagedev released this 19 Oct 12:14
· 32 commits to main since this release
v0.3.5
1ef252d

bloomery 0.3.5 serves DeepSeek V4.1 Flash 1017 to DeepSeek Harness (dsh) with no shim: tool calls in the
spelling the model writes, tool-call arguments that stream as they are written, --chat-template-file on every seat,
and thinking from reasoning_effort.

Install

Linux x86-64 with AVX2 (glibc 2.34+), an NVIDIA GPU of compute capability 8.6+ and driver R580 or newer (CUDA 13).
On Windows, run inside WSL2. Docker and Homebrew as in the README.

curl -fsSL https://raw.githubusercontent.com/midagedev/bloomery/main/tools/release/install.sh | sh
bloomery-serve -m PATH/DeepSeek-V4.1-Flash-1017-Q3_K_M-00001-of-00010.gguf --port 8080

Use it with dsh

The setup the release clip ran (dsh @deepseek-ai/dsh 0.2.0-rc.2, one RTX 3090).

  1. Start the server with a model id for dsh to name:

    bloomery-serve -m PATH/DeepSeek-V4.1-Flash-1017-Q3_K_M-00001-of-00010.gguf \
  --host 127.0.0.1 --port 8080 --alias deepseek-v4.1-flash-1017

    The server picks the context from the card's room (262,144 tokens on a 3090 for this model) and prints it in its
    ctx … ctx=N log line; --ctx-size N sets it. --chat-template-file FILE is optional: the model file's own template
    already gives tool calls and thinking.

  2. Install dsh (npm install @deepseek-ai/dsh) and add a provider to the profile you run. Write this into
    $DSH_HOME/profiles/<profile>/cordis.patch.yml (web for dsh web, headless for dsh --profile headless):

    - id: llm-pi-ai
  config:
    providers:
      bloomery:
        apiKeyEnv: BLOOMERY_DSH_KEY
        api: openai-completions
        baseURL: http://127.0.0.1:8080/v1
        reasoning: high
        models:
          - id: deepseek-v4.1-flash-1017
            contextWindow: 262144     # the server's ctx=N
            maxTokens: 65536
            reasoningEfforts:
              off: none
              high: high
              max: max
- id: agent-default-model
  config:
    provider: bloomery
    model: deepseek-v4.1-flash-1017

  3. Set the key variable and start dsh: export BLOOMERY_DSH_KEY=none (any value: the server checks no key unless it
    was started with --api-key, and then the value must be that key), then dsh web.

No compat block and no proxy are needed. The server parses V4.1's tool-call spelling itself and streams the arguments
as they are written. reasoning picks dsh's default thinking level; off sends reasoning_effort: none, which turns
thinking off. The first ~400 decode steps after a load are slower (Known limits).

New

  • DeepSeek V4.1 Flash 1017 with dsh: V4.1 DSML tool calls are parsed in the spelling the model writes, and
    thinking follows reasoning_effort (#471). Tool-call arguments stream as the model writes them, on every seat that
    parses tool calls; a call whose sent text the parser would have to revise is a named error, never a silent rewrite
    (#477).
  • --chat-template-file works on every seat: DeepSeek V4.1, GLM, Qwen3.8, Qwen3, MiMo and Nemotron (#477).
  • reasoning_effort outside none, minimal, low, medium, high, xhigh, max is a named 400 (#477).
  • Structured output: json_schema response formats and forced tool calls are held by a program over the
    vocabulary (#467).
  • Qwen3.8-27B dense generation, and Qwen XML tool calls with the think span on every Qwen3.5-family variant (#472,
    #475). Ornith-1.5-35B-A3B runs on the Qwen3 seat (#425).
  • Quant types: a generic host dot serves a type that has a host activation and no hand kernel (#450); a card quant
    type's expert-select entries are one macro line (#460).

Changed

  • Lightning's ungated relu² experts run on the card (#437). The host tier's card leg runs its host list while the
    step's arena fills are in flight (#424).
  • Multi-node (by @juliankang4): a tier node may sit on a back-to-back link in 198.18.0.0/15 (#440), and a tier's done
    over a mirrored link posts only the columns its chain uses (#443, closes #421).

Fixed

  • Named faults instead of silent values in rope_neox, the DS41 kernels, the fused rope tail and the NVMe split (#435).
  • Reference writers, ptx-scan and the nightly runner stop on a failed read or a TERM instead of reporting success
    (#444, #446, #451, by @juliankang4).
  • Developer tooling and gates: see the merged pull requests since v0.3.4.

Known limits

  • V4.1 on two cards (--place bp) is not faster than one card (a). In 0.3.5 two cards decode at about the speed of one. Unset --place picks a.
  • V4.1's first ~400 decode steps after a load run at 25–40 tok/s while adaptive residency lands experts on the card
    (24 experts every 4 steps); after that, the dsh run on bp decoded at 45–49 tok/s.
  • From 0.3.4: Bolt needs a 24 GB card and refuses --place bp. From 0.3.3: Qwen3.6 has no DSpark draft; V4.1 DSpark
    serves one slot and refuses images; Nemotron 3 Nano 4B serves one slot on one card with no prompt cache.

Checked before the tag

Release head 1ef252d13 (f9b380607 and the README status). The release batch ran 199 gates on the fixture tier (3 red, none in code this release changed:
gate-gpu-nemotron-serve was red on main before the cut, from card room; gate-qdot's IQ4_XS clause compares with an
ik reference dump the box rewrote on the day of the cut, and the IQ4_XS code is unchanged since v0.3.4;
gate-gpu-qwen38-nodes-remote needs a second tier node this machine does not have), then the 75 gates that
run on the real model files (all green).
Every needs-gpu pull request since v0.3.4 is covered by those gates.

The tarball's bloomery-serve with DeepSeek V4.1 Flash 1017 Q3_K_M at the serving defaults (every lever unset,
--ctx-size 32768), one load per placement, on an RTX A6000 + RTX 3090 (250 W) machine, the box quiet (no other
process on either card). These are each load's first requests, not a timing lease; they run while adaptive
residency is still landing experts on the card (below):

Request unset (picks a): first token, decode bp: first token, decode
Short chat (20 tokens) 0.49 s, 25.5 tok/s 0.57 s, 24.9 tok/s
Long prompt (4,108 tokens) 11.20 s (367 tok/s prefill), 31.3 tok/s 10.47 s (393 tok/s prefill), 35.6 tok/s
One tool call 2.07 s, 32.4 tok/s, tool_calls, arguments parse 1.94 s, 35.9 tok/s, tool_calls, arguments parse
reasoning_effort: high 0.28 s, 29.1 tok/s, thinking in reasoning_content 0.25 s, 34.0 tok/s, thinking in reasoning_content

The dsh run in the clip (--place bp, 46 requests, 50 tool calls, 96,829 tokens in 45 minutes): all 50 tool calls
parsed, the arguments of 49 of 50 streamed as written, and its median decode was 45.4 tok/s.

Thanks

  • @juliankang4 for the multi-node link and column fixes, and the reference-writer, ptx-scan and nightly-runner fixes.

Contributors

juliankang4
Assets 4
Loading