v0.5.4 — context cache for agents at --tp 2: RAM tier, growing KV reservations

A context cache built for agents on two GPUs: a RAM tier for evicted conversations, and KV reservations that grow with the answer.

• RAM tier with two GPUs (issue #3): --host-kv-mib and --host-state-slots now work with --tp 2, like llama.cpp's --cache-ram. Each pinned record holds both GPUs' halves of a checkpoint, and each GPU copies its own half on its own copy stream. An evicted conversation comes back from RAM instead of being prefilled again; restored turns match an all-VRAM run token for token (MTP with int8 KV, DFlash2 with nvfp4 KV).
  • --profile 27b turns it on with 8 state slots and 8 GiB. Without the profile it stays off at --tp 2 unless you set the flags.
  • Qwen3.8-Flash-Next keeps it off: its experts already pin most of the RAM, and 8 GiB of cache cost ~8 % of prefill speed on long prompts (it shrinks the page cache the PLE table is read through).
• KV reservations that grow: a request reserves its prompt plus a 4,096-token window of KV and grows it as the answer grows, instead of reserving its whole max_tokens up front. An agent asking for 64k tokens and answering in 300 no longer pushes your other conversations out of VRAM.
  • An answer that outgrows its window with the GPUs full first moves idle cached conversations to RAM. It drops them only when there is no room for them, and it ends early with finish_reason: length only if no page can be freed.
  • Adapted from Wallawalla47/ninfer-custom 9f108db (Apache-2.0, see NOTICE).
• Measured on Qwen3.8-27B with an agentic replay (3 interleaved sessions, 11 subagents) and production flags (4 lanes):
  • median time to first token 2.14 → 0.98 s, p90 22.2 → 13.3 s;
  • prompt tokens served from cache 67 → 78 %;
  • prompt tokens recomputed −35 %;
  • decode unchanged: two concurrent decodes run at 220 tok/s each, with and without the RAM tier.
• Guardrails:
  • --kv-lease-window 0 restores the full reservation of v0.5.3.
  • The server refuses to start when the RAM tier would leave less than max(4 GiB, 5 % of RAM) available, and says how many MiB fit (NINFER_HOST_TIER_RAM_GUARD=0 skips the check).
  • The request log reports leased pages, cache owners moved or released for a lease, and answers cut short.
• Known limit: copying a conversation between RAM and the GPUs shares PCIe with the two GPUs' exchange, so other requests decode more slowly while a copy is in flight.

Unchanged computation: no quality gate needed (Flash-Next numbers as in v0.5.3).

tar -xzf capy-flash-v0.5.4-linux-x86_64-sm120a-ubuntu26.04.tar.gz && cd capy-flash-v0.5.4-*/
sha256sum -c SHA256SUMS
./serve-tp2.sh /path/to/qwen3_8_27b_quasar_nvfp4.ninfer          # Qwen3.8-27B, RAM tier on
./serve-flash-next.sh /path/to/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer

v0.5.3 — faster first prefill chunk and start, tensor-core router, installer

Faster first prefill chunk and server start, the router on tensor cores, and an installer.

• Prefill: the first chunk of new text waited for its n-gram (PLE) table rows, two random disk reads per row. The rows' multipliers (640 MB) now stay in RAM and each page is read once: a ~15k-token first chunk waits 0.3-0.5 s instead of 0.8-1.3 s. The router of prompt chunks runs on tensor cores. Same prompts as the README table, server side: ~3.5k / ~15k / ~55k tokens 1,985 / 3,778 / 3,977 tok/s (v0.5.2: 1,856 / 3,302 / 3,715; Strata 0.1.44.1 IQ3_S 1,745 / 2,906 / 4,037).
• Quality, same gate as before (the tensor-core router sums in another order, so tokens can differ near ties): GSM8K 1319 0.9765 (GB300 0.9810, not worse, McNemar p 0.070), MMLU-Pro 1008 0.8452 (GB300 0.8452). NINFER_QWEN4_ROUTER_MMA=0 restores the previous router.
• Server start: 26-29 s cold (was 56-141 s). The artifact directory is parsed in 0.5 s instead of 26 s, the 64.6 GiB of experts are read with direct I/O, and the load progress counts them (it used to show an ETA of an hour).
• Easier to run: ninfer-serve --profile flash-next|27b applies the tested configuration; install.sh checks the machine (GPUs, driver, AVX-512, RAM, disk), downloads and verifies the release and the model and prints a test request; the launchers serve the model as capy-flash / qwen3.8-27b; startup errors say what to do (port in use, a missing part of the artifact, GPU memory held by another process); docker/Dockerfile.release; examples/serving (systemd unit, llama-swap entry).
• Italian drafts: the MTP draft shortlist written by tools/capy_flash/convert.py gains the 8,192 most frequent Italian tokens it lacked (Italian answers: draft acceptance 0.37 -> 0.45, ~+11 % decode; English and code unchanged; drafts are verified, so the output does not change). The Hugging Face artifact still ships the previous list; to use the new one, run python tools/capy_flash/convert.py --model <NVIDIA checkpoint> --out <artifact dir> --skip-convert next to it.
• Process: pre-commit hooks and a lint workflow on the changed lines; AGENTS.md / CONTRIBUTING.md rewritten for capy-flash.

Verified: ctest 187/187 with the synthetic Flash-Next artifact for the Engine tests (before the router change), then on the release build the router tests on both paths, the artifact, serve and Qwen4 tp2/MTP Engine tests; the quality gate above; tarball extracted, SHA256SUMS checked, Qwen3.8-Flash-Next served with serve-flash-next.sh (now on --profile) and answered a test prompt correctly.

Tarball: same layout as v0.5.2 (serve-flash-next.sh, serve-tp2.sh, stripped binaries, CUDA runtime, SHA256SUMS; Ubuntu 26.04, sm_120a). Artifact unchanged: Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash. Details of every change: development log, T170-T186.

v0.5.2 — faster prefill of long prompts of new text

Faster prefill of long prompts of new text. No change to the model output: token identity verified against v0.5.1 with the expert residency frozen.

• The next chunk's n-gram (PLE) rows are read while the GPUs run the current chunk. Before every prefill chunk the host gathers the chunk's rows from the 47.7 GiB PLE table, which stays on disk (mapped), with both GPUs idle; rows that are not in the page cache cost 0.4-6 s per 16k-token chunk. A helper thread now starts those reads one chunk ahead, so they overlap the GPU work; the gather itself is unchanged.
• Measured (2x RTX 5070 Ti, server side): the ~55k-token prompt of the README table goes from 2,791 to 3,715 tok/s (Strata 0.1.44.1: 4,037 IQ3_S, 3,821 NVFP4); cold random ~80k-token prompts from 32 s to 21.4 s median (3 runs per side, same binary, prefetch off/on). PLE wait per chunk after the first: 0.4-5.9 s -> ~0.11 s.
• Not covered yet: the first chunk of a request (and so short agentic turns that fit in one chunk) still waits for its rows (~0.4 s at 3k, ~1.2 s at 15k tokens, cold).
• Diagnostics: NINFER_QWEN4_PLE_TIMING=1 prints every prefill gather/prefetch duration; NINFER_QWEN4_PLE_NEXT_PREFETCH=0 turns the prefetch off (diagnostics).

Verified: token identity vs v0.5.1 (2k and 16k prompts, 1,024-token chunks, frozen residency): identical; ctest 186/187, the remaining one (ninfer_elastic_arena_test) fails only under ctest -j 4 and passes 3/3 alone on this build and on v0.5.1; tarball extracted, SHA256SUMS checked, Qwen3.8-Flash-Next served with serve-flash-next.sh and answered a test prompt correctly.

Tarball: same layout as v0.5.1 (serve-flash-next.sh, serve-tp2.sh, stripped binaries, CUDA runtime, SHA256SUMS; Ubuntu 26.04, sm_120a). Artifact unchanged: Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash.

v0.5.1 — prebuilt tarball for Qwen3.8-Flash-Next, artifact on Hugging Face, logo

Packaging release on top of v0.5.0 (ninfer-tp2 renamed capy-flash). No engine change: the binaries behave exactly like v0.5.0.

• Prebuilt tarball for both models: capy-flash-v0.5.1-linux-x86_64-sm120a-ubuntu26.04.tar.gz (stripped ninfer, ninfer-serve, ninfer-perplexity, CUDA 13 runtime, SHA256SUMS). New launcher serve-flash-next.sh with the Qwen3.8-Flash-Next profile (INT8 KV with the elastic KV / expert-slot budget, 262,144-token context, MTP K = 3, one request at a time), next to serve-tp2.sh for Qwen3.8-27B (unchanged profile). Qwen3.8-Flash-Next also needs an AVX-512 VNNI CPU and about 70 GiB of free RAM (checked at startup).
• Ready-to-serve Flash-Next artifact on Hugging Face: Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash (126.6 GB, NVIDIA's NVFP4 experts exactly as published, MTP layer and draft shortlist included, SHA256SUMS). It is the artifact behind the README's quality and speed numbers; converting nvidia/Qwen3.8-Flash-Next-NVFP4 yourself with tools/capy_flash/convert.py gives the same result.
• Logo at the top of the README (docs/logo).
• Tests: guarded device buffers wait for their guard fill, so a parallel ctest can no longer see guard bytes in an Op's output (test-only).

hf download Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash --local-dir capy-fn
tar -xzf capy-flash-v0.5.1-*.tar.gz && cd capy-flash-v0.5.1-*/ && sha256sum -c SHA256SUMS
./serve-flash-next.sh ../capy-fn/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer

Verified: full test suite 187/187; the tarball extracted, SHA256SUMS checked, Qwen3.8-Flash-Next served with serve-flash-next.sh and answered a test prompt correctly; the Hugging Face files match the local SHA256SUMS (6/6).

v0.5.0 — ninfer-tp2 becomes capy-flash: Qwen3.8-Flash-Next on two 16 GB GPUs

ninfer-tp2 is now capy-flash. Same repository, same issues and history; the old URL redirects here. From this release the engine serves Qwen3.8-Flash-Next (MoE 125B-A6B, NVFP4 experts) on two 16 GB consumer GPUs, and keeps serving Qwen3.8-27B exactly as v0.4.9 did. The work since v0.4.9 is one commit (79d969d4 → v0.5.0); the experiments behind it are explained in docs/capy-flash/development-log.md.

Qwen3.8-Flash-Next (new)

• Routed-expert cache in VRAM, partitioned by expert id between the two GPUs; the experts that do not fit live in host RAM and are computed on the CPU (AVX-512 NVFP4 kernel with the GPU's W4A4 activation codec, bit-identical to the GPU path), inside the captured CUDA graph.
• MTP speculative decoding (K = 3, draft vocabulary shortlist), INT8 KV with an elastic KV / expert-slot budget up to 262,144 tokens, prefix cache across turns.
• One-command conversion from nvidia/Qwen3.8-Flash-Next-NVFP4 (tools/capy_flash/convert.py); startup check that the 64.6 GiB of host-resident experts fit in free RAM.
• Quality against NVIDIA's NVFP4 checkpoint served by vLLM on a GB300: GSM8K 0.9780 vs 0.9810 (McNemar p 0.29), MMLU-Pro 0.8482 vs 0.8452 (non-inferior, margin 2.5 pt).
• Speed, agentic replay (53 turns, prompts to ~81k tokens, bench/capy_flash) on 2x RTX 5070 Ti + Ryzen 9 9900X + 96 GB DDR5: TTFT 1.82 s, decode 100.8 tok/s, 243 s total; Strata 0.1.44.1 on the same NVFP4 weights 2.42 s, 78.3 tok/s, 314 s.
• Requirements: two sm_120 16 GB GPUs, AVX-512 VNNI CPU, about 70 GiB of free RAM. See the README.

Qwen3.8-27B (unchanged behaviour)

• tp2 gate against v0.4.9: golden 3/3 identical, perplexity identical at 4k and 64k, MTP3 greedy 60/60 texts identical (16.87 vs 16.70 ms/round), DFlash2 10/10 identical.
• Agentic replay with the v0.4.9 production flags: TTFT 1.42 s, 171.2 tok/s, 156 s (v0.4.9: 1.40 s, 172.2 tok/s, 154 s), same token counts on all 53 turns.
• --turn-anchors now defaults by model: 1 on the 27B, as before (a cancelled turn resumes from the previous turn, #3), 0 on Flash-Next (there each anchor costs one extra ~0.8 s prefill pass per turn). --turn-anchors N overrides.

Verified: full test suite 187/187 (ctest -j 4), tp2 gate on the 27B as above, Flash-Next quality and replay as above.

No prebuilt tarball in this release: build from source (README). For the 27B, the v0.4.9 tarball behaves identically.

v0.4.9

Seventeenth packaged release. Changes since v0.4.8 (382a8a60 → v0.4.9):

• A short side conversation no longer evicts a long session's cache at --tp 2 (#3, reported by @fantasyz). At tp 2 there is no Host tier, so every checkpoint lives in the Device StateImages (--device-state-slots). When a new conversation wanted a checkpoint of its own (turn-closure rewrite or long anchor) and the pool was full, the cache released idle conversations oldest first, whole. In the reported pattern the only idle conversation was a long agent session, so a 1.8k-token helper call released all of it, endpoint and anchors, and the session's next turn re-prefilled from a shared prefix far behind. The reclaim now:

  • never takes a checkpoint that costs more to rebuild than the one being stored; if that is the only way, the small checkpoint is skipped (its conversation re-prefills a few thousand tokens on its next turn);
  • drops long anchors, lowest first, before it releases whole conversations;
  • releases the conversation cheapest to rebuild first, with GreedyDual aging so a long conversation nobody returns to still ages out.

  Measured at tp 2 with 4 Device state slots (27B, an agent session growing from 20k to 47k tokens, then a two-turn 1.8k side conversation and a 342-token one): the session's next turn went from the shared prefix at 26,057 tokens, TTFT 6.9–7.4 s, to its own endpoint at 44,057 tokens, TTFT 1.1 s. The side conversations behave as before.

Verified: ninfer_resource_manager_test with three new cases; the full test suite (139 passed, 0 failed); the tp2 real tests 6/6 (engine, MTP, DFlash2, vision, text context); the end-to-end repro above; behavior gate on the release binary: PASS (shards, golden 3/3, perplexity identical to every digit at 64k and 4k, MTP3 greedy 60/60 texts identical at 16.704 ms/round vs 16.693, DFlash2 10/10 at 18.204 vs 18.204).

Not changed: kernels, defaults, the artifact format, and tp 1 (the reclaim only runs without a Host tier, i.e. at tp 2).

Tarball: same layout as v0.4.8 (stripped ninfer, ninfer-serve, ninfer-perplexity, CUDA 13.2 runtime, serve-tp2.sh, SHA256SUMS; Ubuntu 26.04, sm_120a).

v0.4.8

Sixteenth packaged release. Changes since v0.4.7 (d8013fab → v0.4.8):

• Artifacts with a q8 output head start at --tp 2 (#6). At tp 2 the vocabulary head is split by rows, and the q8 linear had no entry for the [124160,5120] half (fp8 had one), so a conversion that stores text/output_head as q8_g32_fp16 — e.g. kaushikvira/Qwen3.8-27B-swift15-nvfp4full-dflash2-NInfer-v3 — stopped at startup with q8 linear: unsupported shape. The half reuses the full head's kernel schedules (not re-measured at this shape). Checked with that artifact on 2× RTX 5070 Ti: v0.4.6 fails at startup under MTP and DFlash2; v0.4.8 starts under both and answers coherently (greedy, MTP3 acceptance 88.0 %, DFlash2 K=7 84.7 %). The q8 conformance test now covers the shape for every token count it already checked.

• --prefill-miss-dump DIR (off by default; proposed by @fantasyz in #3): a diagnostic for prefix-reuse misses. The server keeps the token contexts of its last 16 requests (prompt plus committed output; cancelled requests included). When a request recomputes at least --prefill-miss-min-tokens prompt tokens (default 10,000), shares at least --prefill-miss-min-match of its prompt (default 0.5) with the closest of them, and either the server recomputed that many tokens identical to it or that many of its tokens follow the divergence point, it writes a new directory under DIR:

  • new_prompt.txt and old_context.txt (chat template and special tokens included) and both token-ID lists;
  • summary.json: token counts, the divergence point with the text on both sides, and the reading — client: … when the request changed or dropped earlier text, server: … when tokens identical to the earlier context were recomputed (both can hold, since checkpoints are discrete).

  The request also logs a prefill miss warning with the numbers and the path. A request that only appends text, however long, is not dumped; at most 64 dumps per process. The files contain the conversation verbatim. End to end at tp 2 (34k-token conversation, thinking on): a normal follow-up and an edited last message are not dumped (34,044 and 34,042 tokens reused); a history rewrite at 70 % of the long message is dumped with both readings.

• The public Engine gains detokenize_text().

Verified: ninfer_linear_q8_a16_test, ninfer_linear_split_test, ninfer_prefill_miss_dump_test (6 cases) and ninfer_request_log_test; the end-to-end runs above; behavior gate on the release binary: PASS (shards, golden 3/3, perplexity identical to every digit at 64k and 4k, MTP3 greedy 60/60 texts identical at 16.697 ms/round vs 16.693, DFlash2 10/10 at 18.193 vs 18.204), and the greedy stage again with --prefill-miss-dump on: 60/60 identical, 16.702 ms/round, no dump written.

Not changed: the kernels used by the published QUASAR artifacts, defaults (the new option is off), the artifact format.

Tarball: same layout as v0.4.6 (stripped ninfer, ninfer-serve, ninfer-perplexity, CUDA 13.2 runtime, serve-tp2.sh, SHA256SUMS; Ubuntu 26.04, sm_120a).

v0.4.7 — MTP layer in NVFP4 as the production artifact, paired perplexity test in the gate

Fifteenth release, without new binaries: the C++ tree is the same as v0.4.6, so the v0.4.6 tarball runs it. Changes since v0.4.6 (47d47864 → v0.4.7):

• MTP layer in NVFP4, as an opt-in conversion recipe. _mtp_nvfp4_layer in tools/convert/official_recipes.py (with tests) stores the MTP layer's MLP gate/up, MLP down and attention output as NVFP4 (nvfp4_mse from the source's BF16 MTP weights, 16-bit activations); at tp 2 their halves run the NVFP4 linear routes the target already uses. The QUASAR model card recipe gains configure_mtp_nvfp4 (--recipe quasar_recipe.py:configure_mtp_nvfp4); it reproduces the measured artifact object for object (1,524/1,524). Against the published artifact on 2× RTX 5070 Ti (eco clocks, MTP3, --lm-head-draft, 262,144 tokens, T=0): the MTP round is 1.9 % shorter at C=1 and 1.4 % at C=4; outputs, target perplexity and GSM8K (500 problems, thinking on: 483/500 on both, the same answer on every problem, McNemar p = 1.000) are identical; decode is 1.5 % faster on average at C=1 (2.2–2.9 % on code, math and English prose, 0.6 % on Italian, unchanged on agent prompts of 55–171K tokens, where acceptance drops by 1.7 points). Weights −142.5 MiB, +68 MiB of free VRAM on rank 0. The production profile of this fork now runs this variant (qwen3_8_27b_quasar_nvfp4_mtpnv.ninfer, SHA-256 3363b1ab64c5693c8dcb0cb283b01bcb1a19de946846799e53bca7ac8f2533ac); the artifact on the Hub is unchanged.
• Gate: a paired perplexity test for commits that change bits on purpose (GATE_PPL_PAIRED=1, tools/tp2/ppl_paired.py, standard library only). An identical table still passes as before; a different one is compared window by window with the reference report.json (mean ΔNLL per token, standard error across windows, paired t, 95 % interval of the perplexity change, exact sign test): PASS if not worse at p > 0.05 and |Δ| < 0.5 %, WARN otherwise, FAIL if the windows differ. --record also stores ppl-*.report.json. Documented in tools/tp2/README.md with the GSM8K/McNemar rule for bit changes outside the gate.
• Gate reference recorded again on the MTP-NVFP4 artifact (GATE_ARTIFACT default …_mtpnv): golden 3/3, perplexity tables identical to every digit (the per-window reports differ only in timings and the artifact path: perplexity measures the target, which this artifact does not change), MTP3 greedy 60/60 texts identical at 16.693 ms/round (was 17.000) with 73.70 % acceptance (73.62 %), DFlash2 unchanged (10/10, 18.204 ms/round).
• Two other draft-head reductions were measured and not adopted: a 65,536-row MTP proposal head (acceptance −3.5 points on average, −8.9 on long agent prompts, and its tp 2 half has no linear route) and the DFlash2 drafter's down projection in NVFP4 (+0.6 % decode, below the 1 % bar; it also does not run with the whole drafter on rank 0).

Verified: conversion tests; gate of the candidate artifact on the v0.4.6 runtime (shards PASS, golden 3/3, perplexity identical to every digit, MTP3 greedy 60/60 texts identical, 16.694 vs 17.000 ms/round); paired-perplexity check end to end on a build that changes bits; production smokes after the switch (text, thinking, image).

Not changed: the runtime (C++/CUDA), defaults, the artifact format, the published artifacts.

v0.4.6 — request log labels and telemetry, speculation detail option, residual rounding independent of tile occupancy

Fourteenth packaged release. Changes since v0.4.5 (ff6b336f → v0.4.6):

• Request log (--request-log-jsonl, schema 21 → 27):
  • request.client carries the optional X-Ninfer-Client header (1–64 characters of A-Z a-z 0-9 . _ : / @ + -; a malformed or repeated header is a 400 invalid_client_label), so benchmarks and smokes can be told apart from real traffic. The fork's own tools (gate client, corpus, concurrency, TTFT, agentic A/B, smokes) now send <kind>/<tool> labels.
  • speculative.accepted_length_histogram: how many rounds accepted exactly i draft tokens.
  • model_thinking_tokens is counted for every response that starts in thinking; it was only counted when a thinking budget was set, so without one it read 0.
  • throughput.gpus: per board, SM and memory clocks, power, energy over the interval, temperature and the clock-throttle reasons, read through NVML (loaded at runtime with dlopen, matched by UUID) on the statistics thread: 4.2 ms per sample of two boards every 5 s, nothing on the request path.
  • --log-speculation-detail (off by default; --log-speculation-detail-max-steps N, default 4096): adds speculative.accepted_lengths, one character per generation step in order (a hex digit = draft tokens accepted in that round, - = a step without a draft; accepted_lengths_truncated past the cap), and result.prompt_ngram_overlap, the share of the output's 4-token windows that also occur in the prompt (thinking and tool calls included; 0.016 ms at 8k prompt tokens, about 1 ms at 262k, on the finishing request only). Off, the decode round is unchanged (16.996 vs 16.998 ms); the public API gains PreparedPrompt::token_ids().
• Fused residual rounding no longer depends on tile occupancy (NVFP4 A4 and FP8 A8 linear_add, the prefill routes). The FullTokens kernel instance, chosen only when the launch width is an exact multiple of the tile, let nvcc contract the scaled accumulator and the residual into one FMA while every other launch (tail tiles, the 1024-token TMA chunks) rounded twice, so a column's result depended on the batch width. Reported and fixed by MirkoCovizzi (87f0689805, 37a1ed49d4); ported here keeping, per format, the rounding the long chunks already used — NVFP4 explicit MUL then ADD, FP8 FMA — so the fix changes nothing that the gate reference measured: perplexity identical to every digit at tp 2 and tp 1, golden 3/3, MTP3 greedy 59/60 texts (one prompt whose prefill segment was tile-aligned), DFlash2 10/10, GSM8K 1,319 problems 96.59 % vs 96.44 % (McNemar p = 0.50), speed unchanged. A new CTest (ninfer_linear_add_residual_rounding_test) checks a residual column keeps its bits at every batch width (9 NVFP4 and 14–21 FP8 widths differed before). Mirko's literal port (FMA everywhere) was measured and not adopted: +0.10 % perplexity at 64k, 14/60 texts.
• Gate reference re-recorded on this tree (greedy 17.000 ms/round; the DFlash2 stage now defaults to the NVFP4-drafter artifact _df2nv: 18.200 ms/round, 62.66 % acceptance); ninfer_resource_manager_test compiles again and covers the cancelled-request path (it had been excluded from the ctest stage since v0.4.4).
• Documentation: the README summarizes the 2026-10-05 changes; docs/serving.md, docs/maintainer/logging.md and docs/weight-conversion.md cover the new options, the log schema and the NVFP4 drafter gate/up conversion.

Verified: ctest 148/148 on the rounding tree; request-log, transport, options, public-API and frontend tests; the two-device MTP engine test for the recorded accepted lengths; behavior gate on main with and without --log-speculation-detail (golden 3/3, perplexity identical to every digit, MTP3 greedy 60/60 texts identical at 16.996–16.999 ms/round, DFlash2 10/10); HTTP smokes on the chat, Anthropic and Responses APIs with the detail on. The release binary's gate is recorded in the production dossier.

Not changed: the decode (A16) kernels, defaults (every new option is off), the artifact format.

Tarball: same layout as v0.4.5 (stripped ninfer, ninfer-serve, ninfer-perplexity, CUDA 13.2 runtime, serve-tp2.sh, SHA256SUMS; Ubuntu 26.04, sm_120a).

v0.4.5 — token embedding in host memory, Anthropic thinking.display, NVFP4 DFlash2 drafter MLP

Thirteenth packaged release. Changes since v0.4.4 (ae667d61 → v0.4.5):

• --embedding-host: the token_embedding table (FP8 rows, 1.18 GiB) is kept in one mapped page-locked host buffer and read by both boards over PCIe instead of being replicated in each board's VRAM. The gather kernels are unchanged, so results are identical bit for bit (eight full gates, with and without the option, all identical to the reference). Frees 1.18 GiB per board: with the production flags the free VRAM goes from 920 MiB to 2.08 GiB on qwen27b (MTP3, 262,144) and from 208 MiB to 1.38 GiB on qwen27b-code (DFlash2, 229,376). Cost: prefill −0.9 % at 8k and −0.5 % at 115k tokens (every board reads 5 KiB per prompt token over PCIe); decode rounds unchanged (MTP3 −0.02 %, DFlash2 +0.13 %, 3 runs each, 2.1 GHz cap). What it buys: the DFlash2 profile reaches the full 262,144 context (a 244,725-token request served), MTP3 at four concurrent requests takes --kv-capacity 327680 int8 (ceiling ≈360k), and at --tp 1 the 27B artifact fits one 16 GB board for text (≈22–26k context, estimated). Default off. Not exercised: vision prompts, WSL2.
• Anthropic thinking.display: thinking: {type: adaptive, display: omitted} is accepted (Claude Code 2.1.289 sends it on every request together with context_management; v0.4.4 answered 400 and the client stopped at its first turn). Ported from Wallawalla47/ninfer-custom ca1f50ed. Verified with the schema tests, an HTTP smoke (11/11) and a real two-turn Claude Code session with a tool call.
• DFlash2 drafter MLP gate/up as NVFP4 (ported from ninfer-custom 1f2f5e97, on the rank-split drafter). Needs a reconverted DFlash2 artifact (5 of 1,590 objects change, −425 MiB; 4 minutes of CPU with the conversion tools); with the existing artifact nothing changes. Measured at C=1 against v0.4.4: decode +2.1 % (code), +2.4 % (math), +1.6 % (prose), +1.5 % (agent 55k), −0.5 % (agent 128k); acceptance within ±1.2 points; round −1.3…−1.7 %; free VRAM on rank 0 at 229,376 from 208 to 420 MiB. At C=4 the prose round is +0.6 % (a 17–32-token route the fork never tuned for 70-SM boards).
• Artifact loader: the direct-I/O staging slots are aligned to the payload alignment themselves (a small page-locked allocation is not always 4,096-byte aligned; the 64 MiB production slots were).

Verified: behavior gate PASS on the ported branch and on --embedding-host on/off (shards, golden 3/3, perplexity 64k/4k identical to every digit, MTP3 greedy 60/60 texts identical at 16.99 ms/round, DFlash2 10/10 identical on the existing artifact); the release binary's gate is recorded in the production dossier.

Not changed: kernels of the dense model, defaults (both --embedding-host and the NVFP4 drafter are opt-in: the latter by artifact), the gate reference.

Tarball: same layout as v0.4.4 (stripped ninfer, ninfer-serve, ninfer-perplexity, CUDA 13.2 runtime, serve-tp2.sh, SHA256SUMS; Ubuntu 26.04, sm_120a).