Skip to content

History / Benchmarks

Revisions

  • Tracker round 10: TT-007 closed — steady-state decode after ANE prefill costs nothing Every ANE decode figure here was a ~60-token window, because the pinned qualification prompt hits end-of-turn and `--max-new` never bound; the window contains the one-time handover cost, so those figures were upper bounds and the steady-state cost had never been measured. The prompt fix — the qualification body plus "continue, do not stop early" — makes the cap bind. `benchmark/ane_steady_state_decode.py` then differences two generation lengths so every one-time cost cancels. On AgentWorld 35B-A3B 4-bit, an 11,948-token prompt, two runs per arm and length: ANE prefill is 3.32x faster and costs **+0.7%** of steady-state decode (7.26 against 7.21 tok/s) — nothing, inside the spread — with no penalty even in the 64-token windows. Engineering Notes records the method, the numbers, the caveats (medians of 2; the GPU arm's decode follows a 10-minute GPU prefill against ANE's 3-minute one; Ornith is not installed, so AgentWorld-4 stands in; its row wires the cache, so the pageable case is TT-005's question via the new `TINYTITAN_KEEP_WIRED=0`), and a new trap: the first ANE prefill of a process pays a 2.1x compile warm-up, so a single cold-process figure overstates the sidecar. Benchmarks carries the table.

    André Borchert committed Sep 18, 2026
  • Wiki audit: correct 12 pages against the code, and retire a command that cannot run Every page was read against the 5.7 tree. Three systemic causes account for most of what the audit found. **Docs described tuning that is no longer shipped.** FAQ and Runtime-Controls quoted the family fallback and pre-repair arms, while the defaults are the per-install `ModelProfile.table`: the 35B rows are 10 GiB/160 slots at 4-bit and 12 GiB/96 at 8-bit with prefetch on, and Qwen3.8 is 12 GiB/96 at 4-bit and 9.5 GiB/40 at 8-bit with prefetch off. Also corrected: Home's Swift floor (6.3 -> 6.4) and its 35B disk range; Getting-Started's dense sizes, and its claim that a pipe sets up the browser window (that prompt needs a terminal); Features' missing 8-bit Qwen3.8, its three clients where there are five, and "binary thinking" where Qwen3.8 has four levels. **Version labels had drifted.** Benchmarks' "v4.7" section is 5.0's work — there is no v4.7 tag — and it claimed a v4.6 baseline for Qwen3.8, which did not exist then. Its three percentages were withdrawn rather than re-guessed: their baselines cannot be re-derived from anything in the repository. **The LAN Manager's remote-access recipe was impossible.** The harness refuses `--host 0.0.0.0`, and a specific address fails the webserver plugin's schema (`expected "127.0.0.1" | "0.0.0.0"`), so the profile does not load. Verified by trying both on a throwaway DSH_HOME. The page, the plugin README and the plugin's own boot log all told operators to do it; all three now say what is true, and the boot log no longer advertises addresses nothing is listening on. Also: Cookbook's launcher keys, OpenAI-Compatible-Server's `-fast` listing and its per-surface reasoning field, System-Design's ANE default, MTP numbers and interface list, TinyTitan-Plugin's `--web` owner and its session-title overclaim, the LAN fence's IPv6 ranges and its missing config keys, an unquoted receipt error, and a duplicated sentence. Verified after: links, anchors and the sidebar clean; every cited path still resolves; both plugin suites pass; all six lint gates green.

    André Borchert committed Sep 18, 2026
  • The wiki is TinyTitan Every page except the Changelog is renamed by the same rule the checkout used (`NVMAI_` -> `TINYTITAN_`, `NVMAI` -> `TinyTitan`, `nvmai` -> `tinytitan`), and the wiki's own `assets/wordmark.svg` is fixed the way the repository's was: the name was split across two coloured spans, so it read `NVM` + `AI` and had to become `Tiny` + `Titan`, with the canvas widened from 132 to 176 so the longer name is not clipped. The Changelog keeps its entries as they shipped -- they are the record of releases made under the old name -- and gains one line at the top saying the project was renamed on 2026-09-14, so the page is not a mystery. Its release links still resolve: GitHub redirects a renamed repository. Home now leads with the project's brand image (`assets/tinytitan-hero.png`), the same artwork the README leads with, and the tracker gains the closed row.

    @agent agent committed Sep 14, 2026
  • Add the "Capital of Paris" measurement page The report moves here from the repository, where it was a markdown file rather than a wiki page: one deliberately ambiguous prompt plus a plain control question, every served model and both engines, three repeats each, with every reply quoted verbatim. It records that one install thinks with the switch off, what the runtime now does with that (C90), and what the repeats settled about the earlier single-run numbers. Linked from the sidebar under Reference and from Benchmarks.

    André Borchert committed Sep 11, 2026
  • Head every page with the NVMAI wordmark The wiki carried no visual identity at all, so the README and the wiki did not look like the same project. Every content page and the sidebar now open with the same two-colour wordmark the README slogan uses -- NVM in cyan, AI in orange. One file serves both themes here. The slogan needs a light and a dark variant because its body text is neutral, but the wordmark is entirely brand colours and both carry enough contrast on #ffffff and #0d1117, which was checked against each before committing. Deliberately not applied to the hundreds of inline "NVMAI" mentions in prose: GitHub cannot colour inline text, so each one would have to become an image, which would break text selection, search and screen readers for no gain.

    André Borchert committed Aug 30, 2026
  • Add the v4.7 benchmark section Every shipped model re-measured at its shipped defaults on the same workload as v4.1 and v4.6, now that the expert-cache budget and prefetch are per family. Qwen3.8-Flash-Next 6.82 tok/s, Ornith 22.65 / 11.89, Qwen 3.6 23.28 / 12.72. Records two things the numbers do not show on their own. The gains exceed the interleaved A/B that motivated them, because interleaving keeps every configuration page-cache-warm and that flatters the baseline more than the tuned configuration; and the 4-bit 35B rows needed seven runs, because at three Qwen 3.6 4-bit returned two transient fast readings that would have published a 13% gain for a configuration that had not changed.

    André Borchert committed Aug 30, 2026
  • Document Qwen3.8-Flash-Next, per-family tuning, and the new model ids The wiki was last updated before three shipped changes, so it described a runtime that no longer exists. - Qwen3.8-Flash-Next 125B-A6B appeared only in the changelog. It is now in Home, Features and System Design, including the install size, which is about 161 GB and dominated by a 95 GB hashed n-gram table, and the point that it runs on 24 GiB because the experts are streamed. - NVMAI was described throughout as a "Qwen3.5-MoE 35B-A3B" runtime. There are three families now, and Qwen3.8-Flash-Next does not share the tensor contract the other two do. - Model ids gained the routed-expert width (`_4-Bit`, `_8-Bit`). Every example in the server guide and FAQ still showed the bare id, which the server no longer accepts. - ANE prefill was documented as experimental and opt-in; it is the default, and falls back to the GPU when a model has no sidecar. - The expert-cache budget was documented as a flat 8 GiB deriving 128 or 64 slots. It is per family now, with a clamp to half of physical memory, and speculative expert prefetch was not documented at all. The dated v4.1 and v4.6 benchmark sections are left as they are: they record what was measured at the time, and their 8 GiB budget note was accurate for those runs. Only the claim that ANE "stays off by default", which a reader would take as current, is corrected there. Current numbers follow in a separate commit once the re-measurement finishes.

    André Borchert committed Aug 30, 2026
  • Audit the wiki: current v4.6 state, remove engineering history records - Move the v4.2/v4.3 experiment qualification record out of the wiki (now docs/v4.2-experiments.md in the source repository) and drop its navigation entries; the wiki keeps user-facing documentation only. - Trim Benchmarks to current, decision-relevant results: v4.6, its v4.1 baseline, the MTP and coding/tooling qualifications behind the shipped defaults, and fast-alias latency. Remove superseded 6-bit-era tables, the stale prompt-length record, and the historical M5 reference. - Bring Features and System Design to v4.6 (tiled Top-K sampling, opt-in ANE prefill) and remove release-tag phrasing from runtime descriptions. - Remove the empty Unreleased changelog section, the legacy 6-bit row in Getting Started, and a duplicated server snippet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

    André Borchert committed Aug 27, 2026
  • Document v4.6: sampler rewrite, ANE prefill, benchmarks, runtime controls

    André Borchert committed Aug 22, 2026
  • Document NVMAI 4.1 release

    @agent agent committed Aug 21, 2026
  • Document NVMAI 4.0 defaults and qualification

    André Borchert committed Aug 21, 2026
  • Record concise qualification commands

    André Borchert committed Aug 20, 2026
  • Document Ornith concise tool qualification

    André Borchert committed Aug 20, 2026
  • Document Ornith native MTP support and benchmarks

    André Borchert committed Aug 20, 2026
  • Add Ornith prompt-length benchmarks

    André Borchert committed Aug 20, 2026
  • docs: refocus wiki on core user workflows

    André Borchert committed Aug 20, 2026
  • docs: update launchers and release history through 3.9

    André Borchert committed Aug 20, 2026
  • launchers: rename to server_launcher.sh + cli_launcher.sh (two launchers, same five questions)

    André Borchert committed Aug 11, 2026
  • docs: add 'physical limit' clarification at the top of FAQ, Home, Performance, and Benchmarks — decode is SSD-configuration-bound, not chip-bound

    André Borchert committed Aug 11, 2026
  • 3.3: marketing lead — headline speed-up, Paris table first

    André Borchert committed Aug 11, 2026
  • 3.3: state fast-alias time savings as x-factors (10x/37x/65x, 90-98%)

    André Borchert committed Aug 11, 2026
  • 3.3: <model>-fast alias strips coding-CLI boilerplate (docs + CLI matrix)

    André Borchert committed Aug 11, 2026
  • docs: fix dangling links after experiment-page removal

    André Borchert committed Aug 11, 2026
  • Refresh for Qwen-only NVMAI: fix binary names, defaults, and broken links - Getting Started and the server guide: rename the old fork's binary names (TurboFieldfareRepack/CLI/Server/Mac) to NVMAIRepack/NVMAICLI/NVMAIServer/ NVMAIMac and the defaults suite to NVMAI; pgrep patterns updated - Runtime Controls: expert-cache default 32 (64 accepted), prefill chunk default 4096, new 4-bit chunk-sizing measurement added alongside the older 8-bit data point - Qwen performance notes: profile updated to 32 slots; slot cap note updated - System Design: Qwen-only banner, broken Sources/TurboFieldfare links fixed to sources/NVMAI, trusted-install load default documented - Optimization Journey: 2026-08 Qwen acceleration measurements (chunk sizing, trusted-install startup, slot neutrality, rejected prefetch/mmap)

    André Borchert committed Aug 7, 2026
  • Migrate project documentation into structured Wiki

    André Borchert committed Aug 3, 2026