v0.38.0

Added

• A rented 16 GB card with 30 GiB of RAM, against llama.cpp and Strata
  (docs/bench-data/2026-10-27-rented-16gb/, docs/benchmarks.md, the README table). The class of
  a home PC with a mid-range card, where Qwen3.8-Flash-Next streams: --auto decodes 19.3-20.8 tok/s
  over 1024 tokens with #431, 27 from the 768th token, against llama.cpp's own offload at 16.6-18.9
  over 256 tokens and Strata at 36-67 (50-66 over 1024). The prefetchers, predictive prefetch and
  route-ahead all lose beside the device cache, measured; the head drafting (--mtp-head) does not
  pay yet. What separates the two engines there is written down with the numbers. The README's table
  also gains the 24 GB card's rows of 2026-10-26.

• An MTP head from a file of its own (--mtp-head FILE). Some quantised files leave the trained
  head out (the GSQ-RCO quants of Qwen3.8-Flash-Next), and llama.cpp's converter writes it on its own
  (convert_hf_to_gguf.py --mtp). The head is opened as a model of its own, whole on the host, and
  drafts in a context that reads the target's hidden states, as llama.cpp's speculative driver does
  with a draft head; the target loads as it is. scripts/fetch-hf-subset.py fetches only the
  tensors a set of patterns names from a Hugging Face checkpoint, by HTTP range requests (each must
  come back as the range asked for), into a directory the converter reads like the checkpoint, so a
  head is a few GB of download instead of the whole model, eight range requests at a time.
  scripts/convert-mtp-head.py runs llama.cpp's converter with the pre-tokenizer name taken from the
  model file the head drafts for, where the converter's own table does not know the tokenizer yet.
  docs/mtp.md has the three commands. Measured on a 24 GB desktop GPU with Qwen3.8-Flash-Next:
  45.1 tok/s against 30.5-33.8 without the head.
  Where the plan arms a device expert cache, the head follows the experts to that device
  (mtp-head-on-devices, a measured decision) and takes its bytes from the cache, never from the
  target's layers; on a 12 GB card with 8 slow cores the head drafting on the host cost 35 ms of a
  65 ms token and lost to no head at all.

• The stack measured on rented desktop GPUs (docs/bench-data/2026-10-26-rented-gpu/,
  docs/benchmarks.md, docs/community-benchmarks.md). A 24 GB card, --auto only: Qwen3.6-35B
  resident on the card at 217 tok/s, Flash-Next 125B with its hottest 18 GB of experts cached on the
  card at 70, DeepSeek V4 Flash streamed from the drive at 9.3 with 94% hits. The same DeepSeek file
  through llama.cpp's own offload decoded at 18.5 from the same cold page cache, which is the margin
  rule of #419 and the control-group page cache of #418, both filed with the numbers. A 12 GB card
  on a 42.6 GiB control group: --auto armed the device tier and was killed at 43 GB (its host-side
  memory is not in the ledger, #424); with the tier off, Flash-Next ran resident at 15.5 tok/s, and
  the MTP head drafting on 8 slow cores cost more than it saved. With CUDA and Vulkan built together,
  the Vulkan device reported no PCI id, so the backends were not grouped and the tier priced the
  slower one (#421).

Fixed

• --auto --overlap no longer refuses to start when the plan does not stream. Asking for
  --overlap keeps the capacity fitter's whole layers on the card instead of the device cache, and
  where what is then left on the host fits in RAM the plan turns streaming off, leaving --overlap
  set: the run stopped with moe.overlap requires moe.enabled (a 16 GB card with 30 GiB of RAM and
  Qwen3.8-Flash-Next). Every knob that acts only on the streamer's reads (--overlap,
  --row-stream, --route-ahead, --predict-log, --predict-prefetch, the overlap trial) is now
  turned off with a decision saying why wherever the plan does not stream.
• bmoe-server loads a model the plan gives a device expert cache. The server arms --overlap
  by default, and the device tier left it on beside the cache it armed, which the session refuses
  (--device-cache-mb computes the experts on the device; --overlap waits for them on the CPU): on a
  16 GB card every server load of Qwen3.8-Flash-Next failed where the CLI ran. The device tier now
  turns a default overlap off with a decision where it arms the cache; one the caller asked for
  still keeps the fitter's placement, as before.
• The expert cache in RAM takes what the plan leaves unspent. With a routing record, --auto
  sized the cache where the record's hit curve stopped rising and left the rest of the RAM idle; the
  record is a few hundred tokens, so its curve is flat past the experts those tokens touched, and a
  longer answer reads every expert it had not seen from the drive. On a 16 GB card with 30 GiB of RAM
  and Qwen3.8-Flash-Next over 1024 tokens, the plan's 15.7 GB cache held 83% and decoded 11.6 tok/s;
  22 GB held 96% and decoded 20.7 (27.5 over the last 256 tokens). The ranking still prices the cache
  on the curve against the resident groups; what none of them wanted now goes to the cache, with a
  cache-remainder decision. The planner test that asserted the old stop is updated.
• The model files' read-ahead is handed back after load. llama.cpp asks for the whole mapping to
  be read ahead when it loads (every non-lazy range), so a streamed model filled the page cache with
  bytes the run reads otherwise: on a 30 GiB box, 23 GB of one shard and 21 GB of the other. Under
  pressure the kernel swapped the expert cache out beside them, and the cache brake halved it twice
  (20.7 GB to 5.2 GB, the last 256 tokens of 1024 at 9.8 tok/s against 27.4 without). Where the dense
  set is held apart from the mapping (--dense-weights anon or pinned), the session now advises the
  kernel that the files' cached pages are not needed (posix_fadvise DONTNEED; a page still mapped
  stays). A no-op on Windows and Darwin.
• Speculation's memory is in the placement. The rollback snapshots speculation keeps in the
  target's context, and the MTP draft context's own state and compute buffers, were not in what
  llama.cpp's capacity fitter placed the model against, so on a card the fitter had filled the
  draft context's 0.9 GB of compute buffers ended the first prompt in a CUDA out-of-memory. The
  placement probe now hands the draft context to the fitter as the second model sharing the
  devices.
• The tables the plan leaves mapped stay mapped. The allocation prices the embedding group on
  its own lane, and where that lane is not residency (the graph gathers rows from it, so a copy buys
  bytes nothing reads) the run's anon/pinned dense policy still copied every dense tensor it was
  handed: a 28.8 GB per-layer table that "fits" by the kernel's count went into memory the ledger had
  sized the expert cache without, and on a 12 GB card's 42.6 GiB control group the run was killed at
  43 GB (#424). The plan now hands the run the tables it priced as mapped (dense-mapped, names read
  off the file's own grouping, a [derived] decision), the run holds them back as it holds back a
  tensor too large, and llama.cpp's lazy prefetch of such a table stays off. Nothing names a tensor
  or a size; a small table the allocation finds worth a resident copy is copied as before.

Changed

• The device expert cache is sized from the card's free memory, not from the fitter's whole
  layers (#428). --auto used to turn exactly the bytes llama.cpp's capacity fitter gave whole
  layers' experts into --device-cache-mb, so the cache inherited the fitter's granularity: the
  memory left under one more whole layer was never used, and a card whose dense part fit with no
  whole layer beside it (the 8-12 GB class) got no cache at all. The placement probe now reads what
  the fitter left free on each card past its margin, off the same free figure the fitter planned
  against, and the cache takes it beside the experts' bytes. On several cards the budget is the
  largest whose share, split by layers as llama.cpp splits it, fits every card. The floor is the RAM
  cache's: one token cycle, under which a cache evicts an expert before the same token needs it
  again; below it the fitter's placement stands, and an MTP head on the device never takes the cache
  under it. Planner tests for the remainder, a card with no whole layer, a card under one cycle and
  two cards. Not yet run on a card.
• The benchmark script measures the plan the planner learns, not only the first load.
  scripts/bench-report.sh ran --auto once with nowhere to keep the model's routing record or
  the lossless knobs' verdicts, so its row was always a cold plan: the cache priced by a coefficient
  rather than the model's own hit curve, --overlap never armed, the thread verdict lost. The app
  and the server keep both, so a community row and an app row measured two different plans. The
  script now keeps the record under its output folder (--routing-dir) and runs twice: the first
  load records and measures, the second plans on the record and is the row for the table. Both are
  printed, each with its mode: line and its plan: predicted ... measured line, so a row also
  says how far the planner was from the machine. A third run is the second with --overlap
  pinned: overlap changes no output, the planner arms it only when its trial measured it 10% faster,
  and the row says what that margin left on the machine. BENCH_RUNS=2 skips it, 1 reports the
  first load alone.
  The script waits between the runs (BENCH_SETTLE, 30 s): the kernel returns the first run's
  locked memory in the seconds after it exits, and a plan made before that sized a cache half the
  right one on a 16 GB laptop; when the second plan still saw much less memory than the first, the
  output says so. Numbers it prints are in the C locale, a decimal point in every language.
• The plan prints the storage curve it priced on. --plan-explain adds a plan: storage:
  line: whether uncached reads were in effect and the measured rate at each request size and lane
  count, with the refault price when measured. On macOS a shell cannot bypass the page cache, so the
  benchmark script's storage column there is this probe's one-lane rate at the engine's request size
  (read with F_NOCACHE), labelled as such, instead of a dd through the cache; the drive's name
  comes from diskutil. The Linux column stays the O_DIRECT dd probe.
• The Plan screen is one screen on every front-end, and its figures arrive when the answer
  does. The plan now lives in each UI's engine state (the web app's server store, the Android
  Engine, the iOS Engine), read when the server is reached, when a session opens and when a
  turn ends, whichever screen is open: before, each Plan screen held its own copy and re-read it
  only while it was the screen showing, so on a phone the measured figures appeared when you came
  back to it, not when the answer ended. The three UIs now share one layout: one banner for
  whatever went wrong with Measure again beside it, the measurement's stages, the Plan-before-
  every-load switch above the plan, the verdict with three figures (decode, cache hits, memory),
  Apply and Measure again together under them and nowhere else (the Android top bar loses its
  refresh icon, the iOS error loses its Dismiss), then memory, devices, decisions, the command and
  Copy diagnostic with the link to how the planner decides. The worst-case tile is gone from all
  three: predicted.s_per_token_worst is the cost model's own bound (one token with nothing
  resident, where every tier's saving is subtracted from), stays in the API and the diagnostic, and
  told a person nothing the expected figure did not. A decode figure is compared to the plan's
  prediction (+26%) only when the loaded model runs that plan.
• GET /api/plan with a model loaded answers the plan Apply would apply. With a model loaded,
  the load's plan is answered (from_last_load) only while nothing was measured after that load;
  ?cached=1 after a measurement made by hand with the model resident answers that measurement
  (cached), the one Apply applies, now with measured beside it too (before, a cached plan never
  carried the model's figures, so a Plan screen showing one never updated after an answer). The
  rule is held_plan in server/plan_view.h, with its test.