Experimental. This project is a research engine, and this release is a snapshot of it rather
than a settled product. The defaults are the configuration that has been measured; several of the
levers it ships are marked experimental in the app and in the docs, which means measured on one
device, measured once, or not yet measured. Treat any number here as a measurement on one phone,
not a specification.
The headline: DeepSeek V4 Flash 0731, a 284B-parameter MoE at roughly 91 GB on disk, generating
on a phone with 12 GB of RAM. More than seven times more model than memory, streamed from flash as
the three shard files Hugging Face ships, with no merge step and no PC in the loop.
What is new
Split models stream natively. Anything over Hugging Face's 50 GB per-file limit ships as
several shards, and until now the streamer assumed one file. Every tensor is resolved to its
(shard, offset), each shard gets its own reader, and you pass the first shard exactly as you would
to llama.cpp. The app downloads a sharded model as one entry, resumable, with a single progress bar.
DeepSeek V4 Flash support, as one recipe row: V3.2-style routing with a per-expert bias and an
always-on shared expert. The V4 attention machinery is dense-side llama.cpp code, invisible to the
streaming seam.
Self-speculative decoding, two sources. The model's own multi-token-prediction head where the
gguf carries one, or an n-gram lookup over the prompt and the answer so far, which needs nothing
from the model and costs exactly a plain decode on the steps it declines to draft. Both verify the
same way and keep only what the model itself would have produced. Off by default.
Route-ahead, which commits a layer's expert choice several layers early so its reads can never
be wasted. Lossy by construction, off by default.
The app declares itself a game. Vendor performance layers read that, and on the test device it
moved the app onto the boosted path: the foreground CPU ceiling went from 1.9/1.65 GHz to the
hardware maximum. The effect belongs to the vendor rather than to Android, so treat it as a
measurement on one phone. In-app numbers from before this release are not comparable with numbers
from after it.
Settings are grouped by purpose, with the experimental levers folded into a collapsed group
under each category rather than interleaved with the settings that are known to work. Descriptions
were rewritten to say what a knob does, without measured figures that would need the device, the
model and the day beside them to mean anything.
What is not in it, and why
--odirect-zero-copy stays out. It reads expert slices straight into the cache with no bounce
copy, and it is correct on Android, but on a Windows host combined with --overlap it produces
wrong bytes silently. It is tracked in #149 with the reproduction and the six hypotheses already
falsified; the branch carries a guard so a host build cannot hit it.
Correctness
The byte-identity gates now also cover the speculative decode loop and route-ahead, which shipped
in this release. Streaming only the routed experts must produce output identical to running with
every expert resident, and that holds across the cache, evictions, overlap, both prefetchers, the
dense-weight rebind, multi-turn sessions and a split model read across shard boundaries.
Full detail in CHANGELOG.md.