v0.8.1 — anon dense weights by default

Flips the default dense-weight policy to anon (O_DIRECT).

On a >RAM model, anon reads the dense (non-expert) weights via O_DIRECT into private buffers, so a memory reclaim swaps them to zram (fast) instead of refaulting them from flash (slow) — and the expert cache finally has room to earn its budget. Measured 0.998 vs 0.711 tok/s on gpt-oss-120b with a 2000 MiB cache.

Warm stays one tap away in Settings for models that fit in RAM, where its page-cache prefetch is the better trade. The fresh-install path now defers to a single source of truth for the default.

Note on reproducing the published numbers: those were measured with warm. To match them, set dense weights to Warm at load in Settings.

Pre-release, like v0.8.0: the session-teardown A/B is still owed, and this default flip's cross-model A/B (Qwen, Gemma) has not been run yet. Promote with gh release edit v0.8.1 --prerelease=false once measured.

Install

BigMoeOnEdge-v0.8.1-dev.apk — dev flavor, arm64. adb install -r to keep any models already on the device.

v0.8.0 — one-tap model downloads

Getting a first model no longer needs adb or a hunt on Hugging Face. The example app now carries a built-in catalog, and the dev shared directory drops a name inherited from an earlier project.

Highlights

• Model catalog — a "Get a model" card offers the models this engine is measured on, each one tap: Qwen3-30B-A3B-Q4_K_M (~18.6 GB) and Gemma-4-26B-A4B-it-Q4_K_M (~17.0 GB). gpt-oss-120b is listed with its PC-side merge recipe (Hugging Face ships that quant split in two, and expert streaming needs a single file). Arbitrary URLs and the file picker stay, under "Other model".
• Downloads that survive the app — tracked by filename and seeded from DownloadManager, so a multi-GB transfer started before the process died is picked back up instead of running unseen. Free space is checked before the download starts.
• /data/local/tmp/shardllm → /data/local/tmp/bmoe (dev flavor). Existing devices: adb shell mv /data/local/tmp/shardllm /data/local/tmp/bmoe.

Fixes found while validating on-device

• Declared INTERNET, without which DownloadManager could never actually run a download — the URL downloader had silently done nothing on every prior build.
• When the same gguf exists in two scanned dirs, prefer the copy on the real filesystem (O_DIRECT works) over emulated storage, instead of whichever the scan hit first.
• Catalog rows no longer wrap to a stack of one-word lines; the card can be collapsed; a refused download reports in its own row, in GB.

Verified on device

OnePlus 8 Pro and 15R: catalog states, the free-space guard, and a full Gemma download → finalize → appears in the picker, all confirmed. Byte-identity host gates pass.

Pre-release, matching v0.7.0: the session-teardown A/B measurement it carries is still owed. Promote with gh release edit v0.8.0 --prerelease=false once measured.

Install

BigMoeOnEdge-v0.8.0-dev.apk — dev flavor, arm64. adb install -r to keep any models already on the device.

v0.7.0 — measuring the decode, and a session that lets go of its RAM

Pre-release. The APK builds, installs and generates on the target device, but the changes below have not had their on-device A/B yet — in particular the session teardown fix, which is a lifecycle change only the app can exercise. Promoted to a stable release once that measurement is in.

Measuring the decode instead of inferring it

--compute-trace PATH and --io-trace PATH. Until now compute_ms was a residual — whatever was left after the parts that were measured — so a large figure could equally mean genuine matmul, a dense-weight fault storm on a >RAM model, or a throttled CPU. Indistinguishable.

The compute trace returns true for every node from the eval callback, which forces ggml to isolate and synchronise each one. The wall delta between callbacks is then that node's real compute, and the major-fault delta attributes a >RAM stall to the node that actually paid for it. The I/O trace emits one row per expert read (latency, aligned window, layer/expert/projection/lane). Read both with scripts/decode-analyze.py.

Both are diagnostics that perturb the run they measure, so only shares are meaningful — never compare their tok/s with the benchmark tables. Done entirely from outside llama.cpp through the public cb_eval: no patch, no fork.

A session that lets go of its RAM before the next one loads

Changing the model or a setting started a new engine while the old process still held its model and its expert cache. The replacement then sized its cache against a MemAvailable still deflated by the dying one — so the app was triggering "two engines at once" on itself at every settings change, silently starving the very cache being retuned, and the combined footprint could be OOM-killed. The new session now reaps the old process off the main thread before probing memory.

Also fixed: the delayed force-kill is now cancellable, so an unload followed quickly by a new prompt can no longer let a stale kill land on the fresh process (exited 137).

Smaller cache rungs, to find where the cache stops paying

Settings now offer 500 and 1000 MiB expert-cache budgets. They were unavailable because the engine rejects a fixed budget under its 1500 MiB floor — a floor that says a cache smaller than one token's routed working set can only thrash. Sound, but it was measured on models whose cache pays for itself. On gpt-oss-120b at top-2 (~886 MB routed per token, an 8–13% hit from a 2000–3000 MiB budget covering ~5% of a 56.8 GB expert bank) the question is live, so the small rungs route through the floor's own escape hatch.

Also in this release

• Markdown-rendered answers in the Android chat, and a scroll that no longer fights the user.
• --route-trace: which experts every token actually routed to, per layer, for offline analysis (scripts/route-analyze.py, scripts/route-viewer.py).
• docs/android-memory.md — what reclaims a >RAM engine's memory on a phone, which levers exist (almost none), and why the cache hit rate is the signal the kernel judges you by: the LRU promotes a page only on a second reference, and a cache hit is that reference. Records the watermarks, the vendor's swappiness-160 override, and the 65536-byte RLIMIT_MEMLOCK ceiling that makes mlock unusable here.
• docs/benchmarks.md split into the Android matrix and docs/benchmarks-gpt-oss.md.

Performance

Unchanged, and not re-measured for this release. No throughput claim here is new — the tables in docs/benchmarks.md still stand as measured for v0.6.0. Everything added here is either a diagnostic (off by default), an app-lifecycle fix, or a new Settings rung.

Install

app-dev-debug.apk is the sideload (dev) build for arm64-v8a, built from 0f1d503. It reads models adb-pushed to /sdcard/Download or /data/local/tmp; installing it does not touch already-downloaded models.

adb install -r app-dev-debug.apk

v0.6.0 — dense weight warm-up

Dense weight warm-up

Models larger than device RAM no longer pay a slow-start: the non-expert weights (embeddings,
attention, norms, lm_head), which stay mmap-resident while experts stream, are now page-cached with
one sequential sweep at load instead of faulting in lazily during the first tokens.

Measured (OnePlus 15R, 11.4 GB RAM): on gpt-oss-120b (~5.2× RAM) the first token drops from
~18 s to ~1 s and the first-five-token wall average improves ~20×. Smaller-dense models (Qwen, Gemma)
are neutral — hit rate and cache budget unchanged, streaming path byte-identical.

On by default; opt out via the Dense weight warm-up toggle in Settings, or --no-warm-dense
on the CLI. Details and per-token data in docs/warmup-analysis.md and
docs/bench-data/2026-07-14-warmup/.

Install

app-dev-debug.apk is the sideload (dev) build for arm64-v8a. It reads models adb-pushed to
/sdcard/Download or /data/local/tmp; installing it does not touch already-downloaded models.

v0.5.0

Android demo — richer telemetry & CPU temperature

• CPU temperature during generation now comes from the kernel thermal zones (/sys/class/thermal), which track compute load directly, instead of the battery pack (which lags and reflects charging). Best-effort, no permission, falls back to battery temp if no CPU zone is readable. (#18)
• Richer live telemetry: prefill rate, time-to-first-token, streamed MB, cache footprint, temperature.
• gpt-oss-120b on-device streaming benchmarks + drivers.

App version: versionName 0.5.0 (versionCode 6).

v0.4.0

v0.4.0 — universal armv8.2 APK baseline (no i8mm), Auto cache, hybrid…

v0.3.0

v0.3.0: reusable Session, adaptive cache budget, Turbo top-k, multi-t…

v0.1.1 — baseline @ f3371aa

Testing baseline: the app built at commit f3371aa (HEAD of main). Contains all functionality up to and including the O_DIRECT internal-storage import; none of the reverted warm-up changes. In-app panel shows git SHA f3371aa (versionName reads 0.1.0-dev). Pre-release: for on-device validation.

v0.1.0

First tagged release of BigMoeOnEdge — a ports-and-adapters engine that streams MoE experts from flash so models larger than device RAM run on-device, built on top of llama.cpp's public API with no fork.

Highlights

• MoE expert-selective streaming for qwen3moe (Qwen3-30B-A3B and siblings), qwen2moe, and llada-moe: only the routed experts per token are read from flash, with an optional LRU cache and a parallel read pool.
• Lossless — byte-identical to a full in-memory run, proven by the synthetic gates and confirmed on a real 64-expert 4 GiB model.
• Zero-fork llama.cpp: streaming rides entirely on the public eval-callback and gguf accessors; third_party/llama.cpp stays a stock upstream submodule.
• bmoe-cli host tool with machine telemetry and a CSV sink.
• Android example app with a live telemetry panel (tok/s, compute-vs-flash-I/O split, cache hit rate).

Assets

The attached app-debug.apk (arm64-v8a) is built by CI and debug-signed for on-device validation, not Play distribution. Push a .gguf to the app's files dir and pick it in the model picker. See examples/android/README.md.

Full details in CHANGELOG.md.