Benchmark v3 — configurations 17–24

Eight ARM64 APKs for benchmark v3, configurations 17–24. Install the APK for your assigned configuration and tap Start / resume. Model and retrieval assets download separately and are verified by SHA-256.

Configurations	Model	RAG
17 / 18	Energy-Mobile mixed 4/8-bit	MobileRAG+ / F6
19 / 20	Gemma 4 E2B-IT mixed 2/4/8-bit	MobileRAG+ / F6
21 / 22	LFM3 fine-tuned Q4_K_M	MobileRAG+ / F6
23 / 24	LFM3 RLVR v3s400 Q4_0	MobileRAG+ / F6

Energy and Gemma request LiteRT-LM GPU; LFM requests Vulkan GPU. Embeddings run on CPU. All configurations include live tool calling and record retrieval candidates, final context, and per-question timings for retrieval and QA evaluation. F6 retains its NR/manual corpus without internal CEMIG documents.

All eight actual-model CPU emulator smoke tests passed with their assigned retrieval pipelines. Physical-phone GPU stability and complete 120-question benchmark runs are pending. This release therefore remains a prerelease.

Protocol and configuration details · Validation evidence

APK version: 3.0 (code 12). Android source: 25d29aecd05df1664d39b24a262aafb53d43111a; evaluation/documentation source: 86976e8440179587d4411ed868403b9a16606beb. Download SHA256SUMS and manifest.json for APK checksums and model pins. Installing another configuration updates the same app; finish or stop the current run before switching.

2.5 · LFM3 live retrieval benchmark A–F

Six APKs for the 120-question on-device, model-controlled retrieval benchmark. Install one assigned APK, tap Start / resume, and let it download its pinned assets. No Hugging Face account or token is needed.

Case	Answer model	Retrieval	Backend
A	LFM3 base Q4_K_M	MobileRAG+	Vulkan GPU
B	LFM3 fine-tuned Q4_K_M	MobileRAG+	Vulkan GPU
C	LFM3 base Q4_K_M	F6	Vulkan GPU
D	LFM3 fine-tuned Q4_K_M	F6	Vulkan GPU
E	LFM3 fine-tuned Q4_0	MobileRAG+	Vulkan GPU
F	LFM3 fine-tuned Q4_0	F6	Vulkan GPU

Run A again: earlier frozen-context answers are a different protocol. All cases use the same 120 questions, shared tool prompt, 4096-token context, greedy decoding and up to two searches. Query embeddings run on CPU. The final answer has a 512-token limit; tool-generation turns have separate limits.

The phone saves and sends a checkpoint after every completed question. The server retains partials and publishes the assigned case to Hugging Face after 120 completed attempts. Reports preserve exact tool queries, evidence, conversations and separate timing for decisions, retrieval and final generation.

Instructions, downloads and time budgets · Protocol and model pins · Validation · Published sessions

This is an experimental release for physical-device testing. Retrieval parity, real-model CPU integration, checkpoint/resume and server checks passed. Samsung GPU stability and full 120-question durations are not yet measured for these live combinations. The APKs retain the existing package/signing identity; stop a run before installing another case.

Benchmark 2.4 — per-question server checkpoints

Each APK runs one configuration with 120 questions. Install your assigned APK from the README table; all nine configurations, model artifacts and inference settings are unchanged.

After every completed question, the app saves and queues a cumulative report to the replacement upload server. The server retains partial progress durably. The app shows Server saved N / 120 answers; after all 120 assigned questions are complete, the server publishes the report to a public HF dataset and the app shows Hugging Face synced. No HF token or setup is needed on the phone.

• Install over the previous APK to preserve downloaded models and completed answers. Reopening queues the current saved run to the new server. Old upload credentials/checkpoints remain separately as backup.
• Network transfer is asynchronous; offline cumulative checkpoints retry automatically. Save a ZIP backup before an RTL reservation ends.
• The server host must remain awake and online. The HTTPS tunnel is temporary; endpoint discovery follows URL changes while checking server identity.
• This change protects progress already received by the server. It does not establish the cause of the reported phone freeze or claim new GPU performance results.

Validation: 31 Android regression tests reported OK (four optional integrations skipped); server suite 67 passed, one optional example skipped; 24 live server checks passed; Android-to-live-server-to-HF completion test passed. All nine APKs were checked for versions, configuration IDs, signatures, fixture identity and absence of the HF credential.

Configuration table and instructions · HF progress index · Validation

Every APK requires ARM64 / Android 12+. Verify downloads using SHA256SUMS.

Mobile benchmark 2.3 — one configuration per APK

One APK now runs exactly one model configuration on the same 120 frozen prompts. This replaces the five batch APKs with nine assignments for two-hour RTL reservations.

Download your assigned APK from the configuration table, install it, and tap Start / resume. Configurations 1 and 2 are the previously successful GPU paths; all five GPU configurations come before the four CPU configurations.

• One required-model download and one expandable benchmark progress card per APK.
• Android 12+ ARM64. Version 2.3 / version code 9. About 58 MB per APK, plus model weights and runtime caches.
• All APKs update the same app and preserve models, answers and recovery data. Finish or stop before switching APKs.
• Reports upload automatically: phone → upload relay → one public Hugging Face dataset per session. No participant HF token or account is required. Session index.
• Legacy batch sessions retain their original labels; each new configuration gets its own upload session.
• README times include 30 extra minutes per APK. Slow configurations may still require checkpoint/import/resume beyond one reservation. Gemma Q8 GPU remains conditional on available RAM.

All nine APKs passed signature, profile, label, architecture and fixture verification; Android regression and server suites passed. Validation details. These packaging and synthetic tests do not constitute new physical-phone performance measurements.

SHA256SUMS identifies the exact APK downloads.

Mobile benchmark 2.2 — install, start, automatic HF reports

Install the assigned APK and tap Start / resume selected. Version 2.2 creates a public HF results session automatically through the upload server. Participants need no Hugging Face account, token, repository setup, Python or Android Studio.

Batch	Configurations in order	Maximum answers
1 · gpu-known	Energy mixed48 LiteRT GPU; LFM3 Q4_K_M Vulkan GPU	240
2 · gpu-new	LFM3 mixed4 LiteRT GPU; original Gemma mobile LiteRT GPU	240
3 · gemma	Original Gemma Q8 Vulkan GPU if RAM permits; Gemma mobile LiteRT CPU	240
4 · lfm-cpu	LFM3 Q4_K_M CPU	120
5 · energy-cpu	Energy int8 CPU; Energy mixed48 CPU	240

The five APKs share their package and signing key. Installing another batch updates the app and preserves downloaded models and answers. All configurations retain the same 120 prompts and 4096/512 token settings from 2.1. Requires Android 12+ and ARM64; models download separately.

Open Report upload status to copy the public results link. Keep the app foreground and wait for Hugging Face synced before ending the reservation when possible. Received by server means the checkpoint is durably queued; publication to HF follows asynchronously. Both phone and server retry network failures. Local ZIP export/import remains available.

Fresh installations, runs and batch changes create separate upload sessions. Resuming the same run/batch reuses its session. If an existing installation/run/batch is reused for another reservation, choose New upload session while stopped.

Verification passed: 28-test Android suite, 55 server tests, actual Android → public HTTPS relay → HF uploads, independent checkpoint/hash/download checks, and live server restart recovery. Test reports are labelled synthetic and do not establish model quality or physical GPU speed.

Initial ingress is a temporary Cloudflare quick tunnel. The app follows the current URL from the public HF index while verifying the server identity pinned in GitHub. It retains checkpoints during outages. Server credential renewal remains an operator task; no HF credential is embedded in the APKs.

Run instructions · Public session index · Validation evidence

Download your assigned batch APK below. SHA256SUMS contains all five checksums.

Mobile benchmark 2.1 — five RTL batches and HF checkpoints

Five Android APK presets split the 120-question generation benchmark across RTL sessions. Each APK selects its batch, keeps the screen on while foreground, and can upload recoverable checkpoints to a public or private Hugging Face dataset repository.

Batch	Configurations in order	Maximum answers
1 · gpu-known	Energy mixed48 LiteRT GPU; LFM3 Q4_K_M Vulkan GPU	240
2 · gpu-new	LFM3 mixed4 LiteRT GPU; original Gemma mobile LiteRT GPU	240
3 · gemma	Original Gemma Q8 Vulkan GPU if RAM permits; Gemma mobile LiteRT CPU	240
4 · lfm-cpu	LFM3 Q4_K_M CPU	120
5 · energy-cpu	Energy int8 CPU; Energy mixed48 CPU	240

Start with batch 1 on a single phone. Colleagues can run separate batches concurrently. Every configuration uses the same 120 prompts, 4096-token context, and 512-token output cap.

All APKs share the application ID and signing certificate. Installing the next batch updates the app and preserves models, reports, and HF settings on that phone. For another RTL device, import the previous report ZIP and download the needed models again.

Create one HF dataset repository per reservation and connect it in Hugging Face upload settings using your own repository-scoped write token. Uploads occur about every minute during a run and after each configuration. Network failures retry; local checkpoints remain available. The source includes session creation and public progress-monitoring scripts. No access token is included in an APK.

Keep the app foreground and confirm its last-sync state before leaving. A batch is not guaranteed to finish within RTL's two-hour limit; resume unfinished answers in another session. Phone GPU timing/quality still needs physical validation.

Validation: all five release builds and signatures passed; emulator suite OK (24 tests), targeted cloud/batch regressions OK (4 tests), separate real-HF upload OK (1 test). The full queue test produced 1,080 synthetic answers. Public report download/checksum verification worked without authentication.

Setup and protocol · Validation details · Register a public session

Each APK is approximately 58 MB, requires Android 12 or later, and targets ARM64. Models download separately. SHA256SUMS lists the exact asset checksums.