Repository navigation
2.5 · LFM3 live retrieval benchmark A–F
Pre-releaseSix APKs for the 120-question on-device, model-controlled retrieval benchmark. Install one assigned APK, tap Start / resume, and let it download its pinned assets. No Hugging Face account or token is needed.
| Case | Answer model | Retrieval | Backend |
|---|---|---|---|
| A | LFM3 base Q4_K_M | MobileRAG+ | Vulkan GPU |
| B | LFM3 fine-tuned Q4_K_M | MobileRAG+ | Vulkan GPU |
| C | LFM3 base Q4_K_M | F6 | Vulkan GPU |
| D | LFM3 fine-tuned Q4_K_M | F6 | Vulkan GPU |
| E | LFM3 fine-tuned Q4_0 | MobileRAG+ | Vulkan GPU |
| F | LFM3 fine-tuned Q4_0 | F6 | Vulkan GPU |
Run A again: earlier frozen-context answers are a different protocol. All cases use the same 120 questions, shared tool prompt, 4096-token context, greedy decoding and up to two searches. Query embeddings run on CPU. The final answer has a 512-token limit; tool-generation turns have separate limits.
The phone saves and sends a checkpoint after every completed question. The server retains partials and publishes the assigned case to Hugging Face after 120 completed attempts. Reports preserve exact tool queries, evidence, conversations and separate timing for decisions, retrieval and final generation.
Instructions, downloads and time budgets · Protocol and model pins · Validation · Published sessions
This is an experimental release for physical-device testing. Retrieval parity, real-model CPU integration, checkpoint/resume and server checks passed. Samsung GPU stability and full 120-question durations are not yet measured for these live combinations. The APKs retain the existing package/signing identity; stop a run before installing another case.