Skip to content

2.5 · LFM3 live retrieval benchmark A–F

Pre-release
Pre-release

Choose a tag to compare

@vinyteles vinyteles released this 06 Nov 21:40

Six APKs for the 120-question on-device, model-controlled retrieval benchmark. Install one assigned APK, tap Start / resume, and let it download its pinned assets. No Hugging Face account or token is needed.

Case Answer model Retrieval Backend
A LFM3 base Q4_K_M MobileRAG+ Vulkan GPU
B LFM3 fine-tuned Q4_K_M MobileRAG+ Vulkan GPU
C LFM3 base Q4_K_M F6 Vulkan GPU
D LFM3 fine-tuned Q4_K_M F6 Vulkan GPU
E LFM3 fine-tuned Q4_0 MobileRAG+ Vulkan GPU
F LFM3 fine-tuned Q4_0 F6 Vulkan GPU

Run A again: earlier frozen-context answers are a different protocol. All cases use the same 120 questions, shared tool prompt, 4096-token context, greedy decoding and up to two searches. Query embeddings run on CPU. The final answer has a 512-token limit; tool-generation turns have separate limits.

The phone saves and sends a checkpoint after every completed question. The server retains partials and publishes the assigned case to Hugging Face after 120 completed attempts. Reports preserve exact tool queries, evidence, conversations and separate timing for decisions, retrieval and final generation.

Instructions, downloads and time budgets · Protocol and model pins · Validation · Published sessions

This is an experimental release for physical-device testing. Retrieval parity, real-model CPU integration, checkpoint/resume and server checks passed. Samsung GPU stability and full 120-question durations are not yet measured for these live combinations. The APKs retain the existing package/signing identity; stop a run before installing another case.