💡 Real-time brood frame scoring via on-device multimodal AI during zero-tap inspections

[github-actions[bot]]

Summary

Use on-device multimodal AI (Gemini Nano v3 or React Native ExecuTorch) to score brood frames in real-time as the beekeeper points their phone camera at each frame during inspection. Voice announces findings: 'Strong brood pattern, approximately 85% capped, no queen cells visible.' Builds longitudinal frame health data with zero manual entry, deepening Broodly's zero-tap moat beyond what any competitor offers.

Market Signal

Gemini Nano v3 (2026) now supports multimodal on-device inference — text, image, audio, video — with as little as 2GB RAM, fully offline. Software Mansion's react-native-executorch brings edge AI directly to React Native apps in production. Varroa mite AI detection has achieved sub-1% error rates in peer-reviewed research (Insects 2025, Agronomy 2026). HiveSense offers offline Whisper voice but zero AI image analysis capabilities. BroodMinder's 2026 roadmap includes no real-time visual AI features. This capability gap is Broodly's strongest potential hardware-free differentiator.

On-device AI inference has reached sub-20ms latency on modern Android devices (AlephZero Labs benchmarks, 2026), making real-time frame scoring practically feasible during the natural inspection flow.

User Signal

The PRD elevates Vision AI to MVP scope based on 'visual interpretation uncertainty as a primary driver of newbie decision paralysis.' Existing ideas #382 (ExecuTorch on-device AI) and #406 (longitudinal visual tracking) validate the direction but address offline inference and historical comparison respectively — neither covers REAL-TIME frame scoring during live zero-tap inspections. This proposal bridges those ideas into a coherent zero-tap experience where quantitative frame data is captured as a byproduct of normal inspection behavior.

Technical Opportunity

React Native ExecuTorch is production-ready and open-source from Software Mansion, directly compatible with Broodly's Expo stack. The architecture already specifies Vertex AI for image analysis — on-device inference is a latency and offline improvement to that cloud pipeline. The VoiceLogCapture component pattern (FAB + state machine) extends naturally to a camera-based scoring flow with voice output. Frame scoring results feed directly into the existing observation JSONB schema (Story 3.2). The Gemini Nano v3 multimodal model can process both the visual frame image and the beekeeper's spoken context simultaneously.

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires on-device vision model fine-tuned for brood frame conditions. Training data curation and model accuracy validation are the primary risks. React Native integration path is proven.
Impact	high	Eliminates manual frame assessment — the most time-consuming data entry during inspections. Creates longitudinal quantitative data that currently doesn't exist in any beekeeping app. Deepens zero-tap moat.
Urgency	med	Strong differentiator but not crisis-driven. Timing aligns with Gemini Nano v3 device availability expanding through 2026.

Adversarial Review

Strongest objection: If the on-device model misidentifies a diseased frame as healthy, it could cause beekeepers to miss critical interventions. Model accuracy on diverse frame conditions (variable lighting, hive types, bee races, camera angles) may be insufficient for reliable field use.

Rebuttal: Broodly's recommendation contract requires confidence scoring on every output. Low-confidence frame scores would display as 'Unable to assess clearly — manual check recommended.' The system is additive (providing data that didn't exist before), not replacing expert judgment. MVP can focus on high-confidence categories only (brood pattern quality, queen cell presence/absence) and expand coverage as model accuracy improves. The PRD already specifies a 'minimum 70% interpretable-result rate' as the Vision AI quality gate — the same threshold applies here.

Suggested Next Step

Evaluate Gemini Nano v3 vs ExecuTorch for frame image classification on target devices (Pixel 10, recent Samsung). Create a labeled training dataset of 500+ brood frame images across conditions (lighting, hive types, seasons). Build a proof-of-concept with react-native-executorch showing on-device frame quality classification with confidence scores and voice output.