💡 On-device produce recognition for one-tap vendor catalog updates

[github-actions[bot]]

Summary

Add an optional "Snap your stall" capture flow: vendor takes a single photo of their table at setup, an on-device vision model detects produce categories (tomatoes, eggs, herbs, bread...), and pre-fills today's product availability list for one-tap confirmation. Targets the longest-pole vendor task — keeping today's product list current — without typing.

Market Signal

• 2026 retail computer-vision adoption is now mainstream: hybrid on-device (Apple Vision / TensorFlow Lite) plus cloud fallback is the standard pattern, and 94% of retail shoppers actively choose produce recognition at self-checkout (Toshiba Elera deployment, Weis Markets). The pattern has crossed the chasm; nothing about it is exotic anymore.
• The team already commissioned domain research on this exact capability: _bmad-output/planning-artifacts/research/domain-real-time-object-detection-research-2026-03-15.md. The intent exists; the FRs and stories never landed.
• No incumbent in the farmers-market vertical uses computer vision for vendor-side capture. MarketWurks' new vendor app focuses on attendance and sales tracking, not product capture. This is a clean differentiation lane.

User Signal

• The PRD's Brad persona requirement is "<10 seconds" for any market-day action (NFR1). Typing a product list is the friction wall the entire exception-first design tries to route around. A photo + confirm is the only path that beats typing in cold/wet/gloved outdoor conditions.
• FR16 ("Vendor can update availability state for products or categories without replacing full baseline profile data") explicitly anticipates a quick partial-update path; today the only implementation is manual toggles.

Technical Opportunity

• React Native already supports on-device inference via react-native-fast-tflite or Apple Vision (react-native-vision-camera + frame processors). Model size for produce-class classification is <10 MB; first-inference latency on a mid-range phone is <500ms.
• The vendor product catalog (Epic 3) already exposes a stable category taxonomy — the model only needs to map detected labels onto that existing taxonomy, no schema changes required.
• The detection runs entirely on-device, so there's no privacy concern, no Supabase storage cost for stall photos, and no degradation when the venue has poor signal (NFR6 graceful degradation requirement).
• Ship as opt-in Speed Mode for produce-heavy vendors first; art and craft markets can be excluded by market-type flag until model coverage extends.

Assessment

Dimension	Score	Rationale
Feasibility	med	Real eng work — model selection, on-device runtime integration, taxonomy mapping, confirmation UI — but each piece is well-understood and bounded. The team already invested in the research.
Impact	high	Removes the single biggest source of friction in the vendor day, reinforces the "exception-first UX" pillar, and creates a story-grade feature for early-customer marketing.
Urgency	med	Not an emergency, but the longer the catalog stays manual, the more vendor abandonment data accumulates. Worth slotting before the Phase 3 expansion features.

Adversarial Review

Strongest objection: "Model accuracy on real farmer-market stalls — varied lighting, mixed produce, occlusions, regional varieties — will be too low to be trustworthy, and a single wrong auto-fill destroys the trust the platform spent the whole MVP building."

Rebuttal: The flow is "detect → pre-fill → vendor confirms with one tap" — the human is always in the loop, and the worst case is no worse than today (vendor types it themselves). Confidence threshold gates which detections appear; below threshold, the field stays empty. Failure mode is identical to current state, while the success mode is a 10x speedup. Combine that with a Speed Mode opt-in and there is no downside risk surface.

Suggested Next Step

Re-read domain-real-time-object-detection-research-2026-03-15.md, then run bmad-quick-spec on a one-vendor pilot scope: one model, one taxonomy mapping, one confirmation screen, behind a feature flag. Validate with 2 produce vendors in a friendly market before broadening.

---

Proposed by the BMAD Analyst (Mary) on 2026-04-07.
Workflow run

[github-actions[bot]]

Weekly Update

What Changed

• Expo SDK 56 released (May 2026): Expo UI APIs are now stable for production, including SwiftUI (iOS) and Jetpack Compose (Android) integration. This materially improves the feasibility of integrating on-device vision via frame processors — the native API layer is no longer experimental.
• React Native 0.85 (April 2026): New Architecture is now the assumed-stable default, not optional. This eliminates the bridge overhead that previously added latency to native module calls like vision inference.
• On-device AI benchmark data (July 2026): Image classification (MobileNetV3) now achieves 4-8ms on-device on iPhone 15+ and recent Android flagships; object detection (optimized YOLO variants) achieves 12-20ms. The original proposal estimated "<500ms first-inference latency" — actual performance is 25-125x better than projected.
• Market adoption: 70-80% of AI features shipped in 2026 mobile apps are on-device. iOS expanded system-level Vision and Image Analysis APIs for real-time object detection and segmentation. The pattern has fully crossed the chasm.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	med-high	↑ Expo SDK 56 stable native APIs + 25-125x better latency than projected reduces integration risk
Impact	high	high	→ Unchanged — still the #1 friction point in vendor day
Urgency	med	med	→ Still not blocking, but Expo SDK 56 makes this cheaper to prototype now

Recommendation

Advance. The Expo SDK 56 release removed the largest technical uncertainty (unstable native API surface). Recommend scheduling a 1-sprint prototype spike using react-native-vision-camera + frame processor with a MobileNetV3 produce classifier, targeting 3 common produce categories (tomatoes, berries, leafy greens) on a single friendly vendor's stall.

[github-actions[bot]]

Weekly Update

What Changed

• React Native On-Device ML Maturity (2026): The React Native ecosystem now has mature on-device ML support. Key frameworks available: react-native-vision-camera + react-native-fast-tflite + vision-camera-resize-plugin for camera-based image classification. Core ML (iOS) and TensorFlow Lite (Android) both supported with good Expo compatibility.
• YOLOv8 on React Native: A working example of React Native Core ML image classification using YOLOv8 has been published, demonstrating real-time object detection in React Native apps. This is directly applicable to produce recognition.
• Software Mansion Guide: Software Mansion (maintainers of react-native-reanimated) published a comprehensive guide on on-device AI/ML in React Native, covering model optimization, inference performance, and battery efficiency. They confirm that on-device inference in 2026 is production-ready for image recognition use cases.
• Hybrid AI Pattern: The emerging best practice is hybrid on-device + cloud API: run fast classification on-device for instant UX response, with optional cloud refinement for edge cases. This pattern reduces latency and works offline (critical for outdoor market environments with spotty connectivity).

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	high	↑
Impact	high	high	→
Urgency	low	med	↑

Recommendation

Advance. Technical feasibility has materially improved since this idea was first proposed. The specific toolchain (react-native-vision-camera + react-native-fast-tflite + YOLOv8) is now documented and production-tested. The hybrid on-device/cloud pattern addresses the outdoor connectivity constraint. Consider creating a technical spike story to build a proof-of-concept: camera → on-device classification → vendor catalog category suggestion. This spike would also inform the new Product-Level Following idea (#394), which relies on a product category taxonomy that produce recognition could help populate.

---

Weekly feature ideation update — 2026-08-07.