Peekify 🔍

A React Native (CLI) app that points your camera at the world and names what it sees in real time, drawing a labeled box on top of each object right in the camera canvas. Tap any object for a detailed AI identification powered by cloud vision.

Splash  →  Home ("Open Camera")  →  Live Camera
                                     ├─ on-device detection (TensorFlow Lite) → boxes + labels @ ~8 fps
                                     └─ tap an object → still frame → cloud vision → precise name

How it works (hybrid detection)

Layer	Tech	Role
On-device (offline, real-time)	react-native-vision-camera frame processor + react-native-fast-tflite running SSD MobileNet v1 (COCO, 90 classes) + vision-camera-resize-plugin	Detects & labels objects live, draws boxes on the canvas. No internet, no cost.
Tap to identify, FREE (default)	A second bundled TFLite model: MobileNet ImageNet (1000 classes), run on the live frame on-device	More specific name than the 90-class detector, fully offline, no API key, no cost.
Tap to identify, Cloud (optional)	A still photo → tiny backend proxy → cloud vision	Brand-aware naming ("bottle of Jack Daniel's", "carton of whole milk"). Opt-in.

No API key needed to run the app. Everything works free and offline out of the box, live detection + on-device tap-to-identify. The cloud path is an optional upgrade: flip CLOUD_VISION_ENABLED to true in src/config.ts and run the /server proxy with your key (the key never ships in the app).

Project layout

App.tsx                      Navigation (Splash → Home → Camera)
src/
  screens/SplashScreen.tsx   Animated splash
  screens/HomeScreen.tsx     "Open Camera" landing
  screens/CameraScreen.tsx   Live camera, TFLite frame processor, tap-to-identify
  components/DetectionOverlay.tsx  Bounding boxes + labels
  ml/labels.ts               COCO label map (model class index → name)
  ml/types.ts                Detection type
  services/cloudVision.ts    POSTs a frame to the proxy
  config.ts                  Proxy URL + detector thresholds
  assets/model/
    ssd_mobilenet.tflite     The on-device model (bundled)
    labelmap.txt             Original COCO labelmap (reference)
server/
  index.js                   Express proxy → cloud vision (holds the API key)

Prerequisites

• Node ≥ 22, the React Native CLI environment (Xcode 16+/CocoaPods for iOS, Android Studio + SDK for Android).
• A physical device is recommended, simulators/emulators have no real camera.
• An Anthropic API key is optional, only needed if you turn on the cloud naming upgrade. The app is fully functional and free without it.

1. Install

npm install            # app deps (already done if you received node_modules)
cd ios && bundle install && bundle exec pod install && cd ..   # iOS only

2. (OPTIONAL) Enable the cloud vision upgrade

Skip this entirely to stay free/offline, tap-to-identify uses the on-device ImageNet classifier by default.

To turn on richer, brand-aware naming: set CLOUD_VISION_ENABLED = true in src/config.ts, then run the proxy:

cd server
npm install
ANTHROPIC_API_KEY=sk-ant-xxxx npm start      # listens on :8787

Then point the app at it in src/config.ts → CLOUD_PROXY_URL:

• iOS simulator → http://localhost:8787
• Android emulator → http://10.0.2.2:8787
• Physical device → http://<your-computer-LAN-IP>:8787 (same Wi-Fi)

The live on-device detection works with no proxy at all, the proxy is only needed for the "Tap to identify" precise naming. Set CLOUD_VISION_ENABLED to false in src/config.ts to hide that feature.

3. Run the app

npm start            # Metro
npm run ios          # or: npm run android

Grant the camera permission when prompted, tap Open Camera, and point it at things. Labels appear over each detected object; tap anywhere to get the cloud service's detailed identification.

Modes (top of the camera screen)

A segmented control switches between three modes:

Mode	What it does	Tech
Objects	Live labeled boxes + tap-to-identify (top-3 / optional cloud)	TFLite, on-device
Label	Point at a product, tap Read label → product name + See details showing all the text on the packet	Google ML Kit OCR, on-device, free
Barcode	Point at a barcode → product name + what it is + See details (ingredients, categories, Nutri-Score…)	VisionCamera scanner + Open Food Facts API (free, no key)

Barcode lookup uses Open Food Facts, excellent for groceries/packaged food (e.g. Oreo). Non-food or uncatalogued barcodes show "not found" with the raw code. OCR works on any printed text, fully offline.

In-app controls

• Sensitivity slider (bottom of the camera screen), live-adjusts the confidence threshold for which boxes are shown. The detector keeps everything above a low floor (MIN_SCORE); the slider filters for display, so dragging it updates instantly without rebuilding the frame processor.
• Tap to identify, sends the current frame to the cloud vision service for a precise name.

Tuning

src/config.ts:

• DETECTION.scoreThreshold, the slider's initial value (default 0.5).
• DETECTION.throttleMs, how often the model runs (default 120ms ≈ 8 fps).
• DETECTION.maxResults, max boxes per frame.

Box orientation: boxes are rotated from sensor space to the portrait preview in src/ml/orientation.ts using frame.orientation. If on your specific device boxes look rotated/mirrored, tweak the rotatePoint mapping there (geometry is isolated and unit-tested).

To use a different on-device model, drop a .tflite into src/assets/model/, update the require(...) in CameraScreen.tsx, and adjust the output-tensor parsing + labels.ts to match the new model.

To switch the cloud model (e.g. cheaper), change model in server/index.js (claude-opus-4-8 → claude-sonnet-4-6).

Notes / known limitations

• Box alignment assumes a portrait, cover-fit preview; extreme rotations may offset boxes slightly. Fine for a demo; tighten with frame orientation math for production.
• SSD MobileNet recognizes 90 COCO classes on-device. Anything outside that set shows no live box, but tap-to-identify still names it via the cloud vision service.
• The model file (~4 MB) is bundled with the app; first model load takes a moment ("Loading detector…").

Key implementation gotchas (already handled)

• Pinned react-native-vision-camera@^4, v5 is a Nitro rewrite the TFLite plugins don't yet target.
• react-native-fast-tflite v3 is Nitro-based: runSync(ArrayBuffer[]) → ArrayBuffer[]. Inputs/outputs are wrapped via .buffer / new Float32Array.
• metro.config.js adds tflite to assetExts.
• babel.config.js adds react-native-worklets-core/plugin (last), plus the legacy @babel/plugin-proposal-* + @babel/preset-typescript packages that plugin expects to be hoisted.
• iOS NSCameraUsageDescription, Android CAMERA permission, and Android minSdkVersion 26 are set.
• Android noCompress "tflite" is set so the model can be memory-mapped in release APKs.
• Barcode scanning needs VisionCamera_enableCodeScanner=true in android/gradle.properties (set), pulls the MLKit barcode model.
• OCR (@react-native-ml-kit/text-recognition) requires iOS 16+, so the Podfile pins the deployment target to 16.0.

⚠️ The OCR + barcode features add native modules, so after pulling these changes you must do a full native rebuild (npm run android / npm run ios) , a JS reload is not enough.

Networking in release builds

The proxy runs over http. Debug builds allow that automatically; release builds don't (cleartext is blocked by default).

• iOS: already release-safe for a local proxy via NSAllowsLocalNetworking (cleartext to localhost / LAN hosts is permitted without TLS).
• Android: handled by android/app/src/main/res/xml/network_security_config.xml, which permits cleartext only to the proxy hosts (localhost, 10.0.2.2). For a physical device, uncomment and set your computer's LAN IP there.

For production, serve the proxy over HTTPS and set CLOUD_PROXY_URL to the https:// URL, then no cleartext exception is needed on either platform (remove the Android <domain> lines).