Snap a photo, describe a world, walk through it in 3D.
On-device AI segments your subject, then WorldLabs renders a navigable Gaussian Splat environment around them — live on the web in under a minute.
iPhone
→ SSD-MobileNet auto-detects person, feeds click to SAM 2
→ SAM 2 (CoreML) pixel-precise subject segmentation
→ CLIP ViT-B/32 scene understanding
→ SmolLM2 135M generates creative world prompts
→ Whisper tiny optional voice hint
→ transparent cutout PNG
Backend (FastAPI)
→ WorldLabs Marble API → .spz Gaussian Splat
Web Dashboard (Next.js + Three.js)
→ live SSE progress → navigable 3D world
All five models run entirely on-device. No cloud ML.
| Layer | Tech |
|---|---|
| Mobile | Expo SDK 54 · React Native 0.81 · New Architecture |
| On-device ML | react-native-executorch (SSD, CLIP, SmolLM2, Whisper) + SAM 2 Swift/CoreML |
| Backend | FastAPI · SQLite · SQLModel |
| 3D rendering | Three.js + SparkJS (.spz Gaussian Splats) |
| Real-time sync | Supabase Realtime · SSE |
| 3D generation | WorldLabs Marble API |
# Backend (port 8024)
cd backend
cp .env.example .env # add WORLDLABS_API_KEY
uvicorn main:app --reload --port 8024
# Frontend (port 3000)
cd frontend && npm install && npm run dev
# Mobile
cd mobile && npx expo start --iosGet a WorldLabs API key at worldlabs.ai.