Physical tiles. Live camera. Embodied learning. No app store required.
Math Tiles is a full-stack educational web app that uses AI-powered computer vision to read physical number tiles through a device camera in real time. A student places tiles on a desk to answer a math problem — the camera detects the number, validates the answer, and responds with immediate feedback, points, and streaks.
The core premise is grounded in embodied cognition research: physical interaction with math tiles creates richer, multi-modal memory traces than tapping a touchscreen. Every design decision — from the FocusBurst movement break to the spaced repetition engine — is backed by peer-reviewed neuroscience.
"We didn't add movement as a feature. We built the app because movement is the learning."
Built for iPad. Runs entirely in Safari. No app store, no download, no hardware beyond a set of number tiles.
Math Tiles is built on 11 peer-reviewed studies spanning embodied cognition, cognitive load theory, neurochemistry, and ADHD research. The full citation list is available at /science.
| Mechanism | Timeframe | Source |
|---|---|---|
| Dopamine, norepinephrine & serotonin release from movement bursts | Seconds to minutes | Basso & Suzuki (2017), Brain Plasticity |
| BDNF upregulation — grows new neural connections | Hours to days | Sleiman et al. (2016), eLife |
| Richer multi-modal memory traces, flatter forgetting curve | Days to weeks | Lindgren & Johnson-Glenberg (2013), Frontiers in Psychology |
| Physical interaction offloads working memory (cognitive offloading) | Immediate | Goldin-Meadow & Wagner (2005), Trends in Cognitive Sciences |
| Embodied cognition + cognitive load theory synergy | Foundational | Zou et al. (2025), Nature Human Behaviour |
| Movement enables focus for ADHD learners | Foundational | Hoy et al. (2025), Sports |
The OCR pipeline sends a JPEG frame to Gemini 2.0 Flash every 600ms via a server-side tRPC procedure. The model returns a structured JSON response with a detected digit (0–99) and a confidence score.
- Two-tile support — placing two tiles side by side returns a two-digit number (e.g. tiles "1" + "3" → submits
13) - 2-consecutive-read confirmation — the same number must appear in two successive frames before auto-submit fires, preventing false positives
- Confidence threshold of 0.78 — tuned to balance speed and accuracy on iPad in varied lighting
- 1200ms grace period on each new problem — prevents the camera from re-reading the previous tile before the student has cleared the desk
- 92% JPEG quality — maximises OCR accuracy without excessive payload size
- No video is ever stored or transmitted beyond the single-frame API call
| Feature | Trigger | Research Basis |
|---|---|---|
| FocusBurst | 2 consecutive wrong answers | Basso & Suzuki (2017) — neurochemical reset via movement |
| Science Badge | Mid-game pop-up | Zou et al. (2025) — reinforces embodied learning rationale |
| AI Tutor (Ciri) | Player request or wrong answer | Worked example + conversational explanation |
| Worked Example | After incorrect answer | Reduces extraneous cognitive load |
| Spaced Repetition | Background, every session | SM-2 algorithm on weak-fact deck |
| High Energy Mode | Settings toggle | Amplified feedback for high-arousal learners |
The engine maintains a rolling window of the last 15 answers and promotes difficulty when either a streak milestone or a 93%+ accuracy threshold is reached. Difficulty never decreases — the system only promotes, keeping the experience encouraging.
| Level | Name | Operations |
|---|---|---|
| 1 | Novice | Addition |
| 2 | Apprentice | + Subtraction |
| 3 | Scholar | + Multiplication |
| 4 | Expert | All four operations |
| 5 | Legend | Full mental math |
When a streak gets hot, the math pauses and a retro arcade challenge unlocks as a movement and engagement break:
| Challenge | Description |
|---|---|
| Tap Frenzy | Tap falling tiles as fast as possible |
| Memory Flash | Memorise and recall a number sequence |
| Frogger | Full Frogger clone — 8 lanes, 3 lives, 30s timer |
| Reaction Blast | Hit targets the instant they appear |
| Pac-Man | Full Pac-Man clone with ghost AI and power pellets |
- Daily streak tracker with fire animations and streak freeze mechanic
- Achievement system — 20 badges unlocked by gameplay milestones
- Global leaderboard with paginated rankings
- XP & level system — earn XP per correct answer, level up with animated overlays
- Daily Challenge — fresh problem set every 24 hours
- Weekly Challenge — longer format, resets each Monday
- Session History — full log of every game with score, accuracy, and problems solved
- Avatar builder — choose icon, colour, frame, and display title
- 8 Game World Themes — Cyber, Ocean, Forest, Space, Candy, Volcano, Arctic, Neon
- Sound Pack system — unlock new audio themes through achievements
- Reward Store — spend earned XP on cosmetic upgrades
- Race Mode — create or join a room with a 6-character code; compete head-to-head in real time via WebSocket
- Live race HUD — shows opponent progress during the race
- Deep-link sharing — share a direct URL to a specific race room
- Teacher Dashboard — create classrooms, assign problem sets, track student progress per assignment
- Parent Portal — link child accounts, view weekly digest reports with accuracy trends
- Curriculum Standards Badge — problems tagged to Common Core math standards
| Layer | Technology |
|---|---|
| Frontend | React 19 + Vite + TypeScript |
| Styling | Tailwind CSS 4 + shadcn/ui |
| OCR / Vision | Google Gemini 2.0 Flash (server-side, structured JSON output) |
| API Layer | tRPC 11 + Superjson (end-to-end type safety) |
| Backend | Express 4 + Node.js |
| Realtime | In-memory WebSocket race store |
| Database | TiDB (MySQL-compatible) + Drizzle ORM |
| Auth | Manus OAuth + JWT session cookies |
| Animations | Framer Motion |
| Testing | Vitest — 46 tests, 0 failures |
client/src/
pages/ ← Full-page views (Home, GameScreen, SciencePage, TeacherDashboard, …)
components/ ← Reusable UI (TileDetectionView, GameHUD, FocusBurst, ScienceBadge, …)
hooks/ ← useComputerVision, useGameEngine, useWakeLock
lib/ ← audioSystem, trpc client
drizzle/
schema.ts ← Database tables (users, game_sessions, fact_mastery, classrooms, …)
server/
routers.ts ← tRPC procedures (auth, game.recognizeDigit, leaderboard, teacher, race)
gameDb.ts ← Game session & spaced repetition queries
teacherDb.ts ← Classroom & assignment queries
shared/
gameTypes.ts ← Shared types & adaptive level logic
achievements.ts ← Achievement definitions
Each frame capture follows this sequence:
- A 640×480 JPEG is drawn from the live
<video>element to an off-screen canvas - A brightness check skips blank or dark frames before sending
- The base64 image is posted to the
game.recognizeDigittRPC procedure - The server calls Gemini 2.0 Flash with a structured JSON schema requiring
{ digit: 0–99 | null, confidence: 0–1 } - The prompt includes explicit disambiguation rules (6 vs 9, 1 vs 7, 0 vs 8, etc.) and two-tile reading instructions
- The client validates the response: confidence ≥ 0.78, digit 0–99, two consecutive matching reads
- On confirmation,
onAutoSubmit(digit)fires — no button press required
The video element is created inside the useComputerVision hook and returned via videoRef to TileDetectionView for rendering. This V36 architecture is the proven pattern for iOS Safari camera stability.
- Node.js 22+
- pnpm 9+
- A MySQL-compatible database (TiDB, PlanetScale, or local MySQL)
- A Gemini API key (or Manus Forge API key)
git clone https://github.com/Theesamkos/osmo-clone.git
cd osmo-clone
pnpm installCopy .env.example to .env and fill in:
DATABASE_URL=
JWT_SECRET=
BUILT_IN_FORGE_API_KEY=
BUILT_IN_FORGE_API_URL=
VITE_FRONTEND_FORGE_API_KEY=
VITE_FRONTEND_FORGE_API_URL=
Apply database migrations by running the SQL files in drizzle/ in order, then start the dev server:
pnpm devThe app runs at http://localhost:3000.
pnpm test # Run all 46 tests
pnpm check # TypeScript check (0 errors)Live app: osmoclone-x8atjcun.manus.space
Best experienced on iPad in Safari. Tap Share → Add to Home Screen for the full PWA experience. A Guest mode is available — no account required.
- Landscape lock on game start (
screen.orientation.lock) - Science Moment pop-up after FocusBurst (citing the specific study)
- Classroom analytics export (CSV / PDF report cards)
- Parent push notifications for weekly digest
- Custom-trained on-device digit recognition model
MIT — see LICENSE for details.