A browser-based Human-Machine Interface for a simulated UR5 robot arm — built as a portfolio artifact demonstrating the full Intrinsic Flowstate engineering stack.
ScreenRecording2026-04-22at10.14.38PM-ezgif.com-video-to-gif-converter.mov
When you run the app, two processes start: a React frontend and a Python robot simulator.
The simulator runs a fake UR5 robot — it moves the arm's 6 joints in a continuous idle motion and broadcasts the joint positions to the browser 30 times per second over WebSocket as Protobuf binary frames. The browser decodes each frame, computes the TCP pose locally via a Rust/WASM forward kinematics solver, and applies the angles directly to the 3D arm with no React re-renders in between — staying smooth at 60 FPS.
Clicking a skill block (e.g. "Pick stock") sends a command to the Python backend. The backend uses roboticstoolbox-python inverse kinematics to solve the arm's joint angles for the target Cartesian pose, then plays back a multi-phase trajectory (approach → grasp → lift → place) at 100 Hz. The block glows blue while the motion runs.
The telemetry panel shows all 6 joint angles, TCP position (computed in-browser via WASM FK), gripper state, and cycle count — updating live at 30 Hz.
Faults fire automatically every ~45 seconds, or you can trigger one manually. When a fault occurs, a panel slides up from the bottom of the screen, the affected joint pulses with a GLSL shader (breathing red glow + rim lighting, running entirely on the GPU), and you can click Diagnose — which calls the Claude API and streams back a structured diagnosis (likely causes, recommended actions, confidence level). Clicking Acknowledge & Reset clears it and returns the arm to home.
Kill the backend and the arm freezes on its last position, the status indicator turns amber, and the UI shows a "Reconnecting…" banner. The frontend retries automatically with exponential backoff. Restart the backend and it reconnects on its own.
Reload the page with no network and the app shell still loads from the service worker cache — the UI comes up with an "Offline" banner rather than a blank browser error.
Intrinsic (Google's AI robotics group) builds Flowstate — a web-based IDE and digital twin for industrial robots. Their frontend role requires TypeScript, React, Three.js, real-time state management, gRPC/Protobuf, and offline-first PWA architecture.
Flowdeck is a compressed, weekend-scoped implementation of the operator-facing HMI layer: a live 3D digital twin of a UR5 arm on the factory floor, where an operator can watch the robot run, dispatch skill commands, and get AI-assisted fault diagnosis — all in the browser.
Every feature maps directly to a bullet in the JD:
| JD requirement | Where it lives |
|---|---|
| TypeScript + React | frontend/src/ — strict mode throughout |
| Three.js / WebGL | Viewer3D.tsx, RobotArm.tsx, WorkcellScene.tsx |
| Real-time state management | Zustand store + direct THREE.Object3D mutation at 30 Hz |
| WebSocket + Protobuf | FastAPI WS + Protobuf binary frames (proto/robot.proto) |
| Service workers / offline-first | Vite PWA plugin (Workbox), ?debug=1 FPS overlay |
| AI agent integration | Claude API streaming diagnosis modal |
| Python backend | FastAPI + asyncio robot simulator + IK via roboticstoolbox-python |
| WebAssembly + Rust | wasm-fk/ Rust crate — FK solver compiled to WASM, runs in browser |
| GLSL shaders | faultHighlight.ts — pulsing fault glow + rim lighting on GPU |
┌──────────────────────┐ WebSocket ┌──────────────────────┐
│ Browser (React) │◄─── joint states ─────────│ Python FastAPI │
│ │ (Protobuf, 30 Hz) │ │
│ Three.js / R3F │ │ asyncio robot sim │
│ Behavior tree UI │──── skill commands ───────►│ IK trajectory player│
│ Telemetry panel │ │ fault injector │
│ Fault panel │ └──────────┬───────────┘
│ Diagnosis modal │◄────── SSE stream ──────────────────┘
│ WASM FK solver │ /diagnose endpoint Anthropic API (Claude)
│ GLSL fault shader │
│ Service worker │
└──────────────────────┘
Key architectural decisions:
- Direct
THREE.Object3Dmutation inuseFrame(not React state) for joint updates — avoids React reconciler overhead at 30 Hz. The Zustand store is read viagetState()(notuseStore()) insideuseFrameto prevent re-renders. - Protobuf binary WebSocket over gRPC-Web — typed wire format with
proto/robot.proto; in production I'd layer gRPC-Web on top for bidirectional streaming and HTTP/2 multiplexing. - TCP pose computed in browser via Rust/WASM FK — the backend only sends raw joint angles; the browser runs the DH-parameter chain locally with zero network round-trip.
- GLSL fault shader — fault highlight is a GPU program (
uTimeuniform + rim lighting), not a CPU material swap. Zero JS overhead at 60 FPS. - Vite over Bazel — right-sized for portfolio scope. A production Intrinsic deployment would use Bazel for hermetic builds and remote caching across a monorepo.
| Layer | Tech | Why |
|---|---|---|
| Language | TypeScript (strict) | JD requires TS; strict mode catches real bugs at the boundary |
| Framework | React 18 | JD lists React; widest ecosystem for 3D + testing |
| Build | Vite | Fast DX; Bazel migration path documented above |
| 3D engine | Three.js + React Three Fiber | JD calls out Three.js; R3F makes 3D composable in React |
| Shaders | GLSL (ShaderMaterial) | GPU-side fault highlight — pulsing glow + rim lighting via uTime uniform |
| State | Zustand | Minimal boilerplate; getState() pattern avoids render overhead at 30 Hz |
| Wire format | Protobuf (proto/robot.proto) |
Typed binary frames; matches gRPC schema discipline from JD |
| WASM | Rust + wasm-bindgen | UR5 FK solver in browser — DH-parameter chain, zero network round-trip |
| IK | roboticstoolbox-python | Cartesian target → joint angles; multi-phase pick-and-place trajectories |
| Styling | Tailwind CSS | Fast iteration; consistent design tokens |
| Offline | Vite PWA (Workbox) | JD: "service workers / offline-first architecture" |
| Testing | Vitest + RTL | Co-located with Vite; no separate Jest config |
| Backend | FastAPI + asyncio | JD: "read Python backend code"; built-in WebSocket; async sim loop |
| AI | Anthropic Claude | Best-in-class reasoning for fault diagnosis; SSE streaming |
| Package mgr | uv | Fast, modern Python tooling |
# Clone and install
git clone https://github.com/suhaasteja/Flowdeck.git && cd Flowdeck
npm install && cd backend && uv sync && cd ..
# Terminal 1 — frontend
npm run dev
# Terminal 2 — backend
cd backend && uv run python server.pyOptional: set ANTHROPIC_API_KEY before starting the backend for real AI diagnosis. Without it, a structured mock response is returned.
The UR5 arm is driven entirely by WebSocket state — no local animation. Joint angles from the backend are applied directly to THREE.Group rotations in useFrame, bypassing React for zero-overhead 30 Hz updates.
Clicking a skill block sends a SkillCommand over WebSocket. The backend interpolates a pre-baked trajectory at 100 Hz and streams the motion back. The active block highlights until the trajectory completes.
Six joint angles (rad + °), TCP pose (x/y/z + rx/ry/rz), gripper state, cycle count — all updating at 30 Hz via direct Zustand store writes without triggering React re-renders in unrelated components.
Skills are defined as Cartesian target poses, not hand-tuned joint angles. The backend solves inverse kinematics via roboticstoolbox-python and plays back a multi-phase trajectory (approach → grasp → lift → move → place) at 100 Hz. Gripper state syncs through the Protobuf stream.
All WebSocket frames are encoded with proto/robot.proto — generated TypeScript and Python classes via scripts/gen-proto.sh. No JSON on the wire; binary frames are visible in DevTools → Network → WS.
TCP pose is computed in the browser by a Rust function compiled to WebAssembly (wasm-fk/). The backend only sends raw joint angles — the browser runs the UR5 DH-parameter chain locally at 30 Hz with zero network round-trip. FK compute time logs to console at ~0.08ms.
When a fault fires (randomly every ~45s, or via curl -X POST localhost:8000/admin/inject-fault):
- The affected joint pulses with a GLSL shader — breathing red glow + rim lighting, entirely on the GPU
- A fault panel slides up with code, message, and timestamp
- Clicking Diagnose streams a structured Claude response (likely causes → recommended actions → confidence)
- Acknowledge & Reset clears the shader and returns the arm to home
The app shell, JS chunks, and CSS are precached via Workbox on first load. Killing the backend or going offline: the app still loads from cache with an appropriate banner. The robot's last known state remains visible.
Add ?debug=1 to the URL to show a live FPS counter driven by useFrame via direct DOM mutation — no React re-renders.
Measured on MacBook Air M2 (Chrome 124):
| Metric | Value |
|---|---|
| Steady-state FPS | 60 FPS |
| WebSocket message rate | 30 Hz |
| JS bundle (gzipped) | 275 KB app + 175 KB Three.js |
| WS parse + store write latency | < 1 ms per frame |
| Cold load time (preview build) | ~1.2s |
The Three.js chunk (683 KB uncompressed) is the main bundle cost — unavoidable for a WebGL app. In production: LOD meshes, Draco-compressed URDF geometry, and URDF mesh precaching would bring perceived load time down significantly.
# Frontend (21 tests)
cd frontend && npx vitest run
# Backend (34 tests)
cd backend && uv run pytest tests/ -vCI runs on every push via GitHub Actions (.github/workflows/ci.yml): typecheck → unit tests → build (frontend) and ruff → pytest (backend).
| Feature | Why |
|---|---|
| gRPC-Web transport | Layer gRPC-Web on top of the existing Protobuf schema — HTTP/2 multiplexing, bidirectional streaming, browser-native cancellation |
| Real UR5 URDF + STL meshes | urdf-loader integration is scaffolded; blocked by xacro dependency in the ROS URDF package |
| Bazel migration | Hermetic builds, remote caching, shared proto/ package across frontend and backend without shell scripts |
| ROS 2 driver | Swap the Python sim for a real ros2_control interface — the Protobuf WebSocket protocol stays identical |
| Scene tree → 3D highlight | Clicking a node in the right panel selects the corresponding THREE.Object3D |
| "Ask follow-up" in Diagnosis Modal | Multi-turn Claude conversation anchored to the active fault context |
- gkjohnson/urdf-loaders — de-facto standard for URDF in Three.js (used by NASA JPL)
- Foxglove Studio — architectural reference for real-time robotics HMI in the browser
- ros-industrial/universal_robot — UR5 URDF and kinematic parameters
- Intrinsic Flowstate — the product this is inspired by