Relay

A real-time multimodal lab assistant for learning electronics.

Built for the Gemini Live Agent Challenge — Live Agents category.

Relay watches your physical workspace through a webcam and guides you through electronics builds with live voice and vision. Instead of reading tutorials, you talk to Relay while it sees your breadboard, components, and wiring — correcting mistakes as they happen.

How it works

1. Pick a build from the library (e.g. Light-Activated Alarm)
2. Show your components — Relay verifies inventory via camera
3. Wire step-by-step — Relay guides and corrects in real time
4. Generate Arduino code — Relay writes and debugs sketches
5. Run and validate — Relay watches serial output and confirms
6. Get a 3D digital twin of the completed build in the sandbox

No hardware? Demo Mode lets judges experience the full live session without an Arduino. Turn on your camera and hold up any object — Relay identifies it in real time.

What's in the app

Tab	What it does
Build	Live workspace — camera feed, voice interaction, step-by-step guidance
Learn	AI-generated adaptive quizzes, drawing exercises with Gemini Vision assessment, wrong-answer requeue, and review history
Sandbox	3D breadboard workspace — place components, edit wires, write Arduino code in Monaco, run circuit validation and simulation with buzzer audio
Community	Share builds, react, comment — persisted via Firestore
Library	Starter projects with 3D Twin presets for the sandbox

Multi-model architecture

One voice session, multiple models working behind the scenes:

Role	Model
Live tutor (voice + vision)	gemini-2.5-flash-native-audio
Quick checks (component ID)	gemini-2.5-flash
Code generation (Arduino)	gemini-2.5-pro
Deep reasoning (debugging)	gemini-2.5-pro
Drawing assessment (vision)	gemini-2.5-pro

Architecture

┌─────────────────────────────────────────────────────┐
│                   Relay Frontend                     │
│        React + TypeScript + Three.js + Monaco        │
│                                                      │
│  ┌──────────┐ ┌──────────┐ ┌────────┐ ┌───────────┐ │
│  │  Build   │ │  Learn   │ │Sandbox │ │ Community  │ │
│  │(Live AI) │ │(Quizzes) │ │  (3D)  │ │  (Feed)   │ │
│  └────┬─────┘ └────┬─────┘ └────────┘ └───────────┘ │
│       │             │                                │
│  WebSocket     HTTP POST          Firestore REST     │
│       │             │                   │            │
└───────┼─────────────┼───────────────────┼────────────┘
        │             │                   │
   ┌────▼────┐   ┌────▼─────┐    ┌───────▼──────┐
   │  Live   │   │  Quiz    │    │   Firestore  │
   │ Bridge  │   │ Engine   │    │  (relay-     │
   │(Cloud   │   │(Cloud    │    │   gemini)    │
   │  Run)   │   │  Run)    │    └──────────────┘
   └────┬────┘   └────┬─────┘
        │              │
   ┌────▼────┐    ┌────▼─────┐
   │  ADK    │    │ Gemini   │
   │  Bidi   │    │ Flash    │
   │Streaming│    │(genai)   │
   └────┬────┘    └──────────┘
        │
   ┌────▼──────────────────┐
   │   Gemini Models       │
   │  (Vertex AI)          │
   │                       │
   │  • Flash Native Audio │
   │  • Flash 2.5          │
   │  • Pro 2.5            │
   └───────────────────────┘

Tech stack

• Frontend: React 18, TypeScript, Vite, Tailwind CSS
• 3D: Three.js via @react-three/fiber + @react-three/drei
• Code editor: Monaco Editor
• Live backend: FastAPI + Google ADK (bidirectional WebSocket streaming)
• Quiz backend: FastAPI + google-genai SDK
• Persistence: Firestore REST API
• Deployment: Google Cloud Run
• GCP Project: relay-gemini

Google Cloud services used

• Cloud Run — live-bridge and quiz-engine as independent microservices
• Vertex AI — Gemini model access via service account (no API key)
• Firestore — user state, quiz history, community data
• Google ADK — agent framework for bidirectional audio/video streaming

Local development

Prerequisites

• Node.js 18+
• Python 3.13 (not 3.14 — pydantic-core wheels are missing)
• Google Cloud SDK (for deployment)

1. Frontend

npm install

Create .env.local:

VITE_RELAY_API_BASE_URL=http://localhost:8082
VITE_RELAY_WS_URL=ws://localhost:8082
VITE_RELAY_QUIZ_API_BASE_URL=http://localhost:8083
VITE_RELAY_DEFAULT_USER_ID=faith

# Firestore persistence (optional)
VITE_FIREBASE_API_KEY=your_firebase_web_api_key
VITE_FIREBASE_PROJECT_ID=relay-gemini

npm run dev

Opens at http://localhost:3000. The app workspace is at /relay-app.

2. Live bridge backend

cd backend/live-bridge
python3.13 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
PYTHONPATH=app uvicorn app.main:app --host 0.0.0.0 --port 8082 --reload

Environment variables:

GOOGLE_GENAI_USE_VERTEXAI=TRUE
GOOGLE_CLOUD_PROJECT=relay-gemini
GOOGLE_CLOUD_LOCATION=europe-west1
RELAY_LIVE_MODEL=gemini-live-2.5-flash-native-audio
RELAY_FLASH_MODEL=gemini-2.5-flash
RELAY_PRO_MODEL=gemini-2.5-pro
RELAY_REASONING_MODEL=gemini-2.5-pro

If not using Vertex AI locally, set GOOGLE_API_KEY instead.

3. Quiz engine backend

cd backend/quiz-engine
python3.13 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 8083 --reload

Google Cloud deployment

Deployment is automated via deploy.sh:

./deploy.sh              # Deploy all services (live-bridge + quiz-engine + frontend build)
./deploy.sh live-bridge  # Deploy live-bridge only
./deploy.sh quiz-engine  # Deploy quiz-engine only
./deploy.sh frontend     # Build frontend only
./deploy.sh verify       # Health-check deployed services

gcp_proof.mp4

The script handles project configuration, deploys both backend microservices to Cloud Run, builds the frontend, and verifies service health — all in a single command.

Configuration is via environment variables (defaults to production):

export GOOGLE_CLOUD_PROJECT=relay-gemini   # GCP project ID
export GOOGLE_CLOUD_REGION=europe-west1    # Cloud Run region

Manual deployment (if preferred):

gcloud run deploy relay-live-bridge \
  --source=backend/live-bridge \
  --region=europe-west1 \
  --allow-unauthenticated \
  --set-env-vars=GOOGLE_GENAI_USE_VERTEXAI=TRUE,GOOGLE_CLOUD_PROJECT=relay-gemini,GOOGLE_CLOUD_LOCATION=europe-west1,RELAY_LIVE_MODEL=gemini-live-2.5-flash-native-audio,RELAY_FLASH_MODEL=gemini-2.5-flash,RELAY_PRO_MODEL=gemini-2.5-pro,RELAY_REASONING_MODEL=gemini-2.5-pro

gcloud run deploy relay-quiz-engine \
  --source=backend/quiz-engine \
  --region=europe-west1 \
  --allow-unauthenticated \
  --set-env-vars=GOOGLE_GENAI_USE_VERTEXAI=TRUE,GOOGLE_CLOUD_PROJECT=relay-gemini,GOOGLE_CLOUD_LOCATION=europe-west1

Build verification

npm run build                                          # Frontend (must pass)
python3 -m py_compile backend/live-bridge/app/main.py  # Backend syntax check
python3 -m py_compile backend/quiz-engine/app/main.py

Gemini API session limits

The Gemini Live API imposes per-session limits:

• 15 minutes for audio-only sessions
• 2 minutes with video active

These are hard API limits. Ending a session and starting a new one gives fresh limits. The UI surfaces these limits so users understand disconnections are API-imposed, not bugs.

Third-party libraries

• React, Vite, Tailwind CSS (MIT)
• Three.js, @react-three/fiber, @react-three/drei (MIT)
• Monaco Editor (MIT)
• Lucide React icons (ISC)
• FastAPI, Pydantic, Uvicorn (MIT/BSD)
• Google ADK, google-genai (Apache 2.0)

Known limitations

• The sandbox validates circuit structure (component presence, wiring, connections) but does not compute voltages/currents like a SPICE simulator
• Firestore persistence requires a configured Firebase web API key
• Gemini Live API imposes hard session limits (15 min audio, 2 min video)

Links

• Blog: I Built a Real-Time AI Lab Partner for Electronics Learning with Gemini Live
• GDG Profile: developers.google.com/profile/u/faithogundimu
• Deployment automation: deploy.sh