A 3D game development IDE powered by AI. Think Replit/Scratch but for Three.js games — with AI coding, 3D model generation, sound effects, and voice interaction.
- AI Coding Assistant — Chat with Claude Haiku 4.5 (thinking, tool-calling) to build and modify Three.js games. Uses sandboxed tools:
edit_file,create_file,bash,read_file, and more. - 3D Sprite Generation — 2-step flow: Flux generates a 2D preview (~5s), you confirm, then Meshy.ai generates the 3D model (~60s). Add PBR textures later with one click (~80s).
- AI Sound Effects — Generate sound effects with ElevenLabs (
generate_sound). Gunshots, footsteps, ambient — saved as MP3, playable in the asset panel. - Live Preview — Game runs in a Docker-served iframe with auto-refresh on code changes. Open in new tab for full-screen play.
- Docker Sandbox — All AI code execution happens inside a sandboxed container. No network, limited resources. The AI can't touch your machine.
- Voice Brainstorm — Talk to an ElevenLabs conversational agent to brainstorm game ideas. When ready, it calls
buildto send the idea to the coding AI. - Accounts & Projects — Email/password auth, multiple projects per user, chat history saved to SQLite.
- SSE Streaming — Real-time tool call visibility with OpenCode-style terminal blocks, color-coded diffs, and progress indicators.
- Sprite Preview & Confirm — See a 2D Flux preview before committing to 3D generation. Expand to full size, confirm or deny.
- Console Capture — Game errors/logs forwarded to the AI via
get_console_logsfor self-debugging. - Resizable Panels — Drag the chat panel border to resize.
┌──────────────────────────────────┐ ┌─────────────────┐
│ Game Panel │ │ Chat Panel │
│ (iframe → Docker:3000) │ │ Haiku 4.5 │
│ │ │ + Tool Calling │
│ Three.js Game │ │ + Voice (11Labs)│
├──────────────────────────────────┤ │ + Image Paste │
│ Sprites & Sounds Panel │ │ + Stop Button │
│ (3D assets + audio + confirm) │ │ │
└──────────────────────────────────┘ └─────────────────┘
│ │
│ Express Server (:8080) │
│ ┌──────────────────────────┐ │
└────│ /api/p/:id/chat (SSE) │────┘
│ /api/p/:id/sprites │
│ /api/p/:id/sounds │
│ /api/auth/* │
│ /api/voice-agent │
└──────────┬───────────────┘
│
┌──────────┴───────────────┐
│ Docker Sandbox │
│ python:3.12-slim │
│ Serves game on :3000 │
│ /workspace (volume) │
└──────────────────────────┘
- Node.js 18+
- Docker (via Colima or Docker Desktop)
- API keys for: Meshy.ai, ElevenLabs, NanoGPT
git clone https://github.com/phuang/GameVibes.git
cd GameVibes
npm install
cp .env.example .env # Add your API keysMESHY_API_KEY=your_meshy_key
ELEVENLABS_API_KEY=your_elevenlabs_key
NANOGPT_API_KEY=your_nanogpt_key
# If using Colima:
colima start --cpu 1 --memory 2
# Build sandbox image:
docker build -t gamevibes-sandbox ./sandboxnpx ts-node src/server.ts- Sign up with email/password
- Create a new project
- Ask the AI to build a game: "Build a simple platformer with a blue sky and green ground"
- The AI writes Three.js code, the game appears in the preview
- Iterate: "Add clouds", "Make the player jump higher", "Generate a dragon sprite"
| Tool | Description |
|---|---|
read_file(path) |
Read files in workspace |
edit_file(path, old, new) |
Targeted string replacement (shows color-coded diff) |
create_file(path, content) |
Create/overwrite files |
list_files(path) |
List workspace files |
bash(command) |
Run shell commands (no network) |
generate_sprite(name, desc) |
Generate 3D model: Flux 2D preview → confirm → Meshy 3D |
generate_sound(name, desc, dur) |
Generate AI sound effect via ElevenLabs |
list_sprites |
See available 3D assets with GLB paths |
get_console_logs |
See JS errors/logs from the running game |
refresh_game |
Reload the game preview |
- AI calls
generate_sprite("dragon", "cute kawaii dragon with wings") - Flux generates a 2D preview image (~5 seconds)
- Preview appears in sprite panel with ✓ Use / ✗ Skip buttons
- Click image to expand full size
- ✓ Use → Meshy generates untextured 3D model (~60s) → immediately usable
- Click 🎨 Add Textures later → PBR refine (~80s) → full quality
Ask the AI to add sounds: "Add a gunshot sound when I shoot"
- AI calls
generate_sound("gunshot", "tactical rifle shot, reverb", 0.5) - ElevenLabs generates MP3 (~1 second)
- Sound appears in the asset panel with play button
- AI wires it into the game:
new Audio("/models/gunshot.mp3").play()
- Click 🔊 in the chat panel
- Speak your game idea to the ElevenLabs agent
- Brainstorm back and forth via voice
- Say "build it" → agent calls
buildtool → coding AI takes over
- Frontend: Vanilla JS, CSS (OpenCode-inspired dark theme)
- Backend: Express.js, TypeScript
- AI Coding: NanoGPT API (Claude Haiku 4.5 thinking, tool-calling)
- 3D Generation: Meshy.ai (meshy-6, preview + optional refine)
- 2D Preview: NanoGPT Flux (flux-schnell, ~5s per image)
- Sound Effects: ElevenLabs Sound Generation API
- Voice: ElevenLabs Conversational AI (@11labs/client SDK)
- Sandbox: Docker (python:3.12-slim, serves game on :3000)
- Database: SQLite (users, projects, chat history, sprites)
- Auth: JWT + bcrypt
- Rendering: Puppeteer (headless 6-angle sprite screenshots)
GameVibes/
├── public/
│ ├── ide.html # IDE shell (3-panel layout)
│ ├── ide.css # OpenCode-inspired dark theme
│ ├── ide.js # Frontend logic (auth, chat, sprites, voice)
│ ├── render.html # Headless 6-angle renderer
│ ├── models/ # Downloaded GLBs + generated MP3s
│ └── sprites/ # Sprite render screenshots
├── src/
│ ├── server.ts # Express server + all API endpoints
│ ├── nanogpt-client.ts # NanoGPT API client with tool definitions
│ ├── sandbox.ts # Docker sandbox manager
│ ├── auth.ts # JWT auth (signup/login)
│ ├── db.ts # SQLite schema + connection
│ └── projects.ts # Project/sprite/chat CRUD
├── workspace/
│ └── SKILL.md # AI system prompt + game framework template
├── workspaces/ # Per-project workspace directories
├── sandbox/
│ └── Dockerfile # Sandbox container image
├── data/
│ └── gamevibes.db # SQLite database
└── .env # API keys (not committed)
MIT