One picture in. A live, lip-synced VTuber out.
Avatar Studio turns a single character image — or a prompt, or a 3D VRM model — into a streaming-ready VTuber avatar that lip-syncs to your microphone, entirely in the browser. No GPU required to run it, no capture plumbing: the runtime renders in an OBS browser source.
Open alpha: https://vtuber.jagthehero.com — sign in with X and start building.
Instead of baking mouth motion into generated video (choppy loops, canned talking), the avatar is decomposed:
base loops (mouth-neutral) ⊕ viseme mouth sprites = final frame
pregenerated, smooth picked per-frame by
real-time audio analysis
- Design — generate portrait candidates (Grok Imagine, billed to your own xAI key), upload your own art, or import a VRM model
- Animate — chained idle clips per emotion set (neutral, happy, sad, angry, surprised), generated or uploaded
- Mouths — a full viseme sprite set in the character's own art style, inpainted, rendered from the VRM, or hand-drawn
- Build — mouth tracking across every frame, chroma key, loop-cycle detection, sprite alignment → one portable bundle
- Go live — the web runtime picks a mouth shape per audio frame (MFCC viseme classification + energy-driven openness) and composites it onto the idle loops; a mic-relay page feeds OBS browser sources
The VRM path is fully AI-free: portraits, expressions, per-emotion mouth sets and idle sway clips are rendered straight from the model's own blendshapes, with exact per-frame mouth anchors projected from the rig.
| Path | What |
|---|---|
server/ |
FastAPI backend: auth (X OAuth + per-user Grok OAuth), job queue, avatar/asset access control |
web/ |
React wizard (Vite) |
runtime/ |
browser runtime: bundle renderer, MFCC lip-sync, mic relay |
tools/ |
pipeline stages: generation, tracking, viseme synthesis, bundle import, VRM rendering |
avatars/ |
the public demo cast (user avatars are private server data, never committed) |
deploy/ |
k8s/tunnel deployment |
Design docs: the bundle format is specified by the importer/runtime pair
(tools/import_bundle.py, runtime/bundle.js).
PRs welcome — from pipeline improvements to new emote animations. Start with CONTRIBUTING.md (5-minute dev setup, test gates, PR flow).
Third-party notices: the runtime vendors MediaPipe Tasks (Apache-2.0) for camera-driven emotion tracking; VRM rendering uses three.js and @pixiv/three-vrm (MIT).
MIT — see LICENSE.