Avatar Studio

One picture in. A live, lip-synced VTuber out.

Avatar Studio turns a single character image — or a prompt, or a 3D VRM model — into a streaming-ready VTuber avatar that lip-syncs to your microphone, entirely in the browser. No GPU required to run it, no capture plumbing: the runtime renders in an OBS browser source.

Open alpha: https://vtuber.jagthehero.com — sign in with X and start building.

This is a fork of JagTheHero/VTuberAvatarStudio with added Docker support for easy self-hosting (especially on OMV / NAS).

Docker / Self-hosting (recommended)

See DOCKER.md for full instructions.

Quick start:

git clone https://github.com/Manya3084/VTuberAvatarStudio.git
cd VTuberAvatarStudio
mkdir -p data/server data/avatars
cp .env.example data/server/.env
# edit data/server/.env (set STUDIO_DEV_LOGIN=0 + X OAuth keys for prod)
docker compose up -d --build

Then open http://your-server:8330.

How it works

Instead of baking mouth motion into generated video (choppy loops, canned talking), the avatar is decomposed:

base loops (mouth-neutral)  ⊕  viseme mouth sprites  =  final frame
     pregenerated, smooth        picked per-frame by
                                 real-time audio analysis

1. Design — generate portrait candidates (Grok Imagine, billed to your own xAI key), upload your own art, or import a VRM model
2. Animate — chained idle clips per emotion set (neutral, happy, sad, angry, surprised), generated or uploaded
3. Mouths — a full viseme sprite set in the character's own art style, inpainted, rendered from the VRM, or hand-drawn
4. Build — mouth tracking across every frame, chroma key, loop-cycle detection, sprite alignment → one portable bundle
5. Go live — the web runtime picks a mouth shape per audio frame (MFCC viseme classification + energy-driven openness) and composites it onto the idle loops; a mic-relay page feeds OBS browser sources

The VRM path is fully AI-free: portraits, expressions, per-emotion mouth sets and idle sway clips are rendered straight from the model's own blendshapes, with exact per-frame mouth anchors projected from the rig.

Repo layout

Path	What
server/	FastAPI backend: auth (X OAuth + per-user Grok OAuth), job queue, avatar/asset access control
web/	React wizard (Vite)
runtime/	browser runtime: bundle renderer, MFCC lip-sync, mic relay
tools/	pipeline stages: generation, tracking, viseme synthesis, bundle import, VRM rendering
avatars/	the public demo cast (user avatars are private server data, never committed)
deploy/	original k8s/tunnel deployment
Dockerfile + docker-compose.yml	New — production Docker setup

Design docs: the bundle format is specified by the importer/runtime pair (tools/import_bundle.py, runtime/bundle.js).

Contributing

PRs welcome — from pipeline improvements to new emote animations. Start with CONTRIBUTING.md (5-minute dev setup, test gates, PR flow).

Third-party notices: the runtime vendors MediaPipe Tasks (Apache-2.0) for camera-driven emotion tracking; VRM rendering uses three.js and @pixiv/three-vrm (MIT).

License

MIT — see LICENSE.