Streaming RealTime Demo

English | 简体中文

This browser demo generates talking-head video in real time. Animate an image using uploaded audio or a live microphone, dub a source video while preserving its pose and timing, or connect a chat agent for interactive speech-driven animation. Preset habit IDs let you switch between speaking styles without a reference video.

On an NVIDIA RTX 3090, the demo achieves 30+ FPS generation throughput while using less than 4 GB of GPU memory, with high-clarity facial details.

The source release includes the inference code, portrait runtime, and frontend. Model checkpoints are installed separately under the local checkpoints/ directory; runtime code never reads files from another repository.

Watch the demo video

The interface provides three modes:

• Audio animates an image from an uploaded audio file or live microphone.
• Video dubbing replaces the speech motion of a source video while retaining its original pose and timing.
• Chat agent connects text or recorded speech to an OpenAI-compatible chat/STT service and streams synthesized speech through the avatar.

All three modes share cached avatar preprocessing and the same streaming renderer. Audio and Chat agent accept image avatars; Video dubbing accepts video avatars and does not require a motion-reference or habit-reference file.

Linux, an NVIDIA GPU, and FFmpeg are required. The tested environment uses Python 3.10, CUDA 12.1, and cuDNN 9.

conda env create -f environment.yaml
conda activate talklikeyou-demo

Alternatively, install a CUDA-compatible PyTorch build and then run:

pip install -r requirements.txt

cd TalkLikeYou-Streaming-RealTime-Demo
CUDA_VISIBLE_DEVICES=0 python app.py --host 0.0.0.0 --port 5070

Open http://127.0.0.1:5070. Uploading a source starts one-time avatar registration; subsequent runs reuse the cached features. Image sources are used by Audio and Chat agent, while video sources are used by Video dubbing. No motion-reference file is required for a video source.

Select Audio, upload an image avatar, then choose an audio file or Live microphone. Once registration finishes, click Start Streaming. Recommended preset habit IDs are 192, 166, and 202.

Select Video dubbing, upload a source video, choose an audio file, and start streaming. The source video's pose and timing are retained while its speech motion is replaced. The bundled data/neutral.pkl motion shell supplies the neutral mouth expression; no neutral reference image is required.

Configure the services below, select Chat agent, upload an image avatar, and enter text or record speech. The response text, synthesized speech, and avatar frames are streamed incrementally. Browser speech recognition may be used when available; otherwise the configured STT endpoint handles recorded audio.

The equivalent launcher is:

CUDA_VISIBLE_DEVICES=0 ./run.sh

The command-line client covers Audio and Video dubbing:

python demo_client.py --source path/to/source.jpg --audio path/to/audio.wav
python demo_client.py --source path/to/source.mp4 --audio path/to/audio.wav

data/neutral.pkl stores only the pre-extracted portrait motion shell and expression. It replaces the previous neutral reference image, avoids an extra face-detection pass during registration, and can be overridden with --neutral_motion or the NEUTRAL_MOTION environment variable.

Runtime uploads and avatar caches are stored under /tmp/talklikeyou_demo by default. Set TALKLIKEYOU_STORAGE_DIR to choose another location.

Edge TTS works after installing the requirements. Chat and server-side speech recognition use OpenAI-compatible endpoints. Export the variables you need before starting the server, or place them in a local .env file:

export TALKLIKEYOU_LLM_BASE_URL=https://example.com/v1
export TALKLIKEYOU_LLM_API_KEY=your_key
export TALKLIKEYOU_LLM_MODEL=qwen-flash

export TALKLIKEYOU_STT_BASE_URL=https://example.com/v1
export TALKLIKEYOU_STT_API_KEY=your_key
export TALKLIKEYOU_STT_MODEL=whisper-1

An OpenAI-compatible TTS service can replace Edge TTS with TALKLIKEYOU_TTS_PROVIDER=openai_compatible, TALKLIKEYOU_TTS_BASE_URL, TALKLIKEYOU_TTS_API_KEY, TALKLIKEYOU_TTS_MODEL, and TALKLIKEYOU_TTS_VOICE.

The paper-release checkpoints are hosted on Hugging Face. This demo expects the following local layout:

checkpoints/
├── motion/
│   ├── audio_encoder.pth
│   ├── lip_motion.pt
│   └── pose_motion.pt
├── runtime/
│   ├── aux_models/
│   └── models/
└── runtime_config.pkl

• checkpoints/motion/: audio encoder and habit-conditioned lip/pose generators.
• checkpoints/runtime/: real-time portrait renderer, landmarks, and audio features.
• checkpoints/runtime_config.pkl: local relative-path runtime configuration.

Download only the shared paper assets and the two demo-specific motion checkpoints, then arrange them in the local layout above:

hf download doubi-killer/TalkLikeYou \
  --include "checkpoints/audio_encoder.pth" \
            "checkpoints/ditto/ditto_pytorch/**" \
            "checkpoints/motion/lip_motion.pt" \
            "checkpoints/motion/pose_motion.pt" \
  --local-dir hf_assets

mkdir -p checkpoints/motion checkpoints/runtime
cp hf_assets/checkpoints/audio_encoder.pth checkpoints/motion/
cp hf_assets/checkpoints/motion/lip_motion.pt checkpoints/motion/
cp hf_assets/checkpoints/motion/pose_motion.pt checkpoints/motion/
cp -a hf_assets/checkpoints/ditto/ditto_pytorch/. checkpoints/runtime/

runtime_config.pkl is included in the source release, so it does not need to be downloaded separately.

The local paper repository and this demo were compared checkpoint by checkpoint:

The demo-specific lip_motion.pt and pose_motion.pt are now available in the same Hugging Face repository. The paper model's motion_generator.pt remains a separate model and must not be renamed or substituted for lip_motion.pt.

The default paths can be overridden with --lip_ckpt, --pose_ckpt, --data_root, and --cfg_pkl. Do not rename motion_generator.pt to lip_motion.pt; their model definitions and parameters differ.

The portrait renderer incorporates components adapted from Ditto, and portrait preprocessing incorporates components adapted from LivePortrait. We thank both teams for releasing their work. See Third-Party Notices for the applicable licenses.

The demo code is released under the MIT License. Third-party code and model files retain their original terms; see THIRD_PARTY_NOTICES.md.

For questions, contact wangbaiqin0511@gmail.com.