Live Avatar Set up

[karthiksreenivasanp]

🏷️ Discussion Type

Question

Body

I wanna make affordable avatar to communicate with customers with unlimited concurrent session. I have TTS, LLM, STT APIs but not know how to use it in a avatar. What type of file does make it lipsync to speech signals. What the Technique?
Hey gen like platform cost 99 dollars with unlimited concurrency. I wanted around 20 dollars rate with unlimited concurrency and sarvam ai integration to support multilingual or any multilingual support. Also can I may my own avatar ? What type of file is that make real time lipsync. Can I make I my PC by custom

Guidelines

• I have read and understood this category's guidelines before making this post.

[NavidZamanKhan]

You can definitely build your own avatar and integrate it with your existing STT, LLM, and TTS APIs without relying on expensive avatar platforms.

The important thing to understand is that lip-sync isn't controlled by a specific audio file format. It's usually handled through audio analysis or phoneme-to-viseme mapping (converting speech sounds into mouth shapes).

Here's how I'd approach it:

1. Create your avatar

For 3D, use Blender and export as .glb, .gltf, or .vrm with facial blendshapes.

For 2D, consider Live2D or a custom rigged avatar with mouth animations.

2. Implement lip-sync

You have two main options:

Audio amplitude analysis: Analyze the TTS audio and animate the mouth based on volume. Cheap and relatively simple, but less accurate.

Phoneme-to-viseme mapping: Use speech timing information to animate specific mouth shapes. More natural, but requires additional processing.

For prerecorded audio, you can also explore Rhubarb Lip Sync, which generates timed mouth-shape data.

3. Connect your APIs

The architecture could look like this:

User Speech → STT → LLM → TTS → Audio Playback + Lip-Sync Animation

Sarvam AI can handle the multilingual speech processing independently of your avatar renderer.

4. Handle concurrency and costs

This is where architecture matters most.

Instead of generating and streaming avatar video from your server, render the avatar directly in each customer's browser using Three.js or a similar technology.

That way, your server primarily handles API orchestration rather than expensive video rendering.

However, unlimited concurrent sessions for $20/month isn't realistic if STT, TTS, and LLM usage are included. Those services still have usage costs and rate limits.

My recommendation: Start with a browser-rendered 3D avatar, basic audio-driven lip-sync, and your existing APIs. Once the pipeline works, improve the lip-sync using phoneme timing and add multilingual optimizations.

You can absolutely prototype the avatar on your own PC. You don't need a dedicated GPU server just to animate a rigged avatar in the browser.

Hope this gives you a useful starting point!