Mandarin output is much worse than VibeVoice

[xqiu]

I tried to reference a 30 seconds stanard mandarin wav file, but the output sounds like a a dialect from some region and feels very strange, way worse than VibeVoice. Also, it takes about 2-3 times amount of gpu memory than VibeVoice (7B)

[makhlwf]

How much GPU memory did it take? I'm thinking of trying it on my 6 GB VRAM GPU, but I am concerned.

[deepblue96]

this model is trash and their bench are a complete lie, 1 takes around 24gb of vram, 2 it's slow as f, 3. the multi language is so bad that even the old default robotic tts is better.

[ghost-agent-250]

Also for multilingual check if you use the correct aligner and if its outputs look decent. It sounds fine in German for me but for some noisy prompt the alignment messed up and it sounded gibberish.

[ghost-agent-250]

This surely needs some optimization. The encoder part is heavy and it's loaded unnecessarily in tada.py. By preprocessing the prompt and removing the encoder in the main model file I could get the memory down to 9gb for the 3b model so 6g may be achievable for the 1b.
The speed is ok at 0.2 rtf for the 3b on an rtx 5090, I had to reduce the num flow matching steps down to 8, which actually sounds the same.

Edit: RTF is actually 0.096 now with torch compile for the 3b on rtx 5090, 10 flow matching steps. Quite nice! Will be sharing a PR soon. Benchmarking with this script

import os
import time
import torch
import torchaudio

from tada.modules.encoder import Encoder, EncoderOutput
from tada.modules.tada import InferenceOptions, TadaForCausalLM

device = "cuda"
PROMPT_CACHE = "prompt_cache.pt"
SAMPLE_RATE = 24000

audio_path = "samples/ljspeech.wav"
prompt_text = "The examination and testimony of the experts, enabled the commission to conclude that five shots may have been fired."

if os.path.exists(PROMPT_CACHE):
    state = torch.load(PROMPT_CACHE, map_location=device, weights_only=False)
    prompt = EncoderOutput(**state)
else:
    encoder = Encoder.from_pretrained("HumeAI/tada-codec", subfolder="encoder").to(device)
    audio, sample_rate = torchaudio.load(audio_path)
    audio = audio.to(device)
    prompt = encoder(audio, text=[prompt_text], sample_rate=sample_rate)
    torch.save(vars(prompt), PROMPT_CACHE)

for field in vars(prompt):
    v = getattr(prompt, field)
    if isinstance(v, torch.Tensor) and v.is_floating_point():
        setattr(prompt, field, v.to(torch.bfloat16))

model = TadaForCausalLM.from_pretrained(
    "HumeAI/tada-3b-ml",
    torch_dtype=torch.bfloat16,
).to(device)
model._decoder.to(torch.bfloat16)
model.compile()

torch.cuda.empty_cache()

torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()

with torch.inference_mode():
    # Warmup (must share inference_mode context with timed run)
    for i in range(2):
        model.generate(
            prompt=prompt,
            text="Please call Stella. Ask her to bring these things with her from the store.",
            inference_options=InferenceOptions(
                num_flow_matching_steps=8
            ),
        )

    torch.cuda.synchronize()
    t0 = time.perf_counter()
    output = model.generate(
        prompt=prompt,
        text="Please call Stella. Ask her to bring these things with her from the store.",
        inference_options=InferenceOptions(
            num_flow_matching_steps=10
        ),
    )
torch.cuda.synchronize()
elapsed = time.perf_counter() - t0

out_samples = output.audio[0].shape[-1]
out_duration_sec = out_samples / SAMPLE_RATE
rtf = elapsed / out_duration_sec
peak_gb = torch.cuda.max_memory_allocated() / 2**30
print(f"Peak GPU memory: {peak_gb:.2f} GB")
print(f"RTF: {rtf:.3f} ({elapsed:.2f}s gen / {out_duration_sec:.2f}s audio)")

torchaudio.save("output.wav", output.audio[0].cpu().float().unsqueeze(0), SAMPLE_RATE)

[srao25]

Thanks @ghost-agent-250
Yeah there were some optimization misses that we were testing from last week. Implemented most of the optimization that @ghost-agent-250 suggested and we made some more clean up, added instructions to readme.

@xqiu could you check the alignment for Mandarin prompt as mentioned in the readme? We noticed that it's easy to miss alignment issues with multilingual prompts, and this generally leads to inferior performance. But having said that we had <2-3% mandarin in our training mix. So we only offer tier 2 support for Mandarin at the moment.