Repository navigation
|
Hi @pnnbao97, Following our previous discussion about Style Control Tags, I’m interested in how we can technically prepare or contribute data to help the model learn these styles (e.g., Since VieNeu-TTS uses an LM-based architecture, I have a few specific questions regarding the data pipeline:
I’d love to understand the requirements so that the community could potentially help in gathering or labeling the right kind of data. Best regards, |
Replies: 1 comment
|
Hi @chucly112, Great follow-up question — this touches one of the most important aspects when moving LM-based TTS toward controllable and expressive speech. Below are some practical guidelines on how stylistic and non-verbal data can be prepared for VieNeu-TTS. ✅ 1. Annotation FormatFor VieNeu-TTS, style and non-verbal information should be embedded directly inside the transcript as special tokens, rather than stored in external metadata. Example: This approach works best because the model is trained as a speech language model predicting speech codes from phoneme/text tokens. External metadata files are generally not recommended unless used only for dataset organization. ✅ 2. Non-verbal Cues (laugh, cough, breath…)Non-verbal events do not require phoneme dictionary entries. Tokens such as: are treated as semantic control tokens. Important note: Example (good): Example (not recommended): The model mainly learns timing, interruption, and recovery patterns rather than standalone sounds. ✅ 3. Recommended Data Volume (Fine-tuning Stage)Approximate ranges that typically start showing stable separation:
Here, the 5h refers to total audio duration where events appear inside speech — not 5h of isolated effects. A healthy batch mixture is roughly:
Mixed samples (style + emotion + events together) are especially valuable. ✅ 4. Synthetic Stylistic DataSynthetic data can be useful only as augmentation, not as the primary supervision signal. Recommended usage:
However, authentic human recordings remain strongly preferred for stable prosody learning. ✅ Practical Contribution GuidelineIf the community would like to contribute stylistic datasets, the most helpful format would be: with natural conversational timing preserved. Expressive controllability (style, emotion, delivery behavior) is an active exploration direction for future VieNeu-TTS versions, and community contributions in this area are highly appreciated. Thanks again for raising this — it’s exactly the kind of discussion that helps move the project toward production-grade speech systems. Best regards, |
Hi @chucly112,
Great follow-up question — this touches one of the most important aspects when moving LM-based TTS toward controllable and expressive speech.
Below are some practical guidelines on how stylistic and non-verbal data can be prepared for VieNeu-TTS.
✅ 1. Annotation Format
For VieNeu-TTS, style and non-verbal information should be embedded directly inside the transcript as special tokens, rather than stored in external metadata.
Example:
This approach works best because the model is trained as a speech language model predicting speech codes from phoneme/text tokens.
Placing control signals inline a…