Training Approach and Strategies for adding Catalan

[elaredo8]

Hello!

I am trying to teach Qwen3-TTS (1.7B version) Catalan language. My end goal is to create a Base-type model that may succesfully clone other Catalan voices and speaks the language without the heavy Spanish accent it introduces as-is.

After patching the critical bugs in the training code discussed in #179 , I have developed the following dataset and training conditions:

45 000 clips of multiple speakers (duration ranging between 2 and 29 seconds, with avg of 8.8s) :
90% of which is Catalan, obtained from https://huggingface.co/datasets/projecte-aina/parlament_parla
10% English, obtained from peoples_speech. Adding this English data aims to reduce the risk of catastrophic forgetting.

The structure is {"audio", "text", "ref_audio"}. ref_audio exists in 100% of the rows and I made sure, thanks to the speaker id tag in the parlament_parla dataset that every audio is referenced against one random other audio of the same speaker.

I am trying to run SFT on this dataset only to find that the Catalan pronunciation keeps being overridden by Spanish-like pronunciation. These are my training parameters: train_sft: batch_size: 2 lr: 0.00002 num_epochs: 5 num_workers: 16 clip_grad_norm: 1.0 gradient_accumulation_steps: 16 mixed_precision: bf16 # "no" | "fp16" | "bf16" num_processes: 2 weight_decay: 0.0001 sub_talker_loss_weight: 0.2 freeze_code_predictor: true

I would very much appreciate the help to get out of this block, there must be something I am missing, either with my dataset config or my training parameters. Should I introduce other languages, such as Spanish, French or Italian instead of English?

Another question would be, anyone have experience with cascading stage training? I see in this discussion from VibeVoice somebody had sucess with this strategy when implementing an Arabic dialect:
https://huggingface.co/Mohcinimohamed/vibevoice-asr-darija-lora

My two options right now are:

Strategy 1:
SFT in wide, Catalan + English dataset with 90/10 distribution. optionally, Spanish or other romance langs.

Strategy 2:
Three-Stage LoRa training, with a varying rate of Catalan presence in the dataset
Round 1: 70/30 Spanish/Catalan
Round 2: 30/70 Spanish/Catalan
Round 3: 0/100 Spanish/Catalan

Let me know if you have tried any of these strategies and what dataset sizes and training epochs you recommend for teaching this amazing model a new language.

[arpit763]

hi, can you tell me how were you able to merge the adapter so that you fine tuned it repeatedly ?

[danialzivehdadr]

✍️😂🤝 oft

[elaredo8]

not the answer you expected?

[danialzivehdadr]

WnI 🤔 ea bhtr ahtrak Uo mnzor

[danialzivehdadr]

Io 01 =!
a,s

[danialzivehdadr]

0x5= 0
01!
as ok
جواب کوانتم