Qwen3-TTS-12Hz-1.7B-Base SFT Fine-tuning Reproduction Report Successful Marie Models vs Failed Tremblay/Gagnon Models

[MinfangZhou]

1. Summary

I am testing Qwen3-TTS-12Hz-1.7B-Base SFT fine-tuning for French speaker customization.

The interesting phenomenon:

• The same base model
• The same GPU environment
• The same official SFT pipeline
• Similar dataset preparation process

produced:

Successful:

• Marie_QC_v1
• Marie_QC_v1_GOLD

Failed:

• Tremblay_QC_v1_train
• Gagnon_QC_v1_train
• Gagnon_QC_v2
• Gagnon_patch_test_e1

The failed models show:

• training loss decreases normally

• checkpoint saves successfully

• inference runs successfully

• but generated audio becomes:

  • severe noise
  • robotic artifacts
  • unstable waveform
  • sometimes unusable speech

---

2. Environment

Hardware

GPU:

NVIDIA GeForce RTX 4090D
24GB VRAM

OS:

Ubuntu 20.04

Python:

Python 3.10

CUDA:

CUDA 13 driver environment

Framework:

PyTorch
Transformers
Accelerate

Model:

Qwen3-TTS-12Hz-1.7B-Base

---

3. Dataset Preparation

All datasets were prepared into Qwen3-TTS SFT JSONL format.

Example:

{
 "audio":"audio/Gagon_chunk_00001.wav",
 "text":"...",
 "ref_audio":"audio/ref.wav",
 "speaker":"gagnon",
 "language":"French",
 "duration":8.837,
 "audio_codes":[...]
}

Audio:

• 24kHz
• mono
• WAV
• normalized

Preparation pipeline:

raw audio
    |
segmentation
    |
metadata jsonl
    |
prepare_data.py
    |
train_with_codes.jsonl

---

4. Successful Case #1: Marie_QC_v1

Training

Model:

Qwen3-TTS-12Hz-1.7B-Base

SFT:

official sft_12hz.py

Speaker:

Marie

Speaker slot:

3000

Result:

SUCCESS

Inference:

generate_custom_voice()

Output:

• stable speech
• correct pronunciation
• recognizable speaker identity
• no noise explosion

---

5. Successful Case #2: Marie_QC_v1_GOLD

A second Marie training result was also successful.

Same:

• base model
• training pipeline
• inference pipeline

Result:

SUCCESS

This is important because it proves:

• Qwen3-TTS architecture works
• SFT can work
• inference pipeline works
• hardware/environment are not the root problem

---

6. Failed Case #1: Tremblay_QC_v1_train

Training

Model:

Qwen3-TTS-12Hz-1.7B-Base

Dataset:

French speaker dataset

Training:

Official:

finetuning/sft_12hz.py

Epochs:

10

Checkpoints:

checkpoint-epoch-0
...
checkpoint-epoch-9

Size:

approximately:

4.3GB each checkpoint

Training behavior:

Normal.

Loss decreased.

No NaN.

No crash.

Inference:

Checkpoint loads successfully.

Speaker map:

speaker available

Generation works.

However:

Output audio quality:

FAILED

Symptoms:

• unstable voice
• artifacts
• incorrect acoustic output

---

7. Failed Case #2: Gagnon_QC_v1_train

Dataset

Prepared dataset:

1709 training samples

Audio:

~590MB wav

Reference:

ref.wav
24kHz mono
8.83 seconds

Metadata:

train_with_codes.jsonl

Generated successfully:

1709 lines
15MB

---

Training:

Official:

sft_12hz.py

Speaker:

Gagnon

Speaker ID:

3000

Epochs:

0-7

Training:

Normal.

Example:

Epoch 7
Loss decreasing

Checkpoint:

Saved successfully.

---

Inference:

Checkpoint loaded.

Speaker:

gagnon

Generation:

Successful execution.

But audio:

FAILED

Result:

• explosive noise
• unusable speech

---

8. Failed Case #3: Gagnon_QC_v2

I repeated training with a cleaned dataset and verified:

• reference audio
• metadata
• speaker name
• file paths

Training:

Successful.

Checkpoint:

Gagnon_QC_v2/checkpoint-epoch-4

Loss:

normal.

Inference:

Successful.

Audio:

FAILED

Output:

pure noise / explosion.

---

9. Additional Debugging

Reference Audio Verification

Checked:

dataset/Gagon_corpus/audio/ref.wav

dataset/gagnon_v1/audio/ref.wav

Both:

24kHz
mono

MD5 verification:

same file

So reference audio mismatch is unlikely.

---

10. Testing Official Script Modification

Based on community reports, I tested the missing text projection issue.

Original:

input_text_embedding = model.talker.model.text_embedding(input_text_ids)

Patched:

input_text_embedding = model.talker.model.text_embedding(input_text_ids)
input_text_embedding = model.talker.text_projection(input_text_embedding)

Training:

Gagnon_patch_test_e1

Parameters:

lr=2e-6
epoch=1

Training:

Normal.

Loss:

5.x-6.x

Checkpoint:

saved successfully.

---

Inference result:

FAILED.

Audio:

• severe distortion
• explosive noise

Conclusion:

The text_projection issue alone does not solve this problem.

---

11. Current Observations

The pattern:

Model	Training	Loss	Inference
Marie_QC_v1	OK	normal	SUCCESS
Marie_QC_v1_GOLD	OK	normal	SUCCESS
Tremblay_QC_v1	OK	normal	FAILED
Gagnon_QC_v1	OK	normal	FAILED
Gagnon_QC_v2	OK	normal	FAILED
Gagnon_patch_text_projection	OK	normal	FAILED

---

12. Important Finding

The problem is not:

• GPU
• CUDA
• model loading
• inference API
• reference audio format
• basic dataset preparation

Because Marie models work.

The failure happens after SFT.

Possible causes:

Hypothesis 1: Label shift issue

The official SFT script may have a mismatch between:

training forward pass

and

generation forward pass.

Possible impact:

• loss decreases
• codec prediction incorrect
• generated waveform destroyed

---

Hypothesis 2: Speaker embedding / codec embedding update

Official script:

speaker embedding
        |
codec_embedding.weight[3000]

is modified during training.

Possible problem:

• speaker slot injection
• checkpoint save
• inference lookup

---

Hypothesis 3: Dataset characteristics

Although datasets are valid:

Possible differences:

• speaker consistency
• acoustic distribution
• recording style
• duration distribution

may interact badly with current SFT implementation.

---

13. Questions for Qwen3-TTS Team

1. Is the current official sft_12hz.py fully compatible with inference generate_custom_voice()?

2. Are there known fixes for:

   • text_projection mismatch
   • label shifting
   • codec embedding speaker slot update?

3. Is full fine-tuning recommended for speaker customization?

4. Is LoRA fine-tuning supported/recommended for 1.7B Base?

5. Are there recommended hyperparameters for single-speaker French fine-tuning?

6. Why can the same pipeline successfully train Marie but fail with other speakers?

---

14. Reproduction Files Available

I can provide:

• dataset metadata
• training JSONL
• failed checkpoints
• successful checkpoint comparison
• training logs
• inference scripts

---

Final Summary

Qwen3-TTS-12Hz-1.7B-Base SFT can successfully create a custom French voice (Marie cases).

However, repeated SFT attempts with other French speakers consistently produce noisy / corrupted audio despite:

• successful training
• decreasing loss
• valid checkpoints
• successful inference execution

The issue appears to be related to the SFT training pipeline rather than the base model or inference system.

Further guidance from the Qwen3-TTS team would be appreciated.

[arpit763]

i faced 3 challenges -

1. merging the finetuned adapter on african language with base model
2. trying to finetune the newly merged model.
3. multi speaker finetuning.

i was able to solve the first one but i still dont know how can i fix the 2 and 3