Skip to content

v1.3 — LoRA Voice Training & Voice Designer Integration

Choose a tag to compare

View all tags
@Finrandojin Finrandojin released this 09 Feb 22:13
· 130 commits to main since this release
v1.3
72bf9e2

What's New

Voice Design Voice Type

Characters can now use the VoiceDesign model directly for on-the-fly voice generation. Set a base voice description per character (e.g., "Young strong soldier") and each line's instruct is appended as delivery/emotion direction. Ideal for minor or throwaway characters that don't need a persistent voice identity.

  • 5 voice types — Custom, Clone, LoRA, Voice Design, and saved designed voices as clone references
  • Base description + per-line instruct combined into VoiceDesign prompt
  • Sequential in batch mode (each line may have a unique voice)

LoRA Voice Training

Train custom voice identities by fine-tuning the Base model with LoRA adapters.

  • Training tab in the web UI with dataset upload, training configuration, live progress, and adapter management
  • Generate datasets directly from Voice Designer descriptions — emotion+text pair rows for emotionally diverse training data
  • Dataset form persistence — sample rows, name, and description saved to localStorage across page reloads
  • Reference audio — first generated sample saved as ref.wav for consistent speaker embedding during training
  • Standalone training script (train_lora.py) aligned with official Qwen3-TTS fine-tuning approach
  • LoRA inference with instruct passthrough — trained voices respond to style/emotion prompts
  • Character style support — persistent style traits (e.g., "heavy Scottish accent") appended to every instruct, same as Custom voices
  • Batch generation routing — LoRA voices handled correctly alongside Custom and Clone voices
  • Settings info panel explaining how each training parameter affects voice quality

Training Alignment with Official Qwen3-TTS

  • Sub-loss weighting at 0.3x (matching official sft_12hz.py)
  • Speaker embedding extracted from consistent reference audio via mel spectrogram, not per-sample
  • No instruct conditioning during training (matching official approach — instruct is only used at inference time)
  • Dataset format supports both Alexandria (audio_filepath) and official (audio, ref_audio) field names

Batched Inference for Clone & LoRA Voices

Clone and LoRA voices now use the Base model's native list-based batch API instead of sequential per-chunk generation. Chunks are grouped by speaker (clone) or adapter (LoRA), sorted by text length, and sub-batched using the same length-ratio strategy as custom voices.

  • 11x speedup — LoRA voices go from ~0.5 RTF (sequential) to ~5.5 RTF (batched)
  • Clone voices get the same batching benefit when multiple chunks share a speaker

Voice Designer Integration

  • Designed voices can now be used as training data sources for LoRA fine-tuning
  • Clone voice dropdown populated from saved Voice Designer outputs

Wiki Documentation

Comprehensive documentation now available at the project wiki:

Bug Fixes

  • Fix VRAM fragmentation hanging batch generation when running a second batch — gc.collect() + torch.cuda.empty_cache() before first sub-batch
  • Fix race condition when loading a saved script during active audio generation — returns 409 error instead of crashing
  • Bounds-check chunk indices after reload to prevent IndexError if chunks change mid-generation
  • Fix Windows file-locking race condition — atomic JSON writes now retry with exponential backoff when os.replace() fails due to Windows file locking during parallel TTS generation. Temp file cleanup also retries. Based on fix by @ElBiggus (969d563)

API Endpoints

New endpoints for LoRA training workflow:

  • POST /api/lora/upload_dataset — Upload training dataset (ZIP)
  • POST /api/lora/generate_dataset — Generate dataset from Voice Designer description
  • GET /api/lora/datasets / DELETE /api/lora/datasets/{id} — Manage datasets
  • POST /api/lora/train — Start training (subprocess, auto-unloads TTS models)
  • GET /api/lora/models / DELETE /api/lora/models/{id} — Manage trained adapters
  • POST /api/lora/test — Test a trained adapter with custom text/instruct

Requirements

  • peft package required for LoRA training (install with pip install peft --no-deps to avoid replacing ROCm torch)
Assets 2
Loading