# [Show and Tell] Local LoRA fine-tuning pipeline for Fish-Speech S2 Pro

[brickheadbs]

Hi everyone,

Here is a local training and fine-tuning pipeline for Fish-Speech S2 Pro written in MLX, co-authored with an AI developer crew (Fable 5, ChatGPT 5, Claude 4, and Gemini 3).

Official training is CUDA-only, but this project allows you to run the entire fine-tuning loop (from raw audio to merged weights) locally on Apple Silicon.

Key Implementation Details

• Dual-AR Causal Alignment: Fixes the off-by-one training target leak in prior ports by applying a causal shift (seq[:-1] predicting seq[1:]) to both slow and fast loss components.
• Trainable Slow Transformer: Unlocks LoRA training for the 36-layer slow transformer, enabling voice accent and prosody adaptation.
• Memory Management: Auto-configures batch sizes and cache limits based on unified memory size to prevent OS swapping.
• Objective Validation: Includes a validator script that generates test clips and computes MFCC similarity stats to score voice quality objectively.

Repository

The code is hosted publicly at: github.com/brickheadbs/mlx-fish-speech-finetuning

Feedback, optimization suggestions, and contributions are welcome.