Training support for mlx-audio — closing the inference-only gap on Apple Silicon

[akashicMarga]

Hey @Blaizzy and contributors 👋

Opening this as an RFC to gauge interest and get design input on something I think could meaningfully expand what the mlx-audio ecosystem enables.

The gap
mlx-audio has become the definitive audio inference library for Apple Silicon — TTS, STT, STS, voice cloning, streaming, the works. But the story ends at inference. If you want to fine-tune a TTS model on your own voice, adapt an ASR model to a domain, or train a custom audio codec — there's no established MLX path for any of it.
This matters more now than it did a year ago. M3/M4 chips with 64–128GB unified memory can realistically run fine-tuning workloads that would have required a cloud GPU not long ago. The hardware is there. The inference stack is there. Training is the missing layer.
The mlx-lm trajectory is instructive here: it started inference-only, then mlx-lm train and LoRA support got formalized, and that's now what most people reach for when they want to fine-tune an LLM on Apple Silicon. Audio deserves the same arc.

What I've been building: mlx-audio-train
To explore this concretely, I've been working on an open-source LoRA fine-tuning pipeline for audio models on Apple Silicon, starting with Qwen3-TTS (a natural first target given it's already a flagship model in mlx-audio).
Current state:

LoRA adapter training for Qwen3-TTS via MLX
Dataset preparation and preprocessing for TTS fine-tuning
Checkpointing and adapter export compatible with mlx-audio inference
Native M-series execution, no CUDA dependency

This is early but functional, and Qwen3-TTS is just the starting point. The broader scope I'm thinking about:

Fine-tuning for other TTS models already in mlx-audio (Orpheus, Fish Audio S2, etc.)
STT fine-tuning (Qwen3-ASR, Parakeet) for domain adaptation
Multilingual voice adaptation — particularly relevant for low-resource languages
QLoRA / quantization-aware training for tighter memory budgets on smaller devices

What I'm not proposing (yet)
I'm deliberately leaving the integration model open. Some options that have crossed my mind — a mlx_audio.train subpackage living in this repo, a sibling repo analogous to mlx-audio-swift, an officially linked external project, or something else entirely. Each has tradeoffs around maintenance surface, versioning, and how tightly training and inference need to share internals.
I can open a PR for a README link as a no-commitment starting point while the bigger question gets figured out. Mostly wanted to surface this and hear what the project maintainers and community think before building further in isolation.
— Akash
🔗 mlx-audio-train

[akashicMarga]

@Blaizzy i have added parler tts support and training for mlx. i can raise parler tts pr here.