mlx (Dart)

A monorepo of Dart packages for running MLX models on Apple Silicon, through Dart FFI bindings to mlx-c. It publishes two packages to pub.dev:

Package	Path	What it is
mlx	packages/mlx	Dart bindings for Apple's MLX array framework: tensors, ops, FFT, quantization, safetensors I/O, and an MLXNN-style neural-network layer (Module, layers, KV-cache).
mlx_audio	packages/mlx_audio	On-device speech models built on mlx: Parakeet-TDT and Whisper STT, PocketTTS TTS, plus transcribe/say CLIs.

mlx_audio depends on mlx. The rest of this README covers the audio stack; see each package directory for its own README and CHANGELOG.

This is a pure-Dart port of mlx-audio-swift. Milestone 1 targets Parakeet and Whisper speech-to-text (STT) and PocketTTS text-to-speech (TTS).

Setup

Consuming the published packages (from your own app):

dart pub add mlx          # or: dart pub add mlx_audio
dart run mlx:setup        # build the native mlx-c libs into a user cache, once

dart run mlx:setup provisions the native mlx-c artifacts into ~/Library/Caches/mlx-dart/<tag>/ (override the base with MLX_DART_CACHE); the loader finds them automatically afterwards — no environment variables. It needs git, cmake, and the Xcode command line tools and takes a couple of minutes the first time (idempotent thereafter; --force rebuilds).

Developing this repo:

make setup && make test

make setup builds the native mlx-c libraries into the in-repo packages/mlx/native/lib/ and resolves Dart dependencies; make test runs both packages' suites (network-free, under two minutes). Both build paths pin the same mlx-c tag — the single source of truth is the mlxcTag constant in packages/mlx/lib/src/ffi/native_version.dart.

Status

Works: Parakeet-TDT and Whisper speech-to-text, and PocketTTS text-to-speech, end-to-end.

• Parakeet-TDT loads the real mlx-community/parakeet-tdt-0.6b-v2 checkpoint (log-mel front-end → Conformer encoder → RNN-T predict/joint → TDT greedy decode) and transcribes audio.
• Whisper loads openai/whisper-* (HuggingFace layout, fp32), mlx-community/whisper-*-fp16 (OpenAI/mlx-whisper layout, fp16 safetensors), and quantized mlx-community/whisper-*-mlx-{4,8}bit (packed safetensors), with a mel front-end → encoder → greedy KV-cache decode and >30 s tiled chunking. Weight-only mlx-community repos (no tokenizer.json) get their tokenizer from the sibling openai/whisper-* repo automatically, chosen by vocab size. npz-only mlx-community whisper repos (e.g. mlx-community/whisper-tiny, which ships only weights.npz) are unsupported — the loader reads .safetensors only, the same limitation as the Swift reference.
• PocketTTS loads the real mlx-community/pocket-tts checkpoint (FlowLM backbone + flow-matching latent sampler → Mimi neural codec) and synthesizes a 24 kHz waveform from text with eight predefined voices. It reproduces the python oracle sample-for-sample under a fixed seed, and the closing end-to-end signal is a round-trip: PocketTTS synthesis → our own Parakeet ASR transcribes it back word-perfect (see test/e2e_tts_test.dart).

The STT models match the Swift reference CLI transcript on the 16 kHz fixtures and stay within a 2% word error rate on resampled audio (see the e2e suites).

Quantized-checkpoint support

Group-wise affine quantization (packed uint32 .weight + .scales/.biases, selected per layer by a .scales companion in the checkpoint) is wired into the model loaders via a construction-time QuantizationContext (see packages/mlx_audio/doc/PORT_NOTES.md for the mechanism vs. the Swift's post-load swap).

Model	Quantized loading	Proof
Whisper (4-/8-bit)	✅ (our extension — no upstream Swift path)	mlx-community/whisper-tiny-mlx-4bit transcribes false-turn.wav with a transcript identical to the python mlx-whisper oracle (weights-tagged e2e)
Parakeet-TDT	✅ infra-ready (Swift-faithful)	synthetic 4-bit fixture: strict packed bind + encoder/decode goldens vs. parakeet-mlx; a real-checkpoint e2e is deferred until an upstream quantized Parakeet ships
PocketTTS	❌ unsupported	a quantized repo is rejected up front with an actionable error pointing at the fp mlx-community/pocket-tts

Quick start

Transcribe a WAV with the CLI:

cd packages/mlx_audio
# Print the transcript to stdout (downloads the model on first run):
dart run mlx_audio:transcribe --audio path/to/audio.wav

# Or point --model at a local checkpoint directory, choose a format,
# and write to a file:
dart run mlx_audio:transcribe \
  --audio path/to/audio.wav \
  --model .cache/models/parakeet-tdt-0.6b-v2 \
  --format json --output-path out --verbose

# Whisper (multilingual): pass a whisper repo id or local dir, plus a
# --language hint (whisper-tiny has no auto-detect, so this is required for
# coherent output).
dart run mlx_audio:transcribe \
  --audio path/to/audio.wav --model openai/whisper-tiny --language en

--format is one of txt (default), json, srt, vtt. Without --output-path the transcript prints to stdout.

Programmatically:

import 'package:mlx_audio/mlx_audio.dart';

final model = await STT.loadModel('mlx-community/parakeet-tdt-0.6b-v2');
final audio = await loadAudioMono16k('audio.wav');
final out = model.generate(audio);
print(out.text);

Text-to-speech

Synthesize speech to a float32 WAV (24 kHz) with the say CLI:

cd packages/mlx_audio
# Downloads mlx-community/pocket-tts on first run, or point --model at a dir:
dart run mlx_audio:say --text "Hello from the Dart port." \
  --voice alba --output hello.wav --verbose

Voices (predefined): alba (default), azelma, cosette, eponine, fantine, javert, jean, marius. --verbose prints load/generate ms, frame count, RTFx, and peak memory.

--seed is a port extension. The Swift/python CLIs have no seed flag; MLX's RNG draws are bit-identical across bindings of the same core version, so passing --seed N makes a whole synthesis reproducible. Omit it for fresh output.

Programmatically:

import 'package:mlx_audio/mlx_audio.dart';

final tts = await TTS.loadModel('mlx-community/pocket-tts');
final out = tts.generate('Hello world.',
    voice: 'alba', params: const TtsGenerateParameters(seed: 42));
writeWavFloat32('hello.wav', out.samples, out.sampleRate);

Caveats. PocketTTS loads fp32/bf16 .safetensors only. The quantized mlx-community/pocket-tts-{4bit,8bit} repos are unsupported — they store packed (.weight uint32 + .scales/.biases) tensors and a quantization config block. TTS.loadModel detects that block and rejects it up front with an actionable error pointing at the fp mlx-community/pocket-tts (rather than failing deep in the load). Quantized loading is implemented for the STT families (Whisper, Parakeet) — see the support matrix above; extending it to PocketTTS is out of scope for now.

Tests

make test        # both packages, network-free
make test-e2e    # L4 transcript parity vs the Swift oracle (loads the checkpoint)
make analyze