Nemotron 3.5 Transcribe: bump mlx-audio to v0.6.4 and expose speaker attribution + MTP on /v1/audio/transcriptions

[felsnerino]

Summary

mlx-audio v0.6.4 (released 2026-11-02) adds two features for Nemotron 3.5 Transcribe
(Neywa-Labs/mlx-audio#1092): word timestamps from the CTC head and speculative decoding with the MTP
weights that ship in the official NVIDIA checkpoints. The model also does speaker attribution
(task="saa"). oMLX 0.7.3 pins mlx-audio 0.6.3 (8c244a1), which drops the llm.mtp.* tensors in
sanitize and has no word timestamps for this model. And /v1/audio/transcriptions has no way to ask
for speaker attribution.

I'd like to run Nemotron 3.5 Transcribe with speakers in oMLX instead of a separate server. Today I have
to run a second copy of the model (GGUF, outside oMLX) just for speakers and word timings.

Requests

1. Bump mlx-audio to ≥ v0.6.4 (e0833581 or later). Right now this happens together with mlx-vlm
   bumps (e.g. feat: bump mlx-vlm to 138d761 (v0.7.11) and support Unlimited-OCR-2 #4934), so it could ride along with the next one if a separate bump is too much.
2. Speaker attribution on /v1/audio/transcriptions. For example an oMLX extension field
   diarize: bool = Form(False) (or a generic task) that maps to model.generate(..., task="saa") for
   backends that support it, and is ignored for the others, the same way prompt is. The speaker field
   that mlx-audio puts in each word segment should then show up in the response segments.
   _normalize_segment already keeps dict keys, so this may mostly be the form field plus passing it
   through.
3. MTP for Nemotron Transcribe. generate(..., speculative=True, num_draft_tokens=4), ideally as a
   per-model setting (like vlm_mtp_enabled) so clients don't need to know about it.

word_timestamps already passes through to generate() on the non-streaming path, so with v0.6.4 that
part should work for Nemotron as is.

Measurements (mlx-audio 0.6.4 in a separate venv, not in oMLX)

Apple M5 Ultra, macOS 27.0.1. Model converted with mlx_audio.convert -q --q-bits 6 from
nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B BF16 (26 GB, keeps llm.mtp.* and ctc_timestamp.*).
Five German test recordings (28 s to 6 min), task="saa", temperature 0:

	without MTP	with MTP (num_draft_tokens=4)
34 s dialog	1.0 s	0.7 s
266 s	7.0 s	4.7 s
352 s	7.7 s	4.8 s
draft acceptance	—	78–98 % (≥ 96 % on long audio)

WER was lower than with the GGUF IQ4_NL we run today on 4 of 5 files (e.g. 33.8 % vs 64.2 % on the
352 s file) and slightly higher on one short file (8.9 % vs 7.1 %). With word timestamps and
task="saa", speaker time attribution was 95–99 % on 4 of 5 files, the same as the GGUF run, and better
on the fifth (52 % vs 38 %).

Environment

oMLX 0.7.3 (Homebrew formula jundot/omlx/omlx), bundled mlx-audio 0.6.3, mlx 0.32.4, mlx-vlm 0.7.11.

Happy to test a branch.