v1.1.0

[ZachNagengast]

Highlights

• WhisperKit: incremental file loading for arbitrary-length audio.
• TTSKit: ~1.3× speedup end-to-end with improved models.
• SpeakerKit: speaker centroid embeddings are now available from a public interface, useful for matching speakers across separate diarization runs.
• Bug fixes for promptTokens, Chinese word timestamps, transcribeWithOptions, TTS chunk boundaries, and the model load/cache path.

Warning

TTSKit's default model variants changed (a one-time download on upgrade) and speech generation now requires macOS 15 / iOS 18. The previous assets remain supported - see Model Assets.

WhisperKit: Incremental File Loading

By default, WhisperKit decodes an entire audio file into memory before transcribing, which can lead to OOM situations for multi-hour audio files. .incremental streams it from disk in bounded-memory chunks instead:

let pipe = try await WhisperKit()

let results = try await pipe.transcribe(
    audioPath: "path/to/large-audio.wav",
    audioInputOptions: AudioInputOptions(audioLoadingMode: .incremental)
)

Chunks are cut at silence (VAD) boundaries, so the transcript matches a full-file run with chunkingStrategy: .vad - only peak memory differs. Tune with .incremental(chunkDuration:chunkBufferSize:) (defaults: 120s chunks, 2 chunks buffered at a time).

Also available via CLI:

swift run argmax-cli transcribe \
  --model large-v3-v20240930_626MB \
  --audio-path "path/to/large-audio.wav" \
  --incremental-loading

This will become a default option in a future release, so let us know how it works for you!

TTSKit: Faster Models

Both Qwen3-TTS decoders now ship as multifunction Core ML assets, which lets us load slightly different model forward pass configurations using the same set of weights.

SpeechDecoder (TTSKitConfig.speechDecoderMode): .latencyOptimized (default, 1 audio frame/call, ~80ms in length) for lowest time-to-first-audio, or .throughputOptimized (4 frames/call, ~320ms) to speed up throughput overall at the cost of a slower initial buffer.

MultiCodeDecoder (TTSKitConfig.multiCodeDecoderMode): a talker frame expands into 15 residual codes. .stepped (default) does that in 16 Core ML calls with host-side sampling; .fused does the whole frame in one call with sampling and lookups inside the graph.

Benchmarks in #494 and #513.

// Defaults: latency-optimized SpeechDecoder, stepped MultiCodeDecoder
let tts = try await TTSKit()

// Opt in
let config = TTSKitConfig(
    speechDecoderMode: .throughputOptimized,
    multiCodeDecoderMode: .fused
)
let fasterTTS = try await TTSKit(config)

Via CLI:

swift run -c release argmax-cli tts --text "Hello there." \
  --speech-decoder-mode throughputOptimized \
  --multi-code-decoder-mode fused --play

The mode is read once at model load since it's considered an entirely separate model to the OS. Set it before constructing TTSKit, or reload to switch at runtime. The two modes don't produce byte-identical audio, but it's audibly indistinguishable - fused sampling draws a different but equally valid sequence. The TTSKitExample app gains sidebar pickers for both modes.

The layout is detected from the asset at load time, so the legacy single-function W8A16 variants keep working (#520) - see Model Assets.

SpeakerKit: Speaker Centroid Embeddings

DiarizationResult now carries speakerCentroidEmbeddings: [Int: [Float]] which can be used to link speakers across separate diarization runs without re-running the embedder:

Within a single result - compare two local speaker ids:

let result = try await speakerKit.diarize(audioArray: multiSpeakerAudioFloats)
if let distance = result.centroidCosineDistance(between: 0, and: 1) {
    // Distance ranging from 0 - 2, where 0 is an exact match
    print("speakers 0 and 1: cosine distance \(distance)")
}

Across separate runs - each diarize(...) assigns its own local speakerIds (0, 1, …). To link speakers in a later chunk to an earlier one, take a centroid from the earlier result and pass it to nearestSpeakerCentroid(to:) on the later result:

let speakerKit = try await SpeakerKit()
let meeting1 = try await speakerKit.diarize(audioArray: meeting1Floats)
let meeting2 = try await speakerKit.diarize(audioArray: meeting2Floats)

// Centroid from an earlier run for speaker 0, can be nil
guard let speakerCentroidMeeting1 = meeting1.speakerCentroidEmbeddings[0] else { return } // handle no speakers found 

// Which meeting2 speaker sounds most like meeting1 speaker 0?
if let match = meeting2.nearestSpeakerCentroid(to: speakerCentroidMeeting1) {
    print("meeting1 speaker 0 most similar to meeting2 speaker \(match.speakerId) (distance \(match.distance))")
}

Centroids are in raw embedder space, so pick your own distance threshold that works best for your data.

Model Assets

TTSKit's default variants moved to multifunction assets:

Component	Before	After
SpeechDecoder	W8A16	W8A16-multifunction
MultiCodeDecoder	W8A16	W8A16-multifunction

Both are published on argmaxinc/ttskit-coreml and download automatically on your next setupModels() / first launch from huggingface. Expect a one-time download on upgrade if the models were downloaded previously, or:

Keeping the previous W8A16 assets

The decoders detect the model types at load time, so the legacy single-function variants keep working if you pin them to the pre-v1.1.0 variants:

let config = TTSKitConfig(
    speechDecoderVariant: "W8A16",
    multiCodeDecoderVariant: "W8A16"
)
let tts = try await TTSKit(config)

If the assets are already in your cache, this will use them instead of downloading the new models.

The legacy assets only implement the default modes. Requesting .throughputOptimized or .fused against them throws TTSError.invalidConfiguration naming the multifunction variant to use, rather than silently falling back.

New minimum OS for TTSKit

TTSKit speech generation now requires macOS 15, iOS 18, watchOS 11, or visionOS 2 — the decode path moved to MLTensor, and the pre-macOS 15 path was removed. This applies to both asset layouts, so pinning W8A16 doesn't avoid it; loadModels() throws TTSError.modelLoadingFailed on older OS versions. The package's declared platforms are unchanged (iOS 16 / macOS 13) and WhisperKit and SpeakerKit are unaffected - but a TTSKit app targeting iOS 17 will build and then fail at model load. We recommend available flags on your TTS entry points with #available(iOS 18, macOS 15, *), if you support iOS 17 or below.

API Changes

Deprecations

• AudioInputConfig -> AudioInputOptions (typealias kept).
• WhisperKit.audioInputConfig -> WhisperKit.audioInputOptions.
• WhisperKitConfig.audioInputConfig -> pass audioInputOptions per call to transcribe(...). The stored value still works as the instance default.

// before
let config = WhisperKitConfig(audioInputConfig: AudioInputConfig(channelMode: .sumChannels(nil)))
let pipe = try await WhisperKit(config)
let results = try await pipe.transcribe(audioPath: path)

// after
let pipe = try await WhisperKit()
let results = try await pipe.transcribe(
    audioPath: path,
    audioInputOptions: AudioInputOptions(channelMode: .sumChannels(nil))
)

Breaking changes

Callers of TTSKit are unaffected, but if you have a custom class that conforms to the SpeechDecoding protocol, the following changes are needed:

• decodeFrame(codes:cache:) / decodeFrameAsync(codes:cache:) take codes: [[Int32]] instead of [Int32] - an outer array of codesPerStep frames.
• New required codesPerStep: Int, read from the loaded model's audio_codes input shape.

New CLI flags

argmax-cli transcribe --incremental-loading
                      --incremental-chunk-duration <seconds>
                      --incremental-chunk-buffer-size <count>

argmax-cli tts --speech-decoder-mode      latencyOptimized|throughputOptimized
               --multi-code-decoder-mode  stepped|fused

Community

Big thanks to everyone who shipped code, filed issues, and dug into reproductions for this release. 🙏

• The fused MultiCodeDecoder proposed by @mjfrey's #506 landed, collapsing the frame into one call with in-graph Gumbel-max sampling, which gave a nice speedup to the QwenTTS pipeline.
• @leecrossley contributed speaker centroid embeddings (#463), closing a long-standing request for cross-run speaker matching, with unit and integration tests.
• @freecodetiger tracked down the NLLanguage normalization bug that had been breaking Chinese word timestamps for quite some time (#511).
• Thanks to @hakanensari, @yangzichao, @sborisov88, and @alan890104 for reproductions and investigation on the promptTokens empty-transcription bug (#514).

This is a big release so let us know how it goes in your testing. Open an issue or join us in Discord. 🚀

What's Changed

• chore: Pin GitHub Actions to commit SHAs by @pgoslatara in chore: Pin GitHub Actions to commit SHAs #426
• Expose speaker centroid embeddings on DiarizationResult by @leecrossley in Expose speaker centroid embeddings on DiarizationResult #463
• Update README with WhisperKit model recommendations by @atiorh in Update README with WhisperKit model recommendations #490
• Bump urllib3 for Python examples and scripts by @ardaatahan in Bump urllib3 for Python examples and scripts #441
• Support optimized multifunction SpeechDecoder for Qwen3-TTS by @EduardoPach in Support optimized multifunction SpeechDecoder for Qwen3-TTS #494
• Harden model load/cache, vendor transformers tests, fix macOS 14 segmenter by @a2they in Harden model load/cache, vendor transformers tests, fix macOS 14 segmenter #495
• Update idna in Python uv locks by @ardaatahan in Update idna in Python uv locks #496
• transcribeWithOptions: index per-element options globally, not per batch by @atiorh in transcribeWithOptions: index per-element options globally, not per batch #512
• fix: normalize NLLanguage code so Chinese hits the no-space word split path by @freecodetiger in fix: normalize NLLanguage code so Chinese hits the no-space word split path #511
• Support incremental file loading by @a2they in Support incremental file loading #507
• Fix empty transcription when promptTokens are set by @a2they in Fix empty transcription when promptTokens are set #514
• Use Unicode sentence segmentation for TextChunker boundaries by @ZachNagengast in Use Unicode sentence segmentation for TextChunker boundaries #515
• Support multifunction MultiCodeDecoder for Qwen3-TTS (stepped + fused) by @EduardoPach in Support multifunction MultiCodeDecoder for Qwen3-TTS (stepped + fused) #513
• Detect MultiCodeDecoder dimensions per multifunction schema by @EduardoPach in Detect MultiCodeDecoder dimensions per multifunction schema #521
• Support the legacy single-function SpeechDecoder and MultiCodeDecoder assets by @EduardoPach in Support the legacy single-function SpeechDecoder and MultiCodeDecoder assets #520

New Contributors

• @pgoslatara made their first contribution in chore: Pin GitHub Actions to commit SHAs #426
• @leecrossley made their first contribution in Expose speaker centroid embeddings on DiarizationResult #463
• @ardaatahan made their first contribution in Bump urllib3 for Python examples and scripts #441
• @freecodetiger made their first contribution in fix: normalize NLLanguage code so Chinese hits the no-space word split path #511
• @EduardoPach made their first contribution in Support multifunction MultiCodeDecoder for Qwen3-TTS (stepped + fused) #513

Full Changelog: v1.0.0...v1.1.0

---

This discussion was created from the release v1.1.0.