fix: KokoroAne zh-CN noise reduction via atan2 phase correction by Alex-Wengg · Pull Request #569 · FluidInference/FluidAudio

Alex-Wengg · 2026-05-03T23:12:02Z

Summary

Documents the regenerated KokoroNoise.mlmodelc now live on HuggingFace at FluidInference/kokoro-82m-coreml/ANE/, which eliminates audible high-frequency background noise that was most noticeable on Mandarin (zh) synthesis.

No Swift code changes are required — the fix is entirely in the model conversion.

Root cause

CoreMLForwardSTFT in mobius/models/tts/kokoro/laishere-coreml/convert-coreml.py reconstructed phase from real/imaginary parts in a way that produced an audible HF noise floor on the synthesizer's source signal. The noise survived every weight-quantization, compute-precision, and dispatch-mode change tried during conversion, which is why it shipped in the first KokoroAne release.

Fix

mobius PR #50 replaces the manual phase reconstruction with atan2(imag, real) so the inverse-STFT phase is computed correctly. The output KokoroNoise.mlmodelc keeps the same:

filename (KokoroNoise.mlmodelc)
inputs (F0_curve fp32, style_timbre fp32)
outputs (x_source_0, x_source_1)
shapes / dtypes

so the existing KokoroAneSynthesizer Noise stage call site at KokoroAneSynthesizer.swift:108-113 consumes the regenerated model unchanged.

Distribution

The fixed KokoroNoise.mlmodelc is now uploaded to FluidInference/kokoro-82m-coreml/ANE/. Fresh installs pick it up automatically on first KokoroAneManager.initialize().

Cache invalidation for existing users

KokoroAneResourceDownloader.ensureModels only checks file existence, not version/etag/hash, so users who ran KokoroAne before the fix landed will keep using the stale local copy. The fix for them is to delete the cached bundle:

# macOS
rm -rf ~/.cache/fluidaudio/Models/kokoro-82m-coreml/ANE/KokoroNoise.mlmodelc

# iOS — under <App caches>/fluidaudio/Models/kokoro-82m-coreml/ANE/

The next initialize() will re-download the corrected file.

Changes

Documentation/TTS/KokoroAne.md — new "Model Updates" section explaining the atan2 phase correction and the cache-invalidation step for existing users.

Test plan

Confirm no Swift source changes in this PR (git diff main --stat)
Spot-check the rendered Markdown on GitHub for broken links
Sanity: regenerate locally with cache cleared, listen to a zh sample before/after

Documents the regenerated `KokoroNoise.mlmodelc` from mobius PR #50, which fixes audible high-frequency background noise (most noticeable on zh voices) via an atan2 phase correction in `CoreMLForwardSTFT`. Model I/O contract is unchanged so no Swift update is needed. Adds a cache-invalidation note for users who ran KokoroAne before the fix — the downloader does file-existence checks only, so a stale `KokoroNoise.mlmodelc` won't be re-fetched without manual deletion.

github-actions · 2026-05-03T23:17:18Z

PocketTTS Smoke Test ✅

Check	Result
Build	✅
Model download	✅
Model load	✅
Synthesis pipeline	✅
Output WAV	✅ (142.5 KB)

_{Runtime: 0m21s}

_{Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon.}

github-actions · 2026-05-03T23:17:23Z

Kokoro TTS Smoke Test ✅

Check	Result
Build	✅
Model download	✅
Model load	✅
Synthesis pipeline	✅
Output WAV	✅ (634.8 KB)

_{Runtime: 0m46s}

_{Note: Kokoro TTS uses CoreML flow matching + Vocos vocoder. CI VM lacks physical ANE — performance may differ from Apple Silicon.}

github-actions · 2026-05-03T23:21:27Z

Qwen3-ASR int8 Smoke Test ✅

Check	Result
Build	✅
Model download	✅
Model load	✅
Transcription pipeline	✅
Decoder size	571 MB (vs 1.1 GB f32)

Performance Metrics

Metric	CI Value	Expected on Apple Silicon
Median RTFx	0.06x	~2.5x
Overall RTFx	0.06x	~2.5x

_{Runtime: 3m25s}

_{Note: CI VM lacks physical GPU — CoreML MLState (macOS 15) KV cache produces degraded results on virtualized runners. On Apple Silicon: ~1.3% WER / 2.5x RTFx.}

github-actions · 2026-05-03T23:22:45Z

ASR Benchmark Results ✅

Status: All benchmarks passed

Parakeet v3 (multilingual)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.57%	0.00%	5.76x	✅
test-other	1.56%	0.00%	3.66x	✅

Parakeet v2 (English-optimized)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.80%	0.00%	5.66x	✅
test-other	1.00%	0.00%	3.26x	✅

Streaming (v3)

Metric	Value	Description
WER	0.00%	Word Error Rate in streaming mode
RTFx	0.67x	Streaming real-time factor
Avg Chunk Time	1.386s	Average time to process each chunk
Max Chunk Time	1.619s	Maximum chunk processing time
First Token	1.619s	Latency to first transcription token
Total Chunks	31	Number of chunks processed

Streaming (v2)

Metric	Value	Description
WER	0.00%	Word Error Rate in streaming mode
RTFx	0.58x	Streaming real-time factor
Avg Chunk Time	1.641s	Average time to process each chunk
Max Chunk Time	2.524s	Maximum chunk processing time
First Token	1.786s	Latency to first transcription token
Total Chunks	31	Number of chunks processed

_{Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming}

_{25 files per dataset • Test runtime: 5m45s • 05/03/2026, 08:00 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

github-actions · 2026-05-03T23:23:19Z

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric	Value	Target	Status	Description
DER	15.1%	<30%	✅	Diarization Error Rate (lower is better)
JER	24.9%	<25%	✅	Jaccard Error Rate
RTFx	18.68x	>1.0x	✅	Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage	Time (s)	%	Description
Model Download	9.796	17.4	Fetching diarization models
Model Compile	4.198	7.5	CoreML compilation
Audio Load	0.057	0.1	Loading audio file
Segmentation	16.844	30.0	Detecting speech regions
Embedding	28.073	50.0	Extracting speaker voices
Clustering	11.229	20.0	Grouping same speakers
Total	56.172	100	Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method	DER	Notes
FluidAudio	15.1%	On-device CoreML
Research baseline	18-30%	Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

M2 MacBook Air (2022): Runs at 150 RTFx real-time
Performance scales with Apple Neural Engine capabilities

_{🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 56.1s diarization time • Test runtime: 2m 31s • 05/03/2026, 07:53 PM EST}

github-actions · 2026-05-03T23:25:12Z

VAD Benchmark Results

Performance Comparison

Dataset	Accuracy	Precision	Recall	F1-Score	RTFx	Files
MUSAN	92.0%	86.2%	100.0%	92.6%	606.3x faster	50
VOiCES	92.0%	86.2%	100.0%	92.6%	635.8x faster	50

Dataset Details

MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

✅: Average F1-Score above 70%

github-actions · 2026-05-03T23:27:30Z

Parakeet EOU Benchmark Results ✅

Status: Benchmark passed
Chunk Size: 320ms
Files Tested: 100/100

Performance Metrics

Metric	Value	Description
WER (Avg)	7.03%	Average Word Error Rate
WER (Med)	4.17%	Median Word Error Rate
RTFx	11.12x	Real-time factor (higher = faster)
Total Audio	470.6s	Total audio duration processed
Total Time	42.8s	Total processing time

Streaming Metrics

Metric	Value	Description
Avg Chunk Time	0.043s	Average chunk processing time
Max Chunk Time	0.086s	Maximum chunk processing time
EOU Detections	0	Total End-of-Utterance detections

_{Test runtime: 0m48s • 05/03/2026, 07:47 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O}

github-actions · 2026-05-03T23:28:56Z

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Metric	Value	Target	Status	Description
DER	10.4%	<20%	✅	Diarization Error Rate (lower is better)
RTFx	13.85x	>1.0x	✅	Real-Time Factor (higher is faster)

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Stage	Time (s)	%	Description
Model Download	11.485	15.2	Fetching diarization models
Model Compile	4.922	6.5	CoreML compilation
Audio Load	0.042	0.1	Loading audio file
Segmentation	21.208	28.0	VAD + speech detection
Embedding	75.563	99.7	Speaker embedding extraction
Clustering (VBx)	0.093	0.1	Hungarian algorithm + VBx clustering
Total	75.769	100	Full VBx pipeline

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Method	DER	Mode	Description
FluidAudio (Offline)	10.4%	VBx Batch	On-device CoreML with optimal clustering
FluidAudio (Streaming)	17.7%	Chunk-based	First-occurrence speaker mapping
Research baseline	18-30%	Various	Standard dataset performance

Pipeline Details:

Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
Segmentation: VAD-based voice activity detection
Embeddings: WeSpeaker-compatible speaker embeddings
Clustering: PowerSet with VBx refinement
Accuracy: Higher than streaming due to optimal post-hoc mapping

_{🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 96.9s processing • Test runtime: 1m 39s • 05/03/2026, 07:41 PM EST}

github-actions · 2026-05-03T23:30:37Z

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Metric	Value	Target	Status
DER	30.3%	<35%	✅
Miss Rate	28.2%	-	-
False Alarm	0.9%	-	-
Speaker Error	1.2%	-	-
RTFx	7.9x	>1.0x	✅
Speakers	4/4	-	-

_{Sortformer High-Latency • ES2004a • Runtime: 3m 37s • 2026-05-03T23:44:39.126Z}

Chronological log covering THCHS-30 benchmark, error analysis, beam search A/B, Cohere encoder fix, the Kokoro v1.1-zh atan2 noise root-cause investigation, the Devin review audit, and PRs landed.

## Summary Phase 1 — variant plumbing + phonemes-bypass synthesis for Kokoro-82M-v1.1-zh on the existing 7-stage CoreML chain. Callers that supply pre-computed Bopomofo (e.g. via misaki[zh] in Python or a future Swift G2P) can now synthesize Mandarin audio. Mandarin text-to-Bopomofo G2P is deferred to a separate Phase 2 PR. The 7-stage chain is **language-agnostic by construction** — input ids, voice slices, and per-stage I/O contracts are identical across v1.0 (English) and v1.1-zh (Mandarin). Only the embedding vocab (177 → 171), the HF subdir (`ANE/` → `ANE-zh/`), the voice-file layout (flat → `voices/<voice>.bin`), and the default voice (`af_heart` → `zf_001`) differ. ## Changes - New `Repo.kokoroAneZh` → `FluidInference/kokoro-82m-coreml/ANE-zh` with `subPath = ANE-zh`, `folderName = kokoro-82m-coreml/ANE-zh`. - `ModelNames.KokoroAne.requiredModelsZh` references `voices/zf_001.bin` so the downloader's all-files-present check resolves correctly when the file lands at `<repoDir>/voices/zf_001.bin`. - New `KokoroAneVariant` enum (`.english` / `.mandarin`) with `defaultVoice`, `useVoicesSubdir`, and `repo` accessors. - `KokoroAneResourceDownloader.ensureModels` and `ensureVoicePack` accept a `variant` param (default `.english` keeps existing callers source-compatible). Mandarin voice fetch creates the `voices/` parent directory on demand. - `KokoroAneModelStore` and `KokoroAneManager` thread the variant through to download + load. - `KokoroAneManager.synthesize(text:)` and `synthesizeDetailed(text:)` reject Mandarin with a clear error directing callers to `synthesizeFromPhonemes()`. The phonemes-bypass entry point already works for any vocab via `vocab.encode → 7-stage chain`. - CLI `--variant` flag accepts `en` / `english` / `zh` / `mandarin` for the `kokoro-ane` backend. Mandarin runs treat the input text as pre-computed Bopomofo and call `synthesizeFromPhonemesDetailed`. - 12 new unit tests (`KokoroAneVariantTests`): variant defaults, repo wiring, required-files set routing, manager init signatures, and Mandarin text-path rejection on both `synthesize` and `synthesizeDetailed`. End-to-end Mandarin synthesis verified against PyTorch ground truth on `zf_001` and `zm_009`. Background-noise investigation tracked separately in #569 (atan2 phase correction in upstream `CoreMLForwardSTFT`). ## Test plan - [x] `swift build` clean - [x] `swift test --filter KokoroAneVariantTests` — 12/12 pass - [x] `swift format lint` clean (only pre-existing warnings on `fastV2_1`/`balancedV2_1`/`highContextV2_1` enum cases unrelated to this PR) - [ ] After HF upload of `ANE-zh/` bundle, end-to-end smoke test: `swift run fluidaudiocli tts "ㄋㄧˇㄏㄠˇㄕˋㄐㄧㄝˋ。" --backend kokoro-ane --variant zh --voice zf_001 --output /tmp/zh.wav` - [ ] No regressions on existing English path (default-arg behavior preserved) ## Out of scope - Mandarin text-to-Bopomofo G2P — Phase 2 (separate PR). - HF upload of `ANE-zh/` bundle — handled outside this repo. - Updating `Documentation/` with Mandarin voice list — defer to Phase 2 when the path is fully usable end-to-end.

Alex-Wengg mentioned this pull request May 3, 2026

feat(tts/kokoro-ane): add Mandarin (v1.1-zh) variant #570

Merged

5 tasks

docs: session report — Mandarin ASR + Kokoro v1.1-zh noise fix

2cdee77

Chronological log covering THCHS-30 benchmark, error analysis, beam search A/B, Cohere encoder fix, the Kokoro v1.1-zh atan2 noise root-cause investigation, the Devin review audit, and PRs landed.

Alex-Wengg closed this May 4, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: KokoroAne zh-CN noise reduction via atan2 phase correction#569

fix: KokoroAne zh-CN noise reduction via atan2 phase correction#569
Alex-Wengg wants to merge 2 commits into
mainfrom
fix/kokoro-ane-zh-noise

Alex-Wengg commented May 3, 2026

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Alex-Wengg commented May 3, 2026

Summary

Root cause

Fix

Distribution

Cache invalidation for existing users

Changes

Test plan

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PocketTTS Smoke Test ✅

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kokoro TTS Smoke Test ✅

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Qwen3-ASR int8 Smoke Test ✅

Performance Metrics

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ASR Benchmark Results ✅

Parakeet v3 (multilingual)

Parakeet v2 (English-optimized)

Streaming (v3)

Streaming (v2)

Expected RTFx Performance on Physical M1 Hardware:

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Diarization Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VAD Benchmark Results

Performance Comparison

Dataset Details

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Parakeet EOU Benchmark Results ✅

Performance Metrics

Streaming Metrics

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Offline VBx Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading

github-actions Bot commented May 3, 2026 •

edited

Loading