Streaming for ASR by BrandonWeng · Pull Request #79 · FluidInference/FluidAudio

BrandonWeng · 2025-08-26T05:28:37Z

Why is this change needed?

Adding support for ASR streaming. Majority of the change is in Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift

e.g swift run fluidaudio transcribe GLM\ 4.5.wav --streaming

Clipboard-20250826-052557-193.mp4

stt.mp4

claude

Comprehensive code review completed. This PR significantly improves FluidAudio's streaming ASR capabilities with excellent API design and examples.

Key Strengths:

Dual-stream architecture (results vs snapshots) elegantly separates detailed events from UI updates
Production-quality SwiftUI and UIKit examples with proper patterns
Well-tuned .realtime configuration for responsive transcription

Critical Issue to Address:

Memory leak risk: finalizedSegments and segmentFinalTexts collections grow unbounded during long sessions (could accumulate thousands of segments in memory)

Recommendations:

Implement segment cleanup mechanism for long-running sessions
Consider structured concurrency patterns in audio callback examples
Add migration guide for API changes

Overall, this is a solid architectural improvement that addresses real-world streaming ASR needs effectively. The memory management issue should be resolved before merge.

github-actions · 2025-08-26T05:34:22Z

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric	Value	Target	Status	Description
DER	15.1%	<30%	✅	Diarization Error Rate (lower is better)
JER	24.9%	<25%	✅	Jaccard Error Rate
RTFx	19.64x	>1.0x	✅	Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage	Time (s)	%	Description
Model Download	8.953	16.8	Fetching diarization models
Model Compile	3.837	7.2	CoreML compilation
Audio Load	0.090	0.2	Loading audio file
Segmentation	16.011	30.0	Detecting speech regions
Embedding	26.685	50.0	Extracting speaker voices
Clustering	10.674	20.0	Grouping same speakers
Total	53.421	100	Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method	DER	Notes
FluidAudio	15.1%	On-device CoreML
Research baseline	18-30%	Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

M2 MacBook Air (2022): Runs at 150 RTFx real-time
Performance scales with Apple Neural Engine capabilities

_{🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 53.4s diarization time • Test runtime: 1m 35s • 09/08/2025, 11:58 PM EST}

github-actions · 2025-08-26T05:37:00Z

ASR Benchmark Results ✅

Status: All benchmarks passed

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.57%	0.00%	3.05x	✅
test-other	1.19%	0.00%	2.03x	✅

🚀 Streaming Performance Test

Metric	Value	Status	Description
WER	0.00%	✅	Word Error Rate in streaming mode
RTFx	0.37x	✅	Streaming real-time factor
Avg Chunk Time	2.396s	✅	Average time to process each chunk
Max Chunk Time	2.967s	✅	Maximum chunk processing time
First Token	2.726s	✅	Latency to first transcription token
Total Chunks	31	✅	Number of chunks processed

_{Streaming test validates --streaming option with 5 files using 0.5s chunks to simulate real-time audio streaming}

_{25 files per dataset • Test runtime: 3m41s • 09/09/2025, 12:01 AM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

github-actions · 2025-08-26T05:39:52Z

VAD Benchmark Results

Performance Comparison

Dataset	Accuracy	Precision	Recall	F1-Score	RTFx	Files
MUSAN	88.0%	80.6%	100.0%	89.3%	84.3x faster	50
VOiCES	88.0%	80.6%	100.0%	89.3%	82.3x faster	50

Dataset Details

MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

✅: Average F1-Score above 70%

BrandonWeng · 2025-08-27T01:22:32Z

@claude review the whole PR

BrandonWeng · 2025-08-27T01:23:06Z

+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/FluidInference/FluidAudio)
+

 Fluid Audio is a Swift framework for fully local, low-latency audio processing on Apple devices. It provides state-of-the-art speaker diarization, ASR, and voice activity detection through open-source models (MIT/Apache 2.0 licensed) that we've converted to Core ML.


@claude double check the readme for me

Claude finished @BrandonWeng's task —— View job

README.md Review Complete ✅

I've thoroughly reviewed the updated README.md and cross-referenced it with the actual implementation. The documentation is accurate and well-written, with only one minor typo to fix:

Issues Found

Line 27: Typo

- **Cross-platform**: Support for macOS 14.0+ and iOS 17.0+ and Apple Sillicon device + **Cross-platform**: Support for macOS 14.0+ and iOS 17.0+ and Apple Silicon device

(Note: "Sillicon" should be "Silicon")

What I Verified ✅

Streaming API Examples: All code examples match the actual StreamingAsrManager implementation

Configuration Options: The StreamingAsrConfig.realtime and .default presets exist exactly as documented

CLI Commands: The --streaming flag and commands are implemented correctly

Performance Numbers: RTFx and benchmarking data aligns with CI results

API Consistency: Both single-stream and multi-stream examples are accurate

Parameter Documentation: All config parameters like chunkSeconds, leftContextSeconds, etc. match the implementation

Strengths

Comprehensive Coverage: Excellent documentation of both batch and streaming modes

Practical Examples: Real-world usage patterns with proper async/await syntax

Performance Context: Clear RTFx explanations and hardware-specific expectations

Configuration Guidance: Well-explained parameter impacts on latency vs accuracy

The streaming ASR documentation is production-ready and accurately reflects the implementation. Just fix the "Sillicon" typo and it's perfect.

BrandonWeng · 2025-08-27T01:24:00Z

-      - name: Cache iOS Platform
-        uses: actions/cache@v4
-        with:
-          path: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform
-          key: ios-platform-${{ runner.os }}-xcode-${{ hashFiles('/Applications/Xcode.app/Contents/version.plist') }}
-          restore-keys: |
-            ios-platform-${{ runner.os }}-xcode-


not a huge file so easier to redownload, notice that hte cahce doesn't work all the time anyways

BrandonWeng · 2025-08-27T01:24:28Z

@@ -1,441 +0,0 @@
-Hello! I’m Donovan, an engineer on the Speech framework team and I’m Shantini, an engineer on the Notes team.


not very useful for us here

BrandonWeng · 2025-08-27T01:25:17Z

 [![Platform](https://img.shields.io/badge/Platform-macOS%20%7C%20iOS-blue.svg)](https://developer.apple.com)
 [![Discord](https://img.shields.io/badge/Discord-Join%20Chat-7289da.svg)](https://discord.gg/WNsvaCtmDe)
 [![Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue)](https://huggingface.co/collections/FluidInference/coreml-models-6873d9e310e638c66d22fba9)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/FluidInference/FluidAudio)


Moved around the sections a bit and cleanedu p the phrases as well

BrandonWeng · 2025-08-27T02:48:54Z

Artificial delay when streaming or else its too fast, and you can't even see it

Alex-Wengg · 2025-08-27T04:05:38Z

is this all from the parakeet python code

nope, parakeet python code's streaming implementation requires a bit too much time, its not responsive enough. Takes inspiration from it and the apple streaming API

where is the apple streaming api doc from

Alex-Wengg · 2025-08-27T04:16:34Z

is this gonna be frequent ? removeFirst is like O(n)

Alex-Wengg · 2025-08-27T04:29:21Z

i gues we will watch out for deadlocks , race conditions and possible memory leaks. the keep priority code being asrmanager.swift

The Cross-Platform Build Check was failing on main branch pushes due to a race condition where iOS simulators were listed as available but not yet accessible to xcodebuild. Changes: - Add xcodebuild verification step with 5-attempt retry mechanism - Wait for simulator to be fully available before build attempts - Add diagnostic output showing available destinations - Implement UUID-based fallback as final attempt - Improve error messages and debugging information This eliminates the timing issue causing "Unable to find a device matching the provided destination specifier" errors. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

Alex-Wengg · 2025-08-27T14:53:51Z

are the streaming prefixes necessary for the streaming asr files ?

BrandonWeng · 2025-08-27T15:17:44Z

are the streaming prefixes necessary for the streaming asr files ?

Lets keep it, or else hard to know which is which.

ozansz · 2025-08-28T09:10:58Z

stt.mp4

Hi, is the code running in this demo video listening to the speakers of your device or did you somehow duplicated the sound output to a virtual device and redirected it to the running streaming ASR? I'm asking because I'm currently trying to find easy-to-implement native ways to do so.

BrandonWeng · 2025-08-28T15:51:08Z

e.g swift run fluidaudio transcribe GLM\ 4.5.wav --streaming

Oh no, in this case it was just listening to the microphone, swift run fluidaudio transcribe --microphone. You should be able to use CoreAudio to tap into the system audio if that's what you're looking for

benbjurstrom · 2025-09-17T16:33:05Z

Thanks for the awesome package, it's been great to work with so far. I know this feature isn't final yet, but just wanted to share some issues I'm seeing.

The screenshot below is a csv dump of streamingAsr.segmentUpdates taken from playing this video into my microphone: https://www.youtube.com/watch?v=s0EbxhQYeCA

If you look at lines 33 and 34, as the segment moves from volatile to stable a bunch of the segment text is lost. I see this often when using the .balanced profile.

There's also other weirdness between volatile and stable updates. Seems to happen most often around the boundaries of chunkDuration. For example, volatile line 17 ends in "But", but it's removed from the stable line 18, and only appears again at the beginning of the next stable update on line 24.

BrandonWeng · 2025-09-17T16:42:35Z

Thanks for the awesome package, it's been great to work with so far. I know this feature isn't final yet, but just wanted to share some issues I'm seeing.

The screenshot below is a csv dump of streamingAsr.segmentUpdates taken from playing this video into my microphone: https://www.youtube.com/watch?v=s0EbxhQYeCA

If you look at lines 33 and 34, as the segment moves from volatile to stable a bunch of the segment text is lost. I see this often when using the .balanced profile.
There's also other weirdness between volatile and stable updates. Seems to happen most often around the boundaries of chunkDuration. For example, volatile line 17 ends in "But", but it's removed from the stable line 18, and only appears again at the beginning of the next stable update on line 24.

Thanks a bunch for the feedback @benbjurstrom - I've been distracted with the VAD model improvements. Hoping to revisit streaming ASR next week 🙏

BrandonWeng · 2025-09-23T16:53:58Z

Going to close this for now, drifted too far from main and haven't been able to find time to work on this

chinarui-na · 2026-02-25T08:59:21Z

Why is this change needed?

Adding support for ASR streaming. Majority of the change is in Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift

e.g swift run fluidaudio transcribe GLM\ 4.5.wav --streaming

Clipboard-20250826-052557-193.mp4
stt.mp4

Are there any examples to demo the stt.mp4 example?

BrandonWeng added documentation Improvements or additions to documentation enhancement New feature or request labels Aug 26, 2025

claude Bot reviewed Aug 26, 2025

View reviewed changes

BrandonWeng force-pushed the asr-streaming branch 2 times, most recently from 72a877b to 260e7f3 Compare August 26, 2025 19:59

Initial vibes for streaming

69434d5

BrandonWeng force-pushed the asr-streaming branch from 260e7f3 to afb238a Compare August 26, 2025 20:01

FluidInference deleted a comment from claude Bot Aug 26, 2025

BrandonWeng commented Aug 27, 2025

View reviewed changes

BrandonWeng requested review from Alex-Wengg and Bharat0091 August 27, 2025 02:43

BrandonWeng assigned Bharat0091 and Alex-Wengg Aug 27, 2025

BrandonWeng marked this pull request as ready for review August 27, 2025 02:47

FluidInference deleted a comment from claude Bot Aug 27, 2025

BrandonWeng commented Aug 27, 2025

View reviewed changes

Comment thread Sources/FluidAudioCLI/Commands/TranscribeCommand.swift Outdated

Alex-Wengg reviewed Aug 27, 2025

View reviewed changes

BrandonWeng force-pushed the asr-streaming branch 2 times, most recently from 2abad6e to eb93191 Compare August 27, 2025 05:06

BrandonWeng force-pushed the asr-streaming branch from a40d91e to dd8b8c9 Compare August 27, 2025 06:04

format

c8eb750

Alex-Wengg reviewed Aug 27, 2025

View reviewed changes

Comment thread Sources/FluidAudio/ASR/Streaming/StreamingTranscriptionResult.swift Outdated

Alex-Wengg reviewed Aug 27, 2025

View reviewed changes

Comment thread Sources/FluidAudio/ASR/Streaming/StreamingTranscriptionResult.swift Outdated

BrandonWeng added 2 commits August 27, 2025 11:44

Better left context for final chunk

f64a3d3

better logging for cli

60145bb

Merge branch 'main' into asr-streaming

76a22d2

BrandonWeng added 9 commits September 3, 2025 14:46

Merge remote-tracking branch 'origin/main' into asr-streaming

b70d4bf

Merge remote-tracking branch 'origin/main' into asr-streaming

2fef897

Merge remote-tracking branch 'origin/main' into asr-streaming

d3a1a6a

deprecate tracker

29ef9cc

Cleaned up some of the issues

e6f57c8

Streaming in a decent state

903890d

Streaming decent

9a1dfdf

Adjust tokenier

dfeffaf

lint

f4800dd

BrandonWeng added the speech-to-text issues related to transcription/asr label Sep 8, 2025

BrandonWeng and others added 4 commits September 8, 2025 12:00

Fix tests

7f8fac1

format

fcb6a72

Merge remote-tracking branch 'origin/main' into asr-streaming

77f05ea

Merge branch 'main' into asr-streaming

91323c7

BrandonWeng closed this Sep 23, 2025

BrandonWeng mentioned this pull request Sep 25, 2025

StreamingASRManager Token level timestamps for diarization #120

Closed

		[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/FluidInference/FluidAudio)


		Fluid Audio is a Swift framework for fully local, low-latency audio processing on Apple devices. It provides state-of-the-art speaker diarization, ASR, and voice activity detection through open-source models (MIT/Apache 2.0 licensed) that we've converted to Core ML.

		@@ -1,441 +0,0 @@
		Hello! I’m Donovan, an engineer on the Speech framework team and I’m Shantini, an engineer on the Notes team.

Uh oh!

Conversation

BrandonWeng commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why is this change needed?

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Diarization Pipeline Timing Breakdown

Speaker Diarization Research Comparison

Uh oh!

github-actions Bot commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ASR Benchmark Results ✅

🚀 Streaming Performance Test

Expected RTFx Performance on Physical M1 Hardware:

Uh oh!

github-actions Bot commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VAD Benchmark Results

Performance Comparison

Dataset Details

Uh oh!

BrandonWeng commented Aug 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude Bot Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

README.md Review Complete ✅

Issues Found

What I Verified ✅

Strengths

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Alex-Wengg commented Aug 27, 2025

Uh oh!

Uh oh!

Uh oh!

Alex-Wengg commented Aug 27, 2025

Uh oh!

BrandonWeng commented Aug 27, 2025

Uh oh!

ozansz commented Aug 28, 2025

Uh oh!

BrandonWeng commented Aug 28, 2025

Uh oh!

benbjurstrom commented Sep 17, 2025

Uh oh!

BrandonWeng commented Sep 17, 2025

BrandonWeng commented Aug 26, 2025 •

edited

Loading

github-actions Bot commented Aug 26, 2025 •

edited

Loading

github-actions Bot commented Aug 26, 2025 •

edited

Loading

github-actions Bot commented Aug 26, 2025 •

edited

Loading

claude Bot Aug 27, 2025 •

edited

Loading

chinarui-na commented Feb 25, 2026 •

edited

Loading