show benchmarks

[laralove143]

the advantage of this project is that it uses CoreML for a performance gain, so showing benchmarks would solidify how much this advantage is

[atiorh]

Hi @laralove143, please give https://www.takeargmax.com/blog/whisperkit a read for the value proposition of WhisperKit.
That being said, performance is definitely a big part and we are working on a "Performance Benchmark Tab" in the example app. Will follow up here shortly.

[laralove143]

that blog is very useful maybe it could be shown more clearly in the readme, for example talking about its contents

alternatively, some stuff from the blog could be included in the readme as well, like that demo video is very useful

[atiorh]

Thanks for the feedback! We will think about a better way to organize information about WhisperKit that is more accessible. We will definitely flesh out the README and docs more before stable release.

[ZachNagengast]

Tracking this here: #28

[aehlke]

My understanding from running llama.cpp on iOS/macOS (via Swift, including streaming) is that Metal is faster than CoreML or Metal+CoreML. There may be some other benefits to using CoreML. Maybe battery? I don't know myself

[atiorh]

Metal is faster than CoreML or Metal+CoreML

This is certainly possible in specific cases but can not be a generally true statement. For context, WhisperKit is currently tuned for mobile and lower-end Macs where the Neural Engine is much more powerful with respect to the GPU (that Metal can harness) and Core ML is the primary framework for deploying to the Neural Engine. That being said, we are actively working on a Metal backend to complement the Core ML backend.

[aehlke]

Thanks for the context. Would be great to see a benchmark later. I recall the Metal-only whispercpp being faster even on lower spec devices such as iPhone but can't find the numbers at the moment.

[aehlke]

Here are some numbers I hadn't seen before showing far better Metal performance on an iPhone using Metal instead of CoreML. So it looks like it holds true for mobile...

https://www.bjnortier.com/2023/11/17/Hello-Transcribe-3.2.html

It mentions some other downsides to CoreML such as the slow caching step and unpredictable cache ejection by the OS