Maybe offer a function to pick "the best model"?

[dscho]

I just performed a quick & dirty (very dirty) test with the latest recordings I had in my setup anyway, to see what the performance characteristics are of the three models I end up switching between: Parakeet v2, v3 and Canary. I have a Windows laptop with a very weak GPU, so I was particularly interested in seeing how CPU vs GPU would compare. Since Canary consistently failed with DirectML, there are no numbers to share about that combination. Here are the numbers (inference times, I specifically excluded the model loading times from the analysis):

Recording	Parakeet v2 DML	v2 CPU	v3 DML	v3 CPU	Canary CPU
11.5s clip	7521ms (1.5x)	1727ms (6.7x)	4220ms (2.7x)	1880ms (6.1x)	1631ms (7.0x)
16.1s clip	5087ms (3.2x)	2801ms (5.8x)	5290ms (3.1x)	2743ms (5.9x)	3557ms (4.5x)
4.5s clip	2230ms (2.0x)	666ms (6.7x)	2107ms (2.1x)	651ms (6.9x)	534ms (8.4x)
12.9s clip	4341ms (3.0x)	1919ms (6.7x)	4028ms (3.2x)	2160ms (6.0x)	1689ms (7.6x)
31.9s clip	ERROR	5995ms (5.3x)	ERROR	6010ms (5.3x)	9683ms (3.3x)

That paints an interesting picture!

• CPU beats DirectML across the board by roughly 2-3x for inference alone. DirectML adds per-inference overhead (GPU
  memory copies, kernel dispatch) that only pays off with much longer audio or larger models (for which the transcription would fail due to memory errors!). On the 32s clip, DirectML even fails for Parakeet with MemcpyToHost!
• Canary on CPU is the fastest for short/medium clips (up to ~15s), but scales worse on the 32s clip (3.3x real-time vs
  Parakeet's 5.3x). Canary is a seq2seq decoder, so its cost grows super-linearly with length.
• Parakeet v2 vs v3 are nearly identical in speed. The choice between them is about accuracy, not performance. Having said that, I often dictate in my native language, therefore v3 is really my go-to choice for now.

Another interesting point are quality differences: Canary produces slightly better punctuation ("Like, I want..." vs "Like I want..."). Parakeet
v2 preserves commas better than v3 but v3 sometimes gets words right that v2 misses ("records" vs "recogns").

Now, the quality differences can only really be assessed by a human or by an advanced LLM. But the timings are something that I could be determined automatically, I imagine, via a dedicated button helping Handy users to determine in a semi-scientific fashion which model to use and whether or not to prefer CPU over GPU.

Thoughts?

[cjpais]

I think it's just a really hard problem overall because different models have different characteristics like you're pointing out. Like punctuation or just generally how they handle speech and your speech in particular, especially people with accents or using multiple languages. I think this problem just isn't very clear cut and is hard to automatically determine for someone and a huge variety of people who use this application.

[dscho]

Fair enough!