How to increase speech to text speed when using whisper cpp?

[ITHealer]

Currently I am using whisper_tiny.en.tflite for my android project. And the inference time for speech to text is quite long.
https://github.com/usefulsensors/openai-whisper/tree/main/android_app/Whisper-TFLIte-Android-Example

---

I/flutter ( 3449): Model name: /assets/whisper-tiny.en.tflite
I/flutter ( 3449): Load success
I/flutter ( 3449): Load model time 0:00:03.274847
I/htotext_exampl( 3449): ProcessProfilingInfo new_methods=1340 is saved saved_to_disk=1 resolve_classes_delay=8000
I/tflite ( 3449): Initialized TensorFlow Lite runtime.
W/1.ui ( 3449): type=1400 audit(0.0:15783): avc: denied { read } for name="u:object_r:vendor_default_prop:s0" dev="tmpfs" ino=14533 scontext=u:r:untrusted_app:s0:c127,c257,c512,c768 tcontext=u:object_r:vendor_default_prop:s0 tclass=file permissive=0
E/libc ( 3449): Access denied finding property "ro.hardware.chipname"
I/tflite ( 3449): Created TensorFlow Lite XNNPACK delegate for CPU.
I/flutter ( 3449): Result process audio file: I love you.
I/flutter ( 3449): Processing time 0:00:11.894501

---

Is there any way to shorten the time, please help me?
Thanks!

[ITHealer]

@ggerganov Can you give me directions?

[dgm3333]

buy a more powerful processor?
Given that you haven't even given basic details of your setup it seems to me you are being pretty unreasonable to target the primary dev for something which is always going to be limited by hardware and is a totally different repo to this one :-(

[themanyone]

I am the author of Caption Anything and Whisper Dictation. In #1653, I mentioned how speed can be improved by compiling Whisper.cpp with acceleration like CLBlast or cuBLAS, using the tiny models, or employing a client-server setup. Connect slow clients like android to faster computers running Whisper.cpp ./server instances on the network.

I also ran into a case where generation was inexplicably slow, and it was no longer using the GPU. The solution was to reload the video driver module (or reboot). (I was testing -allow-unsupported-compiler thinking I could outsmart the system).

sudo modprobe -r nvidia_uvm; sudo modprobe nvidia_uvm

[jordibruin]

Android phones won't be able to run this fast.

[zhouwg]

Xiaomi 14 could do that(with Xiaomi's proprietary 6B device-side AI model) because it contains a very powerful mobile SoC