GPU support via NVBLAS

[ggerganov]

ref #220

WIP IN PROGRESS

[vakkov]

@ggerganov Just "ported" the good work from ggerganov/llama.cpp#1044 to whisper and the performance improvement is significant... shall I PR it? Is this the way to go with Whisper as well?

[Green-Sky]

NVBLAS or cuBLAS ?

[dakk]

let me know if someone else is working on this feature; otherwise I'll put my hands on it

EDIT: I was too curious and I implemented it already: https://github.com/dakk/whisper.cpp using cublas. Let me know if I can create the PR.

[DanielusG]

let me know if someone else is working on this feature; otherwise I'll put my hands on it

EDIT: I was to curious and I implemented it already: https://github.com/dakk/whisper.cpp using cublas. Let me know if I can create the PR.

I want to try your code, what should I do? I've cloned your repo and i've the ggml model. And now? (i'm on linux if it can be relevant)

[dakk]

let me know if someone else is working on this feature; otherwise I'll put my hands on it
EDIT: I was to curious and I implemented it already: https://github.com/dakk/whisper.cpp using cublas. Let me know if I can create the PR.

I want to try your code, what should I do? I've cloned your repo and i've the ggml model. And now? (i'm on linux if it can be relevant)

mkdir build
cd build
cmake .. -DWHISPER_SUPPORT_CUBLAS=TRUE
make
cd ..
./build/bin/main -m models/ggml-tiny.en.bin -f samples/jfk.wav

[DanielusG]

@dakk I can confirm it works well with my GPU, but I notice one thing. With standard whisper I can use medium model with my RTX 2060 (6GB). Now with whisper.cpp CUDA enabled it runs out of memory.

[dakk]

@dakk I can confirm it works well with my GPU, but I notice one thing. With standard whisper I can use medium model with my RTX 2060 (6GB). Now with whisper.cpp CUDA enabled it runs out of memory.

Forgot to free some memory; now should run fine.

git pull && cd build && make

[vakkov]

@DanielusG https://bitbucket.org/sensika/whisper.cpp/branch/cublas

medium consumes 580 MB of vmem for me:
WHISPER_CUBLAS=1 make

[DanielusG]

@dakk I can confirm it works well with my GPU, but I notice one thing. With standard whisper I can use medium model with my RTX 2060 (6GB). Now with whisper.cpp CUDA enabled it runs out of memory.

Forgot to free some memory; now should run fine.

git pull && cd build && make

Yes! it work very well. Like a charm :)

[ggerganov]

I will add the cuBLAS support from the llama.cpp repo to whisper.cpp for the next release

[ggerganov]

Hi all, I am backporting the cuBLAS support from llama.cpp here: #834

Unfortunately, I don't observe significant speed-up. It's quite small actually
@vakkov Will appreciate if you take a look if I did something wrong. Maybe it is just my system and it actually works better on others - not sure

Edit: nevermind - it is much faster actually. Was using too old GPU before

[ggerganov]

Superseded by #834