How could I use a build of whisper.cpp with CUBLAS

[pkese]

I'm seeing better performance with CUBLAS and would like to use a version of whisper with CUDA support.

Is it possible to somehow build a custom whisper.so with this project and provide custom build parameters?

[NickDarvey]

Yes, something like #20.

Right now I'm stuck on

nvcc fatal : A single input file is required for a non-link phase when an outputfile is specified

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\BuildCustomizations\CUDA 12.1.targets(799,9): error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc.exe" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.35.32215\bin\HostX64\x64" -x cu -ID:\a\whisper\whisper\ext\whisper.cpp. -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" --keep-dir x64\Release -maxrregcount=0 --machine 64 --compile -cudart static /Zi -std=c++14 -Xcompiler="/EHsc -Ob2" -D_WINDOWS -DNDEBUG -D_CRT_SECURE_NO_WARNINGS -DGGML_USE_CUBLAS -D"CMAKE_INTDIR="Release"" -D_MBCS -DWIN32 -D_WINDOWS -DNDEBUG -D_CRT_SECURE_NO_WARNINGS -DGGML_USE_CUBLAS -D"CMAKE_INTDIR="Release"" -Xcompiler "/EHsc /W1 /nologo /O2 /FS /MD " -Xcompiler "/FdD:\a\whisper\whisper\out\x64-windows\ext\whisper.cpp\Release\whisper.pdb" -o whisper.dir\Release\ggml-cuda.obj "D:\a\whisper\whisper\ext\whisper.cpp\ggml-cuda.cu"" exited with code 1. [D:\a\whisper\whisper\out\x64-windows\ext\whisper.cpp\whisper.vcxproj]

https://github.com/NickDarvey/whisper/actions/runs/5171997631/jobs/9315926479#step:6:252

It's the same issue that's described in ggerganov/whisper.cpp#840 but I already have the fix described as the submodule is at a later commit than ggerganov/whisper.cpp#867 🫤

NVlabs/instant-ngp#39

It sounds like something in that nvcc.exe command is misformatted.

I don't have a device to test this on so I'm stuck with a 20 minute or more feedback cycle with GitHub actions. If you have a moment and have a device to test with, it would be helpful if you could try to identify what's wrong with that comand.

[pkese]

Sorry, I wasn't aware this library is Windows only.

The machine with GPU that I intend to run Whisper on is Linux.
I found out only after trying to build this library on Linux.

[NickDarvey]

It's something I'd like to add, #6

You could:

1. Add support for it, starting here
2. Use https://github.com/sandrohanea/whisper.net which already supports linux
   (You'll need to contribute cuBLAS support there, probably publishing another runtime package like is done for CoreML.)

[pkese]

I've tried to get build to work on Linux, but then had a problem of how to cleanly install Ninja and Vcpkg.
So I figured out I'll get more quickly to results with https://github.com/sandrohanea/whisper.net
It has an option to provide my own build of libwhisper.so when loading the library, which fits my purpose.

Otoh, I appreciate the cleanliness of your approach: a c++ like interface with least added ceremony (I could simply read the original c++ source files and know how to use the dotnet version).