bug: Quantized Models not Loading

[loukylor]

Describe the bug

When I pass the path of a quantized model into Whisper, it attempts to load the model however fails with the message whisper_model_load: ERROR not all tensors loaded from model file - expected 245, got 3. I don't believe this is an issue with the bindings or whisper.cpp, but rather an issue with the version of whisper.cpp built by Bazel.

A few lines before the error is printed, whisper_model_load: f16 = 8 is printed, however f16 should be ftype according to this commit on whisper.cpp. I confirmed this by loading the quantized version into whisper.cpp, also confirming the quantized model is valid. I've built and installed the whisper.cpp bindings directly from source, so I am unsure why Bazel isn't building the right commit.

To reproduce

• Install and build whisper.cpp bindings according to the readme
• Quantize a model according to the instructions on whisper.cpp's readme
• Attempt to load the quantized model by passing the full path of the quantized model into from_pretrained

Expected behavior

The model loads and runs without error.

Environment

Python: 3.10.9
OS: WSL Ubuntu 22.04
Bazel: 6.0.0 (installed from the Bazel script in ./tools/)
Bindings commit: ca5863c04c9e0eb3359ef0330dde06747cef406c
whisper.cpp submodule commit: 14bee39b290f81381a259c53a169ffac8fcce987

[dfaisalmahmood]

It seems to be an issue related to the download mechanism. If you check ~/.local/share/whispercpp, you will see the files being downloaded are much smaller than they should be. For now a solution is to download the models manually from https://huggingface.co/ggerganov/whisper.cpp/tree/main and place in this folder.

By the way, I am on Mac so on Windows and Ubuntu the folder path might be different. Try to figure out where the model files are getting downloaded to.

[AIWintermuteAI]

No, it is definitely not an issue with the download mechanism.
The model downloaded manually fails the same way.

[AIWintermuteAI]

Fixed here
AIWintermuteAI@512426f
Need to build and install locally of course.

@aarnphm are you still interested in maintaining the package? I see there is a cron job updating submodules, but it has no effect on build, since target commit is specified in rules/deps.bzl.
If you do not plan on maintaining the package, perhaps I will be able to at least set up a CI as GH Action to periodically test with new commits of whisper.cpp.