Converting fine-tuned ggml model to CoreML. Invalid model name error.

[Sogl]

I want to convert this model: https://huggingface.co/mitchelldehaven/whisper-large-v2-ru/tree/main to CoreML

First, I successfully converted bin to ggml using the following instructions: https://github.com/ggerganov/whisper.cpp/tree/master/models#fine-tuned-models

But when trying to convert ggml to CoreML version, an error occurs:

rina@Irinas-MacBook-Air whisper.cpp % ./models/generate-coreml-model.sh ./models/ggml-model.bin
Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 
is the most recent version that has been tested.
/opt/homebrew/lib/python3.11/site-packages/whisper/timing.py:57: NumbaDeprecationWarning: The 'nopython' 
keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument
is currently False, but it will be changed to True in Numba 0.59.0. 
See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit
Traceback (most recent call last):
  File "/Users/irina/code/whisper.cpp/models/convert-whisper-to-coreml.py", line 306, in <module>
    raise ValueError("Invalid model name")
ValueError: Invalid model name
coremlc: error: Model does not exist at models/coreml-encoder-./models/ggml-model.bin.mlpackage -- file:///Users/irina/code/whisper.cpp/
mv: rename models/coreml-encoder-./models/ggml-model.bin.mlmodelc to models/ggml-./models/ggml-model.bin-encoder.mlmodelc: No such file or directory

I also tried @akeybl hack from here: #1088 (comment)

but I don't have .pt file or URL for it 😢

So I ended up with another error:

irina@Irinas-MacBook-Air whisper.cpp % ./models/generate-coreml-model.sh large.ru
Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
/opt/homebrew/lib/python3.11/site-packages/whisper/timing.py:57: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit
100%|██████████████████████████████████████| 55.1k/55.1k [00:00<00:00, 876kiB/s]
Traceback (most recent call last):
  File "/Users/irina/code/whisper.cpp/models/convert-whisper-to-coreml.py", line 308, in <module>
    whisper = load_model(args.model).cpu()
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/whisper/__init__.py", line 135, in load_model
    checkpoint_file = _download(_MODELS[name], download_root, in_memory)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/whisper/__init__.py", line 89, in _download
    raise RuntimeError(
RuntimeError: Model has been downloaded but the SHA256 checksum does not not match. Please retry loading the model.
coremlc: error: Model does not exist at models/coreml-encoder-large.ru.mlpackage -- file:///Users/irina/code/whisper.cpp/
mv: rename models/coreml-encoder-large.ru.mlmodelc to models/ggml-large.ru-encoder.mlmodelc: No such file or directory

https://github.com/ggerganov/whisper.cpp/blob/master/models/convert-whisper-to-coreml.py#L308

Related issue: #947

Any thoughts?

[AlienKevin]

Workaround (HF -> PT -> CoreML)

I found a workaround by converting a HuggingFace model to PyTorch first and then modify the existing convert-whisper-to-coreml.py script slightly to accomodate for a custom model name.

1. Convert HuggingFace model to the PyTorch pt format
   Begin by installing a conversion utility created by bayartsogt-ya. It basically renames the keys in the state_dict of the HuggingFace model so they are compatible with OpenAI's convention.

   pip install git+https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets

   Then, import the utility and use it to convert an existing Huggingface model to PyTorch.

   from multiple_datasets.hub_default_utils import convert_hf_whisper
   convert_hf_whisper("username/whisper-small", "small.pt")

   This should output a small.pt file that obeys the OpenAI convention.
   Copy small.pt to the root directory of the whisper.cpp project in preparation for the 3rd step.
   This allows you to pass the string small as the path to the pt file and also use it as the model name in the generated coreml model. However, I'm sure there's a more ergonomic solution by optionally passing in a model_path argument in addition to the existing model parameter.
2. Modify the model name parsing logic slightly in the convert-whisper-to-coreml.py script:

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", type=str, help="model to convert (e.g. tiny, tiny.en, base, base.en, small, small.en, medium, medium.en, large, large-v1)", required=True)
    parser.add_argument("--encoder-only", type=bool, help="only convert encoder", default=False)
    parser.add_argument("--quantize",     type=bool, help="quantize weights to F16", default=False)
    parser.add_argument("--optimize-ane", type=bool, help="optimize for ANE execution (currently broken)", default=False)
    args = parser.parse_args()

    ############ Comment out the model name checking below ############
    """
    if args.model not in ["tiny", "tiny.en", "base", "base.en", "small", "small.en", "medium", "medium.en", "large", "large-v1"]:
        raise ValueError("Invalid model name")
    """

    whisper = load_model(args.model).cpu()
    hparams = whisper.dims
    print(hparams)

3. You are all set! Now you can run ./models/generate-coreml-model.sh small to convert your small.pt to a ggml-small-encoder.mlmodelc!

Tests

I tested workaround with a fine-tuned small model and it worked quite well.

[Sogl]

@AlienKevin Thank you! Can you clarify about these lines?

1. Then, import the utility and use it to convert an existing Huggingface model to PyTorch.

   from multiple_datasets.hub_default_utils import convert_hf_whisper
   convert_hf_whisper("username/whisper-small", "small.pt")

Do I need to create some py file and put those lines in there and then run it in console? Sorry, I'm not very familiar with Python.

[AlienKevin]

@Sogl Yeah, you can create a python file and run the two lines I mentioned. Alternatively, you can add those to the end of the jupyter notebook that you use to fine-tune the model. Either way works.

[bobqianic]

How about creating a PR for this?

[AlienKevin]

@bobqianic Submitted a PR. I tried to keep the script simple by reusing the existing convert-whisper-to-coreml.py. But since its file name uses dashes instead of underscores, I have to do a dynamic import... I tested the script on a fine-tuned tiny model, see the PR for the detailed steps. Thanks.