Any way to port fine-tuned model from huggingface to this whisper package

[tpmmthomas]

I have successfully fine-tuned the whisper model on a cantonese dataset on huggingface, which can be loaded with the WhisperForConditionalGeneration class.

I then wanted to make use of this whisper package to perform inference in production, in order to take advantage of the transcribe function, and the initial_prompts is also useful to me for guiding the model to output written / spoken cantonese scripts.

However, it seems that the model on huggingface and on the whisper package of this repo is not directly compatible (I cannot load the huggingface model with whisper.load_model(). Is there any way around this? Thank you!

[luigisaetta]

@tpmmthomas you can have a look at this utility I have developed https://github.com/luigisaetta/whisper-app/blob/main/match_layers.ipynb

[huynhthanh98]

Hi!
Can i convert a pt file to huggingface?

[PeterGilles]

Or try this conversion script: https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets

[huynhthanh98]

Hi!
Can i convert a pt file to huggingface?

[huynhthanh98]

Hi @PeterGilles !

Can i convert a pt file which contains weights of whisper model to huggingface model?

Thank you very much.

[afsara-ben]

any easier way to convert WhisperForConditionalGeneration to WhisperModel?

[XiaoQiu2451]

I think what @PeterGilles said is already eazy enough.
https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets
with the code:
from multiple_datasets.hub_default_utils import convert_hf_whisper model_name_or_path = 'path/your_WhisperForConditionalGeneration_model_path' whisper_checkpoint_path = './WhisperModel.pt' convert_hf_whisper(model_name_or_path, whisper_checkpoint_path)
Bascially what the function convert_hf_whisper does is map the encoder , the decoder, etc to the dict which can be directly used by whisper.
If you want to directly convert WhisperForConditionalGeneration to WhisperModel, then you can refer to how convert_hf_whisper function works and how whisper.load_model works.