How can I teach whisper new words?

[MrEnigmamgine]

Hi! I am working on a project that will use whisper to transcribe recordings of my Dungeons and Dragons games. Since we use a lot of made up words and weird names I'd love to include a solution for training them. For example, we often reference NPC names like Zerthimon, Vlaakith, and Mordenkainen as well as place names like Agni'hotri, Tu'narath, and Niam'd'regal.

I understand that I would need to collect labeled samples of these words. But once I have that done, how can I feed that data into Whisper? (Preferably in a way that allows me to store the customization in a separate file and doesn't require retraining every time the app is loaded.)

Any help you guys can provide would be greatly appreciated. I'm not expecting you to write a full tutorial, but if you could point me in the right direction I'm sure I could figure it out.

[jongwook]

You could make it work to some degree by prompting, by specifying --initial_prompt "The following is a conversation during a Dungeons and Dragons game, which includes NPC names like Zerthimon, Vlaakith, and Mordenkainen as well as place names like Agni'hotri, Tu'narath, and Niam'd'regal.", then the model will have a slightly better chance of spelling those words correctly.

If that doesn't work very reliably, the next bet is to fine-tune a Whisper model with your own examples. #364 has some pointers on how to do this.

[MrEnigmamgine]

Thank you so much! The tutorial for fine tuning is going to be incredibly helpful, but I'll be sure to try prompting first just in case.

[chen-rn]

Hey! I'm curious how well prompting worked for you in this case?

[whc2001]

Not working really well for me, or my accent is way too crude...

[rahulbansal16]

Does the fine-tuned model work best for you, your voice, or does it become a general model? If somebody with a different voice says the same words, will it be detected reliably?