Improving whisper timestamp per word by fine tunning

[RaulKite]

Hi,

I need a good timestamp per word accuracy with the transcription of whisper

I have seen that fine tunning whisper with hugging face 🤗 seems easy for other languages so I have thought that maybe to have better accuracy is a feasible task this way.

It could be “easy” to create a dataset with aligned long audios with tools like Gentle( https://github.com/lowerquality/gentle ) I have experience with this.

Also add some layers in the top of the model to train this new output seems possible.

Is there anyone working with this? I’m wrong?

If someone is working on this please ping me. I will be exploring this path next weeks...

P.d. stable ts project is nice but not solve my problems. I have been testing it but I need real accuracy and stable ts works on raw whisper data been sometimes not really accurate.

Thanks

[wwbnjsace]

hugging face

where is the fine tunning whisper code in hugging face?why i cant not find it?

[RaulKite]

https://huggingface.co/blog/fine-tune-whisper