Long-form (including timestamps) for whisper

[JeffreyWardman]

Feature request

504cd71

• Inference is currently only implemented for short-form i.e. audio is pre-segmented into <=30s segments. Long-form (including timestamps) will be implemented in a future release.

When would the ETA be for this?

Motivation

Whisper is not usable for long audio of speech, or for chunking audio based on timestamps determined by the ASR.

Your contribution

Guidance/PR in longer term future if not picked up by others in the next month or so

[sgugger]

cc @sanchit-gandhi and @ArthurZucker

[sanchit-gandhi]

Hey @JeffreyWardman! I believe @ArthurZucker has started looking into this, see #19490 (comment) for context!

[JeffreyWardman]

Thanks @sanchit-gandhi! By the looks of it, it would still be missing the timestamps. This is quite an important feature for me. I'm not completely familiar with the underlying code for huggingface. How does the chunking work? Does it calculate the first break between words after a given duration?

[sanchit-gandhi]

cc @ArthurZucker who knows more about timestamp generation!

This blog highlights quite nicely how chunking works in Transformers: https://huggingface.co/blog/asr-chunking

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[kurianbenoy-sentient]

Does whisper implementation of hugging support timestamps to generate SRT files like openai/whisper implementation?

https://github.com/openai/whisper/blob/main/whisper/utils.py#L64

[ArthurZucker]

Not yet! Working on this you can follow #20620 !