Timecode per/ word a future option?

[samelie]

Right now we get a range printed , eg: [00:00.000 --> 00:05.720]. Would be great to have a way to print out the timecode next to the word. This would unlock lots of interactive behavior.
Thank you for releasing this achievement

[RaulKite]

It will be a killer option to use in may many cases

[kospl]

name one use case for this feature

[DAVIDSystems]

audioediting with text, subtitling with word highlighting and many more.

[DAVIDSystems]

[jongwook]

(Duplicate of #3) Getting word-level timestamps are not directly supported, but it could be possible using the predicted distribution over the timestamp tokens or the cross-attention weights.

Currently, the predicted timestamps tend to be biased towards integers, and there are some failure modes where the timestamps can be constantly shifted, making reliable word-level timestamp prediction difficult. Once this is solved by us or the community, I agree that it'd be a great addition to this repo.

[melindadevins]

When transcript is displayed on video, the word being spoken is often highlighted. The word level timestamp makes it possible. Really appreciate it if the timestamp can be obtained.

[jayn1985]

word level timestamp is really helpful in many cases, hope it can be supported in the later version.

[edgartaor]

Could something like this work?

https://nvidia.github.io/OpenSeq2Seq/html/speech-recognition/speech-to-text-align.html

[sandalwoodsh]

I agree that such a feature would be tremendously awesome!

[jongwook]

I've made a demo of obtaining word-level timestamps using the cross-attention patterns in the multilingual ASR notebook

[octimot]

@jongwook Thanks for this update! It seems to work pretty well, but I'm having trouble scaling it up to audio longer than 30 seconds:

The call whisper_model(mel.unsqueeze(0), tokens.unsqueeze(0)) triggers IndexError: index out of range in self in
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

What would be the best approach to make it take each 30sec window at a time? Would you mind giving a rough example?

Cheers!

[RaulKite]

Have you tried this repo?

https://github.com/linto-ai/whisper-timestamped

It is based in @jongwook notebook and have advanced in that way

[octimot]

@RaulKite
Yes, thank you!

I believe whisper-timestamped is doing the word-alignment on the fly which is cool, but I'd like to understand how to scale the basic implementation that @jongwook proposed, just to have a bit more flexibility.