How to simplify text corrections especially when using word timestamps?

[g3grau]

I just created subtitles for a bunch of children videos to simplify learning. It's a really wonderful tool and produces very good results (testing with German) which require only a handful of fixes, mostly with spelling of names or fantasy words, but sometimes missing obvious words for some unexpected choices (I guess that the large model would already fix that).
For a grown-up those fixes may not be necessary at all, but for learning purposes I want the text to be correct. Without word timestamps, fixing those errors by editing the vtt file is fairly fast and straight forward.

However, after enabling timestamps the required fixes multiply and I'm wondering if/how this flow could be improved. Is it possible to keep the "most likely" words along with the approximate time stamps, pass it to an editor which allows to select a choice or add a correction before the actual text file with word timestamps is written? Maybe a json file with time data and recognition candidates along with their likelihood could be a good intermediate output for postprocessing. It could also help to build a map to merge alternating spellings to a consistent one, e.g. Schmit vs. Schmidt, Tomas vs. Thomas etc.

P.S. Side note for running whisper locally (since the error messages are not that obvious), hope it helps someone:
nvitop was helpful to figure out that all opened windows on my laptop occupied almost 750MB VRAM.
I can run the medium model on my 6GB GPU when closing browser and many other windows, and after setting

export LD_LIBRARY_PATH="/usr/local/cuda-11.7/lib64:/usr/lib/python3.10/site-packages/nvidia_nccl_cu11-2.14.3-py3.10-linux-x86_64.egg/nvidia/nccl/lib/"
# RuntimeError: Unable to find a valid cuDNN algorithm to run convolution   -> might be memory issue:
export PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:64"
whisper "$infile" --model medium --word_timestamps True --highlight_words True --output_format vtt

[EtienneAb3d]

@g3grau
If I understand your question in the right way, you may edit your pure-text output without timestamps, and then resync timestamps with the fixed text using:
https://github.com/EtienneAb3d/WhisperTimeSync
;-)

[phineas-pta]

keep the "most likely" words

this should be possible but require advanced modifications to source code (file whisper/decoding.py) to get other candidate output tokens (rather the default which has the highest confidence score - argmax() in the source file)

[g3grau]

Hi @EtienneAb3d,
thanks for the quick reply! That may already solve it, although it seems to care for the phrases, not the individual words (underligning)? I had some trouble to run it, I'll need some time to figure out the reason.

Hi @phineas-pta, good to know. I guess it could be a useful extension, though.

I'll first try to get Etiennes code running and see how far I get. Thanks again :)

[g3grau]

Hi @EtienneAb3d,
ok, getting closer :) I forgot that word_timestamps and highlight_words are different things. Timestamps indeed work and I can align it with a corrected plain text file using your java code. That's already helpful, thanks!

If I input an srt with highlighted words (using whisper with --word_timestamps True --highlight_words True), it gets confused. That may be a relatively small step as the input essentially contains copies of each sentence with a moving tag pair for different time stamps .. but Java is not my language :-S
If I understand the mechanism right, I can perhaps create a modified input srt which contains only the highlighted word for each timestamp, run your java, and then modify the output txt.srt and add a bit of context and the tags around the word (unless breaking down into single words prevents mapping to the plain text).

[EtienneAb3d]

Can you put a file example here?

[g3grau]

Try this one in German: Peppa
Bad example for text edits, almost nothing to fix from the medium model ;)
If I run whisper with --word_timestamps True --highlight_words False, the output from WhisperTimeSync is fine (though I think it missed an added sentence in the middle of the txt which whisper "forgot" to transcribe).
If I run whisper with --word_timestamps True --highlight_words True, the original srt has more timestamps to underline each word as it is spoken. This input makes it go nuts.
peppa.tar.gz

[EtienneAb3d]

@g3grau
Your goal is to get the good timestamps over the proper corrected text. Right?
So, I do not understand why you want to use such a "highlight_words" SRT, with all sentences repeated at each word?
What is the idea behind this?

[EtienneAb3d]

@g3grau
Forget my last post... I think I understand, now.
I will study a way to process such a "highlight_words" SRT file...