Utilizing Whisper for Audio Segmentation into Sentence Pieces

[comtomru]

What can be used with Whisper to be utilized to separate an audio file with spoken language into audio files containing specific sentences?

[glangford]

You could do this in 4 steps:

• transcribe the file using whisper using --word_timestamps True and output json
• read the json and collect the transcription into a document, and store the start and stop times of each word
• process the document using an NLP toolkit (spaCy or nltk for example) to break the document into sentences, recording the start time of the first word and the end time of the last word for each sentence
• use ffmpeg with the sentence start-stop times to extract the audio of the original file into individual audio fragments

[dgoryeo]

Hi @glangford , would you have a sample code on how to do step 3? Thanks.

[maks554]

run
whisper --language ru --model medium -o ./result -- 2out.wav --word_timestamps True
and got
--word_timestamps: No such file or directory

what should I do to remove the error?

[glangford]

Hi @maks554 --word_timestamps was added in a recent version (20230314) so probably you need to upgrade to that version.

[comtomru]

Thank you for your replies. After about 10 minutes, I found what I was looking for. I wanted to create an Anki deck containing sentences and audio. Since I can already transcript speech to text (a subtitle file with timestamps), I can easily achieve my goal with a tool named Subs2SRS that is specially designed for Anki. Here is the link: https://subs2srs.sourceforge.net/

[ryanheise]

spaCy is probably overkill if you just need sentence segmentation. What you can do instead is use the Whisper options --prepend_punctuations '' --append_punctuations '' which force punctuation symbols to be treated as separate words.

In your code, you're basically going to have a variable current_sentence in which you'll collect a list of words in the current sentence. The algorithm would proceed by looping over every word in the JSON transcript adding each word to current_sentence. Whenever you encounter a sentence-ending punctuation word (i.e. ".", "。", "!", "！", "?", "？") , it means you've reached the last word in the current sentence. It is time to observe the start timestamp of the first word and the end timestamp of the last word, and pass those timestamps to ffmpeg to extract that part of the audio into its own file(*). Finally, clear current_sentence, and you're ready to do it again for the next sentence, and the next, and the next, all the way to the end of the transcript.

(*) Actually, I think ffmpeg wants the start timestamp and the duration (which you can compute by lastWordEnd - firstWordStart.

(If you copy and paste that description into ChatGPT and ask it to write that as a Python program, it'll probably give you something halfway decent.)

[dgoryeo]

@glangford , I would appreciate sample code for the spaCy approach. My source language is in Japanese so I'm hoping that spaCy can do a decent job on that --many of the proposed approaches in the community work for western languages but not exactly a good fit for Asian languages.

[ryanheise]

If your use case is Japanese, the simple approach will work perfectly (that's also my use case, actually). The 。！？ characters are used only to indicate the end of the sentence. And in particular, "。" does not have other uses such as decimal points or abbreviations, so you can be confident that whenever you encounter a "。" that you've really reached a sentence boundary.

In Japanese, numbers are typically represented in Arabic numerals, but conveniently Whisper will break down a number like 2.5 as "2" and ".5". Abbreviations are another matter, although again conveniently those are not an issue for Japanese.

(Note that I am actually using spaCy myself, but not because I needed it for sentence segmentation, but rather because I needed word segmentation. If you don't require word segmentation and just care about the whole sentence, spaCy is a bit of a heavyweight solution. On the other hand, if you think you might need word segmentation or other spaCy features down the line, it wouldn't hurt to start using spaCy from the outset.)

[glangford]

@dgoryeo Here is a code fragment adapted from a different project. It has only been used for English and is an excerpt of work in progress, so apologies for the rough code. Hopefully it is helpful.

def offset_whitespace(s):
    return(len(s) - len(s.lstrip()))

def load_whisper_json(file):
    doc_timing = {}
    doc_text = ""
    js = open(file)
    jsdata = json.load(js)
    for s in jsdata["segments"]:
        for word_timed in s["words"]:
            # print(word_timed['start'], word_timed['end'], word_timed['word'])
            word = word_timed['word']
            if len(doc_text) == 0:
                word = word.lstrip() # remove any leading whitespace from first word in Whisper
                start_index = 0
            doc_text += word 
            start_index = len(doc_text) - len(word) + offset_whitespace(word) # align the timing index with the spaCy token index
            doc_timing[start_index] = (word_timed['start'], word_timed['end'])
    return doc_text, doc_timing

The idea is to create a document of the text with a timing dictionary which are in sync. The timing dictionary is a map from character index to a (start, stop) tuple, which is the timing generated using --word_timestamps. In English text, whisper often (but not always) has whitespace prior to the actual word. So offset_whitespace is used to align to the first character of the word. This makes it consistent with spaCy's idea of a Token.

When iterating through a document by sentences in spaCy, it will return a Span for each sentence. The idx of the first Token in the span will be the key into the timing dictionary. The idx of the last token may not be in the timing dictionary if it is punctuation, for example. So backing up from the last token in the sentence is needed to find the end time for a Span.

So, skipping over the spaCy setup and model loading you can do something like this to get the boundary for a sentence:

for sent in doc.sents:
       start_token = sent[0]
       end_token = sent[-1]
       end_token = sent[-2] if end_token.is_punct else sent[-1]
       start_time, _ = doc_timing[start_token.idx]
       _, end_time   = doc_timing[end_token.idx]

More flexibly, to get to the last token which has timing in whisper,

while end_token.is_punct or not timing.get(end_token.idx, None): # back up in tokens to get timing entry
       end_token = end_token.nbor(-1)

[dgoryeo]

@glangford , @ryanheise , thank you both for your help. Much appreciated!

My usecase is for transcription of long-form Japanese anime videos. I have a two-fold dilema:

(a) I get a rather close transcription when using a VAD and Whisper with well tuned hyper-parameters. But the output still has quite a few lines with wrong-times. They have wrong start time, and wrong duration. However the endtime is almost always correct. So somehow I need to find a way to adjust the wrong-timed lines by keeping the end-time fixed and recalculate the duration and start-time.

(b) I have noticed that an approach used by WhipserHallu produces closest transcription, but it looses the time-stamps. It would be ideal to find a way to map back the transcription to right timestamps.

[ryanheise]

Timestamps should be improved with #1114 when setting --word_timestamps True.