Hallucinations - how to recognize and solve?

[nikola1975]

Hello. I am using Whisper API for some time now. It works very good for big languages and almost acceptable for small ones.

However, occasionally it hallucinates and as part of the transcription, it sends back repeated words or phrases. Sometimes, this can be one word repeated many times, other times it is few words one after the other and then repeated again (like a repeated phrase).

I am trying to recognize this pattern and in that case I would send the audio back to transcription again, as it usually does not repeat for the same audio.

I have tried two approaches:

1. Send the text to OpenAI GPT 3.5 API to recognize suspicious transcription. As I am doing it as part of the other action, this would not bring additional cost, but unfortunately, it is not able to report back consistently. I instructed it to return “[SUSPICIOUS]” if it believes there could be mistakes, but it does so for perfectly good transcriptions as well.
2. Recognize repeated words with Python function. If one word is repeated, it has worked ok, but the problem is with phrases. I need to cut the text and then recognize 2 repeated words, 3 repeated words, 4 repeated words. I am not even sure this will work. Is there more intelligent way than that?

Thanks,
Nikola

[phineas-pta]

it is possible to pre-process audios before sending to api?

[phineas-pta]

idk much about VAD, but silero vad & pyannote are open source, you can actually look at source code instead of wondering

2ndly, it's called voice activity detection, not silence detection, that's how it's different from volume-based detection with ffmpeg, non-silence non-speech segments also cause hallucination

on detecting repeated words with sed, good luck tinkering with regex + repetition thresholds, you'll soon find edge cases all over the place with each new transcript

[nikola1975]

@phineas-pta Yes, you are right about false positives and false negatives. I have created this function to decrease number of problems. At this moment, with the lack of proper solution, I will try to catch most problems - it will still be better then nothing. That was also the reason of my posting here, to see if there are some better ideas.

If I catch the problem, I am trying to transcribe it again - and if it happens once more, I will let the transcription as it is.

[phineas-pta]

the solution for now is to pre-process audios with VAD

if you want to process subtitles post-transcription, you can try alignment with other speech-to-text models like wav2vec2 as in https://github.com/m-bain/whisperX ; other not-NLP text processing not recommended to remove repetition

other interesting point is that when you re-transcribe, hallucinations disappear, maybe it's something special to openai api, because with local-pc (colab) whisper still hallucinates

[mrfragger]

on detecting repeated words with sed, good luck tinkering with regex + repetition thresholds, you'll soon find edge cases all over the place with each new transcript

yea you're correct it's almost impossible to filter repeating words like there are many many issues which would be legit, but at the end of the day it's not a transcript for a book per se so gotta just figure out what's worse. 100 instances of repeating words that shouldn't be or a few instances of repeating words that should but get taken out to a single case. Anyway here's the sed script for pro-nouns for whisper when it goes berserk

[phineas-pta]

you should properly write a script with variables/functions etc. because a one-line 35k-character command may be helpful to some non-technical users who want something ready-to-use, but it's difficult and painful to other people who want to modify/adapt to their needs

also you should use something like gist.github.com to share code, don't share zip (because of security concerns)