A possible solution to Whisper hallucination #679
Replies: 31 comments 134 replies
-
|
Thanks @KaiserChr #current working name for a threshold to determine permissible chunk length for a healthy transcript
lucid_threshold = 0.3
#first chunk, ergo no context or next chunk will be fully within num_frames
if ((seek+N_FRAMES)/num_frames < 1.0) or (seek == 0):
if "prompt" in decode_options:
decode_options["prompt"] = all_tokens[prompt_reset_since:]
else: #next chunk is not first chunk and will not be fully within num_frames i.e. last chunk, calculate lucid_score
lucid_score = (num_frames - seek) / N_FRAMES
if lucid_score < lucid_threshold and "prompt" in decode_options:
decode_options["prompt"] = []
else:
decode_options["prompt"] = all_tokens[prompt_reset_since:] |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much! |
Beta Was this translation helpful? Give feedback.
-
|
Small update: I further debugged and updated the code in the initial post, so the one you kindly formatted for me (looked up how to do it better myself in the meantime) is deprecated. |
Beta Was this translation helpful? Give feedback.
-
|
Well done @KaiserChr! What is the right step to suggest merging this into main Whisper code? Would be great to get a solution like this as early as possible. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the kind words! |
Beta Was this translation helpful? Give feedback.
-
|
I tried using this solution: But the output is still the same: I'm doing something wrong?
|
Beta Was this translation helpful? Give feedback.
-
|
Have you tried setting 'condition_on_previous_text' to False? If with this setting the problem still persists, my (possible) fix unfortunately wont help, bc its just a way to do the same thing, not generally but only for chunks that, as to my current understanding, may cause the problem. My working hypothesis is that, when the problem is underdefined (in that there is too little information provided by the encoder) the decoder may show an aberrant behavior. Especially for very short or very quiet chunks this seems to be the case. I am still trying different approaches to push the model "over the brink" in a reproducible manner to verify my hypothesis. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I didn't set
in the previous example. I have now launched Whisper with:
but the result is still not good: Thank you for your effort |
Beta Was this translation helpful? Give feedback.
-
|
In my shop we hacked together a python script to clean up the vtt output to make it a bit more normative (always add 00: hour timestamps), add cue IDs, shorten the line lengths, add some NOTE metadata and misc other changes. For this problem we read the file into the webvtt library and compare the text of each cue to the previous. If they are exactly equal, we drop the current and move on to the next. Not perfect by any stretch and you still will have the 1st ghost but it beats having the endless repeats. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @whicks1 , by any chance would you be able to share your script? |
Beta Was this translation helpful? Give feedback.
-
|
question: couldnt it detect really poor transcription but just summarizing all of the match probabilitiea and if it is less than 75%, throw out the transcription? i am trying to find a way that if the recording is really bad, instead of just making stuff up it returns nothing. |
Beta Was this translation helpful? Give feedback.
-
|
The best way is to split the audio file up into audio slices using VAD and feeding into whisper. There is a parameter "condition_on_previous_text", set it to false to force it to forget, the problem is that when this parameter is true, it remembers what it output previously and if the current output cannot produce anything, it will just use the last output. With the parameter set to false, it will forget and if it cannot decipher the output it will just output blank. But I find that it is still best to just transcribe slices of audio, especially if it is conversational, because every statement itself will express an idea unlike long essays when the whole paragraph expresses an idea. Sometimes being able to infer from the previous statement may not be the best idea for conversations. Remember whisper is trained using mostly subtitles from youtube, so you will get funny outputs sometimes. |
Beta Was this translation helpful? Give feedback.
-
|
thx, testing it out now! |
Beta Was this translation helpful? Give feedback.
-
|
@silvacarl2 , take a look at this implementation with VAD: it does a reasonably good job: |
Beta Was this translation helpful? Give feedback.
-
|
@dgoryeo Does this work with 2 and 3 seconds of audio files? because I'm making real-time transcription. |
Beta Was this translation helpful? Give feedback.
-
|
@iamironman0 , I don't believe so --I think one shouldn't need VAD for any short audio segments. |
Beta Was this translation helpful? Give feedback.
-
|
@dgoryeo thanks for replying. however is there a solution for short audio files? |
Beta Was this translation helpful? Give feedback.
-
|
yeah this is a riot, trying it against a few difficult cases now./ |
Beta Was this translation helpful? Give feedback.
-
|
options = {"task":"transcribe","language":"English","fp16":fp16,'no_speech_threshold':0.1, "condition_on_previous_text": False, "logprob_threshold": -1.00, "without_timestamps":True} |
Beta Was this translation helpful? Give feedback.
-
|
that might work. the other idea i had was if the overall average of the transcription words is less than say 65%, it is probably wrong or somewhat wrong. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
fp16 is boolean, it can be set to true only if running on GPU, otherwise set it to false. It is more to save memory on the gpu if your gpu has little memory. If you set to true and running on CPU, it will still default to fp32 but will throw out some warning message to say that fp16 is only available on GPU. no_speech_threshold is to tell whisper that as long as whisper thinks there is 10% chance of no-speech, just treat it as no speech. Too high a value and whisper will think there is speech it cannot figure out and it will just throw some random rubbish. I find it good to set a low value especially if the audio file has music or lots of noise. logprob_threshold, I can't remember that this really means, it has been sometime since I spent time tweaking the parameters. I think it has to do something with the logmel processing. |
Beta Was this translation helpful? Give feedback.
-
|
I have uploaded my code to https://github.com/fleek/VADtransciber, you can give it a try. |
Beta Was this translation helpful? Give feedback.
-
|
cool, will have a look at that. |
Beta Was this translation helpful? Give feedback.
-
|
as soon as i get a chance to test this, i will post here. we have 10M samples to work from LOL. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I looked into the problem some more and found out a couple of things: kwargs['language'] = 'de'
kwargs['verbose'] = True
kwargs['task'] = 'transcribe'
kwargs['temperature'] = 0
kwargs['best_of'] = None
kwargs['beam_size'] = None
kwargs['patience'] = None
kwargs['length_penalty'] = None
kwargs['suppress_tokens'] = "-1"
kwargs['initial_prompt'] = None
kwargs['condition_on_previous_text'] = False
kwargs['fp16'] = True #for GPU
kwargs['compression_ratio_threshold'] = 2.4
kwargs['logprob_threshold'] = -0.5
kwargs['no_speech_threshold'] = 0.2seem to work really well IF you use Cuda (for medium model), as the performance on CPU was so slow that the queue was filling up in an unacceptable manner, because the model sometimes tries to understand a silent part of the audio multiple times (a behavior I tried to curb with the parameters given above), and without GPU that is just too slow. Further debugging makes me believe the decoder is the source for this slowdown, as the With GPU this loop is so fast that no significant slowdown is noticable, albeit at a large hardware cost. Once we have a working version with Silero I plan to put the code we use up here on Github so anyone interested can take a look. Please note that this version does not require the code I posted on the start of the Thread, as with a VAD typically utterances are much less than 30 seconds and therefore its seems more stable to deactivate |
Beta Was this translation helpful? Give feedback.
-
Try cmder, it is the only shell that I use |
Beta Was this translation helpful? Give feedback.
-
|
Yes, under cmd and cmder VADtranscriber working without errors in terminal. But if i run VADtranscriber with cuda parameter process is handled by the CPU, not the GPU. I dont know why... Whisper on GPU working correctly Second problem is that the json output is created but does not contain any words en_temp.wav is converted correctly and contains voice. |
Beta Was this translation helpful? Give feedback.
-
|
Run cmder as administrator help. but the output file is still empty: I will probably look for the problem in:
|
Beta Was this translation helpful? Give feedback.
-
|
DLL library exists, I have no idea why the problem occurs |
Beta Was this translation helpful? Give feedback.
-
|
OK. Working. Only speaker identification is not perfect |
Beta Was this translation helpful? Give feedback.
-
|
I am sorting out the worst case recordings we have to be able to test this, will post back when done. |
Beta Was this translation helpful? Give feedback.
-
|
I also think there must be a top_p parameter somepalce, which could be lowered a bit to help as well. |
Beta Was this translation helpful? Give feedback.
-
top_p as in ? probability setting? |
Beta Was this translation helpful? Give feedback.
-
|
correct. it may not be possible be we are also looking at the inference side of whisper to see which settings could be optimized. |
Beta Was this translation helpful? Give feedback.
-
|
I looked into the best combination of parameters some more and right now find utterance splitting via a vad is the most stable solution for clean chunking. The current version of the code I use for real time transcription in german is using gpu and the medium model, as this gives a very nice combination of speed and precision. Find the code attached below if you want to try it for yourself! # asr using whisper and Silero-VAD (https://github.com/snakers4/silero-vad)
# structure based on the very nice work of Oliver Guhr over at https://github.com/oliverguhr/wav2vec2-live
import pyaudio
import numpy as np
import threading
import time
from sys import exit
from queue import Queue
import matplotlib.pylab as plt
import wave
import whisper
import struct
import multiprocessing
import torch
filename = 'audio_provided.wav' #for Debugging: save the audiostream that was provided to whisper after sending through queue
filename_orig = 'audio_recorded.wav' #for Debugging: save the audiostream that was actually recorded pre sending.
class Realtime_Whisper():
exit_event = threading.Event()
def __init__(self, model_name, device_name="default"):
self.model_name = model_name
self.device_name = device_name
def stop(self):
"""stop the asr process"""
Realtime_Whisper.exit_event.set()
self.asr_input_queue.put("close")
print("asr stopped")
def start(self):
"""start the asr process"""
manager = multiprocessing.Manager()
self.asr_output_queue = Queue()
self.asr_input_queue = Queue()
self.visualization_input_queue = manager.Queue() #currently not used, the queue is still in for convenience...
self.asr_process = threading.Thread(target=Realtime_Whisper.asr_process, args=(
self.model_name, self.asr_input_queue, self.asr_output_queue,))
self.asr_process.daemon = True
self.asr_process.start()
time.sleep(5) # start vad after asr model is loaded
self.vad_process = threading.Thread(target=Realtime_Whisper.vad_process, args=(
self.device_name, self.asr_input_queue, self.visualization_input_queue, ))
self.vad_process.daemon = True
self.vad_process.start()
#Debug optional visualization
#self.visualization_process = multiprocessing.Process(target=Realtime_Whisper.plot_stream, args=(
# self.visualization_input_queue,))
# self.visualization_process = threading.Thread(target=Realtime_Whisper.plot_stream, args=(
# self.visualization_input_queue,))
#self.visualization_process.daemon = True
#self.visualization_process.start()
def int2float(sound):
"""convert the wav pcm16 format to one suitable for silero vad"""
_sound = np.copy(sound) # may be not necessary
#abs_max = np.abs(_sound).max()
abs_max = 32767
_sound = _sound.astype('float32')
if abs_max > 0:
_sound *= 1 / abs_max
_sound = _sound.squeeze() # depends on the use case
return _sound
def plot_stream(instream):
"""plot audio stream via matplotlib"""
CHUNK = 160
CHANNELS = 1
RATE = 16000
fig, ax = plt.subplots()
x = np.arange(0, 2 * CHUNK, 2)
line, = ax.plot(x, np.random.rand(CHUNK), 'r')
ax.set_ylim(-20000, 20000)
ax.set_xlim(0, CHUNK)
fig.show()
while True:
data = instream.get()
dataInt = struct.unpack(str(CHUNK) + 'h', data)
line.set_ydata(dataInt)
fig.canvas.draw()
fig.canvas.flush_events()
def vad_process(device_name, asr_input_queue, vis_input_queue):
"""voice activity detection using silero-vad"""
model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
model='silero_vad',
force_reload=False,
onnx=False)
(get_speech_timestamps,
save_audio,
read_audio,
VADIterator,
collect_chunks) = utils
#not sure this is useful, but I leave it in for now...
vad_iterator = VADIterator(model)
audio = pyaudio.PyAudio()
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000
FRAME_DURATION = 60
CHUNK = int(RATE * FRAME_DURATION / 1000)
SPEECH_PROB_THRESHOLD = 0.2 # This probably needs a bit of tweaking
microphones = Realtime_Whisper.list_microphones(audio)
selected_input_device_id = Realtime_Whisper.get_input_device_id(
device_name, microphones)
print('input device id')
print(microphones)
print(selected_input_device_id)
#this should be the default mic, tweak as needed ...
selected_input_device_id = 1
stream = audio.open(input_device_index=selected_input_device_id,
format=FORMAT,
channels=CHANNELS,
rate=RATE,
input=True,
frames_per_buffer=CHUNK)
#framebuffer for queue
frames = b''
#masterframebuffer for saving the data send to asr
masterframes_asr = b''
last_speech_prob = 0
while True:
if Realtime_Whisper.exit_event.is_set():
break
frame = stream.read(CHUNK, exception_on_overflow=False)
frame_tensor = torch.from_numpy(Realtime_Whisper.int2float(np.frombuffer(frame, dtype=np.int16)))
speech_prob = model(frame_tensor, RATE).item()
#turn this on for debugging and tweaking the threshold...
#print(speech_prob)
#accumulate frames in frame buffer if speech is detected and the total length is < 30 sec (max size of whisper chunk)
if speech_prob > SPEECH_PROB_THRESHOLD and len(frames) < 480000: #THIS NEEDS TO BE LOOKED AT AGAIN MAYBE A FULL 30s WHISPER CHUNK IS TOO MUCH
frames += frame
#if there was speech and now there is none (i.e. an utterance has finished or the max length is exceeded, write to queue
elif (speech_prob <= SPEECH_PROB_THRESHOLD < last_speech_prob) or (len(frames) >= 480000):
asr_input_queue.put(frames)
masterframes_asr += frames
frames = b''
last_speech_prob = speech_prob
stream.stop_stream()
stream.close()
audio.terminate()
# Open and Set the data of the WAV file
file = wave.open(filename_orig, 'wb')
file.setnchannels(1)
file.setsampwidth(2)
file.setframerate(16000)
# Write and Close the File
file.writeframes(b''.join(np.frombuffer(masterframes_asr, dtype=np.int16)))
file.close()
def asr_process(model_name, in_queue, output_queue):
"""transcribe using whisper"""
model = whisper.load_model(model_name, device='cuda') #use cuda for everything > base model
#with current settings always excepts to 0, but left in to play around with the setting...
temperature_increment_on_fallback = 0
temperature = 0
try:
temperature = tuple(np.arange(temperature, 1.0 + 1e-6, temperature_increment_on_fallback))
except:
temperature = 0
kwargs = {}
kwargs['language'] = 'de'
kwargs['verbose'] = True
kwargs['task'] = 'transcribe'
kwargs['temperature'] = temperature
kwargs['best_of'] = None
kwargs['beam_size'] = None
kwargs['patience'] = None
kwargs['length_penalty'] = None
kwargs['suppress_tokens'] = "-1"
kwargs['initial_prompt'] = None
kwargs['condition_on_previous_text'] = False # seems source of false Transcripts
kwargs['fp16'] = True #set false if using cpu
kwargs['compression_ratio_threshold'] = None #2.4
kwargs['logprob_threshold'] = None #-1.0 #-0.5
kwargs['no_speech_threshold'] = None #0.6 #0.2
#masterframes = ''
masterframes = b''
while True:
audio_file = in_queue.get()
if audio_file == "close":
break
print("\nlistening to your beautiful voice\n")
masterframes += audio_file
audio_tensor = torch.from_numpy(Realtime_Whisper.int2float(np.frombuffer(audio_file, dtype=np.int16)))
result = model.transcribe(audio_tensor, **kwargs)
if result != "":
output_queue.put(result["segments"])
# Open and Set the data of the WAV file
file = wave.open(filename, 'wb')
file.setnchannels(1)
file.setsampwidth(2)
file.setframerate(16000)
file.writeframes(b''.join(np.frombuffer(masterframes, dtype=np.int16)))
file.close()
def get_input_device_id(device_name, microphones):
for device in microphones:
if device_name in device[1]:
return device[0]
def list_microphones(pyaudio_instance):
info = pyaudio_instance.get_host_api_info_by_index(0)
numdevices = info.get('deviceCount')
result = []
for i in range(0, numdevices):
if (pyaudio_instance.get_device_info_by_host_api_device_index(0, i).get('maxInputChannels')) > 0:
name = pyaudio_instance.get_device_info_by_host_api_device_index(
0, i).get('name')
result += [[i, name]]
return result
def get_last_text(self):
"""returns the text, sample length and inference time in seconds."""
return self.asr_output_queue.get()
if __name__ == "__main__":
print("Live ASR")
#param is model size
asr = Realtime_Whisper("medium")
asr.start()
last_text = 'Start'
try:
while True:
lastresult = asr.get_last_text()
for segment in lastresult:
print('ID: ' + str(segment['id']) + ' START: ' + str(round(segment['start'], 1)) + ' END: ' + str(round(segment['end'], 1)) + ' TEXT: ' + segment['text'])
except KeyboardInterrupt:
asr.stop()
exit()
|
Beta Was this translation helpful? Give feedback.
-
|
@KaiserChr , thanks for the findings. Do I read it right that by default you set both beam and temperatiure to zero, i.e deterministic? |
Beta Was this translation helpful? Give feedback.
-
|
This is correct, as the medium model seems to perform well with it. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure this is the same root cause, but besides the duplicates everything starting at 28:26:00 is total rubbish (translates to "subtitles on behalf of ZDF"). Obviously the source audio does not include someone saying anything about subtitles. I used the small model on CLI. Haven't tried --condition_on_previous_text yet, but trying the medium model now. |
Beta Was this translation helpful? Give feedback.
-
|
It contains actual dialogue. Running the transcribe on the same file earlier worked fine (using the same config) and resulted into a correct result. Feels like this behavior is caused by temporary memory issues or something? The previous run also did not produce these duplicate lines (26:59, 27:59 and 28:07). |
Beta Was this translation helpful? Give feedback.
-
That's the exact behaviour of AI, you can run through the same audio with the same config 200 times and each time will yield different results, it is kind of like the mood of the AI, ahahaha |
Beta Was this translation helpful? Give feedback.
-
|
I don't mind slightly different results, but this just seems broken 🤷♀️ |
Beta Was this translation helpful? Give feedback.
-
I can't remember where I saw it, but if you can retain the same seed value, you can guarantee same results. |
Beta Was this translation helpful? Give feedback.
-
|
I can confirm this behaviour for German language videos ending with music or silence. Transcripts end with lines like "Copyright WDR 2021 Untertitel im Auftrag des ZDF, 2017 Untertitel im Auftrag des ZDF, 2017" or "Untertitel der Amara.org-Community.." that have no basis in any actual utterances, but are apparently artifacts from subtitle files the model was trained on. |
Beta Was this translation helpful? Give feedback.
-
|
My small contribution to this subject: each of the following reduces the amount of "hallucinated texts" (but the problem still occurs).
|
Beta Was this translation helpful? Give feedback.
-
|
100% agree with you. sometimes it will transcribe audio into "music in background", or silence silince silnece, or many times Thanks for Watching! |
Beta Was this translation helpful? Give feedback.
-
|
But to be clear in most of the thousands of test cases we have done, it outperforms all commerical speech to text APIs we have tested with. |
Beta Was this translation helpful? Give feedback.
-
|
Of course, I agree. But, in fact, this question of "best or not best" is not a real subject. It's an evidence that Whisper is better. But, this hallucination issue is a real problem, especially for the final user, not aware of technical considerations, only focusing of his own evidence "it's clear and simple, why the result is not the good one". Above ideas can limit the disappointment, in short:
|
Beta Was this translation helpful? Give feedback.
-
|
I found this thread by searching for "Thanks for watching"! I've been capturing voice for long periods of time with stuff based off #608 and I see soooo much "Thanks for watching" it is funny example from some period of silence or typing: I look forward to trying some things from this thread. What would also be effective is |
Beta Was this translation helpful? Give feedback.
-
|
@digitalsignalperson we plan to do something like that at the office. The issue is there is ~100 languages and we need to find all the same kind of biases (see discussion #928 ) for all languages |
Beta Was this translation helpful? Give feedback.
-
|
agreed with all of your points. we have a ton more testing to do, we have 10M samples. |
Beta Was this translation helpful? Give feedback.
-
|
Loudness normalization may also be useful, by adding PS: updated. With ffmpeg multiple filters should be provided at once comma separated. |
Beta Was this translation helpful? Give feedback.
-
|
Do I understand correctly that ffmpeg pure-silence filtering will trim chunks of the audio and therefore affects the timing/sync of the transcription? If that is the case, is there a way to reconstruct the timing/sync? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it will remove silence parts from the audio file, thus change the timing. |
Beta Was this translation helpful? Give feedback.
-
|
I'm wondering may be one approach would be to use something like this:
But then again, wouldn't a VAD achieve the same outcome? |
Beta Was this translation helpful? Give feedback.
-
Certainly, it should. But, in my experiment, Silero VAD doesn't remove them all properly, especially at beg/end parts, where the observed hallucinations are more likely to occur. |
Beta Was this translation helpful? Give feedback.
-
|
As said above, hallucinations are certainly due to bad sync between sounds and texts in subtitle training data (and copyright added to them). Thus, they are more probably at the beg or end of the output. I observed that hallucinations are very depend on very particular sound configurations. Certainly, if the input is changed a bit, it is highly probable that the hallucination disappears. My new idea is to add markers, easily recognized by Whisper, at the beg and end of the sound. If the markers are well obtained in the beg and end of the output, simply remove them. If not, something was wrong, certainly an hallucination added, try the same inverting the markers (to try something a bit different). If the markers are still not properly obtained in the output, try with the original sound (to avoid trouble possibly added by the markers). If timestamps are needed, it's easy to restore them knowing the time length of the markers. Here is the code: Used markers (GitHub does not accept attached WAV files): |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @EtienneAb3d , wow, this sounds promising. And thanks for the tip on artificial voices. I was just going ahead to use one of those TTS generators for my target language (Japanese). Do I undertand correctly that you insert the markers in each VAD cuts/chunks, right? |
Beta Was this translation helpful? Give feedback.
-
|
I do not chunk the sound, I just remove the noise and blank parts (to get only one file to be processed). In my application (see NeuroSpell), speakers are often making a pause in their dictations to think about what they need to say, and then they are continuing the same sentence. But, yes, if you are chunking the sound (to get several files), you need to add markers before/after each chunk. Additional tip: to get a better recognition of the markers above, you may add "Whisper, ok." and "Ok, Whisper" in the The initial_prompt should not be longer than about 100 words (or it will be silently truncated by Whisper). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @EtienneAb3d . I'll give it a try during the week, will let you know of the result. Happy Sunday! |
Beta Was this translation helpful? Give feedback.
-
|
@EtienneAb3d do you have a modified whisper script that we can use your code directly? |
Beta Was this translation helpful? Give feedback.
-
For the marks method, in short:
|
Beta Was this translation helpful? Give feedback.
-
|
Comparing whisper.cpp to this repo : I took an audio file with no noise at the end and concatened itself to it (so same file *2 in other words), works correctly with whiper.cpp, hallucinations with whisper openai python repo Edit : We tested on a much longer audio files, there is MORE hallucinations so they didn't fix it |
Beta Was this translation helpful? Give feedback.
-
|
These hallucinations are highly random, so, it's not really surprising you are experimenting different results with both code versions. |
Beta Was this translation helpful? Give feedback.
-
|
@EtienneAb3d , back to your original script --removing silence by ffmpeg, would you have a script to recalculate the correct transcription time lines after cutting/trimming? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
It's possible I've read something about this: there is perhaps an ffmpeg option to output silence timestamps. Search for this on this forum, or on the ffmpeg documentation. |
Beta Was this translation helpful? Give feedback.
-
|
@dgoryeo If processing time is not a problem for you, here is perhaps the ultimate way to get a good SRT.
:-) |
Beta Was this translation helpful? Give feedback.
-
|
@skanda1005
@crecos |
Beta Was this translation helpful? Give feedback.
-
|
Hi @EtienneAb3d, I just tested WhisperHallu on several audio files and thought of sharing my observations so far:
The output of all tests were successful in that there were no hallucination and repetitive words. Well done !!!
I tested one of the files (duration 2min) with WhisperTimeSync. The instruction did say that the code is not ready for Japanese language but I just wanted to give it a try. The result was better than I expected --The code was able to align more than 80% of the subs correctly. The other 20% were either grouped wrongly with the previous line, or had a wrong timing --too early. In terms of missing dialogues, I suspect that lowering the VAD threshold might help. My guess is that since the audio is already de-silenced, and has gone through compressor, a lower VAD threshold would be sufficient. A quick question: I couldn't figure out why the code always picks up the standard Whisper and never the ct2-trasnformers. Is that because of the GPU flag? |
Beta Was this translation helpful? Give feedback.
-
|
Feel free to experiment with parameters, and give me your feedback. |
Beta Was this translation helpful? Give feedback.
-
This is because the current tokenizer is cutting text on spaces only. This is not sufficient for Asian texts. To get a more precise result, it should cut on words. |
Beta Was this translation helpful? Give feedback.
-
|
I released a new version of WhisperTimeSync where you specify the language. For Asian languages, it tokenizes on all chars. You should get a much more precise result. |
Beta Was this translation helpful? Give feedback.
-
|
A new version of WhisperHallu is available. It adds Deezer Spleeter to extract voices and eliminate noises. |
Beta Was this translation helpful? Give feedback.
-
|
@malcolmosh , how do you remove common errors? I'd be keen if you could share any of the code. Thnaks. |
Beta Was this translation helpful? Give feedback.
-
|
@dgoryeo - I'm curious why use both Demucs this Silero VAD. Aren't these doing equivalent things? |
Beta Was this translation helpful? Give feedback.
-
|
@jthompson22 |
Beta Was this translation helpful? Give feedback.
-
|
@malcolmosh , By any chance, can you share your code? I am looking for an open-source solution to this problem but haven't found anything yet. I am also using Pyannote's diarization pipeline with whisper-large-v2 transcription for my use case. I am looking for a solution to improve transcription quality by removing hallucinations. Thank you!! |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I can't share the code since we're building it for a client, but I can point you in the right direction. We've actually started using faster-whisper and its integrated VAD (Silero-VAD). Since faster-whisper is 4-10X quicker, we've been able to use the full default set of parameters (beam_size = 5, best_of = 5, temperatures = (0, 0.2, 0.4, etc.) in our pipelines. Using any VAD to remove non-speech segments and then passing the data through the accelerated model with all of the fallback options is almost a foolproof solution. We also tuned the no_speech_threshold and the compression_ratio_threshold. |
Beta Was this translation helpful? Give feedback.
-
works for me. |
Beta Was this translation helpful? Give feedback.
-
"condition_on_previous_text": False, will probably degrade quality |
Beta Was this translation helpful? Give feedback.
-
|
Is an "optimal" value for |
Beta Was this translation helpful? Give feedback.
-
|
Me during an online Blender class, hoping it didn't crash: Meanwhile, Whisper... 01:31:44.240 --> 01:31:44.880 01:32:14.240 --> 01:32:17.240 01:32:44.240 --> 01:32:47.240 01:33:14.240 --> 01:33:17.240 01:33:44.240 --> 01:33:47.240 01:34:14.240 --> 01:34:17.240 01:34:44.240 --> 01:34:47.240 01:35:14.240 --> 01:35:17.240 01:35:18.240 --> 01:35:21.240 01:35:22.240 --> 01:35:25.240 01:35:26.240 --> 01:35:29.240 01:35:30.240 --> 01:35:33.240 01:35:34.240 --> 01:35:37.240 01:35:38.240 --> 01:35:41.240 ... and so on. Language is Italian. Thank you, |
Beta Was this translation helpful? Give feedback.
-
|
with spanish add the same text |
Beta Was this translation helpful? Give feedback.
-
|
Noise removal + VAD + remove segments with low likelihood of speech is working for me. The latter was not mentioned in this thread and it was what has given me a significant improvement in the results, it is mentioned here #928 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
@rodrigoGA , what do you use for noise removal? I've just started looking into UVR5 models UltimateVocalRemoveGUI, but haven't found a way to automate the flow. |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone, this is the first time I'm commenting here but I've been following this topic for some time, and I believe I achieved a very good result in my code, I combined several techniques already mentioned in this topic and others that perhaps haven't been thought of yet. I apologize for some sentences in Portuguese but I believe I have translated all the variables into English, so everyone can understand the code. Anyone who wants to test it will have to adapt it to their own code as this is just an excerpt from my code, my idea is to show how I achieved a good result and almost 100% avoid Whisper's hallucinations and keep it fast enough to make transcriptions In real time. I want to thank the people at Faster Whisper for the excellent work converting it to fast16. class AudioTranscriber: The "speech_filter" function serves to filter some hallucinations, and hide the output, however with my latest modifications, this was no longer necessary. But whoever wants to use it, follow below. def filtro_de_fala(texto): |
Beta Was this translation helpful? Give feedback.
-
|
@ottokarr |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the WhisperHallu application. However, even though your explanations might be clear for a specialist (which I am not) I am unable to figure out how to write the code to import the Whisper AI model and give the right instruction for the transcription. If you have a few minutes, could you please give me a simple example of the sentence to write if, for example, I would like to make a written transcription of a russian video in using a Whisper AI «Large » model ? For your information, until now I was used to obtain srt and vtt files in writing the following : !whisper "PATH/RUSSIAN VIDEO.mp4" --task transcribe --model large --language ru Thank you for your time and consideration in this matter (if you got the time). |
Beta Was this translation helpful? Give feedback.
-
|
@ottokarr |
Beta Was this translation helpful? Give feedback.
-
|
@iichii1 |
Beta Was this translation helpful? Give feedback.
-
|
Normally yes, but I didn't have a good result with it, so I tested webrtc, and it also has a good processing speed, which is essential in my application. |
Beta Was this translation helpful? Give feedback.
-
|
There is a bug introduced with #1279, it can make bad hallucination loops. There is a bugfix -> #1903 [not merged at the moment] |
Beta Was this translation helpful? Give feedback.
-
|
just a quick thought: since this tends to happen on short audios, checking the duration per word might be a way to detect anomalies. for example, when i get:
for a recording of duration 0.128 second then it is definitely an hallucination. |
Beta Was this translation helpful? Give feedback.
and others
-
Hi all,
I was having trouble with whisper creating "ghost transcripts" at the end of a given sound file. These often consist of repeats and shuffles of the text of previous chunks and turned out to be quite detrimental to the overall quality of the transcript.
I looked into this a little and found a possible solution:
The parameter: 'condition_on_previous_text'
This is set to True by default and helps whisper (to my understanding) to keep the context going between chunks.
My working hypothesis is, that the problem arises if the last chunk is short (like a couple of seconds) compared to the text initializing the next chunks transcription. Then the models seems to somehow have a problem with disambiguation and starts "seeing things".
A more elegant solution than just setting condition_on_previous_text to False would be something like this (not properly debugged yet):
After line 178 of whisper/transcribe.py exchange:
decode_options["prompt"] = all_tokens[prompt_reset_since:]with something like this:
Im still tinkering and debugging, but so far it worked for me and I just wanted to post it in case it helps anybody else!
p.s.
I was encountering the problem when working with german audio on the medium and large models
EDIT: 14.12.2022 further debugged and formatted code a bit better
UPDATE: 21.12.2022 Approach using VAD, see my post below
Beta Was this translation helpful? Give feedback.
All reactions