Model ignores the speakers in the audio except for one

[Boulaouaney]

I noticed a weird behaviour when transcribing an audio from a news report that has a couple of interviews.

the model transcribes the speech of the host decently, but completely skips the parts where the guests speak. some segments are over 40 seconds long and are completely absent from the transcript.

I am using large-v2 model for Korean language transcription.
I played around with using the vad filter and even an initial prompt that mentions how many speakers will be in the audio, but nothing fixed it yet.
any idea what could be the cause?

[guillaumekln]

This may be a general issue with the model.

Did you also try with openai/whisper?

[Boulaouaney]

yes it is the same issue with openai/whisper. I think it has to do with the fact that the audio has sections that are super loud.

I found this solution from HF transformers "Rescale layer in whisper processor"
so using do_normalize seems to help a bit.

I couldn't find an implementation of this in the faster-whisper repo so I did it manually using the code below and it helped.

audio, sr = librosa.load("../audios/03.mp3", sr=None)

if sr != 16000:
    audio = librosa.resample(audio, orig_sr=sr, target_sr=16000)

audio_data_normalized = (audio - np.mean(audio)) / np.std(audio)

if it is implemented and I missed it, please let me know how to use it.

[guillaumekln]

We do the same as openai/whisper and normalize the audio between -1 and 1 like this:

https://github.com/guillaumekln/faster-whisper/blob/6a2da9a95cf807d529ea97b2ce1c46103a88e158/faster_whisper/audio.py#L59

However, it is not centered and rescaled like your example.

[mrfragger]

you could normalize it like so
ffmpeg -vn -c:a mp3 -b:a 32k -vbr off -ar 16000 -af dynaudnorm,aformat=channel_layouts=stereo input.mp3

even remove silence and filter out noise (hiss) if that's what you wanna do
ffmpeg -c:a mp3 -vbr off -b:a 32k -ar 16000 -af highpass=500,lowpass=1000,afftdn,aformat=channel_layouts=stereo,volume=12dB,"silenceremove=start_periods=1:stop_periods=-1:start_threshold=-50dB:stop_threshold=-50dB:start_silence=1:start_duration=2:stop_duration=5:detection=peak",dynaudnorm

but I recommend doing EBU R128 algorithm and it's two-pass and is superior to dynaudnorm. Use videomass in AV-Conversions section. After this is done you still might have to convert it to correct sample rate though