Standalone Faster-Whisper-XXL features

[Purfview]

Includes all Standalone Faster-Whisper features +the additional ones mentioned below.
Includes all needed libs.

Vocal extraction model:

--ff_mdx_kim2: Preprocess audio with MDX23 Kim vocal v2 model (thanks to Kimberley Jensen). [Better than HT Demucs v4 FT]

Alternative VAD (Voice activity detection) methods:

--vad_alt_method

Alternative choices:
silero_v3 - Generally less accurate than v4, but doesn't have some quirks of v4.
silero_v4 - Same model as used originally, except that it runs original Silero's code instead of adapted one.
pyannote_v3 - The best accuracy.
pyannote_onnx_v3 - Similar performance to silero.
webrtc - Low accuracy, outdated VAD. Takes only 'vad_min_speech_duration_ms' & 'vad_speech_pad_ms'.
auditok - Actually it's not VAD, it's AAD - Audio Activity Detection.

Latest CTranslate2:

Up to ~26% faster on CPU with the int8 quantizations.
Flash attention support, that's CUDA, but the benchmarks shows no effect on the performance.

[despairTK]

I really like the new parameter --vad_alt_method. Among them, silero_v3/silero_v4/pyannote_onnx_v3 are much better than the original VAD.

For example, there will be some gaps in the original VAD, and for example, sentences starting with "So" will often have a delayed start of the timeline. These issues are resolved in the silero_v3/silero_v4/pyannote_onnx_v3 parameters.

Finally, let me ask, which of the three parameters of silero_v3/silero_v4/pyannote_onnx_v3 has the best test results? Or what are their characteristics?

[Purfview]

First why I've added alternatives, because Silero v4 has problems like false-positive detection on nearly silent parts and other imperfections:
snakers4/silero-vad#396
snakers4/silero-vad#369

...sentences starting with "So" will often have a delayed start of the timeline

Did you evaluate that looking at a transcription?

I really like the new parameter --vad_alt_method. Among them, silero_v3/silero_v4/pyannote_v3 are much better than the original VAD.

Actually the original VAD runs exactly same model as "silero_v4". 😌
"silero_v4" runs original code from silero's repo, when the original VAD runs the same model with adapted code.

Finally, let me ask, which of the three parameters of silero_v3/silero_v4/pyannote_onnx_v3 has the best test results?

Probably neither of them are best, I not tested them much yet.
Did one quick test with "pyannote_onnx_v3", for my disappointment it was less accurate than original, it missed few segments.

[despairTK]

Well...after replying to you, I transcribed dozens of audios, ranging from 10 minutes to 2 hours, and the audio language is English. After comparing the transcription results, I think silero_v3 is the best choice.

[Purfview]

After comparing the transcription results,

That's not how VAD evaluation works, you are looking at whisper's randomness not at VAD's accuracy.

To evaluate VAD you need to look only at VAD's timestamps, you can get them with --vad_dump, then those can be loaded in SE and looked on the waveform.

And VAD params needs to be adjusted so you can see accuracy better, like vad_speech_pad_ms=0 and vad_min_silence_duration_ms=0.
And the same threshold doesn't convert to a same responsiveness on the different methods.

[nexuslux]

any hope of doing something similar for mac in the future?

[Purfview]

Probably there won't be macOS builds for this.

[dgoryeo]

I guess no chance of a python package either?

[Purfview]

I guess no chance of a python package either?

Yeap, it's out of the scope of this repo.

[nexuslux]

Sorry, saw a comment here earlier about 'why post' this here but now its gone -- I was trying to keep it in the thread of this 'XXL' version. For example, if there can be an XXL version why not as if there will be a '-MAC' version in the future... who knows - just asking :)

Separately, for others who may find this thread and dismayed at no mac support there is a lot of work being done with 'whisperkit' which also does great work. https://github.com/argmaxinc/WhisperKit

The future of high-quality transcriptions is very bright.

[Sonnenfleck]

A little annoyance with --ff_mdx_kim2 in r192.3.3 [XXL], which otherwise works great (well, after I realized I had to change my scripts because it's now faster-whisper-xxl.exe instead of whisper-faster-xxl.exe 😉 ):

I'm running Faster-Whisper-XXL in a Nextcloud folder (with a cronjob checking if new audio files have been synchronized, then running faster-whisper-xxl). So far, this worked fine, but in r192.3.3 with MDX filtering enabled, it seems first the *_mdx.wav file is created and then it's moved to a temp folder (?). This move fails because Nextcloud already tries to sync the mdx file, and this leads to whisper-faster-xxl just quitting with an error that the *_mdx.wav file is already in use.

I now set the Nextcloud rules to just ignore *_mdx.wav files, but would it be possible to create them in a temp folder from the start?

[Purfview]

it seems first the *_mdx.wav file is created and then it's moved to a temp folder (?)

It's intermediate file, first it's written to the same folder as the original input file then it's deleted. No "movements" of it.
If Nextcloud app tries to do anything with it then of course deletion fails.

would it be possible to create them in a temp folder

Should be possible.

[Sonnenfleck]

I'm confused: If the mdx file is deleted before transcription starts, then what file is whisper running on?
At least as I observe it, the problem here is: As soon as mdx filtering is finished, Nextcloud grabs the _mdx.wav to sync it, but before it finished syncing (which would be just a few seconds later), faster-whisper tries to delete the _mdx.wav but can't, because it's in use. Shouldn't the intermediate file be deleted after transcription? Or does VAD create another intermediate file (if yes, then it's not visible in the same folder)?

[Purfview]

I'm confused: If the mdx file is deleted before transcription starts, then what file is whisper running on?

The intermediate file is loaded into variables further and it's not needed in the physical form anymore.
Yesterday I've made a quick patch to use a temp folder but not tested it yet, I'll try to release it today.

Or does VAD create another intermediate file (if yes, then it's not visible in the same folder)?

VAD's "intermediate" is created only in memory.

[Sonnenfleck]

Ah, I get it, thanks for clarifying and for implementing the patch. :)

[Purfview]

Implemented in r192.3.4

[Herzfrequenz21]

Do I need to use some kind of tag to make the recognition against a little noise or soft music better?

[Purfview]

This has nothing to do with the audio length.

[dgoryeo]

You're right, I thought the memory error was at the point of vocal isolation.

[Herzfrequenz21]

This has nothing to do with the audio length.

How can I make the memory be used from the CPU as before?
This line now specifies CUDA:
Standalone Faster-Whisper-XXL r192.1.1.1 running on: CUDA

[Purfview]

Use --device cpu

[1001ruchka]

This has nothing to do with the audio length.

How can I make the memory be used from the CPU as before? This line now specifies CUDA: Standalone Faster-Whisper-XXL r192.1.1.1 running on: CUDA

Try using CUDA with int8:
--compute_type=int8_float16 or --compute_type=int8
With these parameters I managed to run large_v2 on 4GB of video memory.

for Russian:

-prompt default

faster-whisper-xxl.exe --language ru --model "large-v2" --compute_type=int8_float16 --sentence -prompt default --beep_off --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cpu "video.mp4"

or

-prompt None

faster-whisper-xxl.exe --language ru --model "large-v2" --compute_type=int8_float16 --sentence -prompt None --beep_off --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cpu "video.mp4"

or

-prompt auto

faster-whisper-xxl.exe --language ru --model "large-v2" --compute_type=int8_float16 --sentence -prompt auto --beep_off --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cpu "video.mp4"