batch execution transcribe in faster-whisper

[eschmidbauer]

Hello,
Is batch execution of faster-whisper's transcribe possible? We've seen in this thread that batch execution should increase the throughput. But it's not clear how to perform batch using faster-whisper if at all. Thanks!

[guillaumekln]

Hi,

The model implemented in CTranslate2 supports batch execution (with some caveats), but faster-whisper currently implements the same transcription logic as openai/whisper which only processes a single audio file.

We could add a batch mode in the future.

Note that there is already a way to increase throughput for CPU execution: increase num_workers and call transcribe from multiple Python threads.

[eschmidbauer]

thanks, i was able to figure it out with threading

[Joepetey]

Hey @eschmidbauer, could you elaborate on how you accomplished the threading?

[eschmidbauer]

here is an example, pass the faster whisper model to your Process thread

        for file in file_list:
            thread: process.Process = process.Process(
                model=model, file_path=file)
            thread.start()
            threads.append(thread)

        for thread in threads:
            thread.join()

[guillaumekln]

Let's keep this issue open. It could be interesting to have an actual batch execution, especially on GPU.

[timnlupo]

+1 here

[junshipeng]

mark

[ezerhouni]

FYI: There is a fork with batch inference for the open ai implementation: openai/whisper#662

[guillaumekln]

WhisperX pushed an experimental branch implementing batch execution with faster-whisper:

m-bain/whisperX#159 (comment)

[natecarlson]

An implementation note - it would be great to be able to both segment large audio files (as WhisperX does), and have the option to pass in a bunch of independent audio files and run those as a batch.

[chainyo]

WhisperX pushed an experimental branch implementing batch execution with faster-whisper:

m-bain/whisperX#159 (comment)

@guillaumekln , The faster-whisper transcribe implementation is still faster than the batch request option proposed by whisperX.
I re-created, with some simplification (I don't use the Binarizer), the entire batching pipeline, and it's like 2x-3x slower than using faster-whisper with num_workers=1, which is sad 🤗
The data exchange between CPU and GPU takes most of the inference time. It could come from my implementation, so I'm still investigating.

ref: https://github.com/Wordcab/wordcab-transcribe/blob/6-investigate-on-a-better-gpu-usage/wordcab_transcribe/services/transcribe_service.py

[aleksandr-smechov]

@guillaumekln Any thoughts on the above?