Repository navigation
batch execution transcribe in faster-whisper #59
Description
Activity
Hi,
The model implemented in CTranslate2 supports batch execution (with some caveats), but faster-whisper currently implements the same transcription logic as openai/whisper which only processes a single audio file.
We could add a batch mode in the future.
Note that there is already a way to increase throughput for CPU execution: increase
num_workersand calltranscribefrom multiple Python threads.Reacted by Ruhollah Majdoddin, Paul Spatariu, Herz3h, Shervin Emami, 01boxer and Chatchai Saratakijthanks, i was able to figure it out with threading
Hey @eschmidbauer, could you elaborate on how you accomplished the threading?
Reacted by Thomas Chaigneauhere is an example, pass the faster whisper model to your Process thread
for file in file_list: thread: process.Process = process.Process( model=model, file_path=file) thread.start() threads.append(thread) for thread in threads: thread.join()Reacted by Ruhollah Majdoddin and Shervin EmamiReacted by Thinh LeLet's keep this issue open. It could be interesting to have an actual batch execution, especially on GPU.
Reacted by DigilConfianz, Roman, Zachary Neveu, Ruhollah Majdoddin, David, joshuabld, DoodleBear, chen-deepdub, Clément Tabary, Paul Spatariu and 16 more+1 here
mark
FYI: There is a fork with batch inference for the open ai implementation: openai/whisper#662
Reacted by Phan Tuấn Anh, Thomas Chaigneau, joshuabld and chien liuWhisperX pushed an experimental branch implementing batch execution with faster-whisper:
Reacted by DigilConfianz, Thomas Chaigneau and khangndSTAn implementation note - it would be great to be able to both segment large audio files (as WhisperX does), and have the option to pass in a bunch of independent audio files and run those as a batch.
Reacted by Rui Santos, Chintella-Esq and MaWhisperX pushed an experimental branch implementing batch execution with faster-whisper:
@guillaumekln , The faster-whisper transcribe implementation is still faster than the batch request option proposed by whisperX.
I re-created, with some simplification (I don't use the Binarizer), the entire batching pipeline, and it's like 2x-3x slower than using faster-whisper withnum_workers=1, which is sad 🤗
The data exchange between CPU and GPU takes most of the inference time. It could come from my implementation, so I'm still investigating.Reacted by Aleksandr Smechov@guillaumekln Any thoughts on the above?
The data exchange between CPU and GPU takes most of the inference time
How did you find that?
The code looks correct to me but you should try simplifying the usage:
- Make sure to use the latest version of
ctranslate2 - Start with
num_workers=1 - You could remove
asynchronous=Truesince you are waiting for the results just after. - If you are not reusing the encoder output, it's better to not run the encoder separately. In that case you can directly pass the input features to the method
generate.
See also my comment in the whisperX issue about the current limitations of batch execution in CTranslate2: m-bain/whisperX#159 (comment)
Reacted by Thomas Chaigneau and Aleksandr Smechov- Make sure to use the latest version of
11 remaining items
I pushed an experimental branch in CTranslate2 to support variable-length text inputs for the Whisper model:
This could allow running the Whisper transcription in batch mode even with
condition_on_previous_text=True.It seems that multiple people in this thread tried to implement some form of batching in faster-whisper. It would be great if you can use the experimental branch and see how far you can go with your batch implementation (and share performance numbers!).
To install this CTranslate2 development build:
- Go to this build page
- Download the artifact "python-wheels"
- Extract the archive
- Install the wheel matching your system and Python version, for example:
pip install --force-reinstall ctranslate2-3.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whlReacted by Thomas Chaigneau, Aleksandr Smechov, Sofian Mejjoute and Ki Hoon KimHi @guillaumekln, I'm checking this for batching support.
Suppose I understand well the
ctranslate2experimental branch allows batching over multiple audio files and then enables transcribing numerous audio files in parallel by extending the VRAM usage. Is it only that? What about batching over one file? (I think it's still an architecture problem?)Batching over one file is typically not possible with
condition_on_previous_text=True(which is default value) because you need the previous transcription to process a 30-second window. In this case the audio needs to be processed sequentially.Batching with
condition_on_previous_text=Falseis already possible and does not require an experimental CTranslate2 branch. See WhisperX for example.Thanks for the WhisperX link. I already checked this implementation and reproduced it (by simplifying the part with HF transformers pipeline to a PyTorch pipeline). Still, this implementation is prone to problems (hallucinations, some words disappearing, wrong transcription...).
I will experiment with audio file batching closer to how the transcription pipeline works. Thanks a lot.
Reacted by Aleksandr Smechov@guillaumekln Oh, I have been watching the ctranslate2 and faster-whisper commits for this update, but missed these messages. Great. I'll give it a try in the next few days.
Reacted by Aleksandr Smechov, Ki Hoon Kim and Arche151Okay, I got a bit of time free, and a machine to play on.
@guillaumekln I followed the special build instructions, looks okay.
So, what is needed is something like WhisperModel.transcribe, but rather WhisperModel.transcribe_batch
I'll take a shot, are there any points/things to look out for?
humhum, it mostly looks pretty straightforward to add _batch versions of existing functions, putting List[] around the function argument types etc... but, there's some logic around temperatures, generate is rerun under certain criteria (if needs_fallback is True).Nevermind, managed to code around it and keep the logic the same.. about 75% done..
Reacted by Aleksandr Smechov and Arche151humhum, it mostly looks pretty straightforward to add _batch versions of existing functions, putting List[] around the function argument types etc... but, there's some logic around temperatures, generate is rerun under certain criteria (if needs_fallback is True).Nevermind, managed to code around it and keep the logic the same.. about 75% done..
Hey, sorry I am kind of a noob, but if I understand correctly you are now working on a pull request to implement batching into faster-whisper and the batching will work by utilizing more VRAM? :)
@hobodrifterdavid any updates on this? Really looking forward to batch transcriptions...
I didn't get to finish it yet, but here is a modified transcribe.py with added _batch functions:
https://gist.github.com/hobodrifterdavid/c437ead4d167b52ca6c0373b0a12529dYou can diff it against the one in the current repo to see the changes (https://github.com/guillaumekln/faster-whisper/blob/master/faster_whisper/transcribe.py). There's not a lot of work to finish it, but I was hoping @guillaumekln would be able to comment on the approach.
Looks like there were a couple of minor changes to transcribe.py from the version I was editing.
I saw in the comments that Guillaume is moving on, but hopefully faster-whisper will get it's batch mode. :)
Reacted by a5723797 and Gabriel MarinkovicI took another look over the weekend. I wasn't satisfied with the approach of making duplicate batch functions.. these functions are less readable that the original.
Control passes:
transcribe() => generate_segments() => generate_with_fallback() => model.generate()
transcribe() <= generate_segments() <= generate_with_fallback() <=Instead of making batch versions of all these functions, it could work better to have a wrapper around transcribe, that passes in a special implemenation of generate_with_fallback (generate_with_fallback_cukoo), that uses async/await to cede control back to the wrapper mid-way through execution, so that a batch function can be called.
Something like this (it's a sketch, not working code):
def transcribe_batch( self, audio_list_passed: List[Union[str, BinaryIO, np.ndarray]], # list language_list_passed: List[Union[str, None]], # list task_list_passed: List[Union[str, None]], # list initial_prompt_list_passed: List[Union[str, Iterable[int], None]], # list beam_size: int = 5, best_of: int = 5, patience: float = 1, length_penalty: float = 1, repetition_penalty: float = 1, temperature: Union[float, List[float], Tuple[float, ...]] = [ 0.0, 0.2, 0.4, 0.6, 0.8, 1.0, ], compression_ratio_threshold: Optional[float] = 2.4, log_prob_threshold: Optional[float] = -1.0, no_speech_threshold: Optional[float] = 0.6, condition_on_previous_text: bool = True, prompt_reset_on_temperature: float = 0.5, prefix: Optional[str] = None, suppress_blank: bool = True, suppress_tokens: Optional[List[int]] = [-1], without_timestamps: bool = False, max_initial_timestamp: float = 1.0, word_timestamps: bool = False, prepend_punctuations: str = "\"'“¿([{-", append_punctuations: str = "\"'.。,，!！?？:：”)]}、", vad_filter: bool = False, vad_parameters: Optional[Union[dict, VadOptions]] = None, batch_size = 10, ) -> List[Tuple[Iterable[Segment], TranscriptionInfo]]: encoder_output_list: List[ctranslate2.StorageView], prompt_list: List[List[int]], tokenizer_list: List[Tokenizer], options_list: List[TranscriptionOptions], futures: List[asyncio.future] = [] async def generate_with_fallback_cukoo(encoder_output: ctranslate2.StorageView, prompt: List[int], tokenizer: Tokenizer, options: TranscriptionOptions): encoder_output_list.append(encoder_output) prompt_list.append(prompt) tokenizer_list.append(tokenizer) options_list.append(options) future = asyncio.get_running_loop().create_future() futures.append(future) result = await future return result rv = [] for i in range(len(audio_list_passed)): ##### EDIT: actually probably don't want to await transcribe here yet, await it after the batch function has executed.. rv.append(await self.transcribe( audio=audio_list_passed[i], language=language_list_passed[i], task=task_list_passed[i], beam_size=beam_size, best_of=best_of, patience=patience, length_penalty=length_penalty, repetition_penalty=repetition_penalty, temperature=temperature, compression_ratio_threshold=compression_ratio_threshold, log_prob_threshold=log_prob_threshold, no_speech_threshold=no_speech_threshold, condition_on_previous_text=condition_on_previous_text, prompt_reset_on_temperature=prompt_reset_on_temperature, initial_prompt=initial_prompt_list_passed[i], # nope prefix=prefix, suppress_blank=suppress_blank, suppress_tokens=suppress_tokens, without_timestamps=without_timestamps, max_initial_timestamp=max_initial_timestamp, word_timestamps=word_timestamps, prepend_punctuations=prepend_punctuations, append_punctuations=append_punctuations, vad_filter=vad_filter, vad_parameters=vad_parameters, ########## pass in the special cukoo function: ########## generate_with_fallback=generate_with_fallback )) if len(encoder_output_list) == batch_size: # Okays time to call the batch function.. results = self.generate_with_fallback_batch(encoder_output_list, prompt_list, tokenizer_list, options_list) for j in range(len(results)): futures[j].set_result(results[j]) encoder_output_list= [] prompt_list= [] tokenizer_list= [] options_list= [] futures = [] results = self.generate_with_fallback_batch(encoder_output_list, prompt_list, tokenizer_list, options_list) for j in range(len(results)): futures[j].set_result(results[j]) # actually need to transform this a bit still.. return rvOne thing.. I don't think it's possible to do this without making transcribe and generate_segments async. I'm not familar enough with Python async/await to say 100%. But you could make transcribe_async() and generate_segments_async(), and wrap them with simple non-async wrappers transcribe() and generate_segments().
If there are transcribe_async and transcribe etc., they should have the same arguments, it's a bit clumsy to maintain two sets of the long argument lists, proabably a bit better to pass in a class with options as an argument, but then the api changes.
Reacted by Chintella-Esq, QuaSoft and a5723797Hello, I'm wondering if there's any progress on this issue? Wish I could help, but unfortunately I'm not an expert. However, I have reviewed the new repos like the relatively new https://github.com/Vaibhavs10/insanely-fast-whisper and WhisperX's implementation, but was wondering if anything similar is going to be implemented directly in faster-whisper like batching for a single file? I know there's been a several issues created this over a couple repositories...
mark
Hello,
Is batch execution of faster-whisper's transcribe possible? We've seen in this thread that batch execution should increase the throughput. But it's not clear how to perform batch using faster-whisper if at all. Thanks!