Enhance faster_whisper Engine

[EvilFreelancer]

Hi, developers!

This pull request introduces the following improvements and fixes:

• Updated NVIDIA CUDA Container: The version of the nvidia/cuda container has been updated from 11.7.0 to 11.7.1. This was necessitated as version 11.7.0 is no longer available.

• Modifications in faster_whisper Engine:

  • The model_converter function within the engine has been enhanced. Previously, the function had hardcoded float16. This has now been replaced with a variable that defaults to float16, allowing for greater flexibility.
  • In core.py, the environment variable ASR_QUANTIZATION has been added. This allows users to specify various quantization levels including float16, int16, int8, int8_float16, and int4. However, it should be noted that there seems to be an issue with the int4 mode which may require further investigation.

Regarding issue #127, if there's an inherent bug in the original codebase, I would prefer to remove the MAPPING. Instead of:

if torch.cuda.is_available():
    model = WhisperModel(model_path, device="cuda", compute_type=MAPPING[model_quantization])
else:
    model = WhisperModel(model_path, device="cpu", compute_type="int8")
model_lock = Lock()

I suggest the following:

model = WhisperModel(model_path, device="cuda", compute_type=model_quantization)

PS. My PR will definitely conflict with #117.

[ayancey]

The diff is kinda messed up for some Git reason I don't understand. Using this to compare to main after 1.2 release.

[EvilFreelancer]

@ayancey hi! Thanks, yeah, seems codebase was updated. I'll fix conflicts.

[ayancey]

Here's my attempt at benchmarking and comparing between @ahmetoner's latest release (1.2.0) and this PR. I bumped the CUDA version on your fork so it would be an even comparison.

Tests done with 23 second audio file using small model and faster_whisper backend. GPU is NVIDIA GeForce GTX 1650.

Branch	CUDA ver	mem before trials	GPU mem before trials	mem after trials	GPU mem after trials	Avg time (seconds)
1.1.1	11.8.0	1.743GiB	1142MiB	2.782GiB	1452MiB	1.82
1.2	11.8.0	613.2MiB	1144MiB	1.606GiB	1454MiB	1.84
EvilFreelancer	11.8.0	1.75GiB	1142MiB	2.808GiB	1452MiB	1.93

I don't understand it, but it looks like Ahmet fixed the problem you referred to in #127. But I couldn't find any improvements in GPU or CPU memory.

[ayancey]

@ayancey hi! Thanks, yeah, seems codebase was updated. I'll fix conflicts.

So it sounds like the model conversion is no longer an issue, but we still want to let users choose their quantization?

[EvilFreelancer]

So it sounds like the model conversion is no longer an issue, but we still want to let users choose their quantization?

I think it's a good idea, in any case we may keep float16 as default value. For example i use int8 version of large-v2 model for my projects.

[ayancey]

Awesome. Once you fix the conflicts I can bug Ahmet to merge this. I'm really excited to start playing around and comparing the different options.

[EvilFreelancer]

@ayancey hi! I've updated my PR, you may try to add ASR_QUANTIZATION env option with one of modes described on this page: https://opennmt.net/CTranslate2/quantization.html

[ahmetoner]

This change has the potential to impact backward compatibility and may result in additional model downloads. Could you please revert to the previous lines? Once that's done, I'll proceed with the merge.
Thank you for your contribution.

[EvilFreelancer]

Okay, ill fix it

[EvilFreelancer]

Done

[ayancey]

👏 👏