[Feature Request] Support M1 Mac's GPU

[0x1FFFFF]

If I pass in mps to device option it will crush. Would be wonderful if M1 GPU can be supported

❯ whisperx assets/test.mp3 --device mps --model large-v2 --vad_filter --align_model WAV2VEC2_ASR_LARGE_LV60K_960H --diarize --hf_token token --language en

torchvision is not available - cannot save figures
Performing VAD...
~~ Transcribing VAD chunk: (01:07:46.006 --> 01:07:50.663) ~~
loc("mps_multiply"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<1x1280x3000xf16>' and 'tensor<1xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
[1]    97334 abort      whisperx assets/test.mp3 --device mps --model large-v2 --vad_filter       en
/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[wllbll]

ye, i got this error too, it seems we need to wait update from pytorch

[skye-repos]

Hi, I believe PyTorch has support for most of the functions now. Plus it can be run with env arg PYTORCH_ENABLE_MPS_FALLBACK=1 for those functions that aren't supported yet to fall back on the CPU.

I'm running pyannote and other projects with PyTorch compiled with support for mps so this should also be do-able

[Herb-sh]

@Harith163 That is correct. Pytorch has implemented "mps"(Metal Performance Shaders) for a at least a year now, Open-AI's Whisper supports "mps" as well but faster-whisper used by WhisperX apparently only supports "cpu" and "gpu".

I myself get "unsupported device mps", here is the error:

--> 128 self.model = ctranslate2.models.Whisper( 129 model_path, 130 device=device, 131 device_index=device_index, 132 compute_type=compute_type, 133 intra_threads=cpu_threads, 134 inter_threads=num_workers, 135 )

For me with Mac M1 "cpu" is extremely slow to the point I have not been able to get e proper transcription.

Any workaround to the issue? I believe this is essential for mac users.

Thanks :)

[7k50]

@Harith163 That is correct. Pytorch has implemented "mps"(Metal Performance Shaders) for a at least a year now, Open-AI's Whisper supports "mps" as well but faster-whisper used by WhisperX apparently only supports "cpu" and "gpu".

I myself get "unsupported device mps", here is the error:

--> 128 self.model = ctranslate2.models.Whisper( 129 model_path, 130 device=device, 131 device_index=device_index, 132 compute_type=compute_type, 133 intra_threads=cpu_threads, 134 inter_threads=num_workers, 135 )

For me with Mac M1 "cpu" is extremely slow to the point I have not been able to get e proper transcription.

Any workaround to the issue? I believe this is essential for mac users.

Thanks :)

Whisper.cpp is fast on Apple Silicon ("Plain C/C++ implementation without dependencies" … "optimized via ARM NEON, Accelerate framework, Metal and Core ML"). However, I believe it only supports very rudimentary diarization currently.

Ideally, WhisperX's solutions for diarization, etc, could be made to work in the fashion of Whisper.cpp.

[purpshell]

@Harith163 That is correct. Pytorch has implemented "mps"(Metal Performance Shaders) for a at least a year now, Open-AI's Whisper supports "mps" as well but faster-whisper used by WhisperX apparently only supports "cpu" and "gpu".
I myself get "unsupported device mps", here is the error:
--> 128 self.model = ctranslate2.models.Whisper( 129 model_path, 130 device=device, 131 device_index=device_index, 132 compute_type=compute_type, 133 intra_threads=cpu_threads, 134 inter_threads=num_workers, 135 )
For me with Mac M1 "cpu" is extremely slow to the point I have not been able to get e proper transcription.
Any workaround to the issue? I believe this is essential for mac users.
Thanks :)

Whisper.cpp is fast on Apple Silicon ("Plain C/C++ implementation without dependencies" … "optimized via ARM NEON, Accelerate framework, Metal and Core ML"). However, I believe it only supports very rudimentary diarization currently.

Ideally, WhisperX's solutions for diarization, etc, could be made to work in the fashion of Whisper.cpp.

That's only ideally, whisper.cpp's creator showed interest in the killer features of WhisperX but stated they are not coming any time soon.

I'd rather fix whisperx to work better on M1/Apple Silicon

[1Dbcj]

Just wanted to second this. I love whisperx on my PC, but on Mac it is just so slow.

Its resulted in fragmentation where if I want my script to be universal I have to look elsewhere. Really wish this could be supported.

[AdrienLF]

Any progress on this so far?

[thibaudbrg]

Any news ?

[colin4k]

Any news?

[aredwine3]

It could be that ctranslate2 needs to be built on your macOS system with WITH_ACCELERATE set to ON. The owner of ctranslate2 says is may be possible here, in reference to this comment.

[ryanfb]

Thanks for the tip! ctranslate2 install instructions are here: https://opennmt.net/CTranslate2/installation.html#install-from-sources

On my macOS, I needed to use the following for the cmake step:

cmake -DWITH_ACCELERATE=ON -DOPENMP_RUNTIME=COMP -DWITH_MKL=OFF ..

After compiling and installing ctranslate2 myself in a fresh pyenv with MPS enabled on my M1 Mac, I installed whisperx (which used my existing ctranslate2 install: Requirement already satisfied: ctranslate2<5,>=4.0 in /Users/ryan/.pyenv/versions/3.12.4/envs/ctranslate-build/lib/python3.12/site-packages (from faster-whisper==1.0.0->whisperx==3.1.1) (4.4.0)) and ran with --device mps but still got:

Traceback (most recent call last):
  File "/Users/ryan/.pyenv/versions/ctranslate-build/bin/whisperx", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/ryan/.pyenv/versions/3.12.4/envs/ctranslate-build/lib/python3.12/site-packages/whisperx/transcribe.py", line 170, in cli
    model = load_model(model_name, device=device, device_index=device_index, download_root=model_dir, compute_type=compute_type, language=args['language'], asr_options=asr_options, vad_options={"vad_onset": vad_onset, "vad_offset": vad_offset}, task=task, threads=faster_whisper_threads)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ryan/.pyenv/versions/3.12.4/envs/ctranslate-build/lib/python3.12/site-packages/whisperx/asr.py", line 288, in load_model
    model = model or WhisperModel(whisper_arch,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ryan/.pyenv/versions/3.12.4/envs/ctranslate-build/lib/python3.12/site-packages/faster_whisper/transcribe.py", line 133, in __init__
    self.model = ctranslate2.models.Whisper(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: unsupported device mps