How to make it work with CUDA enabled GPU? GTX 1050 Ti- 4GB

[FurkanGozukara]

edit : i prepared an excellent tutorial video after my all experience : https://www.youtube.com/watch?v=q8Q8CCDdSKo

I have installed python 3.9.9
Then I have run this command : pip install git+https://github.com/openai/whisper.git

It installed everything and it works fine

However later i learned that I have to installed CUDA enabled Torch

For that what do I need to do ?

First run this command? pip3 uninstall torch

Then which version to install?

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

Any help appreciated

I am using pip and Windows 10

[spcmnspf]

I found this on the github for pytorch: pytorch/pytorch#30664 (comment)

I just modified it to meet the new install instructions. I'm running Windows 11. Seems that you have to remove the cpu version first to install the gpu version. That's my understanding of it at least.

pip uninstall torch
pip cache purge
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

After doing this I was able to run the following command ( choose whatever model fits your needs best ):
whisper "audio.wav" --model medium --device cuda

[DrMemoryFish]

i keep getting:
C:\Users\Abdullah\AppData\Local\Programs\Python\Python310\lib\site-packages\whisper\transcribe.py:78: UserWarning: FP16 is not supported on CPU; using FP32 instead warnings.warn("FP16 is not supported on CPU; using FP32 instead") Detecting language using up to the first 30 seconds. Use --language to specify the language Detected language: English Traceback (most recent call last):

how do I make it use FP16? to use my GPU instead of my CPU. Im a real noob at code, so you'll have to kindly explain in details,
thank you.

[FurkanGozukara]

@AbdullahJames I have an excellent tutorial video : https://www.youtube.com/watch?v=q8Q8CCDdSKo

[maphew]

Thank you for this thread. I'm brand new to Open.ai and this has been helpful.

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

That thread uses pip install torch -f https://download.pytorch.org/whl/torch_stable.html instead. It currently redirects to the url you noted but suspect using the torch_stable link is probably better recommendation long term than using what appears to be a version specific one. Also a heads up for anyone following this route, it's a 2.4 GB download.

Check "whisper --help" to see which device will be used by default.

Use whisper --help | findstr -i pytorch to not have to seek through the voluminous output ;-)

» whisper --help | findstr -i pytorch
  --device DEVICE       device to use for PyTorch inference (default: cuda)

...

Unfortunately for me it turns out that although my video card has CUDA it's tool old for pytorch. :-/

Found GPU0 Quadro K420 which is of cuda capability 3.0.
PyTorch no longer supports this GPU because it is too old.
The minimum cuda capability supported by this library is 3.7.

[FurkanGozukara]

Thank you for this thread. I'm brand new to Open.ai and this has been helpful.

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

That thread uses pip install torch -f https://download.pytorch.org/whl/torch_stable.html instead. It currently redirects to the url you noted but suspect using the torch_stable link is probably better recommendation long term than using what appears to be a version specific one. Also a heads up for anyone following this route, it's a 2.4 GB download.

Check "whisper --help" to see which device will be used by default.

Use whisper --help | findstr -i pytorch to not have to seek through the voluminous output ;-)

» whisper --help | findstr -i pytorch
  --device DEVICE       device to use for PyTorch inference (default: cuda)

...

Unfortunately for me it turns out that although my video card has CUDA it's tool old for pytorch. :-/

Found GPU0 Quadro K420 which is of cuda capability 3.0.
PyTorch no longer supports this GPU because it is too old.
The minimum cuda capability supported by this library is 3.7.

yes I also had to purchase a new card to use Whisper properly.

[el-belgounetos]

I found this on the github for pytorch: pytorch/pytorch#30664 (comment)

I just modified it to meet the new install instructions. I'm running Windows 11. Seems that you have to remove the cpu version first to install the gpu version. That's my understanding of it at least.

pip uninstall torch pip cache purge pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

After doing this I was able to run the following command ( choose whatever model fits your needs best ): whisper "audio.wav" --model medium --device cuda

Thanks, this helped me too.
To be complete it was not working for me without others steps.
To get whisper up and runnig with cuda I had to do the following steps

Install whisper
pip install -U openai-whisper

Uninstall pytorch

pip3 uninstall torch torchvision torchaudio
pip 3 cache purge

Install cuda-python and Torch cuda
pip install cuda-python

To install pytorch you can choose your version from the pytorch website https://pytorch.org/ see INSTALL PYTORCH section

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

[rodgermoore]

It is much easier using Docker for this. Assuming you have the latest Nvidia driver and Docker Engine installed here is a suggested example Dockerfile that works fine for me:

FROM nvcr.io/nvidia/pytorch:22.09-py3
ENV TZ=Europe/Amsterdam
#ENV PYTORCH_NO_CUDA_MEMORY_CACHING=1
ENV PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.8,max_split_size_mb:2048
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && apt update && apt install -y ffmpeg git libsndfile1 sox
&& pip install git+https://github.com/openai/whisper.git datasets datasets[audio] pytube tqdm sentence_transformers pinecone-client google-api-python-client
RUN mkdir /root/.cache/whisper && cd /root/.cache/whisper && { curl -O https://openaipublic.azureedge.net/main/whisper/models/d7440d1dc186f76616474e0ff0b3b6b879abc9d1a4926b7adfa41db2d497ab4f/medium.en.pt --progress-bar ; cd -; } 
RUN cd /root/.cache/whisper && { curl -O https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt --progress-bar ; cd -; }
#SEE https://github.com/pytube/pytube/pull/1409/files and https://github.com/pytube/pytube/pull/1409
COPY opt.conda.lib.python3.8.site-packages.pytube.contrib.channel.py /opt/conda/lib/python3.8/site-packages/pytube/contrib/channel.py
RUN python -c 'from sentence_transformers import SentenceTransformer;model_id = "multi-qa-mpnet-base-dot-v1";model = SentenceTransformer(model_id);model;dim = model.get_sentence_embedding_dimension()'

Build using the command: docker build -t nvcr.io/nvidia/pytorch:22.09-py3-whisper .

To run a container using the current directory as input:

docker run --name whisper --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus all -p 8888:8888 -it -rm -v ${PWD}:/workspace/whisper nvcr.io/nvidia/pytorch:22.09-py3-whisper whisper your_audio_file.wav --translate 

Enjoy the ride

[ttimasdf]

These steps are tested on Windows 11 with CUDA 11.7 + torch 1.12.1

1. Download cuda_11.7.0_windows_network.exe network installer from CUDA Toolkit 11.7 Downloads | NVIDIA Developer
2. Select only Runtime at install.
3. install pytorch with CUDA via pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117
4. Check CUDA availability with python -c 'import torch; print(\"CUDA enabled:\", torch.cuda.is_available());'. if everything OK, the script will output CUDA enabled: True

Reference: Start Locally | PyTorch

[phongku]

use double quotes to prevent a syntax error
python -c "import torch; print('CUDA enabled:', torch.cuda.is_available());"

[oslaksam]

For Windows 11, an important step for me was to figure out the version of CUDA installed by the Driver as outlined here, not installing the matching version caused me trouble.

By calling this command:
nvidia-smi.exe
You can figure out the version supported by the currently installed driver.

[boholder]

And... if you're like me, the nvidia-smi.exe shows that the CUDA version is 12, don't worry, we can still install CUDA 11.x according to this compatibility page.
https://docs.nvidia.com/deploy/cuda-compatibility/index.html

For now (20230305), the pytorch has not out of the box with CUDA 12.
https://discuss.pytorch.org/t/pytorch-for-cuda-12/169447/13
Now you can use the out of box version of pytorch for CUDA 12

[caoccao]

Here is my approach on Windows 10 with 2080 Ti.

• Install the latest Driver
• Install CUDA 118
• Uninstall pytorch via pip uninstall torch && pip cache purge
• Install pytorch via pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
• Verify CUDA is enabled via python -c "import torch; print(\"CUDA enabled:\", torch.cuda.is_available());"

[nikito7]

How to keep model in VRAM?

[caoccao]

How to keep model in VRAM?

That depends on the model size vs VRAM size. My understanding is if the model size is below the VRAM size, it will be completely in VRAM.

[Ozguner]

Additionally, do below if the issue is not fixed after following the answer post here.

sudo apt install nvidia-cuda-toolkit
that fixed for me