Docker error "Could not load library libcudnn_ops_infer.so.8. Error: libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory"

[cod3r0k]

Hi. I have a container which has CUDA and cudnn (nvidia/cuda:11.8.0-base-ubuntu22.04).
But when I want to use it in inference mode, I have a problem:
Could not load library libcudnn_ops_infer.so.8. Error: libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory

[trungkienbkhn]

@cod3r0k , hello. Can you try again with:

sudo apt-get install libcudnn8
sudo apt-get install libcudnn8-dev

I think you should use nvidia/cuda image version 12. Recently, ctranslate2 lib was upgraded to 4.0 to support cuda 12.

[souvikqb]

Getting the same error even after installing libcudnn8. I'm using CUDA 12.2

[trungkienbkhn]

@souvikqb , what version of ctranslate2 did you use? For CUDA 12.2, it should ctranslate2 version 4.0

[souvikqb]

@souvikqb , what version of ctranslate2 did you use? For CUDA 12.2, it should ctranslate2 version 4.0

I'm not explicitly installing ctranslate-2, I'm using the version downloaded from faster-whisper.

[cod3r0k]

@cod3r0k , hello. Can you try again with:

sudo apt-get install libcudnn8
sudo apt-get install libcudnn8-dev

I think you should use nvidia/cuda image version 12. Recently, ctranslate2 lib was upgraded to 4.0 to support cuda 12.

Currently, I am facing an issue that is not resolved yet. It's working fine in non-docker, but in docker, it's not working. Can you please share the docker file that you have used to run it successfully? @trungkienbkhn

[trungkienbkhn]

@souvikqb Uhm, so what version of faster-whisper did you use ? (0.10.1 or latest version 1.0.1)
For version 0.10.1, it doesn't support CUDA 12 => ctranslate2 < 4 (3.24.0)
For latest version 1.0.1, ctranslate2 required >= 4.0
If you use pip package, you should run pip install --upgrade faster-whisper && pip install --upgrade ctranslate2

[souvikqb]

@souvikqb Uhm, so what version of faster-whisper did you use ? (0.10.1 or latest version 1.0.1) For version 0.10.1, it doesn't support CUDA 12 => ctranslate2 < 4 (3.24.0) For latest version 1.0.1, ctranslate2 required >= 4.0 If you use pip package, you should run pip install --upgrade faster-whisper && pip install --upgrade ctranslate2

I'm using the following in requirements.txt -
git+https://github.com/m-bain/whisperx.git
faster-whisper==1.0.1
speechbrain==0.5.16
ctranslate2>=4.0

But still getting the same error on CUDA 12.2

[trungkienbkhn]

@cod3r0k , ok. My CUDA version in host machine is 12.0. Therefor I used image nvidia/cuda:12.0.0-devel-ubuntu20.04. Below is a simple example for FW Dockerfile:

FROM nvidia/cuda:12.0.0-devel-ubuntu20.04

WORKDIR /root

RUN apt-get update -y && apt-get install -y python3 python3-pip libcudnn8 libcudnn8-dev
RUN python3 -m pip install pip --upgrade
COPY infer.py .
COPY jfk.flac .
RUN pip3 install faster-whisper

ENTRYPOINT ["python3", "infer.py"]

infer.py

import time
from faster_whisper import WhisperModel
 
jfk_path = 'jfk.flac'
model_path = 'tiny'
tic = time.time()
model = WhisperModel(model_path, device='cuda')
segments, info = model.transcribe(jfk_path, word_timestamps=True)
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

print("Total time: ", time.time() - tic)

Running:

docker run --gpus all faster_whisper:latest
[0.00s -> 10.48s]  And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.
Total time:  8.151259183883667

Hope it's helpful for you.

[trungkienbkhn]

@souvikqb , could you run apt-cache show libcudnn8 and find libcudnn8 for cuda12. For me:

Package: libcudnn8
Version: 8.8.0.121-1+cuda12.0
Architecture: amd64
Priority: optional
Section: multiverse/libs
Source: cudnn
Maintainer: cudatools <cudatools@nvidia.com>
Installed-Size: 1125269
Filename: ./libcudnn8_8.8.0.121-1+cuda12.0_amd64.deb
Size: 451998740
MD5sum: 4e9d508b2d438a638cc20ca8b5b5c75f
SHA1: 57bed974fcfe5021b8b08aa3257b8718309793f7
SHA256: 87ab4a0a1c5364f594d9edec0a25323621e176e27811e893a54e54c1d1989312
SHA512: 39d2e75dad4f38219e438447b2ebe8df85978940c786e2ec7441abe3648c09e161680550f678b70e89ed6681f1426cb6a1e955167d9fbc8f13236f4bac1f269c
Description: cuDNN runtime libraries
 cuDNN runtime libraries containing primitives for deep neural networks.
Description-md5: 0f50478914081182f91de6bc0162e2

Besides, I found another solution here. You can try it.

[souvikqb]

@souvikqb , could you run apt-cache show libcudnn8 and find libcudnn8 for cuda12. For me:

Package: libcudnn8
Version: 8.8.0.121-1+cuda12.0
Architecture: amd64
Priority: optional
Section: multiverse/libs
Source: cudnn
Maintainer: cudatools <cudatools@nvidia.com>
Installed-Size: 1125269
Filename: ./libcudnn8_8.8.0.121-1+cuda12.0_amd64.deb
Size: 451998740
MD5sum: 4e9d508b2d438a638cc20ca8b5b5c75f
SHA1: 57bed974fcfe5021b8b08aa3257b8718309793f7
SHA256: 87ab4a0a1c5364f594d9edec0a25323621e176e27811e893a54e54c1d1989312
SHA512: 39d2e75dad4f38219e438447b2ebe8df85978940c786e2ec7441abe3648c09e161680550f678b70e89ed6681f1426cb6a1e955167d9fbc8f13236f4bac1f269c
Description: cuDNN runtime libraries
 cuDNN runtime libraries containing primitives for deep neural networks.
Description-md5: 0f50478914081182f91de6bc0162e2

Besides, I found another solution here. You can try it.

Sure let me try

[cod3r0k]

I don't understand why there are issues with cuda and cudnn in nvidia/cuda:12.0.1-cudnn8-runtime-ubuntu22.04 even though everything seems fine. So if there's an issue, the name and tag should not include cuda and cudnn!

[trungkienbkhn]

@cod3r0k , what is the issue you are encountering? I replaced the devel image to the runtime image (nvidia/cuda:12.0.0-runtime-ubuntu20.04) and it still works as expected.

[shuiqingliu]

This issue was resolved by installing libcudnn8 and libcudnn8-dev on ubuntu 22.04 with cuda12

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb 
dpkg -i cuda-keyring_1.0-1_all.deb 
apt update && apt upgrade
apt install libcudnn8 libcudnn8-dev

[kendrickcheung007]

This issue was resolved by installing libcudnn8 and libcudnn8-dev on ubuntu 22.04 with cuda12

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb 
dpkg -i cuda-keyring_1.0-1_all.deb 
apt update && apt upgrade
apt install libcudnn8 libcudnn8-dev

Thank you! It works for me!

[alpcansoydas]

@cod3r0k , ok. My CUDA version in host machine is 12.0. Therefor I used image nvidia/cuda:12.0.0-devel-ubuntu20.04. Below is a simple example for FW Dockerfile:

FROM nvidia/cuda:12.0.0-devel-ubuntu20.04

WORKDIR /root

RUN apt-get update -y && apt-get install -y python3 python3-pip libcudnn8 libcudnn8-dev
RUN python3 -m pip install pip --upgrade
COPY infer.py .
COPY jfk.flac .
RUN pip3 install faster-whisper

ENTRYPOINT ["python3", "infer.py"]

infer.py

import time
from faster_whisper import WhisperModel
 
jfk_path = 'jfk.flac'
model_path = 'tiny'
tic = time.time()
model = WhisperModel(model_path, device='cuda')
segments, info = model.transcribe(jfk_path, word_timestamps=True)
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

print("Total time: ", time.time() - tic)

Running:

docker run --gpus all faster_whisper:latest
[0.00s -> 10.48s]  And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.
Total time:  8.151259183883667

Hope it's helpful for you.

it worked for me

[NightHawkATL]

I will just remove it for now as I will eventually get a better and newer Ampere-series GPU. Thanks for all of the help though.