Torch ops unable to load "image.so"

[epwalsh]

🐛 Describe the bug

Torch ops unable to load "image.so".

I'm able to reproduce this within the following Docker image:

FROM nvidia/cuda:11.3.0-base-ubuntu20.04

ARG DEBIAN_FRONTEND="noninteractive"
ENV TZ="America/Los_Angeles"

RUN rm -rf /etc/apt/sources.list.d/* \
    && apt-get update \
    && apt-get install -y \
        build-essential \
        curl \
        wget \
        git \
        zip \
        libssl-dev \
        software-properties-common \
        libffi-dev \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work/

RUN pip3 install --no-cache-dir virtualenv && \
    virtualenv -p $(which python3) --copies --reset-app-data .venv && \
    .venv/bin/pip install --no-cache-dir torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

ENTRYPOINT [".venv/bin/python", "-c", "import torch; torch.ops.load_library('.venv/lib/python3.8/site-packages/torchvision/image.so')"]

Build and run with:

docker build -t test .
docker run --gpus all --rm test

This prints out:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/work/.venv/lib/python3.8/site-packages/torch/_ops.py", line 110, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libnvjpeg.so.11: cannot open shared object file: No such file or directory

Versions

Collecting environment information...
PyTorch version: 1.10.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.11.0-1020-gcp-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 470.63.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.3
[pip3] torch==1.10.0+cu113
[pip3] torchaudio==0.10.0+cu113
[pip3] torchvision==0.11.1+cu113
[conda] Could not collect

[NicolasHug]

Thanks for the report @epwalsh

Did this work before with torchvision 0.10 instead of 0.11?

Do you know if the nvjpeg lib comes with the cuda docker image? It should come with any CUDA 11 version, but I'm not too familiar with what docker might be doing under the hood

[epwalsh]

Did this work before with torchvision 0.10 instead of 0.11?

Yes. For example this image works fine:

FROM nvidia/cuda:11.1-base-ubuntu20.04

ARG DEBIAN_FRONTEND="noninteractive"
ENV TZ="America/Los_Angeles"

RUN rm -rf /etc/apt/sources.list.d/* \
    && apt-get update \
    && apt-get install -y \
        build-essential \
        curl \
        wget \
        git \
        zip \
        libssl-dev \
        software-properties-common \
        libffi-dev \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work/

RUN pip3 install --no-cache-dir virtualenv && \
    virtualenv -p $(which python3) --copies --reset-app-data .venv && \
    .venv/bin/pip install --no-cache-dir torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

ENTRYPOINT [".venv/bin/python", "-c", "import torch; torch.ops.load_library('.venv/lib/python3.8/site-packages/torchvision/image.so')"]

Do you know if the nvjpeg lib comes with the cuda docker image?

It doesn't look like nvjpeg is on the Docker image (I don't see it in any of the standard locations).

[NicolasHug]

Thanks for the details,

We were still relying on nvjpeg for torchvision 0.10, so this suggests that the issue either comes from a difference between nvidia/cuda:11.1-base-ubuntu20.04 and nvidia/cuda:11.3.0-base-ubuntu20.04, or that we changed something in the lib discovery between 0.10 and 0.11.

Do you mind trying 0.11 with nvidia/cuda:11.1-base-ubuntu20.04 to check whether this is the former?

[epwalsh]

Sure, I just tried and still got the same error. Here is the full Dockerfile:

FROM nvidia/cuda:11.1-base-ubuntu20.04

ARG DEBIAN_FRONTEND="noninteractive"
ENV TZ="America/Los_Angeles"

RUN rm -rf /etc/apt/sources.list.d/* \
    && apt-get update \
    && apt-get install -y \
        build-essential \
        curl \
        wget \
        git \
        zip \
        libssl-dev \
        software-properties-common \
        libffi-dev \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work/

RUN pip3 install --no-cache-dir virtualenv && \
    virtualenv -p $(which python3) --copies --reset-app-data .venv && \
    .venv/bin/pip install --no-cache-dir torch==1.10.0+cu113 torchvision==0.11.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

ENTRYPOINT [".venv/bin/python", "-c", "import torch; torch.ops.load_library('.venv/lib/python3.8/site-packages/torchvision/image.so')"]

[NicolasHug]

Thanks for the details @epwalsh

@malfet @seemethere do you know if anything has changed regarding the packaging of torchvision that could cause such issue?

[malfet]

Hmm, nvjpeg is bundled with 0.11.1 for cu102, but not for cu111 nor cu113

[malfet]

I've fixed the regression by #4752, added extra testing in pytorch/pytorch.github.io#872
and push new cu113 wheel binaries to download.pytorch.org.
@epwalsh can you please try again on your end and let me know if problem is resolved

[epwalsh]

I can confirm that fixed the issue! Thanks for the quick fix 🙌

[malfet]

Thank you for confirming. Linking #4752 as closing fix and closing the issue