[MaskRCNN/Pytorch] ERROR: No supported GPU(s) detected to run this container

[jzhang82119]

Nvidia A100
driver version 470.42.01
Ubuntu 18.04
Cuda 11.4
container
nvcr.io/nvidia/pytorch:20.06-py3

go inside the container, no nvidia driver found. cannot run any gpu related training/inference. cannot run nvidia-smi inside container either.

Looks like driver / container compatibility issue.

[sharathts]

You will have to update your drivers locally for GPUs to be seen in the latest containers. Closing since issue doesn't pertain to the MASKRCNN model. Feel free to reopen if you have more questions.

[jzhang82119]

this issue is just with the latest nvidia 470 driver. It worked with previous older version of drivers.

[hXl3s]

@jzhang82119 You should try newer pytorch container with latest driver. See if 21.06 works fine. It cannot guarantee that results mentioned in readme will be reproducible with 21.06, but functionally everything should work fine. If not fell free to report issues you encounter with container upgrade and MaskRCNN implementation.

[jzhang82119]

I did try building the docker container ( modify the Dockerfile to use the 21.06 PyTorch container instead of the 20.06 PyTorch container)

But got the following error when building it.

Step 3/10 : RUN pip install --upgrade --no-cache-dir pip && pip install --no-cache-dir mlperf-compliance==0.0.10 opencv-python==3.4.1.15 git+https://github.com/NVIDIA/dllogger yacs
---> Running in de662175545b
Requirement already satisfied: pip in /opt/conda/lib/python3.8/site-packages (21.1.2)
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Collecting git+https://github.com/NVIDIA/dllogger
Cloning https://github.com/NVIDIA/dllogger to /tmp/pip-req-build-uixy69ms
Running command git clone -q https://github.com/NVIDIA/dllogger /tmp/pip-req-build-uixy69ms
Requirement already satisfied: mlperf-compliance==0.0.10 in /opt/conda/lib/python3.8/site-packages (0.0.10)
ERROR: Could not find a version that satisfies the requirement opencv-python==3.4.1.15 (from versions: 3.4.8.29, 3.4.9.31, 3.4.9.33, 3.4.10.35, 3.4.10.37, 3.4.11.39, 3.4.11.41, 3.4.11.43, 3.4.11.45, 3.4.13.47, 3.4.14.51, 3.4.14.53, 4.1.2.30, 4.2.0.32, 4.2.0.34, 4.3.0.36, 4.3.0.38, 4.4.0.40, 4.4.0.42, 4.4.0.44, 4.4.0.46, 4.5.1.48, 4.5.2.52, 4.5.2.54)
ERROR: No matching distribution found for opencv-python==3.4.1.15
The command '/bin/sh -c pip install --upgrade --no-cache-dir pip && pip install --no-cache-dir mlperf-compliance==0.0.10 opencv-python==3.4.1.15 git+https://github.com/NVIDIA/dllogger yacs' returned a non-zero code: 1

[jzhang82119]

also tried 21.07. Same error message when building the container.

[jzhang82119]

the opencv-python seems to be too old in DockerFile.
numpy also seems to be very old.

I have to change Dockerfile to the following content so it can build successfully.

ARG FROM_IMAGE_NAME=nvcr.io/nvidia/pytorch:21.07-py3
FROM ${FROM_IMAGE_NAME}

Install Python dependencies

RUN pip install --upgrade --no-cache-dir pip
&& pip install --no-cache-dir
mlperf-compliance==0.0.10
opencv-python==4.5.3.56
git+https://github.com/NVIDIA/dllogger
yacs

Copy detectron code and build

WORKDIR /workspace/object_detection
RUN mkdir -p /datasets/data
RUN mkdir /results
COPY . .
RUN pip install -e .

RUN /opt/conda/bin/conda install -y numpy==1.21.1

ENV OMP_NUM_THREADS=1

[jzhang82119]

when I am inside the container, it still does not support my gpu. I am using the latest version of nvidia driver 470.57.02

bash scripts/docker/interactive.sh /data

=============
== PyTorch ==

NVIDIA Release 21.07 (build 25165078)
PyTorch Version 1.10.0a0+ecc3718

Container image Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Copyright (c) 2014-2021 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
ERROR: No supported GPU(s) detected to run this container

NOTE: MOFED driver was detected, but nv_peer_mem driver was not detected.
Multi-node communication performance may be reduced.

root@5b150dc133d1:/workspace/object_detection# nvidia-smi
No devices were found

[jzhang82119]

Please keep this issue open. It is not fixed.

[sharathts]

@jzhang82119 you don't need to install opencv and numpy in the Dockerfile explicitly, they are installed in the latest PyTorch base containers. So they can be removed altogether from the Dockerfile.

Will reopen this issue for GPU not being seen and close the duplicate #978

[cliffwoolley]

R470 should work fine with A100 and 20.06 containers (or 21.07 for that matter).

What does nvidia-smi show from outside the container? What about nvidia-smi topo -m?

[kuangllbnu]

I just came across this Error because of not starting the fabric-manager service. After starting the fabric-manager, it works well.