Installing NVTX on python fails in nvidia docker image.

[golanor]

Hi,

I'm trying to install nvtx using pip inside an nvidia docker image nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04
I'm installed python 3.9 alongside with pip, and when running

python3.9 -m pip install nvtx

I get:

 #13 79.02 Failed to build nvtx
#13 79.02 ERROR: Could not build wheels for nvtx, which is required to install pyproject.toml-based projects

At first I thought this is the same as #43 , but I see these are not exactly the same issue. I'd rather not build the wheel inside the docker, or install conda.

A minimal dockerfile that fails:

FROM nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04

RUN apt update --fix-missing
RUN apt install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Jerusalem
RUN apt install -y python3.9 python3-pip python3.9-dev python3.9-distutils curl unzip
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/
RUN python3.9 -m pip install --upgrade pip \
    && python3.9 -m pip install --upgrade setuptools \
    && python3.9 -m pip install --upgrade distlib \
    && python3.9 -m pip install nvtx

[shwina]

Thanks for reporting. Is there any more information reported indicating exactly what went wrong in the pip install command?

[golanor]

#12 6.576 Successfully installed distlib-0.3.4
#12 6.576 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#12 7.169 Collecting nvtx
#12 7.198   Downloading nvtx-0.2.3.tar.gz (10 kB)
#12 7.377   Installing build dependencies: started
#12 11.60   Installing build dependencies: finished with status 'done'
#12 11.60   Getting requirements to build wheel: started
#12 12.21   Getting requirements to build wheel: finished with status 'done'
#12 12.22   Preparing metadata (pyproject.toml): started
#12 12.69   Preparing metadata (pyproject.toml): finished with status 'done'
#12 12.69 Building wheels for collected packages: nvtx
#12 12.69   Building wheel for nvtx (pyproject.toml): started
#12 13.24   Building wheel for nvtx (pyproject.toml): finished with status 'error'
#12 13.24   ERROR: Command errored out with exit status 1:
#12 13.24    command: /usr/bin/python3.9 /usr/local/lib/python3.9/dist-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmphinsk2ir
#12 13.24        cwd: /tmp/pip-install-49av2fr8/nvtx_482cf1aeddd34265aa84388a242a4646
#12 13.24   Complete output (25 lines):
#12 13.24   running bdist_wheel
#12 13.24   running build
#12 13.24   running build_py
#12 13.24   creating build
#12 13.24   creating build/lib.linux-x86_64-3.9
#12 13.24   creating build/lib.linux-x86_64-3.9/nvtx
#12 13.24   copying nvtx/nvtx.py -> build/lib.linux-x86_64-3.9/nvtx
#12 13.24   copying nvtx/colors.py -> build/lib.linux-x86_64-3.9/nvtx
#12 13.24   copying nvtx/__init__.py -> build/lib.linux-x86_64-3.9/nvtx
#12 13.24   creating build/lib.linux-x86_64-3.9/nvtx/_lib
#12 13.24   copying nvtx/_lib/__init__.py -> build/lib.linux-x86_64-3.9/nvtx/_lib
#12 13.24   creating build/lib.linux-x86_64-3.9/nvtx/utils
#12 13.24   copying nvtx/utils/cached.py -> build/lib.linux-x86_64-3.9/nvtx/utils
#12 13.24   copying nvtx/utils/__init__.py -> build/lib.linux-x86_64-3.9/nvtx/utils
#12 13.24   copying nvtx/_lib/lib.pxd -> build/lib.linux-x86_64-3.9/nvtx/_lib
#12 13.24   running build_ext
#12 13.24   creating build/temp.linux-x86_64-3.9
#12 13.24   creating build/temp.linux-x86_64-3.9/nvtx
#12 13.24   creating build/temp.linux-x86_64-3.9/nvtx/_lib
#12 13.24   x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include -I/usr/include/python3.9 -c nvtx/_lib/lib.c -o build/temp.linux-x86_64-3.9/nvtx/_lib/lib.o
#12 13.24   nvtx/_lib/lib.c:701:10: fatal error: nvtx3/nvToolsExt.h: No such file or directory
#12 13.24    #include "nvtx3/nvToolsExt.h"
#12 13.24             ^~~~~~~~~~~~~~~~~~~~
#12 13.24   compilation terminated.
#12 13.24   error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
#12 13.24   ----------------------------------------
#12 13.24   ERROR: Failed building wheel for nvtx
#12 13.24 Failed to build nvtx
#12 13.24 ERROR: Could not build wheels for nvtx, which is required to install pyproject.toml-based projects

[shwina]

Ah -- thanks. We don't have pre-built wheels for Python 3.9 just yet. I just opened #49 to address this. Once that's merged, I'll push the wheels onto PyPI and update here.

[golanor]

Hey, is there anything I can do to expedite the PR merge?

[shwina]

Thanks, and apologies for the delay. I'll aim to get it merged in today.

[shwina]

Hi @golanor - could you please confirm that the pip install now works?

[golanor]

Still fails:

× Building wheel for nvtx (pyproject.toml) did not run successfully.
#13 100.4   │ exit code: 1
#13 100.4   ╰─> [10 lines of output]
#13 100.4       running bdist_wheel
#13 100.4       running build
#13 100.4       running build_py
#13 100.4       running build_ext
#13 100.4       building 'nvtx._lib.lib' extension
#13 100.4       nvtx/_lib/lib.c:701:10: fatal error: nvtx3/nvToolsExt.h: No such file or directory
#13 100.4        #include "nvtx3/nvToolsExt.h"
#13 100.4                 ^~~~~~~~~~~~~~~~~~~~
#13 100.4       compilation terminated.
#13 100.4       error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
#13 100.4       [end of output]
#13 100.4   
#13 100.4   note: This error originates from a subprocess, and is likely not a problem with pip.

[shwina]

Could I please see the full output?

[shwina]

Hmm it could be because you're on Ubuntu 18.04. I'll try to repro in this docker container and report back.

[ovcharenkoo]

Hi @golanor, as a workaround try building your image from NGC PyTorch image(docker pull nvcr.io/nvidia/pytorch:22.01-py3) rather than Cuda image. There is nvtx already pre-installed for Python 3.8.12

[golanor]

Hi @golanor, as a workaround try building your image from NGC PyTorch image(docker pull nvcr.io/nvidia/pytorch:22.01-py3) rather than Cuda image. There is nvtx already pre-installed for Python 3.8.12

Thanks, but installing PyTorch will make my already too bloated docker image even more so.