dockerfile failed at build_wheel.py. "file STRINGS file "/include/NvInferVersion.h" cannot be read." #30
Description
In linux with GPU, run the dockerfile build with
cd tensorrtllm_backend
git submodule update --init --recursive
git lfs install
git lfs pull
# Use the Dockerfile to build the backend in a container
# For x86_64
DOCKER_BUILDKIT=1 docker build -t triton_trt_llm -f dockerfile/Dockerfile.trt_llm_backend .
Which failed at cd tensorrt_llm && python3 scripts/build_wheel.py --trt_root=\"${TRT_ROOT}\" -i -c && cd ..
Rerunning this line runs into file STRINGS file "/include/NvInferVersion.h" cannot be read.
~/tensorrtllm_backend release/0.5.0 !1 ?2 ❯ cd tensorrt_llm && python3 scripts/build_wheel.py --trt_root=\"${TRT_ROOT}\" -i -c && cd ..
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: build in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (1.0.3)
Requirement already satisfied: torch in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (2.0.1)
Requirement already satisfied: transformers==4.31.0 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (4.31.0)
Requirement already satisfied: diffusers==0.15.0 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (0.15.0)
Requirement already satisfied: accelerate==0.20.3 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (0.20.3)
Requirement already satisfied: colored in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 6)) (1.4.4)
Requirement already satisfied: polygraphy in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 7)) (0.49.0)
Requirement already satisfied: onnx>=1.12.0 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 8)) (1.14.1)
Requirement already satisfied: mpi4py in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 9)) (3.1.5)
Requirement already satisfied: tensorrt>=8.6.0 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 10)) (8.6.1.post1)
Requirement already satisfied: numpy in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 11)) (1.24.4)
Requirement already satisfied: cuda-python==12.2.0 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 12)) (12.2.0)
Requirement already satisfied: sentencepiece>=0.1.99 in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 13)) (0.1.99)
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from -r requirements.txt (line 14)) (0.34.2)
Requirement already satisfied: lark in /home/hayley/.local/lib/python3.8/site-packages (from -r requirements.txt (line 15)) (1.1.7)
Requirement already satisfied: filelock in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (3.12.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (0.17.3)
Requirement already satisfied: packaging>=20.0 in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (23.2)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (5.3.1)
Requirement already satisfied: regex!=2019.12.17 in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (2023.10.3)
Requirement already satisfied: requests in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (2.31.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (0.4.0)
Requirement already satisfied: tqdm>=4.27 in /home/hayley/.local/lib/python3.8/site-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (4.66.1)
Requirement already satisfied: Pillow in /home/hayley/.local/lib/python3.8/site-packages (from diffusers==0.15.0->-r requirements.txt (line 4)) (10.0.1)
Requirement already satisfied: importlib-metadata in /home/hayley/.local/lib/python3.8/site-packages (from diffusers==0.15.0->-r requirements.txt (line 4)) (6.8.0)
Requirement already satisfied: psutil in /usr/lib/python3/dist-packages (from accelerate==0.20.3->-r requirements.txt (line 5)) (5.5.1)
Requirement already satisfied: cython in /home/hayley/.local/lib/python3.8/site-packages (from cuda-python==12.2.0->-r requirements.txt (line 12)) (3.0.2)
Requirement already satisfied: pyproject_hooks in /home/hayley/.local/lib/python3.8/site-packages (from build->-r requirements.txt (line 1)) (1.0.0)
Requirement already satisfied: tomli>=1.1.0 in /home/hayley/.local/lib/python3.8/site-packages (from build->-r requirements.txt (line 1)) (2.0.1)
Requirement already satisfied: typing-extensions in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (4.8.0)
Requirement already satisfied: sympy in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (1.12)
Requirement already satisfied: networkx in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (3.1)
Requirement already satisfied: jinja2 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (3.1.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.7.99)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.7.99)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.7.101)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (8.5.0.96)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.10.3.66)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (10.9.0.58)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (10.2.10.91)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.4.0.1)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.7.4.91)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (2.14.3)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (11.7.91)
Requirement already satisfied: triton==2.0.0 in /home/hayley/.local/lib/python3.8/site-packages (from torch->-r requirements.txt (line 2)) (2.0.0)
Requirement already satisfied: setuptools in /home/hayley/.local/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch->-r requirements.txt (line 2)) (49.4.0)
Requirement already satisfied: cmake in /home/hayley/.local/lib/python3.8/site-packages (from triton==2.0.0->torch->-r requirements.txt (line 2)) (3.27.6)
Requirement already satisfied: lit in /home/hayley/.local/lib/python3.8/site-packages (from triton==2.0.0->torch->-r requirements.txt (line 2)) (17.0.2)
Requirement already satisfied: protobuf>=3.20.2 in /home/hayley/.local/lib/python3.8/site-packages (from onnx>=1.12.0->-r requirements.txt (line 8)) (4.24.4)
Requirement already satisfied: fsspec in /home/hayley/.local/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.31.0->-r requirements.txt (line 3)) (2023.9.2)
Requirement already satisfied: zipp>=0.5 in /home/hayley/.local/lib/python3.8/site-packages (from importlib-metadata->diffusers==0.15.0->-r requirements.txt (line 4)) (3.17.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/hayley/.local/lib/python3.8/site-packages (from jinja2->torch->-r requirements.txt (line 2)) (2.1.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/hayley/.local/lib/python3.8/site-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (3.3.0)
Requirement already satisfied: idna<4,>=2.5 in /home/hayley/.local/lib/python3.8/site-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/hayley/.local/lib/python3.8/site-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (2.0.6)
Requirement already satisfied: certifi>=2017.4.17 in /home/hayley/.local/lib/python3.8/site-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (2023.7.22)
Requirement already satisfied: mpmath>=0.19 in /home/hayley/.local/lib/python3.8/site-packages (from sympy->torch->-r requirements.txt (line 2)) (1.3.0)
DEPRECATION: omegaconf 2.0.6 has a non-standard dependency specifier PyYAML>=5.1.*. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of omegaconf or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: mlnx-tools -5.8.0- has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of mlnx-tools or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
[notice] A new release of pip is available: 23.2.1 -> 23.3
[notice] To update, run: pip install --upgrade pip
-- The CXX compiler identification is GNU 9.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- NVTX is disabled
-- Importing batch manager
-- Building PyTorch
-- Building Google tests
-- Building benchmarks
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - /usr/local/cuda/bin/nvcc
-- CUDA compiler: /usr/local/cuda/bin/nvcc
-- GPU architectures: 70-real;80-real;86-real;89-real;90-real
-- The CUDA compiler identification is NVIDIA 12.2.140
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found CUDAToolkit: /usr/local/cuda/include (found version "12.2.140")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- ========================= Importing and creating target nvinfer ==========================
-- Looking for library nvinfer
-- Library that was found nvinfer_LIB_PATH-NOTFOUND
-- ==========================================================================================
-- ========================= Importing and creating target nvuffparser ==========================
-- Looking for library nvparsers
-- Library that was found nvparsers_LIB_PATH-NOTFOUND
-- ==========================================================================================
-- CUDAToolkit_VERSION 12.2 is greater or equal than 11.0, enable -DENABLE_BF16 flag
-- CUDAToolkit_VERSION 12.2 is greater or equal than 11.8, enable -DENABLE_FP8 flag
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- COMMON_HEADER_DIRS: /home/hayley/tensorrtllm_backend/tensorrt_llm/cpp;/usr/local/cuda/include
-- TORCH_CUDA_ARCH_LIST: 7.0;8.0;8.6;8.9;9.0
-- Found Python3: /home/hayley/llm_serving/.venv/bin/python3 (found version "3.8.10") found components: Interpreter Development Development.Module Development.Embed
-- Found Python executable at /home/hayley/llm_serving/.venv/bin/python3
-- Found Python libraries at /usr/lib/x86_64-linux-gnu
-- Found CUDA: /usr/local/cuda (found version "12.2")
-- Caffe2: CUDA detected: 12.2
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 12.2
CMake Warning at /home/hayley/llm_serving/.venv/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:166 (message):
Failed to compute shorthash for libnvrtc.so
Call Stack (most recent call first):
/home/hayley/llm_serving/.venv/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
/home/hayley/llm_serving/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:281 (find_package)
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90
CMake Warning at /home/hayley/llm_serving/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/home/hayley/llm_serving/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
CMakeLists.txt:281 (find_package)
-- Found Torch: /home/hayley/llm_serving/.venv/lib/python3.8/site-packages/torch/lib/libtorch.so
-- TORCH_CXX_FLAGS: -D_GLIBCXX_USE_CXX11_ABI=0
CMake Error at CMakeLists.txt:288 (file):
file STRINGS file "/include/NvInferVersion.h" cannot be read.
CMake Error at CMakeLists.txt:291 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:293 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:291 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:293 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:291 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:293 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:291 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:293 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:297 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:299 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:297 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:299 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:297 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
CMake Error at CMakeLists.txt:299 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
-- Building for TensorRT version: .., library version:
-- Using MPI_CXX_INCLUDE_DIRS: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi;/usr/lib/x86_64-linux-gnu/openmpi/include
-- Using MPI_CXX_LIBRARIES: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so;/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- USE_CXX11_ABI: False
CMake Error at tensorrt_llm/plugins/CMakeLists.txt:106 (set_target_properties):
set_target_properties called with incorrect number of arguments.
-- The C compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /home/hayley/llm_serving/.venv/bin/python3 (found version "3.8.10") found components: Interpreter
-- ========================= Importing and creating target nvonnxparser ==========================
-- Looking for library nvonnxparser
-- Library that was found nvonnxparser_LIB_PATH-NOTFOUND
-- ==========================================================================================
-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
File "scripts/build_wheel.py", line 248, in <module>
main(**vars(args))
File "scripts/build_wheel.py", line 149, in main
build_run(
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'cmake -DCMAKE_BUILD_TYPE="Release" -DBUILD_PYT="ON" -DTRT_LIB_DIR=""/targets/x86_64-linux-gnu/lib -DTRT_INCLUDE_DIR=""/include -S "/home/hayley/tensorrtllm_backend/tensorrt_llm/cpp"' returned non-zero exit status 1.
Any ideas?