Unable to build tensorrt_llm backend; problems with CXX11 ABI

[jlewi]

System Info

• CPU Architecture x86_64

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I'm trying to build the backend using the commands below.

  export TRT_VERSION=10.0.1.6
  export TRT_ROOT=/usr/local/TensorRT-10.2.0.19
 python3 ../tensorrt_llm/scripts/build_wheel.py --trt_root ${TRT_ROOT} \
        --cpp_only \
        -D "CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.3/" -D "ENABLE_MULTI_DEVICE=1"

      cmake "-DTRT_VERSION=${TRT_VERSION}" \
      "-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}" \
      "-DVCPKG_TARGET_TRIPLET=${VCPKG_TARGET_TRIPLET}" \
      "-DTRT_LIB_DIR=${TRT_ROOT}/targets/${ARCH}-linux-gnu/lib" \
      "-DTRT_INCLUDE_DIR=${TRT_ROOT}/include" \
      "-DUSE_CXX11_ABI:BOOL=ON" \
      "-DCMAKE_BUILD_TYPE=Release" \
      "-DCMAKE_INSTALL_PREFIX:PATH=/tmp/tritonbuild/tensorrtllm/install" \
      "-DTRITON_REPO_ORGANIZATION:STRING=https://github.com/triton-inference-server" \
      "-DTRITON_COMMON_REPO_TAG:STRING=r24.05" \
      "-DTRITON_CORE_REPO_TAG:STRING=r24.05" \
      "-DTRITON_BACKEND_REPO_TAG:STRING=r24.05" \
      "-DTRITON_ENABLE_GPU:BOOL=ON" \
      "-DTRITON_ENABLE_MALI_GPU:BOOL=OFF" \
      "-DTRITON_ENABLE_STATS:BOOL=ON" \
      "-DTRITON_ENABLE_METRICS:BOOL=ON" \
      "-DTRITON_ENABLE_MEMORY_TRACKER:BOOL=ON" \
      -S \
      ../inflight_batcher_llm \
      -B \
      .
      
      cmake --build . --config Release -j20  -t install

The cmake install command fails with a linking error complaining about undefined references.

[ 97%] Linking CXX shared library libtriton_tensorrtllm.so
/usr/bin/cmake -E cmake_link_script CMakeFiles/triton-tensorrt-llm-backend.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -O2 -Wall -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -march=x86-64-v2 -mtune=broadwell -O3 -DNDEBUG -Wl,--version-script libtriton_tensorrtllm.ldscript -Wl,-rpath,'$ORIGIN' -Wl,--no-undefined -Wl,--as-needed,-O1,--sort-common -Wl,-z,relro,-z,now,-z,noexecstack -shared -Wl,-soname,libtriton_tensorrtllm.so -o libtriton_tensorrtllm.so "CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o"   -L/usr/local/cuda-12.3/lib64/stubs  -L/usr/local/cuda-12.3/lib64  -Wl,-rpath,/home/build/backend/build:/home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm:/home/build/backend/build/_deps/repo-core-build:/usr/local/cuda-12.3/lib64:/usr/local/cuda-12.3/lib64/stubs:/usr/local/TensorRT-10.2.0.19/targets/x86_64-linux-gnu/lib:/home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins libtriton_tensorrtllm_common.so /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so _deps/repo-core-build/libtritonserver.so _deps/repo-backend-build/libtritonbackendutils.a _deps/repo-common-build/src/libtritonasyncworkqueue.a /usr/local/cuda-12.3/lib64/libcudart.so -ldl /usr/lib/librt.a _deps/repo-backend-build/libkernel_library_new.a /usr/local/cuda-12.3/lib64/libcupti.so /usr/lib64/libmpi.so /usr/local/cuda-12.3/lib64/libcudart.so /usr/local/cuda-12.3/lib64/stubs/libnvidia-ml.so /usr/local/TensorRT-10.2.0.19/targets/x86_64-linux-gnu/lib/libnvinfer.so /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so -lcudadevrt -lcudart_static -lrt -lpthread -ldl
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<float>(float const*, int, int, int, bool)':
libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib]+0x1e7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib]+0x1fe): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<__half>(__half const*, int, int, int, bool)':
libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib]+0x27a): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib]+0x291): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<unsigned int>(unsigned int const*, int, int, int, bool)':
libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib]+0x1d7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib]+0x1ee): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<unsigned long>(unsigned long const*, int, int, int, bool)':
libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib]+0x1d7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib]+0x1ee): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<int>(int const*, int, int, int, bool)':
libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib]+0x1d7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
/usr/lib/gcc/x86_64-pc-linux-gnu/12.4.0/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib]+0x1ee): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/triton-tensorrt-llm-backend.dir/build.make:112: libtriton_tensorrtllm.so] Error 1
make[2]: Leaving directory '/home/build/backend/build'
make[1]: *** [CMakeFiles/Makefile2:303: CMakeFiles/triton-tensorrt-llm-backend.dir/all] Error 2
make[1]: Leaving directory '/home/build/backend/build'
make: *** [Makefile:136: all] Error 2

Those symbols seem to be defined in libtensorrt_llm.so which is included in the command

 nm -D /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so  | c++filt | grep TllmException
00000000007c4a10 T tensorrt_llm::common::TllmException::demangle(char const*)
00000000007c5280 T tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
00000000007c5280 T tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
00000000007c49e0 T tensorrt_llm::common::TllmException::~TllmException()
00000000007c49c0 T tensorrt_llm::common::TllmException::~TllmException()
00000000007c49c0 T tensorrt_llm::common::TllmException::~TllmException()
00000000007c4b50 T tensorrt_llm::common::TllmException::getTrace() const
000000006eb66e90 V typeinfo for tensorrt_llm::common::TllmException
00000000028d2a80 V typeinfo name for tensorrt_llm::common::TllmException
000000006eb66ea8 V vtable for tensorrt_llm::common::TllmException

I came across this earlier issue NVIDIA/TensorRT-LLM#277 which said there was a problem with linking when only building the c++ bindings but that issue claims the issue was fixed in TensorRT-LLM.

I'm only building the cpp bindings because of NVIDIA/TensorRT-LLM#1989

Expected behavior

I expected it to build.

actual behavior

Got the linking error provided above

additional notes

Is an issue similar to NVIDIA/TensorRT-LLM#277 affecting this repository?

[jlewi]

So the problem appears to be not correctly compiling with CXX11 ABI.

Details of linking errors for libtriton_tensorrtllm.so

When it tries to build libtriton_tensorrtllm.so I get undefined errors:

2024/08/07 17:29:40 INFO [100%] Linking CXX shared library libtriton_tensorrtllm.so
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<float>(float const*, int, int, int, bool)':
2024/08/07 17:29:40 WARN libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib]+0x1e7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIfEEvPKT_iiib]+0x1fe): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<__half>(__half const*, int, int, int, bool)':
2024/08/07 17:29:40 WARN libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib]+0x27a): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixI6__halfEEvPKT_iiib]+0x291): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<unsigned int>(unsigned int const*, int, int, int, bool)':
2024/08/07 17:29:40 WARN libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib]+0x1d7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIjEEvPKT_iiib]+0x1ee): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<unsigned long>(unsigned long const*, int, int, int, bool)':
2024/08/07 17:29:40 WARN libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib]+0x1d7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixImEEvPKT_iiib]+0x1ee): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o: in function `void tensorrt_llm::common::printMatrix<int>(int const*, int, int, int, bool)':
2024/08/07 17:29:40 WARN libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib]+0x1d7): undefined reference to `tensorrt_llm::common::fmtstr[abi:cxx11](char const*, ...)'
2024/08/07 17:29:40 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: libtensorrtllm.cc:(.text._ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib[_ZN12tensorrt_llm6common11printMatrixIiEEvPKT_iiib]+0x1ee): undefined reference to `tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
2024/08/07 17:29:40 WARN collect2: error: ld returned 1 exit status
2024/08/07 17:29:40 WARN make[2]: *** [CMakeFiles/triton-tensorrt-llm-backend.dir/build.make:112: libtriton_tensorrtllm.so] Error 1
2024/08/07 17:29:40 WARN make[1]: *** [CMakeFiles/Makefile2:303: CMakeFiles/triton-tensorrt-llm-backend.dir/all] Error 2
2024/08/07 17:29:40 WARN make: *** [Makefile:136: all] Error 2

It looks like that symbol is defined in the file /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so

 nm -D /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so  | c++filt | grep TllmException
00000000007c4b00 T tensorrt_llm::common::TllmException::demangle(char const*)
00000000007c5370 T tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
00000000007c5370 T tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
00000000007c4ad0 T tensorrt_llm::common::TllmException::~TllmException()
00000000007c4ab0 T tensorrt_llm::common::TllmException::~TllmException()
00000000007c4ab0 T tensorrt_llm::common::TllmException::~TllmException()
00000000007c4c40 T tensorrt_llm::common::TllmException::getTrace() const
000000006eb0de90 V typeinfo for tensorrt_llm::common::TllmException
000000000288aa00 V typeinfo name for tensorrt_llm::common::TllmException
000000006eb0dea8 V vtable for tensorrt_llm::common::TllmException

I ran cmake with the verbose flag to see what command it is running.

Here's the command

cmake --build . --config Release -j20 -t install verbose
...
/usr/bin/c++ -fPIC -O2 -Wall -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -march=x86-64-v2 -mtune=broadwell -O3 -DNDEBUG -Wl,--version-script libtriton_tensorrtllm.ldscript -Wl,-rpath,'$ORIGIN' -Wl,--no-undefined -Wl,--as-needed,-O1,--sort-common -Wl,-z,relro,-z,now,-z,noexecstack -shared -Wl,-soname,libtriton_tensorrtllm.so -o libtriton_tensorrtllm.so "CMakeFiles/triton-tensorrt-llm-backend.dir/src/libtensorrtllm.cc.o"   -L/usr/local/cuda-12.3/lib64/stubs  -L/usr/local/cuda-12.3/lib64  -Wl,-rpath,/home/build/backend/build:/home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm:/home/build/backend/build/_deps/repo-core-build:/usr/local/cuda-12.3/lib64:/usr/local/cuda-12.3/lib64/stubs:/usr/local/TensorRT-10.2.0.19/targets/x86_64-linux-gnu/lib:/home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins libtriton_tensorrtllm_common.so /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so _deps/repo-core-build/libtritonserver.so _deps/repo-backend-build/libtritonbackendutils.a _deps/repo-common-build/src/libtritonasyncworkqueue.a /usr/local/cuda-12.3/lib64/libcudart.so -ldl /usr/lib/librt.a _deps/repo-backend-build/libkernel_library_new.a /usr/local/cuda-12.3/lib64/libcupti.so /usr/lib64/libmpi.so /usr/local/cuda-12.3/lib64/libcudart.so /usr/local/cuda-12.3/lib64/stubs/libnvidia-ml.so /usr/local/TensorRT-10.2.0.19/targets/x86_64-linux-gnu/lib/libnvinfer.so /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so -lcudadevrt -lcudart_static -lrt -lpthread -ldl

We can see in this command that is including /home/build/backend/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so

Here are the undefined symbols in libtriton_tensorrtllm_common.so

nm -D libtriton_tensorrtllm_common.so | c++filt | grep TllmException
                 U tensorrt_llm::common::TllmException::TllmException(char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
                 U tensorrt_llm::common::TllmException::~TllmException()
                 U typeinfo for tensorrt_llm::common::TllmException

It looks like the function signatures for TllmException::TllmException might be slightly different. For the third argument in libtriton_tensorrtllm_common it is std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const& whereas in libtensorrt_llm.so it is std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&. Could this explain why the signatures don't match?

According to Claude:

Here's what Claude told me about this error.

The key difference is the presence of __cxx11:: in the namespace for the basic_string in libtriton_tensorrtllm_common.
This difference is directly related to the C++11 ABI change we discussed earlier. Here's what's happening:

The __cxx11:: version is using the new C++11 ABI for std::string.
The version without __cxx11:: is using the older, pre-C++11 ABI for std::string.

How Is ABI Determined

Here's where in the CMakeLists.txt where I think it determines whether or not use CXX11. Notably, it looks like it tries to import pytorch to see how it was compiled and if it was compiled with cxx11_abi. I tested the version of torch that was installed when our build failed and it was not compiled with CXX11 ABI.

I think it was coming from the fact that the build process pip installs this requirements.txt.

I tried installing a version of pytorch that was compiled with CXX11 but it looks like that torch version might have been replaced by the pip command.

So next I tried explicitly setting it to use CXX11 by defining the option

 python3 ../tensorrt_llm/scripts/build_wheel.py --trt_root ${TRT_ROOT} \
        -D "CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.3/" -D "ENABLE_MULTI_DEVICE=1" \
        -D "USE_CXX11_ABI=1" 

This seemed to correctly compile libtensorrt_llm.so with CXX11.

nm -D ./tensorrt_llm/cpp/build/tensorrt_llm/libtensorrt_llm.so | c++filt | grep setBadWords
00000000023c4660 T tensorrt_llm::executor::Request::setBadWords(std::__cxx11::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&)

• Here we can see its using the __cxx11::list

However, I got undefined references when trying to generate the python bindings. See below. It looks like the bindings are not compiled with CXX11 and therefore we get undefined references.

Does anyone know how CXX11 compile options is controlled in the bindings?

Undefined references when linking the bindings.so

2024/08/08 01:08:53 INFO [100%] Linking CXX shared module bindings.cpython-311-x86_64-linux-gnu.so
2024/08/08 01:08:56 WARN In function '__copy_m',
2024/08/08 01:08:56 WARN     inlined from '__copy_move_a2' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_algobase.h:501:30,
2024/08/08 01:08:56 WARN     inlined from '__copy_move_a1' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_algobase.h:528:42,
2024/08/08 01:08:56 WARN     inlined from '__copy_move_a' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_algobase.h:535:31,
2024/08/08 01:08:56 WARN     inlined from 'copy' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_algobase.h:626:7,
2024/08/08 01:08:56 WARN     inlined from '__uninit_copy' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_uninitialized.h:147:27,
2024/08/08 01:08:56 WARN     inlined from 'uninitialized_copy' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_uninitialized.h:185:15,
2024/08/08 01:08:56 WARN     inlined from '__uninitialized_copy_a' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_uninitialized.h:372:37,
2024/08/08 01:08:56 WARN     inlined from '_M_assign_aux' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/vector.tcc:339:35,
2024/08/08 01:08:56 WARN     inlined from 'operator=' at /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_vector.h:787:21,
2024/08/08 01:08:56 WARN     inlined from 'initialize.constprop' at /usr/lib/python3.11/site-packages/torch/include/pybind11/pybind11.h:1343:71:
2024/08/08 01:08:56 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/c++/bits/stl_algobase.h:434:30: warning: '__builtin_memcpy' reading between 9 and 9223372036854775807 bytes from a region of size 8 [-Wstringop-overread]
2024/08/08 01:08:56 WARN   434 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
2024/08/08 01:08:56 WARN       |                              ^
2024/08/08 01:08:56 WARN /usr/lib/python3.11/site-packages/torch/include/pybind11/pybind11.h: In member function 'initialize.constprop':
2024/08/08 01:08:56 WARN /usr/lib/python3.11/site-packages/torch/include/pybind11/pybind11.h:1343:71: note: at offset [0, 7] into source object '<anonymous>' of size 8
2024/08/08 01:08:56 WARN  1343 |         internals.registered_types_py[(PyTypeObject *) m_ptr] = {tinfo};
2024/08/08 01:08:56 WARN       |                                                                       ^
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans9.ltrans.o: in function `tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x545f): undefined reference to `tensorrt_llm::executor::Request::setBadWords(std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x5466): undefined reference to `tensorrt_llm::executor::Request::getBadWords() const'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x5498): undefined reference to `tensorrt_llm::executor::Request::setStopWords(std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x549f): undefined reference to `tensorrt_llm::executor::Request::getStopWords() const'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x5595): undefined reference to `tensorrt_llm::executor::Request::setLogitsPostProcessorName(std::string const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x559c): undefined reference to `tensorrt_llm::executor::Request::getLogitsPostProcessorName() const'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x5c55): undefined reference to `tensorrt_llm::executor::Response::getErrorMsg() const'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x6b27): undefined reference to `tensorrt_llm::executor::OrchestratorConfig::setWorkerExecutablePath(std::string const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x6b2e): undefined reference to `tensorrt_llm::executor::OrchestratorConfig::getWorkerExecutablePath() const'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x7fd3): undefined reference to `tensorrt_llm::executor::ExecutorConfig::setLogitsPostProcessorMap(std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x7fda): undefined reference to `tensorrt_llm::executor::ExecutorConfig::getLogitsPostProcessorMap() const'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans10.ltrans.o: in function `pybind11::cpp_function::initialize<tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(tensorrt_llm::executor::RequestStats const&)#2}, std::string, tensorrt_llm::executor::RequestStats const&, pybind11::name, pybind11::is_method, pybind11::sibling>(tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(tensorrt_llm::executor::RequestStats const&)#2}&&, std::string (*)(tensorrt_llm::executor::RequestStats const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .lto_priv.0]':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x65e8): undefined reference to `tensorrt_llm::executor::JsonSerialization::toJsonStr(tensorrt_llm::executor::RequestStats const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x663e): undefined reference to `tensorrt_llm::executor::JsonSerialization::toJsonStr(tensorrt_llm::executor::RequestStats const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans10.ltrans.o: in function `pybind11::cpp_function::initialize<tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(tensorrt_llm::executor::RequestStatsPerIteration const&)#3}, std::string, tensorrt_llm::executor::RequestStatsPerIteration const&, pybind11::name, pybind11::is_method, pybind11::sibling>(tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(tensorrt_llm::executor::RequestStatsPerIteration const&)#3}&&, std::string (*)(tensorrt_llm::executor::RequestStatsPerIteration const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .lto_priv.0]':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x66f8): undefined reference to `tensorrt_llm::executor::JsonSerialization::toJsonStr(tensorrt_llm::executor::RequestStatsPerIteration const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x674e): undefined reference to `tensorrt_llm::executor::JsonSerialization::toJsonStr(tensorrt_llm::executor::RequestStatsPerIteration const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans10.ltrans.o: in function `pybind11::cpp_function::initialize<tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(tensorrt_llm::executor::IterationStats const&)#1}, std::string, tensorrt_llm::executor::IterationStats const&, pybind11::name, pybind11::is_method, pybind11::sibling>(tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(tensorrt_llm::executor::IterationStats const&)#1}&&, std::string (*)(tensorrt_llm::executor::IterationStats const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .lto_priv.0]':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x6a78): undefined reference to `tensorrt_llm::executor::JsonSerialization::toJsonStr(tensorrt_llm::executor::IterationStats const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x6aee): undefined reference to `tensorrt_llm::executor::JsonSerialization::toJsonStr(tensorrt_llm::executor::IterationStats const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans12.ltrans.o: in function `pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<unsigned long, std::string>::execute<pybind11::class_<tensorrt_llm::executor::Response>, pybind11::arg, pybind11::arg, 0>(pybind11::class_<tensorrt_llm::executor::Response>&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::value_and_holder&, unsigned long, std::string)#1}, void, pybind11::detail::value_and_holder&, unsigned long, std::string, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg>(pybind11::class_<tensorrt_llm::executor::Response>&&, void (*)(pybind11::detail::value_and_holder&, unsigned long, std::string), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&)':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x4850): undefined reference to `tensorrt_llm::executor::Response::Response(unsigned long, std::string)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x48a4): undefined reference to `tensorrt_llm::executor::Response::Response(unsigned long, std::string)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans12.ltrans.o: in function `pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<bool, std::string>::execute<pybind11::class_<tensorrt_llm::executor::OrchestratorConfig>, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::executor::OrchestratorConfig>&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, bool, std::string)#1}, void, pybind11::detail::value_and_holder&, bool, std::string, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<tensorrt_llm::executor::OrchestratorConfig>&&, void (*)(pybind11::detail::value_and_holder&, bool, std::string), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&)':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x4a2d): undefined reference to `tensorrt_llm::executor::OrchestratorConfig::OrchestratorConfig(bool, std::string, std::shared_ptr<tensorrt_llm::mpi::MpiComm>)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x4b29): undefined reference to `tensorrt_llm::executor::OrchestratorConfig::OrchestratorConfig(bool, std::string, std::shared_ptr<tensorrt_llm::mpi::MpiComm>)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans13.ltrans.o: in function `pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<std::string const&, std::string const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&>::execute<pybind11::class_<tensorrt_llm::pybind::executor::Executor>, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, 0>(pybind11::class_<tensorrt_llm::pybind::executor::Executor>&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::value_and_holder&, std::string const&, std::string const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&)#1}, void, pybind11::detail::value_and_holder&, std::string const&, std::string const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg>(pybind11::class_<tensorrt_llm::pybind::executor::Executor>&&, void (*)(pybind11::detail::value_and_holder&, std::string const&, std::string const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&)':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x38c8): undefined reference to `tensorrt_llm::executor::Executor::Executor(std::vector<unsigned char, std::allocator<unsigned char> > const&, std::string const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x3a27): undefined reference to `tensorrt_llm::executor::Executor::Executor(std::vector<unsigned char, std::allocator<unsigned char> > const&, std::string const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans13.ltrans.o: in function `pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<std::filesystem::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&>::execute<pybind11::class_<tensorrt_llm::pybind::executor::Executor>, pybind11::arg, pybind11::arg, pybind11::arg, 0>(pybind11::class_<tensorrt_llm::pybind::executor::Executor>&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::value_and_holder&, std::filesystem::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&)#1}, void, pybind11::detail::value_and_holder&, std::filesystem::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg, pybind11::arg, pybind11::arg>(pybind11::class_<tensorrt_llm::pybind::executor::Executor>&&, void (*)(pybind11::detail::value_and_holder&, std::filesystem::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&)':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x3f65): undefined reference to `tensorrt_llm::executor::Executor::Executor(std::filesystem::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: <artificial>:(.text+0x41c2): undefined reference to `tensorrt_llm::executor::Executor::Executor(std::filesystem::path const&, tensorrt_llm::executor::ModelType, tensorrt_llm::executor::ExecutorConfig const&)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans14.ltrans.o: in function `void pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, std::vector<int, std::allocator<int> >, int, bool, tensorrt_llm::executor::SamplingConfig const&, tensorrt_llm::executor::OutputConfig const&, std::optional<int> const&, std::optional<int> const&, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::Tensor>, std::optional<tensorrt_llm::executor::SpeculativeDecodingConfig>, std::optional<tensorrt_llm::executor::PromptTuningConfig>, std::optional<tensorrt_llm::executor::LoraConfig>, std::optional<std::string> >::call_impl<void, pybind11::detail::initimpl::constructor<std::vector<int, std::allocator<int> >, int, bool, tensorrt_llm::executor::SamplingConfig const&, tensorrt_llm::executor::OutputConfig const&, std::optional<int> const&, std::optional<int> const&, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::Tensor>, std::optional<tensorrt_llm::executor::SpeculativeDecodingConfig>, std::optional<tensorrt_llm::executor::PromptTuningConfig>, std::optional<tensorrt_llm::executor::LoraConfig>, std::optional<std::string> >::execute<pybind11::class_<tensorrt_llm::executor::Request>, pybind11::arg, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::executor::Request>&, pybind11::arg const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::vector<int, std::allocator<int> >, int, bool, tensorrt_llm::executor::SamplingConfig const&, tensorrt_llm::executor::OutputConfig const&, std::optional<int> const&, std::optional<int> const&, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::Tensor>, std::optional<tensorrt_llm::executor::SpeculativeDecodingConfig>, std::optional<tensorrt_llm::executor::PromptTuningConfig>, std::optional<tensorrt_llm::executor::LoraConfig>, std::optional<std::string>)#1}&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, pybind11::detail::void_type>(pybind11::detail::initimpl::constructor<std::vector<int, std::allocator<int> >, int, bool, tensorrt_llm::executor::SamplingConfig const&, tensorrt_llm::executor::OutputConfig const&, std::optional<int> const&, std::optional<int> const&, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::Tensor>, std::optional<tensorrt_llm::executor::SpeculativeDecodingConfig>, std::optional<tensorrt_llm::executor::PromptTuningConfig>, std::optional<tensorrt_llm::executor::LoraConfig>, std::optional<std::string> >::execute<pybind11::class_<tensorrt_llm::executor::Request>, pybind11::arg, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::executor::Request>&, pybind11::arg const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::vector<int, std::allocator<int> >, int, bool, tensorrt_llm::executor::SamplingConfig const&, tensorrt_llm::executor::OutputConfig const&, std::optional<int> const&, std::optional<int> const&, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::Tensor>, std::optional<tensorrt_llm::executor::SpeculativeDecodingConfig>, std::optional<tensorrt_llm::executor::PromptTuningConfig>, std::optional<tensorrt_llm::executor::LoraConfig>, std::optional<std::string>)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul>, pybind11::detail::void_type&&) && [clone .constprop.0]':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x4b2e): undefined reference to `tensorrt_llm::executor::Request::Request(std::vector<int, std::allocator<int> >, int, bool, tensorrt_llm::executor::SamplingConfig const&, tensorrt_llm::executor::OutputConfig const&, std::optional<int> const&, std::optional<int> const&, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<std::list<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::Tensor>, std::optional<tensorrt_llm::executor::SpeculativeDecodingConfig>, std::optional<tensorrt_llm::executor::PromptTuningConfig>, std::optional<tensorrt_llm::executor::LoraConfig>, std::optional<std::string>)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans14.ltrans.o: in function `void pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, int, tensorrt_llm::executor::SchedulerConfig const&, tensorrt_llm::executor::KvCacheConfig const&, bool, bool, int, int, tensorrt_llm::executor::BatchingType, std::optional<tensorrt_llm::executor::ParallelConfig>, tensorrt_llm::executor::PeftCacheConfig const&, std::optional<std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > >, std::optional<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::DecodingMode> >::call_impl<void, pybind11::detail::initimpl::constructor<int, tensorrt_llm::executor::SchedulerConfig const&, tensorrt_llm::executor::KvCacheConfig const&, bool, bool, int, int, tensorrt_llm::executor::BatchingType, std::optional<tensorrt_llm::executor::ParallelConfig>, tensorrt_llm::executor::PeftCacheConfig const&, std::optional<std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > >, std::optional<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::DecodingMode> >::execute<pybind11::class_<tensorrt_llm::executor::ExecutorConfig>, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::executor::ExecutorConfig>&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, int, tensorrt_llm::executor::SchedulerConfig const&, tensorrt_llm::executor::KvCacheConfig const&, bool, bool, int, int, tensorrt_llm::executor::BatchingType, std::optional<tensorrt_llm::executor::ParallelConfig>, tensorrt_llm::executor::PeftCacheConfig const&, std::optional<std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > >, std::optional<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::DecodingMode>)#1}&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, pybind11::detail::void_type>(pybind11::detail::initimpl::constructor<int, tensorrt_llm::executor::SchedulerConfig const&, tensorrt_llm::executor::KvCacheConfig const&, bool, bool, int, int, tensorrt_llm::executor::BatchingType, std::optional<tensorrt_llm::executor::ParallelConfig>, tensorrt_llm::executor::PeftCacheConfig const&, std::optional<std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > >, std::optional<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::DecodingMode> >::execute<pybind11::class_<tensorrt_llm::executor::ExecutorConfig>, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::executor::ExecutorConfig>&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, int, tensorrt_llm::executor::SchedulerConfig const&, tensorrt_llm::executor::KvCacheConfig const&, bool, bool, int, int, tensorrt_llm::executor::BatchingType, std::optional<tensorrt_llm::executor::ParallelConfig>, tensorrt_llm::executor::PeftCacheConfig const&, std::optional<std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > >, std::optional<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::DecodingMode>)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul>, pybind11::detail::void_type&&) && [clone .constprop.0]':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x6644): undefined reference to `tensorrt_llm::executor::ExecutorConfig::ExecutorConfig(int, tensorrt_llm::executor::SchedulerConfig const&, tensorrt_llm::executor::KvCacheConfig const&, bool, bool, int, int, tensorrt_llm::executor::BatchingType, std::optional<tensorrt_llm::executor::ParallelConfig>, std::optional<tensorrt_llm::executor::PeftCacheConfig> const&, std::optional<std::unordered_map<std::string, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::function<void (unsigned long, tensorrt_llm::executor::Tensor&, std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > const&, std::shared_ptr<tensorrt_llm::runtime::CudaStream>&)> > > > >, std::optional<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > >, std::optional<tensorrt_llm::executor::DecodingMode>, float)'
2024/08/08 01:09:00 WARN /usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/ccggXlF0.ltrans15.ltrans.o: in function `void pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, std::filesystem::path const&, tensorrt_llm::batch_manager::TrtGptModelType, int, tensorrt_llm::executor::SchedulerConfig const&, std::function<std::list<tensorrt_llm::pybind::batch_manager::InferenceRequest, std::allocator<tensorrt_llm::pybind::batch_manager::InferenceRequest> > (int)>, std::function<void (unsigned long, std::list<tensorrt_llm::pybind::batch_manager::NamedTensor, std::allocator<tensorrt_llm::pybind::batch_manager::NamedTensor> > const&, bool, std::string const&)>, std::function<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> > ()>, std::function<void (std::string const&)>, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&, std::optional<unsigned long> >::call_impl<void, pybind11::detail::initimpl::constructor<std::filesystem::path const&, tensorrt_llm::batch_manager::TrtGptModelType, int, tensorrt_llm::executor::SchedulerConfig const&, std::function<std::list<tensorrt_llm::pybind::batch_manager::InferenceRequest, std::allocator<tensorrt_llm::pybind::batch_manager::InferenceRequest> > (int)>, std::function<void (unsigned long, std::list<tensorrt_llm::pybind::batch_manager::NamedTensor, std::allocator<tensorrt_llm::pybind::batch_manager::NamedTensor> > const&, bool, std::string const&)>, std::function<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> > ()>, std::function<void (std::string const&)>, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&, std::optional<unsigned long> >::execute<pybind11::class_<tensorrt_llm::pybind::batch_manager::GptManager>, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::pybind::batch_manager::GptManager>&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::filesystem::path const&, tensorrt_llm::batch_manager::TrtGptModelType, int, tensorrt_llm::executor::SchedulerConfig const&, std::function<std::list<tensorrt_llm::pybind::batch_manager::InferenceRequest, std::allocator<tensorrt_llm::pybind::batch_manager::InferenceRequest> > (int)>, std::function<void (unsigned long, std::list<tensorrt_llm::pybind::batch_manager::NamedTensor, std::allocator<tensorrt_llm::pybind::batch_manager::NamedTensor> > const&, bool, std::string const&)>, std::function<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> > ()>, std::function<void (std::string const&)>, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&, std::optional<unsigned long>)#1}&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, pybind11::detail::void_type>(pybind11::detail::initimpl::constructor<std::filesystem::path const&, tensorrt_llm::batch_manager::TrtGptModelType, int, tensorrt_llm::executor::SchedulerConfig const&, std::function<std::list<tensorrt_llm::pybind::batch_manager::InferenceRequest, std::allocator<tensorrt_llm::pybind::batch_manager::InferenceRequest> > (int)>, std::function<void (unsigned long, std::list<tensorrt_llm::pybind::batch_manager::NamedTensor, std::allocator<tensorrt_llm::pybind::batch_manager::NamedTensor> > const&, bool, std::string const&)>, std::function<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> > ()>, std::function<void (std::string const&)>, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&, std::optional<unsigned long> >::execute<pybind11::class_<tensorrt_llm::pybind::batch_manager::GptManager>, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, 0>(pybind11::class_<tensorrt_llm::pybind::batch_manager::GptManager>&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::value_and_holder&, std::filesystem::path const&, tensorrt_llm::batch_manager::TrtGptModelType, int, tensorrt_llm::executor::SchedulerConfig const&, std::function<std::list<tensorrt_llm::pybind::batch_manager::InferenceRequest, std::allocator<tensorrt_llm::pybind::batch_manager::InferenceRequest> > (int)>, std::function<void (unsigned long, std::list<tensorrt_llm::pybind::batch_manager::NamedTensor, std::allocator<tensorrt_llm::pybind::batch_manager::NamedTensor> > const&, bool, std::string const&)>, std::function<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> > ()>, std::function<void (std::string const&)>, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&, std::optional<unsigned long>)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul>, pybind11::detail::void_type&&) && [clone .constprop.0]':
2024/08/08 01:09:00 WARN <artificial>:(.text+0x7b3): undefined reference to `tensorrt_llm::batch_manager::GptManager::GptManager(std::filesystem::path const&, tensorrt_llm::batch_manager::TrtGptModelType, int, tensorrt_llm::executor::SchedulerConfig const&, std::function<std::list<std::shared_ptr<tensorrt_llm::batch_manager::InferenceRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::InferenceRequest> > > (int)>, std::function<void (unsigned long, std::list<tensorrt_llm::batch_manager::NamedTensor, std::allocator<tensorrt_llm::batch_manager::NamedTensor> > const&, bool, std::string const&)>, std::function<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long> > ()>, std::function<void (std::string const&)>, tensorrt_llm::batch_manager::TrtGptModelOptionalParams const&, std::optional<unsigned long>, std::optional<int>, bool)'
2024/08/08 01:09:00 WARN collect2: error: ld returned 1 exit status
2024/08/08 01:09:00 WARN make[3]: *** [tensorrt_llm/pybind/CMakeFiles/bindings.dir/build.make:251: tensorrt_llm/pybind/bindings.cpython-311-x86_64-linux-gnu.so] Error 1
2024/08/08 01:09:00 WARN make[2]: *** [CMakeFiles/Makefile2:1309: tensorrt_llm/pybind/CMakeFiles/bindings.dir/all] Error 2
2024/08/08 01:09:00 WARN make[1]: *** [CMakeFiles/Makefile2:1316: tensorrt_llm/pybind/CMakeFiles/bindings.dir/rule] Error 2
2024/08/08 01:09:00 WARN make: *** [Makefile:374: bindings] Error 2
2024/08/08 01:09:00 WARN Traceback (most recent call last):
2024/08/08 01:09:00 WARN   File "/home/build/backend/build/../tensorrt_llm/scripts/build_wheel.py", line 352, in <module>
2024/08/08 01:09:00 WARN     main(**vars(args))
2024/08/08 01:09:00 WARN   File "/home/build/backend/build/../tensorrt_llm/scripts/build_wheel.py", line 166, in main
2024/08/08 01:09:00 WARN     build_run(
2024/08/08 01:09:00 WARN   File "/usr/lib/python3.11/subprocess.py", line 571, in run
2024/08/08 01:09:00 WARN     raise CalledProcessError(retcode, process.args,
2024/08/08 01:09:00 WARN subprocess.CalledProcessError: Command 'cmake --build . --config Release --parallel 32 --target tensorrt_llm nvinfer_plugin_tensorrt_llm th_common bindings  executorWorker ' returned non-zero exit status 2.
2024/08/08 01:09:00 ERRO Step failed: exit status 1

[jlewi]

@Shixiaowei02 I saw you commented on a similar issue NVIDIA/TensorRT-LLM#277 so I'm wondering if you have any insight?

[jlewi]

I think I solved this problem. The problem was my version of torch is 2.3.1 which doesn't solve the version constraint in the requirements.txt. So when the wheel installed the requirements.txt torch ended up getting replaced with a version which doesn't have CXX11 ABI Bindings enabled.

To work around this, I commented out the line in the requirements.txt file to install torch so that my version would be used.

So I suspect the problem was that using a torch version without CX11 ABI enabled leads to the aforementioned linking problems even if USE_CXX11_ABI=1 explicitly enables CXX11.

[AnnaYue]

Thanks for your records, really help me out!!

[arturttm]

@jlewi Do you remember with what version of pytorch your build passed? 2.3.1?

[jlewi]

I don't sorry