setup_cuda.py: expected template-name before ‘<’ token

[dcbishop]

I'm trying this without conda since that is giving me another whole set of issues.

Arch Linux. Possibly due to Cuda 12? Torch 2.0.0

[1/1] /opt/cuda/bin/nvcc  -I/usr/lib/python3.10/site-packages/torch/include -I/usr/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.10/site-packages/torch/include/TH -I/usr/lib/python3.10/site-packages/torch/include/THC -I/opt/cuda/include -I/usr/include/python3.10 -c -c /home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/quant_cuda_kernel.cu -o /home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/quant_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1017"' -DTORCH_EXTENSION_NAME=quant_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 -std=c++17
FAILED: /home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/quant_cuda_kernel.o 
/opt/cuda/bin/nvcc  -I/usr/lib/python3.10/site-packages/torch/include -I/usr/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.10/site-packages/torch/include/TH -I/usr/lib/python3.10/site-packages/torch/include/THC -I/opt/cuda/include -I/usr/include/python3.10 -c -c /home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/quant_cuda_kernel.cu -o /home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/quant_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1017"' -DTORCH_EXTENSION_NAME=quant_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 -std=c++17
/usr/include/pybind11/detail/../cast.h: In function ‘typename pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type>::cast_op_type<T> pybind11::detail::cast_op(make_caster<T>&)’:
/usr/include/pybind11/detail/../cast.h:45:120: error: expected template-name before ‘<’ token
   45 |     return caster.operator typename make_caster<T>::template cast_op_type<T>();
      |                                                                                                                        ^
/usr/include/pybind11/detail/../cast.h:45:120: error: expected identifier before ‘<’ token
/usr/include/pybind11/detail/../cast.h:45:123: error: expected primary-expression before ‘>’ token
   45 |     return caster.operator typename make_caster<T>::template cast_op_type<T>();
      |                                                                                                                           ^
/usr/include/pybind11/detail/../cast.h:45:126: error: expected primary-expression before ‘)’ token
   45 |     return caster.operator typename make_caster<T>::template cast_op_type<T>();
      |                                                                                                                              ^
/home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/quant_cuda_kernel.cu: In lambda function:
/home/dcbishop/src/text-generation-webui/repositories/GPTQ-for-LLaMa/quant_cuda_kernel.cu:128:104: warning: ‘T* at::Tensor::data() const [with T = double]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
  128 |   AT_DISPATCH_FLOATING_TYPES(
      |                                                                                                        ^ 
/usr/include/ATen/core/TensorBody.h:244:1: note: declared here
  244 |   T * data() const {
      | ^ ~~

log.txt

[qwopqwop200]

plese use cuda 11.x

[codito]

If anyone is hitting this in archlinux: I was able to compile with cuda11, get it with pacman -U https://archive.archlinux.org/packages/c/cuda/cuda-11.8.0-1-x86_64.pkg.tar.zst. If needed, do a pip install -r requirements.txt to get the *cu11 packages necessary.

[dcbishop]

I managed to get this to work with Cuda 12. The underlying bug is in pybind11 but a small change to the header file fixes it. Everything else seems to be working find now. pybind/pybind11#4606