Cannot compile the Meilisearch binary with cuda enabled

[illutany10]

Describe the bug
I cannot compile the Meilisearch binary with cuda enabled

To Reproduce
Steps to reproduce the behavior:

command:
cargo build --release --features cuda

error:

error: failed to run custom build command for `candle-kernels v0.4.1`

Caused by:
  process didn't exit successfully: `/home/khl/search_engine/meilisearch/target/release/build/candle-kernels-e9399fd1b457aa7f/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=build.rs
  cargo:info=["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]
  cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP
  cargo:rustc-env=CUDA_COMPUTE_CAP=75
  cargo:info=Builder { cuda_root: Some("/usr"), kernel_paths: ["src/affine.cu", "src/binary.cu", "src/cast.cu", "src/conv.cu", "src/fill.cu", "src/indexing.cu", "src/quantized.cu", "src/reduce.cu", "src/ternary.cu", "src/unary.cu"], watch: [], include_paths: ["src/binary_op_macros.cuh", "src/compatibility.cuh", "src/cuda_utils.cuh"], compute_cap: Some(75), out_dir: "/home/khl/search_engine/meilisearch/target/release/build/candle-kernels-4a081886eb0ed0a5/out", extra_args: [] }
  cargo:rustc-env=CUDA_INCLUDE_DIR=/usr/include
  cargo:rerun-if-changed=src/binary_op_macros.cuh
  cargo:rerun-if-changed=src/compatibility.cuh
  cargo:rerun-if-changed=src/cuda_utils.cuh
  cargo:rerun-if-env-changed=NVCC_CCBIN
  cargo:rerun-if-changed=src/cast.cu
  cargo:rerun-if-changed=src/reduce.cu
  cargo:rerun-if-changed=src/conv.cu
  cargo:rerun-if-changed=src/quantized.cu
  cargo:rerun-if-changed=src/unary.cu
  cargo:rerun-if-changed=src/ternary.cu
  cargo:rerun-if-changed=src/indexing.cu
  cargo:rerun-if-changed=src/fill.cu
  cargo:rerun-if-changed=src/affine.cu
  cargo:rerun-if-changed=src/binary.cu

  --- stderr
  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  6 errors detected in the compilation of "src/cast.cu".
  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  6 errors detected in the compilation of "src/conv.cu".
  src/compatibility.cuh(11): error: identifier "__hmax" is undefined

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(11): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: identifier "__hmin" is undefined

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  src/compatibility.cuh(14): error: ambiguous "?" operation: second operand of type "__half" can be converted to third operand type "<error-type>", and vice versa

  6 errors detected in the compilation of "src/affine.cu".
  6 errors detected in the compilation of "src/unary.cu".
  6 errors detected in the compilation of "src/reduce.cu".
  6 errors detected in the compilation of "src/ternary.cu".
  6 errors detected in the compilation of "src/indexing.cu".
  6 errors detected in the compilation of "src/binary.cu".
  thread 'main' panicked at /home/khl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen_cuda-0.1.5/src/lib.rs:391:13:
  nvcc error while compiling "src/affine.cu":

  # CLI "nvcc" "--gpu-architecture=sm_75" "--ptx" "--default-stream" "per-thread" "--output-directory" "/home/khl/search_engine/meilisearch/target/release/build/candle-kernels-4a081886eb0ed0a5/out" "-Isrc" "-I/usr/include" "src/affine.cu" 

  # stdout


  # stderr

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Meilisearch version:
1.8

Additional context

OS: Ubuntu 22.04
Cuda version: 11.5

[curquiza]

Hello @illutany10

Really weird issue. We have multiple users like you compiling, but without any issue.

Maybe community can help on it?

In the meantime, if you want better support, you can go on our discord to reach the opensource community faster.

[dureuill]

Hello,

It is as @curquiza said, but I would like to add that you might have a better chance with CUDA >=12, I have seen issues on our dependency candle that were fixed by upgrading.

[curquiza]

I close this issue due to a lack of answers and more information. Nothing is definitive of course, feel free to ask to re-open if needed 😊