Cuda13 images do not work with Nvidia V100

[derjohn]

Hello,
the "latest" containers für cuda13 dont work with my V100. I run Nvidia GPU operator with 580x drivers and cuda13. This works with llama.cpp (natively installed without localAI), but fails to load any backend with localAI. Without any backend logs as far as I can see.
The cuda12 images however work.

Describe the bug
Make the cuda13 images work with older volta architecture, they are SM70.

To Reproduce
Start a backend on a v100

Expected behavior
It should run

Additional context
Maybe related: (#9156
Furthermore SM70 does not support BF16, just FP16 and FP32. Some models have pytoch variables set to BF16. Maybe that can be auto-detected? But that's a different issue ....

[mudler]

CUDA Toolkit 13.0 removed offline compilation for Maxwell/Pascal/Volta, so nvcc in CUDA 13 cannot produce code for sm_70, the CUDA 13 LocalAI image will never run on V100. The CUDA 12 image is the supported path for V100; that's an NVIDIA upstream decision, not a LocalAI one:

the support for offline compilation for architectures prior to compute capability (CC) 7.5 will be removed in the next major CUDA Toolkit release, CUDA 13.0.

https://developer.nvidia.com/blog/navigating-gpu-architecture-support-a-guide-for-nvidia-cuda-developers/

Are you sure you are not using cuda 12 images with llama.cpp?

[derjohn]

Honstly: I don't know. I used "ghcr.io/ggml-org/llama.cpp:server-cuda" and supposed, it the current "13" version. What's also odd: They driver for V100/Volta worked well in "cuda13" version. Strange decision from Nvidia. Same thing is true for the open-source drivers. They also support only newer generations.

I can close this ticket or make it a feature wish: Can you log out something on startup like "cuda: Wooooong feature set detected. Consider different cuda version" ?

[derjohn]

@mudler Another thing that came to my mind: I desperately tried to make diffusers running (with z-image-diffusers), but it didnt work. Might it also be the reason that models are built to need > sm70 ? Or are only backends affected by that requirement?

[localai-bot]

CUDA Toolkit 13 dropped offline compilation support for compute capability sm_70 (Volta / V100) - this is an upstream NVIDIA change, not a LocalAI bug. Use the CUDA-12 images on V100 hardware. Closing as upstream/wontfix; we may add a startup warning when a pre-7.5 GPU is used with a CUDA-13 image.