A10 vs A100 Ubuntu AKS Images Nvidia Driver and Cuda Version differences

[lynkz-matt-psaltis]

I'm trying to understand my path forward with our CUDA workloads for both A10 machines (e.g. Standard_NV6ads_A10_v5) which are currently pinned to NVIDIA-SMI 510.73.08 Driver Version: 510.73.08 CUDA Version: 11.6 where as A100 machines are at CUDA 12.2?

From what I understand, the 510 driver is no longer supported by Nvidia and is not compatible with the CUDA 12 Compatibility Package (See: https://docs.nvidia.com/deploy/cuda-compatibility/ Table 3)

When using nvidia-smi and the latest AKS Ubuntu 22.04 images, my understanding is the Azure team are preinstalling nvidia drivers for the GPU enabled workloads. What's the right path here? Should the team be upgrading the preinstalled driver version to an Nvidia supported version? Should I cut my own images with newer drivers?

[lynkz-matt-psaltis]

Also found: #3364

[caffeinism]

Ridiculously low driver version. Any update?

[lynkz-matt-psaltis]

Looks like the latest image has been updated to v535.54.03 - CUDA 12.2

VM image: AKSUbuntu-2204gen2containerd-202312.06.0

Thanks team!