nvidia-cuda-validator v25.10.0 fails to allocate vector

[davidshen84]

I installed the nvidia-gpu-operator v25.10 on my K3S cluster. Most gpu operator related pods are started successfully, except for the cuda validator, which fails with the following message:

cuda-validation Failed to allocate device vector A (error code no CUDA-capable device is detected)!

cuda-validation [Vector addition of 50000 elements]

stream closed EOF for gpu-operator/nvidia-cuda-validator-r6nsb (cuda-validation)

I downgraded to v25.3.2 and everything worked.

My host system is Gentoo. I installed the nvidia driver and nvidia-container-toolkit directly using the host package manager.

I customised the operator with the following values:

    driver:
      enabled: false
    toolkit:
      enabled: false

    devicePlugin:
      config:
        name: device-plugin-config
        create: true
        default: "time-slicing"
        data:
          time-slicing: |-
            version: v1
            flags:
              migStrategy: none
            sharing:
              timeSlicing:
                renameByDefault: false
                failRequestsGreaterThanOne: true
                resources:
                  - name: nvidia.com/gpu
                    replicas: 4

[rahulait]

Hi @davidshen84 , we would need the must-gather logs to analyze and identify what might have gone wrong.

Once you have a broken cluster with v25.10.0, run the must-gather script and attach the zipped dir where those logs are captured to this issue.

curl -fsSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/heads/main/hack/must-gather.sh | bash