does device plugin support GB10 (NVIDIA DGX Spark)

[sceneryback]

I deployed nvidia device plugin on a new NVIDIA DGX Spark machine, the output of nvidia-smi is

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   40C    P8              4W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|

And device plugin can not be started with error logs:

E1029 10:26:46.968061       1 main.go:173] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: error building device map: error building device map from config.resources: error building GPU device map: error visiting device: error building Device: error getting device memory: Not Supported

It seems that it can not get the memory of GB10, same to nvidia-smi.
And I notice the driver 580.95.05 is quite new:

so do we support GB10 now?

[shahizat]

Hello,

I was running the vLLM production stack(https://github.com/vllm-project/production-stack) on an NVIDIA DGX Spark, but I observed that some metrics are not unavailable. They are currently displaying as 'No data'.

See here: https://forums.developer.nvidia.com/t/local-k8s-cluster-with-minikube-on-nvidia-dgx-spark/356043/3

kubectl get pods -A
NAMESPACE     NAME                                                     READY   STATUS             RESTARTS          AGE
default       spark-vllm-deployment-router-6f977d446d-d2jcx            1/1     Running            0                 54m
default       spark-vllm-qwen-3-4b-deployment-vllm-7499f9c8b8-fnkjn    1/1     Running            0                 54m
default       spark-vllm-qwen-3-4b-deployment-vllm-7499f9c8b8-q5rlq    1/1     Running            0                 54m
kube-system   coredns-66bc5c9577-k5wjd                                 1/1     Running            2 (178m ago)      40h
kube-system   etcd-minikube                                            1/1     Running            2 (178m ago)      40h
kube-system   kube-apiserver-minikube                                  1/1     Running            2 (55m ago)       40h
kube-system   kube-controller-manager-minikube                         1/1     Running            2 (178m ago)      40h
kube-system   kube-proxy-dkmbw                                         1/1     Running            2 (178m ago)      40h
kube-system   kube-scheduler-minikube                                  1/1     Running            2 (178m ago)      40h
kube-system   nvidia-device-plugin-daemonset-9kt9b                     0/1     CrashLoopBackOff   390 (3m19s ago)   40h
kube-system   storage-provisioner                                      1/1     Running            4 (54m ago)       40h
monitoring    alertmanager-kube-prom-stack-kube-prome-alertmanager-0   2/2     Running            0                 27m
monitoring    kube-prom-stack-grafana-bf764ffc8-sjslf                  3/3     Running            0                 27m
monitoring    kube-prom-stack-kube-prome-operator-86cd65d9ff-wxw4r     1/1     Running            0                 27m
monitoring    kube-prom-stack-kube-state-metrics-6fb6596785-qq7mt      1/1     Running            0                 27m
monitoring    kube-prom-stack-prometheus-node-exporter-cfqpd           1/1     Running            0                 27m
monitoring    prometheus-adapter-7598955f5d-2mx2w                      1/1     Running            0                 27m
monitoring    prometheus-kube-prom-stack-kube-prome-prometheus-0       2/2     Running            0                 27m

cc @elezar

[BostjanBozic]

Hello,

just my 2 cents here, since I tried it with newer driver version.

nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   80C    P0             45W /  N/A  | Not Supported          |     96%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2906      G   /usr/lib/xorg/Xorg                       33MiB |
|    0   N/A  N/A          284858      C   /usr/bin/python3                       1948MiB |
|    0   N/A  N/A         2423663      C   python                                23911MiB |
+-----------------------------------------------------------------------------------------+

As you see the memory usage of the device is still show as "Not Supported". I guess because it is shared memory, the plugin can not build the device map.

Non the less, plugin itself is working, but within MPS DaemonSet following error occurs:

[2026-03-31 18:15:36.288 Control 2912480] Starting new server 2912480 for user 1000
[2026-03-31 18:15:36.291 Control 2837079] Accepting connection...
[2026-03-31 18:15:36.311 Control 2837079] Server encountered a fatal exception. Shutting down
[2026-03-31 18:15:36.316 Control 2837079] Server 2912480 exited with status 1

I was running workload with UID 1000, because running as root, MPS plugin also has issue accessing /etc/passwd since it is empty; which causes plugin to crash:

[2026-03-31 17:56:25.277 Control 2903914] getpwuid() failed for 0
[2026-03-31 17:56:25.399 Control 2837079] Server 2903914 exited with status 0

I guess until memory usage is not available, only option to use for parallel usage is time slicing plugin.

[guptaNswati]

Pls open an issue for MPS separately as it should not be related to GB10 fix. Closing for GB10 as the fix is alreday there.