Using in clusters which contains both GPU nodes and non-GPU nodes

[idealhack]

When using daemon sets in this kind of cluster, non-GPU nodes will complains

Error: failed to start container "nvidia-device-plugin-ctr": Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"process_linux.go:351: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=ALL --utility --compute --pid=16424 /var/lib/docker/overlay/a86473af4c52afb44dfdfdcc817edb45316d520cccfb086d87cc227314d09015/merged]\\\\nnvidia-container-cli: initialization error: load library failed: libcuda.so.1: cannot open shared object file: no such file or directory\\\\n\\\"\""

It's straightforward to use taints (which could be documented), but how about also done it in this plugin (i.e. better error handling)?

[flx42]

Sure, it's a problem. But looks like you also installed nvidia-container-runtime on this node?

[flx42]

@RenaudWasTaken when deploying to a node with no GPU, we should wait indefinitely, right? We already have a case for this, but it assumes NVML is present and working:
https://github.com/NVIDIA/k8s-device-plugin/blob/master/main.go#L46-L49

[RenaudWasTaken]

@RenaudWasTaken when deploying to a node with no GPU, we should wait indefinitely, right?

The way I expected device plugins to work is to stop when they detect that no devices are available on the node.
I also expect the restart policy to be OnFailure.

[flx42]

Fixed by 9b54e91

[idealhack]

@RenaudWasTaken “A Pod Template in a DaemonSet must have a RestartPolicy equal to Always, or be unspecified, which defaults to Always.” https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#pod-template

As for now, a node without GPU will be in a crash loop.

[RenaudWasTaken]

@RenaudWasTaken “A Pod Template in a DaemonSet must have a RestartPolicy equal to Always, or be unspecified, which defaults to Always.” https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#pod-template

Looks like we'll have to update the k8s docs.

As for now, a node without GPU will be in a crash loop.

I just pushed a fix for that :)

[idealhack]

@RenaudWasTaken I‘m getting The DaemonSet "nvidia-device-plugin-daemonset" is invalid: spec.template.spec.restartPolicy: Unsupported value: "OnFailure": supported values: Always when applying the manifest on k8s 1.8.1, which is right according to the docs. Also I failed to find any clues about this have changed in newer version.

[RenaudWasTaken]

@idealhack thanks for noticing this mistake.
Looks like you have to manually label your nodes if you don't want the plugin to be running on every nodes.

[idealhack]

So crashing is actually the right behavior for GPU plugin on non-GPU node, right?

Speaking of docs, how about we add a note about using taints to handle this kind of clusters in README, like https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#example-use-cases