[Bug]: nvidia-container-toolkit exit 255 CrashLoopBackOff after ~18 minutes on AKS / Ubuntu 24.04 / containerd 2.3.2 with GPU Operator v25.10.1

[alecscodehub]

Environment

Component	Version
GPU Operator	v25.10.1
Container Toolkit	v1.18.1
Kubernetes	AKS (Azure Kubernetes Service)
Node OS	Ubuntu 24.04 (AKSUbuntu-2404gen2containerd-202606.19.0)
containerd	2.3.2
GPU	NVIDIA A100 (Standard_NC24ads_A100_v4)
Driver model	Pre-installed (node label: nvidia.com/gpu.deploy.driver=pre-installed)
Driver version	580.105.08-ubuntu24.04
CDI mode	Enabled (cdi.enabled: true)

---

Describe the bug

The nvidia-container-toolkit-daemonset pod runs for approximately 18 minutes, then exits with exit code 255 and enters CrashLoopBackOff. The pod has restarted 149+ times since node provisioning. The node has been NotReady since first provisioning.

The crash does not happen immediately — the toolkit appears to successfully connect to containerd, write the runtime config, and signal containerd to reload. It then dies approximately 18 minutes later. This timing is consistent with the toolkit sending a SIGHUP/reload to containerd, containerd reloading and sending SIGTERM back to the toolkit pod, and the toolkit not surviving the signal on containerd 2.3.2.

The nvidia-operator-validator and all downstream daemonsets (nvidia-device-plugin, nvidia-dcgm-exporter, nvidia-gfd) remain in Init:0/1 waiting for toolkit readiness.

Other GPU nodes in the same cluster running Ubuntu 22.04 / containerd 1.7.33 are fully Ready with no issues.

---

Steps to reproduce

1. Provision an AKS node pool with Ubuntu 24.04 (node image: AKSUbuntu-2404gen2containerd-202606.19.0, containerd 2.3.2)
2. Install GPU Operator v25.10.1 with cdi.enabled: true, driver.enabled: true (pre-installed driver model), operator.defaultRuntime: docker
3. Observe nvidia-container-toolkit-daemonset pod on the Ubuntu 24.04 node

---

Observed behavior

$ kubectl get pods -n gpu-operator -o wide | grep toolkit
nvidia-container-toolkit-daemonset-xxxxx   0/1   CrashLoopBackOff   149   <node-name>

$ kubectl describe pod nvidia-container-toolkit-daemonset-xxxxx -n gpu-operator
...
Last State: Terminated
  Reason: Error
  Exit Code: 255
  Started: <timestamp>
  Finished: <timestamp>  # ~18 minutes after Started

$ kubectl get node <node-name>
NAME          STATUS     ROLES   AGE
<node-name>   NotReady   <none>  4d

containerd reports Unknown version when queried from the toolkit pod context.

---

Expected behavior

The toolkit pod should remain in Running state (waiting-for-signal loop) after writing the containerd runtime config, matching behavior on Ubuntu 22.04 / containerd 1.7.x nodes in the same cluster.

---

Current ClusterPolicy (relevant excerpt)

spec:
  cdi:
    enabled: true
    default: false
  toolkit:
    enabled: true
    version: v1.18.1
    # no env overrides — default paths used
  operator:
    defaultRuntime: docker
  driver:
    enabled: true
    useNvidiaDriverCRD: false
    kernelModuleType: auto

No CONTAINERD_SOCKET or CONTAINERD_CONFIG env var overrides are set in the toolkit spec. The daemonset uses volume mounts: /runtime/sock-dir/ → /run/containerd and /runtime/config-dir/ → /etc/containerd on the host.

---

Root cause hypothesis

containerd 2.3.2 is outside the validated matrix for GPU Operator v25.10.1 (validated up to containerd 2.2.x). The ~18-minute runtime before crash is consistent with the toolkit writing the config, signaling containerd to reload via SIGHUP, and containerd 2.3.x sending SIGTERM back to the toolkit pod — a known signal-handling regression class documented in issues #456, #991, and #1651.

The NRI Plugin introduced in v26.3.x would eliminate this entire failure class by not modifying containerd's config.toml at all.

---

Questions

1. Is containerd 2.3.2 on AKS Ubuntu 24.04 a known-broken combination for v25.10.1?
2. Is upgrading to GPU Operator v26.3.x with NRI Plugin enabled the recommended fix for AKS + containerd 2.3.x?
3. Is there a supported workaround for v25.10.1 (e.g. disabling CDI, switching to legacy mode) that avoids the containerd reload signal issue?

---

Related issues

• #456 — toolkit CrashLoopBackOff after Completed state
• #991 — containerd restart from toolkit causes daemonsets to get stuck
• #1651 — nvidia runtime config removed after containerd restart (containerd 2.0.5)
• #1876 — v25.10.0 GPU pod not starting on containerd 2.1.4 + Ubuntu 24.04

[tariq1890]

If your node is stuck in NotReady during a container-toolkit deploy, there is a high likelihood that the containerd service wasn't restarted successfully. I recommend checking the journalctl -u containerd command output for any errors in containerd.

I also see that you mention containerd v2.3.2, so you MIGHT be running into containerd config version mismatch error. If this is the case, this is explained by some of the inherent limitations in how containerd manages its config. This is a known issue in containerd. To workaround this you have two options:

1. Set the toolkit env var RUNTIME_CONFIG_SOURCE .

spec:
  toolkit:
    enabled: true
    version: v1.18.1
    env:
    - name: RUNTIME_CONFIG_SOURCE
      value: "file=</path/to/containerd/config/file>"

2. Enable NRI Plugin mode. Worth keeping in mind that this feature is in Technical Preview (as of the time of posting this comment)

[alecscodehub]

Hello @tariq1890 , i have tested and issue is solved.

For anyone there who updates AKS nodes from Ubuntu 22.x to Ubuntu 24.x if face this same issue consider this:

spec:
  toolkit:
    enabled: true
    version: v1.18.1
    env:
    - name: RUNTIME_CONFIG_SOURCE
      value: file

Thank you very much!