fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install - #8918
Closed
ganeshkumarashok wants to merge 3 commits into
Closed
fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install#8918ganeshkumarashok wants to merge 3 commits into
ganeshkumarashok wants to merge 3 commits into
Conversation
… install The shared Ubuntu VHD prebakes only the cuda(-lts) driver plus a DKMS marker. A GRID / converged (A10, NVv5) node then installs the grid driver on top, and the stale prebaked cuda module + its /usr/bin/lib64 userspace libs collide with the grid driver, so nvidia-smi fails with "Failed to initialize NVML: Driver/library version mismatch". The existing cleanUpPrebakedGPUDriver only runs on nodes that do NOT install a managed driver (GPU_NODE != true OR skip), so managed GRID nodes keep the stale prebake. This adds cleanUpGridNodeCudaPrebake, run from ensureGPUDrivers before the configGPUDrivers/validateGPUDrivers dispatch (so both paths are covered), which tears down the prebake when this node installs grid and the marker is not grid (a cuda or legacy kind-less marker). Pure driver-kind mismatch, so no version comparison is needed. CUDA nodes are intentionally untouched. Scoped to the GRID mismatch only; the NAP cuda same-kind/different-version case is addressed separately by aligning the driver version NAP requests with the baked image.
ganeshkumarashok
requested review from
AbelHu,
Devinwong,
SriHarsha001,
awesomenix,
calvin197,
cameronmeissner,
djsly,
karenychen,
lilypan26,
mxj220,
pdamianov-dev,
phealy,
r2k1,
runzhen,
sulixu,
surajssd,
timmy-wright,
xuexu6666 and
zachary-bailey
as code owners
July 13, 2026 21:05
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the Linux CSE GPU-driver provisioning flow to proactively remove a mismatched CUDA(-LTS) prebaked driver on Ubuntu VHDs when the node is going to install a GRID driver, preventing NVML “driver/library version mismatch” failures on GRID/A10 (NVv5) nodes that boot shared prebaked Ubuntu images.
Changes:
- Added
cleanUpGridNodeCudaPrebaketo tear down a CUDA-kind prebake marker on GRID-kind nodes before driver install/validation dispatch. - Invoked the new teardown step early in
ensureGPUDrivers(Ubuntu-only) to cover both config and validation paths. - Added ShellSpec coverage for the teardown decision logic.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| parts/linux/cloud-init/artifacts/cse_config.sh | Adds and wires cleanUpGridNodeCudaPrebake into ensureGPUDrivers to remove mismatched CUDA prebake on GRID nodes. |
| spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh | Adds ShellSpec tests validating teardown vs no-op behavior across marker/node-kind combinations. |
sulixu
approved these changes
Jul 13, 2026
Comment on lines
+39
to
+43
| - path: {{GetCSEConfigScriptFilepath}} | ||
| permissions: "0744" | ||
| encoding: gzip | ||
| owner: root | ||
| content: !!binary | |
4 tasks
Addresses Copilot review on #8919: cover the OS != UBUNTU_OS_NAME early return so a regression that removes or inverts the Ubuntu gate is caught.
Contributor
Changes cached containers or packages on windows VHDsPlease get a Windows SIG member to approve. The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.
diff --git a/vhd_files/2022-containerd-gen2.txt b/vhd_files/2022-containerd-gen2.txt
index e187816..db10c9e 100644
--- a/vhd_files/2022-containerd-gen2.txt
+++ b/vhd_files/2022-containerd-gen2.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2022-containerd.txt b/vhd_files/2022-containerd.txt
index 760db3c..94de353 100644
--- a/vhd_files/2022-containerd.txt
+++ b/vhd_files/2022-containerd.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index a901175..4f3c531 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index 4d7aa19..d0ea692 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 1441011..ab44d8b 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1 |
Comment on lines
+38
to
+42
| # ---- hotfix: auto-generated by hotfix-generate GH Action ---- | ||
| - path: {{GetCSEConfigScriptFilepath}} | ||
| permissions: "0744" | ||
| encoding: gzip | ||
| owner: root |
2 tasks
Contributor
Author
Contributor
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172065316 Detective summary: VM-side validation failed before product checks because the test VM could not fetch Likely cause / signature: Known PR merge-ref availability race; wiki signature Confidence: High. Recommended owner/action: Node Lifecycle pipeline infra owner should continue repair item #38759467; no PR author action unless rerun failures show product-specific validation errors. Strongest alternative: PR source branch changed during the run; less likely as a PR code issue because the failing operation is a generic fetch of a transient GitHub PR merge ref across matrix jobs. Evidence: ADO timeline shows many Test/Scan jobs failed with exit code 2; failed task log includes the missing remote ref and git-clone error. Wiki signature: pr-merge-ref-race |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes
nvidia-smifailing with "Failed to initialize NVML: Driver/library version mismatch" on agentpool GRID / converged A10 (NVv5) nodes and NAP GRID nodes booting the CUDA-prebaked shared Ubuntu VHD.Root cause: the shared Ubuntu VHD prebakes only the
cuda(-lts)driver + a DKMS marker (#8786/#8803). The existingcleanUpPrebakedGPUDriveronly runs on nodes that do not install a managed driver (GPU_NODE != trueOR skip). A GRID node installs thegriddriver on top of the stale prebakedcudamodule +/usr/bin/lib64libs, and the two collide → NVML mismatch.Fix
Add
cleanUpGridNodeCudaPrebake, invoked fromensureGPUDriversbefore theconfigGPUDrivers/validateGPUDriversdispatch (so both install paths are covered). It tears down the prebake when:grid/grid-v20), andcudamarker, or a legacy marker with nodriver_kind=line (the only kind the VHD bakes today).This is a pure driver-kind mismatch, so no version comparison is needed. Reuses the existing
cleanUpPrebakedGPUDriverfor removal (which also clears/usr/bin/lib64and the DKMS registration).Scope
GRID mismatch only. CUDA nodes are intentionally untouched — the NAP cuda same-kind / different-version case (
580.126.09vs baked580.159.04) is a separate issue, addressed by aligning the driver version NAP requests with the baked image rather than by teardown.Why this is safe on
official/v20260702Consume/skip-build is not in this branch, so a kept prebake buys no provisioning benefit today; the only effect of the guard is removing a driver that would otherwise break a GRID node. On a matching (grid) or absent marker, or on a CUDA node, it is a no-op.
Test plan
cse_config_spec.sh(cleanUpGridNodeCudaPrebake)🤖 Generated with Claude Code