fix: initialize NVIDIA driver before CDI toolkit - #175
Merged
ganeshkumarashok merged 2 commits intoAug 5, 2026
Conversation
Stage the driver userspace libraries and validate nvidia-smi before installing NVIDIA Container Toolkit, whose package post-install immediately starts nvidia-cdi-refresh. Regenerate CDI after runtime and device-link configuration, and cover the lifecycle ordering with Bats. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b3becde0-b693-48d3-b13b-801b0ebee848
The toolkit post-install and path unit can consume the oneshot service's start budget before the final authoritative restart. Clear that transient counter while continuing to propagate a real restart failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b3becde0-b693-48d3-b13b-801b0ebee848
ganeshkumarashok
marked this pull request as ready for review
August 5, 2026 22:02
sulixu
approved these changes
Aug 5, 2026
sulixu
left a comment
sulixu
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, a better fix than restart CDI after driver install in AB
ganeshkumarashok
merged commit Aug 5, 2026
337cb15
into
gpu-prebake-test-harness
8 of 14 checks passed
ganeshkumarashok
added a commit
that referenced
this pull request
Aug 5, 2026
* test(install): add shellcheck + bats harness for prebake logic The prebake mode dispatch (build-only / install-skip-build / install), the dkms marker write/parse/validate, and the opportunistic fast-path fallback previously had no automated guard -- CI only ran `docker buildx build` (xuexu6666 review on #162). - make install.sh sourceable for unit tests: move set-flags + config/helper sources + EXIT trap into main(), guard main() behind BASH_SOURCE, extract purge_gpu_cache, parameterize config/modules paths. Behavior-preserving (execution still runs main()). - test/install.bats: 15 tests (marker write/parse/validate, fast-path fallback, target-kernel selection, all three dispatch modes). - ci.yaml: new lint-and-test job (shellcheck install.sh+entrypoint.sh, bats test/). Local: shellcheck clean, bats 15/15 pass. Signed-off-by: Ganeshkumar Ashokavardhanan <aganeshkumar@microsoft.com> * fix: initialize NVIDIA driver before CDI toolkit (#175) * fix: initialize NVIDIA driver before CDI toolkit Stage the driver userspace libraries and validate nvidia-smi before installing NVIDIA Container Toolkit, whose package post-install immediately starts nvidia-cdi-refresh. Regenerate CDI after runtime and device-link configuration, and cover the lifecycle ordering with Bats. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b3becde0-b693-48d3-b13b-801b0ebee848 * fix: clear CDI start limit before regeneration The toolkit post-install and path unit can consume the oneshot service's start budget before the final authoritative restart. Clear that transient counter while continuing to propagate a real restart failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b3becde0-b693-48d3-b13b-801b0ebee848 --------- Copilot-Session: b3becde0-b693-48d3-b13b-801b0ebee848 --------- Signed-off-by: Ganeshkumar Ashokavardhanan <aganeshkumar@microsoft.com> Copilot-Session: b3becde0-b693-48d3-b13b-801b0ebee848
ganeshkumarashok
added a commit
to Azure/AgentBaker
that referenced
this pull request
Aug 6, 2026
…260805223419 Rebase onto main and retarget at the first aks-gpu build that contains Azure/aks-gpu#175 (driver fully initialized before the NVIDIA Container Toolkit installs and starts nvidia-cdi-refresh). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6cae2ec3-2398-461a-b5d1-bcb1cb95770e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the NVIDIA CDI startup race introduced by VHD-prebaked kernel modules by making the driver fully usable before NVIDIA Container Toolkit is installed.
ldconfig, and requirenvidia-smibefore toolkit package post-install starts CDI/dev/charlinks before a final synchronous CDI regenerationRoot cause
Toolkit 1.19.1 starts
nvidia-cdi-refreshduring package installation. With a prebaked module, the unit condition passes beforelibnvidia-ml.sois active in the host library path. The service retries, reaches its start limit, and remains failed after the driver later becomes healthy.This change removes that race instead of repairing it afterward. AgentBaker PR #8736 remains the rollout backstop and final CDI-spec validator until this image change has soaked.
Validation
shellcheck -S warning install.sh entrypoint.shbats test/install.bats— 18/18 passedDependency
Stacked on #166, which adds the sourceable install-script test harness. Retarget to
mainafter #166 merges.