fix(gpu): autoload ib_umad so fabricmanager precheck passes on NVL5+ (HGX B200)#700
Merged
Merged
Conversation
…(HGX B200)
nvidia-fabricmanager 570+ ships a systemd unit whose ExecStartPre invokes
`nvidia-fabricmanager-start.sh --mode precheck`. On HGX B200 / GB200
hardware the wrapper takes the "Detected NVL5+ system" branch, which
requires the ib_umad kernel module to be loaded before nv-fabricmanager
can start -- the daemon opens /dev/infiniband/umad* character devices to
send MADs to the NVSwitch fabric (NVSwitch reuses the InfiniBand
management-datagram shape for topology discovery + NVLink route
programming).
Kairos edge images ship the module in the kernel-modules-extra set but
don't auto-load it. Result: on first boot of a Palette Edge HGX B200
appliance, the service fails at ExecStartPre with:
Detected NVL5+ system
Kernel module "ib_umad" has not been loaded, fabric manager cannot be started
Please run "modprobe ib_umad" before starting fabric manager
That cascades: without fabricmanager the NVSwitch fabric never initializes;
CUDA runtime in any container fails cudaMalloc with CUDA_ERROR_NOT_INITIALIZED
("system not yet initialized"); nvidia-cuda-validator crashes at Init:Error;
nvidia-operator-validator hangs at Init:2/4.
Fix: write /etc/modules-load.d/nvidia-fabricmanager.conf listing ib_umad
so systemd's kmod-static-nodes / systemd-modules-load loads it at boot
ahead of nvidia-fabricmanager.service.
The file is scoped to the same conditional that installs the fabricmanager
package (NVIDIA_INSTALL_FABRICMANAGER=true), so hardware where the operator
opts out of fabricmanager doesn't get the module load either.
On non-NVSwitch hosts the module costs ~10 KiB of RSS and has no other
effect; fabricmanager still exits "No NvSwitch found" and the unit stays
inactive as before.
Verified live on 8x HGX B200 running driver 580.159.03: with the drop-in
in place, `systemctl start nvidia-fabricmanager` transitions from failed
to active in ~3s, the NVSwitch fabric initializes, and downstream
nvidia-cuda-validator exits 0 on the next GPU-operator reconcile.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Full stack audit after applying this fix —
|
| Signal | State |
|---|---|
| Kernel modules loaded | nvidia, nvidia_uvm, nvidia_modeset, nvidia_drm, ib_core, ib_umad, ib_uverbs, ib_cm, mlx5_core, mlx5_ib, mlxfw, rdma_cm, nvme_rdma, nvmet, nvmet_rdma |
/dev/infiniband/* device nodes |
✅ umad0, issm10-13 created by ib_umad |
nvidia-fabricmanager.service |
✅ active (running); NVSwitch fabric initialized |
| GPU operator validator chain | ✅ driver / toolkit / cuda / plugin all validated (nvidia-operator-validator Ready 1/1) |
| Node capacity | ✅ nvidia.com/gpu: 8 (all 8 B200s discoverable) |
| Node labels | ✅ Compute capability 10.0 (Blackwell sm_100), driver 580.159.03, CUDA runtime 13.0 |
| Failed systemd services | Only remote-shell.service (Stylus, unrelated to GPU) |
| dmesg errors | Only cosmetic (nvidia MOK-signing taint, Renesas xHCI ROM fallback) — nothing GPU-blocking |
⚠️ One additional module worth flagging (out-of-scope for this PR)
nvidia_peermem is BUILT via DKMS at /lib/modules/6.17.0-35-generic/updates/dkms/nvidia-peermem.ko.zst but not auto-loaded at boot.
- What it does: exposes GPU memory to RDMA devices → enables GPUDirect RDMA (multi-node NCCL, GPUDirect Storage, RDMA-based frameworks touching GPU memory).
- What it doesn't affect: single-node CUDA workloads (Ollama, vLLM, SGLang, llamacpp) — those go through
nvidia_uvm/nvidia_modesetwhich are already loaded. - When it matters: if this appliance ever hosts distributed inference/training across boxes, or Kubevirt VMs using GPUDirect Storage.
- Cost of loading: near-zero on non-RDMA hardware (module just idles).
Section 10 of install-nvidia-drivers.sh already writes /etc/modules-load.d/nvidia.conf with the three core modules (nvidia, nvidia_uvm, nvidia_modeset) — deliberately omitting nvidia_drm and nvidia_peermem per the inline comment. Reversing that omission would need a deliberate call, so I'm not rolling it into this PR.
Suggested follow-up (separate PR)
Auto-load nvidia_peermem when NVIDIA_INSTALL_FABRICMANAGER=true — HGX/DGX/GB200 systems almost universally have Mellanox HCAs and benefit from GPUDirect RDMA. Same modules-load.d pattern as this PR. Different failure class (silent slow-path fallback vs loud precheck-fail), so worth its own reviewer surface.
If you'd like it in this PR instead, happy to append a commit — just say the word.
Verification transcript (live HGX B200, 8× B200 / driver 580.159.03)
# Before the fix (this session)
$ sudo systemctl is-active nvidia-fabricmanager
failed
$ kubectl -n gpu-operator get pods | grep validator
nvidia-cuda-validator-9xdvx 0/1 Init:Error 1 (31s ago) 74s
nvidia-operator-validator-4j2p4 0/1 Init:2/4 0 2m59s
# After: modprobe ib_umad + systemctl start nvidia-fabricmanager
$ sudo systemctl is-active nvidia-fabricmanager
active
$ kubectl -n gpu-operator get pods | grep validator
nvidia-cuda-validator-mscfd 0/1 Completed 0 3m5s
nvidia-operator-validator-4j2p4 1/1 Running 0 9m29s
abhinavnagaraj
added a commit
that referenced
this pull request
Jul 22, 2026
* feat(gpu): pre-install NVIDIA driver in Ubuntu base image for air-gapped GPU Operator
Add scripts/install-nvidia-drivers.sh to bake the NVIDIA data-center driver and
its DKMS kernel modules into the Ubuntu base image, so GPU nodes can run the
NVIDIA GPU Operator with driver.enabled=false in fully air-gapped environments
(no host-side network, no operator driver container).
The script derives the target kernel from /lib/modules (not the builder's
uname -r), installs ABI-exact headers via install-kernel-headers.sh, forces DKMS
build/install/depmod against that kernel, and verifies nvidia*.ko landed in the
image kernel tree. It also blacklists nouveau, autoloads the nvidia stack,
enables nvidia-persistenced, and rebuilds the initrd.
Wire it into the base-image target (Ubuntu/amd64, after the kernel is finalized)
behind INSTALL_NVIDIA_GPU_DRIVERS, with tunable branch/type/repo/fabricmanager/
toolkit args. Document in .arg.template and docs/nvidia-gpu-airgapped.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(gpu): clarify relationship to NVIDIA's Local Package Repository section
Explain that linux-headers/linux-image/linux-modules listed by NVIDIA belong to
the driver-container + local-apt-mirror strategy (driver.enabled=true), whereas
this integration uses the pre-installed-driver model (driver.enabled=false, no
mirror) and satisfies those three requirements at build time in the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(gpu): document Ubuntu 24.04 support and driver-branch versioning semantics
Clarify that the installer is version-agnostic (derives CUDA repo tag and kernel
codename from the image's os-release), so it works on both Ubuntu 22.04 and
24.04. Note that NVIDIA_DRIVER_BRANCH selects a driver branch (latest patch
within it), not an exact point release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(gpu): add AMD Instinct driver pre-install for air-gapped AMD GPU Operator
Add scripts/install-amdgpu-drivers.sh, the AMD counterpart to the NVIDIA
installer. Bakes the amdgpu-dkms kernel driver into the Ubuntu base image so AMD
Instinct nodes can run the AMD GPU Operator with driver.enable=false in fully
air-gapped environments (no host-side network, no operator-built driver).
Reuses the same build-time approach: derive the image kernel from /lib/modules
(not the builder's uname -r), install ABI-exact headers via
install-kernel-headers.sh plus linux-modules-extra, register the AMD driver repo
via the version-matched amdgpu-install package (auto-discovered from
repo.radeon.com/amdgpu-install/<version>/), install amdgpu-dkms, force DKMS
build/install/depmod against the image kernel, and verify amdgpu.ko landed.
No nouveau-style blacklist is needed since amdgpu-dkms replaces the in-tree
amdgpu module.
Wire it into the base-image target behind INSTALL_AMD_GPU_DRIVERS with an
AMDGPU_ROCM_VERSION arg (default 7.2.4), alongside a build-time guard making the
NVIDIA and AMD pre-install paths mutually exclusive. Document in .arg.template,
docs/amd-gpu-airgapped.md, and cross-link from the NVIDIA doc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(gpu): document how to build air-gapped content when driver is pre-installed
Add a "Building the air-gapped content" section to both the NVIDIA and AMD
guides. Clarify that pre-installing the driver only lets you drop the driver
image (NVIDIA) or the KMM/kaniko/ubuntu build images (AMD) -- the rest of the
operator still ships as container images that must be mirrored into the content
bundle.
Cover the easy-to-miss pieces: NVIDIA still needs container-toolkit (unless also
pre-installed on host) and the CUDA validation image, plus the k3s/rke2
containerd config env overrides for Palette Edge; AMD needs cert-manager
installed first, a DeviceConfig CR with driver.enable=false, and node labelling.
Recommend enumerating exact image tags via 'helm template' and note Palette
content-bundle additional-images handling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(gpu): default NVIDIA_DRIVER_BRANCH to 580
Bump the default NVIDIA driver branch from 570 to 580 across the Earthfile ARG,
install-nvidia-drivers.sh, .arg.template, and docs. Refresh point-release
examples to 580.159.03. The 570 entries left in "e.g. 550, 570, 580" lists are
valid-branch examples, not defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Modified install-amdgpu-drivers.sh for fixing build errors
* fix(gpu): block nouveau at kernel cmdline + in initrd to prevent boot hang
On modern NVIDIA data-center GPUs (Ada/Hopper/Blackwell — reproduced on
HPE ProLiant Compute DL380a Gen12 with a GH100-class GPU), a CanvOS image
built with INSTALL_NVIDIA_GPU_DRIVERS=true would hang forever at first
boot inside systemd-udev-settle. Console showed a kernel stack trace
ending at gh100_gsp_oneinit+0x2cf/0x440 [nouveau]: initramfs udev
auto-loaded nouveau before the rootfs modprobe.d blacklist could apply,
and nouveau's experimental GSP-RM path hung on the GPU. udev-settle
never returned; the node never came up; nvidia.ko never got to load.
Three changes, together sufficient to unbrick boot:
* Earthfile: when INSTALL_NVIDIA_GPU_DRIVERS is true, append
rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nouveau.modeset=0
to /etc/cos/bootargs.cfg. This is the actual load-bearing fix:
rd.driver.blacklist= is honored by dracut/initramfs *before* any udev
event fires, so nouveau never binds. Mirrors the existing cgroup-v2
append pattern immediately above.
* install-nvidia-drivers.sh: write /etc/dracut.conf.d/95-blacklist-nouveau.conf
so BOTH the script's own dracut rebuild AND the later Earthfile-driven
dracut rebuild (a) omit nouveau.ko from the initrd entirely and
(b) carry the modprobe blacklist file into initramfs. Belt-and-braces
in case the cmdline args are ever stripped downstream.
* install-nvidia-drivers.sh: drop nvidia_drm from /etc/modules-load.d/nvidia.conf.
On headless GPU nodes it serves no purpose and loading it at early boot
has a class of KMS/DRM-conflict failure modes. Loads on demand if
anything actually needs it.
Follow-up: the UKI build path (--extend-cmdline in the build-uki targets)
also needs the same nouveau cmdline args when INSTALL_NVIDIA_GPU_DRIVERS
is true. Not addressed here since field failures were non-UKI grub boot;
worth adding in a subsequent commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(gpu): default NVIDIA_DRIVER_TYPE to open for Hopper/Blackwell support
Flip the default flavor of the pre-installed NVIDIA kernel modules from
"proprietary" to "open". The closed modules fail RmInitAdapter (0x22:0x56:897)
on Hopper (H100/H200) and Blackwell (RTX PRO 6000 Blackwell, B100/B200/GB200);
`nvidia-smi` on such a host reports "No devices were found" and the entire
GPU Operator stack downstream stays wedged (toolkit init loops, containerd
never registers the nvidia runtime, device plugin never advertises
nvidia.com/gpu, workload pods stay Pending on Insufficient nvidia.com/gpu).
The open modules also work fine on Turing, Ampere, and Ada, so the new
default is safe for every server-class NVIDIA GPU generation Turing and
newer -- essentially every card CanvOS is likely to target. Only Pascal /
Volta (V100/P100/P40) require the closed modules and now need an explicit
NVIDIA_DRIVER_TYPE=proprietary override.
Applied consistently across:
* scripts/install-nvidia-drivers.sh -- runtime default + header comment
* Earthfile -- ARG default
* .arg.template -- user-facing hint
* docs/nvidia-gpu-airgapped.md -- config table plus new "Choosing the
module flavor" section with a hardware/flavor matrix and the RmInitAdapter
symptom description for people googling the error.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(gpu): add inbox driver-source mode + fail-hard AMD DKMS install
The pre-install script silently shipped broken images when AMD's amdgpu-dkms
source did not build against the image kernel (observed on Ubuntu 24.04 HWE
6.17 with the previous 7.2.4 / 30.30.4 default, whose 6.16.13 source fails at
DKMS configure). The verify step accepted the in-tree amdgpu.ko.zst as
sufficient because it only searched anywhere under /lib/modules/$KVER/, and
apt-get failures were papered over. Nodes booted using the in-tree driver
instead of the DKMS one without any signal that this had happened.
Changes:
- Introduce AMDGPU_DRIVER_SOURCE={dkms,inbox}. dkms (default) DKMS-builds AMD's
out-of-tree amdgpu; inbox skips the AMD apt repo entirely and only ensures
the kernel's in-tree amdgpu autoloads. Any other value is a hard error.
- Rename AMDGPU_ROCM_VERSION -> AMDGPU_DRIVER_RELEASE (the URL segment is a
driver-release marker, not a ROCm x.y.z). Bump default 7.2.4 -> 31.30
(amdgpu-dkms 6.19.4, covers 22.04/24.04 kernels through 6.17).
- Fail hard in dkms mode: set -eo pipefail; remove '|| true' shims around
dkms build/install; drop 'dpkg --configure -a --force-all' silent-fallback;
verify amdgpu landed under $MODDIR/updates/dkms AND that dkms status shows
': installed' for the target kernel. Failure prints make.log tail and
points at bumping AMDGPU_DRIVER_RELEASE or switching to inbox.
- Write /etc/canvos/amdgpu-driver-source on-node marker so ops can query
which mode ran and, in dkms mode, the release + module/version + kernel.
- Docs: two-modes explainer up front, snapshot mapping table (2026-07-11)
linking to AMD's canonical system-requirements page, "when the DKMS build
fails" section documenting the bump-or-fallback flow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): skip DKMS module signing in AMD driver install (no UEFI in build container)
amdgpu-dkms >= 6.18 (observed on release 31.30 = amdgpu-dkms 6.19.4) invokes
mokutil from the DKMS sign_tool hook to enroll a Machine Owner Key. mokutil
reads /sys/firmware/efi/efivars, which is not exposed inside Docker/Earthly
RUN containers, so it aborts with:
"EFI variables are not supported on this system /
/sys/firmware/efi/efivars not found, aborting."
and the amdgpu-dkms postinst returns non-zero, failing the base-image build.
Drop /etc/dkms/framework.conf.d/canvos-no-mok-signing.conf with sign_tool=""
immediately before apt-get install amdgpu-dkms. DKMS honors this and skips
signing entirely, sidestepping the mokutil invocation. Modules are unsigned;
this path already documents that Secure Boot / UKI is unsupported.
The step-5 error message now lists this as the third common failure mode so
future regressions surface with a pointer at the drop-in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(gpu): default AMDGPU_DRIVER_RELEASE to 7.2.1 (matches GPU Operator v1.5.0)
Empirical reproduction on the kairos-ubuntu:24.04-core-amd64-generic-v4.0.4
base image (kernel 6.17.0-35-generic) with the AMDGPU_ROCM_VERSION=7.2.4-era
script + Abhijit's masking reverted showed that amdgpu-dkms 6.16.13 builds
cleanly against Linux 6.17: DKMS reports the module `installed`, all 8 amd*.ko
files land under /lib/modules/<kver>/updates/dkms/, and apt-get returns 0.
The "EFI variables are not supported on this system / /sys/firmware/efi/efivars
not found, aborting" line printed during the 30.30.x postinst is a cosmetic
mokutil warning (sign_tool step); the postinst continues past it and finishes
normally. The signing config drop-in from ad4c6fa is still valuable as
defensive belt-and-suspenders for the 31.x line where the same message is
treated as fatal.
Given that, revert the previous 31.30 default (which crossed AMD's tech-
preview boundary) back to 7.2.1 -- the release AMD's GPU Operator v1.5.0
release notes explicitly pair the operator with. AMD publishes both ROCm-alias
and driver-release-marker URL segments at repo.radeon.com/amdgpu-install/, so
"7.2.1" is a valid AMDGPU_DRIVER_RELEASE (equivalent to "30.30.1").
Also refactor the docs' "Version alignment" section to lead with the
operator → ROCm → driver → kernel chain and explicitly label the 31.x line
as tech-preview.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* diag(gpu): dump DKMS make.log + env probes on amdgpu-dkms install failure
Earthly's -i interactive debugger frequently hands out a broken tty in the
buildkit sandbox, making /var/lib/dkms/*/build/make.log unreachable when
amdgpu-dkms's postinst fails. Instead of asking users to debug interactively,
tail the make.log(s) and a few environment probes (kernel headers state,
Module.symvers presence, sign_tool drop-in state, MemAvailable) to stderr
before die-ing. The next non-interactive rebuild surfaces the underlying
compile error directly in the Earthly output.
No behavior change on the happy path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): mark AMD driver-install RUN as --privileged
amdgpu-dkms's ./configure runs a Kbuild probe to detect CFLAGS. In buildkit's
default unprivileged sandbox that probe fails with
configure: error: cannot detect CFLAGS...
grep: amd/dkms/config/config.h: No such file or directory
make: *** No rule to make target 'amd/dkms/config/config.h'
Reproduced against a plain `docker run --privileged` baseline: same script,
same base image (kairos-ubuntu:24.04-core-amd64-generic-v4.0.4), same kernel
(6.17.0-35-generic), same AMDGPU_DRIVER_RELEASE=7.2.1 -- succeeds end-to-end
with modules landing under /lib/modules/<kver>/updates/dkms/. The only
observed difference is --privileged, so mark the AMD RUN step explicitly.
The wrapper (./earthly.sh) already passes --allow-privileged, and other RUN
steps in this repo use --privileged similarly, so no downstream change needed.
NVIDIA's driver install doesn't use the same ./configure sanity-check step
and continues to work unprivileged; its RUN block is left as-is.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): add full Kbuild bootstrap deps before amdgpu-dkms install
amdgpu-dkms's ./configure fails silently at "cannot detect CFLAGS..." when
Kbuild can't complete the `make -f -` heredoc probe it uses to extract
KBUILD_CFLAGS. build-essential covers gcc/make/libc but modern kernel
Makefiles unconditionally reach for bc, bison, flex, libelf-dev, libssl-dev,
and pahole (from dwarves). --no-install-recommends skips all of those.
Enumerate them explicitly to keep the image lean without breaking the AMD
DKMS build.
dwarves is AMD's own `Recommends:` on amdgpu-dkms -- it just wasn't pulled
because we use --no-install-recommends. cpio + xz-utils are added
defensively for the initramfs trigger that runs after DKMS install.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(gpu): add scripts/prebuild-amdgpu-artifact.sh (host-side DKMS build)
Compiles amdgpu-dkms against the kairos base image's kernel in a host-side
`docker run --privileged` container, then tars the produced kernel module +
firmware + config drop-ins into build/amdgpu-artifact-<release>-<kver>-<digest>.tar.gz.
The main Earthly build consumes that artifact via COPY, sidestepping the
buildkit RUN sandbox limitation that breaks AMD's ./configure heredoc probe.
Cache by (base-image digest x kernel version x AMDGPU_DRIVER_RELEASE); repeat
builds skip the ~8-10 min compile. Cache miss reproduces the same behavior we
already verified end-to-end via direct docker run on the jump host.
Prints "AMDGPU_ARTIFACT_PATH=<abs-path>" on the last stdout line so the
earthly.sh wrapper (next commit) can pick it up and thread it into Earthly.
Standalone; no other files touched by this commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(gpu): hook AMD driver prebuild into earthly.sh wrapper
Before dispatching to earthly (build_with_proxy/build_without_proxy), detect
INSTALL_AMD_GPU_DRIVERS=true + AMDGPU_DRIVER_SOURCE=dkms from either .arg or
CLI overrides, derive BASE_IMAGE via the same tag formula the Earthfile uses,
and invoke scripts/prebuild-amdgpu-artifact.sh to compile the amdgpu module
on the host. The last line of prebuild output is parsed for the artifact
path; it's threaded to earthly as --AMDGPU_ARTIFACT_PATH=<abs>, which the
companion Earthfile change consumes via COPY + tar-extract.
Inbox mode skips this step entirely. Non-AMD builds skip this step entirely.
Existing CLI/.arg semantics for INSTALL_AMD_GPU_DRIVERS, AMDGPU_DRIVER_SOURCE,
AMDGPU_DRIVER_RELEASE, and BASE_IMAGE are preserved unchanged.
Ubuntu-only guard: prebuild refuses OS_DISTRIBUTION != ubuntu with a clear
error, mirroring the mutual-exclusion / Ubuntu-only checks already in the
Earthfile for the AMD block.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(gpu): consume AMD driver artifact in Earthfile + install script + docs
Complete the prebuild-artifact flow started in the previous two commits.
Earthfile:
* new ARG AMDGPU_ARTIFACT_PATH (empty default)
* base-image AMD block now branches on (AMDGPU_DRIVER_SOURCE, artifact):
- dkms + artifact present -> COPY tarball, extract, depmod, initrd
(fast path used by ./earthly.sh + prebuild helper -- the release path)
- inbox, or dkms without artifact -> unchanged legacy path (works under
`docker run --privileged`; fails under buildkit's RUN sandbox at
AMD's ./configure -- documented)
scripts/install-amdgpu-drivers.sh:
* new AMDGPU_ARTIFACT_PATH fast path before the in-buildkit DKMS block:
extract tarball, verify updates/dkms/amdgpu.ko*, depmod, rebuild initrd,
write the on-node marker at /etc/canvos/amdgpu-driver-source. In-buildkit
DKMS legacy path retained for direct `docker run` invocations and for
scripts/prebuild-amdgpu-artifact.sh's own inner container.
* docstring updated to describe the three execution paths.
docs/amd-gpu-airgapped.md:
* new "two-stage build" explanation covering why the prebuild exists
(buildkit RUN sandbox breaks AMD ./configure) and how it's wired up
behind ./earthly.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): invoke install script via bash (RO bind mount can't be chmod'd)
The prebuild's inner container bind-mounts the install scripts read-only
(-v ...:ro). chmod +x fails on RO mounts. Bash can execute scripts without
+x when invoked as `bash <path>`; no functional difference otherwise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): preserve artifact on verify failure + dump tar listing
Previous verify path deleted the tarball via STAGE_DIR trap before the user
could inspect it. Now:
* mv to final path FIRST (before verify), so any failure leaves the file
behind for inspection
* on verify failure, dump the first 40 tar entries to stderr so we can
see what path structure actually landed in the archive
* grep pattern loosened to `updates/dkms/.*amdgpu\.ko` (regex) in case
tar mangled paths in an unexpected way
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): pass artifact path repo-relative to Earthly (COPY reads build context)
Earthly's COPY expects paths relative to the Earthfile's directory (the
repo root), not absolute host paths. The prebuild helper emits an absolute
path for its own file-existence checks; the wrapper now converts that to
a repo-relative path before threading it into Earthly as
--AMDGPU_ARTIFACT_PATH=build/amdgpu-artifact-...tar.gz.
Fails fast with a clear message if the artifact somehow lands outside the
repo tree.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): whitelist AMD driver artifact in .earthlyignore
`build/*` in .earthlyignore was hiding the pre-built AMD driver tarball
from Earthly's build context, so COPY build/amdgpu-artifact-...tar.gz
failed with "no such file or directory" even though the artifact existed
on the host.
Add a negation pattern for build/amdgpu-artifact-*.tar.gz so this specific
artifact is visible while everything else under build/ (Earthly SAVE
ARTIFACT outputs, etc.) stays ignored.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): omit amdgpu from initrd; multi-GPU init storms dracut-initqueue
Reported: node with baked-in DKMS amdgpu (8x MI325X) fails to boot -- drops
into dracut-emergency after `dracut-initqueue: Timed out for waiting the udev
queue being empty`. Screenshots show amdgpu still emitting per-XCP-partition
and per-ring init messages long after boot has given up.
Root cause: rebuilding the initrd (default true) pulled the DKMS amdgpu
modules under /lib/modules/<kver>/updates/dkms/ into the initrd. amdgpu then
loads *before* switch-root, and the sheer volume of udev events it emits on
multi-GPU MI silicon (8 GPUs x 8 XCP partitions + JPEG rings + SMU/PSP/KFD
handshakes) keeps systemd-udev-settle from ever seeing an empty queue.
initqueue times out (~5 min), boot fails, node reboots, loops.
amdgpu is not required to mount rootfs -- NVMe/SATA use their own drivers.
Load it *after* switch-root via /etc/modules-load.d/amdgpu.conf (which the
script was already writing), where there is no timeout pressure. This
matches kairos-a176's observed behavior with the in-tree driver.
Two guards:
1. Flip default AMDGPU_REBUILD_INITRD to `false` (was `true`) in
Earthfile, .arg.template, and install script.
2. Even when someone sets AMDGPU_REBUILD_INITRD=true, drop
/etc/dracut.conf.d/98-canvos-amdgpu-omit.conf with
`omit_drivers+=" amdgpu amdttm amdkcl amd-sched amddrm_ttm_helper
amddrm_buddy amddrm_exec amdxcp "` so dracut still leaves amdgpu out.
Applied consistently across all three code paths in install-amdgpu-drivers.sh
(artifact fast path, inbox mode, in-buildkit DKMS fallback).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat: support installer boot from USB media on Renesas xHCI hardware (#687)
* feat: support installer boot from USB media on Renesas xHCI hardware
Bundle the Renesas xHCI (USB 3.0) host controller driver into the
initramfs and disable PCI reallocation on the installer boot entries so
the Palette Edge installer can boot and install from USB media on
hardware using that chipset.
- Earthfile: write /etc/dracut.conf.d/99-usb-media.conf in base-image
before the distro dracut regeneration so xhci_pci_renesas is included
in the generated initramfs (add_drivers + force_drivers).
- grub.cfg: add pci=realloc=off to the installer menu entries.
* feat(fips): bundle Renesas xHCI driver into initramfs for USB media boot
The Earthfile skips dracut regeneration for FIPS builds, so the USB media
driver config must be added in the ubuntu-fips Dockerfiles where
kairos-init regenerates the initramfs (-s init --fips).
Add a 99-usb-media.conf (xhci_pci_renesas via add_drivers + force_drivers)
to each Ubuntu FIPS version (20.04/22.04/24.04) and COPY it into
/etc/dracut.conf.d/ before the kairos-init init step that runs dracut.
* fix(boot): block nouveau + qat_4xxx on installer ISO and installed OS
Installer ISO was hanging on the same NVIDIA data-center GPUs previously
addressed for the installed OS in 1c3a089: nouveau auto-loads in the live
ISO's initramfs before switchroot, hits GSP init, and stalls udev-settle
indefinitely. Runtime bootargs.cfg had the fix; installer ISO did not.
Also block qat_4xxx unconditionally: Xeon Scalable 4th/5th gen hosts with
Intel QuickAssist devices load qat_4xxx in initramfs and its probe /
firmware-load path adds minutes to boot. CanvOS does not consume QAT
acceleration, so blacklisting is safe on every build.
Three sites, one arg pair each (rd.driver.blacklist= is the load-bearing
flag — dracut honors it before udev fires; modprobe.blacklist= is
belt-and-braces for post-switchroot):
* overlay/files-iso/boot/grub2/grub.cfg: append
rd.driver.blacklist=nouveau,qat_4xxx modprobe.blacklist=nouveau,qat_4xxx
nouveau.modeset=0 to all three menuentries (default install, manual
install, interactive install). Unconditional — the same ISO installs
onto NVIDIA and non-NVIDIA hardware, and blacklisting an absent module
is a no-op.
* Earthfile CMDLINE default: same three args appended, so enki
--extend-cmdline bakes them into the UKI stub for both signed and
unsigned installer paths. Addresses the follow-up flagged in 1c3a089
for the UKI build path.
* Earthfile base-image bootargs.cfg block: new unconditional RUN for
qat_4xxx. Two-branch sed — if the nouveau block above already added
rd.driver.blacklist=, splice ,qat_4xxx onto the existing list rather
than duplicating the arg key (some cmdline parsers treat repeated
keys as last-wins and silently drop the earlier value). Otherwise
add a fresh rd.driver.blacklist=qat_4xxx modprobe.blacklist=qat_4xxx
pair. Idempotent via outer grep guard.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(gpu): enable NVIDIA fabricmanager + nscq + imex by default for HGX/DGX/GB200
HGX H100/H200, HGX B200, DGX and GB200 need nvidia-fabricmanager to bring up
multi-GPU NVLink; GB200 NVL72 additionally needs nvidia-imex for multi-node
NVLink Sharp. Both daemons are userspace, have no kernel side effect, and exit
cleanly on hardware that does not need them (fabricmanager: "No NvSwitch found"
-> failed/inactive; imex: no nodes_config.cfg -> exits, unit inactive), so
default-on is safe on non-target fleets at a ~70-110 MB image cost.
Also install libnvidia-nscq-<branch> explicitly alongside fabricmanager -- it
is a transitive dep today, listed explicitly so the build fails loudly if the
CUDA repo ever drops the auto-dep. IMEX is best-effort: pre-570 driver branches
do not publish the package and the install step warns and skips.
Opt-out per node type via --NVIDIA_INSTALL_FABRICMANAGER=false /
--NVIDIA_INSTALL_IMEX=false.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): install infiniband-diags with fabricmanager so ibstat is present
The nvidia-fabricmanager-<branch> package ships a unit whose ExecStart is a
wrapper (/usr/share/nvidia/fabricmanager/nvidia-fabricmanager-start.sh) that
NVIDIA added to gate FM startup on the IB fabric being up -- needed for
GB200 NVL72 multi-node NVLink Sharp. The wrapper shells out to `ibstat`
(from infiniband-diags) to probe IB device state; without that binary the
unit fails immediately on every boot, regardless of hardware, with:
"ibstat" command not found! Please install "ibstat".
nvidia-fabricmanager.service: Control process exited, code=exited, status=1
Install infiniband-diags whenever fabricmanager is installed. ~5-8 MB, no
runtime side effect on non-IB hosts. On non-NVSwitch hardware the wrapper
now proceeds to nv-fabricmanager which exits "No NvSwitch found" as
originally expected -- the unit stays failed with no kernel effect and no
restart loop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): bypass GB200 NVL72 wrapper on fabricmanager so FM starts on HGX/DGX
NVIDIA's 570+ nvidia-fabricmanager-<branch> unit invokes a wrapper --
/usr/share/nvidia/fabricmanager/nvidia-fabricmanager-start.sh -- that gates
FM startup on the GB200 NVL72 NVLink subnet coming up. The wrapper shells
out to `ibstat` (infiniband-diags) and `nvlsm` (NVIDIA Subnet Manager,
shipped in UFM/MLNX-OFED, not in this package). Installing infiniband-diags
gets past the first gate but the wrapper then dies on missing nvlsm, on
every non-NVL72 topology (standalone HGX H100/B200, single-node DGX,
workstations). FM never runs on any of that hardware.
Install a systemd drop-in at
/etc/systemd/system/nvidia-fabricmanager.service.d/10-bypass-wrapper.conf
that overrides ExecStart back to calling nv-fabricmanager directly (the
pre-wrapper behavior). FM then correctly initializes NVSwitch on HGX/DGX,
and cleanly exits "No NvSwitch found" on non-NVSwitch hosts.
GB200 NVL72 deployments must remove this drop-in via user-data so the
wrapper's IB/nvlsm precondition applies. infiniband-diags is retained
defensively (~5-8 MB) so a user reverting to the wrapper doesn't
re-encounter the ibstat gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): install nvlsm with fabricmanager so vendor wrapper works on all hardware
The NVIDIA 570+ fabricmanager unit invokes a wrapper
(/usr/share/nvidia/fabricmanager/nvidia-fabricmanager-start.sh) that probes
`ibstat` (infiniband-diags) and `nvlsm` (NVIDIA Subnet Manager) before
starting nv-fabricmanager -- needed to gate on the GB200 NVL72 NVLink
subnet coming up. Without both binaries the unit dies before FM is ever
invoked.
Prior fix bypassed the wrapper via a systemd drop-in. That worked, but
diverges from the vendor path and would break GB200 NVL72 deployments
that legitimately need the IB/nvlsm precondition. Verified on a live
HGX host that installing nvlsm (2025.10.14-1, from the CUDA repo we
already add -- unversioned package name) lets the vendor wrapper
succeed on standalone HGX topologies too. FM comes up and GPU Operator
pods reach Ready.
Install nvlsm alongside fabricmanager + libnvidia-nscq + infiniband-diags,
and drop the 10-bypass-wrapper.conf systemd override. Same vendor code
path on HGX/DGX and GB200 NVL72 alike.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(boot): apply nouveau + qat_4xxx + pci=realloc=off to every installed OS
Follow-up to 83cfcf6. The previous nouveau block was gated on
INSTALL_NVIDIA_GPU_DRIVERS=true (inherited from 1c3a089), which meant
non-NVIDIA CanvOS builds that later landed on NVIDIA hardware still
hung in GSP init on first boot. The image doesn't know its target
hardware at build time — the hang is a property of the host, not the
build flag — so blacklisting must be unconditional. Same reasoning
as qat_4xxx, which was already unconditional in the same block.
Collapses the two prior sed passes (gated nouveau + unconditional
qat_4xxx-splice) into one unconditional sed that appends both
blacklists plus pci=realloc=off in a single append, mirroring the
installer ISO grub.cfg cmdline. Idempotent via the grep guard on
"rd.driver.blacklist=nouveau".
pci=realloc=off pins the PCI BAR layout to firmware assignments;
kernel-side reallocation has caused resource conflicts on some
server platforms, and the installer ISO already carries it.
UKI CMDLINE default reverted — UKI installer path is out of scope
for this change per user direction.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gpu): autoload ib_umad so fabricmanager precheck passes on NVL5+ (HGX B200) (#700)
nvidia-fabricmanager 570+ ships a systemd unit whose ExecStartPre invokes
`nvidia-fabricmanager-start.sh --mode precheck`. On HGX B200 / GB200
hardware the wrapper takes the "Detected NVL5+ system" branch, which
requires the ib_umad kernel module to be loaded before nv-fabricmanager
can start -- the daemon opens /dev/infiniband/umad* character devices to
send MADs to the NVSwitch fabric (NVSwitch reuses the InfiniBand
management-datagram shape for topology discovery + NVLink route
programming).
Kairos edge images ship the module in the kernel-modules-extra set but
don't auto-load it. Result: on first boot of a Palette Edge HGX B200
appliance, the service fails at ExecStartPre with:
Detected NVL5+ system
Kernel module "ib_umad" has not been loaded, fabric manager cannot be started
Please run "modprobe ib_umad" before starting fabric manager
That cascades: without fabricmanager the NVSwitch fabric never initializes;
CUDA runtime in any container fails cudaMalloc with CUDA_ERROR_NOT_INITIALIZED
("system not yet initialized"); nvidia-cuda-validator crashes at Init:Error;
nvidia-operator-validator hangs at Init:2/4.
Fix: write /etc/modules-load.d/nvidia-fabricmanager.conf listing ib_umad
so systemd's kmod-static-nodes / systemd-modules-load loads it at boot
ahead of nvidia-fabricmanager.service.
The file is scoped to the same conditional that installs the fabricmanager
package (NVIDIA_INSTALL_FABRICMANAGER=true), so hardware where the operator
opts out of fabricmanager doesn't get the module load either.
On non-NVSwitch hosts the module costs ~10 KiB of RSS and has no other
effect; fabricmanager still exits "No NvSwitch found" and the unit stays
inactive as before.
Verified live on 8x HGX B200 running driver 580.159.03: with the drop-in
in place, `systemctl start nvidia-fabricmanager` transitions from failed
to active in ~3s, the NVSwitch fabric initializes, and downstream
nvidia-cuda-validator exits 0 on the next GPU-operator reconcile.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(pam): bake libpam-pwquality into Ubuntu base image (#702)
Adds libpam-pwquality to the Ubuntu apt install list so the pam_pwquality.so
module is available for password-complexity enforcement in downstream user-data
stages (e.g. AI Launchpad's default password policy in launchpad-ai#314).
Airgap-safe: package is baked into the base image, not fetched at install.
libpam-modules 1.4.0 on Ubuntu 24.04 does not support pam_exec + expose_authtok
for password-type stanzas, so pam_pwquality is the correct primitive here.
---------
Co-authored-by: Vipin Sharma <sharma.vipin@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: abhijithspectro <abhijith.balan@spectrocloud.com>
Co-authored-by: Akhilesh Verma <akhilesh@spectrocloud.com>
Co-authored-by: Nianyu Shen <nianyu@spectrocloud.com>
Co-authored-by: Akhilesh Verma <akhlesh.vermaofc@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
/etc/modules-load.d/nvidia-fabricmanager.conflistingib_umad, inside theNVIDIA_INSTALL_FABRICMANAGER=trueblock ofscripts/install-nvidia-drivers.sh. One kernel module, three lines of shell, ~30 lines with rationale + comments.Why
nvidia-fabricmanager 570+ships a systemd unit whoseExecStartPreinvokesnvidia-fabricmanager-start.sh --mode precheck. On HGX B200 / GB200 the wrapper takes the "Detected NVL5+ system" branch, which requiresib_umadto be loaded beforenv-fabricmanagercan start — the daemon opens/dev/infiniband/umad*character devices to send MADs to the NVSwitch fabric (NVSwitch reuses the InfiniBand management-datagram shape for topology discovery + NVLink route programming).Kairos edge images ship
ib_umadin the kernel-modules-extra set but don't auto-load it. Result on a fresh Palette Edge HGX B200 appliance:That cascades all the way down:
Fix
Inside the existing
NVIDIA_INSTALL_FABRICMANAGER=trueblock (right aftersystemctl enable, in theapt install succeededpath):Scoped to the same conditional that installs the fabricmanager package, so hardware where the operator opts out (
NVIDIA_INSTALL_FABRICMANAGER=false) doesn't get the module load either.What this doesn't do (and doesn't need to)
ib_core,mlx5_ib,mlx5_core— those are already auto-loaded by the Mellanox NIC hotplug/DRBD paths (verified vialsmodon the failing box: they were all loaded, onlyib_umadwas missing).ib_umadis compiled into the kernel-modules-extra set that Kairos already ships."No NvSwitch found"and the unit stays inactive.Verification
Applied live on a running HGX B200 (8× B200, driver 580.159.03) before opening this PR:
Test plan for maintainer
bash -n scripts/install-nvidia-drivers.shcleanNVIDIA_INSTALL_FABRICMANAGER=trueconditionalsudo systemctl status nvidia-fabricmanagershould beactive (running)without operator interventionlsmod | grep ib_umadshows the module loaded on the HGX box after rebootRelated upstream context
This is really a kernel-modules-extra autoload gap in Kairos, not a fabricmanager bug. The fabricmanager wrapper's error message is helpful — it literally names the fix. Long term, upstream Kairos should probably ship
ib_umadin a default modules-load.d set for images that carry the NVIDIA fabricmanager package (mirroring how it already shipsnvidia,nvidia_uvm,nvidia_modesetinscripts/install-nvidia-drivers.shsection 10). This PR is the minimal same-file fix for now.🤖 Generated with Claude Code