fix(installer): enable GPU on Windows — CPU fallback + real GPU support (#616)

[shujaatTracebloc]

What & why

On Windows, a machine that has a working NVIDIA GPU still ran experiments on CPU — and silently. The training pod requested nvidia.com/gpu=1, the k3d node advertised 0 GPUs, so it sat Pending ("Insufficient nvidia.com/gpu") until the SINGLE_NODE fallback downgraded it to CPU. GPU present, GPU wasted, no signal to the user.

Root cause — an asymmetry in install-k8s.ps1: the installer requested a GPU for every job whenever it merely detected a GPU + driver, but the GPU is only provisioned in the cluster (--gpus=all, device plugin, node verify) when K3D_GPU_FLAG is set — and that's set only if the "install NVIDIA Container Toolkit into WSL" step fully succeeds, which has ~6 ways to silently bail (no WSL, distro needs first-run setup, apt/gpg blocked on a filtered network, timeouts).

Closes #616.

Layer 1 — never request a GPU the cluster can't provide ✅ (this commit)

Safe, always-correct, CPU fallback preserved:

• Gate GPU_REQUESTS/GPU_LIMITS on K3D_GPU_FLAG — the same condition that provisions the GPU, not mere detection. Empty ⇒ no GPU request ⇒ training runs on CPU. The SINGLE_NODE GPU→CPU downgrade stays as a belt-and-suspenders net.
• Loud skip: capture a specific $GPU_SKIP_REASON at each toolkit early-return (WSL not responding / Ubuntu needs first-run setup / toolkit apt blocked or timed out / verify failed), surface it in the summary (CPU (GPU detected but not enabled: <reason>)) and warn during install.
• Gate the doctor's GPU-test hint on K3D_GPU_FLAG (don't suggest a test that can't run).
• Tests: 4 new Pester tests; full suite 456 pass. manifest.sha256 regenerated.

Layer 2 — actually enable the GPU in k3d on Windows ✅ (implemented; needs GPU-box validation)

Makes a Windows NVIDIA GPU genuinely usable by training pods, CPU fallback preserved whenever it can't be wired up.

2a — custom k3s-CUDA node image (docker/k3s-cuda/): the stock rancher/k3s image is Alpine with no NVIDIA runtime, so GPU pods can never schedule on it. New image rebuilds the same pinned k3s on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the nvidia runtime, and bakes in the device plugin + nvidia RuntimeClass (auto-deployed on boot). Published by the manual build-k3s-cuda workflow to ghcr.io/tracebloc/k3s-cuda:<k3s>-cuda-<base>.

2b — installer wiring (install-k8s.ps1):

• Confirm-DockerGpu — the authoritative gate: docker run --rm --gpus all nvidia/cuda:<tag> nvidia-smi. Docker Desktop uses its own WSL distro, so toolkit-in-Ubuntu was never a reliable signal — actually running a GPU container is. GPU is enabled iff this passes (so we never create a --gpus cluster that would fail, and DD-GPU works even without the user's Ubuntu toolkit). Failure ⇒ CPU (Layer 1) with a clear reason.
• Cluster create uses the custom CUDA image when GPU is enabled (stock k3s otherwise; same pinned k3s).
• Sets RUNTIME_CLASS_NAME=nvidia so jobs-manager threads runtimeClassName: nvidia into every spawned pod.
• Tests: Confirm-DockerGpu behavioral + source guards; full Pester 464 pass.

Remaining to close #616 (needs your GPU Windows box): run build-k3s-cuda with push: true to publish the image, then re-install on the GPU machine and confirm nvidia.com/gpu becomes allocatable and a training pod runs on the GPU. Kept as draft until that end-to-end validation passes.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes Windows cluster creation, private image pull/auth, and training GPU/runtime Helm values—wrong gating could still strand jobs or misreport GPU mode, but CPU fallback is explicit and bounded by probes before --gpus.

Overview
Fixes #616: on Windows, GPU detection no longer implies GPU requests in Helm values or “NVIDIA GPU” in the install summary when the cluster was never wired for GPUs (which left training jobs pending on nvidia.com/gpu).

Provisioning gate: After WSL toolkit setup, GPU is turned on only if Confirm-DockerGpu (docker run --gpus all + nvidia-smi) and Confirm-GpuImagePullable (optional docker login via registry env, then pull of $K3S_CUDA_IMAGE) both succeed. Failures set $GPU_SKIP_REASON at each toolkit early-exit and in the post-probe block; install warns and summary shows CPU (GPU detected but not enabled: …).

Cluster + chart: When $K3D_GPU_FLAG is set, k3d uses the custom k3s-CUDA image (env/mirror overrides) with --gpus=all; otherwise stock rancher/k3s. GPU_LIMITS/GPU_REQUESTS and RUNTIME_CLASS_NAME=nvidia are set only when the flag is set.

Tests: New Pester coverage for summary text, Helm values, Confirm-DockerGpu, Confirm-GpuImagePullable, and source guards; manifest.sha256 updated for install-k8s.ps1.

^{Reviewed by Cursor Bugbot for commit 46d90a4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1a49577. Configure here.}

[cursor]

Host probe misrepresents cluster GPU readiness

High Severity

K3D_GPU_FLAG reflects host Docker capability, not actual cluster readiness. Existing clusters are reused without the CUDA image or --gpus, and failed node verification never clears the flag, yet Helm requests the GPU runtime and the summary claims acceleration. Training jobs can remain unschedulable.

Additional Locations (2)

• scripts/install-k8s.ps1#L2755-L2757
• scripts/install-k8s.ps1#L3764-L3770

 

^{Triggered by project rule: Bugbot guide — tracebloc/client}

^{Reviewed by Cursor Bugbot for commit 1a49577. Configure here.}

[cursor]

GPU Docker calls can hang installer

Medium Severity

The new docker run, docker login, and docker pull calls are unbounded, violating the installer external-call timeout rule. A wedged Docker daemon, registry, or proxy can leave Step 2 hanging forever instead of falling back to CPU.

Additional Locations (2)

• scripts/install-k8s.ps1#L1729-L1730
• scripts/install-k8s.ps1#L1734-L1735

 

^{Triggered by learned rule: Installer scripts: kubectl/curl/helm calls must have explicit timeouts}

^{Reviewed by Cursor Bugbot for commit 1a49577. Configure here.}

[cursor]

GPU probe bypasses configured image mirror

Medium Severity

Although the CUDA node image is re-homed through TRACEBLOC_IMAGE_REGISTRY, the authoritative capability probe always pulls nvidia/cuda directly from Docker Hub. Sites using an internal mirror because Docker Hub is blocked therefore always disable GPU support.

Additional Locations (1)

• scripts/install-k8s.ps1#L638-L642

 

^{Reviewed by Cursor Bugbot for commit 1a49577. Configure here.}

[cursor]

Toolkit failure masks Docker probe failure

Low Severity

Confirm-DockerGpu does not set a failure reason, while the earlier toolkit step always seeds one. When both steps fail, the authoritative Docker failure preserves an unrelated WSL toolkit reason, directing the operator toward a fix that cannot enable Docker GPU passthrough.

Additional Locations (1)

• scripts/install-k8s.ps1#L1553-L1558

 

^{Reviewed by Cursor Bugbot for commit 1a49577. Configure here.}