fix(installer): stop k3s version pin from silently drifting (#547) - #565
Merged
Conversation
Root-caused from the Windows stuck-install incident: a client ran k3s v1.35.5 while the pin was v1.29.4-k3s1. Addresses three of the four compounding gaps the audit found (F3 left as a tracked checklist item): - F1: the header docs advertised `default: latest`, inviting users to set K8S_VERSION=latest, which floats to k3d's bundled default k3s. Fix the docs in both installers, and warn loudly at create time when `latest` is used. - F2: the reuse/adopt path never re-checked the running node's k3s version, so a cluster born unpinned (old installer / latest / manual create) persisted forever across later correctly-pinned re-runs — the single best explanation for the observation. Add _check_existing_cluster_k8s_version (bash) and a parity check in New-K3dCluster (PowerShell): warn + recreate remedy on drift. - F4: check-facts.sh only compared the pinned version STRINGS, not the create wiring, so `--image rancher/k3s:` could be dropped while CI stayed green. Add a structural guard asserting the pin is wired in cluster.sh + install-k8s.ps1. With --image now guaranteed on create, k3d's own version no longer floats k3s, so F3 (winget installs unpinned k3d) is de-risked and tracked in #547. Tests (only added): +7 bats for _check_existing_cluster_k8s_version, +5 Pester source guards, check-facts.bats fixture extended with the wiring line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc
marked this pull request as ready for review
August 3, 2026 13:38
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
shujaatTracebloc
added a commit
that referenced
this pull request
Aug 3, 2026
…bot #565) Bugbot: the drift check only lived on the full reuse path (_handle_existing_cluster / New-K3dCluster), but both installers short-circuit earlier when a re-run classifies as healthy (bash assess_existing_install, PS completed+healthy fast-path). A healthy-but-drifted cluster — the #547 STEADY STATE — would hit "already set up / nothing to do" and never see the warning, exactly the population the check is meant to help. - bash: assess_existing_install's healthy branch now calls _check_existing_cluster_k8s_version before the handoff (guarded by declare -F). - PS: extracted the inline reuse-path check into Test-K3sVersionDrift and call it from BOTH New-K3dCluster and the completed+healthy fast-path in main. Tests: +2 assess.bats (healthy runs it; --force skips it); Pester #547 block updated to assert the shared function + both call sites. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bot #565) Bugbot: the drift check only lived on the full reuse path (_handle_existing_cluster / New-K3dCluster), but both installers short-circuit earlier when a re-run classifies as healthy (bash assess_existing_install, PS completed+healthy fast-path). A healthy-but-drifted cluster — the #547 STEADY STATE — would hit "already set up / nothing to do" and never see the warning, exactly the population the check is meant to help. - bash: assess_existing_install's healthy branch now calls _check_existing_cluster_k8s_version before the handoff (guarded by declare -F). - PS: extracted the inline reuse-path check into Test-K3sVersionDrift and call it from BOTH New-K3dCluster and the completed+healthy fast-path in main. Tests: +2 assess.bats (healthy runs it; --force skips it); Pester #547 block updated to assert the shared function + both call sites. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc
force-pushed
the
fix/547-k3s-pin-reuse-drift
branch
from
August 3, 2026 13:59
0367d02 to
c571c45
Compare
…run --write" (Bugbot #565) Bugbot: the F4 wiring guard incremented the same `drift` counter as version-string mismatches, so a missing create-time --image pin ended with "fact(s) drifted... Run 'check-facts.sh --write'". But --write only restamps version strings and cannot restore create-time wiring — the summary pointed developers at a no-op fix. Track wiring failures in a separate counter and emit a wiring-specific message (this is a WIRING gap; restore the --image rancher/k3s:${K8S_VERSION} flag by hand). +1 check-facts.bats: a missing --image pin fails with the WIRING message and never the --write hint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Linux (#547) A client re-running the dataset-copy step hit "already exists" from non-idempotent `mkdir` + `Copy-Item -Recurse`. The repo only documented the Linux `kubectl cp` staging path, with no hostpath/Windows guidance. Add an idempotent hostpath staging section: Windows uses `New-Item -Force` + `robocopy /E` (merges into an existing target, safe to re-run); macOS/Linux use `mkdir -p` + `cp -R`. Notes the plain-mkdir "already exists" error is harmless (data already staged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bugbot: _check_existing_cluster_k8s_version (bash) and Test-K3sVersionDrift (PS) ran a bare `docker inspect` with no deadline, and both healthy fast-paths now call them — a wedged Docker engine could hang a headless "already healthy" re-run AFTER success was printed, violating the installer's bounded-probe rule. - bash: wrap the inspect in _bounded (timeout/gtimeout; 124 on timeout → the existing `|| return 0` makes it a silent no-op). - PS: run it via Start-Job + Wait-JobWithProgress -TimeoutSec 15 (mirrors Test-ClusterRunning); on timeout, skip the check with a log line. Tests: cluster.bats setup overrides _bounded so the docker shell-function mock is exercised on Linux CI too (timeout can't exec a function); Pester asserts the bounded Start-Job pattern tied to the "Checking k3s version" probe. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9c501f0. Configure here.
…hell (Bugbot #565) Bugbot: the remediation hint told devs to restore `rancher/k3s:${K8S_VERSION}` in both files, but the PowerShell guard matches the fixed string `rancher/k3s:$K8S_VERSION` (no braces) — following the hint in the PS create path would leave CI red even though --image is correctly wired. Reword the hint to name BOTH shell forms (bash cluster.sh uses ${K8S_VERSION}; PowerShell install-k8s.ps1 uses $K8S_VERSION) and point at the exact literal each ✖ line already prints. +2 assertions in check-facts.bats locking both forms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aptracebloc
approved these changes
Aug 3, 2026
This was referenced Aug 3, 2026
Contributor
|
/fr-pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Root-caused from the Windows stuck-install incident (#545/#547): the affected client's node ran k3s v1.35.5+k3s1 while the installer pin is v1.29.4-k3s1 — a version the chart was never validated against, which widened the timing window the RWO deadlock (#549) exploited.
The post-incident audit found the pin can drift through four compounding gaps. This PR closes three; F3 is de-risked and tracked in #547.
Changes
latestfloat. The header docs advertiseddefault: latest, inviting users to setK8S_VERSION=latest, which skips--imageand floats to k3d's bundled default k3s. Fixed the docs in both installers and added a loud warning at create time whenlatestis used (bash + PowerShell).latest, or a manualk3d create) kept its k3s forever across later correctly-pinned re-runs — the single best explanation for the observation. Added_check_existing_cluster_k8s_version(bash) and a parity check inNew-K3dCluster(PowerShell): warn + recreate remedy on drift, mirroring the existing proxy/CA/storage drift checks. This is what surfaces a drifted cluster (like the incident client's) on its next installer run.check-facts.shcompared only the pinned version strings, not that the create command actually wires the pin. Added a structural guard asserting--image rancher/k3s:is present incluster.shandinstall-k8s.ps1, so a refactor can't silently unpin k3s while CI stays green.F3 (Windows winget installs an unpinned k3d) is intentionally not in this PR: with
--imagenow guaranteed on create, k3d's own version can no longer float k3s, so F3 is de-risked to a k3d-CLI-determinism nicety — tracked as a checklist item in #547.Tests (added only)
scripts/tests/cluster.bats: +7 for_check_existing_cluster_k8s_version(empty/latest no-op, match, drift → warn, registry+digest tag parse, unparseable ref, docker-fails).scripts/tests/install-k8s.Tests.ps1: +5 source guards (create--image,latestwarning, reuse inspect+compare, recreate remedy, docs no longer saydefault: latest).scripts/tests/check-facts.bats: fixture extended additively with the wiring line so the new F4 guard is exercised.Local: cluster.bats 74/74, check-facts.bats 13/13, Pester 387/0/9,
shellcheck --severity=errorclean,bash -nclean, PS parses clean, PSScriptAnalyzer adds nothing.Closes #547
Note
Medium Risk
Touches core cluster creation and reuse paths in bash/PowerShell installers; behavior is warn-only on drift but changes defaults and CI gates for k3s pinning.
Overview
Hardens the local installer so the validated k3s pin is actually used at cluster create time and drift on reused clusters is visible instead of silently reused (#547).
Create path: Bash and PowerShell now document the pinned default (not
latest), warn loudly whenK8S_VERSION=latest, and pass--image rancher/k3s:<pin>on freshk3dcreates. Reuse / healthy fast-path: New checks compare the running node image to the pin and print a non-fatal recreate hint (including when install would otherwise exit on “already healthy”).check-facts.shadds a wiring guard so CI fails if the create-time--imageliteral is removed fromcluster.sh/install-k8s.ps1, with messaging that--writecannot fix wiring gaps.Docs:
INSTALL.mdadds idempotent hostpath staging steps for laptop installs (PowerShellrobocopy/ Unixcp -R). Tests cover drift checks, assess healthy-path behavior, check-facts wiring failures, and PowerShell source guards; installer script hashes inmanifest.sha256are updated.Reviewed by Cursor Bugbot for commit 8ba7cb6. Bugbot is set up for automated code reviews on this repo. Configure here.