fix(update): only offer Kubernetes upgrades the distribution has promoted - #6487
Conversation
…oted An OCI registry serves an image the moment its tag is pushed. k3s pushes rancher/k3s:<tag> at tag time and promotes the release to its channels only later, so the registry tag list is not a list of released versions. Version discovery treated every tag as an upgrade candidate, so within hours of k3s pushing v1.36.3-k3s1 -- which k3s had not promoted, and still has not -- 'ksail cluster update' recreated K3s clusters onto it with no configuration change from the user. Cluster recreation is destructive, and every K3s System Test failed on the same assertion. Constrain discovery to versions the distribution has actually promoted, read from k3s' own channel index. Images with no registered promotion source pass through unchanged, and a broken source fails closed rather than falling back to raw tags. Fixes #6486
Verification recordRED/GREEN with a control on the ablation. With
Restored, all tests pass. Exercised against the real endpoint, not a fixture: Cross-checked against three independent sources agreeing that End-to-end proof is this PR's own K3s System Tests. They are the jobs that were failing on the assertion; passing here is the behavioural confirmation, so I am waiting on them rather than claiming the fix works from the unit tests. Local lint note — pre-existing, not introduced here. |
✅MegaLinter analysis: Success✅ Linters with no issuesactionlint, bash-exec, git_diff, hadolint, jscpd, jsonlint, lychee, markdown-table-formatter, markdownlint, prettier, prettier, shellcheck, shfmt, stylelint, syft, trivy-sbom, trufflehog, v8r, v8r, yamllint Notices📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining See detailed reports in MegaLinter artifacts
|
@coderabbitai review |
|
✅ Action performedReview finished.
|
📝 WalkthroughWalkthroughThe change adds a K3s channel source that retrieves promoted release versions and converts them to OCI tags. A promotion-aware resolver filters registry versions for registered image sources and propagates channel errors. Unknown image sources retain their base resolver results. Cluster reconciliation and version-drift calculations now use the promotion-aware resolver. Tests cover promoted and unpromoted K3s releases, passthrough behavior, error handling, channel parsing, deduplication, and empty responses. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/svc/versionresolver/k3schannel.go`:
- Around line 92-98: The K3s channel source currently treats every channel’s
Latest value as promotion evidence; update K3sChannelSource in
pkg/svc/versionresolver/k3schannel.go (lines 92-98) to include only
stable-channel releases, excluding tags present solely in latest or
minor-version channels. Extend the promotion test in
pkg/svc/versionresolver/promotion_test.go (lines 169-174) with a non-stable-only
tag and assert that K3sChannelSource excludes it.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 36d3adfa-07e8-4e2c-b260-1cd0003d911e
📒 Files selected for processing (5)
pkg/cli/cmd/cluster/orchestrator.gopkg/cli/cmd/cluster/version_drift.gopkg/svc/versionresolver/k3schannel.gopkg/svc/versionresolver/promotion.gopkg/svc/versionresolver/promotion_test.go
📜 Review details
⏰ Context from checks skipped due to timeout. (5)
- GitHub Check: 🧪 System Test (Docker) (Talos, Docker, true, --name system-test-cluster-with-image-verificatio...
- GitHub Check: 🧪 System Test (Docker) (Talos, Docker, true, --gitops-engine Flux --local-registry ghcr.io/dev...
- GitHub Check: 🧪 System Test (Docker) (Talos, Docker, true)
- GitHub Check: 🧪 System Test (Docker) (Talos, Docker, true, --name system-test-cluster --cni Cilium --csi Ena...
- GitHub Check: 🧪 System Test (Docker) (Talos, Docker, true, --cni Calico --csi Disabled --load-balancer Disab...
🧰 Additional context used
📓 Path-based instructions (4)
**/*.go
📄 CodeRabbit inference engine (AGENTS.md)
**/*.go: Use Go 1.26.1 or newer, matching the version declared ingo.mod.
All user-supplied file path arguments in CLI commands must be canonicalized withfsutil.EvalCanonicalPathbefore use; create parent directories first for new output paths.
Usefsutil.ReadFileSafefor constrained file reads instead of reimplementing path-containment checks.
Do not manually register MCP or Copilot tool handlers; runnable Cobra commands are exposed through automatic generation inpkg/toolgen.
Use a typedexperimentalfield inksail.yamlfor configuration-gated behavior that is not an entire command; regenerate the schema and CRD.
Graduate validated experimental features by deleting the singleGuardcall; do not retain unnecessary experimental scaffolding.
Run formatting and linting withgolangci-lint run --fixandgolangci-lint run --timeout 5m; validate withgo buildandgo test ./....
Files:
pkg/cli/cmd/cluster/orchestrator.gopkg/svc/versionresolver/k3schannel.gopkg/cli/cmd/cluster/version_drift.gopkg/svc/versionresolver/promotion_test.gopkg/svc/versionresolver/promotion.go
pkg/cli/**/*.go
📄 CodeRabbit inference engine (AGENTS.md)
pkg/cli/**/*.go: New not-yet-stable commands must be wrapped withexperimental.Guard(cmd), remain disabled by default, and require the global--experimentalflag.
Test experimental commands in both states: enabled with--experimentaland disabled withexperimental.ErrDisabled.
Files:
pkg/cli/cmd/cluster/orchestrator.gopkg/cli/cmd/cluster/version_drift.go
**/*.{go,yaml,yml,md,mdx,ts,tsx,json}
📄 CodeRabbit inference engine (AGENTS.md)
Generated files must not be hand-edited; run
make generateas the canonical regeneration command.
Files:
pkg/cli/cmd/cluster/orchestrator.gopkg/svc/versionresolver/k3schannel.gopkg/cli/cmd/cluster/version_drift.gopkg/svc/versionresolver/promotion_test.gopkg/svc/versionresolver/promotion.go
**/*_test.go
📄 CodeRabbit inference engine (AGENTS.md)
Add regression tests for confident bug fixes and run flaky-test candidates repeatedly with
go test -run <T> -count=10 ./....
Files:
pkg/svc/versionresolver/promotion_test.go
🧠 Learnings (1)
📚 Learning: 2026-08-02T19:26:41.922Z
Learnt from: devantler
Repo: devantler-tech/ksail PR: 6434
File: pkg/cli/clusterapi/eks_create_identity_test.go:0-0
Timestamp: 2026-08-02T19:26:41.922Z
Learning: In Go tests using Testify v1.11.1, do not flag require.NoError(t, err) inside an Eventually condition solely because the condition may run in another goroutine: require calls t.Errorf before FailNow, and t.Errorf marks the outer test as failed. However, prefer decomposing Eventually conditions so service errors, missing resources, and state or phase mismatches are reported separately for clearer diagnostics.
Applied to files:
pkg/svc/versionresolver/promotion_test.go
🔇 Additional comments (5)
pkg/svc/versionresolver/k3schannel.go (1)
1-90: LGTM!Also applies to: 100-115
pkg/svc/versionresolver/promotion_test.go (1)
1-162: LGTM!Also applies to: 176-216
pkg/svc/versionresolver/promotion.go (1)
1-91: LGTM!pkg/cli/cmd/cluster/orchestrator.go (1)
179-179: LGTM!pkg/cli/cmd/cluster/version_drift.go (1)
54-54: LGTM!
k3s publishes "testing" channels alongside the stable ones, and their latest names a release candidate rather than a release. Live index on 2026-08-05: both testing and v1.18-testing point at v1.36.2's stream predecessor v1.18.2-rc3+k3s1. Unioning every channel's latest therefore admitted that rc as promotion evidence, so a cluster on v1.18.x could be offered an upgrade onto a release candidate — the same class of unreleased-version upgrade this source was added to prevent. Filter on the version's own semver pre-release segment rather than the channel name, so a differently-named pre-release channel is caught too. Minor-stream channels (v1.18 -> v1.18.20+k3s1, v1.34 -> v1.34.9+k3s1) name real releases and are deliberately still admitted; restricting to the stable channel alone would strand every cluster not on the newest minor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai review |
|
✅ Action performedReview finished.
|
Readiness record — head
|
Correction — the pre-release fix is defense-in-depth, not a live user-facing bugI checked reachability after the fact and I overstated the CodeRabbit finding. Correcting it before it informs a release decision. What is still exactly as described: the original defect (#6486). What I got wrong: I implied the release-candidate leak was equally reachable. It is not. So CodeRabbit's finding is accurate about The change stays, on its merits rather than on urgency: an allow-list named Net effect on merge urgency: unchanged — driven by #6486, which is real and destructive. |
This bug is now reddening unrelated PRs across the repoEvidence gathered while this PR was waiting on CI. Two unrelated open PRs fail on the same K3s
Both fail identically at
In #6489's run the cluster was created with That makes a fairly clean natural experiment: three PRs, three different diffs, one shared failure, Unrelated to the above: this PR's own one red leg was the |
1 participant
Why
ksail cluster updatewas recreating people's K3s clusters onto a Kubernetes version that k3s had not released.k3s publishes its container image the moment a release is tagged, but only promotes that release to its update channels later — sometimes much later. KSail read the image registry's tag list as though it were the list of released versions, so a tag appearing upstream was immediately treated as an available upgrade. On 2026-08-04 k3s pushed
v1.36.3-k3s1without promoting it, and from that momentksail cluster updatedeleted and recreated any K3s cluster it was pointed at — no configuration change from the user, and a destructive operation. It also turned every K3s System Test red.What
Version discovery now only offers versions the distribution has actually promoted, read from k3s' own release-channel index — the same source the k3s installer uses. Distributions with no promotion source behave exactly as before, and if the channel lookup fails KSail declines to upgrade rather than falling back to raw registry tags, so the check cannot quietly stop protecting anything.
Verified against the live k3s index:
v1.36.2-k3s1(promoted) is offered,v1.36.3-k3s1(not promoted) is not.Fixes #6486