🤖 Generated by the Agentic Engineer
Evidence
The prod deploy fails intermittently in 🔑 Stage Flux and consumer GHCR pull credential, evicting the PR from the merge queue:
##[error]Could not atomically claim and cordon Talos node autoscale-cx33-12ac9ae5b0b9effe; refusing to drain it.
cordon-claim: The request is invalid: the server rejected our request due to an error in our request
Measured on 2026-08-05 across ci.yaml runs: 4 merge_group failures interleaved with 8 successes on unchanged code —
| run |
PR |
time |
| 30997196586 |
pr-2970 |
10:23Z |
| 31003661088 |
pr-2971 |
11:58Z |
| 31006527422 |
pr-2970 |
12:38Z |
| 31011548385 |
pr-2971 |
13:43Z |
Intermittency on unchanged code rules out a malformed patch, which would fail every time.
Diagnosis
claim_node_cordon_ownership (scripts/refresh-flux-ghcr-auth.sh) builds an RFC 6902 patch whose first op is
{"op": "test", "path": "/metadata/resourceVersion", "value": "<version read earlier>"}A failed JSON Patch test op is exactly what the API server rejects as 422, which kubectl renders as "The request is invalid: the server rejected our request due to an error in our request".
That test asserts nothing whatsoever changed on the Node object, but a Node has several independent writers. Read live from the failing node's managedFields:
| manager |
operation |
last write |
kubelet |
Update status |
15:21:32Z (≈3 min before sampling) |
kube-controller-manager |
Update |
14:01:21Z |
Go-http-client |
Update |
14:01:40Z |
hcloud-cloud-controller-manager |
Update status |
03:17:32Z |
cilium-operator-generic |
Update status |
03:17:32Z |
So a routine kubelet status heartbeat — which has no bearing on whether the node is safe to cordon and drain — invalidates the claim and fails the whole deploy. There is no retry: claim_node_cordon_ownership is called once per node and returns non-zero on the first conflict.
The guard is simultaneously too strict (any unrelated status write aborts it) and only indirectly protective: resourceVersion is a proxy for "the invariants I validated still hold", not the invariants themselves.
Why it matters
This is the delivery path. Every eviction costs a full merge-queue cycle, and because the trigger is an unrelated background write, re-queuing without a fix is a coin flip — the exact "re-queue without root-causing" trap the contract warns about. It is not specific to the two dependency PRs that surfaced it; any PR can hit it.
Expected behaviour
The claim survives writes that do not affect drain safety, and still refuses when a genuinely conflicting change occurred.
Acceptance criteria
Size
Small–medium. Contained to claim_node_cordon_ownership and its two call sites, plus tests.
⚠️ This touches the path that drains prod nodes. The retry must re-validate rather than blindly re-patch — a retry that skips revalidation would convert a safe refusal into an unsafe drain, which is strictly worse than the current flake.
Evidence
The prod deploy fails intermittently in
🔑 Stage Flux and consumer GHCR pull credential, evicting the PR from the merge queue:Measured on 2026-08-05 across
ci.yamlruns: 4merge_groupfailures interleaved with 8 successes on unchanged code —Intermittency on unchanged code rules out a malformed patch, which would fail every time.
Diagnosis
claim_node_cordon_ownership(scripts/refresh-flux-ghcr-auth.sh) builds an RFC 6902 patch whose first op is{"op": "test", "path": "/metadata/resourceVersion", "value": "<version read earlier>"}A failed JSON Patch
testop is exactly what the API server rejects as 422, whichkubectlrenders as "The request is invalid: the server rejected our request due to an error in our request".That test asserts nothing whatsoever changed on the Node object, but a Node has several independent writers. Read live from the failing node's
managedFields:kubeletstatuskube-controller-managerGo-http-clienthcloud-cloud-controller-managerstatuscilium-operator-genericstatusSo a routine kubelet status heartbeat — which has no bearing on whether the node is safe to cordon and drain — invalidates the claim and fails the whole deploy. There is no retry:
claim_node_cordon_ownershipis called once per node and returns non-zero on the first conflict.The guard is simultaneously too strict (any unrelated status write aborts it) and only indirectly protective:
resourceVersionis a proxy for "the invariants I validated still hold", not the invariants themselves.Why it matters
This is the delivery path. Every eviction costs a full merge-queue cycle, and because the trigger is an unrelated background write, re-queuing without a fix is a coin flip — the exact "re-queue without root-causing" trap the contract warns about. It is not specific to the two dependency PRs that surfaced it; any PR can hit it.
Expected behaviour
The claim survives writes that do not affect drain safety, and still refuses when a genuinely conflicting change occurred.
Acceptance criteria
testops to the actual invariants —metadata.uid,spec.unschedulable== capturedwas_cordoned,spec.taints== capturedinitial_taints, and the owner annotation being unclaimed/ours — or keepresourceVersionbut re-read and re-validate on retry.scripts/tests/refresh-flux-ghcr-auth/using the existing fakekubectl: a first-attempt 422 followed by success must fail before the fix and pass after, and a persistent conflict must still refuse to drain.Size
Small–medium. Contained to
claim_node_cordon_ownershipand its two call sites, plus tests.