feat(cilium): release the homogeneous-device rollout gate - #3035
Merged
Conversation
All nine Cilium agents now run the widened device set ("en+ eth+") at the
current DaemonSet generation, verified per node: each kept a private
primary address (10.0.1.x) and the fleet reports zero stale pods and zero
non-private InternalIPs.
Remove the temporary operator-stepping overrides the component carried for
the rollout — `updateStrategy: OnDelete` and the Helm
`spec.upgrade.disableWait` handoff — so the DaemonSet returns to the
chart's normal rolling update. Every agent is already at the current
template, so this rolls nothing.
Removing the OnDelete override is what releases the deploy gate: the guard
computes it as the component being referenced AND the component carrying
`type: OnDelete`. The component reference stays, so the widened device set
stays. `ksail cluster update` therefore resumes on the next deploy, ending
the 14-day machine-config sync suppression that had reached its hard-fail
bound and was blocking every prod deploy.
Drop the activation marker with it, and describe the component's current
state rather than the rollout that produced it.
Fixes #3028
Contributor
✅
|
…ease Releasing the rollout gate re-enables `ksail cluster update`, and the gate suppressed that sync long enough for Renovate to queue two upgrades behind it: Kubernetes v1.36.3 and Talos v1.13.7. With both pins forward the first deploy would roll one fleet-wide upgrade carrying the OS and Kubernetes at once, across eight of nine nodes. This deployment rolls those separately. Hold Kubernetes at v1.36.2 — the version eight of the nine nodes are already running — so the resumed sync performs the Talos OS upgrade alone. Renovate tracks this pin and does not automerge it, so its existing bump PR is the reviewed second step once the OS upgrade has settled. prod-worker-1 already runs v1.36.3, so it is the one node this pin moves backward by a patch release before the follow-up returns it.
Three guards pinned the transitional state, so completing the rollout made them red. Each now pins the completed state. The activation test required a top-level `updateStrategy` carrying `type: OnDelete` and a `disableWait: true` upgrade handoff. It now asserts their ABSENCE rather than dropping the assertions: a silently reintroduced gate would suppress the pipeline's only Talos machine-config sync again, which is the failure that took prod deploys down for fourteen days. The device-selection guards are untouched — they protect the datapath whatever the rollout state. The autoscaler-gate test copied the live component into its fixture and so inherited whatever gate state the repository shipped (platform#3031). With the overrides removed the fixture started inactive and its active-gate assertions failed for the wrong reason. It now constructs the active gate it exercises, so it keeps covering the gate no matter what the repository currently ships. The EKS authorization surface fingerprint moved because the rendered Cilium HelmRelease legitimately changed. Re-approved with the measurement the constant's own convention requires: all five roots rendered from both trees and diffed — four byte-identical, membership unchanged, and exactly four lines moving, all of them rollout mechanics inside one HelmRelease, none reaching an identity, binding, policy document, or service account. The value is CI's, because the local toolchain is a refused renderer.
Contributor
Author
@coderabbitai review |
Contributor
|
|
Contributor
📝 WalkthroughWalkthroughThe production Cilium rollout gate is released. Normal rolling updates and Talos machine-config synchronization resume. The deployment action restores autoscaling after Flux revision readiness. Tests verify released and active-gate states. Kubernetes is pinned to ChangesCilium rollout completion
Release baseline updates
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant DeployAction
participant Flux
participant RolloutGuard
participant Talos
DeployAction->>Flux: wait for exact published revision
Flux-->>DeployAction: return revision Ready
DeployAction->>RolloutGuard: invoke --after-revision-ready
RolloutGuard->>RolloutGuard: restore owned autoscaler suspension
DeployAction->>Talos: run cluster update
Possibly related issues
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
CodeRabbit reports @codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a17998a316
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
AGENTS.md is the repository's first-reference operational guide, and it still stated that the temporary component sets `disableWait` and `OnDelete`, that Cilium pods remain operator-stepped, and that the deploy skips `cluster update`. With the gate released all three are false, and a stale description here misleads exactly when it is read: during maintenance and incident response. State that no gate is active and that every deploy runs its Talos machine-config sync. The guard's behaviour is retained as conditional machinery for the next staged rollout, with the condition that actually governs it — the component referenced AND carrying `type: OnDelete` — written down rather than implied.
Code Coverage OverviewLanguages: Go Go / code-coverage/goThe overall coverage in commit c8d6692 in the Show a code coverage summary of the most covered files.
Updated |
Contributor
Author
The AGENTS.md finding from the last round is fixed and its thread resolved. New head, so re-reviewing. @coderabbitai review |
Contributor
|
✅ Action performedReview finished.
|
Contributor
Author
The previous @coderabbitai full review |
Contributor
|
|
Contributor
Author
CodeRabbit reports Since the last Codex pass, the only change is the AGENTS.md fix for its own finding — the gate release and the Kubernetes hold are unchanged. @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8da566116a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Removing the numbered runbook left a dangling reference: report-cilium-rollout-gate-suppression.sh tells an operator to "follow step 4-5 of the runbook" in this component, and two more messages point at "the component runbook". Those fire exactly when the gate's 7-day warning or 14-day failure does — the situation that needs the procedure most, and the one prod was in this morning. Restore it as forward-looking guidance for the next change to this device set rather than a record of the completed rollout, since the gate machinery is still here and reintroducing the two overrides is what re-arms it. Steps 4 and 5 remain the release and post-release verification the reporter names. Comment-only: the rendered controllers root is byte-identical, so the approved authorization surface fingerprint is untouched.
Contributor
Author
The runbook finding is fixed and its thread resolved. New head, so re-reviewing. @codex review |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Contributor
Author
ReadinessProgrammatically tested. 26/26 checks green. The four gate-aware guards pass with the completed state pinned rather than deleted, and the EKS authorization surface was re-approved from CI's own render after diffing all five roots from both trees — four byte-identical, membership unchanged, four lines moving, none of them an identity, binding, policy document, or service account. The runbook restoration is comment-only and the controllers render is byte-identical to that approved measurement. Reviewed. Codex green at this exact head ( Exercised. The rollout this releases was executed and verified node by node: all nine agents at DaemonSet generation 26, zero stale, and zero non-private CiliumNode InternalIPs — the property the gate existed to protect. I confirmed the guard's release condition directly in Not exercised: that the next deploy actually resumes What merging does: the |
devantler
marked this pull request as ready for review
August 9, 2026 17:59
The deploy that released the gate failed at its own config sync: waiting for cluster-autoscaler ... rollout: failed to poll for readiness: ... client rate limiter Wait returned an error: context deadline exceeded The rate limiter is the symptom, not the cause. KSail's readiness check returns not-ready for `Status.Replicas == 0` (pkg/k8s/readiness/deployment.go), so it polls to its deadline. The Deployment it was waiting on was held at zero replicas by the rollout gate's own suspension: releasing the gate re-enables `ksail cluster update`, but the suspension was only restored in the post-deploy phase — after the step that waits on it. Restore in a new phase between the exact-revision proof and cluster update. Both bounds matter. Not earlier: autoscaling may not resume before the safe artifact is deployed, which the pre-publish assertion pins. Not later: the step that waits on the autoscaler runs before the post-deploy phase. An active gate stays suspended in every phase, unchanged. The post-deploy restore stays as the failure-path net, and is a no-op once this phase has cleared the ownership marker.
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Aug 9, 2026
Contributor
Author
New head: the first merge-queue attempt failed at @coderabbitai review |
Contributor
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== HEAD and changed files =='
git rev-parse HEAD
git diff --name-only HEAD^ HEAD
printf '%s\n' '== Guard script outline and relevant sections =='
ast-grep outline scripts/guard-cilium-homogeneous-device-rollout.sh --items all || true
rg -n -C 8 -- '--after-revision-ready|restore_autoscaler_if_owned|suspend_autoscaler|gate_active|usage' \
scripts/guard-cilium-homogeneous-device-rollout.sh
printf '%s\n' '== Deploy action ordering =='
ast-grep outline .github/actions/deploy-prod/action.yml --items all || true
rg -n -C 12 -- 'guard-cilium-homogeneous-device-rollout|revision|ksail cluster update|cluster-autoscaler' \
.github/actions/deploy-prod/action.yml
printf '%s\n' '== Focused test assertions =='
ast-grep outline scripts/tests/test-cilium-homogeneous-devices-autoscaler-gate.sh --items all || true
rg -n -C 8 -- 'after-revision-ready|cluster update|revision|restore|autoscaler|guard' \
scripts/tests/test-cilium-homogeneous-devices-autoscaler-gate.sh
printf '%s\n' '== Exact patch for relevant files =='
git diff HEAD^ HEAD -- \
.github/actions/deploy-prod/action.yml \
scripts/guard-cilium-homogeneous-device-rollout.sh \
scripts/tests/test-cilium-homogeneous-devices-autoscaler-gate.shLength of output: 40699 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== Restore and readiness helper implementation =='
sed -n '294,370p' scripts/guard-cilium-homogeneous-device-rollout.sh
printf '%s\n' '== Full guard phase dispatch =='
sed -n '385,440p' scripts/guard-cilium-homogeneous-device-rollout.sh
printf '%s\n' '== Deploy action conditions around revision, restore, and cluster update =='
sed -n '206,265p' .github/actions/deploy-prod/action.yml
printf '%s\n' '== Focused test fixture: kubectl scale/status emulation =='
sed -n '130,275p' scripts/tests/test-cilium-homogeneous-devices-autoscaler-gate.sh
printf '%s\n' '== Search for the guard phases at every call site =='
rg -n -C 4 --glob '*.yml' --glob '*.yaml' --glob '*.sh' \
'guard-cilium-homogeneous-device-rollout\.sh (?:--before-publish|--after-revision-ready|--after-deploy)' \
.Length of output: 17073
The new phase restores only a suspension owned by the released gate. It runs after the exact Flux revision is Ready and before An active The focused test covers the required ordering, restoration before cluster update, ownership-marker removal, and the no-op post-deploy restore path. I did not execute repository scripts because the sandbox policy prohibits executing repository files.
|
Contributor
Author
CodeRabbit's commit status reads @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c8d669236f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+114
to
+115
| # the post-deploy guard restores the recorded autoscaler replica count. | ||
| # Verify the FIRST autoscaler-created node afterwards like a stepped one — |
There was a problem hiding this comment.
For the next stepped rollout, this release instruction still says the post-deploy guard restores the autoscaler after ksail cluster update, but the new --after-revision-ready phase was added precisely because KSail waits indefinitely on the zero-replica autoscaler during that update. Following this canonical runbook would therefore reproduce the failed release ordering; direct operators to run the new phase after the exact-revision readiness proof and before cluster update.
AGENTS.md reference: AGENTS.md:L3-L5
Useful? React with 👍 / 👎.
| shell: bash | ||
| env: | ||
| HCLOUD_TOKEN: ${{ inputs.hcloud-token }} | ||
| run: ./scripts/guard-cilium-homogeneous-device-rollout.sh --after-revision-ready |
There was a problem hiding this comment.
When a future gate is released while the cluster is below its node limit and pending or CapacityBuffer demand exists, this call starts Cluster Autoscaler before the following cluster update refreshes its machine template. Because the gate may have suppressed that sync for days, the autoscaler can provision a server from stale Talos or pull-credential data in this gap, while the pre-publish bridge verified only nodes that already existed; refresh the template before allowing scale-up, or keep scale-up disabled while satisfying KSail's readiness check.
AGENTS.md reference: AGENTS.md:L187-L188
Useful? React with 👍 / 👎.
| # the very deploy that releases the gate. Restoring in --after-deploy is too | ||
| # late for that wait, and restoring in --before-publish would do it before the | ||
| # safe artifact is deployed, which the gate deliberately forbids. | ||
| restore_autoscaler_if_owned |
There was a problem hiding this comment.
If the autoscaler was already manually scaled to zero when a gate was activated, suspend_autoscaler accepts and records previous_replicas=0; this release phase then “restores” zero and clears the ownership marker. The immediately following cluster update consequently sees the same zero-replica Deployment that this patch says KSail treats as never-ready, reproducing the timeout while losing the state needed for a retry. Preserve the manual suspension by skipping the update in this case, or avoid treating zero as a releasable owned count.
AGENTS.md reference: AGENTS.md:L210-L210
Useful? React with 👍 / 👎.
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Cilium rollout gate reached its 14-day hard-fail bound and has been failing every prod deploy. It suppresses
ksail cluster update— the pipeline's only Talos machine-config sync — so machine config in Git has been diverging from the nodes, and #2922/#2938 sat blocked behind it.The rollout it was gating is now finished. All nine agents run the widened device set at the current DaemonSet generation, stepped one node at a time and verified individually: every node kept a private primary address, and the fleet reports zero stale pods and zero non-private InternalIPs.
What
1. Releases the gate. Removes the two temporary overrides the component carried only for the operator-stepped rollout —
updateStrategy: OnDeleteand the Helmspec.upgrade.disableWaithandoff. The DaemonSet returns to the chart's normal rolling update; every agent is already at the current template, so this rolls nothing. RemovingOnDeleteis precisely what releases the gate (the guard computes it as component referenced AND component carriestype: OnDelete). The component reference stays, so the widened device set stays.2. Holds Kubernetes at the running version, so the resumed sync upgrades the OS alone. The gate suppressed machine-config sync long enough for Renovate to queue two upgrades behind it — Kubernetes v1.36.3 and Talos v1.13.7. With both pins forward, the first deploy would roll one fleet-wide upgrade carrying both, across 8 of 9 nodes. This deployment rolls Kubernetes and OS upgrades separately, so Kubernetes is held at v1.36.2 and Renovate's existing bump PR becomes the reviewed second step.
What merging this actually does
The next deploy performs an in-place rolling Talos upgrade to v1.13.7 — 8 of 9 nodes, one at a time, each rebooting. Worth knowing before it runs:
instance-managerpods (the classic cause of a stalled Talos drain) and 4 single-replica CNPG primaries (backstage,coroot,umami,wedding), which take brief downtime as their node reboots.prod-worker-1already runs Kubernetes v1.36.3, so the hold moves that one node back a patch release until the follow-up returns it.Fixes #3028
🤖 Generated with Claude Code
Summary by CodeRabbit
Deployment
Configuration
Documentation
Tests