fix(apiserver): hold the cutover until the new API server is ready - #5143
Merged
Conversation
xiumozhan
added
enterprise
3 tasks
caseydavenport
approved these changes
Aug 6, 2026
The v3.projectcalico.org APIService is only repointed once the API server it would point at reports ready replicas.
An error reading the APIService or the Deployment is not evidence that the cutover is safe.
…is held Holding the cutover leaves the previous API server in service, but the reconcile still narrowed calico-apiserver-access-calico-crds to the new service account, and that binding is how an API server reads its own CRD-backed storage. v1.38 created that binding under the same name with the previous service account, so this is a subject rewrite rather than a rename, and no gate keyed on resource names can see it. On a direct upgrade the previous API server stopped serving the moment the new one was rendered: the aggregated API went down, the deprecated policy could no longer be deleted, the new pod stayed denied and never became ready, and the hold never cleared. Both service accounts are now subjects while the cutover is held, and the binding narrows to one when it clears. Refs: EV-6821
xiumozhan
force-pushed
the
EV-6821-readiness-gate
branch
from
August 6, 2026 16:33
1a9ac01 to
dc05bca
Compare
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Takes over #5139 and adds the change it needed to work on a live cluster. Casey's two commits are unchanged; the third is the fix.
#5139 holds the APIService repoint, and the removal of the old API server's namespace and RBAC, until the new API server reports ready, so the old one keeps serving and the installation controller's existing delete of the deprecated policy lands. On a direct upgrade that did not happen:
ClusterRoleBinding/calico-apiserver-access-calico-crdsis how an API server reads its own CRD-backed storage, v1.38 creates it under that same name with the previous service account, and v1.42 renders it with only the new one. The reconcile re-points it, the previous API server loses access while it is still the one in service, and the API goes down. Because the name never changes this is a subject rewrite rather than a rename, so holding deprecated resources by name cannot see it.That binding now carries both service accounts while the cutover is held, and narrows to one when it clears.
Background, including the failing run and the root cause: https://claude.ai/code/artifact/bba535da-e439-4edf-a5cd-93ec7d336247
Validated on the release-v1.42 build of this change (see the companion branch): a direct CE 3.21.4 to 3.23.1 upgrade with the aggregated API probed every two seconds throughout, 158 of 158 samples up and no unavailability window. The same code without the binding fix was 371 of 388 samples down and never recovered.
Release Note
For PR author
make gen-files- no API changes.make gen-versions- no version changes.