fix(apiserver): hold the cutover until the new API server is ready - #5144
Merged
caseydavenport merged 4 commits intoAug 7, 2026
Merged
Conversation
The v3.projectcalico.org APIService is only repointed once the API server it would point at reports ready replicas.
An error reading the APIService or the Deployment is not evidence that the cutover is safe.
ControllerOptions has no Variant field on this branch and operatorv1.CalicoEnterprise does not exist here; both arrived with the product-variant refactor that this branch does not carry. Use the EnterpriseCRDExists idiom the neighbouring tests use. Refs: EV-6821
…is held Holding the cutover leaves the previous API server in service, but the reconcile still narrowed calico-apiserver-access-calico-crds to the new service account, and that binding is how an API server reads its own CRD-backed storage. v1.38 created that binding under the same name with the previous service account, so this is a subject rewrite rather than a rename, and no gate keyed on resource names can see it. On a direct upgrade the previous API server stopped serving the moment the new one was rendered: the aggregated API went down, the deprecated policy could no longer be deleted, the new pod stayed denied and never became ready, and the hold never cleared. Both service accounts are now subjects while the cutover is held, and the binding narrows to one when it clears. Refs: EV-6821
xiumozhan
added
enterprise
This was referenced Aug 6, 2026
caseydavenport
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cherry-pick of the master change to
release-v1.42, which ships CE 3.23. This is the build the live validation below was run on.Takes over #5139 and adds the change it needed to work on a live cluster. Casey's two commits are unchanged; the fourth is the fix, and the third only adapts his tests to this branch, which has no
ControllerOptions.Variant.#5139 holds the APIService repoint, and the removal of the old API server's namespace and RBAC, until the new API server reports ready, so the old one keeps serving and the installation controller's existing delete of the deprecated policy lands. On a direct upgrade that did not happen:
ClusterRoleBinding/calico-apiserver-access-calico-crdsis how an API server reads its own CRD-backed storage, v1.38 creates it under that same name with the previous service account, and v1.42 renders it with only the new one. The reconcile re-points it, the previous API server loses access while it is still the one in service, and the API goes down. Because the name never changes this is a subject rewrite rather than a rename, so holding deprecated resources by name cannot see it.That binding now carries both service accounts while the cutover is held, and narrows to one when it clears.
Background, including the failing run and the root cause: https://claude.ai/code/artifact/bba535da-e439-4edf-a5cd-93ec7d336247
Validated on this build: a direct CE 3.21.4 to 3.23.1 upgrade with the aggregated API probed every two seconds throughout, 158 of 158 samples up and no unavailability window. The same code without the binding fix was 371 of 388 samples down and never recovered.
Release Note
For PR author
make gen-files- no API changes.make gen-versions- no version changes.