feat: adopt the shared expand-migrate-contract upgrade flow for glance#736
Merged
Conversation
The expand-migrate-contract database upgrade flow is being lifted out of the keystone operator into internal/common so glance becomes its second consumer. Its phase enum must live where the shared flow can reference it, so move UpgradePhase and its four constants into internal/common/types beside the other shared database vocabulary. Keystone keeps a type alias (UpgradePhase = commonv1.UpgradePhase) and a re-exported constant block, mirroring the GatewaySpec alias pattern, so every existing keystonev1alpha1.UpgradePhase reference and bare UpgradePhase* constant keeps compiling unchanged. The generated CRD is byte-identical: the schema description is driven by the status field godoc, not the moved type godoc, and the enum marker travels with the type, so make generate/manifests/sync-crds produces no diff. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Port keystone's private four-phase database upgrade machine (Expanding -> Migrating -> RollingUpdate -> Contracting) into internal/common/database as a parameterized flow so glance becomes its second consumer. The flow generalizes over a single seam, the spec-side release string (keystone's spec.image.tag, glance's spec.openStackRelease), supplied through UpgradeFlowParams alongside the owner CR, the condition vocabulary, the phase Job builder, and the terminal-metric callback. The exported surface mirrors keystone's semantics exactly: IsUpgrade, InitiateUpgrade, ReconcileUpgrade, AbortUpgrade, and CompleteRollingUpdate emit the same condition reasons, event reasons, and message wording as today, so the eventual keystone rewiring is behavior-preserving. Two service-specific phrasings are neutralized: "Image tag changed" becomes "Spec release changed" and "spec.image.tag reverted" becomes "spec release reverted"; the release-value substrings and the reasons keystone asserts on are unchanged. The expand-migrate-contract condition reasons move out of their keystone-private home into this package next to the steady-state Reason* set, and the package doc gains a bullet for the upgrade flow. Keystone's private machine stays in place and compiling; a later commit deletes it and rewires keystone onto this flow. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Delete keystone's private expand-migrate-contract machine and route the DatabaseReady upgrade path through internal/common/database, now that the shared flow added in the previous commit carries the same logic. reconcileDatabase delegates upgrade initiation, the active-phase dispatch, and the abort to database.InitiateUpgrade and database.ReconcileUpgrade; reconcileDeployment drives the RollingUpdate-to-Contracting transition through database.CompleteRollingUpdate. A new upgradeFlowParams binds the keystone-specific inputs: the owner CR, the DatabaseReady vocabulary, spec.image.tag as the requested release, the per-phase Job builders, and the terminal-metric callback. Behavior is frozen. The condition reasons, lifecycle events, db_sync metric suffixes, and the #468 abort semantics are unchanged, and the keystone unit, envtest, and e2e suites pass with no test edits. Two seams stay keystone-local to keep that behavior frozen: - The target-changed guard remains in reconcileDatabase so its error keeps the "image tag changed during active upgrade" prose the unit test pins; the shared flow phrases that guard generically as "spec release". - Only the seven upgrade reason constants keystone still emits locally or the tests reference by name are aliased to the shared database.Reason* set; aliasing the rest would leave unused constants the linter rejects. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Glance is about to adopt the shared expand-migrate-contract database upgrade flow, which tracks the in-flight phase on the owning CR via a status pointer. Add the status.upgradePhase field to GlanceStatus so the shared flow can persist Expanding -> Migrating -> RollingUpdate -> Contracting and clear it when no upgrade is in flight, mirroring KeystoneStatus.UpgradePhase. The field reuses the shared commonv1.UpgradePhase type, so the CRD gains the same four-value enum keystone already carries. Regenerate the two Glance CRD copies (config/crd/bases and the Helm chart) accordingly; no other operator's CRD or deepcopy output changes. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Glance becomes the second consumer of the shared expand-migrate-contract database-upgrade flow (internal/common/database), replacing its bespoke single-pass release handling. A release transition (a non-patch change from the installed release, e.g. 2025.2 -> 2026.1) now walks Expanding -> Migrating -> RollingUpdate -> Contracting with glance-manage db expand|migrate|contract phase Jobs, driven identically to keystone; status.upgradePhase tracks the active phase and TargetRelease is owned by the flow (set on initiate, cleared on completion or abort) instead of being stamped every pass. Fresh installs and patch bumps stay on the single-pass glance-manage db sync path. The accepted set of transitions is unchanged (patch-only or a single sequential step, no downgrade); only the failure reporting changes. The bespoke InvalidReleaseTransition reason and its non-zero-requeue short-circuit are gone: an invalid transition now surfaces through the shared vocabulary (VersionParseError, DowngradeNotSupported, UpgradePathInvalid) as a returned error, so the controller backs off with exponential retry exactly like keystone. The Deployment reconcile flips RollingUpdate -> Contracting once the rollout is ready, without stamping the endpoint on the flip pass. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add an integration test that drives a full glance release upgrade (2025.2 -> 2026.1) through the shared expand-migrate-contract flow against envtest, mirroring keystone's upgrade-cycle test. envtest runs no Job or Deployment controllers, so every phase Job completion and Deployment readiness is simulated. The test locks four properties no unit test observes together: - the Expanding -> Migrating -> RollingUpdate -> Contracting phase walk, with each glance-manage db expand|migrate|contract phase Job carrying the target-release image; - the rollout-gated contract flip: the phase stays RollingUpdate until the re-imaged Deployment reports ready, asserted with a short Consistently before simulating the rollout; - the eventlet -> uWSGI launch-command switch the Deployment template takes on the release bump; - the post-upgrade steady-state db-sync Job re-running with the new image on pod-spec-hash drift. Only the glance integration test file changes. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Cover the 2025.2 -> 2026.1 cross-release upgrade end-to-end, making glance the second consumer of the shared expand-migrate-contract database upgrade flow exercised by a chainsaw e2e suite. The suite deploys glance-release-upgrade at 2025.2 with a default S3 backend, asserts the pre-upgrade steady state (Ready/AllReady, installedRelease 2025.2, and the eventlet glance-api launch command), then patches spec.openStackRelease and spec.image.tag to 2026.1. It then asserts the upgrade drains: Ready/AllReady with installedRelease promoted to 2026.1, status.upgradePhase cleared, the db-expand, db-migrate and db-contract phase Jobs left in place, the Deployment flipped to the uWSGI launch command on the :2026.1 image with the rollout converged, and the steady-state db-sync Job re-run on the :2026.1 image (its db load_metadefs pass loads the 2026.1 metadata definitions). The suite is auto-discovered by the per-operator chainsaw run, so no CI wiring changes. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Glance now shares Keystone's expand-migrate-contract upgrade machine (internal/common/database), so a spec.openStackRelease bump with the image in lockstep walks Expanding -> Migrating -> RollingUpdate -> Contracting instead of a single-pass db sync. The reference docs and the prepare-new-service skill still described the old single-pass design, which no longer matches the code. Add a Glance Upgrade Flow reference page mirroring the Keystone one: trigger, version format, status fields, the phase table with the exact glance-manage commands and Job names, the eventlet -> uWSGI launch-mode flip during RollingUpdate, the DatabaseReady condition reasons and events, abort semantics, the image/release lockstep contract with its digest-pinning nuance, and the post-upgrade metadefs reload. Update the glance CRD, reconciler, events, and index pages: drop the "no upgradePhase field" and "reverse window is not closed" claims and every InvalidReleaseTransition mention, add the upgradePhase status row and the new condition reasons and events, and link the new page from the sidebar and reference lists. Note the shared machinery on the Keystone upgrade-flow page. In prepare-new-service, extend the Database profiling bullet to ask which zero-downtime upgrade mechanism the manage tool supports and to adopt the shared database.ReconcileUpgrade flow, and drop expand-migrate-contract from the keystone-specific classification now that it is shared. On-behalf-of: @SAP Assisted-by: Claude:claude-fable-5 Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
berendt
force-pushed
the
implement/issue-734-shared-upgrade-flow
branch
from
July 24, 2026 05:35
aec287b to
7056e72
Compare
berendt
marked this pull request as ready for review
July 24, 2026 06:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Glance's release upgrades ran a single
glance-manage db syncbefore the workload rolled — anddb syncexecutes expand → migrate → contract in one pass, so the destructive contract half ran while previous-release pods were still serving. Keystone already solved this with a four-phase upgrade machine (Expanding → Migrating → RollingUpdate → Contracting) that was keystone-private. This branch extracts that machine intointernal/common/database, rewires Keystone onto it with frozen behavior, and adopts it for Glance as the second consumer, then corrects the onboarding skill and docs so the next database-backed service decides this at onboarding time.Closes #734
Change set (commit order)
The eight commits are ordered so each is independently compiling — the shared flow lands first while keystone's private machine still works, then keystone is cut over, then glance adopts it, then tests and docs.
7f0dd9d4refactor: move the UpgradePhase type into the shared commonv1 packageMoves
UpgradePhaseand its four constants (Expanding/Migrating/RollingUpdate/Contracting) fromkeystone_types.gointointernal/common/types/types.go, so the shared flow can reference them. Keystone keeps atype UpgradePhase = commonv1.UpgradePhasealias plus a re-exported constant block (theGatewaySpecalias pattern), so every existing reference compiles unchanged. The generated CRD is byte-identical — the enum marker travels with the type and the schema description comes from the status-field godoc.a6f37258feat: add the shared expand-migrate-contract upgrade flowPorts keystone's private four-phase machine into
internal/common/database/upgrade.go, parameterized over a single seam — the spec-side release string — viaUpgradeFlowParams(owner CR, condition vocabulary,SpecRelease,BuildPhaseJobclosure, terminal-metric callback). ExportsIsUpgrade,InitiateUpgrade,ReconcileUpgrade,AbortUpgrade, andCompleteRollingUpdatewith keystone's exact condition/event reasons and message wording. Two service-specific phrasings are neutralized ("Image tag changed" → "Spec release changed"). Keystone's private machine stays in place and compiling — a later commit deletes it. Ships with the shared unit suite (upgrade_test.go, +538) covering phase transitions, abort, target-changed, unknown phase, and the parse/downgrade/sequential rejections.d0ac12d0refactor: rewire keystone onto the shared database upgrade flowDeletes
reconcile_upgrade.go(−316) and routesreconcileDatabase/reconcileDeploymentthroughInitiateUpgrade/ReconcileUpgrade/AbortUpgrade/CompleteRollingUpdate.buildExpandJob/buildMigrateJob/buildContractJobbecome theBuildPhaseJobclosure. Behavior is frozen — reasons, events,db_syncmetric suffixes, and the abort semantics are unchanged; the keystone unit/envtest/e2e suites pass with no test edits. Two seams stay keystone-local deliberately (documented in the commit): the target-changed guard keeps its "image tag changed" prose that a unit test pins, and only the reason constants keystone still emits or tests reference by name are aliased (aliasing the rest would leave unused constants the linter rejects).a623a085feat: add status.upgradePhase to the Glance CRDAdds
UpgradePhase commonv1.UpgradePhasetoGlanceStatusand regenerates the two Glance CRD copies (config/crd/bases+ the Helm chart). Reuses the shared type, so the Glance CRD gains the same four-value enum keystone already carries; no other operator's CRD or deepcopy output changes.a64a8be4feat: adopt the shared upgrade flow for glance release transitionsGlance becomes the second consumer. A release transition (non-patch step, e.g. 2025.2 → 2026.1) now walks the four phases with
glance-manage db expand|migrate|contractphase Jobs, driven identically to keystone;status.upgradePhasetracks the active phase andTargetReleaseis owned by the flow. Fresh installs and patch bumps stay on the single-passdb syncpath. Documented behavior change in failure reporting only: the bespokeInvalidReleaseTransitionreason and its non-zero-requeue short-circuit are gone — an invalid transition now surfaces through the shared vocabulary (VersionParseError,DowngradeNotSupported,UpgradePathInvalid) as a returned error, so the controller backs off with exponential retry exactly like keystone. The accepted set of transitions is unchanged (patch-only or a single sequential step, no downgrade). Deployment reconcile flips RollingUpdate → Contracting once the rollout is ready, and the eventlet → uWSGI launch-mode switch derives fromspec.openStackRelease.e41bec21test: cover the glance upgrade phase walk in envtestAdds a glance integration test driving a full 2025.2 → 2026.1 upgrade against envtest (which runs no Job/Deployment controllers, so completions are simulated), mirroring keystone's upgrade-cycle test. Locks four properties together: the Expanding → Migrating → RollingUpdate → Contracting walk with each phase Job carrying the target image; the rollout-gated contract flip (stays RollingUpdate until the re-imaged Deployment reports ready, asserted with a
Consistently); the eventlet → uWSGI launch-command switch; and the post-upgrade steady-state db-sync re-run on pod-spec-hash drift.e4012b03test: add the glance release-upgrade e2e suiteAdds
tests/e2e/glance/release-upgrade/— the cross-release upgrade end-to-end, making glance the second consumer exercised by a chainsaw suite. Deploys at 2025.2 with a default S3 backend (glance cannot go Ready without one), asserts the pre-upgrade steady state, patchesspec.openStackRelease+spec.image.tagto 2026.1, then asserts the drain: Ready/AllReady,installedReleasepromoted to 2026.1,upgradePhasecleared, the three phase Jobs left in place, the Deployment flipped to the uWSGI command on the :2026.1 image, and the steady-state db-sync re-run (itsdb load_metadefsloads the 2026.1 definitions). Auto-discovered by the per-operator chainsaw run — no CI wiring changes.aec287bddocs: document the glance upgrade flow and fix the onboarding skillAdds
docs/reference/glance/glance-upgrade-flow.mdmirroring the keystone page (phase table with exact commands/Job names, the launch-mode flip, condition reasons/events, abort semantics, the image/release lockstep contract with its digest-pinning nuance, the post-upgrade metadefs reload). Updates the glance CRD/reconciler/events/index pages (drops the "no upgradePhase field" and everyInvalidReleaseTransitionclaim, adds the new rows/reasons/events, links the page from the sidebar). Adds a shared-machinery note to the keystone upgrade-flow page. Inprepare-new-service, extends the database profiling bullet to ask which zero-downtime upgrade mechanism the manage tool supports and to adopt the shareddatabase.ReconcileUpgradeflow, and drops expand-migrate-contract from the keystone-specific category-3 list now that it is shared category-1 machinery.Divergences from the issue
operators/glance/internal/controller/db_job_metrics.goand keystone'sreconcile_database_test.go/integration_test.goas touch points, but they do not appear in the diff. This is consistent with the commits: glance's DB-Job metrics were already suffix-generic (sodb-expand/db-migrate/db-contractflow throughrecordDBJobTerminalStateunchanged), and the keystone rewire was behavior-preserving enough that its existing upgrade assertions needed no edits — the acceptance criteria explicitly require those keystone suites to pass without modification. The one new keystone test file in the diff isreconcile_deployment_test.go(+55), covering theCompleteRollingUpdatecutover.Verification
Acceptance criteria are covered by the shared-flow unit suite (
internal/common/database/upgrade_test.go), the glance unit/envtest additions, and the new glance release-upgrade e2e suite. Full CI (keystone + glance unit, envtest, and the upgrade e2e suites) runs on this PR.Implemented by planwerk-agent 494b8a9 with Claude:claude-opus-4-8
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.