Skip to content

feat: adopt the shared expand-migrate-contract upgrade flow for glance#736

Merged
berendt merged 8 commits into
mainfrom
implement/issue-734-shared-upgrade-flow
Jul 24, 2026
Merged

feat: adopt the shared expand-migrate-contract upgrade flow for glance#736
berendt merged 8 commits into
mainfrom
implement/issue-734-shared-upgrade-flow

Conversation

@berendt

@berendt berendt commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Glance's release upgrades ran a single glance-manage db sync before the workload rolled — and db sync executes expand → migrate → contract in one pass, so the destructive contract half ran while previous-release pods were still serving. Keystone already solved this with a four-phase upgrade machine (Expanding → Migrating → RollingUpdate → Contracting) that was keystone-private. This branch extracts that machine into internal/common/database, rewires Keystone onto it with frozen behavior, and adopts it for Glance as the second consumer, then corrects the onboarding skill and docs so the next database-backed service decides this at onboarding time.

Closes #734

Change set (commit order)

The eight commits are ordered so each is independently compiling — the shared flow lands first while keystone's private machine still works, then keystone is cut over, then glance adopts it, then tests and docs.

  1. 7f0dd9d4 refactor: move the UpgradePhase type into the shared commonv1 package
    Moves UpgradePhase and its four constants (Expanding/Migrating/RollingUpdate/Contracting) from keystone_types.go into internal/common/types/types.go, so the shared flow can reference them. Keystone keeps a type UpgradePhase = commonv1.UpgradePhase alias plus a re-exported constant block (the GatewaySpec alias pattern), so every existing reference compiles unchanged. The generated CRD is byte-identical — the enum marker travels with the type and the schema description comes from the status-field godoc.

  2. a6f37258 feat: add the shared expand-migrate-contract upgrade flow
    Ports keystone's private four-phase machine into internal/common/database/upgrade.go, parameterized over a single seam — the spec-side release string — via UpgradeFlowParams (owner CR, condition vocabulary, SpecRelease, BuildPhaseJob closure, terminal-metric callback). Exports IsUpgrade, InitiateUpgrade, ReconcileUpgrade, AbortUpgrade, and CompleteRollingUpdate with keystone's exact condition/event reasons and message wording. Two service-specific phrasings are neutralized ("Image tag changed" → "Spec release changed"). Keystone's private machine stays in place and compiling — a later commit deletes it. Ships with the shared unit suite (upgrade_test.go, +538) covering phase transitions, abort, target-changed, unknown phase, and the parse/downgrade/sequential rejections.

  3. d0ac12d0 refactor: rewire keystone onto the shared database upgrade flow
    Deletes reconcile_upgrade.go (−316) and routes reconcileDatabase/reconcileDeployment through InitiateUpgrade/ReconcileUpgrade/AbortUpgrade/CompleteRollingUpdate. buildExpandJob/buildMigrateJob/buildContractJob become the BuildPhaseJob closure. Behavior is frozen — reasons, events, db_sync metric suffixes, and the abort semantics are unchanged; the keystone unit/envtest/e2e suites pass with no test edits. Two seams stay keystone-local deliberately (documented in the commit): the target-changed guard keeps its "image tag changed" prose that a unit test pins, and only the reason constants keystone still emits or tests reference by name are aliased (aliasing the rest would leave unused constants the linter rejects).

  4. a623a085 feat: add status.upgradePhase to the Glance CRD
    Adds UpgradePhase commonv1.UpgradePhase to GlanceStatus and regenerates the two Glance CRD copies (config/crd/bases + the Helm chart). Reuses the shared type, so the Glance CRD gains the same four-value enum keystone already carries; no other operator's CRD or deepcopy output changes.

  5. a64a8be4 feat: adopt the shared upgrade flow for glance release transitions
    Glance becomes the second consumer. A release transition (non-patch step, e.g. 2025.2 → 2026.1) now walks the four phases with glance-manage db expand|migrate|contract phase Jobs, driven identically to keystone; status.upgradePhase tracks the active phase and TargetRelease is owned by the flow. Fresh installs and patch bumps stay on the single-pass db sync path. Documented behavior change in failure reporting only: the bespoke InvalidReleaseTransition reason and its non-zero-requeue short-circuit are gone — an invalid transition now surfaces through the shared vocabulary (VersionParseError, DowngradeNotSupported, UpgradePathInvalid) as a returned error, so the controller backs off with exponential retry exactly like keystone. The accepted set of transitions is unchanged (patch-only or a single sequential step, no downgrade). Deployment reconcile flips RollingUpdate → Contracting once the rollout is ready, and the eventlet → uWSGI launch-mode switch derives from spec.openStackRelease.

  6. e41bec21 test: cover the glance upgrade phase walk in envtest
    Adds a glance integration test driving a full 2025.2 → 2026.1 upgrade against envtest (which runs no Job/Deployment controllers, so completions are simulated), mirroring keystone's upgrade-cycle test. Locks four properties together: the Expanding → Migrating → RollingUpdate → Contracting walk with each phase Job carrying the target image; the rollout-gated contract flip (stays RollingUpdate until the re-imaged Deployment reports ready, asserted with a Consistently); the eventlet → uWSGI launch-command switch; and the post-upgrade steady-state db-sync re-run on pod-spec-hash drift.

  7. e4012b03 test: add the glance release-upgrade e2e suite
    Adds tests/e2e/glance/release-upgrade/ — the cross-release upgrade end-to-end, making glance the second consumer exercised by a chainsaw suite. Deploys at 2025.2 with a default S3 backend (glance cannot go Ready without one), asserts the pre-upgrade steady state, patches spec.openStackRelease + spec.image.tag to 2026.1, then asserts the drain: Ready/AllReady, installedRelease promoted to 2026.1, upgradePhase cleared, the three phase Jobs left in place, the Deployment flipped to the uWSGI command on the :2026.1 image, and the steady-state db-sync re-run (its db load_metadefs loads the 2026.1 definitions). Auto-discovered by the per-operator chainsaw run — no CI wiring changes.

  8. aec287bd docs: document the glance upgrade flow and fix the onboarding skill
    Adds docs/reference/glance/glance-upgrade-flow.md mirroring the keystone page (phase table with exact commands/Job names, the launch-mode flip, condition reasons/events, abort semantics, the image/release lockstep contract with its digest-pinning nuance, the post-upgrade metadefs reload). Updates the glance CRD/reconciler/events/index pages (drops the "no upgradePhase field" and every InvalidReleaseTransition claim, adds the new rows/reasons/events, links the page from the sidebar). Adds a shared-machinery note to the keystone upgrade-flow page. In prepare-new-service, extends the database profiling bullet to ask which zero-downtime upgrade mechanism the manage tool supports and to adopt the shared database.ReconcileUpgrade flow, and drops expand-migrate-contract from the keystone-specific category-3 list now that it is shared category-1 machinery.

Divergences from the issue

  • The issue lists operators/glance/internal/controller/db_job_metrics.go and keystone's reconcile_database_test.go / integration_test.go as touch points, but they do not appear in the diff. This is consistent with the commits: glance's DB-Job metrics were already suffix-generic (so db-expand/db-migrate/db-contract flow through recordDBJobTerminalState unchanged), and the keystone rewire was behavior-preserving enough that its existing upgrade assertions needed no edits — the acceptance criteria explicitly require those keystone suites to pass without modification. The one new keystone test file in the diff is reconcile_deployment_test.go (+55), covering the CompleteRollingUpdate cutover.

Verification

Acceptance criteria are covered by the shared-flow unit suite (internal/common/database/upgrade_test.go), the glance unit/envtest additions, and the new glance release-upgrade e2e suite. Full CI (keystone + glance unit, envtest, and the upgrade e2e suites) runs on this PR.


Implemented by planwerk-agent 494b8a9 with Claude:claude-opus-4-8


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

berendt added 8 commits July 23, 2026 22:20
The expand-migrate-contract database upgrade flow is being lifted out of
the keystone operator into internal/common so glance becomes its second
consumer. Its phase enum must live where the shared flow can reference
it, so move UpgradePhase and its four constants into
internal/common/types beside the other shared database vocabulary.

Keystone keeps a type alias (UpgradePhase = commonv1.UpgradePhase) and a
re-exported constant block, mirroring the GatewaySpec alias pattern, so
every existing keystonev1alpha1.UpgradePhase reference and bare
UpgradePhase* constant keeps compiling unchanged. The generated CRD is
byte-identical: the schema description is driven by the status field
godoc, not the moved type godoc, and the enum marker travels with the
type, so make generate/manifests/sync-crds produces no diff.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Port keystone's private four-phase database upgrade machine
(Expanding -> Migrating -> RollingUpdate -> Contracting) into
internal/common/database as a parameterized flow so glance becomes its
second consumer. The flow generalizes over a single seam, the spec-side
release string (keystone's spec.image.tag, glance's spec.openStackRelease),
supplied through UpgradeFlowParams alongside the owner CR, the condition
vocabulary, the phase Job builder, and the terminal-metric callback.

The exported surface mirrors keystone's semantics exactly: IsUpgrade,
InitiateUpgrade, ReconcileUpgrade, AbortUpgrade, and CompleteRollingUpdate
emit the same condition reasons, event reasons, and message wording as
today, so the eventual keystone rewiring is behavior-preserving. Two
service-specific phrasings are neutralized: "Image tag changed" becomes
"Spec release changed" and "spec.image.tag reverted" becomes "spec
release reverted"; the release-value substrings and the reasons keystone
asserts on are unchanged.

The expand-migrate-contract condition reasons move out of their
keystone-private home into this package next to the steady-state Reason*
set, and the package doc gains a bullet for the upgrade flow. Keystone's
private machine stays in place and compiling; a later commit deletes it
and rewires keystone onto this flow.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Delete keystone's private expand-migrate-contract machine and route the
DatabaseReady upgrade path through internal/common/database, now that
the shared flow added in the previous commit carries the same logic.

reconcileDatabase delegates upgrade initiation, the active-phase
dispatch, and the abort to database.InitiateUpgrade and
database.ReconcileUpgrade; reconcileDeployment drives the
RollingUpdate-to-Contracting transition through
database.CompleteRollingUpdate. A new upgradeFlowParams binds the
keystone-specific inputs: the owner CR, the DatabaseReady vocabulary,
spec.image.tag as the requested release, the per-phase Job builders,
and the terminal-metric callback.

Behavior is frozen. The condition reasons, lifecycle events, db_sync
metric suffixes, and the #468 abort semantics are unchanged, and the
keystone unit, envtest, and e2e suites pass with no test edits.

Two seams stay keystone-local to keep that behavior frozen:
- The target-changed guard remains in reconcileDatabase so its error
  keeps the "image tag changed during active upgrade" prose the unit
  test pins; the shared flow phrases that guard generically as "spec
  release".
- Only the seven upgrade reason constants keystone still emits locally
  or the tests reference by name are aliased to the shared
  database.Reason* set; aliasing the rest would leave unused constants
  the linter rejects.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Glance is about to adopt the shared expand-migrate-contract database
upgrade flow, which tracks the in-flight phase on the owning CR via a
status pointer. Add the status.upgradePhase field to GlanceStatus so the
shared flow can persist Expanding -> Migrating -> RollingUpdate ->
Contracting and clear it when no upgrade is in flight, mirroring
KeystoneStatus.UpgradePhase.

The field reuses the shared commonv1.UpgradePhase type, so the CRD gains
the same four-value enum keystone already carries. Regenerate the two
Glance CRD copies (config/crd/bases and the Helm chart) accordingly; no
other operator's CRD or deepcopy output changes.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Glance becomes the second consumer of the shared expand-migrate-contract
database-upgrade flow (internal/common/database), replacing its bespoke
single-pass release handling. A release transition (a non-patch change
from the installed release, e.g. 2025.2 -> 2026.1) now walks Expanding ->
Migrating -> RollingUpdate -> Contracting with glance-manage
db expand|migrate|contract phase Jobs, driven identically to keystone;
status.upgradePhase tracks the active phase and TargetRelease is owned by
the flow (set on initiate, cleared on completion or abort) instead of
being stamped every pass. Fresh installs and patch bumps stay on the
single-pass glance-manage db sync path.

The accepted set of transitions is unchanged (patch-only or a single
sequential step, no downgrade); only the failure reporting changes. The
bespoke InvalidReleaseTransition reason and its non-zero-requeue
short-circuit are gone: an invalid transition now surfaces through the
shared vocabulary (VersionParseError, DowngradeNotSupported,
UpgradePathInvalid) as a returned error, so the controller backs off with
exponential retry exactly like keystone. The Deployment reconcile flips
RollingUpdate -> Contracting once the rollout is ready, without stamping
the endpoint on the flip pass.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Add an integration test that drives a full glance release upgrade
(2025.2 -> 2026.1) through the shared expand-migrate-contract flow
against envtest, mirroring keystone's upgrade-cycle test. envtest runs
no Job or Deployment controllers, so every phase Job completion and
Deployment readiness is simulated.

The test locks four properties no unit test observes together:

- the Expanding -> Migrating -> RollingUpdate -> Contracting phase
  walk, with each glance-manage db expand|migrate|contract phase Job
  carrying the target-release image;
- the rollout-gated contract flip: the phase stays RollingUpdate until
  the re-imaged Deployment reports ready, asserted with a short
  Consistently before simulating the rollout;
- the eventlet -> uWSGI launch-command switch the Deployment template
  takes on the release bump;
- the post-upgrade steady-state db-sync Job re-running with the new
  image on pod-spec-hash drift.

Only the glance integration test file changes.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Cover the 2025.2 -> 2026.1 cross-release upgrade end-to-end, making
glance the second consumer of the shared expand-migrate-contract
database upgrade flow exercised by a chainsaw e2e suite.

The suite deploys glance-release-upgrade at 2025.2 with a default S3
backend, asserts the pre-upgrade steady state (Ready/AllReady,
installedRelease 2025.2, and the eventlet glance-api launch command),
then patches spec.openStackRelease and spec.image.tag to 2026.1. It
then asserts the upgrade drains: Ready/AllReady with installedRelease
promoted to 2026.1, status.upgradePhase cleared, the db-expand,
db-migrate and db-contract phase Jobs left in place, the Deployment
flipped to the uWSGI launch command on the :2026.1 image with the
rollout converged, and the steady-state db-sync Job re-run on the
:2026.1 image (its db load_metadefs pass loads the 2026.1 metadata
definitions).

The suite is auto-discovered by the per-operator chainsaw run, so no
CI wiring changes.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
Glance now shares Keystone's expand-migrate-contract upgrade machine
(internal/common/database), so a spec.openStackRelease bump with the
image in lockstep walks Expanding -> Migrating -> RollingUpdate ->
Contracting instead of a single-pass db sync. The reference docs and
the prepare-new-service skill still described the old single-pass
design, which no longer matches the code.

Add a Glance Upgrade Flow reference page mirroring the Keystone one:
trigger, version format, status fields, the phase table with the exact
glance-manage commands and Job names, the eventlet -> uWSGI launch-mode
flip during RollingUpdate, the DatabaseReady condition reasons and
events, abort semantics, the image/release lockstep contract with its
digest-pinning nuance, and the post-upgrade metadefs reload.

Update the glance CRD, reconciler, events, and index pages: drop the
"no upgradePhase field" and "reverse window is not closed" claims and
every InvalidReleaseTransition mention, add the upgradePhase status row
and the new condition reasons and events, and link the new page from
the sidebar and reference lists. Note the shared machinery on the
Keystone upgrade-flow page.

In prepare-new-service, extend the Database profiling bullet to ask
which zero-downtime upgrade mechanism the manage tool supports and to
adopt the shared database.ReconcileUpgrade flow, and drop
expand-migrate-contract from the keystone-specific classification now
that it is shared.

On-behalf-of: @SAP
Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
@berendt
berendt force-pushed the implement/issue-734-shared-upgrade-flow branch from aec287b to 7056e72 Compare July 24, 2026 05:35
@berendt
berendt marked this pull request as ready for review July 24, 2026 06:50
@berendt
berendt merged commit 65e3d3b into main Jul 24, 2026
53 checks passed
@berendt
berendt deleted the implement/issue-734-shared-upgrade-flow branch July 24, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt expand-migrate-contract database upgrades for Glance and encode the pattern in the service-onboarding skill

1 participant