DBInstance controller redesign: from phase-pointer flow to level driven logic

[Yohansenanayake]

Context

The current DBInstance controller works, but its core movement is still mostly driven by status fields such as status.provisioningPhase.

That creates a few problems as we add M1 features:

• status can become the controller’s memory of past actions, not proof of current reality;
• deleted or drifted child resources may not be re-verified;
• stop/start/resize are special branches before provisioning flow;
• future features like TLS, credentials reset, image repave, backups, replicas, etc. will keep adding more branches.

The main redesign goal is:

Reconcile from observed Kubernetes/Harvester state, not from a phase pointer.

Status should report progress. It should not be the source of truth.

[Yohansenanayake]

Proposal: convergence pipeline

This redesign proposal replaces the current status.provisioningPhase dispatcher with a fixed convergence pipeline.

Instead of moving like this:

Pending → NetworkProvisioned → StorageProvisioned → VMCreated → DatabaseReady → Available

the controller runs ordered idempotent steps:

preflight → credentials → VM → power → resize → health → monitoring

See attached design diagram for the proposed flow.

What this improves:

• provisioning logic no longer depends on a phase-pointer switch;
• each concern has a clear step;
• stop/start/resize become part of normal reconciliation;
• future work like repave, reset, TLS, backups, and replicas gets a natural extension point;
• status phase becomes descriptive instead of being the main control gate;
• resource builders and owner refs become the path for owned Kubernetes children.

I believe this is a strong improvement over the current controller design.

[tgcjananga]

I like the direction of replacing the phase-pointer dispatcher with an ordered convergence pipeline, but I have two concerns about the happy path.

First, could this make each reconcile run too long and keep controller workers busy?

If one reconcile pass tries to walk the whole pipeline, it may spend time checking or triggering multiple concerns: credentials, VM creation, power transition, resize state, health, monitoring, and status. Even if each operation is individually reasonable, the combined reconcile duration could become long. With many DBInstances, this means worker slots stay occupied longer, other queued events wait behind them, and the controller becomes less responsive under load.

Second, will users lose status visibility during provisioning?

In the happy path, several important things happen before the database is actually ready: credentials created, VM requested, VM running, database booting, endpoint discovered, monitoring created. If status is mainly patched after the full pipeline completes, users may not see useful intermediate progress. From their point of view, the instance may look stuck until the final ready condition appears.

[Yohansenanayake]

Good point. I agree that allowing a single reconcile pass to act across the entire pipeline could make reconciliations too long, occupy worker slots, and reduce responsiveness as the number of instances grows. Delaying status updates until the end would also reduce progress visibility.
The design should therefore place explicit boundaries around each pass, returning after meaningful progress and reporting that progress incrementally

[Yohansenanayake]

Bounded Reconciliation Model

Following the concern about long-running reconcile passes, the convergence pipeline uses a bounded reconciliation model.

The goal is to keep each pass short without forcing every already-satisfied step into a separate reconciliation. A pass may continue through multiple validation or read-only steps, but it stops when it reaches a meaningful convergence boundary.

Reconciliation flow

General model

Each step:

1. Observes the current Kubernetes or provider state.
2. Compares it with the desired state.
3. Performs at most one bounded unit of related work.
4. Returns a StepResult.

observe → compare → optionally act → report outcome

The runner continues only while steps return Satisfied. The first Pending, Terminal, or Transient outcome stops the pipeline. Status is then derived and patched before reconciliation returns.

Status describes observed progress; it is not used as memory of previously executed actions.

Step outcomes

Outcome	Meaning	Runner behavior
Satisfied	The step’s prerequisite is proven, and no re-observation boundary is required	Continue to the next step
Pending	External state was changed, or asynchronous convergence is still in progress	Patch status and stop the pass
Terminal	Retrying the same observed state cannot make progress; a relevant specification, configuration, or external-state change is required	Patch status and stop automatic retries until a new event triggers reconciliation
Transient	A retryable API, provider, or infrastructure error occurred	Patch status and return the error for controller-runtime backoff

Pending supports two requeue strategies:

• Event-driven: after a mutation, rely on an owned-resource or related watch.
• Timed fallback: use RequeueAfter when the awaited transition may not produce a reliable watch event.

Pending does not necessarily indicate failure. A resource may have been successfully created or updated, but the pass stops so the persisted result can be observed during the next reconciliation.

Rules of thumb

• Derive decisions from observed Kubernetes or provider state, not from status conditions or a persisted phase pointer.
• For a mutating convergence step, operation success alone does not normally mean Satisfied:
  • Desired state already observed or operation reports no change → Satisfied.
  • Resource created or updated → Pending.
• Starting an asynchronous transition such as VM creation, start, stop, or resize → Pending.
• Waiting for an asynchronous transition to complete → Pending, usually with a timed fallback.
• Pure validation, read-only observation, deterministic bookkeeping, and in-memory status projection may return Satisfied and allow the pipeline to continue.
• Do not block or poll inside reconciliation. Request the transition, return, and observe it again later.
• Each step owns its component condition and uses declared, typed reason constants.
• Status is patched on every stopped or completed pass so users can see incremental progress.
• Top-level ObservedGeneration advances only after the complete step chain is satisfied.

Ensuring that a step can converge

A mutating step must have a stable way to distinguish converged state from work still required.

Prefer one of the following:

• Observe and compare the current state before performing the operation.
• Make the underlying helper report whether it created, updated, or performed no work.
• Observe a stable operation or runtime result that proves completion.

The normal pattern is:

observe current state
    ├─ already matches desired → Satisfied
    └─ drift detected
          ├─ mutation succeeds → Pending
          └─ mutation fails    → Transient

next pass:
observe persisted state
    └─ matches desired → Satisfied

A step must not blindly return Pending after an operation that produces the same ambiguous result on every pass. That would prevent the runner from ever reaching the next step.

If no observable convergence signal can be established, the operation cannot safely act as a blocking bounded step. Its API contract should be improved, or it should be explicitly modeled as a non-gating action with documented semantics.

Summary

Bounded reconciliation does not mean one function or one action per reconcile. A pass may continue through multiple cheap, read-only, validation, or already-satisfied steps. The boundary occurs when a step changes external state, waits for asynchronous convergence, or encounters an error. This keeps reconcile passes responsive while preserving clear, incremental progress reporting.