v3.1.0

Restate Operator v3.1.0 Release Notes

See the v3.0.1 release notes for changes prior to this release.

Highlights

• deletePolicy for RestateDeployment. Deleting a deployment now has a deadline, a way out, and a .status.deletion that says what the deletion is waiting for.
• Suspend reconciliation per resource. restate.dev/reconcile: disabled makes the operator leave a resource and its children alone — Flux's suspend, without scaling the operator to zero.
• Credential canaries retry instead of looping or stalling. The Pod Identity canary no longer deletes a Job out from under its own retry, and both canary Jobs now have a five-minute active deadline.
• Rolled-back versions keep their drain delay instead of being scaled to zero on the first reconcile.
• ApplySet bookkeeping labels no longer propagate to operator-owned children, so kubectl apply --prune --applyset cannot take a draining Service with it.

Table of Contents

• Upgrade Notes
• New Features
• Improvements
• Bug Fixes

Upgrade Notes

• Apply the new CRDs before rolling the operator image. Two changes need them: the new spec.restate.deletePolicy / spec.restate.drain fields on RestateDeployment, and a status subresource on RestateCloudEnvironment. With the default installCrds: true the bundled chart handles this; if you manage CRDs yourself (installCrds: false), apply crd/*.yaml or upgrade the standalone restate-operator-crds chart first.
• Metric label change. restate_operator_reconciliation_errors_total now labels RestateDeployment failures with the error the reconciler actually raised (DeletionDrainOverdue, DeploymentInUse, AdminCallFailed, …) instead of collapsing them all into FinalizerError. Alerts or dashboards matching error="FinalizerError" for this controller need updating.
• A blocked deletion reports differently after one hour. A drain deletion that is still held raises DeletionDrainOverdue with phase: Overdue instead of DeploymentInUse. The Kubernetes event reason is unchanged (FailedReconcile); only its message changes.
• Rollback followed by another rollout now takes drainDelaySeconds longer to tear the intermediate version down. That delay is the fix, not a regression.

New Features

deletePolicy for RestateDeployment (#198)

Deleting a RestateDeployment is now governed by spec.restate.deletePolicy, whose terms are set by spec.restate.drain:

• deletePolicy: drain (default): the existing behaviour — wait for in-flight invocations to finish before deregistering. drain.timeoutSeconds (default 3600) sets the deadline and drain.onTimeout decides what happens there:
  • hold (default): keep waiting, and report the drain as overdue.
  • force: deregister anyway, abandoning whatever is left.
• deletePolicy: force: deregister and tear down immediately, skipping the drain, drainDelaySeconds and the revision history limit. drain is ignored entirely.

Progress is reported on .status.deletion, which names the versions holding the deletion and their pinned/unpinned invocation counts, alongside the phase (Draining, Overdue, Forcing), the deadline, what happens at it, and a total. kubectl get rsd -o wide shows the phase and total. A force deletion that walked over unfinished invocations raises a ForcedDeletion warning event naming what it abandoned.

Why this matters. A deletion blocked by a scheduled invocation days out, or by a workflow that will never complete, previously had no deadline and no visible reason: the only signal was a warning event and the operator log, and the only way out was to find and cancel the invocations. There is now a bounded option, and a status field that answers "what is this waiting for".

Impact. Existing deployments keep the current behaviour — no deletePolicy means drain, which still waits indefinitely. deletePolicy: force skips the wait, not the deregistration: it still needs the Restate admin API to be reachable, so it will not unstick a deletion blocked because Restate is down. A paused RestateDeployment (restate.dev/reconcile: disabled) still deletes normally, and now drops the Disabled reconciliation state and its Reconciling condition when the deletion starts, rather than reporting itself suspended throughout the teardown.

Migration. Apply the updated CRDs (helm upgrade of restate-operator-crds, or kubectl apply --server-side -f crd/restatedeployments.yaml). No changes are required to existing RestateDeployment resources. To unstick a deletion that is already blocked, patch the terminating object:

kubectl patch rsd greeter --type=merge \
  -p '{"spec":{"restate":{"deletePolicy":"force"}}}'

See docs/delete-policy.md.

Suspend reconciliation by annotation (#197)

Annotating a RestateCluster, RestateDeployment or RestateCloudEnvironment with restate.dev/reconcile: disabled makes the operator leave that resource and everything it owns alone until the annotation is removed — Flux's suspend, per resource, for hand-editing generated objects during an incident without scaling the whole operator to zero.

Only the exact value disabled suspends, so a typo cannot silently stop reconciliation. .status.reconciliation reports Reconciling, Disabled or ResumingReconciliation (annotation gone, not Ready again yet), and the rest of the status stays frozen as of the last real reconcile.

Deletion is not suspended — the annotation stops the operator managing a resource, not tearing it down, so a suspended resource still deletes and cleans up normally.

RestateCloudEnvironment gained a status subresource for this, so if you manage CRDs yourself (installCrds: false), apply the new CRDs before rolling the operator image.

Improvements

CRDs

• RestateCloudEnvironment gained printer columns, so kubectl get rce shows its state without -o yaml (#133).

Bug Fixes

RestateCluster

Stalled credential canaries are retried (#194, #200)

The Pod Identity pod shortcut now only accelerates success: missing credentials on an individual pod no longer delete the Job before its retry can run. Terminally failed Jobs are deleted with their pods — deletion cascades rather than orphaning them — and retried on the next reconcile. Pod checks are scoped to the current Job, so a stale pod from an earlier attempt can no longer decide the verdict.

Both Pod Identity and Workload Identity canary Jobs now carry a five-minute active deadline, so a Job that makes no progress (a pod stuck in Pending, say) fails and is retried instead of waiting forever. Successful Jobs are retained.

Why this matters. Credential propagation and scheduling delays no longer cause premature Job deletion or leave a canary waiting indefinitely. Before this fix, one orphaned canary pod could put the check into a loop that created pods at roughly three per second for as long as it lasted.

Impact. New and existing clusters receive the deadline on reconciliation. Previously orphaned canary pods are not removed by this fix and may still need separate cleanup:

kubectl -n <namespace> delete pods -l job-name=restate-pia-canary \
  --field-selector=status.phase!=Running

RestateDeployment

Rolled-back versions keep their drain delay (#174, #179)

A version that is rolled back to — or reintroduced with an identical spec — no longer carries a stale removal deadline into its next rollout.

When a version is superseded, the operator stamps restate.dev/remove-version-at on its ReplicaSet (Knative mode: its Configuration) to schedule teardown after spec.restate.drainDelaySeconds. Because ReplicaSets and Services are named by a content hash of the pod template, and Configurations by tag, rolling back re-adopts that exact object rather than creating a new one — and the stamp came with it. Nothing removed it: the annotation has its own field manager, and the cleanup pass skips whichever version is currently latest. The next time that version was superseded, its deadline had already passed, so it was scaled to zero on the first reconcile instead of being given its drain window.

The operator now clears the annotation when it re-adopts a version as the latest one, in both deployment modes. Clearing it also works now: in ReplicaSet mode the old clear left the annotation behind with an empty value instead of removing it, so a version that became active again while draining kept an empty deadline for good and was re-patched on every reconcile.

Why this matters. The drain delay exists so that invocations already pinned to a version can finish on it. A version that skipped the delay was scaled to zero while that work was still in flight, which surfaces as invocations retrying against a version with no pods behind it until they are re-pinned or time out. The window needed a rollback (or a re-applied identical spec) followed by another rollout — the ordinary shape of "revert, fix forward".

Impact. No configuration change. A version currently holding a stale or empty deadline has it cleared the next time it is reconciled as the latest version.

ApplySet bookkeeping labels are no longer propagated (#170, #171)

RestateDeployment no longer copies labels in the applyset.kubernetes.io/ namespace onto its operator-owned Services, Knative Configurations, or Knative Routes. Other user labels continue to propagate.

Why this matters. kubectl uses applyset.kubernetes.io/part-of to decide which resources belong to an ApplySet. Copying that label made operator-owned child resources look like direct members of the ApplySet that contained the RestateDeployment, so a later kubectl apply --prune --applyset=... could delete a versioned Service before Restate had drained the corresponding deployment.

Impact. No manifest changes. The operator removes the bookkeeping labels from children on reconciliation, and kubectl continues to manage and prune the RestateDeployment itself.

No panic on the Knative reconcile path (#174, #181)

RegistrationAction::AlreadyLatest now carries the deployment id it was decided from, so the Knative reconciler reads it out of the verdict instead of unwrapping the Option it had just passed in. The invariant held — the planner reaches AlreadyLatest only after matching a recorded id against Restate's usage map — so the .expect() was not reachable in practice, but it sat on the most frequently taken branch of the reconciler, where the cost of being wrong is the controller task unwinding rather than an error condition. Moving the id into the variant makes the bad state unrepresentable.

Impact. None: no behaviour change, no CRD change, and no new status field or annotation.