v3.0.0

Restate Operator v3.0.0 Release Notes

See the v2.8.1 release notes for changes prior to this release.

This is a major release: upgrading an existing install requires a one-time CRD ownership hand-off (see Breaking Changes), and several deployment-lifecycle fixes change behavior on the first reconcile after upgrade (see Upgrade Notes).

Highlights

• CRDs now upgrade with the chart. The three CRDs ship as a templated, standalone restate-operator-crds chart, so helm upgrade applies schema changes instead of silently skipping them. The first upgrade from ≤ 2.8.1 needs a one-time ownership hand-off.
• The operator waits for its CRDs on startup instead of exiting/crashlooping, and surfaces the wait via /ready, a metric, and a Kubernetes event.
• RestateDeployment deletion no longer hangs forever on its own latest version.
• Rolling back now moves Restate's routing, and Ready=True is stricter — inconsistent deployments are promoted on the next reconcile after upgrade.
• GCP workload-identity IAMPolicyMember server-side-apply wedge is fixed.

Table of Contents

• Upgrade Notes
• Breaking Changes
• Improvements
• Bug Fixes

Upgrade Notes

Read these before upgrading — each is expanded below.

• CRDs: one-time ownership hand-off. The first upgrade to templated CRDs fails with invalid ownership metadata until you hand ownership to Helm. See Breaking Changes. Nothing is deleted if you skip it — the upgrade just aborts.
• Chart and image move in lockstep. The readiness probe now targets /ready, which older operator images do not serve. If you pin the chart version to an image older than 3.0.0, pin the chart to a matching older version too — otherwise the /ready 404 leaves the pod NotReady forever.
• Inconsistent RestateDeployments are promoted on the next reconcile. If a deployment's desired revision is not the one Restate currently routes to, the operator will move routing to it — a routing change that happens without being asked for. Upgrade when that is acceptable. Endpoints registered outside their RestateDeployment now stall at Ready=False with reason ForeignDeployment instead of being silently tolerated.
• Deletion now waits out the drain window. Deleting a RestateDeployment now takes at least spec.restate.drainDelaySeconds (default 300s) and still blocks on unfinished invocations (including scheduled ones, which have no upper bound).

Breaking Changes

CRD installation now upgrades with the chart — action required on first upgrade

Through 2.8.1 the CRDs shipped through Helm's native crds/ directory, which is install-only: once the CRDs were in the cluster, helm upgrade never touched them again. That bit a customer — after an operator upgrade, new schema fields were silently pruned at admission until the CRDs were reapplied by hand.

The CRDs now live in their own restate-operator-crds chart, rendered as ordinary templates, so upgrading the chart applies the new schema. They carry helm.sh/resource-policy: keep, so helm uninstall will not take the CRDs (or the custom resources under them) down with it. restate-operator-helm bundles this chart behind installCrds (default true), so a standalone operator install is unchanged. The CRD chart is also published on its own:

helm upgrade --install restate-operator-crds \
  oci://ghcr.io/restatedev/restate-operator-crds --version 3.0.0

One-time ownership hand-off. CRDs installed through the old crds/ directory (2.8.1 or earlier) carry no Helm ownership metadata, so the first upgrade to the templated CRDs fails with invalid ownership metadata until ownership is handed over. Pick one:

• Helm ≥ 3.17 — add --take-ownership to the upgrade. Where you pass it depends on how you run the CRDs:
  • Bundled (default, installCrds=true): helm upgrade restate-operator … --take-ownership
  • Standalone CRD chart (installCrds=false): helm upgrade restate-operator-crds … --take-ownership
• Older Helm — set the app.kubernetes.io/managed-by=Helm label plus the meta.helm.sh/release-name and meta.helm.sh/release-namespace annotations on the three CRDs, for the release that owns them.
• ArgoCD / Flux — no action needed; they adopt the resources on sync and never hit Helm's ownership gate.

Either way no custom resources are deleted — the failure aborts the upgrade, it does not cascade.

If you don't want Helm upgrading your CRDs. CRDs are cluster-scoped, so on a shared or multi-tenant cluster a helm upgrade now mutates a cluster-wide object that can affect other teams. To gate CRD changes behind manual review — or if you already manage CRDs via GitOps — set installCrds=false on the operator and own the CRD lifecycle yourself: apply crd/*.yaml directly, or install the standalone restate-operator-crds chart on your own cadence. This is the recommended path for ArgoCD/Flux.

Improvements

Startup: wait for CRDs instead of exiting (#166, #169)

If a required CRD (RestateCluster, RestateDeployment, RestateCloudEnvironment) is not yet installed when the operator starts, it no longer logs an error and exits. It now polls the apiserver's discovery endpoint and waits for each CRD to appear, then begins reconciling. This matches how most Kubernetes operators behave and removes the ordering/restart dance in GitOps flows where the operator and its CRDs are applied together.

Missing group/versions, 404/429/5xx responses, and an unreachable apiserver are all treated as "not ready yet, keep waiting". Genuine configuration errors (401/403, bad kubeconfig) still log and exit. The PodIdentityAssociation check (when aws-pod-identity-association-cluster is set) still exits, since that is a configuration error rather than a race.

The wait is observable three ways:

• Readiness endpoint. /ready returns 200 only once every controller has its CRD and has started reconciling, otherwise 503 with the pending controllers listed:

  {"ready":false,"pendingControllers":["RestateCluster","RestateDeployment"]}

  /health keeps its always-200 behavior and is now the liveness probe.

• Metric. restate_operator_crd_missing{crd="<plural>.<group>"} is 1 while waiting and 0 once available — alert with e.g. restate_operator_crd_missing > 0 for 5m.

• Event. A single Warning/WaitingForCRD event is recorded against the operator's own pod naming every CRD being waited on, closed out by a Normal/CRDsAvailable event. Nothing is emitted if the CRDs land within the first 10 seconds (the common case).

No new RBAC is required.

Helm Chart

• The CRDs are delivered by the templated, standalone restate-operator-crds chart — see Breaking Changes.
• The readiness probe now targets /ready; a liveness probe targeting /health was added. See the chart/image lockstep note.
• The Deployment passes OPERATOR_POD_NAME and OPERATOR_POD_UID via the downward API so operator-level events attach to the right pod. If you deploy the operator without the chart, set both (or neither — without them the operator still logs and reports the metric, but emits no events).
• The operator Service sets publishNotReadyAddresses: true, so a NotReady operator — exactly when restate_operator_crd_missing needs scraping — stays in the ServiceMonitor's targets.

Examples & Docs

• Example manifests, Pkl templates, and docs now reference Restate server 1.7 (#151). No forced default: the operator does not hardcode a server image, so existing clusters keep whatever version their manifests pin.

Bug Fixes

RestateDeployment

Deletion no longer hangs forever (#172). Cleanup previously conflated "a service still points at this deployment" with "it has unfinished invocations", so during a deletion the latest version stayed "active" forever, the finalizer never completed, and kubectl delete restatedeployment never returned (namespace deletion inherited the hang). The operator now tracks the two facts separately: being a service's current endpoint holds a version through a rollout but is ignored during a deletion; only unfinished invocations can hold a deletion, and they drain on their own.

• Deletion now takes at least spec.restate.drainDelaySeconds (default 300s), and still blocks on unfinished invocations — scheduled invocations days out have no upper bound. A blocked deletion now backs off (30s → 5m) and the DeploymentInUse event names each version and its pinned/unpinned invocation counts.

• No migration needed: a currently-stuck deletion proceeds on the next reconcile after upgrade (no manual finalizer edits). To see what is holding a deletion:

  kubectl describe restatedeployment <name> -n <namespace>

Rolling back now moves Restate's routing (#174). Because ReplicaSets/Services are named by a content hash, v1 → v2 → v1 re-adopts the original Restate deployment id, so restoring the Kubernetes revision was never enough on its own — Restate also has to be told the restored deployment is latest again. The operator now decides on whether Restate routes new invocations to the recorded deployment, re-registers with overwrite when it does not (preserving the deployment id and pinned invocations), verifies routing via GET /services before reporting Ready, and emits a Promoted event.

• Ready=True is stricter: it now means Restate routes new invocations to the desired revision, not merely that the pods are up. See the routing-promotion upgrade note for the on-upgrade impact (unrequested promotions, ForeignDeployment stalls, and that a promoted deployment's registration timestamp is reset in Restate).

• To check for the inconsistent state this fixes:

  kubectl get restatedeployment <name> -n <namespace> -o jsonpath='{.status.deploymentId}'
  curl -s "$RESTATE_ADMIN/services/<ServiceName>" | jq -r .deployment_id

RestateCluster

Request-identity config emitted under the nested worker.invoker key (#158). With request signing enabled (spec.security.requestSigningPrivateKey), the operator now emits RESTATE_WORKER__INVOKER__REQUEST_IDENTITY_PRIVATE_KEY_PEM_FILE (→ worker.invoker.request-identity-private-key-pem-file) instead of the flat alias restate-server deprecated. Only the env var name changes; signing behavior is unchanged. No action required — this removes the startup deprecation warning and keeps working once restate-server drops the alias.

GCP workload-identity IAMPolicyMember SSA wedge fixed (#136). The restate-workload-identity IAMPolicyMember CR is now applied with spec.resourceRef.apiVersion set, matching what Config Connector defaults at creation. Previously SSA stripped the defaulted value on every reconcile, Config Connector's deny-immutable-field-updates webhook rejected it, and the reconcile loop blocked unrelated RestateCluster updates. Affects BYOC installs on GKE / Config Connector; existing wedged installs unblock once the upgraded operator reconciles.