Skip to content

v2.6.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Jun 11:16
· 2 commits to main since this release
v2.6.0
dd62d2c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Restate Operator v2.6.0 Release Notes

Highlights

  • Per-version autoscaling for draining RestateDeployment versions — an opt-in spec.autoscaling block lets the operator run a HorizontalPodAutoscaler per non-latest version, so old versions shed compute as their invocations drain instead of holding full replicas for the entire (potentially multi-hour) drain window.
  • tunnelMode: in-process for Restate Cloud deployments — pods hold their own outbound tunnel connections (e.g. via @restatedev/restate-sdk-tunnel), serving a Restate Cloud environment with zero inbound networking, while keeping the operator's per-revision registration and draining.
  • Fix: HPA metrics on a RestateDeployment are no longer polluted by draining versions — the scale subresource's label selector is now scoped to the latest version's pods, fixing under-provisioning of a busy latest version during drains.
  • tolerations and nodeSelector in the Helm chart — schedule the operator pod through chart values instead of forking the chart.

New Features

Per-version autoscaling for draining RestateDeployment versions

A new optional spec.autoscaling field on RestateDeployment (ReplicaSet mode)
lets the operator manage a HorizontalPodAutoscaler per non-latest version.
It is a pass-through HPA .spec (minReplicas, maxReplicas, metrics,
behavior); the operator injects scaleTargetRef per version and owns the HPA,
so it is garbage-collected with the RestateDeployment. A non-latest version
gets an HPA while Restate still reports it active, and the HPA is removed when it
goes inactive — before the operator scales the ReplicaSet to zero.

Previously every old version was held at its full replicas for the entire
drain window — which can last hours for long-running or stuck workflows —
multiplying compute cost across concurrently-draining versions, with no lever to
reduce it (an HPA on the RestateDeployment scale subresource only ever scales
the latest version). Now old versions shed compute as their load falls, while
remaining available for in-flight invocations.

Impact on Users:

  • Existing deployments: no impact — the field is optional, and omitting it
    preserves today's behaviour (draining versions stay at full replicas).
  • Knative mode: unaffected (Knative's own autoscaler handles this); the field is
    rejected in Knative mode.

Usage:

spec:
  autoscaling:
    minReplicas: 1
    maxReplicas: 10
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 70

Notes:

  • The latest version is not covered here — autoscale it with your own HPA
    targeting the RestateDeployment scale subresource (now correct thanks to the
    selector fix below).
  • minReplicas is floored at 1 (there is no scale-to-zero in ReplicaSet mode).
    Only Resource/ContainerResource/Pods metrics are per-version (scoped via
    the target's pod selector); an Object/External metric in the shared
    template is read globally, so every draining version would scale on the same
    value.
  • CPU/memory metrics require container resource requests; prefer CPU (memory
    does not scale back down).
  • RBAC: the bundled Helm chart now grants the operator
    get,list,watch,create,patch,delete on horizontalpodautoscalers (the
    autoscaling API group). If you deploy the operator with your own RBAC,
    add this, or per-version HPAs will silently fail to be created.

Related: Issue #140, PR #143

tunnelMode: in-process for Restate Cloud deployments

RestateDeployment gained spec.restate.tunnelMode (takes effect only with
register.cloud). With tunnelMode: in-process, the deployment's pods hold
their own outbound tunnel connections to Restate Cloud — for example with the
@restatedev/restate-sdk-tunnel npm package — instead of being reached through
the tunnel-client pods and each version's Service. Workloads in private
networks can therefore serve a Restate Cloud environment with zero inbound
networking, while keeping the operator's transparent per-revision registration
and draining.

For such deployments the operator injects RESTATE_INPROC_TUNNEL_NAME (the
versioned name of the revision), RESTATE_INPROC_ENVIRONMENT_ID,
RESTATE_INPROC_CLOUD_REGION and RESTATE_INPROC_SIGNING_PUBLIC_KEY into every
container and init container of the pod template (declaring one of these
yourself is a reconcile error), and registers each version under its tunnel URL
(https://tunnel.<region>.restate.cloud:9080/<env>/<versioned-name>/http/in-process/9080/)
instead of the Service URL. Credentials are never injected —
RESTATE_INPROC_AUTH_TOKEN_FILE is reserved for your own Secret mount. A change
to the referenced RestateCloudEnvironment's environment id, region or signing
key mints a new version, exactly like a pod-template change (these values are
folded into the revision hash only when the mode is set, so existing deployments
keep their hashes). tunnelMode: in-process is rejected in Knative mode.

Impact on Users:

  • Existing deployments: no impact. The field is optional, and hashes of
    deployments that don't set it are unchanged, so no ReplicaSets roll on upgrade.

Usage:

spec:
  restate:
    register:
      cloud: my-cloud-environment
    tunnelMode: in-process

Run an in-process tunnel client in your pods and mount an API key Secret (point
RESTATE_INPROC_AUTH_TOKEN_FILE at it). To adopt the mode on an existing
cloud-registered deployment, add tunnelMode: in-process — this creates a new
version (new hash), registered through the tunnel while the old Service-routed
version drains as usual.

Related: PR #144

Tolerations and nodeSelector in the Helm chart

tolerations and nodeSelector are now declared in
charts/restate-operator-helm/values.yaml, so the operator pod can be scheduled
onto tainted or dedicated nodes through chart values instead of forking the
chart or layering a post-render patch. The tolerations template was already
wired; this exposes it via the values file and adds a matching nodeSelector
block.

Impact on Users: existing deployments — no impact (both fields are optional).

Usage:

tolerations:
  - key: "service"
    operator: "Exists"
    effect: "NoSchedule"
nodeSelector:
  workload: controllers

Related: Issue #127

Bug Fixes

RestateDeployment

  • Scale-subresource labelSelector scoped to the latest version
    (#139). The
    RestateDeployment scale subresource's .status.labelSelector now appends the
    latest pod-template-hash. Previously it was written verbatim from
    spec.selector with no version filter, so an HPA targeting a
    RestateDeployment averaged metrics across every ReplicaSet the deployment
    had ever owned — the latest plus any old ones still draining pinned
    invocations. Over the (potentially multi-hour) drain window this polluted the
    averaged metric, under-provisioning — or even scaling down — the genuinely-busy
    latest version. No action required; deployments without an HPA are unaffected.

Upgrading

Reapply the CRDs before using the new fields:

kubectl apply --server-side -f crd/restatedeployments.yaml
Assets 5
Loading