Document durability model, delivery health signals, and team isolation - #1
Merged
Conversation
Findings from a customer-style trial of this reference on a live EKS cluster, written up for the platform team the guide addresses. Architecture gains a data durability model: which data on the volumes is irreplaceable (replicated log, Raft metadata) and which is derived (partition stores), why snapshots are not a backup, and a strong recommendation to keep metadata in S3 through the object-store provider, chosen at first deployment because a later migration stops invocation processing. The services guide gains the operator's Ready condition reasons and how Terraform, Argo CD, and Flux each surface a rejected revision, with an Argo CD health check, plus a team isolation section stating that one RestateCluster is one trust domain. Operations gains the fact that stage 02's terraform apply does not wait for a Restate roll, the rollout gate to use instead, and the three snapshot trigger modes with their log-trimming consequence. terraform/03-services is an optional example of applying the RestateDeployment from Terraform, kept separate from the cluster stages. The compute manifest and services guide now point at restatedev/examples instead of a service this repository never shipped.
pcholakov
marked this pull request as ready for review
September 2, 2026 16:13
…maintenance Timings and counts from the validation run were useful to the authors but say nothing to an operator reading the guide, so the roll measurement and the narrated rejected-revision test become plain statements of behavior. Adds the observability section the guide lacked: metrics on 5122 and the networkPeers.node entry a scraper needs, JSON logs, and OTLP tracing with the egress rule a private collector needs. Adds a node maintenance section describing the operator's PodDisruptionBudget and the three-node floor, and notes that deleting an idle RestateDeployment still waits drainDelaySeconds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I did a test-drive (using Claude Code, Fable 5.1) of the reference at 64189a8 - here are some updates I thought might be useful additions.
Summary
Documentation from a trial of the reference on a live EKS cluster, plus an optional Terraform example. Installation held up. Noted some gaps in what we say about durability, delivery tooling, and trust boundaries – these things might be handy to a platform team deploying Restate for the first time on.
I also wanted to raise it whether to make S3 metadata the default (see Open question below).
Changes
docs/00-architecture.md: data durability model. Which data on the volumes is irreplaceable (the replicated log and the Raft metadata), which is derived (partition stores), why snapshots are not a backup, and that no supported backup and restore procedure exists yet(!!!). Includes a strong recommendation to use the S3 object-store metadata provider on AWS, decided at first deployment because migrating from the replicated store later requires restarting all nodes in migration mode and stops invocation processing for the duration.docs/03-deploying-services.md: health signals for delivery tools. TheReadycondition reasons the operator emits, and how a rejected revision surfaces in Terraform (waits to its timeout, then a generic error), Argo CD (needs a health check, one is included), and Flux (kstatus, Flux's status library, keeps waiting onReady=False). Recommends gating on the condition and describes the Terraform-for-cluster, Argo CD-for-applications split as the one that fits the guide's existing cadence argument.docs/03-deploying-services.md: team isolation. One RestateCluster is one trust domain, service names are cluster-global, and separate trust domains belong on separate clusters behind authenticated ingress.docs/05-operations.md, three additions:terraform/02-restate'sterraform applyreturns before a Restate roll starts, withkubectl rollout statusas the gate to use instead.num-retained(snapshots kept per partition) trade-off against Restate's recommendation of 1.terraform/03-services/. Optional example applying the RestateDeployment from Terraform with a digest-pinnedservice_imagevariable. The Terraform guide's "no service variable" sentence now points at it. Added toscripts/validate.sh.resources/05-restate-compute.yamland the services guide point at restatedev/examples instead of a service that does not exist in this repository.Open question: default metadata backend?
Whether object-store metadata should become the shipped default rather than a recommendation. For: it removes the one irreplaceable datum that shares a volume with the log, and it makes an object store a day-one dependency, which removes the usual reason snapshots get deferred until the volumes fill. Against: the shipped manifest is a validated translation of the Restate Cloud profile, and the object-store path was not exercised (see Validation).
Related: restatedev/restate-operator#199 (terminal handling of permanent registration rejections), filed from the same trial.