Production report: GoAkt as the whole distribution layer of a multi-replica Go monolith

[StringKe]

What we run

An internal analytics platform (self-hosted, Google-Analytics-shaped) that collects behavioural and telemetry data from clients across several of our product lines. One Go application, deliberately kept a monolith, running on Kubernetes at roughly 31 pods in steady state, with about 16 new pods joining the existing cluster simultaneously during a rolling update. GoAkt is the entire distribution layer: no MQTT or AMQP broker, no service mesh, no external actor registry. The only stateful things next to the app are a database and Redis.

How the workload maps onto the actor model

Four shapes, each answered by a primitive rather than by a piece of infrastructure:

• Inbound long connections. WebSocket/SSE clients land on a random replica via the k8s Service. Each connection is an ephemeral actor with a cluster-wide name (WithRelocationDisabled plus a long-lived passivation strategy), so "deliver this to user X" is a location-transparent Tell instead of a Redis lookup plus a routing table we maintain ourselves.
• Outbound long connections. Integrations that each hold one long-lived connection to a third party. Each must exist exactly once cluster-wide, yet the set of them must spread across replicas because their resource profiles differ. Grain single activation plus relocation is exactly that semantics, and it is the one piece we would otherwise have hand-built out of Redis locks and gotten subtly wrong.
• Fan-out computation. Reporting work like "aggregate everything a reseller's whole downline produced" decomposes naturally into map/reduce over messages, with mailbox back-pressure for free.
• Scheduled and durable work. Cron, delayed tasks, at-least-once processing. This is where we contributed the most: with cluster single-fire cron in place, the external scheduler we used to run next to the app can retire.

How it performs

The headline is that the actor core has not been a source of surprises. Message passing, supervision, placement, and grain single activation behave exactly as documented under steady state, and none of our production incidents in this stack originated there. Everything that bit us lived at the edges of the lifecycle: how the system starts, and what happens when a node goes away. That is a good place for the sharp edges to be, and it is why the two issues below were worth digging into.

The two edges we hit

1. Remoting inherited cancelation from the Start context (#1252, merged in #1253).

We passed our DI framework's bounded OnStart context to ActorSystem.Start - the ordinary "give startup 30 seconds" pattern. The node started fine and served traffic. Then the startup budget elapsed: because the startup context was retained as the remoting server's base context, every inbound handler context from that moment on was born already DeadlineExceeded. All cross-node traffic to that node failed permanently, while the process itself looked perfectly healthy. The fix is a one-line context.WithoutCancel on the server's base context. The more general lesson: a library that holds a caller's context for the lifetime of a server creates a footgun no amount of care at the call site can defuse.

2. Cluster bootstrap had no in-process retry (#1257, PR #1258).

During a rolling update, about 16 new pods join a 31-node cluster that already holds data. Partition redistribution plus routing-table convergence can exceed the 10s bootstrap timeout, at which point Start fails and the process exits - so Kubernetes CrashLoopBackOff becomes the de facto retry loop: several restarts and a few minutes of alert noise for a condition that heals in seconds. We raised the bootstrap timeout on our side as a stopgap, but the real fix is a bounded in-process retry with a full engine teardown between attempts. Olric's own join retries do not cover this, because they only retry the memberlist join, not the initial sync and dmap creation that follow it.

Crash recovery on the default replicaCount: an observation we would like to open up

Registry-derived crash recovery is a genuinely nice addition and we adopted it immediately. What we want to share is how it behaves in combination with the default configuration - we did read the note in the docs, and we still had to work around it in practice, which is why we think it is worth putting on the table.

The mechanism. Registry records are keyed by actor name (PutActor -> dmap.Put(composeKey(namespaceActors, addr.Name()), ...)), so a record's partition is chosen by hash(actorName) and has nothing to do with which node the actor actually runs on. ReplicaCount defaults to 1, which in Olric means one primary copy and no backups. So when a node crashes, the partitions it owned are gone with it - and those partitions hold a share of every node's records, including its own.

The consequence. deriveRelocationSetFromRegistry scans the surviving registry and finds the crashed node's records that happened to hash onto other nodes' partitions - statistically about (N-1)/N of them - and silently misses the rest: no error, no warning, and the rebalance still reports success. Silent partial recovery is harder to notice than no recovery at all.

Why "just raise the replica count" is not a free answer. The docs recommend replica count > 1, but GoAkt enforces readQuorum + writeQuorum > replicaCount, so moving to 2 forces writeQuorum = 2 (or readQuorum = 2). With writeQuorum = 2, every registry write must reach two replicas synchronously - so while a node is down, registry writes fail, and grain activation writes the registry. To make recovery-after-crash work, you make the registry fail-during-crash. For a registry that is inherently rebuildable (an actor re-registers on spawn), that is a trade we did not feel able to make. Memory is not the objection - our registry is a few hundred records - the quorum coupling is.

What we do instead. We run an opt-in placement journal on our fork: relocatable placements are recorded to a store we own (Redis, which we already run), and on an unexpected NodeLeft the leader replays that node's journal to synthesize the same PeerState a graceful shutdown would have produced. It needs no quiescence wait, so it runs synchronously in the NodeLeft handler, and it is complete at replicaCount = 1 because it does not depend on Olric's partition replication at all. It also survives something replicaCount = 2 does not: a full-cluster restart wipes the in-memory DMap registry entirely, while the journal is still there.

The proposal. We would be glad to offer this upstream as a purely additive feature: a placement.Journal interface plus a ClusterConfig.WithPlacementJournal(store) option, defaulting to nil so behavior is unchanged for anyone who does not opt in; when configured, relocation prefers the journal's complete record and otherwise falls back to registry derivation exactly as today. The interface is storage-agnostic (an in-memory implementation ships with it; ours in production is Redis-backed), so no specific store enters core. It has been running in production, and it has already survived a full upstream rewrite of the relocation machinery - re-landing it on the new dispatcher/worker architecture cost two one-line hooks and one wiring change, which is a fair proxy for its long-term maintenance weight.

If that shape does not belong in core, we understand - it sits close to the boundary drawn in #1245, and the storage contract is precisely the debatable part. Even without the feature, a startup warning when replicaCount == 1 and relocation is enabled might save someone a bad afternoon.

What we are building on top

Per the boundary write-up in #1245: the runtime-shaped capabilities we need beyond the actor core (durable at-least-once jobs with retry and DLQ, a cluster gateway that terminates TLS for the whole cluster, a KV/lock surface, rate limiting) live on our fork today and will move into a companion library built on GoAkt's public API, in the eGo mould. We agreed with that rubric then and we still do: the core staying small is exactly what let us embed it in our own process instead of adopting an entire platform.

Two questions we would like your view on

1. Does the placement journal shape (an optional interface, off by default) belong in core, or must it live in a satellite library?
2. If it is a satellite concern, what is the minimum core should do so that users on the default configuration do not believe they have complete crash recovery when they do not?

[Tochemey]

Thanks for taking the time to write this up. The two issues you filed (#1252, #1257) already made the library better, and the crash recovery observation is worth a detailed answer, so here it is.

One correction first, because it changes the picture: GoAkt does not enforce readQuorum + writeQuorum > replicaCount. That formula only appears in the godoc as guidance against stale reads. The enforced constraints (from Olric) are quorum >= 1 and quorum <= replicaCount. So replicaCount=2, writeQuorum=1, readQuorum=1 is legal today, and for the registry it's the right setting:

• Quorum 1 doesn't weaken grain single activation. That's arbitrated by an atomic put-if-absent on the partition's primary owner, not by read/write quorum overlap. The quorum-sum rule buys read-your-writes consistency, which a rebuildable registry doesn't need. GoAkt also runs Olric with read repair on, so a replica that missed a write gets healed on the next read.
• Replication is synchronous (GoAkt pins Olric's sync replication mode) and the backup receives every write before the primary acknowledges it. Quorum 1 doesn't skip the backup.
• On writes failing during a crash: that only happens in the failure detection window, the few seconds between the crash and the routing table dropping the dead member. In that window, writes to partitions where the dead node was primary or backup fail. After the routing table updates, writes go to the reassigned owners and succeed for the rest of the outage. A comparable window already exists at replicaCount=1 for partitions whose primary died, so this isn't a new cost of raising the count.

There was a real hole behind your concern until recently, though. Before Olric v0.3.13, after a crash the survivor's only copy of a partition sat in its backup fragment, where the balancer could ship it away before promoting it. A second failure could destroy data even at replicaCount=2, and the registry scan could run before the surviving copy was visible to primary reads. v0.3.13 fixes this: backup fragments are merged into the local primary as the first step of every balancing cycle, through the version-aware merge path. And since a member only acks a rebalance epoch after a balancing cycle with nothing left to move, and GoAkt emits NodeLeft only after the epoch completes plus a quiet period, the crash recovery scan now runs after promotion has finished cluster-wide. Not by luck, by ordering.

So with v0.3.13 and replicaCount=2, registry-derived crash recovery is complete for a single-node crash. To be exact about what stays open, since your point about silent partial recovery being the worst failure mode is correct:

1. Losing both nodes that hold the two copies of a partition at the same time loses its records, and the derivation can't know what it missed.
2. A write whose backup replication failed at the command level gets acknowledged with a single copy. If that key is never read again before its primary crashes, it's lost. A narrow race, not steady-state behavior.
3. A full-cluster restart starts with an empty registry. Relocation doesn't cover that case today anyway, since no leader survives to rebalance.

What I'm going to change in core:

1. Move to tochemey/olric@v0.3.13.
2. Change the default replicaCount from 1 to 2, both quorums staying at 1. People who configure nothing get the behavior described above out of the box. You're right that the current default silently under-delivers.
3. Warn at startup when clustering runs with an explicit replicaCount=1, since crash recovery will be partial there.
4. Mark registry-derived recovery as best-effort in the relocation events instead of only in a log line, so you can see programmatically that a recovery was reconstructed rather than snapshot-based, and react to the open cases above.
5. Fix the docs: the relocation page gets the concrete replica and quorum recipe with its trade-offs, and the WithReplicaCount and quorum godoc will say plainly that the quorum-sum formula is a guideline, not a validated constraint.

On question 1, the placement journal: with the new default, the gap it closes narrows to the three cases above. They're real, but they don't justify an external storage contract in core's default path. GoAkt's value is that it runs on nothing but itself. A store the developer has to provide, even optionally, moves the reliability story outside the library for everyone who reads about it.

So: the journal belongs in a companion library, per the boundary from #1245, but I'd rather upstream the hooks than the feature. You said re-landing it after the relocation rewrite cost two one-line hooks and one wiring change. File a feature request describing exactly those integration points. I expect they come down to a notification on relocatable spawn and stop, plus a way to hand in a PeerState for a departed node before the registry derivation runs. If they're as small as that, I'll take them, and your journal lives entirely on public API with no fork.

On question 2, items 2 through 5 above are my answer: the default has to be safe, deviating from it has to warn, partial recovery has to be observable, and the docs shouldn't require reverse-engineering the quorum rules.

[Tochemey]

@StringKe I have an issue that I will work in the coming days. #1260

[StringKe]

Thank you for the correction - and it is a correction we needed. We read readQuorum + writeQuorum > replicaCount in the godoc as an enforced constraint rather than stale-read guidance, concluded that replicaCount=2 would force writeQuorum=2, and reasoned ourselves into a trade-off that does not exist. That was our error, not the docs' alone, and it is exactly the kind of thing we should have verified against Olric's actual validation instead of inferring from prose. replicaCount=2 with both quorums at 1 is the setting we will move to once #1260 lands.

The five changes in #1260 answer our second question completely, and the olric v0.3.13 work behind them is more than we hoped for.

On the journal: we have filed #1262 with the exact integration points, per your request. It comes down to the two you predicted - placement notifications on relocatable spawn/stop, and a chance to hand in a departed node's relocation set before the registry derivation runs - plus a note on the one wire-type question they raise. If those land, our journal moves onto the public API and the fork goes away entirely, which is the outcome we want too.

Separately: the two capabilities that had no business being in core are now out. gateway and jobs are standalone libraries on upstream GoAkt's public API (no fork, no internal imports), and the cluster KV, the distributed lock and the rate limiter are simply gone - your point in #1245 about not publishing a guarantee the substrate cannot honor was right, and the certificate issuance lock in particular now runs on Redis where mutual exclusion is real.

[Tochemey]

@StringKe I am closing this discussion. For feedback on how GoAkt is performing at your organisation, you can kindly use the feedback issue. Thank you very much for helping enhance the framework.