Vision: GoAkt as a complete runtime for multi-replica Go applications

[StringKe]

We run a single Go application deployed as N replicas, built on GoAkt. Today, teams in this position typically compose several external systems around the actor runtime: a task queue plus Redis for durable jobs and cron (Asynq/River), Redis or database locks for coordination, nginx/caddy plus cert tooling in front for TLS and long-lived connections, and hand-rolled registries for presence. Yet the embedded cluster already has the primitives all of these need: membership, consistent placement, put-if-absent, pub/sub. Our wish, and the point of this discussion: GoAkt (core plus its ecosystem) could cover the distributed multi-instance feature set end to end, so applications do not have to stitch together half a dozen dependencies for capabilities the runtime is one narrow API away from providing.

We implemented the full set on our fork (branch self-park on StringKe/goakt), each piece additive-only, zero new dependencies, with unit + multi-node tests, benchmarks, and docs. We are offering all of it upstream, in whatever form you prefer.

Filed today as small, independent PRs (each with its own issue): leader status + change events (#1233/#1239), ephemeral spawn option for high-churn connection actors (#1234/#1240), scheduler introspection (#1235/#1241), cluster single-fire schedules (#1236/#1242), non-actor PubSub subscriptions (#1237/#1243), topic presence introspection (#1238/#1244).

Implemented and ready, but we would like your guidance before filing anything:

1. Persistent scheduler JobQueue - expose go-quartz's quartz.WithQueue via an ActorSystem option, with a serializable delivery-intent envelope so schedules survive restarts (adapters would live outside core, following the discovery.Provider pattern).
2. Public cluster KV + TryLock - a narrow metadata-registry surface (Get/Put/PutIfAbsent with TTL, TryLock) over the embedded Olric store, the same primitive grain single-activation already uses.
3. Cluster-wide rate limiting - a small token-bucket/window limiter on top of that KV surface.
4. Crash relocation - an opt-in placement journal so actors that opted into relocation are respawned after kill -9/OOM node loss, not only after graceful shutdown.
5. jobs package - embedded at-least-once execution: lease/ack with fencing tokens, retry/backoff, dead-lettering, first-class fan-out/fan-in (map/reduce over actors), and an inspector API. We know conveyor covers app-level task queueing as a standalone server, and see this as the complementary embedded, actor-native primitive - happy to align naming/concepts or scope it differently.
6. gateway package - the cluster acting as its own edge tier: WS/SSE connection registry with two-tier delivery (local socket write fast path, cluster routing only cross-node), cluster-shared TLS where each domain is issued exactly once cluster-wide (pluggable issuers; Cloudflare Origin CA implemented with net/http only), and graceful connection draining for rolling deploys.

These interlock: single-fire + a persistent queue give HA cron; the KV/lock surface underpins both the limiter and the gateway's certificate arbitration; ephemeral actors + presence + the subscriber bridge are what make the gateway's connection model clean; the placement journal closes the crash-recovery gap for exclusive long-lived workloads (bot connections and similar).

For items 5 and 6 especially, we are open to any structure you prefer: in-tree packages, separate repos under your org (the ego/ego-contrib pattern), or scoped down to just the core hooks. Everything is runnable on the fork today if you want to inspect behavior before deciding. Which of these would you take as core proposals, which belong in ecosystem repos, and which do you consider out of scope?

[Tochemey]

@StringKe there is already a task queue built on top of GoAkt: https://github.com/conveyorq/conveyor

[Tochemey]

Position

GoAkt should not absorb the runtime feature set proposed (public cluster KV and locks, cluster-wide rate limiting, an at-least-once jobs engine, a persistent scheduler with durable delivery, a TLS edge gateway). These belong in separate libraries built on top of GoAkt, exactly the way eGo (event sourcing, CQRS, saga) and goakt-ebpf (distributed tracing) already are. This is not a "maybe later" or a resourcing decision. It is a boundary decision, and the boundary is load-bearing.

The short version: GoAkt is an engine. The proposal asks the engine to also be the transmission, the fuel system, and the car body. Each of those is a real part of the car. None of them is the engine, and bolting them to the engine block makes the engine heavier, harder to trust, and worse at being an engine.

1. What GoAkt actually is (the bounded context)

GoAkt's domain is the actor model: typed message passing, supervision hierarchies, location transparency, and clustering of actors and grains. Its cluster layer (internal/cluster, backed by the github.com/tochemey/olric fork and Hashicorp Memberlist) exists to serve that domain and nothing wider:

• Membership via SWIM-style gossip (Memberlist). An eventually consistent view of who is in the cluster.
• Placement: a quorum-tunable actor and grain registry (read/write quorum, replica count) that answers "where does actor X live," reconciled continuously by the runtime.
• Coordination for actor concerns: leader election (IsLeader), leadership-change events, and exclusive cluster-wide schedule-fire claiming (ClaimScheduleFire).
• Topics: local and cluster fan-out pub/sub for actor messaging.

Every one of these primitives exists because an actor needs it, and every one is allowed to be advisory and self-healing, because the actor runtime reconciles the truth on top of it. That property is the whole reason GoAkt can sit on an availability-favoring, tunably-consistent substrate (Olric is PA/EC; GoAkt raises the read/write quorum for the placement registry to get stronger reads) and still be correct: placement is reconciled, leadership is self-correcting, a duplicated schedule claim is deduplicated. The moment we expose those same primitives as general-purpose infrastructure with their own guarantee contracts, that safety net disappears, because there is no longer a reconciling runtime above them; the user's code sits directly on the raw guarantee.

This is the crux of the whole discussion. Everything in GoAkt orbits the actor model. The goodies that already ship (leader election, cluster-wide schedule claiming, non-actor pub/sub) exist to serve actors, and the fact that they exist is not a mandate to absorb the rest of a distributed runtime. A few useful primitives falling out of the cluster layer is not a recipe for putting everything into GoAkt.

2. Steelman: why the proposal is attractive

The proposal is not naive, and the pull is real:

• It removes genuine setup friction. Teams running N replicas do stitch together Redis, a queue, a gateway, and a cert story. "Batteries included" is a real developer-experience win for newcomers.
• The primitives look free. Membership, placement, pub/sub, and an embedded KV (Olric DMaps) already exist in-process. Exposing them feels like near-zero marginal cost and "no new dependencies."
• The adoption pitch is clean. "One library runs your whole multi-replica app" is easy to say and easy to sell.

These are true. They are also exactly the arguments that produce a distributed monolith. Every one of them optimizes the first hour of adoption at the expense of the guarantees, the security posture, and the maintainability of the next five years.

3. The load-bearing objection: one substrate, many incompatible guarantee contracts

The central engineering fact is this: the proposed features each publish a different correctness contract, and those contracts require guarantees the substrate does not provide. GoAkt's substrate (the Olric fork) is, by its own README, a PA/EC system under PACELC: under a network partition it favors availability over consistency, resolving conflicts by last-write-wins over gossip membership, with no consensus layer (there is no Raft, Paxos, or linearizability anywhere in the fork). GoAkt tunes the read/write quorum up for the placement registry, which buys Cassandra-class tunable consistency (raising readQuorum + writeQuorum above the replica count prevents stale reads), but tunable quorum is still not consensus, and the fork's own distributed lock is, verbatim, "recommended for efficiency purposes in general, rather than correctness." That profile is the correct foundation for a reconciled actor placement layer. It is the wrong foundation for the user-facing correctness primitives being requested.

3.1 Consistency-model mismatch

Consider what each requested feature actually promises a user:

Requested feature	Contract a user will assume	Guarantee it truly needs
Public distributed lock (Lock)	mutual exclusion (only one holder)	linearizable consensus (Raft/etcd)
Cluster-wide rate limiter	a global ceiling is enforced	a linearizable or CRDT counter with bounded error
At-least-once jobs, leasing, DLQ	no job silently lost or double-run past its guarantee	durable, fenced, consensus-backed queue
Persistent scheduler / durable delivery	a scheduled fire survives crashes and fires once	durable storage plus fencing

The Olric fork's own README is explicit about the lock: "there is no guarantee about reliability in the case of network partitioning. The lock implementation is recommended for efficiency purposes in general, rather than correctness," and "if you cannot tolerate losing strong consistency under network partitioning, you need to use a different tool for locking." A distributed lock that can be held by two nodes at once is not a lock; it is a race with good intentions. The same reasoning applies to a rate limiter: the fork's Incr/Decr are atomic on a partition's primary owner but still last-write-wins across a partition, and there is no compare-and-swap, so during a split each side admits the full budget and increments are lost on merge. An at-least-once queue built on the same store can silently overwrite a lease the same way.

Newcomers, the exact audience the proposal courts, will read "distributed lock" and "at-least-once" and trust them the way they trust Redis or etcd. They will build correctness on them. They will get bitten during a partition, in production, and the bug report lands on the GoAkt maintainer, who then owns the impossible task of making PA/EC primitives behave like CP ones. A guarantee you cannot honor is a liability you signed, not a feature you shipped.

Note the asymmetry with what GoAkt already does. Its own ClaimScheduleFire is built on exactly the primitive a public lock would expose: an atomic put-if-absent with a TTL (Put with NX + EX, the same SETNX-plus-expiry that underlies Olric's Lock). GoAkt relies on it comfortably because it uses it only where best-effort is acceptable: a duplicated schedule fire is deduplicated by the runtime, and a stale claim is reclaimed when its TTL lapses (single grain activation, reconciled). It never dresses that primitive up as a fenced lock and hands it to users. Leader election is safe for the same reason (a stale leader is superseded, not catastrophic); the registry is safe because the runtime treats it as advisory and reconciles it. A public Lock, KV, or rate limiter has no reconciling layer above it, so the user's correctness rests directly on the raw primitive. The guarantee is the product, and at that layer the guarantee is not there.

3.2 Public-API ossification (Hyrum's Law)

"No new dependencies" is the most expensive sentence in the proposal. It is true only because it works by exposing Olric as a public API surface. Today Olric lives in internal/cluster. It is an implementation detail: a fork we control and can swap for another engine (a different KV, a Raft library, a rewrite) without any user noticing. The registry, topics, and placement are all mediated by GoAkt's own types.

Publish a cluster.KV / Lock surface and that freedom is gone permanently. By Hyrum's Law, every observable behavior of Olric (its TTL semantics, its LWW timing, its scan ordering, its failure modes under partition) becomes a contract that real users depend on. We would never again be able to change the cluster engine without a major version break and a migration guide. We would be converting a reversible internal decision into an irreversible public one, in exchange for a convenience we can deliver from a satellite instead.

3.3 Trust boundary and security blast radius

The gateway package (WebSocket/SSE registry, cluster-wide TLS, cert lifecycle, graceful draining) is a category error inside an actor library. It places an internet-facing edge tier inside the same process, trust boundary, and release cadence as the message-passing core.

An edge proxy and an actor runtime have different threat models, different audit requirements, and different CVE surfaces. TLS termination and certificate management alone are a serious, never-ending security liability. Coupling them means every actor-core patch drags the gateway's attack surface along for review, and every gateway CVE forces a core release. A library that is embedded deep inside other people's processes must keep its blast radius small. An edge tier is the opposite of small.

3.4 Coupled release cadence and versioning

A monolith forces lockstep versioning. The jobs engine's semantics will evolve (backoff policy, DLQ format, leasing model). The gateway's TLS defaults will change as the ecosystem moves. In a monolith each of those is a GoAkt (v4 -> v5) event that also drags the actor core, its users, and its API stability along with it. As independent satellites, each subsystem versions on its own clock and its churn never destabilizes the actor core. The core's job is to be the boring, stable thing everything else stands on. You cannot be the stable foundation and the fast-moving jobs engine in the same go.mod.

3.5 Verification and support burden

Each of these primitives is only trustworthy if it is verified to a high bar. An at-least-once queue, a fenced lease, a distributed lock, and a rate limiter are precisely the class of system that needs Jepsen-grade partition testing before anyone should build correctness on it. The core team cannot own that depth of verification for six new subsystems on top of the actor model itself. Shipping them unverified under trustworthy names ("at-least-once," "lock") is worse than not shipping them, because it manufactures false confidence. As satellites, each library owns and states its own guarantees and its own test regime, and can be as rigorous (or as explicitly best-effort) as its authors choose without putting the core's credibility on the line.

4. The pattern is already in the codebase

GoAkt already solved this problem, twice, and the proposal would undo the solution.

• discovery.Provider is the template. Core defines a narrow interface (ID, Initialize, Register, Deregister, DiscoverPeers, Close) and depends only on the interface. Every integration (kubernetes, nats, consul, etcd, dnssd, mdns, static, selfmanaged) is a satellite package that implements it. Core never imports Kubernetes or Consul. This is the Open-Closed and Dependency-Inversion principles made concrete: the runtime is open to extension and closed to modification, and it depends on abstractions, not on the plugins.
• eGo (github.com/Tochemey/ego) is the proof at the feature level. Event sourcing, durable state, CQRS projections, and saga orchestration, all genuinely heavier and more opinionated than the actor core, live in a companion framework on top of GoAkt. Nobody argues they should be in core.
• goakt-ebpf does the same for distributed tracing, as an out-of-process sidecar.

The ecosystem's own precedent is unambiguous: heavyweight, opinionated, guarantee-bearing capabilities are satellites; the actor core stays small and stable. There is a crucial constraint on this pattern, and it is the whole point: discovery.Provider is admissible in core because discovery is an actor-model concern (a cluster of actors must find its peers to form). Core earns an SPI only when the extension point is itself expressed in actor-model terms. It must never grow a KV SPI, a lock SPI, a job-store SPI, or a scheduler-store SPI, because those contracts have nothing to do with the actor model. Everything in #1245 that is generic infrastructure is built by a satellite on the public actor and cluster API, with no new core interface at all.

5. Decision rubric: core vs satellite

A capability belongs in the actor core only if all hold:

1. It manipulates actor concerns directly (lifecycle, messaging, supervision, placement, or the clustering thereof).
2. It needs only guarantees the actor runtime already provides and reconciles (advisory, self-healing, no new user-facing correctness contract).
3. If it is an extension point (an SPI), its contract is expressed in actor-model terms, the way discovery.Provider is about finding peers for a cluster of actors. Core never adds an SPI shaped like generic infrastructure (a KV, a lock, a store, a queue), even "just an interface."

Everything else is a satellite, built on GoAkt's public actor and cluster API. A capability is a satellite if it does any of:

• Introduces a new consistency or durability guarantee the substrate does not honor as a user-facing contract (locks, rate limits, at-least-once, durable delivery).
• Exposes a new public protocol or storage surface that ossifies an internal choice (public KV over Olric).
• Opens a new trust boundary (an internet-facing gateway, TLS termination).
• Carries an independent release cadence or verification regime (a jobs engine, an edge tier).

6. Feature-by-feature verdict

#	Proposed feature	Verdict	Why
1	Persistent scheduler / durable delivery	Satellite	Scheduling messages to actors is core and already exists. Durability is a storage contract with nothing actor-shaped about it, so it earns no core SPI; a satellite persists and replays over the public API.
2	Public cluster KV + lock	Hard no in core	Ossifies Olric as public API (3.2) and publishes as a correctness primitive a lock the substrate honors only best-effort (3.1). A KV and a lock are not actor concerns. Wrong on every axis.
3	Cluster-wide rate limiting	Satellite	Correctness needs a linearizable or bounded-error CRDT counter; an eventually-consistent counter leaks the budget per partition.
4	Crash relocation / placement journal	Satellite (policy already core)	The respawn policy is placement, already core. The durable journal is generic storage, not an actor concern, so it gets no PlacementJournal SPI. A satellite observes lifecycle + membership events and re-drives Spawn on the public API. See section 7.
5	Jobs package (at-least-once, leasing, retry, DLQ, map/reduce)	Satellite	This is a workflow engine (Temporal territory). It is exactly what eGo-style companions are for.
6	Gateway (WebSocket/SSE, cluster TLS, draining)	Separate project entirely	New trust boundary and security blast radius (3.3). Does not belong in the same process family as the actor core.

7. The item that looks like it needs a core hook, and does not

Crash relocation (4) is the proposal that comes closest to a core concern, because placement is core. Deciding where a dead node's actors respawn is squarely the actor runtime's job, and GoAkt already does it. What tempts people toward core machinery is the durable memory of "what was running where," so placement can be rebuilt after a full-cluster restart. That durability is generic storage, not an actor concern, so it does not earn a core SPI, not even "just an interface."

The resolution needs nothing new in core. GoAkt already emits the actor-model signals a satellite needs: actor and grain lifecycle, membership change events, and the placement queries. A relocation satellite subscribes to those, persists placement to whatever store it chooses (with whatever durability guarantee it is willing to stand behind), and on restart re-drives the ordinary public Spawn path. Core stays actor-shaped; the storage engine lives outside it. This is the discipline the whole document argues for: extend GoAkt through the actor model, not by bolting a storage contract onto it.

8. Field precedent

This is not a GoAkt-specific opinion; it is how every serious actor and runtime ecosystem is factored.

• Erlang/OTP: the core is processes, links, supervisors, and gen_server. Mnesia (storage), cowboy (HTTP), and the rest are separate applications, not part of the process runtime.
• Akka: akka-actor is the core. HTTP (akka-http), persistence/event-sourcing (akka-persistence), and even cluster sharding are separate modules with separate guarantees, versioned independently. Even the reference implementation of the model modularizes exactly along these lines.
• Dapr / Temporal: the platform-runtime feature set the proposal describes (state stores, locks, rate limits, durable jobs, gateways) is what these systems are. They are entire platforms with dedicated teams and their own operational contracts. Reimplementing a slice of them inside an embeddable actor library gets the worst of both: platform-sized promises with library-sized verification.

9. Recommendation

1. Keep the actor core small. Its value is being a well-behaved, embeddable library that composes into other people's processes. Absorbing the runtime turns it into a platform users must adopt wholesale, which raises the cost of adoption rather than lowering it.
2. Add SPIs only for actor-model concerns. discovery.Provider is the bar: an extension point earns a place in core only when its contract is about actors, messaging, placement, or clustering. Durable scheduling, placement journaling, KV, and locks are not actor concerns and get no core SPI, not even as an interface; a satellite builds them on the public API.
3. Build the rest on top. KV/locks, rate limiting, the jobs engine, and the gateway become companion libraries (the eGo / goakt-ebpf model), each owning and honestly stating its own guarantees.
4. Never publish a guarantee the substrate cannot honor. No public lock or KV sold as a correctness primitive, no "at-least-once" label, and no "distributed rate limit" unless and until it sits on a consensus-backed foundation, and that foundation is a deliberate, separately-verified choice, not an accidental side effect of exposing Olric.

[StringKe]

Thank you for taking the time to write this up - the rubric and the guarantee-contract analysis are genuinely valuable, and we agree with the boundary. We will keep building on the public actor and cluster API, following the eGo pattern, and take our time to get it right before anything is published. And thanks again for landing the four PRs so quickly - the collaboration this week has been outstanding.