lockstep v0.2.0

lockstep v0.2.0 runs lockstep.New() on a pool of worker goroutines, so turns of different actors run in parallel. It also adds event-sourced persistent actors, helpers for duplicate and stale messages, logs and metrics, more lockstep-vet checks, and a benchmark suite. Code that compiles against v0.1.0 compiles against v0.2.0, but lockstep.New() behaves differently in three ways, listed under Breaking changes. The README shows each feature with an example.

Requirements

• Go 1.26 or later.
• faultline v0.1.2, pinned in go.mod as in v0.1.0. Only lockstep/sim imports it; a program that does not import sim never links faultline.

Modules and tags

This release tags two modules:

• github.com/hmdsefi/lockstep at v0.2.0: the runtime, lockstep/sim, lockstep/persist, lockstep/persist/filestore, and lockstep/dedup.
• github.com/hmdsefi/lockstep/analysis at analysis/v0.2.0: the analyzers, lockstep-vet, and the golangci-lint plugin.

examples/raft, examples/orders, and bench are modules of their own, built against the lockstep code in the same checkout. They are not tagged.

Breaking changes

The API only adds to v0.1.0. These three changes are in what lockstep.New() does at run time.

Turns run in parallel

In v0.1.0, one goroutine ran every actor turn. In v0.2.0, lockstep.New() runs turns on runtime.GOMAXPROCS(0) workers, so two actors that share memory instead of exchanging messages now race. The turns of one actor still never overlap.

Run your tests with -race, and run lockstep-vet, whose new LS200 and LS210 checks report memory shared through messages and package variables. Until the sharing is gone, WithWorkers(1) runs every turn on one worker:

sys := lockstep.New(lockstep.WithWorkers(1))

Long turns fail

In v0.1.0, WithWatchdog was accepted and had no effect. In v0.2.0, a turn that runs for 5 seconds (DefaultWatchdogTimeout) is logged at Error with its worker's stack. When the turn returns, the incarnation fails with an error that matches ErrTimeout, and its supervisor restarts, stops, or escalates it.

Slow work belongs in an effect started with lockstep.Do. To keep long turns, raise the timeout with WithWatchdog(d) or turn the watchdog off:

sys := lockstep.New(lockstep.WithWatchdog(0))

Memory admission follows the Go memory limit

In v0.1.0, WithAdmission was accepted and had no effect. In v0.2.0, when a Go memory limit is set (GOMEMLIMIT or debug.SetMemoryLimit), admission stops once the live heap reaches 90% of the limit and resumes when it falls to 80%. While it is stopped, Spawn returns an error that matches ErrOverloaded, and each Send and Ask to a user actor is handled as if the target's mailbox were full. Without a memory limit, nothing changes.

To turn memory admission off:

sys := lockstep.New(lockstep.WithAdmission(0, 0))

Added

The production engine

• lockstep.New() runs turns on a fixed pool of worker goroutines. Messages from one sender to one receiver arrive in send order, and timers and effect results run on the same workers.
• WithWorkers sets the pool size (default runtime.GOMAXPROCS(0)). WithThroughput sets how many user messages an actor handles in one turn before it yields its worker (default 32), and the Spawn option WithActorThroughput sets it for one actor.
• WithEffectPool(workers, queue) sets the bounds on effects started with lockstep.Do, which v0.1.0 fixed at 64 running and 1024 waiting.
• Simulation checks these options and then ignores them: each turn there handles one message, in an order the seed decides.

Overload protection

• Memory admission: New samples the live heap every 100 ms. WithAdmission(high, low) sets byte thresholds in place of the memory limit. Actors that called ctx.SubscribeOverload get an Overloaded signal when admission stops and another when it resumes.
• Stuck turns: a watchdog reports every turn that has run for the WithWatchdog timeout. Until the turn returns, user messages to its actor are handled as if the mailbox were full, and a Stop stops the actor once the turn returns, without a restart.
• Goroutine count: once a second, the system counts the goroutines of the whole process. When the count reaches WithGoroutineWarning(n) (default DefaultGoroutineWarning, 10,000), it logs one warning with the ten functions that hold the most goroutines.

Logs and metrics

• WithLogger sets the log/slog logger for the runtime's events: failures and restarts, shutdown, the overload events, and, at Debug, full mailboxes, shed and dropped messages, and dead letters. Without it, the logger is slog.Default() as it is when New runs.
• ctx.Logger() returns that logger with the actor's node, path, and uid attributes, and ctx.Logf logs through it at Info.
• WithMetrics takes your implementation of lockstep.Metrics, whose Counter and GaugeFunc methods register series with MetricLabel labels. lockstep ships no adapter. Architecture lists every series and shows an example adapter.
• In simulation, ctx.Logger() and ctx.Logf write to the node's faultline kernel log, so the lines go into trace.jsonl and the trace hash. In v0.1.0, ctx.Logf reached only the trace. Metrics are off in simulation.

Persistent actors (lockstep/persist)

• persist.Spawn starts an event-sourced actor: an EventSourced[C, E, S] whose Handle turns a command into events and whose Apply applies an event to the state. lockstep writes the events to a persist.Journal and fsyncs them before it applies them and runs the callbacks registered with ctx.Then. A reply sent from a callback therefore means the events are on disk.
• A start or restart recovers the state from the newest snapshot and the events after it. Config.SnapshotEvery and ctx.Snapshot save snapshots to a persist.SnapshotStore, and Config.PruneEvents deletes the events that the previous snapshot covers.
• persist.JSON is a codec for events and state. An actor can implement Recoverer to learn when recovery finished, and Stopper and SignalReceiver for the Stopped hook and signals.
• persist/filestore is the reference journal and snapshot store. filestore.Open(dir) uses the operating system's files. On recovery, it discards a torn write at the end of the journal and reports its size in Bounds.Repaired.
• In simulation, s.Volume(node) returns the node's simulated disk as a filestore.FS, and filestore.New runs a store on it. faultline's disk faults, such as lost or torn writes that were not synced and failed syncs, then run through the real recovery code. lockstep/leak fails a seed when an actor stops with a file still open.

Delivery helpers (lockstep/dedup)

• Window holds idempotency keys and tombstones, Replies caches the first reply to each request, and Versions accepts a version for an entity only when it is greater than the one it holds. Config.TTL and Config.Capacity bound each of them, and each encodes to JSON, so a persistent actor can keep it in its snapshots.
• Message delivery explains what your actors must handle themselves: a retried request that arrives twice, and a Cancel that arrives before its Create. examples/orders is an order service whose simulation tests find both bugs from a seed.

lockstep-vet

lockstep-vet and the golangci-lint plugin have these new checks, so code that passed v0.1.0's checks can get new findings:

• LS200: a variable used after the message that holds it was sent.
• LS201: a type marked //lockstep:immutable that is not immutable.
• LS210: actor code that writes a package variable, or puts the actor's own state in a message.
• LS220: a reply from lockstep.Ask that is never awaited or piped, and an ignored error from lockstep.Send or lockstep.Spawn.
• LS230: a message type with no case in Receive.
• LS240: an actor field that holds a resource and is neither passed to ctx.Own nor released in Stopped.

The markers //lockstep:immutable and //lockstep:resource go in a type's doc comment and apply in every package that imports the type.

go install github.com/hmdsefi/lockstep/analysis/cmd/lockstep-vet@v0.2.0
go vet -vettool=$(which lockstep-vet) ./...

Benchmarks

• The bench module runs ping-pong, fan-out, idle actors, send latency, and ask latency on lockstep, protoactor-go, Hollywood, and GoAkt, and measures how many seeds per second sim.Simulate runs.
• The README's Benchmarks table and bench/RESULTS.md show the results of one CI run on main. CI fails when a lockstep result goes over its budget in bench/budgets.txt.

Fixed

• System.Shutdown took time quadratic in the number of sibling actors under one parent (#298). Stopping children, removing death watches, and removing overload subscriptions now take linear time.

Known limitations

• lockstep cannot preempt a turn. The watchdog reports a stuck turn and isolates its actor, but the turn keeps its worker, and Shutdown with a deadline returns the context's error when the deadline passes.
• No remoting or clustering: actors on different nodes exchange messages only in simulation. lockstep ships no tracing or metrics adapter. The roadmap tracks later releases.
• persist/filestore keeps files on a local disk. Use one filestore.Store per directory: two Stores on the same directory, in one process or in two, are not detected...

lockstep v0.1.0

lockstep v0.1.0 is the first release: typed local actors on lockstep.New(), a seeded multi-node simulation in lockstep/sim built on faultline, and lockstep-vet, which checks that actor code stays deterministic. The README walks through a first actor, a simulation test, and a replayed seed.

Requirements

• Go 1.26 or later.
• faultline v0.1.2, pinned in go.mod. Only lockstep/sim imports it; a program that does not import sim never links faultline.

Modules and tags

This release tags two modules:

• github.com/hmdsefi/lockstep at v0.1.0: the runtime and lockstep/sim.
• github.com/hmdsefi/lockstep/analysis at analysis/v0.1.0: the analyzers, lockstep-vet, and the golangci-lint plugin.

examples/raft is a module of its own, built against the lockstep code in the same checkout. It is not tagged.

Added

Actors and messages

• Typed actors: Actor[M] receives one message type M, and Spawn returns a PID[M], so a message of the wrong type does not compile. Spawn takes a factory that runs at every start and restart.
• Send and SendContext deliver a message; Stop stops an actor by Address.
• Request and reply: Ask returns a *Reply[R] and passes a ReplyTo[R] into the request. ReplyTo.Send and ReplyTo.Fail answer it exactly once, WithTimeout sets the deadline, and NewReply creates a reply without a target.
• Reply.Await waits outside actors. Inside an actor it returns ErrAwaitInActor; Pipe delivers the reply to the actor as one of its own messages.
• Signals from the runtime arrive through SignalReceiver[M], never as an M: Terminated, ChildFailed, and Overloaded.
• Watch and Unwatch subscribe to a Terminated signal when another actor stops for good.
• Names and lookup: Lookup[M] finds a registered actor by path and type, and PIDAt addresses whichever actor is registered at a path on a node.
• Context gives actor code its runtime: Self, Parent, Now, Rand, Logf, Fail, Stop, and Own.

Mailboxes and overload

• Bounded mailboxes with a per-actor WithMailbox or system-wide WithDefaultMailbox capacity and policy. The default is 1024 messages with FailSend.
• Four full-mailbox policies: FailSend (Send returns ErrMailboxFull), DropNewest, DropOldest, and BlockSender, which makes an outside sender wait. Timer messages, piped replies, and effect results never count toward capacity and are never refused.
• Opt-in memory estimates: WithMemoryEstimates, messages that implement Sizer, actors that implement MemSizer, and System.Estimates for the largest actors.
• Admission: when a system sheds, Spawn and Send return ErrOverloaded, and actors that called Context.SubscribeOverload get Overloaded signals. In v0.1.0, shedding happens in simulation only (see Known limitations).

Lifecycle and supervision

• Lifecycle hooks: Starter.Started runs before the first message of an incarnation, and Stopper.Stopped runs once per incarnation, on stop, before every restart, and after a failure. A resource registered with Context.Own is closed after Stopped.
• A panic or Context.Fail fails the actor; a panic arrives as a *PanicError with its stack.
• Supervision strategies OneForOne and AllForOne, with directives DirectiveRestart, DirectiveStop, DirectiveEscalate, and DirectiveResume. Strategy limits restarts per window (MaxRestarts, Window) and backs off between them (MinBackoff, MaxBackoff). WithStrategy sets it per actor and WithGuardianStrategy for top-level actors.
• Stopping a parent stops its children first, and StopReason tells Stopped why: Normal, Failed, ParentStopping, Shutdown, Restarting, or NoActor.
• The actor cap: WithMaxActors limits live actors per system (default DefaultMaxActors, 1,000,000). Past it, Spawn returns ErrActorCap; one percent of the cap is held back for restarts and system actors.
• System.Shutdown stops user actors, then system actors, and waits for every Stopped hook.

Timers

• Context.After delivers a message to the actor once, and Context.Every delivers it on a period, skipping a firing while the previous message is still in the mailbox. Context.CancelTimer cancels either.
• Timers run on the engine clock: wall time on lockstep.New(), the node's virtual clock in simulation.

Effects

• Do runs an Effect[T] (a network call, file I/O, a query) outside the actor's turn and delivers its result as a message through a done function that runs on the actor's turn.
• WithEffectTimeout sets the timeout (default DefaultEffectTimeout, 30 s); Context.CancelEffect cancels a pending effect.
• On lockstep.New(), effects run on worker goroutines, at most 64 at a time with up to 1024 waiting; past that, done gets ErrOverloaded.
• In simulation, Effect.Run is never called: Sim.HandleEffect answers each effect by name with a value, an error, and a virtual latency.

Simulation (lockstep/sim)

• sim.Simulate and sim.SimulateSeeds run a test body once per seed on faultline's simulation kernel, with virtual time and a simulated network. sim.Options embeds faultline.Options.
• Multiple lockstep nodes per run: Sim.Node adds a server and Sim.Client a client node. Each boot gets a new System, and actors on different nodes exchange messages over the simulated network.
• sim.Send and sim.Ask drive the run from the test body. Sim.System, Sim.World, and Sim.Counts expose the run's state.
• Faults: faultline's planners and injector (partitions, crashes and restarts, pauses, link faults, clock faults) apply to lockstep nodes. lockstep adds Sim.DropBetween and Sim.DelayBetween for messages between two actors, Sim.InjectFail and Sim.StopActor for single actors, and memory pressure with Sim.Pressure and Sim.PressureFor.
• Count-based admission budgets: sim.Budget sets per-node limits on live actors and estimated bytes, with a resume mark, so shedding replays exactly.
• Checks that fail a seed: lockstep/deadlock reports a cycle of pending asks as lockstep: deadlock: A → B → A; lockstep/leak reports an actor that stops or restarts with an effect still pending, or a resource that fails to close; lockstep/no-effect-handler reports an effect with no handler (ErrNoEffectHandler); lockstep/misuse reports API misuse such as Await on a pending reply.
• Exact replay: a failing seed prints a FAULTLINE_SEED=... command that reruns it with the same trace hash. LOCKSTEP_CHECK_DETERMINISM=1 runs every seed twice and compares the traces.

lockstep-vet and the golangci-lint plugin

• LS100 reports nondeterminism and blocking in actor code: the wall clock, global and crypto randomness, goroutines, select, map iteration order, environment variables, channel operations, locks, and blocking network, file, database, and process calls.

• LS110 reports Reply.Await in actor code.

• LS001 and LS002 report malformed and unknown //lockstep: directives. //lockstep:allow LSnnn <reason> suppresses one finding.

• lockstep-vet runs the analyzers with go vet -vettool:

  go install github.com/hmdsefi/lockstep/analysis/cmd/lockstep-vet@v0.1.0
  go vet -vettool=$(which lockstep-vet) ./...

• github.com/hmdsefi/lockstep/analysis/golangci registers the same analyzers as the golangci-lint module plugin lockstep, with a disable setting that leaves out analyzers by name.

The Raft demo

• examples/raft runs a Raft key-value cluster of three or five servers, written as lockstep actors, through partitions, crashes, paused servers, and lossy links. It checks Raft's safety properties after every event, and checks the client history against the committed log when the run ends.
• Three planted bugs (vote-ignores-log, stale-read, vote-not-durable) each fail within the first 300 seeds of their fault preset, and TestDemoPartitionBug finds and replays one.
• The demo passes lockstep-vet with no //lockstep:allow directives.

Known limitations

• lockstep.New() runs on a local engine: one goroutine runs every actor turn, so actors interleave but never run in parallel. There is no pooled production scheduler.
• WithAdmission, WithWatchdog, and WithGoroutineWarning are accepted and validated but have no effect: lockstep.New() does not read the live heap, watch for stuck turns, or count goroutines, and never sheds. Admission and Overloaded signals work in simulation through sim.Budget and memory pressure.
• No remoting, clustering, persistence, or OpenTelemetry. Actors on different nodes exchange messages only in simulation. The roadmap tracks later releases.
• System.Shutdown takes time quadratic in the number of sibling actors under one parent (#298).
• An ask deadlock through a PID built with PIDAt is not reported as a deadlock; the seed ends by the ask's timeout instead. It never causes a false failure (#299).
• Replay from a seed (FAULTLINE_SEED) is exact, down to the trace hash. Replay from a schedule.json file (FAULTLINE_SCHEDULE) reproduces the same failure and the same lockstep records, but its trace hash differs from the original run's, because faultline v0.1.2 records schedule-driven faults differently from planned ones.
• Memory pressure (Sim.Pressure, Sim.PressureFor) and the message rules of Sim.DropBetween and Sim.DelayBetween draw from lockstep's own random streams, not faultline faults. A seed replays them exactly, but schedule.json does not list them, and faultline cannot minimize them.
• The API can change before v1.0.