lockstep v0.2.0

lockstep v0.2.0 runs lockstep.New() on a pool of worker goroutines, so turns of different actors run in parallel. It also adds event-sourced persistent actors, helpers for duplicate and stale messages, logs and metrics, more lockstep-vet checks, and a benchmark suite. Code that compiles against v0.1.0 compiles against v0.2.0, but lockstep.New() behaves differently in three ways, listed under Breaking changes. The README shows each feature with an example.

Requirements

• Go 1.26 or later.
• faultline v0.1.2, pinned in go.mod as in v0.1.0. Only lockstep/sim imports it; a program that does not import sim never links faultline.

Modules and tags

This release tags two modules:

• github.com/hmdsefi/lockstep at v0.2.0: the runtime, lockstep/sim, lockstep/persist, lockstep/persist/filestore, and lockstep/dedup.
• github.com/hmdsefi/lockstep/analysis at analysis/v0.2.0: the analyzers, lockstep-vet, and the golangci-lint plugin.

examples/raft, examples/orders, and bench are modules of their own, built against the lockstep code in the same checkout. They are not tagged.

Breaking changes

The API only adds to v0.1.0. These three changes are in what lockstep.New() does at run time.

Turns run in parallel

In v0.1.0, one goroutine ran every actor turn. In v0.2.0, lockstep.New() runs turns on runtime.GOMAXPROCS(0) workers, so two actors that share memory instead of exchanging messages now race. The turns of one actor still never overlap.

Run your tests with -race, and run lockstep-vet, whose new LS200 and LS210 checks report memory shared through messages and package variables. Until the sharing is gone, WithWorkers(1) runs every turn on one worker:

sys := lockstep.New(lockstep.WithWorkers(1))

Long turns fail

In v0.1.0, WithWatchdog was accepted and had no effect. In v0.2.0, a turn that runs for 5 seconds (DefaultWatchdogTimeout) is logged at Error with its worker's stack. When the turn returns, the incarnation fails with an error that matches ErrTimeout, and its supervisor restarts, stops, or escalates it.

Slow work belongs in an effect started with lockstep.Do. To keep long turns, raise the timeout with WithWatchdog(d) or turn the watchdog off:

sys := lockstep.New(lockstep.WithWatchdog(0))

Memory admission follows the Go memory limit

In v0.1.0, WithAdmission was accepted and had no effect. In v0.2.0, when a Go memory limit is set (GOMEMLIMIT or debug.SetMemoryLimit), admission stops once the live heap reaches 90% of the limit and resumes when it falls to 80%. While it is stopped, Spawn returns an error that matches ErrOverloaded, and each Send and Ask to a user actor is handled as if the target's mailbox were full. Without a memory limit, nothing changes.

To turn memory admission off:

sys := lockstep.New(lockstep.WithAdmission(0, 0))

Added

The production engine

• lockstep.New() runs turns on a fixed pool of worker goroutines. Messages from one sender to one receiver arrive in send order, and timers and effect results run on the same workers.
• WithWorkers sets the pool size (default runtime.GOMAXPROCS(0)). WithThroughput sets how many user messages an actor handles in one turn before it yields its worker (default 32), and the Spawn option WithActorThroughput sets it for one actor.
• WithEffectPool(workers, queue) sets the bounds on effects started with lockstep.Do, which v0.1.0 fixed at 64 running and 1024 waiting.
• Simulation checks these options and then ignores them: each turn there handles one message, in an order the seed decides.

Overload protection

• Memory admission: New samples the live heap every 100 ms. WithAdmission(high, low) sets byte thresholds in place of the memory limit. Actors that called ctx.SubscribeOverload get an Overloaded signal when admission stops and another when it resumes.
• Stuck turns: a watchdog reports every turn that has run for the WithWatchdog timeout. Until the turn returns, user messages to its actor are handled as if the mailbox were full, and a Stop stops the actor once the turn returns, without a restart.
• Goroutine count: once a second, the system counts the goroutines of the whole process. When the count reaches WithGoroutineWarning(n) (default DefaultGoroutineWarning, 10,000), it logs one warning with the ten functions that hold the most goroutines.

Logs and metrics

• WithLogger sets the log/slog logger for the runtime's events: failures and restarts, shutdown, the overload events, and, at Debug, full mailboxes, shed and dropped messages, and dead letters. Without it, the logger is slog.Default() as it is when New runs.
• ctx.Logger() returns that logger with the actor's node, path, and uid attributes, and ctx.Logf logs through it at Info.
• WithMetrics takes your implementation of lockstep.Metrics, whose Counter and GaugeFunc methods register series with MetricLabel labels. lockstep ships no adapter. Architecture lists every series and shows an example adapter.
• In simulation, ctx.Logger() and ctx.Logf write to the node's faultline kernel log, so the lines go into trace.jsonl and the trace hash. In v0.1.0, ctx.Logf reached only the trace. Metrics are off in simulation.

Persistent actors (lockstep/persist)

• persist.Spawn starts an event-sourced actor: an EventSourced[C, E, S] whose Handle turns a command into events and whose Apply applies an event to the state. lockstep writes the events to a persist.Journal and fsyncs them before it applies them and runs the callbacks registered with ctx.Then. A reply sent from a callback therefore means the events are on disk.
• A start or restart recovers the state from the newest snapshot and the events after it. Config.SnapshotEvery and ctx.Snapshot save snapshots to a persist.SnapshotStore, and Config.PruneEvents deletes the events that the previous snapshot covers.
• persist.JSON is a codec for events and state. An actor can implement Recoverer to learn when recovery finished, and Stopper and SignalReceiver for the Stopped hook and signals.
• persist/filestore is the reference journal and snapshot store. filestore.Open(dir) uses the operating system's files. On recovery, it discards a torn write at the end of the journal and reports its size in Bounds.Repaired.
• In simulation, s.Volume(node) returns the node's simulated disk as a filestore.FS, and filestore.New runs a store on it. faultline's disk faults, such as lost or torn writes that were not synced and failed syncs, then run through the real recovery code. lockstep/leak fails a seed when an actor stops with a file still open.

Delivery helpers (lockstep/dedup)

• Window holds idempotency keys and tombstones, Replies caches the first reply to each request, and Versions accepts a version for an entity only when it is greater than the one it holds. Config.TTL and Config.Capacity bound each of them, and each encodes to JSON, so a persistent actor can keep it in its snapshots.
• Message delivery explains what your actors must handle themselves: a retried request that arrives twice, and a Cancel that arrives before its Create. examples/orders is an order service whose simulation tests find both bugs from a seed.

lockstep-vet

lockstep-vet and the golangci-lint plugin have these new checks, so code that passed v0.1.0's checks can get new findings:

• LS200: a variable used after the message that holds it was sent.
• LS201: a type marked //lockstep:immutable that is not immutable.
• LS210: actor code that writes a package variable, or puts the actor's own state in a message.
• LS220: a reply from lockstep.Ask that is never awaited or piped, and an ignored error from lockstep.Send or lockstep.Spawn.
• LS230: a message type with no case in Receive.
• LS240: an actor field that holds a resource and is neither passed to ctx.Own nor released in Stopped.

The markers //lockstep:immutable and //lockstep:resource go in a type's doc comment and apply in every package that imports the type.

go install github.com/hmdsefi/lockstep/analysis/cmd/lockstep-vet@v0.2.0
go vet -vettool=$(which lockstep-vet) ./...

Benchmarks

• The bench module runs ping-pong, fan-out, idle actors, send latency, and ask latency on lockstep, protoactor-go, Hollywood, and GoAkt, and measures how many seeds per second sim.Simulate runs.
• The README's Benchmarks table and bench/RESULTS.md show the results of one CI run on main. CI fails when a lockstep result goes over its budget in bench/budgets.txt.

Fixed

• System.Shutdown took time quadratic in the number of sibling actors under one parent (#298). Stopping children, removing death watches, and removing overload subscriptions now take linear time.

Known limitations

• lockstep cannot preempt a turn. The watchdog reports a stuck turn and isolates its actor, but the turn keeps its worker, and Shutdown with a deadline returns the context's error when the deadline passes.
• No remoting or clustering: actors on different nodes exchange messages only in simulation. lockstep ships no tracing or metrics adapter. The roadmap tracks later releases.
• persist/filestore keeps files on a local disk. Use one filestore.Store per directory: two Stores on the same directory, in one process or in two, are not detected and can overwrite each other's committed events.
• An ask deadlock through a PID built with PIDAt is not reported as a deadlock; the seed ends by the ask's timeout instead. It never causes a false failure (#299).
• Replay from a seed (FAULTLINE_SEED) is exact, down to the trace hash. Replay from a schedule.json file (FAULTLINE_SCHEDULE) reproduces the same failure and the same lockstep records, but its trace hash differs from the original run's, because faultline v0.1.2 records schedule-driven faults differently from planned ones.
• Memory pressure (Sim.Pressure, Sim.PressureFor) and the message rules of Sim.DropBetween and Sim.DelayBetween draw from lockstep's own random streams, not faultline faults. A seed replays them exactly, but schedule.json does not list them, and faultline cannot minimize them.
• The API can change before v1.0.