Repository navigation
lockstep v0.1.0 is the first release: typed local actors on lockstep.New(), a seeded multi-node simulation in lockstep/sim built on faultline, and lockstep-vet, which checks that actor code stays deterministic. The README walks through a first actor, a simulation test, and a replayed seed.
Requirements
- Go 1.26 or later.
- faultline v0.1.2, pinned in
go.mod. Onlylockstep/simimports it; a program that does not importsimnever links faultline.
Modules and tags
This release tags two modules:
github.com/hmdsefi/lockstepatv0.1.0: the runtime andlockstep/sim.github.com/hmdsefi/lockstep/analysisatanalysis/v0.1.0: the analyzers,lockstep-vet, and the golangci-lint plugin.
examples/raft is a module of its own, built against the lockstep code in the same checkout. It is not tagged.
Added
Actors and messages
- Typed actors:
Actor[M]receives one message typeM, andSpawnreturns aPID[M], so a message of the wrong type does not compile.Spawntakes a factory that runs at every start and restart. SendandSendContextdeliver a message;Stopstops an actor byAddress.- Request and reply:
Askreturns a*Reply[R]and passes aReplyTo[R]into the request.ReplyTo.SendandReplyTo.Failanswer it exactly once,WithTimeoutsets the deadline, andNewReplycreates a reply without a target. Reply.Awaitwaits outside actors. Inside an actor it returnsErrAwaitInActor;Pipedelivers the reply to the actor as one of its own messages.- Signals from the runtime arrive through
SignalReceiver[M], never as anM:Terminated,ChildFailed, andOverloaded. WatchandUnwatchsubscribe to aTerminatedsignal when another actor stops for good.- Names and lookup:
Lookup[M]finds a registered actor by path and type, andPIDAtaddresses whichever actor is registered at a path on a node. Contextgives actor code its runtime:Self,Parent,Now,Rand,Logf,Fail,Stop, andOwn.
Mailboxes and overload
- Bounded mailboxes with a per-actor
WithMailboxor system-wideWithDefaultMailboxcapacity and policy. The default is 1024 messages withFailSend. - Four full-mailbox policies:
FailSend(SendreturnsErrMailboxFull),DropNewest,DropOldest, andBlockSender, which makes an outside sender wait. Timer messages, piped replies, and effect results never count toward capacity and are never refused. - Opt-in memory estimates:
WithMemoryEstimates, messages that implementSizer, actors that implementMemSizer, andSystem.Estimatesfor the largest actors. - Admission: when a system sheds,
SpawnandSendreturnErrOverloaded, and actors that calledContext.SubscribeOverloadgetOverloadedsignals. In v0.1.0, shedding happens in simulation only (see Known limitations).
Lifecycle and supervision
- Lifecycle hooks:
Starter.Startedruns before the first message of an incarnation, andStopper.Stoppedruns once per incarnation, on stop, before every restart, and after a failure. A resource registered withContext.Ownis closed afterStopped. - A panic or
Context.Failfails the actor; a panic arrives as a*PanicErrorwith its stack. - Supervision strategies
OneForOneandAllForOne, with directivesDirectiveRestart,DirectiveStop,DirectiveEscalate, andDirectiveResume.Strategylimits restarts per window (MaxRestarts,Window) and backs off between them (MinBackoff,MaxBackoff).WithStrategysets it per actor andWithGuardianStrategyfor top-level actors. - Stopping a parent stops its children first, and
StopReasontellsStoppedwhy:Normal,Failed,ParentStopping,Shutdown,Restarting, orNoActor. - The actor cap:
WithMaxActorslimits live actors per system (defaultDefaultMaxActors, 1,000,000). Past it,SpawnreturnsErrActorCap; one percent of the cap is held back for restarts and system actors. System.Shutdownstops user actors, then system actors, and waits for everyStoppedhook.
Timers
Context.Afterdelivers a message to the actor once, andContext.Everydelivers it on a period, skipping a firing while the previous message is still in the mailbox.Context.CancelTimercancels either.- Timers run on the engine clock: wall time on
lockstep.New(), the node's virtual clock in simulation.
Effects
Doruns anEffect[T](a network call, file I/O, a query) outside the actor's turn and delivers its result as a message through adonefunction that runs on the actor's turn.WithEffectTimeoutsets the timeout (defaultDefaultEffectTimeout, 30 s);Context.CancelEffectcancels a pending effect.- On
lockstep.New(), effects run on worker goroutines, at most 64 at a time with up to 1024 waiting; past that,donegetsErrOverloaded. - In simulation,
Effect.Runis never called:Sim.HandleEffectanswers each effect by name with a value, an error, and a virtual latency.
Simulation (lockstep/sim)
sim.Simulateandsim.SimulateSeedsrun a test body once per seed on faultline's simulation kernel, with virtual time and a simulated network.sim.Optionsembedsfaultline.Options.- Multiple lockstep nodes per run:
Sim.Nodeadds a server andSim.Clienta client node. Each boot gets a newSystem, and actors on different nodes exchange messages over the simulated network. sim.Sendandsim.Askdrive the run from the test body.Sim.System,Sim.World, andSim.Countsexpose the run's state.- Faults: faultline's planners and injector (partitions, crashes and restarts, pauses, link faults, clock faults) apply to lockstep nodes. lockstep adds
Sim.DropBetweenandSim.DelayBetweenfor messages between two actors,Sim.InjectFailandSim.StopActorfor single actors, and memory pressure withSim.PressureandSim.PressureFor. - Count-based admission budgets:
sim.Budgetsets per-node limits on live actors and estimated bytes, with a resume mark, so shedding replays exactly. - Checks that fail a seed:
lockstep/deadlockreports a cycle of pending asks aslockstep: deadlock: A → B → A;lockstep/leakreports an actor that stops or restarts with an effect still pending, or a resource that fails to close;lockstep/no-effect-handlerreports an effect with no handler (ErrNoEffectHandler);lockstep/misusereports API misuse such asAwaiton a pending reply. - Exact replay: a failing seed prints a
FAULTLINE_SEED=...command that reruns it with the same trace hash.LOCKSTEP_CHECK_DETERMINISM=1runs every seed twice and compares the traces.
lockstep-vet and the golangci-lint plugin
-
LS100reports nondeterminism and blocking in actor code: the wall clock, global and crypto randomness, goroutines,select, map iteration order, environment variables, channel operations, locks, and blocking network, file, database, and process calls. -
LS110reportsReply.Awaitin actor code. -
LS001andLS002report malformed and unknown//lockstep:directives.//lockstep:allow LSnnn <reason>suppresses one finding. -
lockstep-vetruns the analyzers withgo vet -vettool:go install github.com/hmdsefi/lockstep/analysis/cmd/lockstep-vet@v0.1.0 go vet -vettool=$(which lockstep-vet) ./... -
github.com/hmdsefi/lockstep/analysis/golangciregisters the same analyzers as the golangci-lint module pluginlockstep, with adisablesetting that leaves out analyzers by name.
The Raft demo
examples/raftruns a Raft key-value cluster of three or five servers, written as lockstep actors, through partitions, crashes, paused servers, and lossy links. It checks Raft's safety properties after every event, and checks the client history against the committed log when the run ends.- Three planted bugs (
vote-ignores-log,stale-read,vote-not-durable) each fail within the first 300 seeds of their fault preset, andTestDemoPartitionBugfinds and replays one. - The demo passes
lockstep-vetwith no//lockstep:allowdirectives.
Known limitations
lockstep.New()runs on a local engine: one goroutine runs every actor turn, so actors interleave but never run in parallel. There is no pooled production scheduler.WithAdmission,WithWatchdog, andWithGoroutineWarningare accepted and validated but have no effect:lockstep.New()does not read the live heap, watch for stuck turns, or count goroutines, and never sheds. Admission andOverloadedsignals work in simulation throughsim.Budgetand memory pressure.- No remoting, clustering, persistence, or OpenTelemetry. Actors on different nodes exchange messages only in simulation. The roadmap tracks later releases.
System.Shutdowntakes time quadratic in the number of sibling actors under one parent (#298).- An ask deadlock through a PID built with
PIDAtis not reported as a deadlock; the seed ends by the ask's timeout instead. It never causes a false failure (#299). - Replay from a seed (
FAULTLINE_SEED) is exact, down to the trace hash. Replay from aschedule.jsonfile (FAULTLINE_SCHEDULE) reproduces the same failure and the same lockstep records, but its trace hash differs from the original run's, because faultline v0.1.2 records schedule-driven faults differently from planned ones. - Memory pressure (
Sim.Pressure,Sim.PressureFor) and the message rules ofSim.DropBetweenandSim.DelayBetweendraw from lockstep's own random streams, not faultline faults. A seed replays them exactly, butschedule.jsondoes not list them, and faultline cannot minimize them. - The API can change before v1.0.