Skip to content

lockstep v0.1.0

Latest

Choose a tag to compare

@hmdsefi hmdsefi released this 25 Oct 13:13
· 89 commits to main since this release
2dc2142

lockstep v0.1.0 is the first release: typed local actors on lockstep.New(), a seeded multi-node simulation in lockstep/sim built on faultline, and lockstep-vet, which checks that actor code stays deterministic. The README walks through a first actor, a simulation test, and a replayed seed.

Requirements

  • Go 1.26 or later.
  • faultline v0.1.2, pinned in go.mod. Only lockstep/sim imports it; a program that does not import sim never links faultline.

Modules and tags

This release tags two modules:

  • github.com/hmdsefi/lockstep at v0.1.0: the runtime and lockstep/sim.
  • github.com/hmdsefi/lockstep/analysis at analysis/v0.1.0: the analyzers, lockstep-vet, and the golangci-lint plugin.

examples/raft is a module of its own, built against the lockstep code in the same checkout. It is not tagged.

Added

Actors and messages

  • Typed actors: Actor[M] receives one message type M, and Spawn returns a PID[M], so a message of the wrong type does not compile. Spawn takes a factory that runs at every start and restart.
  • Send and SendContext deliver a message; Stop stops an actor by Address.
  • Request and reply: Ask returns a *Reply[R] and passes a ReplyTo[R] into the request. ReplyTo.Send and ReplyTo.Fail answer it exactly once, WithTimeout sets the deadline, and NewReply creates a reply without a target.
  • Reply.Await waits outside actors. Inside an actor it returns ErrAwaitInActor; Pipe delivers the reply to the actor as one of its own messages.
  • Signals from the runtime arrive through SignalReceiver[M], never as an M: Terminated, ChildFailed, and Overloaded.
  • Watch and Unwatch subscribe to a Terminated signal when another actor stops for good.
  • Names and lookup: Lookup[M] finds a registered actor by path and type, and PIDAt addresses whichever actor is registered at a path on a node.
  • Context gives actor code its runtime: Self, Parent, Now, Rand, Logf, Fail, Stop, and Own.

Mailboxes and overload

  • Bounded mailboxes with a per-actor WithMailbox or system-wide WithDefaultMailbox capacity and policy. The default is 1024 messages with FailSend.
  • Four full-mailbox policies: FailSend (Send returns ErrMailboxFull), DropNewest, DropOldest, and BlockSender, which makes an outside sender wait. Timer messages, piped replies, and effect results never count toward capacity and are never refused.
  • Opt-in memory estimates: WithMemoryEstimates, messages that implement Sizer, actors that implement MemSizer, and System.Estimates for the largest actors.
  • Admission: when a system sheds, Spawn and Send return ErrOverloaded, and actors that called Context.SubscribeOverload get Overloaded signals. In v0.1.0, shedding happens in simulation only (see Known limitations).

Lifecycle and supervision

  • Lifecycle hooks: Starter.Started runs before the first message of an incarnation, and Stopper.Stopped runs once per incarnation, on stop, before every restart, and after a failure. A resource registered with Context.Own is closed after Stopped.
  • A panic or Context.Fail fails the actor; a panic arrives as a *PanicError with its stack.
  • Supervision strategies OneForOne and AllForOne, with directives DirectiveRestart, DirectiveStop, DirectiveEscalate, and DirectiveResume. Strategy limits restarts per window (MaxRestarts, Window) and backs off between them (MinBackoff, MaxBackoff). WithStrategy sets it per actor and WithGuardianStrategy for top-level actors.
  • Stopping a parent stops its children first, and StopReason tells Stopped why: Normal, Failed, ParentStopping, Shutdown, Restarting, or NoActor.
  • The actor cap: WithMaxActors limits live actors per system (default DefaultMaxActors, 1,000,000). Past it, Spawn returns ErrActorCap; one percent of the cap is held back for restarts and system actors.
  • System.Shutdown stops user actors, then system actors, and waits for every Stopped hook.

Timers

  • Context.After delivers a message to the actor once, and Context.Every delivers it on a period, skipping a firing while the previous message is still in the mailbox. Context.CancelTimer cancels either.
  • Timers run on the engine clock: wall time on lockstep.New(), the node's virtual clock in simulation.

Effects

  • Do runs an Effect[T] (a network call, file I/O, a query) outside the actor's turn and delivers its result as a message through a done function that runs on the actor's turn.
  • WithEffectTimeout sets the timeout (default DefaultEffectTimeout, 30 s); Context.CancelEffect cancels a pending effect.
  • On lockstep.New(), effects run on worker goroutines, at most 64 at a time with up to 1024 waiting; past that, done gets ErrOverloaded.
  • In simulation, Effect.Run is never called: Sim.HandleEffect answers each effect by name with a value, an error, and a virtual latency.

Simulation (lockstep/sim)

  • sim.Simulate and sim.SimulateSeeds run a test body once per seed on faultline's simulation kernel, with virtual time and a simulated network. sim.Options embeds faultline.Options.
  • Multiple lockstep nodes per run: Sim.Node adds a server and Sim.Client a client node. Each boot gets a new System, and actors on different nodes exchange messages over the simulated network.
  • sim.Send and sim.Ask drive the run from the test body. Sim.System, Sim.World, and Sim.Counts expose the run's state.
  • Faults: faultline's planners and injector (partitions, crashes and restarts, pauses, link faults, clock faults) apply to lockstep nodes. lockstep adds Sim.DropBetween and Sim.DelayBetween for messages between two actors, Sim.InjectFail and Sim.StopActor for single actors, and memory pressure with Sim.Pressure and Sim.PressureFor.
  • Count-based admission budgets: sim.Budget sets per-node limits on live actors and estimated bytes, with a resume mark, so shedding replays exactly.
  • Checks that fail a seed: lockstep/deadlock reports a cycle of pending asks as lockstep: deadlock: A → B → A; lockstep/leak reports an actor that stops or restarts with an effect still pending, or a resource that fails to close; lockstep/no-effect-handler reports an effect with no handler (ErrNoEffectHandler); lockstep/misuse reports API misuse such as Await on a pending reply.
  • Exact replay: a failing seed prints a FAULTLINE_SEED=... command that reruns it with the same trace hash. LOCKSTEP_CHECK_DETERMINISM=1 runs every seed twice and compares the traces.

lockstep-vet and the golangci-lint plugin

  • LS100 reports nondeterminism and blocking in actor code: the wall clock, global and crypto randomness, goroutines, select, map iteration order, environment variables, channel operations, locks, and blocking network, file, database, and process calls.

  • LS110 reports Reply.Await in actor code.

  • LS001 and LS002 report malformed and unknown //lockstep: directives. //lockstep:allow LSnnn <reason> suppresses one finding.

  • lockstep-vet runs the analyzers with go vet -vettool:

    go install github.com/hmdsefi/lockstep/analysis/cmd/lockstep-vet@v0.1.0
    go vet -vettool=$(which lockstep-vet) ./...
  • github.com/hmdsefi/lockstep/analysis/golangci registers the same analyzers as the golangci-lint module plugin lockstep, with a disable setting that leaves out analyzers by name.

The Raft demo

  • examples/raft runs a Raft key-value cluster of three or five servers, written as lockstep actors, through partitions, crashes, paused servers, and lossy links. It checks Raft's safety properties after every event, and checks the client history against the committed log when the run ends.
  • Three planted bugs (vote-ignores-log, stale-read, vote-not-durable) each fail within the first 300 seeds of their fault preset, and TestDemoPartitionBug finds and replays one.
  • The demo passes lockstep-vet with no //lockstep:allow directives.

Known limitations

  • lockstep.New() runs on a local engine: one goroutine runs every actor turn, so actors interleave but never run in parallel. There is no pooled production scheduler.
  • WithAdmission, WithWatchdog, and WithGoroutineWarning are accepted and validated but have no effect: lockstep.New() does not read the live heap, watch for stuck turns, or count goroutines, and never sheds. Admission and Overloaded signals work in simulation through sim.Budget and memory pressure.
  • No remoting, clustering, persistence, or OpenTelemetry. Actors on different nodes exchange messages only in simulation. The roadmap tracks later releases.
  • System.Shutdown takes time quadratic in the number of sibling actors under one parent (#298).
  • An ask deadlock through a PID built with PIDAt is not reported as a deadlock; the seed ends by the ask's timeout instead. It never causes a false failure (#299).
  • Replay from a seed (FAULTLINE_SEED) is exact, down to the trace hash. Replay from a schedule.json file (FAULTLINE_SCHEDULE) reproduces the same failure and the same lockstep records, but its trace hash differs from the original run's, because faultline v0.1.2 records schedule-driven faults differently from planned ones.
  • Memory pressure (Sim.Pressure, Sim.PressureFor) and the message rules of Sim.DropBetween and Sim.DelayBetween draw from lockstep's own random streams, not faultline faults. A seed replays them exactly, but schedule.json does not list them, and faultline cannot minimize them.
  • The API can change before v1.0.