Replicas coordinate through the database, and size themselves

This release rebuilds how a dwarf fleet finds work, divides resources, and carries state. Both the API and the database schema change. Dwarf is v0.x, so upgrading is a maintenance window, not a rolling deploy — see docs/upgrading.md.

Breaking changes

• Host loses SignalPeers. The engine now sends nothing to its peers at all; hosts implement two methods, LoadGraph and ExecuteTask. Three signal kinds existed here and all three were deleted after measurement — a per-step work doorbell, a per-flow stop broadcast, and a fleet-membership nudge — because in each case the poll the engine already runs converges faster than the message.
• SetShard(ShardSpec{Index, DSN, VirtualCPUs}) replaces SetDSN / SetNumShards / SetWorkersPerConn. You declare facts about each shard's server; the engine derives the tuning. New SetEngineID pins a replica identity so restarts don't inflate the observed fleet count.
• workflow.State is a type, not a map[string]any — on FlowOutcome.State, Flow.Snapshot(), baggage, and interrupt/subgraph payloads. It holds each field as raw JSON and decodes on read. Flow.Delete → Flow.Del; Flow.SetState, Flow.Transform, MergeState, Graph.HasFanIn and the Duration() helpers are gone; Render() no longer returns an error; typed getters panic on a mistyped read instead of quietly returning a zero value.
• NewEngineUnderTest(t) / SetTestName replace RunInTest / SetInTest.
• Schema: a new dwarf_peers registry, new step columns (state_refs, engine_id), fixed-width CHAR tokens and statuses widened to VARCHAR, and SQL Server JSON payloads stored as VARBINARY — which ends a silent corruption of non-ASCII payloads on that dialect. The baseline migrations are edited in place, so a database created by v0.9.x cannot be migrated forward — recreate it.

Coordination is the database, and nothing else

Replicas discover each other through a dwarf_peers registry read per shard, not through the host's transport. From one reading a replica derives two different numbers with deliberately opposite risk postures: how many ways to divide a shard's connection budget (over-count and you collapse a database) and which replicas demonstrably dispatch work (over-count and you strand it). A replica that goes blind holds its last good reading rather than acting on silence.

Sizing derives itself

Connection pools, the worker crew, the candidate cache and the scan cadence are all derived from declared vCPUs and the observed fleet, and every path that moves a pool re-derives what depends on it. Pools sit at 6x vCPUs, or 12x on servers of 32 vCPUs or more. The worker crew grows on demand for long tasks and retires surplus goroutines on their own idleness, bounded by a ceiling derived from the crash-recovery lease margin and a startup RTT probe. Database access is ordered by band and job age through a per-shard turnstile.

Dispatch is per-shard

The single merged refiller is replaced by a fleet of per-shard pistons with no barrier between them (the straggler wait measured 2.02x at six shards). Selection is three-phase and capped, so scan cost tracks fairness-key cardinality rather than backlog depth — a 3M-row single-key flood fell from 15.3s to 2.2s per pass on Postgres. Candidates are partitioned across replicas by residue class (claim misses 41.3% → 11.6% at R=2), and a replica that is alive but not keeping up has its share stolen rather than stranded (137–153 → 458–501 steps/s with one crippled peer).

State is stored once, not once per step

A large carried field is written to one anchor row and referenced from every step that carries it — ~29x fewer stored state bytes on a document-extraction shape. Flow-boundary flattening, Fork remapping and fan-in reduction are handled by the engine; the encoding never reaches task code.

Durability and correctness

A step's outcome is persisted by retrying the write, never re-running the task; when the retries are exhausted the engine asks the database which failed rather than reading driver error codes. Lease generations (lease_seq) fence every post-execution write, so a slow worker can no longer corrupt or terminalize a flow a peer is healthily re-running. Plus fixes to fan-in cohort accounting, Fork's nested-cohort recompute, subgraph failure propagation and orphan recovery.

Observability, docs and benchmarks

33 dwarf_* OTEL instruments, including per-column state byte counters and gauges for state held across a host call. Seven new operator guides — data handling, flows, deployment, runbook, backup and restore, resharding, production checklist, upgrading — and a cloud benchmark campaign with sizing formulas and charts, measured from 1 to 64 database vCPUs.

Testing

Tests rendezvous with engine checkpoints instead of sleeping, the suites run in parallel under a connection budget, and CI sweeps all four dialects — SQLite, PostgreSQL, MySQL, SQL Server.