Reducers That Fold Everywhere, History Stored Once, Priority Bands That Spill

v0.12.0 makes a reduced field mean the same thing wherever a task writes it. Every write to a field with a
reducer is a contribution, folded through the reducer at every hand-off: the next step, the fan-in, the final
state and Continue. A task writes only what it adds, and the outcome no longer depends on the graph's shape. An
agent loop's history is stored once: a field under append, or a list a task keeps rewriting but only extends,
is kept as references to the rows that hold its elements rather than as the whole history at every step.
Priorities are now five bands that spill, so a better band no longer starves a worse one while workers sit
idle. Beneath that, the engine does fewer round trips per step, a shard is dispatched by two replicas at a
budgeted pool, and a step whose transition was lost to a crash is replayed rather than stranded.

This release breaks the schema, the API and the wire format, so upgrading from v0.11.0 needs a maintenance
window and a fresh database or hand-applied schema changes. Read "Upgrading from v0.11.0" first.

Highlights

• Reducers fold at every hand-off. A write to a reduced field is a contribution everywhere; Overwrite and
  Del replace. (Behavior change - see below.)
• An agent loop stores each message once. A 100-step loop of 1 KB messages stored ~37x less under append, and
  ~33x less when a task rewrites a growing list each step, on PostgreSQL.
• Priority bands that spill. Priorities run 1 to 5 by default (SetMaxPriority widens it up to 100), the
  default is 3, and capacity a better band leaves unused goes to the next one. (Breaking - see below.)
• Fewer round trips per step. The transition after a task runs as one pipelined batch on PostgreSQL, a flow
  is created in one batch, and the claim returns what the dispatch needs.
• Two dispatchers per shard, on a budgeted pool. Each shard's connections are a budget of 250-500 by distance
  to the database, clamped by the server, and split between the two replicas that dispatch it.
  ShardSpec.MaxConnections is the lever.
• A lost transition is replayed. A replica that dies between a step's completion and its transition no longer
  leaves the flow stranded; the step's recorded outcome is committed without running the task again.
• A 256 MB ceiling on every value, refused with a 413 instead of failing deep inside a write.
• Many API renames and removals ahead of v1.0. (Breaking - see "API changes".)

Upgrading from v0.11.0

Plan a maintenance window. Three things change at once: the schema, how stored writes to reduced fields are
read, and how replicas share a shard. None of them can be rolled through one replica at a time.

1. Drain in-flight flows

Let every flow finish, or Terminate it, before the upgrade. In v0.11.0 a step's write to a reduced field outside a
fan-out branch was the field's whole value; v0.12.0 reads every stored write as a contribution and folds it in.
Anywhere a v0.11.0 step's stored output is folded again, it would count twice: an interrupted step resumed, a
pending Retry, a Subgraph re-entry, a replay, a cohort still converging, a tail merged into the final state.
Terminal flows are unaffected; a Continue from one folds its final state the new way, which is correct.

2. Start from a fresh database, or apply the schema changes by hand

The migration scripts were edited in place, so a v0.11.0 database does not receive these changes on startup. A
fresh database gets all of them. On an existing one, apply them during the drained window:

Columns added. Without them the engine cannot run:

• dwarf_flows.payload_ver: without it every Create fails.
• dwarf_flows.graph_hash, NOT NULL with no default. A hand-applied ALTER TABLE ... ADD on a populated table
  needs DEFAULT '', which the engine reads as "no hash" and falls back to reading the graph.
• dwarf_steps.advancing and the advance_* columns, state_reduced and payload_deferrals: without
  payload_deferrals every claim fails.
• dwarf_peers.draining and dwarf_peers.joined_at: without draining every registry read fails, and each
  replica dispatches every shard at the full budget.

Use the per-dialect definitions in the migration scripts for the ALTER TABLE ... ADD statements.

Column renamed: dwarf_flows.step_id is stopped_step_id. On an older database every List, Create and
claim fails with "no such column". Rename it after the drain, since a running flow's old value is not a stopped
step: ALTER TABLE dwarf_flows RENAME COLUMN step_id TO stopped_step_id; on SQL Server
EXEC sp_rename 'dwarf_flows.step_id', 'stopped_step_id', 'COLUMN'.

Column type changed: dwarf_steps.subgraph_error is a JSON column (JSON, JSONB, VARBINARY(MAX) or TEXT by
dialect, like subgraph_result), holding {} for no error. Alter it to the type the migration script declares.

PostgreSQL time columns are TIMESTAMP, not TIMESTAMPTZ. An older database keeps TIMESTAMPTZ, which on a
server or role whose time zone is not UTC stores every time shifted by the offset (a flow.Retry give-up horizon
then never expires). After the drain, alter each time column with
ALTER TABLE ... ALTER COLUMN <col> TYPE TIMESTAMP USING <col> AT TIME ZONE 'UTC', or keep the server in UTC.

Rebuild idx_dwarf_steps_due on PostgreSQL, SQL Server and SQLite: drop it and recreate it from the migration
script. Its key and its predicate changed. An older database keeps the old index, which the new scheduling
statements no longer match, so they silently fall back to reading every pending row. MySQL's is unchanged.

Indexes removed. An older database keeps them; they are harmless but written on every step:

• DROP INDEX idx_dwarf_steps_saturation
• DROP INDEX idx_dwarf_steps_status (... ON dwarf_steps on MySQL and SQL Server)

PostgreSQL storage setting: dwarf_flows no longer sets a fillfactor. A table created during this release
cycle may carry fillfactor = 80, which is harmless; ALTER TABLE dwarf_flows RESET (fillfactor) reverts it for
new pages.

3. Priorities run 1 to 5

The default priority is 3, the midpoint, and SetDefaultPriority is gone. SetMaxPriority(n), at most 100 and set
before Startup, widens the range; Create and Run reject a priority above it. Lower numbers still run first.

Work stored above the range is served after every level in it. That includes every flow created at the old default
of 100: in flight if it was not drained, and anything a Continue, Fork or subgraph inherits from a finished one,
since inheritance keeps the stored priority. No migration remaps it. To bring old threads back into range, run
UPDATE dwarf_flows SET priority=3 WHERE priority>5, and the same on dwarf_steps, during the window.

4. Upgrade every replica, and the host, together

Mixed versions disagree on which replicas dispatch a shard and how its pool is sized, on how stored state
references are read, and on the flow's JSON form. In particular, a v0.11.0 replica reads v0.12.0's list and
forEach element references as no references at all: silent data loss. From this release on, a replica fails a
step it cannot read rather than dropping data, and a replica that meets a flow in a newer payload format defers it
to a newer replica.

Reducers fold at every hand-off

A write to a field with a reducer is a contribution, folded through the reducer at the next step, the fan-in,
the final state and Continue. A read returns the combined value.

• A task writes only its contribution. f.Set("messages", []any{m}) under append adds m. A read-modify-write
  (Set(k, append(Get(k), m))) adds the history again.
• The result does not depend on graph shape. A step that changes nothing changes nothing, and wrapping a step in
  a one-item forEach gives the same result.
• Flow.Overwrite replaces a reduced field instead of contributing to it, and Del is Overwrite(k, nil).
  Both are refused on a reduced field inside a fan-out branch, where a reset has no defined meaning for the other
  branches.
• A step's runs compose. On Retry, a resumed Interrupt or a Subgraph re-entry, the task sees its earlier
  runs' writes and adds to them. Write after Interrupt or Subgraph, not before.
• A bare nil contributes nothing to a reduced field.
• Nested fan-outs: an inner cohort's work reaches the outer fan-in and the final state. Before, it could be lost.
• A Fork at an inner fan-in hands on its cohort's original part, whatever the overrides.
• Folds keep numbers exact, including integers past 2^53, and union compares numbers by value, so 1, 1.0
  and a database's respelling of the same number are one element.

An agent loop stores each message once

A field under append, and a list with no reducer that a task rewrites each step but only extends, are stored as
references to the rows holding their elements, instead of the whole history at every step. A 100-step loop of 1 KB
messages measured ~37x less storage under append and ~33x less for the rewrite, on PostgreSQL. A forEach branch
references its element inside the array rather than storing a second copy.

dwarf_state_write_bytes gained column=state_refs, where those references are written; such a workload's state
series falls to near zero.

Priority bands that spill

The scheduler fills the best due band first, then gives whatever capacity remains to the next, and so on. A sparse
but continuous better band therefore no longer starves every worse band while workers sit idle. Priority stays
strict over capacity: worse work gets only what better work leaves unused.

Throughput and dispatch

• Fewer round trips per step. The transition after a task is one pipelined batch on PostgreSQL, a
  single-successor transition is one batch, a flow is created in one batch, and the claim returns the flow row.
  Successors are inserted with multi-row inserts, and a flow-row lock no longer writes a row version on PostgreSQL
  and MySQL.
• Less allocation per step, and a graph cache that parses each graph once per replica.
• Two dispatchers per shard. The dispatchers are a window over the replicas in the order they joined, so the
  role spreads evenly across shards. A replica losing its dispatcher slot keeps its share of the pool until the
  steps it already claimed finish, and a draining replica hands its work to its peers.
• Connection budget. A shard's budget is a constant of 250-500 connections, ramped by the round trip to the
  database and clamped by the server's max_connections, shared by its dispatchers. ShardSpec.MaxConnections sets
  it; ShardSpec.Weight (new) weights flow placement across shards.
• Placement. A new flow goes to a shard the creating replica dispatches, when there is one. Any
  ShardSpec.Weight other than 1 switches the fleet to weighted placement over every non-cordoned shard.
• Scheduling at scale. The scheduling queries read only what each band needs, and the background sweeps no
  longer read the whole step table at once.

Reliability

• A lost transition is replayed. A replica that dies between a step's completion and its transition used to
  leave the flow running with nothing to run. The completion now records what the transition needs, and recovery
  commits it without running the task again.
• Expired leases are recovered every minute (was 5), so a crashed replica's claimed steps run again within the
  step's TimeBudget + 30 s + 1 min. A dead worker's completed step is recovered on the same bound.
• Contention on the completion write is retried, never the task.
• Shutdown no longer waits for background maintenance in flight; an abandoned pass is redone by a peer or the
  next start.
• Payload format. A flow records the format its state is stored in. A replica that meets a newer format answers
  501 on the reads that cannot wait (Snapshot, Step, Fork) and defers the step to a newer replica.
• Unreadable rows fail loudly. A step whose stored state, changes or references cannot be read fails, and
  Snapshot and Step return the error, instead of carrying on with an empty value.

Limits

• Every value over 256 MB is refused: a 413 from Create, Run, Continue, Resume and Fork. A step whose
  output is too large fails at once, and a final state over the limit fails the flow with an empty final state.
  MySQL and MariaDB need max_allowed_packet of at least 512M, since a step row holds its state and its changes,
  each up to 256 MB.
• A Terminate or Cancel reason over 4 KB, or holding a NUL or invalid UTF-8, is a 400.
• Stored error messages are capped at 16 KB, with invalid UTF-8 replaced.
• Create returns a 400 for a FlowOptions.Priority beyond 32 bits or a TimeBudget over just under 24.8
  days, and SetTimeBudget returns a 400 over that ceiling.

API changes

Reducers and the carrier:

• workflow.Changes (contributions and overwrites) is a step's writes. State.Apply and ApplyAll take it, and
  Changes.Compose combines two. Changes.Additions and AdditionSuffix are new.
• Removed: State.Merge, MergeReduce, MergeReduceAll, DelNils.
• RawFlow.RawChanges and SetRawChanges take Changes; RawFlow.SetReducers, RetryDelay and SetInBranch
  are new.
• A field name ending in ! or + cannot be written.
• Flow.SetChanges(source) drops its snapshot parameter and sets every field the source marshals, skipping empty
  omitempty/omitzero fields. A host's generated wrappers drop the Snapshot() call.
• Flow.Get returns (ok bool, err error) (was error). Flow.ParseState is Parse.
• GotoRequested, RetryRequested, SleepRequested, InterruptRequested and SubgraphRequested moved from
  Flow to RawFlow.

Renames:

• Graph.SetEndpoint is SetURL.
• workflow.Node and Graph.Nodes() are Task and Tasks(); workflow.END is End.
• Transition's WithGoto, OnError and Switch booleans are one Kind ("", TransitionSwitch,
  TransitionGoto, TransitionError; JSON kind). Stored graphs with the booleans still decode.
• FlowStep.SubWorkflowURL and SubWorkflowName are SubgraphURL and SubgraphName.
• FlowSummary.TaskName and Query.TaskName are StoppedTaskName, and the meaning changes: the task the flow
  stopped at, empty for a running flow, so the filter matches no running flow.
• ShardSummary.Shard is Index, and LatencyMs int is Latency time.Duration.
• engine.TestProxy and NewTestProxy are hosts.Mux and hosts.NewMux, in a new package:
  HandleGraph(workflowURL, graph), HandleTask(taskURL, func).
• Engine.FlowsTerminated is FlowsEnded.

Added:

• Engine.SetStateCacheSize, Engine.SetMaxPriority.
• FlowStep.SubgraphKey: History names a subgraph caller's child flow.
• ShardSpec.MaxConnections and ShardSpec.Weight.

Removed:

• Engine.SetDefaultPriority.
• Engine.HistoryMermaid: render History with workflow.NewFlowRenderer(steps).WithLinks("step").Render().
• FlowRenderer.WithTitle and GraphRenderer.WithTitleLabel: no renderer draws a title.
• engine.TaskHandler: HandleTask takes a plain func.
• Engine.SetMaxOpenConns: the engine derives each shard's connection budget, and ShardSpec.MaxConnections
  sets it instead, for example lower on a database shared with other workloads.
• Engine.DB, Engine.Seams and the exported fault, checkpoint and variable seam constants.
• Engine.SetRefillInterval.
• workflow.TerminalStatuses: use IsTerminalStatus.
• ShardSpec.VirtualCPUs: pools no longer follow vCPUs.
• ShardSummary.Steps and Flows: ShardInfo reports health only.

Wire and serialized formats

• The flow's JSON, which a host exchanges with a remote task, gains reducers (the graph's) and inBranch.
  changes names an overwritten field with a ! suffix: {"log": [...], "summary!": "x", "gone!": null}. On a
  re-run it may also hold an addition recorded by an earlier run, under a + suffix: {"history+": [m]}.
• FlowStep's JSON drops stepID, predecessorID and successorID; the edges are predecessorKey and
  successorKey, as step keys. subgraphURL, subgraphName and stoppedTaskName follow the renames above.
• ShardSummary's JSON has index and latency (nanoseconds).

Metrics and operations

• Renamed: dwarf_flows_terminated is dwarf_flows_ended. Dashboards and alerts must change.
• New labels: dwarf_flows_started and dwarf_flows_ended have a priority label, so more series; a View can
  drop it.
• New: dwarf_operation_seconds (to see a slow shard), dwarf_state_cache_bytes,
  dwarf_steps_dispatch_lag_seconds, dwarf_shard_dispatching.
• dwarf_steps_pending and dwarf_steps_oldest_pending_age_seconds are reported every 30 s by one leading
  replica per shard. They can be absent for up to 30 s at a handoff; aggregate them with max per shard, never sum.
• No gauge queries the database at collection.
• Memory: the state cache defaults to a tenth of the memory limit (GOMEMLIMIT, else the cgroup, else 1 GiB),
  and without GOMEMLIMIT the heap can reach about twice that, so an upgraded replica's resident memory grows.
  SetStateCacheSize overrides it.
• Tracing: with no OpenTelemetry SDK installed by Startup, the engine traces through a no-op and never picks
  up a provider installed later. Install the tracer before Startup.
• PostgreSQL sessions are pinned to UTC, so stored times are right whatever the server's or role's time zone.

New operating guidance in the docs:

• Set autovacuum_work_mem = 1GB on PostgreSQL.
• Leave statement_timeout, transaction_timeout, idle_in_transaction_session_timeout and MySQL's
  max_execution_time unset for the engine's database, and keep long-running queries off it.
• After a failover to an asynchronous standby, or a restore, restart every replica.

Fixes

• Terminate of an ended flow returns nil (was 409). Construction-time configuration errors are 400 or 409 (were
  500). Resume with data that is not JSON-marshalable is a 400 (was silently dropped).
• Await and Run timeouts are 408s that satisfy errors.Is(err, context.DeadlineExceeded), also when the
  caller's context had no deadline. An Await racing a Resume stays parked instead of returning a 408 while the
  flow is still running.
• List and Purge reject a negative OlderThan or NewerThan with a 400, and Purge accepts every List
  filter.
• SQL Server: an interval past ~24.8 days no longer overflows. Before, a List or Purge age filter of 30 days
  failed, and a flow.Sleep or Retry backoff that long failed its flow.
• SQL Server: an operation over a few thousand flows or steps no longer exceeds the 2,100-parameter limit.
• Long tasks no longer slow Create and the step hand-off.
• Retry returns false for a negative initialDelay or maxIntervalDelay, or a multiplier that is negative, NaN
  or infinite. A backoff with no maximum saturates rather than overflowing, and the engine fails a step whose retry
  delay comes out negative.
• A flow or step key parses only in the spelling the engine issues: 01-5-tok or 1-+5-tok is a 404.
• Snapshot and Resume follow the interrupt the flow row records, so they agree with StoppedTaskName when
  several siblings interrupt.
• During a deletion window, History of a subgraph child's key returns 404 with its root.
• Fork of a flow that a Delete or Purge stamped mid-call is a 409, and of one already removed a 404 (was a
  500).
• Purge no longer marks a root that a Resume revived after it was selected and that has since completed.
• Terminate, Delete and Purge stop a flow whose rows hold an unresolvable state reference or an unparseable
  graph, with an empty final state. They used to retry forever.
• A step covered by Cancel whose changes, with the error added, exceed the payload limit fails on the size.
• A failed child's error reaches its flow.Subgraph caller whole, status code and properties included.
• dwarf_steps_executed{interrupted} no longer counts an interrupt fenced away by a lost lease, and
  dwarf_flows_ended counts only the flows a Terminate itself ended.