Flows Archive by Day, and Lists by State

v1.49.0 moves the foreman to dwarf v0.14.0, by way of v0.13.0, and sequel v1.18.0. Closed flows now move to a
per-day archive about a minute after they close and are kept for a retention you set, so the tables every step reads
hold the open flows and little else. The foreman lists flows by their state, Continue builds on the flow you name,
and the state of a step now crosses the wire once instead of twice. The foreman needs new, empty databases: a
database an earlier release created cannot be upgraded in place. Go 1.27.3 or later is required.

Highlights

• Closed flows are archived and kept 30 days by default. The new Retention config sets how long, and the new
  FlowExpiration config can end flows that never close.
• Lists by state. ListOpen and ListClosed replace List, and the new PurgeBefore drops whole archive days.
• Continue takes the flow to continue. Threads are gone; the caller keeps the chain.
• Leaner task dispatch. A task's reply carries only its changes and control signals, not the whole state.
• Subgraphs that cannot start come back to the calling task as an error it can retry, instead of failing the
  step.
• Agent Studio shows open and closed flows on separate tabs.
• Fresh databases required, for the foreman's shards in every environment.

Fresh Databases Required

dwarf v0.13.0 and v0.14.0 edited migrations that earlier releases shipped, so a database an earlier release created
cannot be upgraded, and no ALTER patches it. Start the foreman on new, empty databases, on every shard and in every
environment, dev and test included. Flows in the old databases do not carry over. Nothing detects an old database:
the foreman starts on one, and every list of flows then fails on a missing column, which reads like a bug.

In a LOCAL deployment with no Shards config, the foreman keeps its flows in shard_1.local.sqlite in the
working directory of the process that runs it, typically main/. Delete it, and its -wal and -shm files, and
the foreman creates a fresh one on its next start. That file now opens with the settings dwarf recommends for a
durable SQLite database: journal_mode(WAL), synchronous(NORMAL) and auto_vacuum(FULL).

The new engine also requires:

• PostgreSQL: every foreman replica connects as the role that owns dwarf's tables, since the archive creates and
  drops a table per day.
• MySQL 8.0 or later, or MariaDB 10.6 or later, since archiving locks with SKIP LOCKED.
• SQL Server is now reached through github.com/microsoft/go-mssqldb, in place of the deprecated
  github.com/denisenkom/go-mssqldb. A program that imports the old package itself must switch, since both register
  the mssql driver and linking both panics at startup. If a development server's self-signed certificate now fails
  the TLS handshake at login, set encrypt=disable in its DSN.

Closed Flows Are Archived

A flow is open while it is running or interrupted, and closed once it has completed, failed, been terminated or been
cancelled. About a minute after a flow closes, the foreman moves it to an archive kept by the UTC day, and drops a
whole day once the retention has passed. An archived flow reads by its key exactly as before: Snapshot, Await,
History, Step, Continue and Fork all find it.

Two new foreman configs control it. Both are read at startup and should match on every replica:

• Retention is how many days the archive keeps closed flows. The default is 30; 0 keeps no archive, and a
  closed flow is deleted instead. In v1.48 every flow was kept until it was deleted.
• FlowExpiration terminates a flow still running or interrupted that long after its creation, with its
  subgraphs. It is off by default, and bounds what abandoned interrupts leave behind.

Lists by State

The foreman's List endpoint is replaced by two:

• ListOpen lists the running and interrupted flows, most recently updated first.
• ListClosed lists the archived flows, most recently archived first. It requires Query.FromTime and
  Query.ToTime, and the window may reach at most 31 days.

Both require Query.Limit, and neither paginates: each returns the first Limit across all shards, so
NextCursor and Query.Cursor are gone, as are Query.NewerThan and Query.OlderThan (use FromTime and
ToTime). A flow that closed in the last minute is in neither list until it is archived; read it by its key.

Removal follows the same split:

• Delete removes only a closed flow. A running or interrupted flow is a 409: Terminate it first.
• Purge removes what ListClosed lists for a query, at most Limit flows a call, and now returns
  (removed, done): call it until done. It never touches an open flow. It deletes row by row, so it is meant for
  erasing specific records.
• PurgeBefore, new, drops whole archive days before a given time, the cheap way to remove in bulk.

Continue Takes the Flow to Continue

Threads are removed. Continue(ctx, flowKey, additionalState) starts a new flow from the final state of exactly the
completed flow you name, and inherits its policy. The caller keeps the chain by continuing the key the last
Continue returned. Continuing one flow twice makes two flows, and continuing a flow that is not completed is a 409.
FlowOptions.ThreadKey, Query.ThreadKey and FlowSummary.ThreadKey are gone. To start the next flow with
different options, call Create with the previous flow's final state, from Snapshot.

Leaner Task Dispatch

The foreman and the generated task handlers now send each direction only what it needs. The foreman sends the state
to the task, and the task replies with its changes and control signals only, so a large state crosses the wire once
per step rather than twice. The generated Executor used in tests still hands back the task's outputs.

The wire format is now versioned, so a foreman and a microservice built against different releases fail the step
loudly instead of misreading each other. Deploy the foreman and every microservice that hosts tasks together.

Subgraphs That Cannot Start

When a task calls flow.Subgraph and the child cannot start, the error now comes back to the task, as a failed
child's does, instead of failing the step at once:

• 503 when no microservice serves the workflow's URL;
• 404 when there is no such graph;
• 400 when the graph is invalid.

Returning the error takes the task's onError transition, as before. To wait out a 503, arm flow.Retry: the next
run's Subgraph call starts the child again.

The same mapping reaches the foreman's own callers: a Create or Run with a workflow URL that no microservice
serves now returns 503 rather than 404. A step whose task host never answers still fails after the foreman's
re-probes, now with a 503, so an onError handler can tell a missing microservice from a task that ran and failed.

Agent Studio

The flows page has two tabs. Open lists the running and interrupted flows. Closed lists the archived flows
within a window: 8 hours, 24 hours by default, 7 days or 30 days. They stay separate rather than merged, because each
has its own order, and a flow that closed in the last minute is in neither. The Continue button is enabled only on a
completed flow, and the dashboard reads both lists.

Other Changes

• flow.Retry's giveUpAfter bounds the total of the backoff delays, not the time since the step was created.
  Time spent queued or running no longer counts, so a step held up behind other work keeps its retries. The foreman's
  re-probe of a missing task host now makes exactly 8 probes across the task's time budget.
• State field names are checked. A name is an ASCII letter or underscore, followed by ASCII letters, digits,
  underscores or hyphens. genservice rejects a task or workflow argument whose JSON name breaks the rule, and a task
  that writes such a name with flow.Set fails its step.
• Fairness keys are at most 256 ASCII letters, digits, -, _ and .. The foreman's default, the caller's
  tenant ID, already qualifies.
• Flow and step keys now carry the flow's creation hour, so a key from an earlier release is a 400.
• Grafana. The metric dwarf_flows_ended is renamed dwarf_flows_closed (dwarf_flows_closed_total in
  Prometheus), and dwarf_operation_seconds's op="list" is now op="list_open" or op="list_closed". The
  workflow dashboard follows the rename and gains an "Archive & Retention" row: records lost to a failing archive
  (alert on it), archive lag, expired flows, archive days kept, flows archived and removed, and reaper pass time.
• sequel v1.18.0. NOW_UTC() now stores whole milliseconds on PostgreSQL, as it already did on the other
  databases. sequel.UnsafeSQL is deprecated but still works as before, and no change is needed in this release.
• Dependencies. dwarf v0.14.0, sequel v1.18.0, Go 1.27.3, and a refresh including pgx v5.12.0 and
  modernc.org/sqlite v1.62.0.

Migration

From inside a Microbus project, ask Claude Code to upgrade Microbus:

Get the latest version of Microbus.

The upgrade-microbus skill first tells you to recreate the foreman's databases, and offers to delete the local
SQLite file. It then rewrites code that calls the foreman's changed endpoints: List, Continue, Purge, Delete
and the removed Query and FlowOptions fields. It renames the metric in dashboards and alerts, and walks through
the behavior changes above. A project that does not call the foreman directly needs no source change beyond
regeneration, but still needs fresh databases.