Skip to content

Archiving That Keeps Up, Trees of Any Size, Recently Closed Flows

Choose a tag to compare

@bw19 bw19 released this 14 Nov 03:30
· 81 commits to main since this release

Archiving, deletion, retention and expiration now run as separate background jobs on each shard, each in batches
that take their turn on the database beside dispatch, so one slow or blocked job holds back no other, and two jobs
share the archiving and the deletion of each shard. A tree too large for one transaction is archived in stages and
deleted in chunks, so a tree of any size reaches the archive or leaves the database; Delete and Purge of an
archived tree return at once and leave the deletion to a background job; and an archive failure never deletes a
record before its retention. On PostgreSQL the database's memory no longer grows with archiving. New calls cover the
minute between a flow closing and being archived (ListRecentlyClosed) and read one flow's summary by its key
(Summary), and every summary names its root and parent step. Upgrading an existing database needs schema changes
by hand
(Breaking), and the archive metrics are reported per job.

New

  • ListRecentlyClosed lists the closed flows not archived yet. A flow that closes leaves ListOpen at once but
    reaches ListClosed only when it is archived, about a minute later and longer while archiving lags, so for that
    minute it was in no list. ListRecentlyClosed covers it: the closed flows still in the live tables, the most
    recently closed first, with the same Query filters as ListOpen, an optional window on UpdatedAt and the
    required Limit; a Status that is not closed is a 400, and a flow marked for deletion is not listed. To see
    every closed flow, call ListRecentlyClosed first and then ListClosed, and drop duplicates by FlowKey. Its
    latency is recorded as op="list_recently_closed" on dwarf_operation_seconds. Like the other lists, it returns
    flow keys, so a host gates it by credential.

  • Summary(flowKey) reads one flow's FlowSummary by its key, open or closed, archived or not - including the
    minute between closing and archiving - so a caller holding only a key can tell a subgraph child from a root
    (Subgraph) and get the flow's workflow name, times and stopped-at task in one call. A flow marked for deletion is
    a 404.

  • FlowSummary gains RootFlowKey and ParentStepKey, filled by Summary and by every list. RootFlowKey is
    the root of the flow's tree - its own key for a root - which Resume, Cancel, Terminate, Delete and
    Continue take; ParentStepKey is the step that called a subgraph child, empty for a root. Like the rest of a
    summary they are keys, so a host that gates the lists by credential gates Summary the same way.

  • Archiving, deletion, retention and expiration run as separate background jobs on each shard. A replica that
    dispatches a shard runs that shard's jobs, each on its own, so a job that is slow, blocked or working through a
    large tree holds back no other; a replica that only reads a shard runs none. Each job works in batches, and each
    batch waits for a turn on the shard's connections at dispatch's priority, its statements taking pooled connections
    as dispatch's do. A fast job that finds nothing waits a second before it looks again; the sweeps,
    large_archiving, live_retention and archive_deletion_sweep, which normally have nothing to do, 30 seconds. The
    jobs, by the value of the job label on their metrics:

    • fast_archiving_1 and fast_archiving_2 archive closed trees as they come due, a minute after they close, each
      taking its share of the shard's trees.
    • sweep_archiving_1 and sweep_archiving_2, each behind its fast job, take the trees it could not: one held by
      another transaction, one whose archiving failed, one that closed late. A sweep takes such a tree once it is 30
      seconds overdue, and tries again every sweep.
    • large_archiving copies trees too large for one transaction into the archive in stages (below).
    • fast_deletion_1 and fast_deletion_2, with sweep_deletion_1 and sweep_deletion_2 behind them, delete the
      trees scheduled for deletion - by Delete, DeleteOnCompletion, expiration, a retention of 0, or given up at
      their retention - a tree larger than a batch in chunks, one tree at a time.
    • live_retention gives up the closed trees archiving never took, once their retention has passed (Fixed).
    • archive_deletion_sweep deletes the archived trees Delete and Purge marked (Fixed).
    • archive_retention makes the archive's days ahead and drops the days past the retention, hourly.
    • expiration ends the flows open past the flow expiration, while SetFlowExpiration is on (Breaking).

    On SQLite, whose single writer a second job would only queue behind, one fast job of each kind runs
    (fast_archiving_1, fast_deletion_1), each with its sweep. A batch is at most 500 roots, about 4,000 rows or 16 MiB of payload, whichever it reaches first. A tree larger
    than a batch but under the staging threshold is archived alone in one longer transaction, and deleted in chunks.

  • Each job is observable on its own. New metrics, reported by each replica dispatching the shard:

    • dwarf_janitor_batch_seconds{shard,job,part}: one batch's time in two parts - turn, the wait for a turn on the
      shard's connections, and work, its statements, their waits for a pooled connection included. A job falling
      behind with most of its time in turn is held back by the shard's load; with most in work, by its statements
      and the pool.
    • dwarf_janitor_progress_age_seconds{shard,job}: how long since the job last made progress, or since it started
      on this replica. It stays near zero while the job has work and grows while it has none, so read it beside the
      job's own readings: growing while its work waits means the job is blocked or failing, and the logs say why.
      Aggregate with max.
    • dwarf_archive_held_roots{shard,kind,job}: the roots a sweep failed on singly, the larger of its current
      sweep's count and its last complete one's. kind is closed (sweep_archiving_k), deleted
      (sweep_deletion_k) or retention (live_retention). One replica's jobs of a kind hold different roots, so sum
      them; two replicas can hold the same root, so aggregate replicas with max. It resets when a replica restarts or
      stops dispatching the shard. Investigate a value that stays above zero.
    • dwarf_archive_oldest_attempt_seconds{shard} and dwarf_archive_staged_rows{shard}, for staged archival
      (below), and dwarf_archive_deletions_pending{shard}, for the archive deletion sweep (Fixed).
  • A tree too large for one transaction is archived in stages. A closed tree of more than 262,144 rows (its flows
    and steps together) or 128 MiB of payload is copied into its archive day a batch at a time, over as many
    transactions as it takes, invisible there until one last small transaction publishes it; its live copy is then
    deleted in chunks. A smaller tree moves in one transaction. Readers and Fork see one whole tree throughout: until
    it is published it is read from the live tables, and ListRecentlyClosed lists it; once published, ListClosed
    lists it under the time its copy began - minutes earlier, or hours for the very largest - so a caller polling
    ListClosed forward sees it late. Large trees are copied one at a time. A copy survives a restart and goes on from
    any replica that dispatches the shard; a Delete during the copy abandons it.

    • dwarf_archive_oldest_attempt_seconds is the age of the oldest copy not yet published, and
      dwarf_archive_staged_rows counts the rows copied: an age that grows while the count moves is a large tree
      copying, one that grows while it is flat is a copy that is stuck, and the logs say why. A tree being copied counts
      in dwarf_archive_lag_seconds for as long as its copy takes.

Fixed

  • An archive failure no longer deletes records before their retention. Archiving that kept failing for a
    minute used to delete closed flows without archiving them, and one bad tree took its batch-mates with it. Now a
    tree the archive cannot take stays in the live tables, readable, and is tried again by its sweep once it is 30
    seconds overdue, and every sweep after; the trees behind it are archived meanwhile. A tree still unarchived once its
    retention has passed - counted from the day it would have been archived, its completion plus the archive grace -
    is given up and deleted, whether or not the archive is reachable. dwarf_flows_reaped{reason="archive_failed"}
    counts its flows when it is given up, and should stay zero; its deletion counts them again under
    reason="deleted", so for such trees the two reasons overlap. Deletion asked for - Delete,
    DeleteOnCompletion, Purge, a retention of 0 - does not wait for the retention. While archiving is held up,
    ListRecentlyClosed lists those flows for longer than the usual minute.
  • A tree of any size can be deleted. Delete, DeleteOnCompletion, a retention of 0 and the retention rule
    above used to delete a tree in one transaction, so a tree large enough - a forEach of hundreds of thousands of
    elements - could never finish and stayed in the live tables for good. A tree larger than a batch is now deleted
    over several transactions of about a batch each, its root last, one tree at a time. It is marked for deletion
    before its first chunk, so a deletion stopped by a restart is finished later, on any replica that dispatches the
    shard. Readers never see part of a tree: a tree marked for deletion - the root and every subgraph flow under it -
    is not listed and is a 404 from Summary, and once its deletion is due (at once after Delete, after the grace for
    DeleteOnCompletion) Step and History of it are a 404 too. Snapshot, Await and Fingerprint of its root
    still serve its outcome until the root is deleted. On PostgreSQL the chunks read a flow's steps through
    idx_dwarf_steps_flow_id, now keyed (flow_id, step_id) (Breaking: schema).
  • Delete and Purge of an archived tree of any size return at once. Each now only marks the tree for
    deletion. The mark hides the tree, its root and every subgraph flow, from
    every read and list when the call returns, and counts it in dwarf_flows_removed. A background job, the archive
    deletion sweep,
    then deletes its rows: the trees a batch holds together, a larger one in chunks, one at a time. Earlier the call
    deleted the rows itself, in one transaction that a large enough tree could not finish. The storage is released
    when the sweep reaches the tree, or when its day drops. dwarf_archive_deletions_pending counts the marked trees a
    sweep found and could not delete - held by another transaction, or failing. It is above zero for a moment after a
    Purge; one that stays above zero across sweeps is the one to look at. Purge's done describes what the call
    found. Delete of a flow still in DeleteOnCompletion's grace now makes it due at once.
  • Listing archived roots past a very wide tree is fast. ListClosed without IncludeSubgraphs, and Purge, read
    a new index of each archive day's roots instead of walking past every subgraph flow archived in their window (79-837
    ms to 0.2-3 ms for a page beside a tree of 300,000 subgraph flows). Each archive day also gets an index of the trees
    marked for deletion, which the archive deletion sweep reads, and on PostgreSQL its steps index becomes
    (flow_id, step_id), as the live one. Days made by v0.14.0 have none of these until they are added (Breaking:
    schema), or until retention drops the days.
  • Scheduled deletion keeps up through a long archive outage. The scan for flows scheduled for deletion (Delete,
    DeleteOnCompletion) read past every closed flow held in the live tables before reaching them; it now reads only
    the scheduled ones (1.29 s to 0.35 ms at 1.2M held flows on PostgreSQL). The index behind it,
    idx_dwarf_flows_delete_after, is keyed on flow_id on PostgreSQL, SQLite and SQL Server (Breaking: schema).
  • On PostgreSQL, the database's memory no longer grows with archiving and deletion. The engine wrote the ids of
    the rows it archived, deleted, marked and recovered into the text of each statement, so nearly every such statement
    was a new one. The PostgreSQL driver keeps the statements it runs prepared on each connection, each holding memory
    on the server, and these filled every connection's cache until the database could run out of memory. Every id is
    now a bound parameter, in a list padded to one of a few lengths, so those statements take a handful of forms.
  • Terminate no longer lets a step it caught mid-transition change the terminated flow. A step whose task
    finished just before Terminate could still write afterwards: a failed transition rewrote the step as failed or
    cancelled, an errored one put it back to pending, a cancellation's error route added onErr to its changes, and a
    fan-in arrival bumped the terminated flow's counters. Terminate now fences those writes, and the step abandons
    them.
  • SQL Server: a cancelled request no longer breaks a pooled connection. A transaction ended by a cancelled
    context - a caller giving up mid-call, or Shutdown - could leave its connection in the pool unusable, and the next
    operation to draw it failed once with Read: EOF. A driver bug (go-mssqldb v1.11.2, microsoft/go-mssqldb#390),
    worked around in sequel v1.18.1: such a connection is now discarded. A timed-out request was never affected.
  • Flow expiration keeps to its time budget when it terminates nothing, and goes on where it stopped. A long run
    of closed flows still in the live tables - while archiving is behind - could hold an expiration pass past its
    30-second budget, and, walked from the oldest every pass, keep the pass from reaching an expired flow behind them.
    Each pass now stops at its budget, and the next goes on at once after the last flow it read. A flow whose
    termination fails is retried once the walk has reached the newest expired flow, not on the very next pass.

Breaking

  • An existing database needs schema changes by hand. Migrations are edited in place before v1.0, so a database
    made by v0.14.0 does not pick these up, and Startup does not check for them; a new database gets them all. Apply
    the first group with every replica stopped, before starting v0.15.0: without it the engine fails. The engine runs
    without the second group, more slowly; apply it in the same window, or later at a quiet time. A shard's archive
    days are the tables dwarf_flows_archive_<n> and dwarf_steps_archive_<n>, and their indexes are named for them
    (idx_dwarf_flows_archive_<n>_root).
    • Before the upgrade.
      • Eight byte-count columns, each the length of a payload column as the engine wrote it, so archiving
        and deletion size a batch by its payload without reading it: graph_bytes, baggage_bytes and
        final_state_bytes on dwarf_flows, and state_bytes, changes_bytes, interrupt_payload_bytes,
        resume_data_bytes and subgraph_result_bytes on dwarf_steps. Without them Create and every step's write
        fail. A row written before the upgrade counts 0 bytes, so it is sized by its rows alone.
      • dwarf_flows.archive_attempt_at, the time a large tree's staged copy began. Without it the engine fails on
        its first read of the column, on every shard.
      • On MySQL and MariaDB, the index idx_dwarf_flows_attempt, which large-tree archiving names: without it no
        large tree is archived. The other databases may add theirs after.
      • Every archive day that exists at the upgrade keeps v0.14.0's definition: Startup migrates a day only for
        a new archive migration, never an edited one. Each day needs the byte columns: without them archiving into
        today and the days ahead fails, closed trees staying live until a day made by v0.15.0 begins; Fork of a flow
        archived on any such day fails; and the archive deletion sweep cannot delete a tree marked there, so its rows
        stay until the day drops. On PostgreSQL adding a column with a constant default rewrites no rows.
    • After, at any time. Without these the engine reads more than it must: idx_dwarf_flows_attempt (but on MySQL
      and MariaDB, above) finds the trees being copied, idx_dwarf_flows_delete_after walks the scheduled trees in the
      deletion's order, and on PostgreSQL idx_dwarf_flows_root (root_flow_id, flow_id) and idx_dwarf_steps_flow_id (flow_id, step_id) keep a large tree's chunks bounded, idx_dwarf_flows_closed_roots serves the archiving scans,
      and dwarf_steps is analyzed at 1% of rows changed (autovacuum_analyze_scale_factor = 0.01; the default is
      10%), so the planner learns a large new flow sooner. A day without the roots and deletion indexes, and on
      PostgreSQL without the re-keyed root and steps indexes, keeps the slower plans until it drops.
    • PostgreSQL. Before:
      ALTER TABLE dwarf_flows ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0, ADD COLUMN archive_attempt_at TIMESTAMP(3) NOT NULL DEFAULT '2000-01-01 00:00:00';
      ALTER TABLE dwarf_steps ADD COLUMN state_bytes INT NOT NULL DEFAULT 0, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0;
      -- every archive day <n>:
      ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN state_bytes INT NOT NULL DEFAULT 0, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0;
      After:
      CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (flow_id) WHERE archive_attempt_at > '2000-01-01 00:00:00';
      CREATE INDEX idx_dwarf_flows_closed_roots ON dwarf_flows (status, updated_at, flow_id) WHERE surgraph_flow_id=0 AND delete_after_ms=0;
      DROP INDEX idx_dwarf_flows_delete_after;
      CREATE INDEX idx_dwarf_flows_delete_after ON dwarf_flows (flow_id) WHERE delete_after_ms > 0;
      DROP INDEX idx_dwarf_flows_root;
      CREATE INDEX idx_dwarf_flows_root ON dwarf_flows (root_flow_id, flow_id);
      DROP INDEX idx_dwarf_steps_flow_id;
      CREATE INDEX idx_dwarf_steps_flow_id ON dwarf_steps (flow_id, step_id);
      ALTER TABLE dwarf_steps SET (autovacuum_analyze_scale_factor = 0.01);
      -- every archive day <n>:
      CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (archived_at, flow_id) WHERE surgraph_flow_id = 0;
      CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (flow_id) WHERE delete_after_ms > 0;
      DROP INDEX idx_dwarf_flows_archive_<n>_root;
      CREATE INDEX idx_dwarf_flows_archive_<n>_root ON dwarf_flows_archive_<n> (root_flow_id, flow_id);
      DROP INDEX idx_dwarf_steps_archive_<n>_flow_id;
      CREATE INDEX idx_dwarf_steps_archive_<n>_flow_id ON dwarf_steps_archive_<n> (flow_id, step_id);
    • MySQL and MariaDB. Before:
      ALTER TABLE dwarf_flows ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0 AFTER graph, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0 AFTER baggage, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0 AFTER final_state, ADD COLUMN archive_attempt_at DATETIME(3) NOT NULL DEFAULT '2000-01-01 00:00:00' AFTER touch;
      CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (archive_attempt_at);
      ALTER TABLE dwarf_steps ADD COLUMN state_bytes INT NOT NULL DEFAULT 0 AFTER state, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0 AFTER changes, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0 AFTER interrupt_payload, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0 AFTER resume_data, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0 AFTER subgraph_result;
      -- every archive day <n>:
      ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0 AFTER graph, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0 AFTER baggage, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0 AFTER final_state;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN state_bytes INT NOT NULL DEFAULT 0 AFTER state, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0 AFTER changes, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0 AFTER interrupt_payload, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0 AFTER resume_data, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0 AFTER subgraph_result;
      After, on the archive days only (the live tables need nothing more):
      CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (surgraph_flow_id, archived_at, flow_id);
      CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (delete_after_ms);
    • SQL Server. Before:
      ALTER TABLE dwarf_flows ADD graph_bytes INT NOT NULL DEFAULT 0, baggage_bytes INT NOT NULL DEFAULT 0, final_state_bytes INT NOT NULL DEFAULT 0, archive_attempt_at DATETIME2(3) NOT NULL DEFAULT '2000-01-01 00:00:00';
      ALTER TABLE dwarf_steps ADD state_bytes INT NOT NULL DEFAULT 0, changes_bytes INT NOT NULL DEFAULT 0, interrupt_payload_bytes INT NOT NULL DEFAULT 0, resume_data_bytes INT NOT NULL DEFAULT 0, subgraph_result_bytes INT NOT NULL DEFAULT 0;
      -- every archive day <n>:
      ALTER TABLE dwarf_flows_archive_<n> ADD graph_bytes INT NOT NULL DEFAULT 0, baggage_bytes INT NOT NULL DEFAULT 0, final_state_bytes INT NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD state_bytes INT NOT NULL DEFAULT 0, changes_bytes INT NOT NULL DEFAULT 0, interrupt_payload_bytes INT NOT NULL DEFAULT 0, resume_data_bytes INT NOT NULL DEFAULT 0, subgraph_result_bytes INT NOT NULL DEFAULT 0;
      After:
      CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (flow_id) WHERE archive_attempt_at > '2000-01-01 00:00:00';
      DROP INDEX idx_dwarf_flows_delete_after ON dwarf_flows;
      CREATE INDEX idx_dwarf_flows_delete_after ON dwarf_flows (flow_id) WHERE delete_after_ms > 0;
      -- every archive day <n>:
      CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (archived_at, flow_id) WHERE surgraph_flow_id = 0;
      CREATE INDEX idx_dwarf_flows_archive_<n>_roots_status ON dwarf_flows_archive_<n> (status, archived_at, flow_id) WHERE surgraph_flow_id = 0;
      CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (flow_id) WHERE delete_after_ms > 0;
    • SQLite. Before:
      ALTER TABLE dwarf_flows ADD COLUMN graph_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_flows ADD COLUMN baggage_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_flows ADD COLUMN final_state_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_flows ADD COLUMN archive_attempt_at DATETIME NOT NULL DEFAULT '2000-01-01 00:00:00.000';
      ALTER TABLE dwarf_steps ADD COLUMN state_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps ADD COLUMN changes_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps ADD COLUMN interrupt_payload_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps ADD COLUMN resume_data_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps ADD COLUMN subgraph_result_bytes INTEGER NOT NULL DEFAULT 0;
      -- every archive day <n>:
      ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN graph_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN baggage_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN final_state_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN state_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN changes_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN interrupt_payload_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN resume_data_bytes INTEGER NOT NULL DEFAULT 0;
      ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN subgraph_result_bytes INTEGER NOT NULL DEFAULT 0;
      After:
      CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (flow_id) WHERE archive_attempt_at > '2000-01-01 00:00:00.000';
      DROP INDEX idx_dwarf_flows_delete_after;
      CREATE INDEX idx_dwarf_flows_delete_after ON dwarf_flows (flow_id) WHERE delete_after_ms > 0;
      -- every archive day <n>:
      CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (archived_at, flow_id) WHERE surgraph_flow_id = 0;
      CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (flow_id) WHERE delete_after_ms > 0;
  • dwarf_reap_pass_seconds, dwarf_archive_days and dwarf_archive_days_dropped are removed: update the
    dashboards and alerts that read them. The background work is timed per job now, by dwarf_janitor_batch_seconds,
    and whether each job keeps moving is dwarf_janitor_progress_age_seconds (New). dwarf_archive_oldest_day_age_seconds
    shows the retention being kept - about the retention plus a day - and dwarf_archive_maintain_failed counts the
    failed passes of archive_retention.
  • dwarf_archive_lag_seconds and dwarf_archive_failing_seconds gain a job label, so a shard reports several
    series of each: aggregate with max for the shard's reading.
    • dwarf_archive_lag_seconds has a series per fast archiving job (fast_archiving_1, fast_archiving_2; one on
      SQLite): how long past the archive grace the oldest closed root of the job's share has waited. It now counts every
      such root, one a failure left behind and a large tree being copied in stages included, and 0 once none is due.
      It is read about once a second and goes on aging between readings. So one failed root shows in it until its sweep
      takes it, up to about a minute, and a large tree's copy for as long as the copy takes.
    • dwarf_archive_failing_seconds has a series per archiving job, each how long that job's archiving has been
      failing: a fast_archiving_k job's until a later move archives something, or until every root of its share due
      at its last failure has left the live tables; a sweep_archiving_k job's until a sweep completes failing nothing;
      large_archiving's until a round over every copy in progress fails none. A tree whose archiving fails stays
      live, never deleted for it before its retention (Fixed).
  • Expired flows are deleted, not archived, and expiration runs every hour. A flow SetFlowExpiration ends is
    terminated as before and, in the same write, marked for deletion; it is deleted within a few minutes, counted in
    dwarf_flows_reaped{reason="deleted"}. It is never archived, so ListClosed never lists it and it cannot be
    forked. Until the deletion, Snapshot and Await report its terminated outcome; the lists and History leave it
    out at once. The check runs on each shard every hour, or every half of the expiration when that is shorter, at least
    a minute apart, where it ran every 5 minutes: a flow is terminated up to that long after it expires.
    dwarf_flows_expire_failed counts a root that fails again on every pass, up to an hour apart.
  • Every duration dwarf writes as JSON is an integer number of milliseconds, as the carrier's sleepMillis and
    backoff*Millis already were. An integer is exact in every language, and a reducer (add, min, max) or a
    when expression treats it as a number.
    • SetDuration stores whole milliseconds, rounded to the nearest one, and GetDuration reads them; it panics on
      anything but an integer. A duration stored in nanoseconds by an earlier version reads a million times too long:
      a flow that is open across the upgrade and holds one should be finished first, or its field rewritten.
    • Set with a time.Duration, or one in a flow's initial state, still stores Go's encoding, nanoseconds, which
      GetDuration does not read: use SetDuration, or store d.Milliseconds().
    • FlowOptions.TimeBudget is "timeBudgetMillis" in JSON (was "timeBudget", nanoseconds), a positive fraction
      of a millisecond rounded up; ShardSummary.Latency is "latencyMillis" (was "latency", nanoseconds), rounded
      to the nearest millisecond. Both stay time.Duration in Go.
  • Engine.FlowsStarted and Engine.FlowsClosed are removed. They counted only one replica's flows - a flow
    started on one replica is closed by whichever dispatches its last step - so their difference was not the number
    of open flows, as their documentation said. Use the dwarf_flows_started and dwarf_flows_closed metrics,
    summed over the fleet, or ListOpen.
  • The correlation ID is {shard}-{hour}-{flowID}, the flow key without its token (was {shard}-{flowID}), in the
    workflow.id span attribute and the flow/childFlow log values. Find its flow with Query.Search set to the ID
    followed by -, and Query.Shard set; a dashboard or log query keyed on the old form needs updating.