Repository navigation
Archiving That Keeps Up, Trees of Any Size, Recently Closed Flows
Archiving, deletion, retention and expiration now run as separate background jobs on each shard, each in batches
that take their turn on the database beside dispatch, so one slow or blocked job holds back no other, and two jobs
share the archiving and the deletion of each shard. A tree too large for one transaction is archived in stages and
deleted in chunks, so a tree of any size reaches the archive or leaves the database; Delete and Purge of an
archived tree return at once and leave the deletion to a background job; and an archive failure never deletes a
record before its retention. On PostgreSQL the database's memory no longer grows with archiving. New calls cover the
minute between a flow closing and being archived (ListRecentlyClosed) and read one flow's summary by its key
(Summary), and every summary names its root and parent step. Upgrading an existing database needs schema changes
by hand (Breaking), and the archive metrics are reported per job.
New
-
ListRecentlyClosedlists the closed flows not archived yet. A flow that closes leavesListOpenat once but
reachesListClosedonly when it is archived, about a minute later and longer while archiving lags, so for that
minute it was in no list.ListRecentlyClosedcovers it: the closed flows still in the live tables, the most
recently closed first, with the sameQueryfilters asListOpen, an optional window onUpdatedAtand the
requiredLimit; aStatusthat is not closed is a 400, and a flow marked for deletion is not listed. To see
every closed flow, callListRecentlyClosedfirst and thenListClosed, and drop duplicates byFlowKey. Its
latency is recorded asop="list_recently_closed"ondwarf_operation_seconds. Like the other lists, it returns
flow keys, so a host gates it by credential. -
Summary(flowKey)reads one flow'sFlowSummaryby its key, open or closed, archived or not - including the
minute between closing and archiving - so a caller holding only a key can tell a subgraph child from a root
(Subgraph) and get the flow's workflow name, times and stopped-at task in one call. A flow marked for deletion is
a 404. -
FlowSummarygainsRootFlowKeyandParentStepKey, filled bySummaryand by every list.RootFlowKeyis
the root of the flow's tree - its own key for a root - whichResume,Cancel,Terminate,Deleteand
Continuetake;ParentStepKeyis the step that called a subgraph child, empty for a root. Like the rest of a
summary they are keys, so a host that gates the lists by credential gatesSummarythe same way. -
Archiving, deletion, retention and expiration run as separate background jobs on each shard. A replica that
dispatches a shard runs that shard's jobs, each on its own, so a job that is slow, blocked or working through a
large tree holds back no other; a replica that only reads a shard runs none. Each job works in batches, and each
batch waits for a turn on the shard's connections at dispatch's priority, its statements taking pooled connections
as dispatch's do. A fast job that finds nothing waits a second before it looks again; the sweeps,
large_archiving,live_retentionandarchive_deletion_sweep, which normally have nothing to do, 30 seconds. The
jobs, by the value of thejoblabel on their metrics:fast_archiving_1andfast_archiving_2archive closed trees as they come due, a minute after they close, each
taking its share of the shard's trees.sweep_archiving_1andsweep_archiving_2, each behind its fast job, take the trees it could not: one held by
another transaction, one whose archiving failed, one that closed late. A sweep takes such a tree once it is 30
seconds overdue, and tries again every sweep.large_archivingcopies trees too large for one transaction into the archive in stages (below).fast_deletion_1andfast_deletion_2, withsweep_deletion_1andsweep_deletion_2behind them, delete the
trees scheduled for deletion - byDelete,DeleteOnCompletion, expiration, a retention of 0, or given up at
their retention - a tree larger than a batch in chunks, one tree at a time.live_retentiongives up the closed trees archiving never took, once their retention has passed (Fixed).archive_deletion_sweepdeletes the archived treesDeleteandPurgemarked (Fixed).archive_retentionmakes the archive's days ahead and drops the days past the retention, hourly.expirationends the flows open past the flow expiration, whileSetFlowExpirationis on (Breaking).
On SQLite, whose single writer a second job would only queue behind, one fast job of each kind runs
(fast_archiving_1,fast_deletion_1), each with its sweep. A batch is at most 500 roots, about 4,000 rows or 16 MiB of payload, whichever it reaches first. A tree larger
than a batch but under the staging threshold is archived alone in one longer transaction, and deleted in chunks. -
Each job is observable on its own. New metrics, reported by each replica dispatching the shard:
dwarf_janitor_batch_seconds{shard,job,part}: one batch's time in two parts -turn, the wait for a turn on the
shard's connections, andwork, its statements, their waits for a pooled connection included. A job falling
behind with most of its time inturnis held back by the shard's load; with most inwork, by its statements
and the pool.dwarf_janitor_progress_age_seconds{shard,job}: how long since the job last made progress, or since it started
on this replica. It stays near zero while the job has work and grows while it has none, so read it beside the
job's own readings: growing while its work waits means the job is blocked or failing, and the logs say why.
Aggregate with max.dwarf_archive_held_roots{shard,kind,job}: the roots a sweep failed on singly, the larger of its current
sweep's count and its last complete one's.kindisclosed(sweep_archiving_k),deleted
(sweep_deletion_k) orretention(live_retention). One replica's jobs of a kind hold different roots, so sum
them; two replicas can hold the same root, so aggregate replicas with max. It resets when a replica restarts or
stops dispatching the shard. Investigate a value that stays above zero.dwarf_archive_oldest_attempt_seconds{shard}anddwarf_archive_staged_rows{shard}, for staged archival
(below), anddwarf_archive_deletions_pending{shard}, for the archive deletion sweep (Fixed).
-
A tree too large for one transaction is archived in stages. A closed tree of more than 262,144 rows (its flows
and steps together) or 128 MiB of payload is copied into its archive day a batch at a time, over as many
transactions as it takes, invisible there until one last small transaction publishes it; its live copy is then
deleted in chunks. A smaller tree moves in one transaction. Readers andForksee one whole tree throughout: until
it is published it is read from the live tables, andListRecentlyClosedlists it; once published,ListClosed
lists it under the time its copy began - minutes earlier, or hours for the very largest - so a caller polling
ListClosedforward sees it late. Large trees are copied one at a time. A copy survives a restart and goes on from
any replica that dispatches the shard; aDeleteduring the copy abandons it.dwarf_archive_oldest_attempt_secondsis the age of the oldest copy not yet published, and
dwarf_archive_staged_rowscounts the rows copied: an age that grows while the count moves is a large tree
copying, one that grows while it is flat is a copy that is stuck, and the logs say why. A tree being copied counts
indwarf_archive_lag_secondsfor as long as its copy takes.
Fixed
- An archive failure no longer deletes records before their retention. Archiving that kept failing for a
minute used to delete closed flows without archiving them, and one bad tree took its batch-mates with it. Now a
tree the archive cannot take stays in the live tables, readable, and is tried again by its sweep once it is 30
seconds overdue, and every sweep after; the trees behind it are archived meanwhile. A tree still unarchived once its
retention has passed - counted from the day it would have been archived, its completion plus the archive grace -
is given up and deleted, whether or not the archive is reachable.dwarf_flows_reaped{reason="archive_failed"}
counts its flows when it is given up, and should stay zero; its deletion counts them again under
reason="deleted", so for such trees the two reasons overlap. Deletion asked for -Delete,
DeleteOnCompletion,Purge, a retention of 0 - does not wait for the retention. While archiving is held up,
ListRecentlyClosedlists those flows for longer than the usual minute. - A tree of any size can be deleted.
Delete,DeleteOnCompletion, a retention of 0 and the retention rule
above used to delete a tree in one transaction, so a tree large enough - aforEachof hundreds of thousands of
elements - could never finish and stayed in the live tables for good. A tree larger than a batch is now deleted
over several transactions of about a batch each, its root last, one tree at a time. It is marked for deletion
before its first chunk, so a deletion stopped by a restart is finished later, on any replica that dispatches the
shard. Readers never see part of a tree: a tree marked for deletion - the root and every subgraph flow under it -
is not listed and is a 404 fromSummary, and once its deletion is due (at once afterDelete, after the grace for
DeleteOnCompletion)StepandHistoryof it are a 404 too.Snapshot,AwaitandFingerprintof its root
still serve its outcome until the root is deleted. On PostgreSQL the chunks read a flow's steps through
idx_dwarf_steps_flow_id, now keyed(flow_id, step_id)(Breaking: schema). DeleteandPurgeof an archived tree of any size return at once. Each now only marks the tree for
deletion. The mark hides the tree, its root and every subgraph flow, from
every read and list when the call returns, and counts it indwarf_flows_removed. A background job, the archive
deletion sweep,
then deletes its rows: the trees a batch holds together, a larger one in chunks, one at a time. Earlier the call
deleted the rows itself, in one transaction that a large enough tree could not finish. The storage is released
when the sweep reaches the tree, or when its day drops.dwarf_archive_deletions_pendingcounts the marked trees a
sweep found and could not delete - held by another transaction, or failing. It is above zero for a moment after a
Purge; one that stays above zero across sweeps is the one to look at.Purge'sdonedescribes what the call
found.Deleteof a flow still inDeleteOnCompletion's grace now makes it due at once.- Listing archived roots past a very wide tree is fast.
ListClosedwithoutIncludeSubgraphs, andPurge, read
a new index of each archive day's roots instead of walking past every subgraph flow archived in their window (79-837
ms to 0.2-3 ms for a page beside a tree of 300,000 subgraph flows). Each archive day also gets an index of the trees
marked for deletion, which the archive deletion sweep reads, and on PostgreSQL its steps index becomes
(flow_id, step_id), as the live one. Days made by v0.14.0 have none of these until they are added (Breaking:
schema), or until retention drops the days. - Scheduled deletion keeps up through a long archive outage. The scan for flows scheduled for deletion (
Delete,
DeleteOnCompletion) read past every closed flow held in the live tables before reaching them; it now reads only
the scheduled ones (1.29 s to 0.35 ms at 1.2M held flows on PostgreSQL). The index behind it,
idx_dwarf_flows_delete_after, is keyed onflow_idon PostgreSQL, SQLite and SQL Server (Breaking: schema). - On PostgreSQL, the database's memory no longer grows with archiving and deletion. The engine wrote the ids of
the rows it archived, deleted, marked and recovered into the text of each statement, so nearly every such statement
was a new one. The PostgreSQL driver keeps the statements it runs prepared on each connection, each holding memory
on the server, and these filled every connection's cache until the database could run out of memory. Every id is
now a bound parameter, in a list padded to one of a few lengths, so those statements take a handful of forms. Terminateno longer lets a step it caught mid-transition change the terminated flow. A step whose task
finished just beforeTerminatecould still write afterwards: a failed transition rewrote the step as failed or
cancelled, an errored one put it back to pending, a cancellation's error route addedonErrto its changes, and a
fan-in arrival bumped the terminated flow's counters.Terminatenow fences those writes, and the step abandons
them.- SQL Server: a cancelled request no longer breaks a pooled connection. A transaction ended by a cancelled
context - a caller giving up mid-call, or Shutdown - could leave its connection in the pool unusable, and the next
operation to draw it failed once withRead: EOF. A driver bug (go-mssqldb v1.11.2, microsoft/go-mssqldb#390),
worked around in sequel v1.18.1: such a connection is now discarded. A timed-out request was never affected. - Flow expiration keeps to its time budget when it terminates nothing, and goes on where it stopped. A long run
of closed flows still in the live tables - while archiving is behind - could hold an expiration pass past its
30-second budget, and, walked from the oldest every pass, keep the pass from reaching an expired flow behind them.
Each pass now stops at its budget, and the next goes on at once after the last flow it read. A flow whose
termination fails is retried once the walk has reached the newest expired flow, not on the very next pass.
Breaking
- An existing database needs schema changes by hand. Migrations are edited in place before v1.0, so a database
made by v0.14.0 does not pick these up, andStartupdoes not check for them; a new database gets them all. Apply
the first group with every replica stopped, before starting v0.15.0: without it the engine fails. The engine runs
without the second group, more slowly; apply it in the same window, or later at a quiet time. A shard's archive
days are the tablesdwarf_flows_archive_<n>anddwarf_steps_archive_<n>, and their indexes are named for them
(idx_dwarf_flows_archive_<n>_root).- Before the upgrade.
- Eight byte-count columns, each the length of a payload column as the engine wrote it, so archiving
and deletion size a batch by its payload without reading it:graph_bytes,baggage_bytesand
final_state_bytesondwarf_flows, andstate_bytes,changes_bytes,interrupt_payload_bytes,
resume_data_bytesandsubgraph_result_bytesondwarf_steps. Without themCreateand every step's write
fail. A row written before the upgrade counts 0 bytes, so it is sized by its rows alone. dwarf_flows.archive_attempt_at, the time a large tree's staged copy began. Without it the engine fails on
its first read of the column, on every shard.- On MySQL and MariaDB, the index
idx_dwarf_flows_attempt, which large-tree archiving names: without it no
large tree is archived. The other databases may add theirs after. - Every archive day that exists at the upgrade keeps v0.14.0's definition:
Startupmigrates a day only for
a new archive migration, never an edited one. Each day needs the byte columns: without them archiving into
today and the days ahead fails, closed trees staying live until a day made by v0.15.0 begins;Forkof a flow
archived on any such day fails; and the archive deletion sweep cannot delete a tree marked there, so its rows
stay until the day drops. On PostgreSQL adding a column with a constant default rewrites no rows.
- Eight byte-count columns, each the length of a payload column as the engine wrote it, so archiving
- After, at any time. Without these the engine reads more than it must:
idx_dwarf_flows_attempt(but on MySQL
and MariaDB, above) finds the trees being copied,idx_dwarf_flows_delete_afterwalks the scheduled trees in the
deletion's order, and on PostgreSQLidx_dwarf_flows_root (root_flow_id, flow_id)andidx_dwarf_steps_flow_id (flow_id, step_id)keep a large tree's chunks bounded,idx_dwarf_flows_closed_rootsserves the archiving scans,
anddwarf_stepsis analyzed at 1% of rows changed (autovacuum_analyze_scale_factor = 0.01; the default is
10%), so the planner learns a large new flow sooner. A day without the roots and deletion indexes, and on
PostgreSQL without the re-keyed root and steps indexes, keeps the slower plans until it drops. - PostgreSQL. Before:
After:
ALTER TABLE dwarf_flows ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0, ADD COLUMN archive_attempt_at TIMESTAMP(3) NOT NULL DEFAULT '2000-01-01 00:00:00'; ALTER TABLE dwarf_steps ADD COLUMN state_bytes INT NOT NULL DEFAULT 0, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0; -- every archive day <n>: ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN state_bytes INT NOT NULL DEFAULT 0, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0;
CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (flow_id) WHERE archive_attempt_at > '2000-01-01 00:00:00'; CREATE INDEX idx_dwarf_flows_closed_roots ON dwarf_flows (status, updated_at, flow_id) WHERE surgraph_flow_id=0 AND delete_after_ms=0; DROP INDEX idx_dwarf_flows_delete_after; CREATE INDEX idx_dwarf_flows_delete_after ON dwarf_flows (flow_id) WHERE delete_after_ms > 0; DROP INDEX idx_dwarf_flows_root; CREATE INDEX idx_dwarf_flows_root ON dwarf_flows (root_flow_id, flow_id); DROP INDEX idx_dwarf_steps_flow_id; CREATE INDEX idx_dwarf_steps_flow_id ON dwarf_steps (flow_id, step_id); ALTER TABLE dwarf_steps SET (autovacuum_analyze_scale_factor = 0.01); -- every archive day <n>: CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (archived_at, flow_id) WHERE surgraph_flow_id = 0; CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (flow_id) WHERE delete_after_ms > 0; DROP INDEX idx_dwarf_flows_archive_<n>_root; CREATE INDEX idx_dwarf_flows_archive_<n>_root ON dwarf_flows_archive_<n> (root_flow_id, flow_id); DROP INDEX idx_dwarf_steps_archive_<n>_flow_id; CREATE INDEX idx_dwarf_steps_archive_<n>_flow_id ON dwarf_steps_archive_<n> (flow_id, step_id);
- MySQL and MariaDB. Before:
After, on the archive days only (the live tables need nothing more):
ALTER TABLE dwarf_flows ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0 AFTER graph, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0 AFTER baggage, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0 AFTER final_state, ADD COLUMN archive_attempt_at DATETIME(3) NOT NULL DEFAULT '2000-01-01 00:00:00' AFTER touch; CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (archive_attempt_at); ALTER TABLE dwarf_steps ADD COLUMN state_bytes INT NOT NULL DEFAULT 0 AFTER state, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0 AFTER changes, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0 AFTER interrupt_payload, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0 AFTER resume_data, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0 AFTER subgraph_result; -- every archive day <n>: ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN graph_bytes INT NOT NULL DEFAULT 0 AFTER graph, ADD COLUMN baggage_bytes INT NOT NULL DEFAULT 0 AFTER baggage, ADD COLUMN final_state_bytes INT NOT NULL DEFAULT 0 AFTER final_state; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN state_bytes INT NOT NULL DEFAULT 0 AFTER state, ADD COLUMN changes_bytes INT NOT NULL DEFAULT 0 AFTER changes, ADD COLUMN interrupt_payload_bytes INT NOT NULL DEFAULT 0 AFTER interrupt_payload, ADD COLUMN resume_data_bytes INT NOT NULL DEFAULT 0 AFTER resume_data, ADD COLUMN subgraph_result_bytes INT NOT NULL DEFAULT 0 AFTER subgraph_result;
CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (surgraph_flow_id, archived_at, flow_id); CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (delete_after_ms);
- SQL Server. Before:
After:
ALTER TABLE dwarf_flows ADD graph_bytes INT NOT NULL DEFAULT 0, baggage_bytes INT NOT NULL DEFAULT 0, final_state_bytes INT NOT NULL DEFAULT 0, archive_attempt_at DATETIME2(3) NOT NULL DEFAULT '2000-01-01 00:00:00'; ALTER TABLE dwarf_steps ADD state_bytes INT NOT NULL DEFAULT 0, changes_bytes INT NOT NULL DEFAULT 0, interrupt_payload_bytes INT NOT NULL DEFAULT 0, resume_data_bytes INT NOT NULL DEFAULT 0, subgraph_result_bytes INT NOT NULL DEFAULT 0; -- every archive day <n>: ALTER TABLE dwarf_flows_archive_<n> ADD graph_bytes INT NOT NULL DEFAULT 0, baggage_bytes INT NOT NULL DEFAULT 0, final_state_bytes INT NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD state_bytes INT NOT NULL DEFAULT 0, changes_bytes INT NOT NULL DEFAULT 0, interrupt_payload_bytes INT NOT NULL DEFAULT 0, resume_data_bytes INT NOT NULL DEFAULT 0, subgraph_result_bytes INT NOT NULL DEFAULT 0;
CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (flow_id) WHERE archive_attempt_at > '2000-01-01 00:00:00'; DROP INDEX idx_dwarf_flows_delete_after ON dwarf_flows; CREATE INDEX idx_dwarf_flows_delete_after ON dwarf_flows (flow_id) WHERE delete_after_ms > 0; -- every archive day <n>: CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (archived_at, flow_id) WHERE surgraph_flow_id = 0; CREATE INDEX idx_dwarf_flows_archive_<n>_roots_status ON dwarf_flows_archive_<n> (status, archived_at, flow_id) WHERE surgraph_flow_id = 0; CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (flow_id) WHERE delete_after_ms > 0;
- SQLite. Before:
After:
ALTER TABLE dwarf_flows ADD COLUMN graph_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_flows ADD COLUMN baggage_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_flows ADD COLUMN final_state_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_flows ADD COLUMN archive_attempt_at DATETIME NOT NULL DEFAULT '2000-01-01 00:00:00.000'; ALTER TABLE dwarf_steps ADD COLUMN state_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps ADD COLUMN changes_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps ADD COLUMN interrupt_payload_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps ADD COLUMN resume_data_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps ADD COLUMN subgraph_result_bytes INTEGER NOT NULL DEFAULT 0; -- every archive day <n>: ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN graph_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN baggage_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_flows_archive_<n> ADD COLUMN final_state_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN state_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN changes_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN interrupt_payload_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN resume_data_bytes INTEGER NOT NULL DEFAULT 0; ALTER TABLE dwarf_steps_archive_<n> ADD COLUMN subgraph_result_bytes INTEGER NOT NULL DEFAULT 0;
CREATE INDEX idx_dwarf_flows_attempt ON dwarf_flows (flow_id) WHERE archive_attempt_at > '2000-01-01 00:00:00.000'; DROP INDEX idx_dwarf_flows_delete_after; CREATE INDEX idx_dwarf_flows_delete_after ON dwarf_flows (flow_id) WHERE delete_after_ms > 0; -- every archive day <n>: CREATE INDEX idx_dwarf_flows_archive_<n>_roots ON dwarf_flows_archive_<n> (archived_at, flow_id) WHERE surgraph_flow_id = 0; CREATE INDEX idx_dwarf_flows_archive_<n>_delete_after ON dwarf_flows_archive_<n> (flow_id) WHERE delete_after_ms > 0;
- Before the upgrade.
dwarf_reap_pass_seconds,dwarf_archive_daysanddwarf_archive_days_droppedare removed: update the
dashboards and alerts that read them. The background work is timed per job now, bydwarf_janitor_batch_seconds,
and whether each job keeps moving isdwarf_janitor_progress_age_seconds(New).dwarf_archive_oldest_day_age_seconds
shows the retention being kept - about the retention plus a day - anddwarf_archive_maintain_failedcounts the
failed passes ofarchive_retention.dwarf_archive_lag_secondsanddwarf_archive_failing_secondsgain ajoblabel, so a shard reports several
series of each: aggregate with max for the shard's reading.dwarf_archive_lag_secondshas a series per fast archiving job (fast_archiving_1,fast_archiving_2; one on
SQLite): how long past the archive grace the oldest closed root of the job's share has waited. It now counts every
such root, one a failure left behind and a large tree being copied in stages included, and 0 once none is due.
It is read about once a second and goes on aging between readings. So one failed root shows in it until its sweep
takes it, up to about a minute, and a large tree's copy for as long as the copy takes.dwarf_archive_failing_secondshas a series per archiving job, each how long that job's archiving has been
failing: afast_archiving_kjob's until a later move archives something, or until every root of its share due
at its last failure has left the live tables; asweep_archiving_kjob's until a sweep completes failing nothing;
large_archiving's until a round over every copy in progress fails none. A tree whose archiving fails stays
live, never deleted for it before its retention (Fixed).
- Expired flows are deleted, not archived, and expiration runs every hour. A flow
SetFlowExpirationends is
terminated as before and, in the same write, marked for deletion; it is deleted within a few minutes, counted in
dwarf_flows_reaped{reason="deleted"}. It is never archived, soListClosednever lists it and it cannot be
forked. Until the deletion,SnapshotandAwaitreport its terminated outcome; the lists andHistoryleave it
out at once. The check runs on each shard every hour, or every half of the expiration when that is shorter, at least
a minute apart, where it ran every 5 minutes: a flow is terminated up to that long after it expires.
dwarf_flows_expire_failedcounts a root that fails again on every pass, up to an hour apart. - Every duration dwarf writes as JSON is an integer number of milliseconds, as the carrier's
sleepMillisand
backoff*Millisalready were. An integer is exact in every language, and a reducer (add,min,max) or a
whenexpression treats it as a number.SetDurationstores whole milliseconds, rounded to the nearest one, andGetDurationreads them; it panics on
anything but an integer. A duration stored in nanoseconds by an earlier version reads a million times too long:
a flow that is open across the upgrade and holds one should be finished first, or its field rewritten.Setwith atime.Duration, or one in a flow's initial state, still stores Go's encoding, nanoseconds, which
GetDurationdoes not read: useSetDuration, or stored.Milliseconds().FlowOptions.TimeBudgetis"timeBudgetMillis"in JSON (was"timeBudget", nanoseconds), a positive fraction
of a millisecond rounded up;ShardSummary.Latencyis"latencyMillis"(was"latency", nanoseconds), rounded
to the nearest millisecond. Both staytime.Durationin Go.
Engine.FlowsStartedandEngine.FlowsClosedare removed. They counted only one replica's flows - a flow
started on one replica is closed by whichever dispatches its last step - so their difference was not the number
of open flows, as their documentation said. Use thedwarf_flows_startedanddwarf_flows_closedmetrics,
summed over the fleet, orListOpen.- The correlation ID is
{shard}-{hour}-{flowID}, the flow key without its token (was{shard}-{flowID}), in the
workflow.idspan attribute and theflow/childFlowlog values. Find its flow withQuery.Searchset to the ID
followed by-, andQuery.Shardset; a dashboard or log query keyed on the old form needs updating.