One vocabulary 🐝🐘
Replicate an Apiary database into
PostgreSQL — backfill the history, then follow it. Standalone: its own repo,
its own release cadence, and no Go dependency on Apiary in either direction.
It reads the daemon's database; it is not a plugin and does not run inside
the daemon.
This release follows Apiary's unified task states. Apiary collapsed four
disjoint state vocabularies onto one canonical set — queued / running /
blocked / done / failed / canceled / skipped — and split the reason
out into blocked_reason and skipped_reason columns
(apiary#465).
The new columns need no work here: the SQLite reader reflects columns live.
What did need changing is the catalog's terminal-state lists, which are
data — and a stale one silently changes which rows get re-read every cycle.
dispatch_jobs learned that succeeded is now done, step_runs that
passed is done and skipped_cached is skipped plus a reason, and
workflow_instances gained canceled.
Every legacy spelling is kept alongside the canonical one. Apiary migrates
only terminal rows in bulk and lets live rows convert on their next
transition, so a database part-way through holds both vocabularies at once
and both have to settle. apiary_compat is unchanged: this release works
against every Apiary version the last one did.
blocked is deliberately not terminal, because a blocked row can still
move. An orphaned instance — now blocked + blocked_reason='interrupted',
where it used to have its own state — therefore stays in the open set until
the daemon's next reconcile rewrites it.
dispatcher_state is gone from the catalog, because Apiary dropped the
table. It was created in every database and never written, so nothing is
lost; an older Apiary that still has it reports a drift warning, not an
error.
pgsink doctor -c pgsink.yaml # check the catalog against your database
pgsink migrate -c pgsink.yaml # create the target tables
pgsink backfill -c pgsink.yaml # load history
pgsink sync -c pgsink.yaml # follow, with --metrics for PrometheusBackfill and sync are the same pipeline — only the starting watermark and the
stop condition differ — so the backfill path is exercised by every test the
follower has.
Before you run it
Same host, as the daemon's user. Apiary serves its event stream over a
Unix socket rather than TCP, and SQLite in WAL mode has no true read-only
reader: even a reader must be able to write the -shm wal-index file. A
locked-down user of its own fails at startup with unable to open database file, which reads like a path bug and is not one. See
Deployment.
The sink is an archive. Apiary prunes old logs and can delete tasks; a
cursor-based follower never observes a delete, so PostgreSQL retains rows
Apiary has dropped. For reporting that is a feature — but it is a stated
choice, not an accident.
Filtering an open_row table on its state column changes when rows
appear. The shipped example filters task_executions on status, so an
execution stays invisible until it settles and then arrives complete. Drop
the filter to watch runs in flight.
Notes
Two of Apiary's tables — task_executions and step_runs — are written
twice: inserted at dispatch with zero cost, updated at completion with the
tokens, cost and timings, and neither carries updated_at. Following them
correctly is most of what this tool does. See
the table catalog.
Timestamps are normalised to UTC before comparison. Apiary writes them with
a local offset, and rows predating its _time_format fix carry a Go
monotonic-clock suffix that its own date functions silently drop — those are
parsed here rather than discarded, so the target holds history the Apiary
dashboard does not show.
Changelog
- 5ba705a: feat(catalog): follow Apiary's canonical task states (#1) (@orlandoburli)
