event log retention archival

Event-log retention and object-store archival

A completed execution older than a configurable window is copied to an object store, indexed for discovery by date, and only then removed from the hot EHDB volume. The hot volume becomes a window rather than an unbounded accumulator.

Program: noetl/ai-meta#459 · closes #457 · spec noetl/ehdb design/retention-archival-tier.md

Source: src/services/event_archive.rs, src/services/archive_store.rs, src/handlers/archive_read.rs.

⚠⚠ What this prunes, and what it does not

The store this prunes — /data/ehdb-embedded, 2.4 GiB on a 9.7 GiB PVC — is a shadow. Verified 2026-10-09 with a control grep: the only readers of engine() are the shadow writer itself and the parity verifier, and GET /api/executions/{id} serves events from Postgres noetl.event, which is append-only and never purged.

So pruning it cannot make API history unreadable — the API never read it. The archive is the only place EHDB-format history survives a prune, which is what the read routes are for.

⚠ Not the same store as the serving tier. NOETL_EHDB_EVENTLOG=primary on the worker pools points at noetl-cmdbus-writer-0:9110 — a different process, a different volume, and that one is primary-serving. Pruning that would need the fall-through helper wired into its read path.

Environment variables

Catalogued on deployment-specification; summarised here:

variable	default	meaning
NOETL_EHDB_RETENTION_HOURS	48	a completed execution older than this is archivable; 24 supported
NOETL_EHDB_ARCHIVE_ENABLED	false	run the archive pass — additive, deletes nothing
NOETL_EHDB_PRUNE_ENABLED	false	allow the floor to advance and parts to drop — deletes hot data
NOETL_EHDB_ARCHIVE_BUCKET	unset	archive target; unset ⇒ archiving is off regardless of the flag
NOETL_EHDB_ARCHIVE_ENDPOINT	falls back to NOETL_OBJECT_STORE_GCS_ENDPOINT	GCS or an emulator
NOETL_EHDB_ARCHIVE_INTERVAL_SECS	900	pass cadence
NOETL_EHDB_ARCHIVE_MAX_PER_PASS	200	work bound per pass

⭐⭐ Archive and prune are two independent flags, deliberately. Archiving is additive; pruning destroys the only other copy. There has to be a reachable state where the archive is provably complete and nothing has been deleted — a single flag makes "verify before deleting" impossible to express. Pruning with archiving off is refused, not obeyed.

⚠ A 0 window is refused, not clamped, and an unparseable flag is an error, not a silent false: a typo in a data-deleting flag must not look like "off".

Layout

gs://<bucket>/execution_id=<id>/events-<min>-<max>.jsonl   zero-padded: lexicographic = sequence order
gs://<bucket>/execution_id=<id>/manifest.json              digest, count, seq range, status, archived_at
gs://<bucket>/date=YYYY-MM-DD/execution_id=<id>.json       the execution's START date

⭐⭐ One object per execution under date=, not one appended per-day manifest. GCS has no append, so a per-day object would be read-modify-write and two concurrent archives would lose an entry with nothing reporting it. One object per execution makes each write independent and idempotent, and discovery becomes list(prefix). A rolled-up daily index may be derived later; it must never be the write path.

The constraint that shapes it

Dataset::partition for D1 is shard_for_execution(execution_id, shard_count) with shard_count = 1 in prod, and index_key is the execution id. A part therefore holds many executions interleaved by append sequence. Consequences:

• an execution owns no part → archival is an extraction, not a file move;
• PartMeta.execution_bloom answers "might X be here" and cannot enumerate;
• ⚠⚠ dropping an old part can destroy the early events of a still-running execution.

Resolution: archive per execution (exact), prune per part (coarse), with a floor as the bridge — min(global_sequence) over every execution not yet safe to prune. A part entirely below the floor holds only archived executions. ehdb_l0::plan_retention never splits a part; one straddling the floor is kept whole.

Order is the guarantee

upload → read-back verify (digest + record count) → record durably → then prune. A crash anywhere leaves the hot copy intact, because the prune's floor derives only from durably recorded, verified archives. There is no window in which the only copy is in flight.

The prune gate (archive_state) requires all three: a verified data object, a matching record count, and the date reference. An execution archived but unindexed is data that exists and cannot be found, so it blocks the prune.

⚠⚠⚠ The blocker that froze the drain AND made prune impossible

Found on prod 2026-10-09, from a log line, after every published gauge had read healthy for hours:

archive pass complete  examined=6511 archivable=6259
                       archived=0  already=3680  out_of_coverage=100  failed=0

out_of_coverage=100 is exactly max_per_pass. The budget is decremented before the empty-records check:

budget -= 1;
let Some(records) = read_records(*id) else { ... };
if records.is_empty() { rep.out_of_coverage += 1; continue; }   // budget already spent

…and the answer was not retained. So every pass spent its entire budget re-deciding the same unarchivable executions, archived sat at 0, and everything further down the list was never reached. It is server#514's shape by a different route: work repeated because its result was not retained.

⚠⚠ And it was worse than a stall. An out-of-coverage execution is archivable by age and never becomes verified_archived, so compute_prune_floor counted it as backlog and returned AwaitingBacklog permanently. Prune could never have run, on any timescale.

Fixed in server#522:

1. known_out_of_coverage persists across passes, so a remembered id is skipped for free. Monotonic by construction — the engine only holds records appended since it opened, so an execution with none now will never acquire any.
2. The floor's satisfied set is archived ∪ out-of-coverage. Safe, and this is the load-bearing half: the floor exists to stop a part being dropped while it still holds an unarchived execution's records, and an execution the engine has no records for is in no part. Nothing to copy, nothing to protect.
3. out_of_coverage is now a pinned metric outcome (10 → 11). It had been a PassReport field reported only in a log line — which is exactly why the block was invisible on /metrics. A count that can halt the system does not belong only in prose.

⚠⚠ The denominator is not archivable

The honest completion criterion is:

already_archived + out_of_coverage  >=  archivable

Measured on prod: already=3686 + out_of_coverage=2578 = 6264 = archivable. 41% of "archivable" executions had no records in this store at all — they predate the volume, so there is nothing to copy. Reporting progress as "X of 6,264" overstates the remaining work, and waiting for archived to reach archivable waits forever.

Nothing is lost by it: those executions' events live in Postgres noetl.event, which is append-only and never purged. They simply do not need archiving from a store that never held them.

⚠ Restarts reset the memory

Both known_archived and known_out_of_coverage are in-memory. A pod restart forces re-establishment: one pass of archive_state GETs for the archived set (free of budget, but ~77 ms each, so minutes), then max_per_pass out-of-coverage per pass until re-enumerated. Batch configuration changes rather than making several small ones.

⚠⚠ The hazard to watch: the pinned floor

One stuck non-terminal execution blocks all reclamation while every other counter reads healthy — archiving keeps succeeding, verification keeps passing, and zero bytes come back. This is the drift hazard of the whole design.

noetl_ehdb_retention_oldest_non_archivable_age_seconds   0 = nothing blocking
noetl_ehdb_retention_floor_blocked_by_execution          the id (0 = none)
noetl_ehdb_retention_floor_sequence
noetl_ehdb_archive_scan_examined / _archivable           the selection WITH its population
noetl_ehdb_prune_bytes_reclaimed_total
noetl_ehdb_archive_total{outcome}                        10 pinned outcomes

⭐ The alert that matters pairs two series: archiving succeeding while reclaimed bytes stay 0. It is invisible in either series alone.

All pinned at 0 unconditionally at startup — a pin inside an if archive_enabled branch is not a pin, it leaves the series absent on exactly the configuration whose value someone would be reading (the server#315 mistake). Both floor gauges are reset when the block clears, because a stale id names an execution that is no longer blocking anything.

The floor, and how it reaches apply_retention

The prune path is prune_pass (src/services/event_archive.rs), deliberately separate from archive_pass: archival behaves identically whether pruning is on, and the floor refuses by itself. It is the only code in the server that deletes from the event log.

prune_pass(cfg, &CandidateSet, now, verified_archived, footprint_fn, prune_fn)
   └─ floor_relevant_ids  ->  compute_prune_floor  ->  ehdb_embedded::prune_below
                                                          └─ L0Engine::apply_retention

The footprint comes from the manifest, not from reading records. The obvious implementation reads each constraining execution's records and takes min/max global_sequence; at prod scale that is tens of MB read under the append lock, every pass, on the one process hosting both buses. PartMeta already carries the part's sequence range and a bloom over its execution ids, so hot_footprint tests the bloom instead and holds the lock only long enough to clone the manifest.

⚠ The bloom's error direction is what makes this sound. A false positive attributes a part to an execution not in it, which lowers that execution's min_sequence, lowers the floor, and prunes less. There is no false negative. A part with no bloom at all is treated as possibly holding everything, for the same reason. Records in the unsealed tail are not in the manifest, which is also fine: the retention plan only drops parts the manifest lists.

Three refusals, each guarding a different way to delete live data

refusal	what it prevents
AwaitingBacklog	any archivable execution not yet verified-archived — a part may hold it
CandidateSet.complete == false	a floor computed from a partial population. collect_candidates caps at 200 pages x 100 and used to return Ok on a truncated list; an execution the listing never returned is invisible to the floor, so its records can lie below it
absent footprint (None)	the engine not being open. Absence of evidence is not evidence of absence — it must never read as "nothing constrains"

MAX_FOOTPRINT_IDS (20,000) refuses rather than truncating, because a truncated footprint places the floor above live data. There is no safe partial footprint.

⚠⚠ The all-archived case is not NoFootprint

An empty footprint and a footprint whose every member is archived are opposites, and conflating them cost the whole feature its purpose:

• empty ⇒ we measured nothing ⇒ refuse;
• non-empty, all archived ⇒ we measured, and all of it is safe ⇒ the floor advances past the highest sequence those executions occupy.

The first implementation returned NoFootprint for both. That is exactly the steady state an idle log reaches once the backlog drains, so the entire prune path would have reclaimed zero while every call returned Ok. It is also why HotExecution::max_sequence exists — added for this case, then never read.

The floor must clear the highest sequence, not the highest first sequence; the latter is far below the log tip and reclaims nothing. Both errors return Ok, which is why the tests assert on surviving records and reclaimed bytes, never on a return code.

Guards: tests/prune_wiring.rs (8 tests, 5/5 mutation-caught) and tests/prune_floor.rs. The prune callback in the former models a real part layout and the engine's actual droppability rule (part.max_sequence < keep_from_sequence).

⚠⚠ Instrumenting the object-store put

Two mistakes were made here and both are worth not repeating.

Time the primitive, not a caller. The put histogram was added to ObjectBackend::put, the wrapper. The archive tier's ArchiveStore impl calls GcsBackend::put directly, so ~300 real objects reached prod GCS while the metric read its seed. The timing now lives inside GcsBackend::put, which makes the bypass unexpressible — there is nowhere else to put the bytes. The Postgres arm keeps its timing in the wrapper (it has no primitive here), so the wrapper's GCS arm must stay a bare delegation or every GCS put is counted twice. tests/object_put_instrumented.rs guards all three facts.

Never pin a histogram by observing. The pin used observe(0.0) per label. inc_by(0) on a counter is harmless; observe(0.0) on a histogram records a measurement that was never taken — _count starts at 1 so "has it fired?" is unanswerable, and the distribution is biased toward zero, worst when n is small. In the first prod reading the only sample below 50 ms was that seed. Absent-vs-zero is now carried by noetl_object_store_put_total{backend,outcome}, a counter, which can be pinned truthfully; the histogram is left to be pruned until it fires, with noetl_server_build_info discriminating "this version has no such metric".

Measured in prod, n=300, same-region: p50 77 ms, p90 98 ms, p95 139 ms, p99 234 ms. Roughly 19x a ~4 ms local fsync — the figure that settles whether a remote put can sit on the append path. It cannot.

Reading archived history

GET /api/archive/executions/{execution_id}
GET /api/archive/executions?date=YYYY-MM-DD[&limit=N]

Both are gated (the by-date route enumerates, which is a capability distinct from reporting on an id the caller already holds) and read-only by construction — the ArchiveStore trait has no delete, so once the hot part is pruned the layer that wrote the archive cannot remove it.

ReadSource is three states, never two: hot, archive, not_found.

• ⚠⚠ An empty hot result is a MISS that falls through, never an answer. After a prune the hot store legitimately holds zero records, and treating that as "no events" would turn every pruned execution into a silent empty success.
• ⚠⚠ A broken archive is an error, never a not_found. A present manifest means the execution was archived, so a missing or corrupt data object is data loss — reporting it as absence would hide that behind a clean 404.
• 503 with a reason when the archive is unconfigured. "Not configured" and "nothing archived" are different facts and must not share a response.
• A malformed date is refused, not turned into a prefix that matches nothing and returns a clean empty — a quiet day that never existed.

Every listing carries its denominator: keys_seen, unrecognised, truncated, limit_applied.

Rolling it out

1. IAM — the server SA needs roles/storage.objectAdmin on the archive bucket. It has no binding there and no project-level roles; archiving fails closed until granted.
2. Deploy.
3. NOETL_EHDB_ARCHIVE_BUCKET + NOETL_EHDB_ARCHIVE_ENABLED=true — archive-only, deletes nothing. Leave it until the output is checked.
4. Wait for the backlog to drain. ⚠⚠ Flipping NOETL_EHDB_PRUNE_ENABLED=true before that reclaims nothing, correctly: the floor returns AwaitingBacklog while any archivable execution is neither verified-archived nor out-of-coverage. ⚠⚠⚠ Do NOT wait for archived to reach archivable — that never happens. The criterion is already_archived + out_of_coverage >= archivable; on prod 41% of the archivable set was out of coverage. Watch both outcomes, and note the pass is bounded by MAX_PER_PASS per INTERVAL_SECS. An operator who flips the flag early and reads zero bytes will reasonably conclude the feature is broken.
5. Only then NOETL_EHDB_PRUNE_ENABLED=true, and measure the volume before and after. noetl_ehdb_retention_floor_sequence becoming non-zero is the signal the floor advanced; reclaimed bytes are reported per pass in the retention pruned parts below the proven floor log line.

Testing it locally

The object client already supports an open emulator (GcsAuth::None for any non-Google endpoint), so the whole tier runs against fake-gcs-server:

podman run -d --name fake-gcs -p 4443:4443 docker.io/fsouza/fake-gcs-server:latest \
  -scheme http -port 4443 -external-url http://127.0.0.1:4443 -public-host 127.0.0.1:4443
curl -X POST localhost:4443/storage/v1/b -H 'Content-Type: application/json' -d '{"name":"ehdb-archive-test"}'

NOETL_TEST_GCS_ENDPOINT=http://127.0.0.1:4443 NOETL_TEST_GCS_BUCKET=ehdb-archive-test \
  cargo test --test archive_execution --test archive_date_index --test archive_retrieval \
  emulator -- --ignored --nocapture

⚠ GcsBackend::open_unauthenticated refuses a googleapis.com host: an unauthenticated client against real GCS fails every request, and the first guess would be the credential rather than the constructor.

Related

• deployment-specification — env-var source of truth.
• runtime-registry — the sibling #455 work.
• ehdb#345 — a full volume is silent; why this matters.