execution chain store

The execution-partitioned chain store on the read path

How noetl-server decides whether an execution can advance, when the chain comes from the ehdb-l0 chain store instead of a Postgres WHERE execution_id read.

Env vars and their arming state live on deployment-specification — this page is the mechanism and, more importantly, the guards and what each one exists to prevent.

Source: src/handlers/chain_populate.rs, src/db/queries/event_chain.rs, src/chain_advance.rs, src/state.rs.

⚠⚠⚠ Status: NOT ARMED — rolled back 2026-09-30

⚠⚠⚠ ROLLED BACK 2026-09-30 00:10Z — the gates are NOT armed. The 772/0 comparator reading was real but the window was too short; an hour later the same deployment read 13 divergences over 839 engagements (98.45%) and all three vars were removed (env 67 → 64). The cause is a THIRD mechanism: an event commits into the MIDDLE of an ORDER BY event_id read, because a snowflake event_id is minted before insert so commit order is not id order (noetl/ai-meta#362). Proven by position arithmetic — the store held at position 72 what the log holds at 73, one missing INTERIOR row. server v3.117.2 remains deployed and inert.

What this page describes is still what the code does; only the "armed in prod" state is withdrawn. The guards table below stands, and so does the StaleLog section — but note that neither of those mechanisms is what failed prod. A third one did, and it is not yet fixed.

⚠ The lesson worth carrying off this page: a comparator denominator of 772 with 0 failures was a real measurement over a window of minutes, against a mechanism with an inter-arrival of about an hour. Volume is not duration. Size the window to the phenomenon, not to convenience.

The shape

LogSourcedChainSource::chain_for answers one question — what is this execution's chain? — in four steps:

1. Read the authoritative log. read_chain selects the execution's events from noetl.event ordered by event_id, with a row limit.
2. Populate the store from that read. populate_from_log recomputes the chain edges from log order. The log's own prev_event_id is never an input — it is NULL on the overwhelming majority of rows.
3. Answer from the STORE, not from the rows just read. This is deliberate: the rows are already in hand, so returning them would leave the store's content untested and the eventual repoint unjustified. Answering from the store means every decision the source makes is a decision the store was capable of.
4. Zip in terminal, which is a property of the event type and so lives in the log, not the store.

Guards, and what each one prevents

guard	prevents
rows.len() >= limit ⇒ refuse (log_truncated)	Populating from a truncated page and reporting the partition complete — rebuilding the truncation defect on the other end. A full page is not a complete log.
A failed read returns None, never an empty view	An empty view says "this execution has no events", so a running execution would read as finished.
chain_if_authoritative — coverage and watermark must agree	Serving a partition populated from the middle as though it were populated from the beginning.
Coverage is monotonic	A stale snapshot recording a smaller total makes a healthy partition unservable. Without this, fixing the divergence misclassification only converts a false divergence into a false refusal.
NOETL_CHAIN_SOURCE=chain resolves to OFF without NOETL_CHAIN_POPULATE	A source reading an unpopulated store makes decide() report Empty on a running execution. Three gates, not two.
Unrecognised NOETL_CHAIN_SOURCE ⇒ off	A typo must never move the execution-advance path.
ChainHeads hydrates from the DB on first touch, Err ⇒ Failed (never Empty)	After a restart the in-memory head map is empty; treating that as "no head" truncates a live chain. Failed and Empty must stay distinguishable.

⚠⚠ StaleLog: why a behind snapshot is not a conflict, and what that cost

chain_for reads the log and then takes the store lock — it cannot hold a std::sync::Mutex across the database await. Under concurrent reads of one execution, the task that read first can reach the lock second and find the store already ahead of its snapshot.

That was classified as Diverged, and it failed a prod ramp at 5 divergences in 79 (6.3%) with nothing actually wrong. It is FromLog::StaleLog now: the extra events are real, committed, and the fresher answer.

⚠ But "the store holds more than my snapshot" is also exactly how a SECOND WRITER looks — a different populator putting events in the store that are not in the log at all. From one snapshot the two are indistinguishable, and absorbing the second would be far worse than the bug being fixed.

So the verdict lives where a second read is possible:

StaleLog { stored_len, log_len }
  └─ re-read the log
       ├─ log caught up (len >= stored_len) ─→ benign; populate again and answer
       └─ log still short                   ─→ those events are NOT in the
                                               authoritative log  ⇒  `diverged`,
                                               refuse, fall through to Postgres

ehdb-l0 reports StaleLog as unresolved rather than deciding, because the crate cannot re-read. That split is the point.

⚠ Populate and the guarded read are ONE lock hold

Releasing the lock between them let a concurrent reader advance the store, so the chain came back longer than the terminal vector built from our log rows — 2 length_disagreement refusals in 800 concurrent reads. Safe (it declines rather than answering wrongly), but it is the same false alarm one step downstream.

⭐ A side effect worth knowing: the longer critical section makes readers queue in read order, so the reordering that creates a behind snapshot largely stops happening at all.

Reading the metrics

noetl_chain_populate_total{outcome} — 13 labels, all pinned at 0 unconditionally, because an unpinned label is an absent series and reads exactly like a binary that has no such outcome.

• Engagement is in_sync + extended, not in_sync alone. Both are FromLog::InSync, split only on whether anything was appended, so a healthy system can read in_sync=0 while extended climbs. Watching in_sync alone reports a false failure.
• Failure is diverged plus every refusal reason. Count them by reason — a bare "refusals > 0" cannot tell a transient DB error from a residual false alarm, and that distinction is what diagnosis turns on.
• stale_log is benign and expected under concurrency; diverged is not.
• opened is the only series separating never ran from ran and rejected everything, because the store opens lazily on the first populate.
• stale_log / stale_log_unresolved exist only from v3.117.2, so their presence is the cheapest proof of which binary is running — more direct than the image tag, which is a different representation and can disagree.

Rollback

Remove the three env vars. Measured at ~17s, no migration, and the store is derived state that populate_from_log rebuilds from noetl.event.

⚠ Prefer this over rolling the image: v3.117.1 predates the #360 fix, so rolling back to it restores the false-divergence classification.

Related

• deployment-specification — the env vars, defaults and arming state.
• Embedded-EHDB-Shadow — the neighbouring shadow slice and the /data mount trap.
• noetl/ai-meta#357 — the repoint umbrella.
• noetl/ai-meta#358 — two writers on one partition.
• noetl/ai-meta#360 — the false divergence and this fix.