reconcile give up

Reconcile give-up — the cap that bounds re-driving

The reconcile poller ticks every 8 s and re-drives every execution in orch_cache. An execution whose WAL chain can never complete — a truncated log, a permanently failed event write, log retention — is a no-op that gets re-driven forever, because the cache entry is evicted only on a terminal event. On 2026-09-01 that was 53 executions pinning the system pool at consumer lag 78–86 and taking a burst from p50 199 ms to 1997 ms, with 23 of 60 requests timing out.

NOETL_RECONCILE_MAX_NOOPS (default 225 ≈ 30 min) bounds it: after that many consecutive polls without progress the poller gives up, records a tombstone, and evicts the slot. Giving up is not destructive — it emits nothing, and any later real event resumes the execution.

What the guards exist to prevent

⚠ Do not delete these tests as redundant. Each one encodes a way this mechanism shipped inert, and three of them went to production before being caught.

guard	prevents
a_stuck_set_decays_to_zero_under_the_cap	the cap never being reached at all
decay_survives_orch_cache_eviction_on_every_poll	the budget living in the evictable slot, where six orch_cache.evict sites reset it
decay_survives_a_real_event_that_does_not_advance	clear() refilling the budget on event arrival
an_advancing_execution_is_never_tombstoned	a cap that swallows executions making progress
clear_still_drops_the_tombstone_while_preserving_the_budget	"preserve the budget" being satisfied by making clear() a no-op

Three rounds, two of them inert

The same defect recurred three times: the budget lived somewhere another code path reset it. Each round was necessary and the first two were not sufficient.

round	moved	still reset by	outcome
#461	the give-up record out of the evictable slot	the budget, still in the slot, via six orch_cache.evict sites	inert
#462	the budget out of the slot	event arrival — clear() dropped it	inert
#463	—	nothing	see below

dispatch_offserver_stateless_drive calls DriveTombstones::clear() as the else if condition, so it runs for every real event whether or not a tombstone exists. Measured 34 minutes after #462 reached prod:

reconcile_giveup{max_noops}       0        (no tombstone_* series at all)
drive{offserver_retry}            24.7/min (pre-deploy baseline 24.0/min)
drive{retrigger_recorded}         79.6/min across ~101 in-flight executions

That is one real event per execution every ~76 s against a budget needing 1800 s of quiet. Unreachable by construction.

A real event is not progress. Progress is an applied drive, which zeroes the budget through reconcile_decision(advanced = true). An execution that keeps receiving events and still never advances is the population the cap exists for. mark() still frees the budget — there it has done its job.

⚠⚠ The cap counts POLLS, not minutes — and the poller does not tick at 8 s

NOETL_RECONCILE_MAX_NOOPS is a count of poller iterations. "225 ≈ 30 minutes" is true only while the loop actually ticks every 8 s, and that is a copy of reality, not a property.

Measured on prod 2026-09-24 on v3.112.8, by two independent signals over the same 299 s window:

signal	ticks observed	expected at 8 s
noetl_orchestrate_in_flight_oldest_seconds changing	1	37
the per-tick held past the stale threshold warn	1	37

So the real interval was ~299 s, about 37x slower than designed, and 225 polls would take 18.7 hours rather than 30 minutes.

Why. The loop awaits every active execution's drive sequentially:

for execution_id in state.orch_cache.active_executions() {
    let advanced = trigger_orchestrator_inner(&state, execution_id, i64::MAX, true).await;
    ...
}

In the same window offserver_retry advanced by 27 — one tick's worth — giving ~11 s per drive. Drives are slow while the EHDB tier is unavailable (EHDB cross-store parity: could not read the tier … outcome="ehdb_unavailable", recurring every ~2 min). This is self-reinforcing: more stuck executions make the tick slower, which pushes the cap further out of reach.

⚠ So a give-up counter of 0 does not by itself mean the budget is broken. Check the tick rate first — measure whether in_flight_oldest_seconds tracks the wall clock. Three consecutive fixes (#461, #462, #463) were aimed at budget resets while this was also true.

How to tell whether the cap is working

Read the counter, not the code:

noetl_reconcile_giveup_total{reason="max_noops"}
noetl_orchestrate_drive_total{stage="tombstone_gaveup_skip"}
noetl_orchestrate_drive_total{stage="offserver_retry"}

⚠ Both tombstone series are label-scoped and therefore absent until they first fire — absence reads identically to a binary that predates them. Confirm the running version with noetl_server_build_info before concluding anything from a missing series.

A giveup of 0 while offserver_retry climbs steadily means the cap is inert.

Related

• deployment-specification — the env var row.
• noetl/ai-meta#315 — the umbrella.