chain certified fold skip

Chain-certified fold skip (NOETL_CHAIN_CERT)

Spec: noetl/ai-meta#366. Landed by noetl/server#483 (timestamp normalisation), noetl/ehdb#379 (the in-engine certificate) and noetl/server#484 (this).

Default OFF. Enabled on prod once, on 2026-10-01, and rolled back within the hour because it was inert — see Prod outcome below before planning anything on top of this.

⚠ Prod outcome, 2026-10-01: enabled, inert, rolled back

Rolled v3.122.0 by digest (flag off, baseline matched: median 15s both sides, 0 restarts), then enabled the flag. The reading after five executions:

observed=290  skipped=0  refolded=0  divergences=0  tracked=1

observed climbing proves the observer ran. skipped and refolded both zero proves no fold decision happened at all — rebuild_or_skip was never reached, so rebuild_state is never called on prod.

Why: prod runs NOETL_STATE_BUILDER=offserver, which routes the drive from the execute-time descriptor with zero noetl.event reads on the drive path. rebuild_state reads noetl.event and folds, so the drive bypasses it. The function has four real callers; none is on prod's drive path.

This is not the earlier "wired to a function with no callers" defect (noetl/server#485) — it is a function that is called, just not in this configuration.

The flag was rolled back the same hour. Nothing diverged, no refusals rose, no latency change: an inert feature is correctness-safe, which is exactly why it needed a metric to be visible at all.

tracked=1 across five executions is terminal eviction working — each completed execution is dropped on its playbook.completed.

Why there is no worker-side consumer either

Measured on the prod system pool (the pod that actually runs offserver): cold_rebuild is the cheapest outcome at 0.23 ms / 11 hops, everything a certificate could remove totals 16.4 s over 2d19h, and 87% of state-builder time is the incomplete outcome — a waiting problem a certificate cannot touch. ExecutionChain::advance_to also already short-circuits an unchanged chain via head_event_id, which is the same O(1) test more cheaply.

Decision and the conditions that would reopen it: noetl/ai-meta#367. Numbers: Off-server state builder cost.

If you want this live

The server-side consumer is merged, guarded and ready. It becomes useful the moment prod stops running offserver — no code change needed, just the flag.

What it does

The server is the producer for noetl.event, so it advances a per-execution rolling SHA-256 once per event as it writes. Deciding "is my folded state still valid?" then becomes a 32-byte compare instead of re-running WorkflowState::from_events.

Measured: validation is 0.33–0.38 ns/compare, flat from n=36 to n=10000 (O(1)); the per-event producer cost is ~0.2 µs.

⚠ The single-replica constraint

The skip is sound only while ONE process ingests the chain.

The roller is a change detector, not a chain identity. A second server replica would ingest events this process never sees, so this process's roller would stop advancing, read "nothing changed", and serve stale state.

Prod runs noetl-server-rust-embedded as a 1-replica StatefulSet (the noetl-server-rust Deployment is scaled to 0). That is what makes the feature sound today — it is a deployment fact, not an invariant the code can assume.

A replica increase must unset NOETL_CHAIN_CERT first.

chain_cert::divergences() is the backstop: reconcile compares the roller's length against what a real fold actually read, and on a mismatch it drops the execution's state and fails the skip closed globally — not just for the execution that revealed it, because a process that missed one event is not a reliable detector for any of them. It is a backstop, not a licence to scale.

⚠ Why a run of skips is bounded

MAX_CONSECUTIVE_SKIPS = 16.

reconcile only learns the chain's real length from a fold. So a process that has stopped seeing an execution's events has a roller that stops advancing, reads "nothing changed", skips forever — and therefore never folds and never reconciles. The miss would go undetected indefinitely while the skip served progressively staler state. Detection required the very fold the skip was removing.

Forcing a real fold every 16 skips bounds that blind window to roughly two minutes at the reconcile poller's 8 s cadence, while still eliminating 94% of folds — above the spec's ≥90% gate. Raising it widens the blind window; lowering it costs folds. Do not change it without re-deriving both numbers.

Why every INSERT site observes, not the "chokepoint"

event_write::emit_events describes itself as "the one chokepoint every authoritative event passes through". It is not. Two sites in handlers::events write noetl.event directly, in-transaction, on the branch should_publish takes when it returns false — the only branch a system-pool execution can take. That assumption already shipped as noetl/ai-meta#263: system/* playbooks mirrored 11 of 13 events on every hourly run.

So the observer is wired at all six INSERT sites, and certified_fold::every_event_insert_site_observes_the_chain counts INSERT sites against observe calls so a seventh cannot be added without one.

The hook sits at the INSERT, not at emit_events, which is what avoids noetl/ai-meta#332's double-count: a published event passes emit_events and then reaches the materializer, so an emit_events-level hook fires twice. Each event is INSERTed exactly once, by exactly one site.

Prerequisite: deterministic stored bytes

The digest is taken over what an event serialises to, so those bytes must be deterministic across producers. Event::serialize_timestamp_micros (noetl/server#483) truncates the timestamp to whole microseconds for exactly this reason — without it, two producers rendering the same instant at different sub-microsecond precision derive different digests (spec guardrail A8).

Truncation, not rounding, because that is already this platform's producer rule: ehdb_projection_fold::truncate_to_micros applies it to the fold's input after the same defect was measured on prod on 2026-08-31.

It fails closed

Only an exact (chain_len, digest) match permits reuse. Absent certificate, chain advanced, cached-ahead, same-length-different-digest, uncalibrated roller, revalidation due, and any prior divergence all fall back to a real fold. The same-length-different-digest case is the dangerous one: a length-only check would call it valid and serve state for a chain that diverged.

Metrics — read observed beside skipped

noetl_chain_cert_decisions_total{outcome} (9 outcomes), noetl_chain_cert_observed_total, noetl_chain_cert_divergences_total, noetl_chain_cert_tracked_executions. All pinned at zero by init_chain_cert_series.

observed	skipped	meaning
0	0	flag off, or no traffic
>0	0	flag on and inert — what happened on 2026-10-01
>0	>0	working

skipped alone cannot separate the first two. This pairing is the only reason the inert run was diagnosed in minutes instead of being recorded as a clean ramp.

Rollback

kubectl set env statefulset/noetl-server-rust-embedded -n noetl NOETL_CHAIN_CERT-

Roll back on any of: divergences() > 0, a new refusal or mismatch verdict in the cross-store comparator, hotel-cards p50 outside its ~10 s baseline band, or any pod restart.