docs: release v3.108.0 -- D3 serve-on-behind shipped inert Records the merge of server#423 (06fcfe70), tag v3.108.0, AR digest 867b822f, and that prod was NOT deployed (still v3.107.1, flag absent from its env). Documents the bounded re-fold as the safety argument for serving a behind snapshot, and that an EXISTING guard caught the first draft calling fold_spine_inner directly -- a real ai-meta#307 bug, since the spine index evicts completed executions and the verification would have refused for exactly those. ⚠ Records the binary-level flag check WITH its control: RECOVERY_SOURCE appears in both builds, which is what makes SERVE_ON_BEHIND=0 on v3.107.1 meaningful. Serve-under-lag remains unproven and gates the prod flip. Refs noetl/ai-meta#332
docs: recovery serving cutover to the tier (RECOVERY_SOURCE verify -> tier) Records the 2026-09-09 prod cutover: pre-flight 42/42 tier-folds vs spine refusals with fold_failed=0 across 14 present-and-pinned series and 0 of 21 control series unexpected; the revert rehearsed in kind BEFORE the forward flip (which also establishes that the mode is confirmed by reading a metric rather than assumed); a full-spec server-side diff showing exactly one spec difference. ⚠ Also records what the cutover does NOT show: served_tier=0 over a ZERO denominator, because recovery only fires when the spine refuses and a fresh pod with healthy in-flight executions never refuses. Those zeros mean not-yet- exercised, not working -- served_tier is only readable alongside a non-zero spine_incomplete. Refs noetl/ai-meta#332, noetl/ai-meta#307, noetl/ai-meta#265
docs: the projection tier already has a working read path Read-only survey 2026-09-09. ehdb_projection_read.rs ships a four-mode ladder (postgres/verify/tier/wal) with enforced controls and prod instrumentation. ⚠ The module doc lists only three modes; prod runs 'wal', which reads as an unrecognised value against the doc but is a real fourth variant in the parser. The doc is stale, not the config. ⚠⚠ Wal mode does not consult Postgres at all -- which is the mechanism behind the 2026-09-09 no-op storm: 'WAL chain incomplete; returning no-op' was this path correctly refusing to serve a state that might be wrong. served_tier=0 / spine_refused=12 is the #307 lifetime mismatch (in-memory spine holds in-flight executions, comparator asks about completed ones) -- coverage ~0 by construction, not a defect. The dark launch already settles whether the tier CAN serve: folded{source=tier}=24 where spine_incomplete{source=spine}=24. Projections are already served via three /api/ehdb/projection-* endpoints, with content-level digest agreement and a built-in negative control. Refs noetl/ai-meta#332, noetl/ai-meta#265, noetl/ai-meta#307
docs: N>1 routing, fail-closed, and PV recovery validated in kind Records the 2026-09-09 off-prod validation against 3.106.1: a 2-shard StatefulSet where the pods independently agree on ownership (each one's forwarded_ok equals the other's forwarded_terminus), fail-closed confirmed three ways (4x503 / metric forward_unavailable=4 / the FAILING CLOSED log), and a volumeClaimTemplate proving the engine RECOVERS on pre-existing data after the pod is deleted -- not merely that the bytes persist. ⚠ Also records why a volume cannot simply be added to the prod Deployment: RWO deadlocks every rollout under maxSurge:1, and RWX removes the deadlock precisely by putting two single-writer engines on one directory. Staged inert as ops#303. Refs noetl/ai-meta#332, noetl/server#419
docs: embedded shadow is reachable and agreeing on prod (v3.106.1) The first hourly execution after the roll gave opened=1, agreed=12, diverged=0, append_failed=0, with du flat at 44KB (0% of the 1GiB quota), 0 restarts and 0 ERROR. publish_skipped{system_execution}=22 in the same window -- the traffic the old call site could not see -- so the fix is demonstrated, not asserted. The engine dir now exists and carries substrate/FORMAT_VERSION, confirming the format-version gate ran and passed at open. ⚠ Corrects this page's own storage prediction: du read 44KB immediately rather than ~0 for days. 44KB is the engine's fixed structure written at open; PART data is what waits for should_seal. Healthy signature is nonzero-but-flat, and growth appears as a step every ~2 days rather than a ramp. Refs noetl/ai-meta#332
docs: hook moved to the emit_events chokepoint (server v3.106.1) Records noetl/server#420: shadow_append moves from the materializer to emit_events, beside mirror_rows -- post-dedup/chain-stamp so rows IS the authoritative set, pre-fork so one call site covers both sides of the gate. Removed from project_events rather than duplicated, because with the gate on both would fire for the same event and double-count it. Documents why the guards read source (emit_events cannot be unit-driven, and a direct call to shadow_append passes whether or not the write path calls it), the four mutations each was proven to catch, and the guard that first counted 6 call sites by matching its own assertion strings. ⚠ Adds how to read the storage gate: should_seal fires at 1024 records / 8MiB, so du reads ~0 for DAYS while agreed climbs -- an empty dir is not evidence the shadow is idle. And df is the wrong instrument: it shows the node overlay (83GB free), not the pod's 1GiB quota. Refs noetl/ai-meta#332
docs: correct the all-zero shadow reading -- no user traffic, not a wrong hook Supersedes the previous section, which gave two wrong mechanisms in turn: 'the CQRS gate is off' (it is ON -- publish_skipped{gate_off}=0) and then 'the hook is on the wrong writer' (it is on the right one). noetl_event_ingest_publish_skipped_total{reason=system_execution}=22 with gate_off=0 and no_transport=0 settles it: should_publish excludes system executions by design, prod's hourly job is system/orchestrate, so those events take the synchronous INSERT and never reach the materializer -- where shadow_append lives. The shadow fires for USER executions; the window had none. Adds an explicit warning against 'fixing' this by moving the hook to emit_events, which would capture system traffic the CQRS design deliberately keeps off the materializer. Refs noetl/ai-meta#332
docs: the armed prod shadow sits on the gate-on writer, which prod does not use Within an hour of arming, one hourly execution persisted 23 events through handlers::events -> event_write::emit_events while events_projected_total stayed 0 and every embedded-shadow series stayed 0. shadow_append is called only from services::internal::project_events, which this module's own header identifies as the materializer -- the writer that is live only when the CQRS gate is ON. Prod runs the gate off, so the shadow is structurally unreachable for the ordinary event flow. Kind measured agreed=24 because kind exercises project_events. A config difference, not a code one -- the ai-meta#326 lesson repeating. ⚠ The failure mode is that an unreachable shadow reports agreed=0 diverged=0, which is exactly what a healthy quiet shadow reports. Records the denominator so no agreement claim rests on an empty population. Refs noetl/ai-meta#332
docs: embedded EHDB shadow armed on prod; the /data default does not fail closed Records the 2026-09-08 arming of NOETL_EHDB_EMBEDDED=true on shastaratech-noetl-prod (server v3.106.0) — an env-only change whose full-spec pre-apply diff was two lines, verified post-apply against a whole-object before-snapshot. Adds the two env vars to deployment-specification.md, which the original change set omitted (wiki-maintenance Rule 2a). ⚠ Corrects a false claim in the code's own doc comment: DEFAULT_EMBEDDED_DIR says /data 'fails loudly at open' with no volume mounted. It does not — LocalFsSubstrate::new calls create_dir_all, so a missing volume is silently substituted by the container's ephemeral layer, which on prod carries a 1Gi limit whose breach EVICTS the pod. Safe at the measured ~500 events/day with manifest retain capped at 32, but safe by rate rather than by guard. Refs noetl/ai-meta#332
Embedded EHDB shadow (v3.106.0): default-off, serves nothing, and what not to break Records the flag and its deliberate strictness, the pinned metric label set and why skipped != agreed, and the three invariants: errors swallowed on purpose because a shadow that can fail a real write is a liability; rows serialised with the same to_stream_json the publish path uses; a poisoned lock recovered rather than unwrapped. Plus the git-tag dependency and the two-line prod diff.