wiki(#155): the async event-log mirror, and correct the stale "primary not activated" claims Adds Runbook-Async-Event-Log-Mirror — the operational surface for the mirror that went asynchronous on prod 2026-08-19, and which this wiki did not mention at all. Leads with the one-flag rollback and with the pairing that makes it dangerous: async on with the tolerance at 0 makes the comparator judge a healthy tier on its own liveness. Records the parts that are easy to get wrong rather than only the settings. That the window must be MEASURED and then set above the observed p99 — prod came back p99 699ms, so the window is 30s. That it sits deliberately below SETTLE_SECS so the background sampler still compares every execution. That the two lag controls are a discrimination rather than a clean-parity check, because a comparator taught to ignore everything passes the naive version perfectly. And that reserve rather than send is load-bearing, because the first implementation lost 32 of 60 records while every counter still read correctly. Corrects two current-state claims that had gone stale: * Home said the reference tiers are O-of-n per op and "not primary-serve". Still true for KV/object/vector; false for the event log since the runtime cache landed and since the tier went primary on 2026-08-13. * Roadmap said the primary cutover "stays a later step" and that primary is "recognised but not activated". Both annotated as superseded rather than deleted, since they remain accurate as the plan of record. Sessions-Log is left untouched — it is history and correct as written. Home gains a dated current-state banner so a reader meets the live state before the older planning prose. Also records that Option 2's batch flag is still OFF on prod, batch=0 on both replicas, so the substrate is landed and measurable but inert. Tracked at noetl/ai-meta#284.
docs(perf): add Design-Performance-and-Load-Testing + baseline; Sessions-Log, Home, Roadmap, _Sidebar
docs(runtime): KV + object tier mirrors now wired into the live worker path (v5.68.0) Home + Roadmap (Phase 8) + Sessions-Log: event-log + KV + object tiers' runtime mirrors wired live (worker#168, v5.68.0, 2ec2e2b); projection + vector deferred with documented seams. Live-drive proof pending redeploy. Refs ehdb#234, ehdb#241.
docs(consolidation): EHDB program to one coherent current state (Roadmap+Home+Sessions) Phase 6: shadow mirror wired into the live emit path + proven on real in-kind drives (worker v5.67.0, worker#167 d310c7b, execs 332760742153424896 + 332760854506246144); eventlog tier only so far. Home gains a Live-in-kind bullet (runtime mirror + query interface end-to-end via server v3.53.0 + durable-backend slice 1 ehdb#253) and an explicit What's-NOT-done-yet bullet (prod cutovers gated, durable later slices, other-tier runtime mirrors, raw-tier query seam). Docs-only; prod unchanged (worker v5.52.0/server v3.52.0, all NOETL_EHDB_* flags off).
docs(design): durable event-log backend — first slice (segment store + crash recovery) New design note Design-Durable-EventLog-Backend.md: the production disk format (segmented append files + CRC framing + offset index + fsync + crash-recovery replay) the Phase-6 note deferred and the prod-cutover runbook §C durability gate names as the hard blocker for Stage C. Covers the on-disk format, durability guarantees, semantics parity, the single-writer-per-shard plan (execution-affinity, later slice), the Phase-10 backend selector, the PVC-vs-object-store recommendation + flagged deploy assumptions, and the remaining-slices checklist. Linked from _Sidebar (Design notes) + Roadmap (Phase 6 → Phase 9 prereq subsection). Sessions-Log entry for the slice-1 landing. Tracks noetl/ehdb#254 · noetl/ehdb#241 · slice-1 PR noetl/ehdb#253
docs(runbook): Phase 9 tier-1 event-log PROD cutover runbook (draft — awaiting user go) Staged A→B→C plan (deploy v5.66.0 flags-off → shadow → primary), prod deploy mechanism grounded in the ops CQRS-flip runbook (GKE ctx, kubectl set env, GMP monitoring), tier-1 rollback levers, and the local_reference pod-local-JSONL durability blocker for Stage C. Nothing executed. Linked from Roadmap Phase 9 tier-1 row + tier-1 section + _Sidebar Runbooks. Refs noetl/ehdb#241
docs(phase10): Backend Configuration reference + Roadmap/Home/Sessions IMPLEMENTED Phase 10 (tunable-backend config surface) landed → EHDB program (Phases 6-10) code-complete. New Backend-Configuration page (per-tier selection, env vars, precedence, coherent/incoherent examples, ehdb-selfcheck config matrix, EHDB-default-not-lock-in). Roadmap Phase 10 IMPLEMENTED + program status banner; _Sidebar link; Home + Sessions-Log entries.
docs(roadmap): Phase 9 tiers 3-5 in-kind dual-run VALIDATED → Phase 9 CODE-COMPLETE Combined tier 3-5 in-cluster dual-run passed on kind-noetl (worker v5.65.0 image v5.65.0-p9 7d30b250d261, Job ehdb-p9-t345 ns ehdb-p9-validate, node noetl-control-plane, container kernel 6.19.7-200.fc43.aarch64 — not macOS host; pod Succeeded 0 restarts). Tier 3 KV served_primary + dual_run_holds (5 parities) + reversible + guard_refused; tier 4 object served_primary + digest-integrity dual-run (4 parities) + reversible + guard_refused; tier 5 vector served_primary + top-k parity (ids/order/monotonic) + bounded cap + reversible + guard_refused; all secret-free, off=disabled no-op. Tiers 1-2 already validated 2026-07-05, so Phase 9 is now CODE-COMPLETE (all 5 tiers implemented + in-kind validated). prod/GKE cutover STILL GATED on user; nothing in prod changed.
docs(roadmap): Phase 9 tier-5 vector primary-serve IMPLEMENTED + MERGED (all 5 tiers done)
docs(roadmap): Phase 9 tier-4 object/blob primary-serve IMPLEMENTED + MERGED Tier-4 (object/blob) primary cutover built + merged (ehdb#250 bb42b8d + worker#164 a100adf → v5.64.0 369e4c1), activated behind NOETL_EHDB_OBJECT= primary, reversible, dual-run-verified. LOCAL only; in-kind dual-run BATCHED (combined tier 3-5); prod/GKE cutover stays gated on the user. Roadmap table row + tier-4 section + rollback procedure + Sessions-Log entry. Refs noetl/ai-meta#234
docs(roadmap): Phase 9 tier-3 KV/state primary-serve IMPLEMENTED + MERGED ehdb#249 (73b1446) + worker#163 (ba9f829, v5.63.0 a7925e0). Tier-3 table row + Tier-3 section (ehdb/worker deliverables, local proof, rollback procedure) + Sessions-Log entry. Kind dual-run BATCHED (combined tier 3-5); prod/GKE cutover STILL GATED on user. Tiers 1-3 activated; tiers 4-5 remain inert. Refs noetl/ai-meta#234
docs: Phase 9 tiers 1 & 2 — in-cluster kind dual-run VALIDATED (2026-07-05) Event-log + projection primary-serve verbs (ehdb-selfcheck, worker v5.62.0 image sha256:9d222db6) ran in a Job pod on kind node noetl-control-plane (context kind-noetl, not GKE): both tiers served_by_ehdb:true + reversible:true + dual_run_holds:true + secret-free metrics (exit 0); control-plane role => guard_refused (exit 4); pod Succeeded, 0 restarts. Roadmap tier-1/tier-2 rows PENDING/BATCHED => VALIDATED. prod/GKE cutover stays gated on the user.
Phase 9 tier-2 projection primary-serve implemented + merged (2026-07-05) Roadmap: tier-2 table row → primary-serve IMPLEMENTED + MERGED; new "Tier 2 (projection / read-model)" section + tier-2 rollback procedure (2 levers); Phase-7 acceptance checkbox flipped. Design-Projection: status + primary-mode section updated to reflect activation + reversibility. Home: Phase-7 line + primary-serve status. Sessions-Log: tier-2 entry prepended. ehdb#248 d08013c + worker#162 a56583c (released v5.62.0 36875e3). Kind dual-run BATCHED (podman VM wedged); prod/GKE cutover STILL GATED.
docs: Phase 9 tier 1 (event log) primary-serve implemented + merged — ehdb#247 7f014c9 + worker#161 v5.61.0; kind dual-run pending; prod/GKE cutover STILL GATED on user; tier-1 rollback procedure documented
docs: Phase 8 vector engine slice landed — Phase 8 shadow-complete (ehdb#246 f6c5a6f + worker#160 c7b8872 v5.60.0); Phase 9 per-tier primary-cutover enumerated
docs: Phase 8 object/blob engine slice landed (ehdb#245 + worker#159 v5.59.0) Second Phase-8 engine slice merged: the content-addressed object/blob engine (ObjectBlobDriver over ehdb-storage ImmutableObjectStore + a key->digest registry stream) with a disabled-by-default worker NOETL_EHDB_OBJECT shadow (digest/length/ retrievability parity) for the state-shard (#166) + result-tier (#104) platform artifacts. Design note + Roadmap Phase 8 updated; vector slice remains; no primary cutover (Phase 9). Standing evidence = 109 ehdb + 13 worker tests + built ehdb-selfcheck object verbs; in-container kind DEFERRED (env build eviction).
docs(design): Phase 8 KV/Object/Vector engines design note + KV slice landed New design note covering all three Phase-8 engines (KvStateDriver / ObjectBlobDriver / VectorDriver), each mirroring the Phase 6/7 driver+shadow shape; KV/state slice merged (ehdb#244 e2547200 + worker#158 7876be06, disabled-by-default shadow). Object/blob + vector enumerated as follow-up slices; no tier cut over to primary. Linked from Sidebar, Roadmap Phase 8 (in-progress), and the RFC.
docs: Phase 7 — reconcile shadow-slice status to reality (worker#157 merged eadc3a5) The prior Phase 7 note (68b90d2) asserted the worker shadow slice had landed before it actually merged. The worker shadow (src/ehdb/projection.rs, NOETL_EHDB_PROJECTION=off|shadow|primary, default off) is now merged as worker#157 (eadc3a5), pinning ehdb-reference to the merged #243 rev e0f1c0f. Correct the design-note / Roadmap / Home / Sessions-Log status wording to cite the actual PRs+SHAs and record the worker-side validation (13 unit tests, clippy -D warnings, ehdb-selfcheck disabled/shadow/guard/primary evidence; in-container kind deferred per the standing env-eviction pattern).
docs: Phase 7 projection / read-model engine — design note + engine slice + shadow slice landed
docs: Phase 6 event-log core engine — design note + shadow slice landed New Design-Event-Log-Core-Engine page (semantics contract, on-disk format, driver interface, shadow/parity strategy, config surface, assumptions, remaining work). Roadmap Phase 6 marked in-progress with what landed (ehdb#242 9a9b28d + worker#156 43c8f0f) and what remains. Home implementation note + Sessions-Log entry + sidebar link.
docs: RFC DECIDED — EHDB completion program (event-log engine bottleneck fix + noetl self-sufficiency, platform-only); Roadmap Phases 6-10
docs: RFC — server↔EHDB coupling (loose) + EHDB as pluggable storage substrate (Phase 8); up for Codex review
docs: EHDB Phase E RAG — label in-container kind validation deferred (env build eviction; ~110-min cold worker-rust image timed out) Local ehdb-selfcheck A->E+RAG drive remains the standing evidence. Refs noetl/ehdb#234 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs: EHDB Phase E second slice DONE — bounded RAG retrieval wired into worker-rust (Phase E COMPLETE) ehdb#240 (c2aaad5) bounded retrieve/ingest helpers + CLI; worker#155 (d1ebaf2) in-process src/ehdb/rag.rs retrieve+ingest + rag-suite selfcheck. Validated via the built ehdb-selfcheck A->E+RAG drive; in-container kind confirmation pending the worker-rust image build. #234 stays open (Phase 6/7 remain).
docs: EHDB Phase E first slice DONE — system WASM store wired into worker-rust (worker#154 3162b1d, ehdb 9bb5928); RAG deferred
docs: EHDB re-home COMPLETE — both PRs merged, #238 closed - Python retire noetl/noetl#691 MERGED (ff3a920f); Roadmap re-home → complete, #238 CLOSED. Architecture "Implemented" note updated. Sessions-Log entry. - worker-rust owns EHDB in-process (noetl/worker#153 d6226a2); Python path gone. Refs noetl/ehdb#234, closes noetl/ehdb#238
docs: EHDB worker integration re-homed into worker-rust (Rust-only); Python retired - Roadmap "Integration debt: re-home Phase B–D" → DONE: worker-rust owns the integration in-process (noetl/worker#153 MERGED d6226a2); Python retire (noetl/noetl#691) open awaiting review; kind-validated. - Architecture "Rust-first EHDB worker integration" decision → Implemented. - Sessions-Log: re-home + kind evidence entry. Refs noetl/ehdb#234, noetl/ehdb#238
docs: Rust-first EHDB worker-integration decision + Phase D pointer finalized Architecture: add 'Architecture Decision: Rust-first EHDB worker integration (Python as thin wrapper)' — engine already Rust; the Phase B–D Python glue in noetl/core is a disabled-default stopgap that does not run in the prod Rust worker; target = worker-rust owns the integration in-process against the ehdb crate; Python = thin binding (the Polars model). Roadmap: mark Phase E (system WASM store -> RAG) as Rust-first; add integration-debt item to re-home Phase B-D worker integration from Python into worker-rust (noetl/ehdb#238). Sessions-Log: entry for the decision + noetl#690 merge / ai-meta repos/noetl pointer finalization. Refs noetl/ehdb#234, noetl/ehdb#238 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(phase-d): event-stream drain — Architecture + Roadmap (Done) + Sessions-Log Document integration Phase D (event-stream integration path): the bounded disabled-by-default worker/playbook/system durable-consumer drain (project/consume/ack), the event-log-authoritative invariant (EHDB is a derived consumer, never writes the NoETL event log), and the control-plane guard. Roadmap Phase D = Done, next = Phase E (system WASM store). EHDB noetl/ehdb#237 (3cefba9); NoETL noetl/noetl#690 (awaiting review). Tracks noetl/ehdb#234.
docs: EHDB integration Phase C — worker/playbook data-plane append/read step