wiki(L1 #208): writer-restart survival — both defects fixed + a third in the publish path; kind before/after; 3 PRs held
wiki(L1 #205): ehdb#301 merged as 03e94be; adoption held to bundle with #208
Refs noetl/ai-meta#205, noetl/ai-meta#208
wiki(L1 #205): dispatch latency root-caused + fixed; T4/T5 status corrected
- Program page: T4 row corrected (full flip SUCCEEDED on shastaratech
prod 2026-07-27 round 3, was still 'PREPARED, NOT EXECUTED'); T5 row
now names its two blockers (#205 latency, #208 writer-restart survival);
new section on the #205 attribution + fix.
- Sessions log: 2026-07-28 entry with the per-hop attribution, the
three-part fix, the loopback before/after, the kind correctness PASS,
and why kind cannot resolve the latency delta.
- Home: implementation note for the fix.
Refs noetl/ai-meta#205, noetl/ai-meta#208
L1 T4 COMPLETE — EHDB is the production command bus (full flip 80/80 on shastaratech prod)
Runbook: status banner rewritten to SUCCESS; T4 go/no-go checklist cleared
with the latency caveat + a new T5 checklist; prod execution log restructured
into three rounds (held-at-canary / failed-on-#203 / succeeded) with the #203
root cause + fix + before-after table, the post-flip prod state, the rollback
command, and the latency regression.
Program page: L1 status T0-T4 done / T5 held; T4+T5 table rows rewritten.
Home: program bullet leads with prod-command-bus state.
Sessions-Log: new 2026-07-27 entry.
Refs noetl/ai-meta#194, #202 (closed), #203 (closed), #205, #206
Refine boundary to write-behind-cache model; re-scope D5/D6 issues (+#198 sink)
EHDB is never the durable system of record for business data (customer's
connector-backed store is), but MAY hold business processing context transiently
as a write-behind cache: two roles — (a) permanent control-plane log stays lean/
reference; (b) transient processing cache bounded + sunk to the customer store +
evictable. Supersedes the 'never holds business payload' absolute. Re-scopes the
fixes: #195 reframed (keep PERMANENT noetl.event lean; transient cache keeps full
context for the drive — dissolves the earlier drive-breaking risk since the drive
reads the transient cache not the permanent log); NEW #198 the write-behind sink
path (transient context -> customer system of record + sink-confirmation-gated
eviction; builds on connector tools + result-tier byte-source + GC); #196
reframed (permanence in noetl.command); #197 unchanged.
Record boundary fix as three tracked issues (#195/#196/#197), Slice A reframed
The D5/D1/D3/D2 boundary finding is now tracked, not unfixed: #195 (design-level
— durable datasets carry only the extracted predicate block, NOT raw payload;
orchestrate-core sufficiency proof, since context/result are load-bearing for
the drive decision per state_builder.rs:124 — NOT a blind 'drop all inline'
refactor), #196 (command-input tiering), #197 (RAG-ingest guard test). All Refs
noetl/ai-meta#194, sequenced behind the live L1 T4 command-bus work.
Record sharpened boundary (operations/references, never payloads) + D1-D10 audit
Sharpened invariant: EHDB holds control-plane metadata + data-plane operational
state + references/pointers, NOT the actual data a playbook processes (business
payloads flow through connectors). Records the code-verified per-dataset audit:
D4/D7/D8/D9/D10 clean; D5-large clean (URN + object-store byte-source); D1/D2/D3
+ D5-small boundary-risk (<=100KB step payloads inline in the durable event log /
slim projection / command input, command.rs:1493 / state_builder.rs:128 /
execute.rs:1218); D6 clean-today with a latent rag::ingest seam. Recommended
fixes recorded (Slice A reference-only for small payloads; Slice B tier command
input; Slice C RAG guardrail) — NOT applied. Umbrella:
https://github.com/noetl/ai-meta/issues/194
docs(ehdb): L1 T0-T3 shadow arc complete + T4 cutover runbook (prepared, gated)
Program page: L1 T0-T3 status table (all merged; latency gate PASS, bus
p99 137us). New Runbook-L1-Command-Bus-Cutover: the NOETL_COMMAND_BUS
flag, cutover sequence + validation gates, go/no-go checklist, and the
one-command rollback — PREPARED, NOT EXECUTED (T4/T5 human-gated).
Sessions-Log + sidebar updated.
Refs noetl/ai-meta#194
docs(ehdb): L0 object-store foundation complete — datasets D1-D10 merged
Program page: status → L0 COMPLETE (engine L0.1-L0.7 + all predefined
datasets D1-D10 merged, main 0af99fc); added the D1-D10 dataset table +
the op-log/fold shape note. Sessions-Log: D6-D10 landing entry.
Refs noetl/ai-meta#194
L0.7 dataset-generic engine MERGED (bb05cf7); next = D2 command queue
L0: L0.1-L0.6 all MERGED to main (3eb2eff); N-way replication implemented + proven
L0: mark L0.1–L0.5 built + merged (D1); noetl-native substrate, #278 re-land note
Add noetl-only scoping invariant + ClickHouse meta-catalog to L0 design
Program invariant (recorded prominently): EHDB is noetl-internal-only, a FIXED
set of predefined datasets (D1-D10: event log, commands, projections, KV,
object/blob, vector, catalog, runtime, system-WASM, provider-facts), NOT a
general DB — no arbitrary schemas/DDL, no query planner, no hosted-DB surface;
business data never in EHDB. It SHRINKS every layer. L0 design now = VM/VictoriaLogs
write engine + a ClickHouse-MergeTree-style meta-catalog (manifest + sparse index
-> object-store pointer; fixed-schema table-format-over-object-storage). Scope
cuts: no general inverted index (few fixed per-dataset indexes only); L3 cut from
a SQL engine to fixed reads + read-only export (~2-3q -> ~1q). Umbrella:
https://github.com/noetl/ai-meta/issues/194
Reframe program page to layered platform (L0-first); NATS takeover = L1
Major reframe: EHDB becomes a layered platform built L0-first — L0 replicated
object store (modeled on VictoriaMetrics/VictoriaLogs: buffered flush ->
immutable parts -> background merge -> inverted-index-first -> columnar) -> L1
streaming (the NATS takeover) -> L2 KV -> L3 append-log SQL. L0 resolves the HA
debate: durability+replication from the object store => fungible writers,
per-shard-Raft (T-RF) retired, only a light L1 ordering lease remains. Records
the honest departure (VM keeps the hot path on local disk; object storage is
backup-only, so L0's live object-store durability tier is net-new beyond VM),
the reuse assessment (#254 segments approx immutable parts; gaps = inverted
index + merge engine + object-store tier), per-layer cost, and the first L0
slice (L0.1). Master RFC: docs/rfc/ehdb-layered-platform.md. Umbrella:
https://github.com/noetl/ai-meta/issues/194
Program page: lock HA posture — shards-only now, RF deferred (replication-ready)
Verified parity (prod NATS replicas:1, streams R1). Ships (c) single-writer-
per-shard; per-shard replication factor deferred to a named non-blocking phase
(T-RF). Adds the replication-ready seam (the #254 durable_eventlog_affinity
single-writer + open_read_only followers + durable_eventlog_shared log-shipping
+ durable external cursors => T-RF additive, no log/cursor rewrite), the
failover sequence + seconds-scale loss-free stall estimate, and the worker
reconnect-on-failover story (unchanged StatefulSet DNS). Umbrella:
https://github.com/noetl/ai-meta/issues/194
Program page: revise to (c) per-shard-writer-as-broker topology
Supersedes (b) co-locate on latency (two delivery hops). (c): the stateful
per-shard writer owns its shard's durable log AND delivery; workers subscribe
directly to the writer; stateless server publishes to the writer and is out of
the delivery path -> one delivery hop, matching NATS. Records the honest premise
correction from the code trace (#166 sharding dormant; system pool is a
single-replica Deployment, no StatefulSet/PVC/identity; drive hands back to the
server), so (c) is a build not a relocation (~2q, more than (b)). Adds the
alternatives ledger, honest downsides (connection fan-out, per-shard HA,
concentration on the #163 OOM pod), revised cost, and the (c)-shaped T0 spec
with a latency-vs-NATS comparison. Umbrella:
https://github.com/noetl/ai-meta/issues/194
Program page: lock (b) co-locate topology for EHDB->NATS takeover
Records the locked topology (2026-07-15): durable log + change-feed stay in
the per-shard writer (#166, stateful); the noetl-server stays stateless (#115)
and owns delivery only (tails the change-feed, pushes over the SSE hub); option
(a) server-embeds-EHDB rejected. Adds the three-component split, re-derived
Track T phases, revised cost (~1.5 quarters), and the (b)-shaped T0 spec.
Program umbrella: https://github.com/noetl/ai-meta/issues/194
Add EHDB→NATS takeover program page (decision + plan, nothing built)
Records the locked decision (2026-07-15): remove NATS via noetl-server-owned
push reusing the gateway SSE ConnectionHub over EHDB's durable log +
change-feed; standalone ehdb-server broker rejected. Track S (storage cutover,
in flight) + Track T (transport, server-owned-push, T0->T5, unbuilt), revised
cost, three open sub-decisions, T0 slice spec. Cross-linked from Home +
_Sidebar. Program umbrella: https://github.com/noetl/ai-meta/issues/194