Skip to content

Architecture Resilient KV Core

Jump to bottom
Kadyapam edited this page Sep 12, 2026 · 2 revisions

Architecture: EHDB as a resilient KV core — CockroachDB's layers, minus SQL

Status: draft overview · Spec artifact for the resilient-core programme · Coordination issue noetl/ai-meta#339

Grounded in noetl/ehdb@24e63b8 (tag v0.2.0, the tag noetl-server pins) and prod readings taken 2026-09-11.

The thesis

CockroachDB stacks five layers: storage (MVCC KV) → replication (Raft) → distribution (ranges) → transactions → SQL. The resilient KV core is layers 5→2; SQL is Postgres-compatibility cost. noetl does not need it: its access patterns are append-event, fold-projection, get/scan-by-key, range-by-sequence. None require a parser, planner or optimiser.

So EHDB's target is the resilient KV core, with event sourcing, projections and context handling directly on top, and no SQL engine — permanently.

  event log (D1)   projections (D3)   context (D4 KV)
  ────────────────────────────────────────────────────
   append              fold                get/put
  ┌──────────────────────────────────────────────────┐
  │ resilient KV core                                │
  │  • compare-and-append at expected sequence       │  L2, narrowed
  │  • quorum-acked unsealed tail  ← THE GAP         │  L4
  │  • static hash partitioning + gossip membership  │  L3, adequate
  │  • immutable parts, N-way copy, cold-load        │  L4, sealed: done
  │  • manifest + sparse index + bloom + merge + GC  │  L5, done
  └──────────────────────────────────────────────────┘
                     NO SQL LAYER

Layer map, as implemented

⚠ Scope note that changes how this reads: noetl-server depends on ehdb-l0 + ehdb-feed only. ehdb-reference (23,949 LOC), ehdb-service and ehdb-transaction are not in the production dependency path. Several capabilities below exist in the tree and are not linked into the running binary.

L5 Storage — done, and the strongest layer

Immutable parts + ClickHouse-style meta-catalog (Manifest / PartMeta), per-part SparseIndex, min/max pruning, per-granule blooms over execution_id, small→big merge with atomic manifest swap, retention as drop-partition + orphan GC, a columnar per-field codec, and fsync-per-append for D1.

Not an MVCC KV, and should not become one. The version axis is global_sequence. Mutable state is modeled as a fold over an append-only op log (D2 command_queue already does this). For event sourcing that is a better fit than timestamps.

L4 Replication — the gap, and narrower than "no replication"

Piece State
N-way copy of sealed parts Implemented (L0.6): ReplicaTarget, PartMeta::replicas, read fallback
Quorum on the write path Missing — an append acks after fsync to one local disk
The unsealed tail Unreplicated by construction — a part replicates only once sealed
Bounding the window ✅ Live: NOETL_EHDB_SEAL_MAX_AGE_MS=5000 on the writer
Measuring the window ✅ Live on both engines
Independent failure domain ❌ Absent in prod — tier dir nested in the writer dir, one PVC
open_replicated in prod ❌ Never called outside tests
validate_replica_domains ❌ Never called outside tests — the guard cannot fire
Single-writer enforcement ❌ Rests on StatefulSet replicas: 1
Election / fencing Implemented in ehdb-reference; not linked into the server, fencing shadow-mode
Leaseholder reads Absent — no concept

Prod readings, 2026-09-11:

writer :9106 (event bus, the D1 tier that is primary)
  ehdb_l0_unreplicated_age_seconds{shard="0"}  0.000
  ehdb_l0_unreplicated_records{shard="0"}      0
  ehdb_l0_replicated_lag_seconds_count         383

writer :9102 (command bus)
  ehdb_l0_unreplicated_age_seconds{shard="0"}  3.667
  ehdb_l0_unreplicated_records{shard="0"}      1

NOETL_EHDB_SEAL_MAX_AGE_MS = 5000
NOETL_EVENT_BUS_WRITER_DIR = /data/eventbus
NOETL_EHDB_TIER_SERVICE_DIR = /data/eventbus/ehdb-tier   ← same PVC

So the window is measured and bounded (~5s + one sweep interval), and 383 parts have genuinely replicated. What it replicates to is a subdirectory of the volume holding the original. failure_domain.rs states the consequence plainly:

An RF of N over one domain is an RF of 1 wearing a larger number.

L3 Distribution — partial, and mostly adequate

Static hash partitioning (shard_for(execution_id)), an affinity layer, foca gossip membership (Scuttlebutt + phi-accrual), D8 RuntimeDataset as the topology projection. Missing: key ranges, auto-split/merge, rebalance.

Ranges are largely SQL-layer tax. Cockroach needs them because a SQL keyspace is unbounded and arbitrarily skewed. noetl's key is a snowflake — uniformly distributed by construction, so it does not develop hot ranges. What EHDB actually lacks is online reshard, not splitting.

L2 Transactional — mostly not needed

Event sourcing wants exactly one primitive: append these N events at this expected sequence, or fail — a compare-and-append, not a transaction manager. Everything downstream is a deterministic fold.

ehdb-transaction is a control-plane metadata mutation log (catalog/stream/retrieval/system/storage), used by ehdb-reference and ehdb-service — not a data-plane engine, and not linked into the server.

The cross-dataset case (append a D1 event and update the D3 projection atomically) is best solved by not needing it: the projection is derived, so it may lag. That is the serve-on-behind design.

L1 SQL — absent, deliberately, permanently

Removing it also removes the reason for most of L2 and much of L3.

The resilience decision

Keep "no consensus" for sealed parts

ehdb-l0/src/lib.rs §2.7 already decided: N-way copy of immutable parts, HDFS block-replication model, no consensus / no Raft, justified by immutability — byte-identical copies cannot conflict. That argument is correct and this programme keeps it. Raft over sealed parts would add cost and no guarantee.

Add Raft only where immutability runs out

The argument covers sealed parts. Between ack and seal there is no immutable object yet — only one node's opinion. So the question is not "Raft or not" but what is the smallest thing that needs agreement: the tail, and who owns it.

A small per-shard Raft group over the unsealed tail only.

  • Makes the ack survivable — the one missing property.
  • The log stays tiny: seal truncates it, so snapshot transfer and compaction mostly vanish. The sealed part is the snapshot; cold-load is the restore.
  • Solves single-writer properly. The leader is the writer and its term is a consensus-backed epoch — exactly what Invariant F's fencing needs. ⚠ election.rs already warns: "a Lease elects; it does not fence", and a paused holder can believe it still holds an expired lease.
  • Self-sufficient: a crate, not a service to operate.

Rejected — full Raft-replicated ranges (Cockroach-style). Re-solves sealed-part replication that immutability already solves more cheaply, and drags in range/split/rebalance machinery whose purpose is a SQL keyspace.

Library: open question. openraft (tokio-native, storage behind a trait — matches EHDB's shape) vs tikv/raft-rs (consensus module only, battle-tested in TiKV). ⚠ Decide against a partition + clock-skew harness, not documentation: consensus bugs fail silently, which is this codebase's most expensive failure mode.

Phase plan

⚠ Corrected against prod rather than assumed. Two phases the original plan listed as work are already shipped — the recurring "built ahead of its consumers" pattern.

Phase What State
0 recovery→tier (#307); D1 mirror at the server chokepoint; embedded engine in shadow; D3 serve-on-behind; AC14 verification independence live as v3.108.1 ✅ shipped
1 Measure the durability window ✅ already live — both engines export it
2 Bound the window ✅ already live — SEAL_MAX_AGE_MS=5000
3a Make the failure-domain guard able to fire (validate_replica_domains wired into open_replicated, shadow-first) ✅ noetl/ehdb#359 — also found the Arc forwarding impl dropped failure_domain
3b Give the tier its own failure domain ⏸ deferred by decision — refill after the mirror is clean, not migrate (Plan)
3c Stop the mirror losing events ⚠ partly — durable repair live+armed (v3.108.3), but delivery still fails (ai-meta#343)
4 Enforce single-writer: link election + fencing into the server's dependency path, promote fencing shadow → enforce ⏳
5 Decide the Raft library, then quorum-ack the tail ⏳
6 Online reshard ⏳

Phase 3 is the first with real risk reduction: the replication API existed and nothing in production called it.

Mirror coverage — measured, root-caused, partly fixed (2026-09-12)

The tier was diverging from Postgres on 78% of recent executions. Two causes, and one was the measurement itself:

cause share status
(b) reference vs inlined context — the verification folded the raw Postgres row under PERMANENT_LOG_LEAN, digesting a pointer against content 30% ✅ fixed (v3.108.2 serve path, v3.108.3 diagnostic)
(a) real missing events — the mirror dropped batches terminally after ~63 s against a single-endpoint relay, and the repair endpoint had no caller 48% ⚠ mechanism fixed + armed, delivery still failing

Parity proof for (b), same fixed 40 executions across the v3.108.3 deploy:

before: agree=10  divergent=30   (15 equal-count / 15 missing-events)
after : agree=26  divergent=14   newly agreeing 16, regressed 0

⚠⚠ The first attempt at this proof failed informatively. v3.108.2 fixed the serve path and the divergence count moved by 0 of 40, because the endpoint being measured (compare_sources) was never on the path that changed. An instrument that cannot see a fix cannot validate one — it is now held to the same rule, with a guard.

⚠ The remaining 14 are real loss and do not close yet: re-mirrored rows fail at the transport (attempt{unavailable} ~47/min while the relay pod is up), and an execution whose tier payload exceeds 1 MiB cannot be read back at all, so it leaves the denominator rather than reporting as divergent (ai-meta#343).

Reading the prod numbers honestly

  • unreplicated_age_seconds = 0.000 is a real reading, not an absence — the shard has nothing pending. Treat absent series and zero differently.
  • ⚠ ehdb_l0_ingest_append_failed is absent on :9106, not zero. A build predating that counter serves no such series, and that reads exactly like a healthy one.
  • ⚠ The writer hosts two L0 engines (:9102 cmdbus, :9106 event bus). Instrumenting or reading one is not the other — a grep finds one wiring and stops.

Related

EHDB

Programs

Kind validation

Query interface

Runbooks

Design notes

GitHub

Clone this wiki locally