Architecture Four Engines

Architecture — the four engines

Scope authority: docs/SCOPE.md (#320, under #324). Invariants: Consistency Invariants.

This page supersedes the five-tier engine list. EHDB owns four engines. OLAP and a standalone ANN/vector engine are removed from scope — a removal, not an externalisation: no external engine is introduced in their place.

The four

flowchart TB
  subgraph W["writer — one per shard"]
    EL["<b>1. event log</b><br/>source of truth<br/>append-only · single-writer/shard<br/>global_sequence total order"]
  end

  EL -->|fold| PR["<b>2. projection</b><br/>every read model<br/>analytical views AND<br/>vectorized retrieval"]
  EL -->|fold| KV["<b>3. KV</b><br/>keyed lookup<br/>clock-free fold"]
  EL -->|seal → upload| OB["<b>4. object</b><br/>immutable parts<br/>content-addressed"]

  PR --> CAT["catalog<br/><i>a projection,<br/>not a fifth engine</i>"]
  PR --> VEC["vectorized retrieval<br/><i>bounded cosine over an<br/>execution-scoped candidate set —<br/>no ANN index</i>"]

  classDef serve fill:#1f6f3f,stroke:#0d3,color:#fff
  classDef shadow fill:#5a4a1f,stroke:#c93,color:#fff
  classDef derived fill:#2b3a55,stroke:#68a,color:#fff
  class EL,PR serve
  class KV,OB shadow
  class CAT,VEC derived

Loading

Green = has a runtime serve path. Amber = no serve path exists. That distinction is not cosmetic; see below.

#	engine	owns
1	event log	the source of truth; append-only, single-writer-per-shard, global_sequence total order
2	projection	every read model — including analytical views and vectorized retrieval
3	KV	keyed lookup
4	object	immutable parts in an object store

The layers the engines are built on

The four engines above are what EHDB owns. The layers below are how it is built — a separate axis, and conflating them is a common source of confusion.

flowchart TB
  L3["<b>L3</b> — append-log SQL<br/><i>fixed schema, no planner</i>"]
  L2["<b>L2</b> — KV<br/><i>keyed lookup over a clock-free fold</i>"]
  L1["<b>L1</b> — streaming<br/><i>the NATS takeover; one-hop delivery</i>"]
  L0["<b>L0</b> — replicated object store<br/><i>durability + replication foundation:</i><br/>buffered flush → immutable parts → background merge<br/>+ a manifest/sparse-index meta-catalog"]

  L3 --> L2 --> L1 --> L0

  classDef built fill:#1f6f3f,stroke:#0d3,color:#fff
  classDef planned fill:#3a3a3a,stroke:#888,color:#ddd
  class L0,L1 built
  class L2,L3 planned

Loading

⚠ Only L0 is a crate (ehdb-l0). L1 exists as the feed (ehdb-feed, L1 T0–T4); L2 and L3 are plan, not code — do not read the diagram as an inventory of what is implemented.

Why L0 first. Durability and replication come from the object store, which makes writers fungible — a replacement writer cold-loads sealed parts from L0 rather than recovering peer state. That is what retired the per-shard-Raft plan; only a light L1 ordering lease remains.

⚠ The honest departure from the VictoriaMetrics model this borrows from: VM keeps the hot path on local disk and treats object storage as backup. L0 uses the object store as a live durability tier, which is net-new and therefore the highest-risk piece.

Node roles

⚠ These are deployment facts, not a code abstraction. There is no NodeRole type in the codebase; a role is simply which workload a process is running as. Observed on prod, 2026-09-16:

role	workload	why it is separate
write	sts/noetl-cmdbus-writer	single-writer-per-shard is the event log's ordering invariant; a StatefulSet at replicas: 1 is what stands in for mutual exclusion (see I-EL-2 — ⚠ this is a preference, not a primitive)
read	deploy/noetl-server-rust, sts/noetl-server-rust-embedded	serve paths and the embedded shadow; holds no authority over the log
compute	deploy/noetl-worker-rust, worker-system-pool*	executes steps; mirrors to shadow tiers
maintenance	cronjob/noetl-scheduled-cleanup, cronjob/noetl-state-builder-watchdog	GC and convergence sweeps, deliberately out-of-band so they cannot stall a serve path

⚠ The write role carries durable state and a pinned digest; the read and compute roles are fungible. Any change whose rollout order depends on the writer moving first is therefore gated on that pin — which is why the durable kv/object shadow store is not deployable on a writer that predates it.

⚠⚠ The writer has no volumeClaimTemplates. Verified 2026-09-16: it mounts three statically-created PVCs by name — noetl-cmdbus-writer-0-data, noetl-eventbus-writer-0-data, noetl-eventbus-kv-0-data (10Gi / 20Gi / 10Gi, premium-rwo). The claims therefore exist independently of the StatefulSet and survive its deletion, which is not the behaviour a reader assumes from a StatefulSet with durable state. See the mirror runbook for why this changes how a roll behaves.

⚠ Only two tiers can actually serve

pub const SERVE_WIRED_TIERS: &[&str] = &["eventlog", "projection"];

The Backend Configuration page presents five tiers each with an off / shadow / primary mode, which reads as though setting primary makes any of them serve. It does not. Only eventlog and projection have a runtime serve path; tier_serves_primary is the guard that keeps a tier from reporting it serves when nothing calls it.

Setting NOETL_EHDB_KV=primary or NOETL_EHDB_OBJECT=primary does not promote anything. Treat the mode table as configuration surface, and this list as capability.

Why vector and OLAP collapse in

EHDB stores a fixed set of internal datasets with predefined access paths, so the analytical questions and retrieval queries over them are known in advance — which is exactly what a read model materialises.

• Analytics become projections. A new analytical question is a new projection, not a new query planner.
• Vector retrieval becomes a vectorized projection. Embeddings are materialised as read models and searched over bounded, execution-scoped candidate sets (tenant / namespace / model / execution). Retrieval is always scoped to a small slice, so exact cosine is sufficient and no ANN index is needed.

This matches what is built. ehdb-reference/src/vector.rs is already "a bounded cosine-similarity search over the collection's live points" — there is no ANN index to remove. The gap was in the promise, not the implementation: the README and roadmap still committed to absorbing Qdrant and ClickHouse. "Future surface" and "out of scope" are different commitments, and only one of them was true.

Engine ≠ tier ≠ StoreTier

Three vocabularies overlap here and are routinely conflated:

term	what it is
engine	one of the four above — an owned storage capability
tier (NOETL_EHDB_<TIER>)	a configuration selector with off/shadow/primary
StoreTier	a worker-side durable store: EventLog, Projection, Catalog
QueryTier	a worker-side shadow read surface; also names kv, object, vector

catalog has a StoreTier but is a projection, not a fifth engine — a thing gains a StoreTier when it gains a store; it does not thereby gain an engine. vector has no StoreTier, no durable store, and is absent from SERVE_WIRED_TIERS.

Related

Consistency Invariants · Backend Configuration · Architecture · Roadmap · spec: election + fencing · spec: durability window