Skip to content

Embedded State Foundations

Jump to bottom
Kadyapam edited this page Sep 8, 2026 · 2 revisions

Embedded-state foundations — what landed in ehdb-l0 and ehdb-gossip

NoETL is moving EHDB from a networked writer service to an embedded library called in-process per shard (noetl/ai-meta#332). This page records what that added to this repo and, for each guard, what it exists to prevent — so it is not later deleted as noise.

Nothing here is deployed. All of it is inert until the server opens a local engine, which is a separate gated step.

ehdb-l0::format_version — refuse a mismatched on-disk layout

FORMAT_VERSION (currently 1) is written to the substrate as a marker at FORMAT_VERSION_KEY. L0Engine::open* refuses to open a substrate whose marker disagrees.

What it prevents. Embedding means two processes — server and worker — each open() an engine directly, and they pin this repo by git rev: at the time this landed they were 70 commits apart. Two builds writing one layout at that divergence is silent corruption, and a git-rev pin has no mechanism that would notice. Refusing is recoverable; misreading is not.

Where it runs. open_replicated_with_metrics — the chokepoint every open* funnels through — before the manifest is read, and on every replica (each is an independent substrate; a single-replica check waves through a replica written by a different build).

⚠ Not manifest_version. That already existed and is a monotonic sequence number for manifest snapshots — "which snapshot", not "which layout".

⚠ Bump FORMAT_VERSION in the same change as any incompatible layout change.

The bug this module's own tests found

The first read_marker used get_range(key, 0, 32) with Err(_) => Ok(None). get_range treats a range past the end as an error by design, and the marker is 1–3 bytes — so the read failed every time, and the blanket arm turned that, a corrupt marker, and an I/O failure into "absent" → re-initialise, writing our marker over another build's layout. Now: exists() then get_all(), and every other error propagates. UnreadableSubstrate in the tests is what pins it.

ehdb-l0::cursor — stored per-shard apply cursor

cursor::load(substrate, shard) / cursor::advance(substrate, shard, seq), a durable u64 per shard at CURSOR/shard-<n>.

What it replaces. ehdb-reference's ProjectionCheckpoint derives the cursor by replaying the whole log and taking the max. Correct and exactly-once-safe, but recovery pays an O(log length) scan before it can start replaying the tail.

Recovery is now cursor::load + L0Engine::read_partition_after(shard, seq). Measured, not asserted: 20 records with the cursor at 15 replays 5.

Invariants:

  • ⚠ Monotonic. A lower advance is ignored, not written. A backwards cursor re-applies an applied window; a forwards one past what was applied skips events, which is worse.
  • An absent cursor reads as 0 and replays everything — losing the pointer costs time, never correctness. That asymmetry is what makes storing it safe.
  • ⚠ An unreadable cursor refuses, rather than reading as 0. Reading as 0 is safe here, but it would hide a substrate failure.

ehdb-l0::projection and ::runtime — now exercised

Both ProjectionStore (D3) and RuntimeStore (D8) had zero consumers and zero tests. They were built ahead of their consumers, so "it exists" and "it works" were independent questions. Both now have the second answered:

D3 projection — append→read-latest; list_executions returns distinct ids; state survives a reopen; cold_load recovers another instance's durable state; the manifest snapshot is storage layout, not the read model.

D8 runtime — register/heartbeat/deregister fold latest-op-wins; a heartbeat for a never-registered or deregistered node returns None and does not resurrect it; list_live_since evicts by watermark with no wall clock, and a quiet node stays registered rather than being confused with a departed one; membership survives a reopen; cold_load reads another instance's membership; and the op log retains all three events after the current state has forgotten the node — the audit property a heartbeat column cannot give.

⚠ proj_seq is engine-assigned from a global sequence — not caller-supplied and not per-execution.

ehdb-l0::runtime::validate_op + OpOrigin — append-time validation

What it prevents. D8 previously appended whatever it was handed, for any id. Membership is persisted, so a poisoned entry does not vanish when whoever wrote it leaves — it is in the log and every fold replays it. Validating at append is the only point at which refusing keeps it out.

Structural half (validate_op, no network trust required): non-empty and length-bounded worker_id; a restricted charset [A-Za-z0-9-_.:]; a dot-run rejection; a bounded contract; and field/event agreement — a Register or Heartbeat carrying heartbeat: 0 is refused, because list_live_since(1) would then silently drop a node that is beating.

⚠ The dot-run check looks redundant and is not: dots are legal in an id (shard.3), so ".." is the one traversal shape the charset rule does not catch. A whitespace check that was redundant was removed — an ungated branch is an untested branch.

Network-trust half (OpOrigin): consulted on the append path, defaulting to TrustedLocalOrigin — named for what it assumes, not described as safe. RuntimeStore::with_origin(..) installs a real one.

⚠ TrustedLocalOrigin accepts every structurally valid op. Correct while every caller is in-process; it must be replaced before gossip feeds this store from the network.

ehdb-gossip — foca, transport only

A new crate, so ehdb-l0 keeps its two dependencies and consumers opt in by not depending on this one.

The split: foca observes, EHDB records. foca 2.0.0 (SWIM + suspicion + indirect probing) carries transport and failure detection only; every transition it reports is appended to D8, and all state, history, recovery and query stay in EHDB. Chosen over chitchat because chitchat is Scuttlebutt + phi-accrual and does not do indirect probing — the property that distinguishes "I cannot reach X" from "X is down".

  • ShardIdentity puts the shard in the gossip identity, so D8's contract comes from gossip rather than being inferred. renew() bumps an incarnation so a wrongly-evicted node can out-rank the corpse its peers hold; ⚠ higher incarnation wins — inverting it means a rejoining node is permanently rejected.
  • MembershipSink maps all 7 foca notifications. ⚠ foca is edge-triggered — no per-peer liveness tick — so D8's heartbeat watermark is advanced by a local tick over the live set, which means list_live_since guards against this adapter stalling. A different property from what gossip guards. ⚠ rename registers the new identity before dropping the old, so the node is never absent from routing mid-rename.
  • GossipOrigin cannot be constructed without a verifier — no Default, no permissive constructor, no relaxing feature. Default posture is RejectUnsigned: an unconfigured cluster trusts no remote membership.

Status: inert. No socket, no runtime, no bring-up.

First consumer: noetl/server v3.106.0 (2026-09-08)

ehdb-l0 is now consumed in-process by the NoETL server, via tag = "v0.2.0" — the first git-tag-semver dependency on this repo, replacing the rev pins that let two consumers drift 70 commits apart.

The server opens an L0Engine<D1EventLog> in shadow: it appends what Postgres accepted and compares counts. It serves nothing from the engine.

What this exercises in this repo, on a real workload:

  • L0Engine::open and, with it, the FORMAT_VERSION gate — this is the open the gate was written for;
  • append_record on D1EventLog from a second, non-writer process;
  • LocalFsSubstrate under a server's write rate.

Measured in kind: engine opened once, 24 batches compared, 0 divergences, 0 append failures, and the server's own dispatch unaffected.

On prod the flag is off, so the engine is not opened there yet. Arming it is a separate checkpoint.

⚠ Consequence for this repo: a breaking change to the on-disk layout now has a real consumer. Bump FORMAT_VERSION in the same change, and cut a new tag — the server pins tags, not revs, so it will not pick the change up silently.

Related

EHDB

Programs

Kind validation

Query interface

Runbooks

Design notes

GitHub

Clone this wiki locally