North Star Distributed Registry

North star: EHDB as NoETL's distributed registry

"etcd, but truly distributed" — plus an append log, catalog relations, a topology / service registry with discovery, secret references, and vectors for AI.

Spec: design/north-star-distributed-registry.md (noetl/ehdb#384) · anchor issue noetl/ai-meta#455.

⚠ The inventory corrected three prior claims

A spec built on a bad inventory specifies work that is already done, so every "exists" was checked against origin/main rather than recalled — and three of my own notes were wrong:

• D8 RuntimeDataset is not "implemented and never wired" — it is in the production ehdb-l0 crate, exported from lib.rs, with register/heartbeat/deregister/get/ list_live/list_live_since already present.
• Vectors are not a greenfield hard part — cosine in ehdb-l0, cosine_similarity plus a cosine-ranked upsert/query in ehdb-reference, and ehdb-retrieval has its own.
• Four of the seven capabilities are substantially built.

capability	state
Append log	✅ done, and now measured
Catalog relations	✅ done — noetl/catalog on EHDB
Service registry	✅ largely done — D8
Vectors	✅ arithmetic exists; integration does not
Gossip membership	⚠ crate exists; foca chosen
Replication	⚠ sealed parts N-way; unsealed tail is RF=1
Leases / TTL	✅ P1 landed (was a heartbeat counter)
Watch / subscribe	❌ zero pub fn watch/subscribe in the workspace
Ephemeral runtime units	❌ D8 is worker-shaped
Secret references	⚠ no reference type

The honest distributed lower bound

A coordination-free data path gives CALM-style availability and not linearizable cross-shard reads. "etcd semantics, distributed better" is wrong. The accurate claim: per-shard ordered, coordination-free, with explicitly-typed staleness at the read boundary — weaker than etcd in places, traded for no external service and no quorum to operate. ehdb-core's plan layer already models nearest vs bounded and refuses rather than silently serving from a non-owner.

Phases

phase	work	risk
P1	wall-clock TTL liveness on D8	low — ✅ done
P2	generic (kind, id) registration	low–med
P3	watch(kind, after_seq) + stated latency	low
P4	ephemeral execution registration with TTL	low
P5	secret_ref type + refuse-material validation	low
P6	vectors on catalog objects + recall@k	med
P7	unsealed-tail replication	high
P8	writer election + fencing on the production path	high

P7 and P8 are the hard parts — not vectors, not the registry. Both change durability or safety semantics and want their own spec plus prod sign-off.

P6, landed — and measuring first changed the deliverable

ehdb#390.

The spec asked for "an honest recall@k number, because semantic search works is unfalsifiable without one". VectorStore::top_k turns out to be an exact brute-force scan — every live point, cosine over all, sort, truncate, no approximation. So:

recall@k is 1.0 by construction, and a reported 1.0 is not evidence that search works — it is evidence that an exhaustive scan was exhaustive.

Recall becomes falsifiable when an ANN index exists and not before. It is therefore kept as a correctness guard with its own control: a complete candidate set scores 1.000, a damaged one 0.333, and an empty expected set is undefined rather than 1.0 — otherwise a known-answer set that failed to load reports a perfect score.

⚠ The first known-answer set was silently degenerate — 64 points over 16 dimensions, so points with axis >= 16 had no dominant component and the query vector was the zero vector. recall@1 read 0.0 and looked like an engine defect.

⭐⭐ The real finding: query cost is O(ops), not O(live points)

collection	ops	live points	top_k
5,000 written once	5,000	5,000	66.1 ms
500 re-embedded 10x	5,000	500	64.9 ms

Within 2% — so re-embedding a collection is as expensive as growing it (378x for the same 500 live points returning the same answer). Dimension is linear (exponent 0.90).

⚠⚠ Compaction cannot fix it

Measured: after a merge actually ran, the 10x collection went 64.8 → 70.5 ms, worse. Verified in code: live_points reads every op ever written then folds latest-wins; VectorDataset has no dedupe_key; and the Dataset trait exposes no supersede or compaction hook at all. Engine merge combines parts, not keys, so it structurally cannot drop a superseded op.

The prerequisite for vectors at scale is a key-level compaction primitive, not an approximate index — and it is broader than vectors: RuntimeStore's D8 folds and the catalog's attribute/relation folds share the same pattern. ehdb#391.

The catalog attachment needs no new dataset

collection is already the partition and the index dimension, and point_id is free-form, so (resource_type, path) maps onto (collection, point_id). Pinned by set equality so cross-type leakage fails; a path containing / round-trips, because a point_id is a payload field, not a substrate key — unlike the runtime registry's deliberately narrow id charset.

P1–P5, landed

phase	what shipped	where
P1	wall-clock TTL liveness on D8 (list_live_at, liveness_coverage)	ehdb#384
P2	RuntimeKind {Server, Gateway, Ehdb, Worker, Playbook, Execution}, register_kind, discover(kind, now, ttl)	ehdb#385
P3	watch_since(after_op_seq) -> (ops, cursor)	ehdb#385
P4	ephemeral unit registration — an Execution registers with a short TTL and vanishes by not being renewed: no tombstone, no reaper	ehdb#385
P5	SecretRef = provider://path[#version], a pointer only — the type never holds a value, so its Display cannot render one	ehdb#385

Discovery by kind is the point of P2. Without it, "where are the gateways" means taking a fleet-wide list and inferring kind from the shape of an id — a naming convention masquerading as a schema.

P3's watch is poll-based and named as such. A consumer's latency is its poll interval. But it is a watch in the property that matters: resuming from the cursor returns nothing when nothing changed. A "watch" that re-delivers the whole log every poll is a scan, and the two look identical in any test that only polls from 0. Departures are delivered too.

⚠⚠ A real bug P2's tests caught

heartbeat reset the kind to Worker, because append_at defaults it. A renewed Execution silently stopped being discoverable as an Execution one heartbeat after registration — a registration that disappears while being actively renewed. The kind is now carried forward on both heartbeat paths, as the contract already was.

⚠ The compatibility asymmetry, stated

RuntimeOp carries #[serde(deny_unknown_fields)], and this repo already documents the consequence in ehdb-stream/tests/record_envelope_additivity.rs: a reader built before a field was added rejects a record carrying it rather than ignoring it.

So adding a field is backward compatible and forward INCOMPATIBLE → an upgrade ordering requirement: readers before writers. Pinned by the_compatibility_asymmetry_is_explicit.

⚠ The id charset is deliberately narrow

[A-Za-z0-9-_.:]. The id becomes a substrate key and an index dimension, so a path-shaped identity must be encoded by the caller (: is permitted). Widening it was considered and rejected: a substrate key containing / is a directory traversal waiting to happen.

P1, landed

list_live_at(now, ttl) replaces the judgement list_live_since(min_heartbeat) pushed onto the caller in units of an opaque monotonic counter. register_at/heartbeat_at stamp last_seen_micros, and the clock is a parameter, not ambient — a registry whose liveness depends on an ambient clock cannot be tested without sleeping.

⚠⚠ The upgrade hazard, pinned by a test. A record written before the field decodes with last_seen_micros = 0. Treating that as expired would make the first list_live_at after an upgrade evict the entire fleet — a total outage caused by a liveness improvement. So 0 is unknown, not dead, and liveness_coverage reports how much of the answer rests on such records: an answer resting on incomplete information must be visible, not inferred.

5 mutants, 5 caught — including <= for < (the off-by-one between a stable registry and a flapping one) and treating unknown as dead.

Scope boundary

Internal EHDB — self-sufficient, no external datastore, ever. The business catalog (travel's domain data in Firestore) is explicitly not this.