Invariants + architecture: the transition semantics and the layer/role axes
noetl/ehdb#323. Both pages existed; both were missing the half the issue asks
for, and it is the half the cutover questions turn on.
Consistency-Invariants gains §5 — what a reader observes during a
shadow -> primary transition:
* the three states, and why the middle one misleads — a shadow tier is fully
written and fully compared and still serves nothing, so append rate and store
size say the mirror is alive and say nothing about whether a flip is correct
* what is dual-written, re-measured on prod 2026-09-16: MINT_AUTHORITATIVE=true,
STORE_DUAL_WRITE=false, PROJECTION_READ_SOURCE=wal, SERVE_ON_BEHIND=true
(so projection reads are NOT read-your-writes), OBJECT_STORE_BACKEND=gcs
* that the NOETL_EHDB_<TIER> mode variables are not set on the prod pods at all,
so live modes come from code defaults rather than any manifest
* what parity checks and what it deliberately does not: three DIVERGENCE_KINDS;
superseded/unmirrored are observations, not divergences; arrival order is
counted rather than judged (#346 — the first definition of "different" was
wrong and produced 8-of-74 false divergence)
* the recovery ladder spine -> tier -> postgres, and why the 2026-09-16 audit
finding (every event-log read still has Postgres authority behind it) is
exactly what a flip would spend
* a four-question checklist to run before any flip
Architecture-Four-Engines gains the two axes it was missing:
* the L0..L3 layer stack, marked for what is CODE vs what is PLAN — only L0 is
a crate, L1 is the feed, L2/L3 are plan
* node roles as DEPLOYMENT facts, stated as such because there is no NodeRole
type in the codebase, with the write role's PVC + pinned digest called out as
the reason writer-ordered rollouts are gated
Documentation only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wiki: record the F1-F5 remediation and refresh the invariants
The Consistency-Invariants page said 'detected by nothing' for two invariants
that now have detectors, which is exactly the drift that page exists to catch.
I-EL-2 (single writer) is now DETECTABLE but still not enforced: fencing runs in
shadow, counting stale-epoch writes while letting them through, and the election
issues tokens without being authoritative. The ordering hazard is recorded on
the page itself -- enforcing before the election issues real tokens is an outage
rather than a degradation, because with no election every writer's epoch is 0.
I-EL-4 (reaching the substrate) now names ehdb_l0_unreplicated_age_seconds as
its detector, live on the writer's /metrics since worker v5.125.0, and keeps the
two misleading readings explicitly labelled: upload_lag_micros_total is
seal-relative and blind to the dominant term, and ehdb_feed_shard_lag is consumer
backlog wearing an adjacent name.
Also records that prod as it stands would FAIL the new replica-domain check --
the tier dir is nested inside the writer dir on one PVC -- so validating at open
before fixing the layout is a startup outage by construction.
Home carries the summary above the fold; Sessions-Log gets the dated entry.
Refs noetl/ehdb#324, noetl/ehdb#328, noetl/ehdb#329, noetl/ehdb#330, noetl/ehdb#331, noetl/ehdb#332
wiki: the four-engine architecture + per-tier consistency invariants
ehdb#323, under #324. Two new pages, plus Home and the sidebar reconciled to
the narrowed scope.
Architecture-Four-Engines documents the four owned engines with a diagram, and
records WHY vector and OLAP collapse into projections -- matching what is built
rather than what was promised: ehdb-reference/src/vector.rs is already a bounded
cosine search over the collection's live points, so there is NO ANN index to
remove. The gap was in the promise. It also separates three vocabularies that
are routinely conflated: engine vs tier vs StoreTier/QueryTier.
⚠⚠ Both pages surface a discrepancy the Backend-Configuration page invites:
it presents five tiers each with an off/shadow/primary mode, which reads as
though setting primary makes any of them serve. SERVE_WIRED_TIERS is
["eventlog", "projection"] -- KV and object accept primary as CONFIGURATION and
cannot serve as CAPABILITY. Home now says so above the fold.
Consistency-Invariants states, per tier, what each guarantee is, WHAT FORCES IT
to hold, and HOW YOU WOULD FIND OUT if it stopped -- because a guarantee with no
enforcement and no detector is a description, not an invariant. Three come out
badly and are labelled rather than smoothed over: I-EL-2 (single writer) is
enforced by nothing stronger than replicas: 1 and detected by nothing;
I-EL-4 (reaching the substrate) is bounded by volume not time and detected by
nothing that works, since upload lag is measured from seal and feed lag is
consumer backlog; and in prod the substrate is the SAME PVC as the primary, so
it buys no independent failure domain.
Home's mission no longer claims EHDB will absorb Qdrant and ClickHouse.
Refs noetl/ehdb#323, noetl/ehdb#324, noetl/ehdb#320, noetl/ehdb#321, noetl/ehdb#322