off box replication and sealing

Off-box replication and the unsealed tail

How the embedded event log stops being a single point of failure, what that does not cover, and the gauges that say which state it is in.

Tracked by noetl/ai-meta#460 (A2, B2). Landed in server#518, #519, #520.

⚠⚠ The durability statement, stated precisely

The imprecise version of this is dangerous, so here it is in full:

sealed parts	uploaded to every configured replica ⇒ recoverable after node loss
the unsealed tail	RF=1. Records appended since the last seal exist only in the local writer. Losing the node loses exactly them.
timing	asynchronous. register_and_upload hands an UploadJob to a background channel and the append returns, so even a sealed part has a window where the only copy is local.

Recovery from node loss therefore yields a bounded-loss consistent prefix of the log. It is not "no data loss". Do not write that anywhere.

B2 is what bounds the loss; B3 (a streaming tail replica, not built) is what would remove it.

The asynchrony is not laziness. Measured in-region GCS put latency is p50 77 ms, p99 234 ms against a ~4 ms local fsync — roughly 19x. A synchronous remote write on the append path would dominate it, which is what #460 decision (i) turned on.

A2 — GcsSubstrate

src/services/gcs_substrate.rs implements ehdb's DurableSubstrate over the server's GCS client. Configured, the engine opens N-way (L0Engine::open_replicated) with replica-0 = the local PVC and replica-1-gcs = the bucket.

Why it is in the server and not in ehdb-l0

ehdb-l0 has no HTTP client and a deliberately lean dependency set (serde, ehdb-core, twox-hash). Adding reqwest to the storage core to reach one backend is weight on every consumer. DurableSubstrate is the extension point for precisely this.

⚠ Consequence: ehdb-core is now a third direct pin that must move with ehdb-l0. The trait's signatures are written in its Result/EhdbError, and ehdb-l0 does not re-export them. A different tag puts two EhdbError types in the graph and the impl silently stops satisfying the trait.

The failure domain is the whole point

failure_domain() returns Remote { provider: "gcs", bucket }. That declaration is what makes survives_node_loss true and what lets validate_replica_domains accept the set.

• ⚠ Declaring it wrongly is worse than not declaring it. Undeclared is refused by the replica-domain check; a false Remote is accepted and makes the single-point-of-failure gauge read 0 on a store that still has one copy.
• ⚠⚠ Two GCS buckets are ONE domain. label() collapses Remote to its provider, so a provider outage takes both. They do still survives_node_loss (both are off-node), and is_single_point_of_failure() is defined as !survives_node_loss — so the gauge that would catch a "spread across two buckets" plan is domains_distinct, not the single-point-of-failure one. Reading redundancy off the wrong gauge is how a correlated replica set reads as safe.

Two primitives with load-bearing contracts

• put_if_absent — ifGenerationMatch=0, so the existence check and the write are one atomic operation. An exists()-then-put() pair is a race, and this contract is what makes a part's bytes immutable once written. HTTP 412 is the "already there" signal, not an error, and carries its own pinned metric outcome already_present.
• get_range — refuses a short read rather than returning it. The manifest states every part's length, so fewer bytes means the store and the manifest disagree, and returning them would let a truncated object read as intact. A 200 instead of 206 (the server ignored the Range header) fails the same check. len == 0 is answered without a request, since bytes=o-(o-1) is malformed.

Degradation is measured, not assumed

If the replica is configured and cannot be opened, the engine still opens at RF=1. Refusing would take the shadow down on a storage problem, which the embedded-shadow design forbids outright. That is acceptable only because the A1 evaluation runs over the replica set that actually exists (replica_reality::evaluate_and_publish_set), so the gauges report the single point of failure including when redundancy was asked for and failed. A silent fallback with the gauges still reading "redundant" is the drift A1 was built to catch.

How it is proved

Unit tests cover the domain algebra — which a substrate that declared Remote while dropping every upload would also pass. The real proof is tests/gcs_substrate_roundtrip.rs (#[ignore], needs a fake-gcs-server): it cold-loads the log from the GCS replica alone, fresh local root, original local substrate not passed in.

Verified 2026-10-09, and confirmed by listing the bucket from outside the test: 6 sealed part objects spanning sequences 1-48, the FORMAT_VERSION guard, and the manifest chain — 14 objects / 26,159 bytes, with the control bucket at 0.

⚠ Controls matter here because the positive result has innocent explanations that look identical to success: a cold-load that read a local directory, or an assert_eq! between two empty vectors. The fixture asserts 48 records before any comparison, and with_seal_max_records(8) makes sealing actually happen — a test that never seals proves nothing while passing.

B2 — the age-based seal trigger

seal_max_age and seal_aged_parts() have existed in ehdb-l0 for some time. Until server#520 the server configured neither and called the sealer from nowhere: it had no caller outside ehdb's own tests. The mechanism existed and was unreachable.

⚠⚠ ehdb's own doc comment names the trap, and it is worth quoting because it is the reason both halves must ship together:

PartWriter::should_seal is only consulted on append, and the shard the age trigger protects is by definition the one taking no appends. Setting seal_max_age without driving this on a timer leaves the trigger inert on exactly the shard it was added for — the flag would be present, the config would look correct, and nothing would ever fire.

So tests/age_seal_wiring.rs::the_trigger_has_both_halves asserts four things in one build: the config call, the sealer call, the task, and the spawn in main.rs.

✅ Verified on prod, 2026-10-09 — the trigger fires and the whole chain completes

Not "configured and quiet". NOETL_EHDB_SEAL_MAX_AGE_SECS=900 on server v3.140.1:

21:17:21  age_sealed=0  unsealed_age=95   unrep_recs=174  parts=321
21:20:22  age_sealed=0  unsealed_age=275  unrep_recs=174  parts=321
21:23:22  age_sealed=0  unsealed_age=455  unrep_recs=174  parts=321
21:26:23  age_sealed=0  unsealed_age=635  unrep_recs=174  parts=321
21:29:23  age_sealed=0  unsealed_age=815  unrep_recs=174  parts=321
21:32:24  age_sealed=1  unsealed_age=59   unrep_recs=0    parts=322   <- FIRED

2026-10-09T21:30:42Z INFO noetl_server::ehdb_embedded:
  age trigger sealed aged parts; they are now eligible for off-box replication  sealed=1

⭐ The timing is the proof of mechanism, not the counter. First append was ~21:15:48; the seal landed at 21:30:42 — 894 s against a configured 900 s, inside one 60 s tick. A record- or byte-limit seal would not land on that boundary, so this is unambiguously the age trigger.

Each signal rules out a different false green, which is why all six are worth reading:

signal	before → after	rules out
age_sealed_total	0 → 1	the inert trigger
oldest_unsealed_age_seconds	815 → 59	a counter that increments without sealing
unreplicated_records	174 → 0	a seal that does not close the durability window
manifest_parts	321 → 322	a sealed part that never enters the manifest (an orphan)
.eslog files	654 → 656	the part existing only in memory (+2 = both copies)
.active files	10 → 9	the active file not actually being promoted

Traced on disk, the same part in both copies:

local/…/shard-0-seq-00367057411841728512-00367057697004068864.eslog   682,772 B  21:16
substrate/…/shard-0-seq-00367057411841728512-00367057697004068864.eslog 682,772 B  21:30

682,772 B is exactly what part-000000.active had reached, and the substrate copy is stamped at the seal.

Why this mattered here: the newest sealed part before the trigger fired was from 19:10 — the active part had been accumulating for over two hours. B2 bounded that to 15 minutes, and manifest_parts moved +1, not a flood, so the counterweight held.

⚠ Note the identity oldest_unsealed_age_seconds == unreplicated_oldest_age_seconds throughout the climb. That is correct at RF=1-local — the durability window closes as fast as a local file write, so it equals the unsealed age. Once a GCS replica is configured the two must diverge; if they do not, the remote upload is not being tracked and the window gauge is lying.

⚠⚠ Orphaned .active files leak — ehdb#396

Measured on the same volume: 10 .active files, 11.2 MiB, oldest from Sep 24. Nine are orphans — records appended, acknowledged, written to the local active file, then never sealed and never entered the manifest because the process restarted first.

reclaim_orphans → collect_eslog_files filters on extension == "eslog", and nothing anywhere deletes a .active file, so they accumulate one per restart, forever. Not referenced by the manifest, so no read path reaches them either.

This is direct measured evidence of the tail-loss shape: B2 reduces it (a shorter window means less in flight at restart) and B3 would close it. ⚠ It also inflates any du-based measurement of the store, which matters when a disk-reclaim number is being taken.

Checked and not broken, so the finding is not overstated: the live active part is recovered rather than clobbered (recover_active() replays intact frames and appends, truncating only a torn tail), and the manifest tolerates out-of-order parts (prune() sorts by min_sequence).

The gauges publish unconditionally

⚠ Including when the trigger is off. The unsealed tail is an RF=1 exposure whether or not anything is configured to bound it, and a gauge that appeared only once the fix was enabled would be invisible in exactly the configuration that has it.

noetl_ehdb_oldest_unsealed_age_seconds   the size of the RF=1 window
noetl_ehdb_age_sealed_total              age-caused seals; 0 while the age gauge climbs
                                         is the INERT-TRIGGER signature
noetl_ehdb_manifest_parts                the COUNTERWEIGHT

⭐ The counterweight is deliberate. B2 makes parts smaller and more numerous, and manifest cost grows with part count — the shape that filled the cmdbus PVC, where snapshot size tracked part count while snapshot count tracked write count. Shipping the trigger without a parts gauge would trade an invisible durability risk for an invisible capacity one, which is not an improvement.

NOETL_EHDB_SEAL_MAX_AGE_SECS=0 is refused, not clamped: it would seal on every tick and grow the manifest without bound, and clamping would hide the misconfiguration.

Environment variables

The deployment-spec page is the source of truth; see deployment-specification for the full catalogue. In brief:

NOETL_EHDB_REPLICA_GCS_BUCKET     unset => RF=1 (prod today)
NOETL_EHDB_REPLICA_GCS_PREFIX     key prefix, normalised to one trailing /
NOETL_EHDB_REPLICA_GCS_ENDPOINT   real GCS or an emulator
NOETL_EHDB_SEAL_MAX_AGE_SECS      unset => trigger off; 0 refused
NOETL_EHDB_SEAL_TICK_SECS         60

Rolling it out

1. Create the replica bucket and grant the server SA roles/storage.objectAdmin on it. ⚠ It has no binding there; the replica fails to open until granted, and the gauges will say RF=1.
2. Deploy. With the vars unset this is byte-for-byte the current path, and the B2 gauges start reporting prod's existing unsealed-tail exposure before anything is enabled — measure that first.
3. Set NOETL_EHDB_SEAL_MAX_AGE_SECS and watch age_sealed_total move while manifest_parts stays sane. ⚠ If age_sealed_total stays 0 while oldest_unsealed_age_seconds climbs, the trigger is inert — do not interpret the quiet as health.
4. Only then set NOETL_EHDB_REPLICA_GCS_BUCKET and confirm survives_node_loss flips to 1 and domains_distinct to 1. ⚠ Confirm on the gauges read off the running pod, not from the configuration.

Related

• Embedded-EHDB-Shadow — the engine this replicates.
• event-log-retention-archival — the archive tier is a different thing: a copy for retrieval after pruning, not a replica for durability.
• deployment-specification — env-var source of truth.