off-box: B2 VERIFIED on prod — the age trigger fires, chain completes
Replaces 'configured' with measured. NOETL_EHDB_SEAL_MAX_AGE_SECS=900 on
v3.140.1: tail climbed 95->815s, seal logged at 21:30:42, 894s after the
first append against a configured 900s — inside one 60s tick, which is
what makes it unambiguously the AGE trigger rather than a record/byte
limit. The counter is the weakest of the six signals; the timing is the
proof of mechanism.
Records all six signals and what each one rules out, because they fail
differently: a counter that increments without sealing, a seal that does
not close the durability window, a sealed part that never enters the
manifest, a part that exists only in memory, an active file not actually
promoted. Traced the part on disk in both copies at 682,772 B — exactly
what the active file had reached.
Context worth keeping: the newest sealed part before the fire was from
19:10, so the tail had been accumulating over two hours; B2 bounded it to
15 minutes and manifest_parts moved +1, not a flood.
⚠ Names the identity unsealed_age == unreplicated_age as CORRECT at
RF=1-local, and that the two MUST diverge once a GCS replica exists — if
they do not, the remote upload is untracked and the window gauge lies.
Also documents ehdb#396: 10 .active files / 11.2 MiB stranded on prod,
oldest from Sep 24, because collect_eslog_files filters on .eslog and
nothing deletes a .active. Direct evidence of the tail-loss shape, and it
inflates du-based store measurements.
off-box replication and the unsealed tail (A2 + B2)
New page covering server#518/#519/#520 in the server's own terms:
- the precise durability statement — sealed parts survive node loss, the
unsealed tail does not, replication is async, so recovery is a
bounded-loss consistent prefix and NOT 'no data loss'.
- why GcsSubstrate lives in the server rather than ehdb-l0, and the
consequence: ehdb-core is now a third pin that must move with ehdb-l0.
- the failure-domain algebra, including that two GCS buckets are ONE
domain and that domains_distinct — not single_point_of_failure — is the
gauge that would catch a two-bucket plan.
- the put_if_absent atomicity and get_range short-read contracts, and why
each is load-bearing.
- B2: the mechanism existed in ehdb and was unreachable from the server.
Both halves must ship together, quoting ehdb's own warning about an
inert trigger, and the manifest-parts counterweight exists because
smaller parts trade a durability risk for a capacity one.
- a rollout that measures prod's existing exposure BEFORE enabling
anything, and warns that a quiet age_sealed_total is not health.
Plus the two B2 env rows on the deployment spec (Rule 2a) and sidebar +
Home links.