Repository navigation
Embedded EHDB Shadow
Since v3.106.0 (noetl/ai-meta#332 step 5). Deployed to prod 2026-09-08 with the flag off.
The server opens an ehdb-l0 event-log engine in-process and appends to it
in parallel with the authoritative Postgres write, then compares the counts. It
serves nothing — project_events returns its rows unchanged, Postgres
remains the system of record, and no read or write is answered from the embedded
engine.
Flipping the serve path is a separate, later checkpoint.
| var | default | meaning |
|---|---|---|
NOETL_EHDB_EMBEDDED |
false |
arms the shadow. ⚠ Strict == "true" — 1, yes, on, TRUE do not arm it |
NOETL_EHDB_EMBEDDED_DIR |
/data/ehdb-embedded |
local root; needs a mounted volume in prod |
⚠ The strictness is deliberate: a flag that accepts many spellings is one nobody can read off a manifest with confidence.
With the flag off the engine is never opened — the OnceLock is not initialised,
no directory is touched, and the module costs nothing. That is what makes the
release inert on arrival and reversible by configuration rather than redeploy.
noetl_ehdb_embedded_shadow_total{outcome} — a closed, pinned label set:
opened · open_failed · agreed · diverged · skipped · append_failed.
⚠ Pinned at 0 in init_embedded_shadow_series() because absent and zero are
different answers: an unpinned metric is missing until it first fires, so "the
shadow never ran" would look identical to "the exporter is broken".
⚠ skipped is distinct from agreed on purpose. A shadow reporting
agreement when it compared nothing is a clean result computed over zero rows —
which reads exactly like a healthy one.
-
Shadow errors are recorded and swallowed. A shadow that can fail a real
write is a liability, not evidence.
append_failedis how a silently-degraded shadow stays visible; a bare swallow here would repeat theserve_ingestmistake that hid a full disk for hours. -
Rows are serialised with
EventRow::to_stream_json— the same function the networked publish path uses. A shadow storing a different shape would prove nothing about the engine's fitness to replace that path. -
A poisoned lock is recovered with
into_inner(), never unwrapped. The shadow must not be able to panic the write path.
ehdb-l0 and ehdb-feed come from tag = "v0.2.0", replacing rev pins that let
two consumers drift 70 commits apart on the same engine. Tolerable while they
talked over a wire protocol; a corruption vector once both open one on-disk
layout. ehdb-l0's FORMAT_VERSION gate is the belt to this braces and fires at
exactly this open.
Kind, flag on: opened=1, agreed=24, diverged=0, append_failed=0, and a
dispatched execution still COMPLETED in ~5 s — serving unaffected.
Prod, flag off (v3.106.0): all six series at 0 (so the engine was never
opened), NOETL_EHDB_EMBEDDED unset, dispatch COMPLETED, 0 ERROR lines, 0
unexpected restarts.
Prod, flag ON — armed 2026-09-08 (shastaratech-noetl-prod, ns noetl,
server v3.106.0). Env-only change, applied after a full-spec diff whose entire
substance was two lines:
- generation: 110
+ generation: 111
+ - name: NOETL_EHDB_EMBEDDED
+ value: "true"
The image digest sat on an unchanged line. Post-apply the whole live object was
re-diffed against the before-snapshot: the only spec change was
env: len 60 -> 61; volumes: ['sm-secrets'] unchanged, image identical,
rollout clean with 0 restarts and 0 ERROR lines.
⚠ The engine opens lazily, on the first shadow_append — so a freshly
rolled pod reports all six series at 0 until prod projects an event. That zero
is "not yet exercised", not "broken", and the two are indistinguishable on
a scrape without knowing this. It is the same absent-vs-zero trap the pinned
series exist to defuse, one level up: pinning makes the series present, but
it cannot make the engine opened.
Deployed by digest (sha256:e9b11d97…, AR, linux/amd64). The full-object
kubectl diff --server-side before the apply had exactly one spec
difference — the image digest; volumes, mounts, containers, resources,
probes, replicas and all 61 env vars identical, no PVC to be missing, and
pod-template annotations unchanged, so the roll was driven by the image alone.
The first hourly execution after the roll exercised it:
| signal | value |
|---|---|
opened |
1 |
agreed |
12 batches |
diverged |
0 |
append_failed |
0 |
du -sk /data/ehdb-embedded |
44 KB, flat (0% of the 1 GiB quota) |
| serving | 0 restarts, 0 ERROR, 0 panic, execution terminal in ~5 s |
⚠ In that same window publish_skipped{system_execution} was 22. Those are
precisely the events the old call site could not see, and the shadow recorded 12
agreed batches over them — the fix, demonstrated rather than asserted.
The directory the previous build never created now exists, and its contents confirm two things at once:
/data/ehdb-embedded/substrate/FORMAT_VERSION <- the format gate ran and passed at open
/data/ehdb-embedded/local/parts/d1_event_log
This page previously said du would read ~0 for days. It read 44 KB
immediately, then stayed flat. Both observations are consistent, and the
distinction is the useful part:
- 44 KB is the engine's fixed structure — format marker, manifest, directories — written once at open.
-
Part data is what waits for
should_seal()(1024 records / 8 MiB).
So the healthy signature is nonzero but flat, and real growth would appear as a step roughly every two days, not a ramp. A reader expecting a literal zero would have mistaken a correct 44 KB for a problem; a reader expecting a ramp would miss a step.
Rollback remains one command — set image back to the previous digest.
noetl/server#420, 2026-09-08.
shadow_append now sits in handlers::event_write::emit_events, beside
mirror_rows, and inherits that call site's two properties verbatim:
-
after terminal dedup and chain stamping ⇒
rowsis exactly the set that becomes authoritative; - before the publish/insert fork ⇒ one call site covers both sides of the CQRS gate, so the gate-off branch cannot rot separately.
It was removed from project_events, not duplicated. With the gate on, an
event passes emit_events and then reaches the materializer; two hooks would
append it twice and report a divergence the engine did not cause — a
self-inflicted finding on the one signal the shadow exists to provide.
emit_events needs an AppState and a live pool, so it cannot be driven from a
unit test. The tempting alternative — call shadow_append directly and assert
it appends — passes whether or not anything on the write path calls it. That is
the mistake this whole page documents, one level up. So four guards read the
call site, and each was proven to fail when it should:
| mutation | caught by |
|---|---|
delete the call from emit_events
|
all four guards |
| move it past the publish/insert fork | the ordering guard |
give shadow_append a return type |
the serving-path guard |
re-add the second hook in project_events
|
the one-call-site guard |
Zero compile errors across the set, so each result is a real verdict rather than a build failure misread as a catch. Every guard asserts its own extraction first — body size plus the presence of the fork — because a slice that came back empty would make all of them pass over zero bytes.
⚠ One guard was wrong on first write: it counted 6 call sites, matching its own assertion strings. It now reads only the non-test region. A guard that matches its own text is the recurring failure on this repo, and it recurred here.
should_seal() fires at 1024 records or 8 MiB. At prod's ~550 events/day
the shadow touches disk roughly every two days (~2 MB per part) and buffers
in memory until then, with manifest retain capped at 32.
So du legitimately reads ~0 for days while agreed climbs. An empty
directory is not evidence the shadow is idle — judge reachability by opened=1
and agreed > 0, never by the directory.
⚠⚠ And df is the wrong instrument entirely: it reports the node's
overlay (94.3 GB, 12% used), not the pod's 1 GiB ephemeral-storage quota,
which the kubelet enforces against the container's writable layer. df would
report ~83 GB free with the pod one byte from eviction. The gate measures
du -sk /data/ehdb-embedded.
For scale: the cluster-autoscaler evicted the armed pod 41 minutes after it started, as routine node consolidation — so the ephemeral layer is reset far more often than a 1 GiB budget could ever fill at this rate.
Not a defect. The window contained no user traffic, and only user traffic reaches the shadow.
Over 14 samples across 25 minutes — including one complete hourly execution —
every one of the six noetl_ehdb_embedded_shadow_total series stayed at 0, and
so did noetl_events_projected_total. The metric built for exactly this
question settles it:
noetl_event_ingest_publish_skipped_total{reason="gate_off"} 0
noetl_event_ingest_publish_skipped_total{reason="no_transport"} 0
noetl_event_ingest_publish_skipped_total{reason="system_execution"} 22
noetl_event_ingest_published_total ABSENT
The chain: NOETL_EVENT_INGEST_PUBLISH_ONLY=ON, so emit_events normally
publishes and project_events — where shadow_append lives — becomes the sole
writer. But should_publish excludes system executions by design
(is_system_execution: any catalog path under system/), and prod's hourly
scheduled job is system/orchestrate. Those events fall through to the
synchronous INSERT, never reach the materializer, and so never reach the
shadow.
So the shadow is wired correctly. It fires for user executions; the overnight window simply had none.
⚠ Do not "fix" this by moving the hook to emit_events. That would capture
system traffic too — traffic the CQRS design deliberately keeps off the
materializer — and would make the shadow diverge from the writer it is meant to
mirror. The hook is where it belongs; what is missing is a user execution.
agreed=0, diverged=0, append_failed=0 is byte-identical to what a healthy
shadow reports on a quiet system. opened is the only series that separates
"never ran" from "ran and agreed" — and it is easy to read past, because five
of the six sit at 0 in the healthy case too.
Two intermediate diagnoses were wrong before the metric settled it: first "the
gate is off" (it is on — gate_off = 0 proves it), then "the hook is on the
wrong writer" (it is on the right one). Both were plausible readings of a
correct observation, and both would have produced a confident wrong writeup.
publish_skipped_total{reason} exists precisely because the publish counter
alone cannot distinguish these cases — its own doc comment says so, and it was
right.
Any agreement claim from prod must publish its denominator. Here the denominator is zero appends, and nothing about engine agreement follows from it either way.
Off-prod, 2026-09-09, against ghcr.io/noetl/server:3.106.1. Prod untouched.
A 2-replica StatefulSet with NOETL_SHARD_INDEX_FROM_HOSTNAME, so each pod
derived its own index (0 and 1) from its ordinal. Six execution ids were sent to
both shards:
| shard 0 | shard 1 | |
|---|---|---|
owned_local |
2 | 4 |
forwarded_ok |
4 | 2 |
forwarded_terminus |
2 | 4 |
That solves to shard 0 owning 2 ids and shard 1 owning 4 — and the two pods
agree independently: shard 0's forwarded_ok equals shard 1's
forwarded_terminus, and vice versa. Every non-owner forwarded; every forward
terminated at the owner in one hop.
⚠ Ownership was deliberately not predicted by reimplementing XxHash64 in
the test harness. A wrong replica of the hash yields a confident wrong test. The
invariant — exactly one shard claims each id, and the other forwards to it —
is checkable without knowing the answer in advance, and it makes the two
processes cross-check each other.
Shard 1 was scaled away and the same six ids replayed at shard 0:
- 4 × HTTP 503, 2 × HTTP 200 — the same 2/4 split, derived by a wholly different method than the metrics above;
-
noetl_execution_affinity_total{outcome="forward_unavailable"} 4; -
WARN execution-affinity: owner unreachable; FAILING CLOSED.
No orphaned id was processed locally. That is the property preventing a failed forward from forking the owning shard's event log.
With a volumeClaimTemplate and the engine at the prod default path
/data/ehdb-embedded, now on a real device rather than the overlay:
- 40 events →
substrate/FORMAT_VERSIONand an active part written; - the pod was deleted, simulating the node consolidation that wipes ephemeral storage;
- md5 of every file identical afterwards, same PV rebound;
- the engine reopened on pre-existing data — the
FORMAT_VERSIONgate passed against state it had not created — and a second round gaveopened=1 agreed=40 diverged=0 append_failed=0.
Step 4 is the one that matters for serving: recovery, not merely persistence.
The prod server is a Deployment with maxSurge: 1, maxUnavailable: 0. Both
obvious options are traps:
- RWO PVC — the new pod starts before the old terminates and a ReadWriteOnce disk cannot be mounted twice, so every rollout deadlocks until the progress deadline fails it. Reads fine, cannot schedule.
- RWX PVC — removes the deadlock precisely by letting both pods mount at once, which puts two single-writer engines on one directory during every rollout. This is the tempting option (smallest diff, keeps zero-downtime rolls) and it is the wrong one.
The target shape is therefore a StatefulSet with a volumeClaimTemplate, staged
inert in noetl/ops#303 and not
applied. Mounting a volume does not make the engine authoritative; the serve
flip remains a separate, gated change.
noetl/server#423 merged as 06fcfe70;
semantic-release cut v3.108.0 (minor, from feat:). AR digest
sha256:867b822f838bd54a9bc13069954f3ae337c836aa497759ba9a624574f6111daf,
GHCR ghcr.io/noetl/server:3.108.0.
Prod was not deployed. It remains on v3.107.1 and the flag is not in its env.
wal_projection_state now intercepts StoredBehindSpine behind
NOETL_EHDB_PROJECTION_SERVE_ON_BEHIND (default off, strict == "true") and
serves the stored body when ServeGrant::evaluate issues a grant. The forward fold
required no new code: rebuild_state already folds every event after
snap.version, so returning the behind snapshot is the forward fold.
The safety argument is the bounded re-fold. A behind snapshot cannot be checked
against the fresh fold — the spine has moved on, so the digests differ for a
legitimate reason. grant_for_behind re-folds truncated to the stored version and
compares digests there; agreement proves the snapshot correct at its own watermark,
and everything after it is replayed. Serving a behind snapshot without that check
would be serving an unverified one.
grant_for_behind originally called fold_spine_inner directly.
recovery_reaches_the_spine_only_through_the_ladder failed with "a recovery path is
reading the in-flight-only index directly again — which is the whole of
ai-meta#307". It was right: the spine index evicts on completion, so the bounded
verification would have refused for exactly the completed executions it needs to
cover. It now goes through the ladder and truncates the ladder's events.
⚠ The same guard then failed on the doc comment, which used the parenthesised
fold_spine_inner( form. The comment was reworded rather than the guard weakened —
though it is worth knowing the guard counts comments as call sites.
| build |
SERVE_ON_BEHIND in binary |
RECOVERY_SOURCE (control) |
|---|---|---|
| 3.108.0 | 1 | 1 |
| 3.107.1 | 0 | 1 |
The control fires in both, which is what makes the 0 meaningful: without it,
"0 occurrences" is indistinguishable from a grep that does not work.
Serve-under-lag — the condition prod actually exhibits (11/11 stored_behind_spine)
— has no evidence. It needs a kind harness that induces mirror lag against a build
carrying the flag. The prod flip is gated on that.
NOETL_EHDB_RECOVERY_SOURCE flipped verify -> tier on prod
(statefulset/noetl-server-rust-embedded, v3.107.1). The tier now serves a
recovery fold when the in-memory spine refuses, instead of only recording what it
would have done.
| signal | value |
|---|---|
recovery_fold_total{folded, source=tier} |
42 |
recovery_fold_total{spine_incomplete, source=spine} |
42 |
recovery_fold_total{fold_failed, *} |
0 |
control series with unexpected != 0
|
0 of 21 |
Perfect 42/42 parity — the tier folded exactly where the incumbent refused. All 14 fold series are present and pinned, so those zeros are real zeros rather than absent families.
Rehearsed in kind before the forward flip: setting tier moved the metric to
{mode="tier"}=1, setting verify moved it back. That rehearsal also establishes
how a revert is confirmed — the mode is published as a metric, so it is read back
rather than assumed.
kubectl -n noetl set env statefulset/noetl-server-rust-embedded \
NOETL_EHDB_RECOVERY_SOURCE=verify
# confirm: noetl_ehdb_recovery_source_info{mode="verify"} 1
Postgres is not what changes here. tier alters only where a recovery fold
resolves its events; the write path and the authoritative store are untouched.
Full-spec kubectl diff --server-side before applying showed three lines: the
last-applied blob and generation (both mechanical) and value: verify -> tier.
A structured whole-object comparison found exactly one spec difference. Image,
volumeClaimTemplates, volumeMounts and the 62-var env count all unchanged.
4/4 executions completed; submissions 0.3–0.4 s; no-op storm 0/min (the
2026-09-09 degradation signal was ~52/min); 0 ERROR; parity divergent=0; controls
unexpected=0.
served_tier = 0 and folded{source=tier} = 0 on this pod, over a zero
denominator. Recovery fires only when the spine refuses, and a fresh pod with
healthy in-flight executions never refuses — so those zeros mean not yet
exercised, not working. The flip is justified by the pre-flip 42/42, not by a
post-flip demonstration. Read served_tier only alongside a non-zero
spine_incomplete; on its own it cannot distinguish a healthy tier from an
unexercised one.
Surveyed 2026-09-09, read-only. Worth stating first because it reframes this whole
page: the D1 embedded shadow documented here is not the most advanced read work
in the server. handlers/ehdb_projection_read.rs (919 lines) has shipped a
projection-tier (D3) read path with a four-mode ladder, enforced controls, and
production instrumentation.
NOETL_EHDB_PROJECTION_READ_SOURCE: postgres (default) · verify · tier ·
wal.
⚠ The module's own doc comment lists only the first three. Prod runs wal, and
read against the doc that looks like an unrecognised value silently resolving to
postgres. It is not — wal is a real fourth variant in the parser. The doc is
stale, not the config. Read the parser, not the prose.
⚠⚠ Wal mode does not consult Postgres at all — "not as a source and not as a
fallback. Every non-Match verdict refuses… the reconciler re-drives." This is the
mechanism behind the 2026-09-09 no-op storm: WAL chain incomplete; returning no-op
was this path correctly refusing to serve a state that might be wrong. The
incident was a safety property working as designed, plus an unbounded re-drive.
noetl_ehdb_projection_read_total{served_tier} = 0, {spine_refused} = 12.
The cause is documented in ehdb_projection_fold.rs and matches
noetl/ai-meta#307: recovery folded
only from the worker's in-memory state-builder index, which holds in-flight
executions and evicts them on completion, while the comparator asks about
completed ones. Two components with incompatible lifetimes ⇒ coverage ~0 by
construction. Nothing was broken.
Prod runs NOETL_EHDB_RECOVERY_SOURCE=verify — attempt the tier when the spine
refuses, record what it would have done, still refuse.
| outcome | source | count |
|---|---|---|
spine_incomplete |
spine | 24 |
folded |
tier | 24 |
fold_failed |
tier | 0 |
The tier folded 24 of 24 exactly where the incumbent refused 24 of 24. The flip
from verify to tier is one env var with that evidence behind it.
Three endpoints, all 200 against prod:
-
/api/ehdb/projection-recovery/equivalence— examined 25, agreed 25,equivalent: true /api/ehdb/projection-fold/executions/{id}-
/api/ehdb/projection-parity/executions/{id}—outcome: match
The fold endpoint returns content-level proof, not an id/type set:
postgres: applied_count=14 digest=a8073a05… version=356166954710474752
tier (ehdb_tier): applied_count=14 digest=a8073a05… version=356166954710474752
digests_agree: true
postgres_without_context: digest=ee7d11ed… <- a DIFFERENT digest
That last line is a built-in negative control: the digest is provably sensitive to
content, so digests_agree: true is a real claim rather than two trivially-equal
values. Enforced positive controls ship here too —
noetl_ehdb_projection_control_total{result="expected"} = 248, unexpected = 0
across nine control kinds.
⚠ Scope: these are comparator/diagnostic endpoints. The mainstream client APIs
(/api/executions, /api/execute) still resolve from Postgres.
DEFAULT_EMBEDDED_DIR is documented in-code as landing "under /data so it
lands on a mounted volume when one exists and fails loudly at open when it
does not". That second clause is false. LocalFsSubstrate::new calls
fs::create_dir_all, so when no volume is mounted the engine silently creates
the path on the container's ephemeral writable layer — and on prod that
layer has a hard ephemeral-storage limit of 1Gi, past which the kubelet
evicts the pod. A shadow taking down serving is precisely what the rest of
this design forbids.
It is safe at the current rate, and the distinction matters: the safety comes from measurement, not from the guard the comment promises.
| measured | |
|---|---|
| prod projection rate | 33 events / 97 min ≈ 500/day |
| disk touched per append | none — buffered until writer.should_seal()
|
| manifest growth | capped, DEFAULT_MANIFEST_RETAIN = 32
|
| ⇒ consumption | low single-digit MB/day against 1Gi |
A pod restart clears the layer as well, so it never accumulates in practice.
If the rate rises, or a retention driver is added, mount a bounded
emptyDir with a sizeLimit before trusting this again — the write would
then fail (and be swallowed into append_failed) instead of evicting the pod.
Tracked as part of noetl/ai-meta#332.
⚠ Full-spec kubectl diff --server-side against live before the apply, per
agents/rules/apply-safety.md. The entire diff was two lines — the image
digest and the server-assigned generation. No phantom volume, no annotation
change, no env change.
- Sharding and Owner Routing — the partition table and fail-closed forwarding this builds on.
- Central: noetl/ai-meta#332.
- Event envelope
- Event-sourced execution
- API surface
- Runtime shape (compiled + plug-in ring)
- Runtime registry (D8)
- Event-log retention + archival
- Off-box replication + the unsealed tail
- Catalog refuses secret material — topology, discovery, leases
- Cursor / claim loop mode
- Static guards
- noetl/cli wiki
- noetl/worker wiki
- noetl/tools wiki
- noetl/noetl wiki — Python implementation (twin during migration)
- noetl/ops wiki