Skip to content

Backend Configuration

Jump to bottom
Kadyapam edited this page Jul 6, 2026 · 1 revision

Backend Configuration (Phase 10 — Tunable-Backend Config Surface)

The per-tier backend-selection surface for NoETL's five internal platform tiers. Phase 10 is the final phase of the EHDB completion program: it consolidates the scattered NOETL_EHDB_* per-tier flags into one coherent, documented schema so an operator can select — and verify — whether each platform tier runs on EHDB or on its external incumbent.

EHDB is the default, not a lock-in. Every tier stays first-class selectable back to the engine it replaced. A deployment can run the log on JetStream+Postgres and vectors on EHDB, or any per-tier mix.

Platform-only boundary. This surface configures NoETL's internal platform storage tiers. Business/tenant data is never in EHDB — it stays in real business systems (PostgreSQL, Snowflake, Kafka, object stores, …) reached via playbook connectors, entirely outside this surface.

The five platform tiers

Tier Runtime env var Default backend (program end) Selectable back to (incumbent)
Event log NOETL_EHDB_EVENTLOG EHDB event-log engine NATS JetStream + Postgres
Projection / read-model NOETL_EHDB_PROJECTION EHDB projection engine Postgres materializer
KV / state NOETL_EHDB_KV EHDB KV engine NATS KV
Object / blob NOETL_EHDB_OBJECT EHDB object engine external object store (GCS/S3) / Postgres
Vector NOETL_EHDB_VECTOR EHDB vector engine Qdrant

The two axes: mode and backend

Each tier is selected on two axes. The mode (NOETL_EHDB_<TIER>) is the operational state; the backend is derived from it.

Mode (NOETL_EHDB_<TIER>) What runs Serving backend
off (default) strict no-op — no EHDB engine opens external incumbent
shadow EHDB dual-writes + parity-compares, never serves external incumbent
primary EHDB serves authoritatively, incumbent dual-run parity-checked EHDB

Backend derivation is exactly: primary ⇒ EHDB serves; shadow/off ⇒ the external incumbent serves. shadow is a migration state — EHDB is being written and verified but the incumbent is still authoritative, so the serving backend is still external.

Resolution precedence

The consolidated config resolves in this order:

  1. Umbrella enable — NOETL_EHDB_ENABLED (truthy). The single gate every tier sits under. When unset, the whole integration is a strict no-op.
  2. Per-tier mode — NOETL_EHDB_<TIER> ∈ off|shadow|primary, read through each tier's own runtime parser.
  3. Derived backend — primary ⇒ EHDB, else the external incumbent.

The role env (NOETL_EHDB_CLIENT_ROLE) is orthogonal but load-bearing for coherence: only data-plane roles (worker/playbook/system) may serve a tier; control-plane gatekeepers (gateway/api/server) never do.

Backward compatible by construction

Phase 10 adds a unifying resolution + validation layer, not a breaking rename. The existing NOETL_EHDB_* env vars remain the source of truth, and the worker resolves each tier's mode through the same <Tier>Mode::from_env parser the runtime dispatch already uses (ehdb_reference::backends in the ehdb crate; noetl_worker::ehdb::backends in the worker). So the consolidated view can never drift from actual behavior: a deployment with no NOETL_EHDB_* env resolves every tier to its current default (external incumbent, off), and an enabled primary tier resolves to EHDB exactly as dispatch serves it.

Coherence validation

The resolver rejects incoherent combinations with clear operator messages (the runtime stays a disabled-by-default no-op regardless — the validation classifies a misconfig, it does not change the strict-no-op guarantee):

  • A tier requesting shadow or primary while NOETL_EHDB_ENABLED is unset — a tier cannot shadow-mirror or serve while the umbrella integration is disabled.
  • A tier requesting shadow/primary on a control-plane role (gateway/api/server) — those roles are gatekeepers and never serve a data-plane tier.

Verifying the resolved matrix

The ehdb-selfcheck config verb (alias backends), shipped in the same image as noetl-worker, prints the resolved per-tier backend+mode matrix. It is read-only (opens no engine), the render is secret-free (only tier keys/modes/backends + the role token — never an env value), and its exit code conveys coherence: 0 coherent, 4 incoherent.

All-external default (no NOETL_EHDB_* env)

$ ehdb-selfcheck config
{"op":"config","coherent":true,"errors":[],"secret_free":true,"matrix":{
  "enabled":false,"role":"worker","control_plane":false,
  "integration_mode":"disabled","coherent":true,"errors":[],
  "tiers":[
    {"tier":"eventlog","env_var":"NOETL_EHDB_EVENTLOG","mode":"off","backend":"external","incumbent":"NATS JetStream + Postgres"},
    {"tier":"projection","env_var":"NOETL_EHDB_PROJECTION","mode":"off","backend":"external","incumbent":"Postgres materializer"},
    {"tier":"kv","env_var":"NOETL_EHDB_KV","mode":"off","backend":"external","incumbent":"NATS KV"},
    {"tier":"object","env_var":"NOETL_EHDB_OBJECT","mode":"off","backend":"external","incumbent":"external object store (GCS/S3) / Postgres"},
    {"tier":"vector","env_var":"NOETL_EHDB_VECTOR","mode":"off","backend":"external","incumbent":"Qdrant"}]}}
# exit 0

Run a tier on EHDB (all-EHDB)

Set the umbrella enable, a data-plane role + local-reference log, and each tier's flag to primary:

$ NOETL_EHDB_ENABLED=true NOETL_EHDB_MODE=local_reference \
  NOETL_EHDB_CLIENT_ROLE=worker NOETL_EHDB_LOCAL_REFERENCE_LOG=/var/lib/ehdb/ref.jsonl \
  NOETL_EHDB_EVENTLOG=primary NOETL_EHDB_PROJECTION=primary NOETL_EHDB_KV=primary \
  NOETL_EHDB_OBJECT=primary NOETL_EHDB_VECTOR=primary \
  ehdb-selfcheck config
# → coherent:true, every tier backend:"ehdb", exit 0

Mixed (per-tier selection)

Log + vector on EHDB, projection dual-writing (shadow), KV/object left on their incumbents:

$ NOETL_EHDB_ENABLED=true NOETL_EHDB_MODE=local_reference \
  NOETL_EHDB_CLIENT_ROLE=worker NOETL_EHDB_LOCAL_REFERENCE_LOG=/var/lib/ehdb/ref.jsonl \
  NOETL_EHDB_EVENTLOG=primary NOETL_EHDB_VECTOR=primary NOETL_EHDB_PROJECTION=shadow \
  ehdb-selfcheck config
# → eventlog:ehdb, vector:ehdb, projection:external(shadow), kv:external, object:external; coherent:true, exit 0

Incoherent — caught with exit 4

$ NOETL_EHDB_EVENTLOG=primary ehdb-selfcheck config
# coherent:false, exit 4
# errors: ["[eventlog] NOETL_EHDB_EVENTLOG requests 'primary' but NOETL_EHDB_ENABLED
#           is not set — a tier cannot serve from EHDB while the umbrella EHDB
#           integration is disabled (set NOETL_EHDB_ENABLED or return
#           NOETL_EHDB_EVENTLOG to 'off')"]

$ NOETL_EHDB_ENABLED=true NOETL_EHDB_CLIENT_ROLE=gateway \
  NOETL_EHDB_MODE=local_reference NOETL_EHDB_LOCAL_REFERENCE_LOG=/tmp/x.jsonl \
  NOETL_EHDB_EVENTLOG=primary ehdb-selfcheck config
# coherent:false, exit 4
# errors: control-plane role 'gateway' never serves a data-plane tier

Rolling a tier back to its incumbent

Because the backend is a runtime choice, rollback is a flag flip with no redeploy: set the tier's NOETL_EHDB_<TIER> back to shadow (keep verifying) or off (fully external). The incumbent is authoritative again immediately; the primary path only ever appends to the EHDB store and never mutates what the incumbent owns, so there is zero data loss. Each tier also carries a compile-time PRIMARY_SERVE_ACTIVATED kill switch as a structural belt-and-suspenders revert. See the per-tier rollback procedures in the Roadmap Phase 9 sections.

Where the schema lives

  • ehdb crate — ehdb-reference::backends: the pure schema (PlatformTier, TierMode, Backend, BackendMatrix), the backend_for_mode derivation, the coherence validate(), and the secret-free to_json() render. Reads no env, opens no engine. (ehdb#252, merged 4c0df81.)
  • worker — noetl_worker::ehdb::backends: resolve(&EnvMap) maps the process env (through the per-tier from_env parsers) into a BackendMatrix, and the ehdb-selfcheck config verb prints it. (worker#166, released v5.66.0.)

Status

Phase 10 — IMPLEMENTED (2026-07-06). The config surface landed (cargo test + clippy -D warnings clean; ehdb-selfcheck config matrix validated in-cluster batched with the Phase-9 kind runs). With Phase 10 the whole EHDB completion program (Phases 6–10) is code-complete and kind-validated — every platform tier has a built + shadow-verified engine, a per-tier reversible primary cutover, and a consolidated backend-selection surface. The only remaining step is the prod/GKE cutover, gated on the user, per tier; nothing in prod has changed (prod runs worker v5.52.0, all NOETL_EHDB_* flags default off).

Related

EHDB

Programs

Kind validation

Query interface

Runbooks

Design notes

GitHub

Clone this wiki locally