automation ehdb topology

EHDB-only platform topology (automation/ehdb/)

NATS was deleted from the NoETL platform on 2026-08-01. Command dispatch, the events feed, the SSE fan-out, and the KV store behind the gateway's session cache and request store are all served by the EHDB writer now.

That topology is declared as noetl playbooks run in local mode, at automation/ehdb/ — not as loose manifests. The playbooks render and apply the Kubernetes objects as steps, but the source of truth and the executor is the playbook plus the noetl CLI, the same shape the rest of automation/ uses.

Landed by noetl/ops#245; tracked as noetl/ai-meta#222.

Why this exists

origin/main carried zero occurrences of NOETL_COMMAND_BUS until #245. The prod EHDB-only topology — noetl-cmdbus-writer-0, the events faces, the per-pool subject filters, the materializer sources — was produced imperatively with kubectl set env and kubectl patch during the 2026-07-31/08-01 cutover and never written back. There was nothing to redeploy prod from, and no file to read to find out what prod actually ran. The EHDB-only kind soak had to reconstruct the whole topology from the worker source before it could test anything, and recorded the gap as one of its defects.

This supersedes noetl/ops#241, which covered only the command-bus writer and predates the events faces entirely.

The playbooks

Playbook	Reconciles
ehdb_platform.yaml	Entry point. Guards the target cluster, then drives the other four.
ehdb_writer.yaml	Writer StatefulSet, per-shard PVCs, headless Service, one ClusterIP per shard. All nine faces.
ehdb_runtime.yaml	Server / user pool / system pool / gateway pointed at those faces, NATS-era env stripped.
ehdb_autoscaler.yaml	User-pool KEDA ScaledObject on EHDB per-subject lag, plus the writer scrape.
ehdb_verify.yaml	The functional proof.

# from repos/ops
noetl run automation/ehdb/ehdb_platform.yaml --runtime local \
  --set action=converge --set profile=kind --set context=kind-noetl

action is plan / converge / status / verify / teardown. profile is kind or prod, and it is a guard as well as a set of defaults: kind refuses any context that is not kind-*, and prod refuses one that is.

The nine faces

9100 cmdbus ingest   9103 events ingest    9106 events lag/metrics
9101 cmdbus claim    9104 events claim     9107 KV
9102 cmdbus lag      9105 events SSE       9108 WAL fan-out
                     9090 worker metrics

:9107 and :9108 appear in no other manifest in the repo. The gateway's session cache and request store depend on the first; the off-server state builder's WAL drain depends on the second.

The writer is a StatefulSet

It is a singleton per command shard over an RWO volume. Prod expressed that as one hand-copied Deployment per shard, which is how a two-shard cluster ended up with two files differing only in an index.

A StatefulSet says it once: the pod ordinal is the shard, volumeClaimTemplates give each shard its own volumes, and the controller will not run two pods with the same ordinal. Pod names come out noetl-cmdbus-writer-0, -1 — the names prod already resolves — so this is a drop-in for the per-shard Deployments rather than a re-address.

Addressing is one ClusterIP per shard (selecting the pod by statefulset.kubernetes.io/pod-name) plus a headless governing Service. There is deliberately no aggregate ClusterIP across shards: at more than one shard it would round-robin claim and ingest traffic across shards, which is silent mis-routing. drain_legacy removes the un-suffixed aggregate Service if a cluster still carries one.

storage_mode=claim mounts pre-existing, explicitly named PVCs instead of templating them. That is the adoption path: it converts a running writer Deployment into a StatefulSet without moving a byte of the durable log.

Never point a tcpSocket probe at an events face

ehdb_feed::serve handshakes inside its accept loop, so a connection that does not complete the handshake escapes the whole function and drops the listener — the face is then dead for the life of the process, with one ERROR line and nothing else (noetl/ehdb#311).

It is not only a malformed frame that does this. A connect that sends nothing and closes is enough. nc -z does it. A kubernetes tcpSocket readiness or liveness probe on :9108 would do it on every period, forever, and the pod would still look healthy.

So the writer's probes point at the command-bus claim face :9101, which does not share that shape, and the playbooks read face liveness from the pod's own listening sockets (netstat -ltn) rather than connecting to anything.

Verification is paired evidence, not an execution count

Every defect the NATS deletion produced failed silently while executions still completed. ehdb_verify.yaml therefore asserts published == projected, every materializer group's cursor advancing by that same delta and ending at lag 0, zero cursor errors, zero out-of-order appends, zero publish errors, zero duplicate execution ids, and every SSE subscriber (three, because the face is a broadcast) receiving frames. An unreadable metric counts as a failure, never as a zero.

Status

Validated in local kind only. Clean state to a working platform in 1m42s; idempotent re-converge with no generation bump on any workload and no pod restart; storage_mode=claim adoption and a prod-shaped Deployment→StatefulSet migration both verified.

Prod has not been converged from these playbooks yet. That is a separate, deliberate step — run action=plan --set profile=prod first and read the diff. The prod section of automation/ehdb/README.md lists what must be set (writer_storage_mode=claim is not optional there, writer_storage_class=premium-rwo, state_builder_shadow=false, gateway_reconcile=true) and what the Deployment→StatefulSet handover costs.

Related

• KEDA Scaler — the user-pool autoscaler this playbook set renders; the NATS-JetStream trigger it used to carry is gone.
• Production monitoring (GMP) — prod is Google Managed Prometheus, which is why the scaler uses a metrics-api trigger reading the writer's :9102 directly rather than a prometheus one.
• Local kind VM (Podman machine) — the cluster these playbooks were validated on.