Skip to content

cmdbus writer tier service

Jump to bottom
Kadyapam edited this page Sep 23, 2026 · 2 revisions

cmdbus-writer: the tier service and the shared Tokio runtime

The noetl-worker binary has a second role beyond the pull loop. With NOETL_COMMAND_BUS_HOST=true and NOETL_EVENT_BUS_HOST=true it becomes the writer — in prod, the single StatefulSet pod noetl-cmdbus-writer-0 that hosts the command bus, the event bus and the EHDB tier service together.

This page records two facts about that role that are not obvious from the code and cost a production diagnosis to establish (noetl/ai-meta#351, 2026-09-22).

1. The tier service and the buses share a RUNTIME, not a lock

⚠ Corrected 2026-09-23. An earlier version of this page said the tier service "takes the engine lock" and that a slow tier operation holding that lock blocks the :9102 scrape. That is wrong, and it was wrong in a page written to representation-drift.md. The two subsystems share no lock:

  • tier_store uses ehdb_reference with a per-store-path tokio::sync::RwLock (src/ehdb/tier_store.rs:163); it is not an L0 engine — the file says so itself at tier_store.rs:211.
  • The command/event buses use ehdb_l0::L0Engine behind a std::sync::Mutex (ehdb-feed/src/lib.rs:57, :500).

Grepping ehdb_l0|L0Engine in tier_store.rs / tier_service.rs matches comments only — no type, no import, no call.

What actually couples them is the shared Tokio runtime. src/main.rs is a bare #[tokio::main], so every face — command-bus ingest/claim/lag, event bus, tier service, metrics — runs on one multi-threaded runtime whose worker count is available_parallelism().

Every tier data path is synchronous and runs to completion on a runtime worker thread, with no spawn_blocking:

path file:line
append_locked tier_store.rs:535
append_batch_locked tier_store.rs:469
read_execution_locked tier_store.rs:643
scan_locked tier_store.rs:1080

The only spawn_blocking in the file is tier_store.rs:713, the off-request index backfill.

Measured in production, noetl-cmdbus-writer-0, 2026-09-23T20:34Z:

tokio-rt-worker threads : 2          (of 8 total in the process)
NOETL_EHDB_TIER_MAX_INFLIGHT : 4
cpu.max                 : 200000 100000   (= 2.0 cores)
nproc (uncapped view)   : 4

available_parallelism() honours the cgroup quota, so the runtime has two worker threads while the tier permits four concurrent synchronous multi-hundred-MB file replays. Four such operations therefore occupy the entire worker pool and nothing else in the process runs — that is the stall, and it needs no shared lock to explain the correlation with the buses, the server's HTTP surface, or the scrape.

⚠ Moving the tier work to spawn_blocking is necessary but not sufficient: measured in kind, 8 concurrent reads of a 973 MiB segment at cpu=2 still starved a cost-free request, because the blocking pool does not create CPU. The same gate with a 243 MiB segment served normally — segment size is the other half of the mechanism.

So a slow tier operation starves the :9102 scrape of CPU, and the scrape's client sees a connection that was accepted and then never answered. KEDA (3s timeout, 10s poll) reports this as:

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

which reads like an unreachable endpoint and is not one.

The same lock is in the path of the server's command- and event-bus calls. The server points NOETL_COMMAND_BUS_WRITER_ADDRS and NOETL_EVENT_BUS_WRITER_ADDRS at this one pod, so when a tier operation stalls, the server's own HTTP surface stalls with it — including request paths that never touch the bus at all. On 2026-09-22 that reached the SPA as a hung UI, with /api/auth/validate (a Postgres lookup) taking 16–68s.

Measured on prod, worker 6.1.6, from one :9090/metrics scrape:

tier_service op count mean >1s tail mean
read_execution 7086 5.587s 15.3% ~36.4s
append 2607 2.217s 12.1% ~18.1s
append_batch 593 3.774s 11.8% ~31.8s
scan 1 75.92s 100% 75.9s
health 367 0.000376s 0% —

health against read_execution is the diagnostic pairing. Both are served by the same process on the same listener; 0.38ms next to 5.6s says the control path is fine and the data path is contended. Reach for that comparison before concluding a writer pod is sick — a stalled writer and a saturated one look identical from outside, and six plausible external theories (disk full, DNS, CPU throttling, cold cache, an autoscaler feedback loop, node pressure) were all wrong on the way to it.

Saturation shows up in the connection outcomes, against NOETL_EHDB_TIER_MAX_INFLIGHT (4 in prod, max_waiters=32, shed_after_ms=1000):

tier_service.conn  accepted=10656  closed=9751
                   shed_busy=805  shed_waiters_full=68  write_error=901

16.6% shed or failed. Averaged over the pod's life the tier ran at ~41% of its inflight capacity, so the shedding is bursty and concentrated in the stall windows rather than a steady overload.

2. The writer's probes cannot see any of this

The deployed StatefulSet uses TCP-only probes, for liveness as well as readiness, and points both at the claim port rather than at any of the faces that stall:

livenessProbe:  { tcpSocket: { port: cmdbus-claim }, timeoutSeconds: 1,
                  periodSeconds: 20, failureThreshold: 3 }
readinessProbe: { tcpSocket: { port: cmdbus-claim }, timeoutSeconds: 1,
                  periodSeconds: 5,  failureThreshold: 3 }

A listening socket answers a TCP probe whatever the process is doing behind it. So the writer stays Ready with Restart Count: 0 through stalls of any duration — the 2026-09-22 incident ran 18 hours without alerting anyone, and was found only because a user reported a hanging UI.

⚠ Do not "simplify" these to match — they are already the weak form. The binary serves /healthz, /readyz and /livez on :9090 (see deployment-specification); the writer's probes should use them. This is the same defect class as noetl/ai-meta#322, which found a TCP readiness probe weaker than its HTTP liveness probe on the system pool — here both are TCP, on the pod that carries both buses.

What to watch

signal where means
ehdb_l0_durability_sample_ok :9102, :9106 ⚠ 0 means the engine mutex was POISONED (a thread panicked holding it), not contended. std::sync::Mutex::lock() blocks; the provider's .lock().ok() maps only PoisonError to None (command_bus.rs:401). On a contended lock the scrape does not degrade to 0 — it blocks. The HELP text (command_bus.rs:438, event_bus.rs:524) asserts a contention check the code does not implement.
noetl_ehdb_tier_service_duration_seconds{operation} :9090 ⚠ do not compare health against read_execution to detect saturation. The measured window starts after the permit is acquired (tier_service.rs:591), so a health that queued 999ms still records ~0.38ms, and one shed for lack of a permit is recorded under conn/shed_busy and never under health at all. health is a format! over a constant with no await point, so it cannot observe runtime starvation either. The series is structurally incapable of showing saturation.
noetl_ehdb_dataplane_ops_total{outcome="shed_busy"} :9090 rising = past NOETL_EHDB_TIER_MAX_INFLIGHT
ehdb_feed_subject_lag{subject} :9102 per-pool, so read the right subject: commands.shared.* is the user pool, commands.system.* the system pool. They diverge.

That last row is worth its own warning. On 2026-09-22 commands.shared.shard.0 sat at 0 for a full day while commands.system.shard.0 climbed past 600. Read only the total (ehdb_feed_total_lag) or only the pool you expected, and you will attribute a backlog to the wrong pool and size the wrong deployment.

Related

noetl-worker

Architecture

Operations

Related repos

External

Clone this wiki locally