runtime registry

Runtime registry (D8) — topology and discovery

The server keeps the fleet's topology in EHDB dataset D8 (RuntimeStore) and serves it over /api/runtime/*. Every component that wants to be discoverable registers itself and renews a lease; discovery is a read against those leases at a given instant.

Source: src/runtime_registry.rs, src/handlers/runtime_topology.rs. Program: noetl/ai-meta#455.

The invariant: liveness is a lease, not a status field

An instance is live when now - last_seen < ttl. Expiry is a read-time verdict, not a delete — nothing marks an instance dead; it stops appearing because nothing renewed it.

This is deliberate and it is the reason the registry exists in this shape. A status column somebody has to remember to update is the drift this platform keeps finding — noetl.execution.status froze at the Python retirement and still claimed 3,225 executions RUNNING (ai-meta#235). A lease cannot drift, because the absence of a heartbeat is the signal.

Do not add a "mark dead" path. If you need to remove an instance promptly, deregister it; the lease is the fallback, not the mechanism to replace.

Kinds

RuntimeKind is Server | Gateway | Ehdb | Worker | Playbook | Execution.

⚠⚠ RuntimeKind::default() is Worker. POST /api/runtime/register therefore returns 400 on an unknown kind rather than deserializing to the default — a silent coercion would file a gateway under the wrong role and make discover(Gateway) wrong in a way nothing reports.

kind	who registers it	how
Server	the server itself, at startup, before the listener binds	self_register() + spawn_heartbeat()
Gateway	the gateway, from its own process	POST /api/runtime/register, renewed forever (noetl/gateway v3.13.0)
Worker	mirrored from the authoritative pool register/heartbeat	handlers::runtime → mirror_pool_*
Ehdb	the server, on a publish that landed	mirror_ehdb_tier_seen
Execution	the execution path, with a longer TTL	mirror_execution_started / _finished
Playbook	not yet wired	—

Endpoints

route	behaviour
GET /api/runtime/topology	live members grouped by kind: {kinds: [{kind, count, members: [{id, contract, age_secs, last_seen_micros}]}]}. 503 when the registry is off, not an empty 200 — an empty 200 would read exactly like a healthy empty fleet.
GET /api/runtime/watch?after=N	ops since op_seq N, plus a cursor. History is required, which is why retention/compaction cannot be applied to D8.
POST /api/runtime/register	upsert {kind, id, contract}; returns {registered, renewed, ttl_secs}. Unknown kind → 400.
POST /api/runtime/deregister	clean departure.

⭐ ttl_secs in the response is the contract for heartbeat pacing. Callers derive their interval from it (the gateway uses ttl/3) rather than configuring a TTL of their own. A TTL duplicated on both sides is a representation that drifts the moment one changes, and an interval computed from a stale copy would expire the lease it exists to renew.

Environment variables

Catalogued on deployment-specification; summarised here:

variable	default	meaning
NOETL_RUNTIME_REGISTRY	on (follows embedded_enabled())	opt-out
NOETL_RUNTIME_TTL_SECS	180	lease length
NOETL_RUNTIME_HEARTBEAT_SECS	60	clamped below the TTL
NOETL_RUNTIME_EXECUTION_TTL_SECS	300	executions get the longer lease

⚠⚠ The Ehdb hook, and why its placement is the load-bearing detail

mirror_ehdb_tier_seen is called from two publish paths, and only on success:

• event_bus.rs — a successful event publish.
• handlers/execute.rs — the command publish, on the PublishOutcome::Landed arm only.

Three things here are easy to get wrong, and two of them already were.

1. Never register from configuration. An env var naming a writer says nothing about whether it is up. Registering from config would claim liveness for a dead address — the exact drift the registry exists to avoid. A completed round-trip is the forcing function.

2. Deferred is not evidence. PublishOutcome::Deferred means the on-path budget elapsed and a background task is still retrying; the writer may well be down. Registering on it would claim liveness from a timeout — the same misclassification as tier-load reporting ok: true for an explicit refusal.

3. ⚠⚠ A correctly-gated hook on an unreached path registers nothing. v3.130.0 shipped this hook on the events publish alone and measured ehdb_registered = 0 on prod, because should_publish skips system executions:

noetl_event_ingest_publish_skipped_total{reason="system_execution"} 22
noetl_event_ingest_total                                            (absent — family pruned)
noetl_command_publish_total{pool="system",route="legacy"}            4

Prod's only regular traffic — the hourly noetl-scheduled-cleanup and the 3-minutely noetl-state-builder-watchdog — is system executions, so publish_event is never called there. Two structural tests covering the hook passed throughout: both assert the call site is correctly gated, neither asserts anything reaches it. That is ai-meta#326 — existence is not reachability. Fixed in v3.130.1 by adding the command-bus call site.

Do not delete either call site. They are two different buses (cmdbus 9100-02, events 9103-08 — hosted by one writer process in prod, but distinct endpoints), and each is the only evidence for its own.

The throttle is keyed by address

EHDB_LAST_SEEN holds last-seen micros per address, bounded at 64 entries with least-recently-seen eviction.

⚠⚠ It was a single global timestamp while there was one call site. With two buses reporting, whichever address published first would throttle the other out for a full floor interval, leaving a live tier unregistered while every counter read healthy. One cap serving two addresses is the same defect as one cap serving two directions (noetl/worker#314). Eviction replaces the oldest rather than refusing, because refusing would make a genuinely new tier permanently invisible.

The decision lives in tier_seen_due(seen, addr, now, floor) — extracted as a pure function so it can be asserted against rather than buried in a static.

The hot path must not take the registry lock. The throttle is a separate, tiny lock and both it and the registry are taken with try_lock: a publish must never wait on the registry, because an observability surface that slows the thing it observes is the same class of mistake as a scrape that stalls appends.

Metrics

noetl_runtime_registry_total{outcome} — 11 outcomes, all pinned at 0 unconditionally at startup (init_runtime_registry_series()):

registered  register_failed  heartbeat  heartbeat_failed  reregistered
deregistered  mirror_failed  heartbeat_skipped
execution_registered  execution_finished  ehdb_registered

⚠ Pinning matters because Registry::gather prunes families with no children, so an unpinned labelled metric is absent until it first fires — indistinguishable from a broken exporter or a pod predating the metric. A pin placed inside a config branch is not a pin (server#315 pinned publish-skip reasons inside if event_bus_mode.publishes_ehdb(), leaving them absent on exactly the configuration whose reason someone would read).

heartbeat_skipped being large relative to heartbeat is normal — it is the throttle working.

Reading it in prod

kubectl --context "$PROD" -n noetl port-forward pod/noetl-server-rust-embedded-0 8082:8082
curl -s localhost:8082/api/runtime/topology | jq '.kinds[] | {kind, count}'

⚠ kubectl port-forward fails open: a local port already bound routes curl to whatever cluster owns it, and --context protects kubectl, not curl. Check the forward log for Forwarding from, and after the read kill the forward and confirm the probe dies — that negative control is what distinguishes "I read this cluster" from "I read some cluster".

Related

• deployment-specification — env-var source of truth.
• runtime-shape — the four-binary runtime.
• ai-meta#455 — the program.
• noetl/gateway wiki → deployment-specification — the gateway side of registration.