Skip to content

deployment specification

Jump to bottom
Kadyapam edited this page Oct 9, 2026 · 4 revisions

noetl-gateway — deployment specification

The contract the noetl-gateway binary expects from its environment, per wiki-maintenance.md Rule 2a. Companion to configuration (which knob does what) and deployment (how to roll it); this page is the runtime contract — what must be true for the process to start and serve.

Measured against shastaratech-noetl-prod on 2026-08-05. Where this page and the manifests disagree, the cluster is right and this page is a bug.

Component summary
Binary noetl-gateway
Namespace gateway (not noetl)
Deployment gateway
Replicas 1
Image …/noetl/noetl-gateway:v3.7.1
Service gateway, type LoadBalancer, 80 → 8090

Runtime contract

The gateway is the gatekeeper only — session auth, authorization, SSE delivery, subscription routing. It never reads or writes domain data on a client's behalf (agents/rules/execution-model.md). Everything it needs it gets from the NoETL server API and the EHDB writer.

It refuses to start without:

  • NOETL_BASE_URL — the server it proxies to.
  • NOETL_EVENT_FEED_ADDR — the events SSE face. Empty is a hard bail! with an explicit message, not a degraded start.
  • NOETL_KV_ADDR — the KV face backing session and request state. Same.

That is worth stating plainly because it is not how the server and worker behave: their equivalents fall back to a quiet default and surface the problem downstream. The gateway is the one component here that fails loudly on missing bus configuration.

Network surface

Ports

Port Name Purpose
8090 http HTTP API, SSE (/events), health (/health)

ROUTER_PORT sets it; the manifest and the container port must agree.

Dependencies (outbound)

Target Where Why
NoETL server http://noetl.noetl.svc.cluster.local:8082 every domain operation
EHDB events SSE noetl-cmdbus-writer-0.noetl.svc.cluster.local:9105 execution-lifecycle feed to the SPA
EHDB KV noetl-cmdbus-writer-0.noetl.svc.cluster.local:9107 session + request buckets
Auth0 GATEWAY_AUTH0_DOMAIN login redirect + token verification

⚠ All three cluster targets are the same writer pod. It hosts both buses over ten named ports; there is no separate events-bus workload, which reads easily as "the events bus is not deployed".

Resources

request limit
CPU 100m 500m
Memory 128Mi 512Mi
Ephemeral storage 1Gi 1Gi

Health probes

Both hit GET /health on 8090.

initialDelay period timeout failureThreshold
liveness 10s 30s 1s 3
readiness 5s 10s 1s 3

⚠ /health is a liveness check, not a dependency check — it does not verify the server, the events feed or KV are reachable. A gateway that has lost its feed stays Ready. The symptom appears in the log as an SSE reconnect loop.

Environment variables

The full catalogue lives in configuration. This section records only what the runtime contract depends on, plus the traps.

Must be set

Variable Prod value If wrong
NOETL_BASE_URL http://noetl.noetl.svc.cluster.local:8082 every request fails
NOETL_EVENT_FEED_ADDR …:9105 refuses to start
NOETL_KV_ADDR …:9107 refuses to start
ROUTER_PORT 8090 must match the container port
GATEWAY_AUTH0_DOMAIN / _CLIENT_ID / _REDIRECT_URI Auth0 tenant login breaks

Optional

Variable Default Effect
NOETL_GATEWAY_RUNTIME_REGISTRY unset = on 0/false/off/no (trimmed, case-insensitive) stops this gateway registering itself in the server's runtime registry. Serving is unaffected either way.

Default-on is deliberate: a registry that has to be switched on is a registry that is off in the one deployment whose topology somebody needed to read.

Runtime registration

The gateway registers itself as the Gateway kind in the server's D8 runtime registry (POST /api/runtime/register) at startup and renews the lease forever after. It is then visible at GET /api/runtime/topology and to discover(Gateway).

Three properties worth knowing before changing it:

  • Liveness is a lease, not a status field. Nothing marks a gateway dead; it disappears because nothing renewed it. A status column someone has to remember to update is the drift shape this platform keeps finding — a lease cannot drift, because the absence of a heartbeat is the signal.

  • The heartbeat interval comes from the server's response, not from gateway config. POST /api/runtime/register returns ttl_secs and the gateway heartbeats at a third of it. A TTL duplicated on both sides is a representation that drifts the moment one side changes, and an interval derived from a stale copy would expire the lease it exists to renew.

  • It is fail-soft and never gates serving. A registration failure is logged and retried on the next tick. If the lease lapses the gateway vanishes from discovery, which is the correct reading — it genuinely could not reach the server.

The id is HOSTNAME (the pod name), sanitised to the [A-Za-z0-9-_.:] set the server's validate_op accepts. Replicas must stay distinguishable: a shared id would report one live gateway no matter how many were running, which is a count that looks plausible and is wrong.

⚠ POST /api/runtime/register is on the server's privileged route list. The auth gate is in shadow mode today, so the unauthenticated call succeeds; when the gate is flipped to enforce, this call needs a service credential or gateway registration starts failing (fail-soft, so the gateway keeps serving — it would simply go missing from discovery).

Traps

  • NOETL_AUTH_SYNC / NOETL_AUTHZ_SYNC accept only exactly "true" or "1". No trim, no case folding — "TRUE", "True", " true", "yes", "on" all mean false, silently, and login reverts to the slow drive-dispatched path (#168 took it from ~7.9s to ~0.4s). The worker accepts 1/true/yes/on case-insensitively for the same concept, so a value copied between components changes meaning. Prod: both true.

  • GATEWAY_AUTH_BYPASS disables authentication. Development only. There is no prod-guard on it beyond not setting it.

  • NOETL_EVENT_SOURCE does nothing. Prod sets it to ehdb. Nothing reads it: EventSourceMode::from_env_value is called only from its own unit tests, and main.rs starts the EHDB feed listener unconditionally once NOETL_EVENT_FEED_ADDR is non-empty. The gateway is EHDB-only by construction, which is correct post-T5 — but the variable, the enum and its Nats variant are residue. The enum's doc comment still claims "anything unrecognised is nats, so a typo can never silently take the SPA's live updates off their working path"; there is no nats path to fall back to. Tracked on #242.

Not runtime

NOETL_LIVE_OIDC_TOKEN, NOETL_LIVE_OIDC_AUD, NOETL_LIVE_OIDC_SA are read only inside an #[ignore]d live-validation test driven by noetl/e2e scripts/live_validate_oidc_verify.sh. They are not deployment configuration and must not be set on the Deployment.

Secrets

NOETL_INTERNAL_API_TOKEN gates the server's /api/internal/* routes. Supply it from a K8s Secret — never inline in a manifest (#188 is the same defect for a different credential). Auth0 client secrets do not belong here: the gateway holds only public tenant identifiers; the secret lives in the keychain and is used by the server-dispatched auth0_login playbook.

Observability

⚠ The gateway is covered by no PodMonitoring — namespace gateway has no scrape configuration at all (verified 2026-08-05: kubectl -n gateway get podmonitoring → no resources). Until that is applied, the log is its only signal in practice.

The gateway does expose a Prometheus endpoint, at GET /metrics on the serving port (http, 8090) — this page previously said otherwise, and the reason it was easy to believe is worth recording. Every metric on the ingress registry is labelled by subscription, and Registry::gather prunes metric families with no children, so before a delivery has ever arrived the endpoint returns HTTP 200 with a zero-byte body. Checking it on a quiet gateway produced exactly the evidence you would expect from "there is no endpoint".

noetl_gateway_build_info{version} (added in noetl/ai-meta#238) is unconditional and needs no traffic, so the endpoint now always answers with at least the running version. That also makes it possible to tell an unscraped gateway from a wrong port from a binary too old to carry a metric you are looking for — three states that an empty body could not distinguish.

Metrics

metric type labels meaning
noetl_gateway_build_info gauge version Always 1; identifies the running binary.
noetl_ingress_received_total counter subscription Push/webhook deliveries received.
noetl_ingress_rejected_total counter subscription, reason Rejected at verification; no execution dispatched.
noetl_ingress_dispatched_total counter subscription Verified deliveries forwarded as one execution.
noetl_gateway_auth_bypass_enabled gauge mode 1 when authentication is DISABLED. GATEWAY_AUTH_BYPASS makes the gateway accept any session token and inject a synthetic user_id: 0 context. Registered in both states, so mode="enforced" reads 0 on a healthy gateway.
noetl_gateway_login_total counter outcome succeeded | callback_failed | not_authenticated. Recorded in finish_login, which both auth flows share.
noetl_gateway_session_check_total counter outcome Per-request session decision: ok | missing_token | invalid_or_expired | validation_error | bypass.

Why the success outcomes are counted too

noetl_gateway_login_total and noetl_gateway_session_check_total both record their success paths, not only failures. That is deliberate: a spike in invalid_or_expired means nothing without ok beside it, and an auth backend failing every request looks identical to one nobody is using. The rate is the signal these exist for — before them, every auth decision the gateway made was visible only as a tracing::warn! on the failing branch.

noetl_gateway_auth_bypass_enabled is registered in both states for the same reason at one remove: a gauge that appeared only when bypass was on would leave "auth is enforced" and "I cannot tell" identical — which is exactly the failure it replaces, since the old per-request warn exists only in the dangerous state.

⚠ This component cannot currently be validated on kind. Its images are amd64-only while the validation cluster is arm64, so the released gateway cannot run there and the kind deployment holds a registry.k8s.io/pause:3.10 placeholder. Tracked as noetl/ai-meta#245; the metrics above are unit-verified but have never been confirmed against a real scrape.

The three noetl_ingress_* counters are absent until the first matching delivery — that is the normal state, not a fault.

The reconnect loop is the one log pattern worth knowing: EHDB lifecycle feed error; reconnecting … Connection refused. Bursts of it coincide with writer restarts, and the backoff caps at ~5s, so a genuinely broken gateway writes ~120 lines per 10 minutes. Date the last line before reacting — silence since the last burst is positive evidence it reconnected.

Related

Gateway

Surfaces

Operations

See also

Clone this wiki locally