architecture

Architecture

How the gateway fits into the NoETL stack and what each internal module does.

For the underlying principle, see Ephemeral Blueprints + Compute-Data Boundary. The gateway is the concrete expression of "gatekeeper only" in that model.

Position in the stack

┌─────────────────────────────┐
│   Client (browser, CLI,     │
│   partner integration)      │
└────┬───────────────────┬────┘
     │ session_token      │ SSE /events
     ▼                    ▼
┌──────────────────────────────────────────────────┐
│   noetl-gateway   (this repo)                    │
│                                                  │
│   - Auth: /api/auth/login, /api/auth/validate    │
│   - GraphQL: executePlaybook                     │
│   - Proxy: /noetl/* -> noetl-server              │
│   - SSE: /events fan-out per client_id           │
│   - Callbacks: /api/internal/callback{,async}    │
│   - NATS bridge: playbook/state frames           │
└────┬─────────────────────────────────────────────┘
     │ HTTP /api/execute
     │ NATS subscribe
     ▼
┌──────────────────────────────────────────────────┐
│   noetl-server (FastAPI) + worker pool           │
│   noetl-server publishes execution events on     │
│   NATS subject prefix `noetl.events.`            │
│   the gateway subscribes and re-emits as         │
│   playbook/state SSE frames.                     │
└──────────────────────────────────────────────────┘

Every arrow is a real network hop. The gateway never reaches a domain database; the worker pool never reaches the browser; the browser never reaches the worker pool directly.

Module map

src/main.rs

Application bootstrap: builds the axum router, wires middleware (auth, CORS, tracing), and starts the server.

src/auth/

• login: Auth0 ID-token exchange. Dispatches the api_integration/auth0/auth0_login playbook in noetl-server via the gateway-to-noetl proxy. On success, persists the session and returns the session token.
• validate: confirms a presented session token is still valid.
• middleware: extracts the session token from every authenticated request, verifies it, and attaches the authenticated user record to the request context.

src/graphql/

GraphQL endpoint at POST /graphql. The primary mutation is executePlaybook(path, workload):

1. Resolves the authenticated user and client_id.
2. Stores request_id → { client_id, session_token, execution_id, playbook_path } in RequestStore (NATS K/V).
3. POSTs to noetl-server's /api/execute to start the playbook.
4. Returns { requestId, executionId } immediately. The client awaits completion over the SSE channel rather than blocking the HTTP request.

src/sse.rs + src/connection_hub.rs

• GET /events?session_token=...&client_id=... opens an SSE stream. The handler validates the session, registers the client in ConnectionHub, sends an init message frame containing the assigned client_id, and merges per-client messages with periodic ping heartbeats.
• ConnectionHub is the in-memory registry of client_id → channel for fan-out. send_to_client is the universal outbound primitive.

src/callbacks.rs

• POST /api/internal/callback receives synchronous callbacks from playbooks (e.g. the send_success_callback step in auth0_login). Looks up the request_id in RequestStore and routes the result to the corresponding SSE client as a playbook/result frame.
• POST /api/internal/callback/async handles asynchronous callbacks the same way.

src/playbook_state.rs

NATS subscriber that listens on subjects matching noetl.events.* and forwards a curated allowlist of event types to interested clients. Forwarded event types (FORWARDED_EVENT_TYPES):

• step.exit
• playbook.completed
• playbook.failed
• calendar.event.touched (added v2.12.0; orchestrator-side signal that a calendar entry was written, so SPA clients can re-read on a specific signal rather than the generic playbook.completed)

For each forwarded frame:

1. Looks up which client_id cares about the execution_id (the subscriber maintains a per-client filter map seeded by executePlaybook and subscribeToExecution calls).
2. Forwards the frame as a playbook/state SSE event to that client.

Added in v2.11.0; the allowlist gained calendar.event.touched in v2.12.0. This lets the SPA-side waitForExecution move off polling onto a push model.

Removed in v2.12.0: src/firestore_subscriptions.rs

The gateway used to expose POST /api/subscriptions/firestore and DELETE /api/subscriptions/{subscription_id} plus a Python sidecar (scripts/firestore_listener.py) that opened Firestore onSnapshot watches on the SPA's behalf. This violated the gatekeeper-only boundary — the gateway was reaching a domain database to satisfy a client request.

Removed in v2.12.0 (noetl/gateway#18

• Dockerfile / config cleanup in noetl/gateway#19). Live-update transport for SPA clients now goes through the NoETL event stream — the orchestrator emits calendar.event.touched events that the gateway forwards via the SSE channel (see playbook_state.rs above). See Subscriptions for the historical reference page.

src/request_store.rs

NATS K/V store keyed by request_id. Holds the in-flight mapping needed for callback routing. Cleared when the corresponding playbook/result frame is delivered.

src/proxy.rs

Proxies authenticated requests under /noetl/* to noetl-server, attaching the session token's user context as headers. Lets CLIs and other clients use the same auth model as the SPA without re-implementing NoETL API calls.

src/session_cache.rs

Short-TTL in-memory cache for validated session tokens. Avoids hammering noetl-server's auth.sessions table on every request.

src/nats.rs

NATS connection setup and helpers. Reads NATS_URL from the environment. NATS connection itself is platform-runtime, not a business-logic credential (see secrets-and-credentials rule).

src/db/

Postgres pool wiring for the auth/session tables. Same platform-runtime classification.

Request flow examples

Authenticated playbook execution

SPA -> POST /api/auth/login { id_token }
SPA <- 200 { session_token, expires_at, user_id }
SPA -> GET /events?session_token=...
SPA <- SSE: message { result: { clientId } }
SPA -> POST /graphql { query: executePlaybook(...), variables }
SPA <- 200 { requestId, executionId }
... worker pool runs the playbook ...
playbook -> POST /api/internal/callback { request_id, status, data }
SPA <- SSE: playbook/result { requestId, executionId, status, data }

Calendar live-update push (post-v2.12.0)

SPA -> POST /graphql { executePlaybook(travel/playbooks/catalog/calendar/list, ...) }
SPA <- 200 { requestId, executionId }
... orchestrator writes a calendar event during a chat turn ...
worker -> emits calendar.event.touched on NATS subject noetl.events.<exec_id>.*
playbook_state.rs receives -> ConnectionHub::send_to_client
SPA <- SSE: playbook/state { execution_id, event_type: calendar.event.touched, ... }
SPA: re-runs the read playbook to fetch the updated calendar list

Pre-v2.12.0 the SPA went through POST /api/subscriptions/firestore and a Python sidecar that watched a Firestore collection. See Subscriptions for the historical reference.

Execution lifecycle push

... worker emits events on NATS subject playbooks.executions.<id>.step.exit ...
playbook_state.rs receives -> ConnectionHub::send_to_client
SPA <- SSE: playbook/state { execution_id, event_type, step_name, at }
... again for playbook.completed ...
SPA <- SSE: playbook/state { execution_id, event_type: playbook.completed, at }

Related

• SSE events — full frame schema reference, including FORWARDED_EVENT_TYPES.
• Subscriptions — historical reference for the removed Firestore subscription endpoint (v2.11.0 → v2.12.0).
• Configuration — env vars.
• Deployment — Docker, GKE, Helm.