architecture

Architecture

How the gateway fits into the NoETL stack and what each internal module does.

For the underlying principle, see Ephemeral Blueprints + Compute-Data Boundary. The gateway is the concrete expression of "gatekeeper only" in that model.

Position in the stack

┌─────────────────────────────┐
│   Client (browser, CLI,     │
│   partner integration)      │
└────┬───────────────────┬────┘
     │ session_token      │ SSE /events
     ▼                    ▼
┌──────────────────────────────────────────────────┐
│   noetl-gateway   (this repo)                    │
│                                                  │
│   - Auth: /api/auth/login, /api/auth/validate    │
│   - GraphQL: executePlaybook                     │
│   - Proxy: /noetl/* -> noetl-server              │
│   - SSE: /events fan-out per client_id           │
│   - Callbacks: /api/internal/callback{,async}    │
│   - Subscriptions: /api/subscriptions/firestore  │
│   - NATS bridge: playbook/state frames           │
└────┬─────────────────────────────────────────────┘
     │ HTTP /api/execute        Python sidecar
     │ NATS subscribe              Firestore watch
     ▼                          ▲
┌──────────────────────────────────────────────────┐
│   noetl-server (FastAPI) + worker pool           │
│   noetl-server publishes execution events on     │
│   NATS subject prefix `playbooks.executions.`    │
│   the gateway subscribes and re-emits as         │
│   playbook/state SSE frames.                     │
└──────────────────────────────────────────────────┘

Every arrow is a real network hop. The gateway never reaches a domain database; the worker pool never reaches the browser; the browser never reaches the worker pool directly.

Module map

src/main.rs

Application bootstrap: builds the axum router, wires middleware (auth, CORS, tracing), and starts the server.

src/auth/

• login: Auth0 ID-token exchange. Dispatches the api_integration/auth0/auth0_login playbook in noetl-server via the gateway-to-noetl proxy. On success, persists the session and returns the session token.
• validate: confirms a presented session token is still valid.
• middleware: extracts the session token from every authenticated request, verifies it, and attaches the authenticated user record to the request context.

src/graphql/

GraphQL endpoint at POST /graphql. The primary mutation is executePlaybook(path, workload):

1. Resolves the authenticated user and client_id.
2. Stores request_id → { client_id, session_token, execution_id, playbook_path } in RequestStore (NATS K/V).
3. POSTs to noetl-server's /api/execute to start the playbook.
4. Returns { requestId, executionId } immediately. The client awaits completion over the SSE channel rather than blocking the HTTP request.

src/sse.rs + src/connection_hub.rs

• GET /events?session_token=...&client_id=... opens an SSE stream. The handler validates the session, registers the client in ConnectionHub, sends an init message frame containing the assigned client_id, and merges per-client messages with periodic ping heartbeats.
• ConnectionHub is the in-memory registry of client_id → channel for fan-out. send_to_client is the universal outbound primitive.

src/callbacks.rs

• POST /api/internal/callback receives synchronous callbacks from playbooks (e.g. the send_success_callback step in auth0_login). Looks up the request_id in RequestStore and routes the result to the corresponding SSE client as a playbook/result frame.
• POST /api/internal/callback/async handles asynchronous callbacks the same way.

src/playbook_state.rs

NATS subscriber that listens on subjects matching playbooks.executions.*.{step.exit,playbook.completed,playbook.failed} (prefix from NATS_UPDATES_SUBJECT_PREFIX). For each frame:

1. Looks up which client_id cares about the execution_id (the subscriber maintains a per-client filter map seeded by executePlaybook and subscribeToExecution calls).
2. Forwards the frame as a playbook/state SSE event to that client.

Added in v2.11.0. This lets the SPA-side waitForExecution move off polling onto a push model.

src/firestore_subscriptions.rs

Implements POST /api/subscriptions/firestore and DELETE /api/subscriptions/{subscription_id}. The gateway itself does not import a Firestore Admin SDK; it spawns a Python sidecar process (see requirements-firestore.txt) that watches the requested Firestore path and streams JSON lines back to the gateway. The gateway forwards each line as a subscription/event SSE frame to the requesting client.

The auth check before opening a watch confirms the requested path is under the authenticated user's tenant (typically chat_threads/<thread_id>/...).

Added in v2.11.0. See Subscriptions.

src/request_store.rs

NATS K/V store keyed by request_id. Holds the in-flight mapping needed for callback routing. Cleared when the corresponding playbook/result frame is delivered.

src/proxy.rs

Proxies authenticated requests under /noetl/* to noetl-server, attaching the session token's user context as headers. Lets CLIs and other clients use the same auth model as the SPA without re-implementing NoETL API calls.

src/session_cache.rs

Short-TTL in-memory cache for validated session tokens. Avoids hammering noetl-server's auth.sessions table on every request.

src/nats.rs

NATS connection setup and helpers. Reads NATS_URL from the environment. NATS connection itself is platform-runtime, not a business-logic credential (see secrets-and-credentials rule).

src/db/

Postgres pool wiring for the auth/session tables. Same platform-runtime classification.

Request flow examples

Authenticated playbook execution

SPA -> POST /api/auth/login { id_token }
SPA <- 200 { session_token, expires_at, user_id }
SPA -> GET /events?session_token=...
SPA <- SSE: message { result: { clientId } }
SPA -> POST /graphql { query: executePlaybook(...), variables }
SPA <- 200 { requestId, executionId }
... worker pool runs the playbook ...
playbook -> POST /api/internal/callback { request_id, status, data }
SPA <- SSE: playbook/result { requestId, executionId, status, data }

Subscription to live Firestore updates

SPA -> POST /api/subscriptions/firestore { path, scope }
SPA <- 200 { subscription_id, client_id }
... Python sidecar opens onSnapshot on the path ...
firestore change -> sidecar stdout JSON -> gateway
SPA <- SSE: subscription/event { subscription_id, doc_id, data, op }
SPA -> DELETE /api/subscriptions/{subscription_id}
... sidecar closes the watch ...

Execution lifecycle push

... worker emits events on NATS subject playbooks.executions.<id>.step.exit ...
playbook_state.rs receives -> ConnectionHub::send_to_client
SPA <- SSE: playbook/state { execution_id, event_type, step_name, at }
... again for playbook.completed ...
SPA <- SSE: playbook/state { execution_id, event_type: playbook.completed, at }

Related

• SSE events — full frame schema reference.
• Subscriptions — Firestore subscription endpoint and sidecar.
• Configuration — env vars.
• Deployment — Docker, GKE, Helm.