Home

noetl-server

The async Rust control plane for distributed NoETL orchestration. Axum + sqlx + PostgreSQL + NATS JetStream.

What this crate is

The Rust port of the NoETL control plane (the Python implementation lives in noetl/noetl under noetl/server/). Both implement the same external contract — same HTTP routes, same event-log table layout, same NATS subjects — so a deployment can flip between them.

noetl-server is responsible for:

• Workflow orchestration — execute playbooks; derive workflow state by replaying events.
• Catalog management — register + retrieve playbooks, tools, resources.
• Credential management — encrypted credential storage at rest; decryption on resolve.
• Event processing — ingest worker events; emit command notifications to NATS.
• Execution management — track + manage playbook executions.

Its primary consumers:

• The Rust noetl-worker pulls commands via NATS, fetches command details via the server's HTTP API, and emits events back to the server.
• The Python orchestrator (when running side-by-side during migration) operates on the same event log + NATS subjects.
• noetl/gateway forwards external requests to this server's HTTP API.

Architecture

Event-sourced. All state derives from events stored in PostgreSQL:

worker / CLI                  noetl-server                 PostgreSQL
    │                              │                            │
    ├── POST /api/events ──────────┤                            │
    │                              ├── INSERT into noetl.event ─►
    │                              │                            │
    │                              ├── orchestrator evaluates ──┤
    │                              │                            │
    │                              ├── INSERT noetl.command ────►
    │                              │                            │
NATS JetStream ◄──────────────────┘                            │
    │                                                            │
    └── notification ──────────────► worker claims               │
                                          │                      │
                                          ├── GET /api/commands/<id> ◄─┘

Module layout

Module	Purpose
config	Environment + DB + NATS configuration.
db	sqlx pool + models (noetl.event + noetl.command + noetl.execution tables).
error	Custom error enum with Axum IntoResponse integration.
handlers::events	POST /api/events — ingests worker events; appends to event log; emits notifications via NATS.
handlers::commands	GET /api/commands/<id> — returns full command details to a claiming worker.
handlers::catalog	Playbook / tool / resource catalog routes.
handlers::credentials	Encrypted credential CRUD; integrates with crypto.
handlers::executions	Execution tracking; replays events to project state.
handlers::variables	GET/POST /api/vars/<execution_id>/<name> — per-execution variable storage.
handlers::keychain	Keychain integration for credential aliases.
handlers::runtime	Runtime capability metadata (/api/runtime/...).
handlers::dashboard	Dashboard / health / system info.
engine::commands	Command generation from events.
engine::evaluator	Case / when / then evaluation (server-side).
engine::orchestrator	Drives execution forward in response to events.
engine::state	Event-log projection helpers.
nats	JetStream publisher for command notifications.
playbook	Playbook parser + types.
crypto	Credential encryption at rest.

Pages

• Event envelope — wire format for POST /api/events; the four-shape divergence currently in flight (server EventRequest ↔ Python EventEmitRequest ↔ worker WorkerEvent ↔ executor ExecutorEvent); reconciliation tracked on noetl/ai-meta#30.
• Event-sourced execution — how events drive state and produce command notifications.
• API surface — full HTTP route inventory (/api/events, /api/commands, /api/catalog, /api/credentials, /api/executions, /api/vars, /api/keychain, /api/runtime).
• Runtime shape (compiled + plug-in ring) — proposed. The four-binary crate layout (server / publisher / projector / system_pool), the compiled-core vs WASM-plug-in split, the system worker pool, and the sequencing for the full Python → Rust migration. Tracked under noetl/ai-meta#45 + noetl/ai-meta#46.
• Sharding design (Phase F) — Phase F of noetl/ai-meta#49. Natural shard boundary is the execution; hash(execution_id) % N on the full snowflake; gateway-aware routing for F; single-master cluster-wide tables. R1 (design + endpoint inventory codification) shipped via noetl/server#40.
• Deployment specification — durable reference for deploying noetl-server: runtime contract, resource sizing, ports, health probes, full env-var catalogue with rationale, secrets handling, observability, kind-validation procedure. Updated in lockstep with code per wiki-maintenance.md.

Relationship to the Python control plane

Both implementations exist concurrently. Python lives in noetl/noetl; Rust is this crate. Migration plan:

1. Today: Python serves prod; Rust serves dev / local-kind validation.
2. R-1.3 → R-2: feature-parity sweep; same event-log table, same NATS subjects, same external API.
3. Eventually: deprecate the Python path once Rust has parity + sufficient cluster runtime.

The migration cadence is driven by the broader Appendix H roadmap.

Distribution channels

• Crates.io: noetl-server
• Container image: GHCR/GCR (recommended primary runtime channel)
• Cloud Build: for image builds + GKE deploy

Related

• noetl/cli wiki — executor-crate-architecture — shared execution core both CLI + worker depend on.
• noetl/worker wiki — NATS pull consumer that emits events here.
• noetl/tools wiki — tool registry both worker + CLI dispatch through.
• noetl/noetl wiki — Python implementation (twin of this crate during migration).
• noetl/ops wiki — Helm + manifests for deploying this server.
• § H — Rust migration roadmap