Skip to content

Home

Jump to bottom
Kadyapam edited this page Jun 15, 2026 · 6 revisions

noetl-server

The async Rust control plane for distributed NoETL orchestration. Axum + sqlx + PostgreSQL + NATS JetStream.

What this crate is

The Rust port of the NoETL control plane (the Python implementation lives in noetl/noetl under noetl/server/). Both implement the same external contract — same HTTP routes, same event-log table layout, same NATS subjects — so a deployment can flip between them.

noetl-server is responsible for:

  • Workflow orchestration — execute playbooks; derive workflow state by replaying events.
  • Catalog management — register + retrieve playbooks, tools, resources.
  • Credential management — encrypted credential storage at rest; decryption on resolve.
  • Event processing — ingest worker events; emit command notifications to NATS.
  • Execution management — track + manage playbook executions.

Its primary consumers:

  • The Rust noetl-worker pulls commands via NATS, fetches command details via the server's HTTP API, and emits events back to the server.
  • The Python orchestrator (when running side-by-side during migration) operates on the same event log + NATS subjects.
  • noetl/gateway forwards external requests to this server's HTTP API.

Architecture

Event-sourced. All state derives from events stored in PostgreSQL:

worker / CLI                  noetl-server                 PostgreSQL
    │                              │                            │
    ├── POST /api/events ──────────┤                            │
    │                              ├── INSERT into noetl.event ─►
    │                              │                            │
    │                              ├── orchestrator evaluates ──┤
    │                              │                            │
    │                              ├── INSERT noetl.command ────►
    │                              │                            │
NATS JetStream ◄──────────────────┘                            │
    │                                                            │
    └── notification ──────────────► worker claims               │
                                          │                      │
                                          ├── GET /api/commands/<id> ◄─┘

Module layout

Module Purpose
config Environment + DB + NATS configuration.
db sqlx pool + models (noetl.event + noetl.command + noetl.execution tables).
error Custom error enum with Axum IntoResponse integration.
handlers::events POST /api/events — ingests worker events; appends to event log; emits notifications via NATS.
handlers::commands GET /api/commands/<id> — returns full command details to a claiming worker.
handlers::catalog Playbook / tool / resource catalog routes.
handlers::credentials Encrypted credential CRUD; integrates with crypto.
handlers::executions Execution tracking; replays events to project state.
handlers::variables GET/POST /api/vars/<execution_id>/<name> — per-execution variable storage.
handlers::keychain Keychain integration for credential aliases.
handlers::runtime Runtime capability metadata (/api/runtime/...).
handlers::dashboard Dashboard / health / system info.
engine::commands Command generation from events.
engine::evaluator Case / when / then evaluation (server-side).
engine::orchestrator Drives execution forward in response to events.
engine::state Event-log projection helpers.
nats JetStream publisher for command notifications.
playbook Playbook parser + types.
crypto Credential encryption at rest.

Pages

  • Event envelope — wire format for POST /api/events; the four-shape divergence currently in flight (server EventRequest ↔ Python EventEmitRequest ↔ worker WorkerEvent ↔ executor ExecutorEvent); reconciliation tracked on noetl/ai-meta#30.
  • Event-sourced execution — how events drive state and produce command notifications.
  • API surface — full HTTP route inventory (/api/events, /api/commands, /api/catalog, /api/credentials, /api/executions, /api/vars, /api/keychain, /api/runtime).
  • Runtime shape (compiled + plug-in ring) — proposed. The four-binary crate layout (server / publisher / projector / system_pool), the compiled-core vs WASM-plug-in split, the system worker pool, and the sequencing for the full Python → Rust migration. Tracked under noetl/ai-meta#45 + noetl/ai-meta#46.
  • Sharding design (Phase F) — Phase F of noetl/ai-meta#49. Natural shard boundary is the execution; hash(execution_id) % N on the full snowflake; gateway-aware routing for F; single-master cluster-wide tables. R1 (design + endpoint inventory codification) shipped via noetl/server#40.
  • Cursor / claim loop mode — loop.spec.mode: cursor orchestrator semantics: entry → claim → fan-out body per row → re-claim → drain; LoopMode::Cursor, CursorClaim, FrameSpec; the output namespace for arc when: / step set:; difference from sequential/parallel loops and from the Python frame-leasing model. Added in v3.8.0 (noetl/server#196); validated via the full PFT flow (noetl/ai-meta#100).
  • Deployment specification — durable reference for deploying noetl-server: runtime contract, resource sizing, ports, health probes, full env-var catalogue with rationale, secrets handling, observability, kind-validation procedure. Updated in lockstep with code per wiki-maintenance.md.

Relationship to the Python control plane

Both implementations exist concurrently. Python lives in noetl/noetl; Rust is this crate. Migration plan:

  1. Today: Python serves prod; Rust serves dev / local-kind validation.
  2. R-1.3 → R-2: feature-parity sweep; same event-log table, same NATS subjects, same external API.
  3. Eventually: deprecate the Python path once Rust has parity + sufficient cluster runtime.

The migration cadence is driven by the broader Appendix H roadmap.

Distribution channels

  • Crates.io: noetl-server
  • Container image: GHCR/GCR (recommended primary runtime channel)
  • Cloud Build: for image builds + GKE deploy

Related

noetl-server

Reference

Design

Operations

Related repos

External

Clone this wiki locally