Home

noetl-worker

The NoETL worker pool: a NATS-driven pull-model executable that consumes command events from JetStream and runs them against the noetl-tools registry.

What the worker is

A stateless atomic compute block, in § H.10's terms: claim a command from NATS, hydrate inputs from the per-execution context, execute one tool, write outputs back to the event log via the Python control plane, release the slot.

• Stateless — no per-tenant resident state between commands.
• Pull model — subscribes to a JetStream durable consumer; no recursion.
• Replaceable — multiple workers process the same queue; KEDA scales them on real consumer lag.

The worker's twin is the noetl CLI: same tool dispatch surface, same execution semantics, but the CLI uses a recursive tree walker for local YAML execution where the worker uses a flat pull loop. See § H.10 for why the two control loops stay separate.

Crate dependency graph

noetl-worker (this crate)
   ├── noetl-tools = "2.16"         # tool registry (rhai, shell, http, duckdb, postgres,
   │                                #   result_fetch, nats, mcp, ...)
   ├── noetl-executor = "0.3"       # shared utilities + types (condition eval, event envelope,
   │                                #   tool dispatch bridge; ships from the noetl/cli workspace)
   ├── async-nats = "0.38"          # JetStream consumer
   ├── reqwest = "0.12"             # HTTP client for the control plane
   └── tokio = "1"                  # async runtime

noetl-executor and noetl-tools are the shared surface: both the CLI and the worker call into them. The worker keeps its NATS pull loop + control-plane HTTP client + pull-loop control flow (case evaluator, command dispatcher) local because those are worker-specific shapes per § H.10.

Module layout

Module	Purpose
worker	Main worker loop — claim, dispatch, ack/nack.
config	Runtime config (NATS URL, server URL, worker id, stream/consumer names).
nats::subscriber	JetStream consumer + CommandNotification envelope.
client::control_plane	HTTP client to the Python control plane — fetch commands, emit events, read/write vars.
events::emitter	EventEmitter with retry logic over ControlPlaneClient.emit_event.
executor::command	CommandExecutor — runs one tool, evaluates cases, emits lifecycle events.
executor::case_evaluator	Worker-specific pull-loop control flow (Case / CaseAction / CaseResult / CaseEvaluator). Condition primitives (Operator, Condition, evaluate_structured_condition) delegate to noetl_executor::condition as of R-1.2 PR-2c.

Pages

• noetl-executor adoption — what surfaces the worker imports from noetl-executor, what stays local, R-1.x landing history (parallels the cli's executor-crate-architecture page from the consumer side).
• Release pipeline — semantic-release → release-worker flow; the GitHub-Actions safeguard gotcha and the workaround.
• Worker credentials — NOETL_KEYCHAIN_ENV_VARS allow-list (5.7.0+, noetl/ai-meta#34) that lifts platform-mounted Secret env vars into the in-process keychain map for playbook ctx.get_secret(alias) lookups.
• nats and mcp tool kinds — playbook config shape, credential wiring, and dispatch path for the nats (2.15.0) and mcp (2.16.0) tool kinds picked up by the noetl-tools bump in this version.
• Deployment specification — durable reference for deploying noetl-worker: runtime contract, NATS layout, KEDA scaling, full env-var catalogue with rationale, snowflake node-id derivation, secrets handling, kind-validation procedure. Updated in lockstep with code per wiki-maintenance.md.

Distribution channels

• Crates.io: noetl-worker
• Container image: GHCR/GCR (recommended primary runtime channel)
• Cloud Build: for image builds + GKE deploy

Related

• noetl-executor crate architecture — the shared crate's design + R-1.x sub-PR history.
• § H.10 — tree walker vs pull model
• Appendix H — Rust migration roadmap
• noetl/tools wiki — tool registry the worker dispatches through.
• noetl/server wiki — Python control plane the worker emits events to + fetches commands from.