Home

noetl-worker

The NoETL worker pool: a NATS-driven pull-model executable that consumes command events from JetStream and runs them against the noetl-tools registry.

What the worker is

A stateless atomic compute block, in § H.10's terms: claim a command from NATS, hydrate inputs from the per-execution context, execute one tool, write outputs back to the event log via the Python control plane, release the slot.

• Stateless — no per-tenant resident state between commands.
• Pull model — subscribes to a JetStream durable consumer; no recursion.
• Replaceable — multiple workers process the same queue; KEDA scales them on real consumer lag.

The worker's twin is the noetl CLI: same tool dispatch surface, same execution semantics, but the CLI uses a recursive tree walker for local YAML execution where the worker uses a flat pull loop. See § H.10 for why the two control loops stay separate.

Crate dependency graph

noetl-worker (this crate)
   ├── noetl-tools = "2.8.7"        # tool registry (rhai, shell, http, duckdb, postgres, ...)
   ├── noetl-executor = "0.2"       # shared utilities + types (condition eval, event envelope,
   │                                #   tool dispatch bridge; ships from the noetl/cli workspace)
   ├── async-nats = "0.38"          # JetStream consumer
   ├── reqwest = "0.12"             # HTTP client for the control plane
   └── tokio = "1"                  # async runtime

noetl-executor and noetl-tools are the shared surface: both the CLI and the worker call into them. The worker keeps its NATS pull loop + control-plane HTTP client + pull-loop control flow (case evaluator, command dispatcher) local because those are worker-specific shapes per § H.10.

Module layout

Module	Purpose
worker	Main worker loop — claim, dispatch, ack/nack.
config	Runtime config (NATS URL, server URL, worker id, stream/consumer names).
nats::subscriber	JetStream consumer + CommandNotification envelope.
client::control_plane	HTTP client to the Python control plane — fetch commands, emit events, read/write vars.
events::emitter	EventEmitter with retry logic over ControlPlaneClient.emit_event.
executor::command	CommandExecutor — runs one tool, evaluates cases, emits lifecycle events.
executor::case_evaluator	Worker-specific pull-loop control flow (Case / CaseAction / CaseResult / CaseEvaluator). Condition primitives (Operator, Condition, evaluate_structured_condition) delegate to noetl_executor::condition as of R-1.2 PR-2c.

Pages

• noetl-executor adoption — what surfaces the worker imports from noetl-executor, what stays local, R-1.x landing history (parallels the cli's executor-crate-architecture page from the consumer side).
• Release pipeline — semantic-release → release-worker flow; the GitHub-Actions safeguard gotcha and the workaround.

Distribution channels

• Crates.io: noetl-worker
• Container image: GHCR/GCR (recommended primary runtime channel)
• Cloud Build: for image builds + GKE deploy

Related

• noetl-executor crate architecture — the shared crate's design + R-1.x sub-PR history.
• § H.10 — tree walker vs pull model
• Appendix H — Rust migration roadmap
• noetl/tools wiki — tool registry the worker dispatches through.
• noetl/server wiki — Python control plane the worker emits events to + fetches commands from.