Skip to content

noetl executor adoption

Jump to bottom
Kadyapam edited this page May 31, 2026 · 14 revisions

noetl-executor adoption

Companion to the cli wiki's executor-crate-architecture page from the worker side. Documents which surfaces the worker imports from noetl-executor, which stay worker-local, and the sub-PR landing history.

Why the worker depends on noetl-executor

§ H.10 of the Rust migration roadmap established that the noetl CLI and noetl-worker are fundamentally different control loops — recursive tree walker vs pull-model consumer — but share the same:

  • YAML playbook types (Step, Tool, NextFormat, …)
  • Template rendering rules ({{ workload.x }} substitution)
  • Case-condition operator semantics (Eq / Gt / Contains / Matches / …)
  • Event envelope shape (ExecutorEvent mirrors Python's noetl.event table)
  • Tool dispatch bridge onto the noetl-tools registry

noetl-executor is the home for that shared surface. The CLI ships it as a workspace member crate in noetl/cli; the worker depends on it via crates.io.

What the worker imports

Surface Used by worker module Notes
noetl_executor::condition::Operator executor::case_evaluator 12-variant enum re-exported for backward compatibility.
noetl_executor::condition::Condition executor::case_evaluator Structured { left, op, right } envelope.
noetl_executor::condition::evaluate_structured_condition executor::case_evaluator::CaseEvaluator::evaluate_conditions Per-condition evaluation.

Planned for R-1.2 PR-2d-2 (NATS subscriber adoption — blocked on noetl-executor 0.3.0 reaching crates.io):

Planned surface Will be used by Status
noetl_executor::worker::source::CommandSource new nats::source::NatsCommandSource (wraps NatsSubscriber + ControlPlaneClient) Trait the wrapper implements; Worker::process_commands driven through source.next() + source.ack() / source.nack().
noetl_executor::worker::source::ClaimOutcome NATS source next() return path 4-state enum (Claimed(Command) / AlreadyClaimed / RetryLater(String) / Failed(String)) — maps 1:1 onto the worker's pre-PR-2d ClaimResult.
noetl_executor::worker::source::Pulled<H> NATS source next() return Generic wrapper { outcome, ack: H } — H is the source's AckHandle associated type (async_nats::jetstream::Message for the NATS source).
noetl_executor::worker::source::Command translated from crate::client::Command at the source seam Enriched in 0.3.0 with render_context: HashMap<String, Value> + attempts: u32 (both #[serde(default)]). Worker's Command.render_context() / meta.attempts map cleanly onto these fields.

What stays worker-local

Worker-specific shapes that don't belong in the shared crate:

Worker-local surface Why
worker::Worker Pull-loop control flow. Claim, dispatch, ack/nack — pull-model semantics that the CLI's tree walker doesn't share.
nats::subscriber::NatsSubscriber NATS JetStream binding. CLI has no NATS surface.
client::control_plane::ControlPlaneClient HTTP client to the Python control plane. CLI uses local YAML, not a server.
events::emitter::EventEmitter Retry logic over ControlPlaneClient.emit_event. Worker-specific retry policy.
executor::case_evaluator::Case / CaseAction / CaseResult / CaseEvaluator Pull-loop control flow. CaseAction::{Continue, Exit, SetVar, Goto, Retry, Fail} are dispatch-side decisions the CLI's tree walker handles differently.
client::control_plane::WorkerEvent Wire format to the Python server's /api/events endpoint. Diverges from noetl_executor::events::ExecutorEvent; reconciliation is a tracked cross-repo follow-up.

R-1.2 sub-PR landing history (worker side)

Sub-PR Scope noetl-worker version PR
R-1.2 PR-2c First worker PR depending on noetl-executor. Adds noetl-executor = "0.2" dep; replaces inline Operator + Condition + per-condition resolution helpers with re-exports + delegation to evaluate_structured_condition. Keeps pull-loop control flow (Case / CaseAction / CaseResult / CaseEvaluator) per § H.10. case_evaluator.rs 437 → 344 LoC (-93 net); 3 new tests lock in worker contract (first_match_wins, no_match_returns_none, and_semantics). 1.1.0 noetl/worker#2

Sub-PRs in flight or pending

  • R-1.2 PR-2d-2 (next) — NatsCommandSource (wrapping subscriber + control-plane client) implements noetl_executor::worker::source::CommandSource 0.3.0; Worker::process_commands becomes generic over the trait. Blocked on noetl-executor 0.3.0 reaching crates.io via noetl/cli#35.
  • Event envelope reconciliation (cross-repo) — WorkerEvent ↔ server's EventEmitRequest ↔ executor's ExecutorEvent all diverge; coordinated PR series needed. Tracked on noetl/ai-meta#30.

Companion sub-PRs on noetl/cli

These ship via noetl/cli's release pipeline; the worker picks them up by bumping the noetl-executor semver requirement.

Sub-PR Scope crates.io version
R-1.2 PR-1 Publish noetl-executor 0.1.0 to crates.io; fix the cli's release pipeline. noetl/cli#32 0.1.0
R-1.2 PR-2a Align execution_id to i64 across events, runtime, worker::source. Breaking → 0.2.0. noetl/cli#33 0.2.0
R-1.2 PR-2b Add structured Condition + 12-variant Operator + evaluate_structured_condition to condition. Minor → 0.2.1. noetl/cli#34 0.2.1
R-1.2 PR-2d-1 Redesign worker::source::CommandSource trait with ack lifecycle + 4-state ClaimOutcome + Pulled<H> wrapper + associated AckHandle type. Enriches Command with render_context + attempts. Adds in-crate MockSource test helper. Breaking → 0.3.0. Bin auto-bumped to noetl 4.0.0. See the cli wiki's executor-crate-architecture page for the design-decisions table. noetl/cli#35 0.3.0

Versioning policy

The worker pins noetl-executor = "0.2" (semver-compatible to any 0.2.x). Patch releases of noetl-executor are picked up automatically on the next worker build; minor / major bumps require an explicit Cargo.toml change in the worker.

The cli + worker release cadences are independent. Worker is currently at 1.1.0 (one minor release ahead of 1.0.0); cli is at 3.1.0.

Related

noetl-worker

Architecture

Operations

Related repos

External

Clone this wiki locally