executor crate architecture

noetl-executor workspace crate

Since v2.17.1+ the CLI is a Cargo workspace. The root crate (noetl, producing the noetl and ntl binaries) is unchanged; a new workspace-member crate noetl-executor hosts the shared execution utilities and types that both this CLI and the noetl-worker daemon (R-1.3) depend on.

This page documents the executor crate's module layout, what moved out of playbook_runner.rs and why, and the deliberate architectural decision that the CLI keeps its own control loop rather than sharing one with the worker.

Source layout

repos/cli/
  Cargo.toml                    # workspace root + noetl binary
  src/
    main.rs
    playbook_runner.rs          # CLI tree walker (control loop stays here)
    ...
  executor/                     # workspace member: noetl-executor
    Cargo.toml
    src/
      lib.rs
      playbook.rs               # YAML playbook types
      template.rs               # render_template + Rhai/JSON helpers
      condition.rs              # evaluate_condition + evaluate_rhai_condition
      capabilities.rs           # validate_capabilities + ValidationReport
      runtime.rs                # ExecutionContext + CredentialResolver trait
      events.rs                 # EventSink trait + ExecutorEvent + EventEmitter
      tools_bridge.rs           # noetl-tools registry bridge (scaffold)
      worker/
        mod.rs
        source.rs               # Command + CommandSource (worker-only)

What lives in noetl-executor and why

Module	Purpose	Used by
playbook	Pydantic-like YAML types: Playbook, Step, Tool, NextFormat, RuntimeCapabilities, etc. All field accessors pub.	CLI + worker
template	render_template, render_template_with_result, get_json_path, json_to_rhai, rhai_to_json_string. Takes &HashMap<String, String> views of the per-execution variables + step results so each binary owns its own context shape.	CLI + worker
condition	evaluate_condition (simple {{ a == b }} / 'in' / truthy) and evaluate_rhai_condition (full Rhai expression eval). Same context-view contract as template.	CLI + worker
capabilities	validate_capabilities + ValidationReport / ValidationError. Pure function: returns the report rather than bail!ing so the CLI can format errors with playbook_path and the worker with execution_id.	CLI + worker
runtime	ExecutionContext (executor-side variant with async step_results + Arc<dyn CredentialResolver>); CredentialResolver trait.	worker (CLI uses its own ExecutionContext in playbook_runner.rs)
events	ExecutorEvent (mirrors the Python noetl.runtime.events.report_event envelope), EventSink trait, NoopSink, EventEmitter.	CLI + worker
tools_bridge	Scaffold for replacing the CLI's inline tool implementations with calls into the noetl-tools registry. Filled in incrementally per Strategy B (one tool kind per sub-PR).	CLI (worker already uses noetl-tools directly)
worker::source	Command envelope + CommandSource trait. Worker-only: the CLI's tree walker doesn't consume this.	worker (R-1.3)

What does NOT live in noetl-executor

	Where it lives	Why
The CLI's recursive tree walker (run, execute_step, execute_next_steps, execute_router_arcs)	repos/cli/src/playbook_runner.rs	Natural fit for local YAML execution; flattening into a pull-model iterator would lose local-debug clarity
The CLI's inline tool implementations (execute_tool, execute_shell_command, execute_http_request, execute_duckdb_query, etc.)	repos/cli/src/playbook_runner.rs (today)	Migrated incrementally to noetl-tools registry via tools_bridge per Strategy B; ~870 LoC of inline tool dispatch
The worker's NATS pull loop	repos/worker/src/	Different shape than tree walker; tied to NATS durable-consumer semantics
RunOutcome (the CLI's JSON output envelope)	repos/cli/src/playbook_runner.rs	Not a YAML input type; worker has a different output envelope (event-log writes)

Architectural finding (§ H.10 of Appendix H)

Mid-implementation discovery, documented in § H.10 of the global hybrid cloud blueprint:

• The CLI is a recursive tree walker. It loads the YAML, walks the workflow, evaluates next arcs / case conditions / then blocks in place, dispatches each step inline. Control flow is the call stack.
• The worker is a pull-model consumer. It subscribes to a NATS durable consumer, pulls one command at a time, executes it, emits events, repeats. No tree. No recursion.
• The original "unified CommandSource trait — both binaries plug in their own impl" was the wrong abstraction for the CLI. Flattening the tree walker into a pull-model iterator loses local-debug clarity, complicates case / then state management, and breaks integration tests written against the tree shape.

noetl-executor was re-scoped from a control-loop crate to a utilities-and-types crate. The CLI's tree walker stays. The worker's pull loop stays. Both call into the executor for the same template rendering, condition evaluation, credential resolution, capability validation, and event shape.

Cross-references

Topic	Where
The migration roadmap	Appendix H of the global hybrid cloud blueprint
The architectural finding	§ H.10 of the same doc
The Polars-pattern endpoint (pip install noetl ships Rust runtime + Python wrapper)	§ H.9
Apache Arrow data plane	§ H.4 + § H.11 (local-mode Feather buffer)
Tracking issues	noetl/cli#19 (this CLI's R-1.1 sub-issue) · noetl/ai-meta#30 (umbrella)

Sub-PR landing history

Sub-PR	Scope	PR
R-1.1 PR-1	Crate skeleton (lib.rs, runtime, events, source/dispatch placeholders)	#20
R-1.1 PR-2a	YAML types → noetl-executor::playbook	#21
R-1.1 PR-2b	Utilities (template + condition + capabilities); § H.10 restructure (placeholder LocalPlaybookSource removed; CommandSource → worker::source)	#22
R-1.1 PR-2c-1	noetl-tools = "2.8.7" dep + tools_bridge scaffold	#23
R-1.1 PR-2c-2	tools_bridge adapters: BridgeContext, to_tools_context, to_tools_config (all 8 Tool variants), from_tools_result; dispatch_via_registry becomes async with per-tool-kind match arms	#24

After PR-2c-2, the next ~6 sub-PRs replace one inline tool implementation at a time so semantic differences surface incrementally. PR-2d closes noetl/cli#19 with documentation + integration-test pass.

Stats since the workspace landed

• playbook_runner.rs: 2,688 → 1,964 lines (-724 net across PR-2a + PR-2b)
• New code in noetl-executor: 7 modules + 1 worker submodule, ~1,850 LoC
• noetl-executor unit tests: 0 → 29 across PR-1 + PR-2a + PR-2b + PR-2c-1 + PR-2c-2
• Workspace-wide tests: 41 passing (29 noetl-executor + 12 noetl bin)