Skip to content

Architecture

Jump to bottom
kadyapam edited this page Oct 17, 2026 · 8 revisions

Architecture

How a kind: agent step executes in the NoETL local runtime, which component owns what, and why it is shaped this way.

Components

flowchart LR
    T[systemd timer] -->|noetl run --runtime local| R
    subgraph CLI ["noetl CLI (noetl/cli)"]
        R[PlaybookRunner<br/>src/playbook_runner.rs]
        B[tools_bridge<br/>executor/src/tools_bridge.rs]
        R --> B
    end
    subgraph TOOLS ["noetl-tools (noetl/tools)"]
        S[AgentSession<br/>src/tools/agent.rs]
    end
    B --> S
    S -->|framework: claude<br/>HTTPS + x-api-key| API[(Anthropic<br/>Messages API)]
    S -->|framework: claude-code<br/>subprocess| CC[claude -p]
    CC --> SUB[(Claude account)]
    R -->|tool call| P[tool sub-playbook<br/>PlaybookRunner]
Loading
Component Owns Source
PlaybookRunner walking the workflow, rendering templates, driving the agent loop, running tool sub-playbooks src/playbook_runner.rs (Tool::Agent arm, run_agent_tool_call)
tools_bridge mapping the YAML step to a noetl-tools config; local credential resolution executor/src/tools_bridge.rs (start_agent_session, local_credential_env_var)
Tool::Agent the YAML shape of the step executor/src/playbook.rs
AgentSession talking to the model: request building, response parsing, turn and token accounting, the Claude Code subprocess src/tools/agent.rs
AgentTool registry entry agent for tool-free agents (used by any host that cannot run tools) same file

Links point at main: noetl-tools 4.1.0 (noetl/tools#112) and noetl CLI 5.1.0 (noetl/cli#95).

framework: claude — the turn loop

sequenceDiagram
    participant R as PlaybookRunner
    participant S as AgentSession
    participant A as Messages API
    participant P as tool sub-playbook
    R->>S: start_agent_session(rendered step)
    Note over S: parse config, resolve auth,<br/>messages = [user: prompt]
    loop until Final or max_turns
        R->>S: next_turn()
        S->>A: POST /v1/messages (model, system, messages, tools)
        A-->>S: content blocks + stop_reason + usage
        alt tool_use blocks present
            S-->>R: ToolCalls([...])
            loop each call
                R->>P: run tool's playbook, input as workload.*
                P-->>R: RunOutcome (final_result or error)
            end
            R->>S: submit_tool_results([...])
        else text only
            S-->>R: Final({text, turns, usage, ...})
        end
    end
    R->>R: store step result
Loading
  1. Render. The runner renders the step's templates (prompt, system, model, tools, auth, …) against the playbook context, the same way every other tool is rendered.
  2. Start. start_agent_session resolves auth.credential locally (see Credentials), builds the noetl-tools config, and calls AgentSession::start, which validates the config and seeds messages with the prompt.
  3. Turn. Each next_turn is one POST /v1/messages. The assistant message is appended to the history; tool_use blocks become AgentToolCalls. A turn with no tool calls is final.
  4. Tools. For each call the runner looks up the tool by name, resolves its playbook: relative to the agent playbook, and runs it with a fresh PlaybookRunner. The model's input object becomes workload.<key> variables. The sub-playbook's final_result (or its step results, or its error) becomes the tool_result content. A failing tool is reported to the model with is_error: true; it does not fail the step, so the model can recover.
  5. Bounds. max_turns counts model calls. The (max_turns + 1)-th next_turn fails the step with agent did not finish within max_turns=N.

Why turn-by-turn instead of one async call

The agent's tools are playbooks, and the CLI's PlaybookRunner::run is synchronous and enters tokio itself (block_in_place + block_on). Running a sub-playbook from inside an async agent loop would nest block_on calls. So the session exposes one model turn per call, and the runner runs tools between turns, outside any block_on, exactly as a kind: playbook step does.

The same shape matters beyond the CLI: the session's whole state is its JSON message history, so a distributed runner could persist it between turns instead of holding a worker slot while a tool runs. That is what NoETL's execution model asks of any long-running block.

framework: claude-code — one subprocess

sequenceDiagram
    participant R as PlaybookRunner
    participant S as AgentSession
    participant C as claude -p
    R->>S: start_agent_session (no auth, no tools allowed)
    R->>S: next_turn()
    S->>C: spawn with args, env minus ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN
    S->>C: prompt on stdin
    Note over C: Claude Code runs its own loop<br/>with allowed_tools / mcp_config only
    C-->>S: one JSON result on stdout
    S-->>R: Final({text, turns, usage, framework, permission_denials})
Loading
  • Arguments: -p --output-format json --no-session-persistence --model <model>, plus --append-system-prompt <system>, --strict-mcp-config --mcp-config <files…>, --allowed-tools <tools…> when set. --append-system-prompt keeps Claude Code's own system prompt, which carries its tool instructions.
  • Prompt on stdin, never as an argument: --allowed-tools and --mcp-config take variable-length lists that would swallow a trailing positional prompt.
  • Billing guard: the two API-key variables are removed from the child's environment, so a key exported for framework: claude playbooks cannot silently switch this run to API billing.
  • Result: the CLI's JSON result becomes text; is_error: true fails the step with that text; non-JSON output fails with the exit status and the tail of stderr.
  • Bound: the step timeout (default 300 s); the child is killed on timeout.
  • No NoETL tools: Claude Code's tool calls happen inside the subprocess, invisible to NoETL. That is why tools: is refused here — to have NoETL see and gate every tool call, use framework: claude.

Credentials in local mode

flowchart LR
    Y["auth:<br/>credential: anthropic"] --> V{"$NOETL_CREDENTIAL_ANTHROPIC<br/>non-blank?"}
    V -->|yes| X[value]
    V -->|no| F{"~/.config/noetl/credentials/anthropic<br/>exists, mode 600?"}
    F -->|yes| X
    F -->|missing or blank| E[step fails before any call]
    F -->|readable by others| E
    X -->|agent| H[x-api-key header]
    X -->|"http google_oauth"| R[refresh → cached access token] --> B[Bearer header]
Loading
  • The playbook names an alias, never a key. This follows NoETL's no-default-connection rule: no auth:, no call.
  • Local mode has no server keychain, so the alias X resolves from the environment variable NOETL_CREDENTIAL_ + X upper-cased with non-alphanumerics as _, else from the file ~/.config/noetl/credentials/X, which must be mode 600. The file store needs noetl 5.2.0 (noetl/cli#96).
  • The same resolver serves agent steps and http steps (type: bearer | api_key | google_oauth); source: executor/src/local_credentials.rs.
  • google_oauth credentials hold a refresh token. The bridge exchanges it at oauth2.googleapis.com/token and caches the access token per alias until a minute before it expires, so an agent making many Gmail calls in one run refreshes once.
  • Keys are never in a unit file, a playbook, the process arguments, or step results.

Data flow and templates

  • The step config is rendered once, before the session starts.
  • Model output is data. Tool inputs from the model are passed to the sub-playbook verbatim — not rendered by the agent runner — and {{ is rewritten to { {. The rewrite was added because render_template up to noetl 5.3.0 re-expanded values it had already substituted, so {{ workload.secret }} inside an email quoted back by the model would have been expanded by the tool playbook's own templates. noetl 5.3.1 renders in a single pass (noetl/cli#98); the rewrite stays as defence in depth.
  • In playbooks that put fetched data into the prompt ({{ fetch.result }}), the CLI substitutes it once (noetl 5.3.1+). Since 5.3.2, shell / http steps are no longer re-rendered by noetl-tools (noetl/ai-meta#477), so that data reaches the tool unchanged.

The step result

Stored as the step's result (a JSON string), so later steps can use {{ <step>.result }}:

Field claude claude-code
text final text blocks joined CLI result
turns model calls made CLI num_turns
usage.input_tokens / output_tokens summed over turns as the CLI reports
stop_reason API stop_reason CLI terminal_reason / subtype
model ✓ ✓
framework — "claude-code"
permission_denials — tools Claude Code tried and was refused

What this does not do (yet)

  • Distributed runtime. The NoETL server rejects agent at playbook registration and workers don't dispatch it. Running agents on the worker pool needs a callback-shaped design so a model turn doesn't hold a slot.
  • Streaming, prompt caching, extended thinking are not wired into the claude loop.

Related

agent-playbooks

Code

Clone this wiki locally