Expressive Language RAG

Expressive Language (.rag)

The expressive language is TinyAgents' declarative blueprint format. A .rag file describes an agent graph — its start node, state channels, nodes, and routing — as compact, side-effect-free source text. It compiles into the same graph and harness structures as hand-written Rust: the runtime never knows whether a graph came from a Rust builder or from a .rag source string.

Crucially, .rag source can only reference capabilities by name — models, tools, subgraphs, routers, reducers — that Rust has already registered and allowed. It can never define behaviour or embed host code. That property is what makes .rag the safe boundary for agent-authored plans: a language model can emit a blueprint, and the same compiler + registry gate that validates a human-written file validates the model's output before anything runs. See Recursion and RLM for where self-authoring fits in the recursive picture.

Source lives in src/language/; the module spec is docs/modules/expressive-language/README.md.

The pipeline

A .rag file flows through four fixed phases. Each phase is a separate submodule so callers can stop at the level of safety they need:

source  -->  lexer  -->  tokens  -->  parser  -->  AST  -->  compiler  -->  Blueprint
        (lexer.rs)              (parser.rs)              (compiler.rs)

Phase	Module	Input	Output	Validates
Lex	language::lexer	&str	Vec<SpannedToken>	tokens, strings, numbers, // comments
Parse	language::parser	tokens	Program (AST)	structure: well-formed blocks, expected tokens
Compile	language::compiler::compile	Program	Vec<Blueprint>	semantics: duplicate names, start/targets, routing
Bind	language::compiler (resolver)	Blueprint + registry	() (checked)	capability references against the registry

The compiler then offers a final, optional phase — build_graph — that materialises a Blueprint into a runnable CompiledGraph using a Rust-supplied NodeFactory. Behaviour is always Rust's job; the blueprint only describes topology.

1. Lexer (lexer.rs)

tokenize(source) turns text into [SpannedToken]s, each carrying a 1-based line/column [Span]. The token set is deliberately tiny:

• identifiers / keywords: [A-Za-z_][A-Za-z0-9_]* (keywords like graph, node, start are lexed as identifiers and recognised contextually)
• numbers: integer or decimal, optionally signed (50, 1.5, -3)
• double-quoted strings with \n, \t, \r, \\, \" escapes
• punctuation: { } [ ] , and the arrow ->
• // line comments (skipped)

Lexical errors (unterminated string, invalid escape, malformed number, stray character) surface as TinyAgentsError::Parse with the offending line/column.

2. Parser (parser.rs)

parse(&tokens) (or the one-shot parse_str(source)) is a small hand-written recursive-descent parser. It performs structural validation only — expected-token checks and well-formed blocks — and produces a Program AST. It does not check whether names exist or routes are valid; that is the compiler's job. Errors are TinyAgentsError::Parse with the span of the offending token.

3. Compiler (compiler.rs)

compile(&program) lowers each graph declaration into one serializable [Blueprint] and runs the semantic checks:

• duplicate node names within a graph are rejected,
• a graph must declare a start node, and it must be defined,
• every next, route, and edge target must be a defined node or the reserved END,
• a node may use static routing (next / an incident edge) or command routing (routes), never both.

Failures are TinyAgentsError::Compile. Routing precedence when lowering a node is: explicit routes > next > top-level edge > terminal. A next END or an edge to END becomes [Routing::Terminal].

4. Capability binding (the registry gate)

A compiled Blueprint is inert until its references are checked against what Rust has registered. This is the safety boundary, and there are two paths:

• Minimal / manual — bind_capabilities(&blueprint, &resolver) checks only model and tool references against a CapabilityResolver allowlist. Build one with CapabilityResolver::from_lists(models, tools) or the chaining allow_model / allow_tool helpers.
• Strict / registry-backed — bind_capabilities_with_registry(&blueprint, &registry) builds a fully populated resolver from a live CapabilityRegistry (models, tools, subgraphs, routers, reducers, plus the default node kinds) and runs CapabilityResolver::bind_blueprint. This additionally validates node kinds, subgraph/router references, and channel reducers.

The default recognised node kinds are agent, model, tool_executor, subgraph, graph, router, human (DEFAULT_NODE_KINDS). Reference conventions used by the strict path:

• subgraph / graph: the node's model field names a registered subgraph blueprint,
• router: the model field names a registered router function,
• everything else: the model field names a registered chat model.

An unknown kind is a Compile error; the first unregistered model/tool/subgraph/ router/reducer reference is a Capability error.

The convenience façade compile_source(source, &registry) runs the whole chain — parse -> compile -> registry-bind — and returns validated blueprints in one call.

Materialising into the runtime

Blueprint  --build_graph(&blueprint, &factory)-->  CompiledGraph<State, State>

build_graph walks each NodeSpec, asks the caller's NodeFactory::make for a runnable handler, and wires routing into durable graph topology:

• Routing::Next(target) → GraphBuilder::add_edge (a static successor),
• Routing::Conditional(_) → GraphBuilder::mark_command_routing (the node decides its route at runtime by returning a Command goto),
• Routing::Terminal → GraphBuilder::set_finish (route to END).

The blueprint's start node becomes the graph entry. Because the factory is the only source of node behaviour, declarative source can never smuggle in arbitrary code.

The Blueprint artifact

A [Blueprint] is the inspectable, fully serializable output of the compiler. It can be stored, diffed, reviewed in a UI, and reloaded independently of the source text — which is exactly what the agent-authoring and review workflows need.

Blueprint {
  graph_id: String,            // the graph name
  start:    String,            // validated start node
  channels: Vec<ChannelSpec>,  // { name, reducer }
  nodes:    Vec<NodeSpec>,     // { name, kind, model?, prompt?, tools, routing }
  edges:    Vec<EdgeSpec>,     // { from, to }
  defaults: Vec<(String, Literal)>,
}

NodeSpec::routing is one of Next(target), Conditional([(label, target), …]), or Terminal. An unspecified node kind defaults to "model" during compilation.

Grammar

The implemented v1 grammar (what the parser actually accepts) is:

program       = graph_decl*
graph_decl    = "graph" ident "{" graph_item* "}"

graph_item    = "start" ident
              | "defaults" "{" ( ident literal )* "}"
              | "channel" ident ident          // channel <name> <reducer>
              | node_decl
              | edge_decl                       // ident "->" ident

node_decl     = "node" ident "{" node_item* "}"
node_item     = "kind"   ident
              | "model"  string
              | "system" string                 // alias for `prompt`
              | "prompt" string
              | "tools"  "[" ( string ("," string)* )? "]"
              | "next"   ident
              | "routes" "{" ( ident "->" ident )* "}"

literal       = string | number | ident

Notes:

• END is a reserved terminal target; it is written as a bare identifier.
• system and prompt both populate a node's prompt; system is accepted as an alias.
• The broader spec sketches future primitives (command, sends, join, interrupt, metadata, retry, timeout, steering, capability allow-lists, state schemas). The v1 parser above is the safe subset that is actually implemented; the AST and Blueprint leave room for the rest.

A worked example

// A support workflow with a tool loop.
graph support_agent {
  start agent

  defaults {
    recursion_limit 50
    backoff "exponential"
    checkpoint inherit
  }

  channel messages messages
  channel tool_calls append

  node agent {
    kind agent
    model "default"
    system "Resolve support requests using tools when useful."
    tools ["lookup_user", "create_ticket"]
    routes {
      tool_call -> tools
      final -> END
    }
  }

  node tools {
    kind tool_executor
    next agent
  }
}

Compiling and binding it from Rust (see the runnable examples/rag_blueprint.rs):

use tinyagents::language::parser::parse_str;
use tinyagents::language::compiler::{compile, bind_capabilities, CapabilityResolver};

let program   = parse_str(SUPPORT_AGENT)?;        // lex + parse
let blueprint = compile(&program)?.remove(0);     // semantic compile

let allow = CapabilityResolver::from_lists(
    ["default".to_string()],                       // allowed models
    ["lookup_user".to_string(), "create_ticket".to_string()], // allowed tools
);
bind_capabilities(&blueprint, &allow)?;            // registry gate

The compiled blueprint reports:

• start = agent
• channels messages (reducer messages) and tool_calls (reducer append)
• node agent (kind agent, model "default", tools [lookup_user, create_ticket]) with conditional routes tool_call -> tools, final -> END
• node tools (kind tool_executor) with next -> agent

This is the textbook agent loop: the agent node calls the model, routes to the tool executor when there is a tool call, and back, until it routes final to END.

Run it:

cargo run --example rag_blueprint

Self-authoring: a model emits, compiles, and runs .rag

The deepest recursion in TinyAgents is a model writing the workflow it runs inside. Because .rag is declarative and registry-bound, a model's output passes through the exact same parse -> compile -> bind -> build_graph path as a human-authored file — with the capability allowlist as the safety boundary. The model never executes code; it only produces source that a Rust-side NodeFactory materialises.

examples/openai_self_blueprint.rs demonstrates the full loop:

1. Ask the model (OpenAI, behind the openai feature) to output only .rag source, handing it the grammar plus a worked example in the system prompt.
2. Strip any ``` fences and feed the text to parse_str → `compile`.
3. Bind the blueprint against a CapabilityResolver allowlist — only allowlisted models/tools pass; anything else is rejected.
4. build_graph with a trivial NodeFactory, then run to END.

If the model's output fails to parse, compile, or bind, the diagnostic and the offending source are surfaced — never executed. This is precisely why generated topology must flow through the compiler and policy checks instead of being installed directly: producing a graph never grants new capabilities.

cargo run --features openai --example openai_self_blueprint

Where .rag fits

• .rag defines graph topology and bindings declaratively.
• .ragsh is the imperative counterpart: it inspects, scripts, and recursively orchestrates harness/graph runs, and it can draft, validate, compile, and (under policy) register .rag blueprints through this same compiler.
• Both lower into the same graph + harness runtime as hand-written Rust.

Errors

Error	Phase	Examples
TinyAgentsError::Parse	lexer / parser	unterminated string, invalid escape, unexpected token
TinyAgentsError::Compile	compile / node kind	duplicate node, missing/undefined start, unknown target, mixed routing, unknown node kind
TinyAgentsError::Capability	binding	unregistered model, tool, subgraph, router, or reducer

See also

• Graph Runtime — the durable runtime blueprints lower into.
• Registry — the capability catalog .rag binds against by name.
• REPL Language (.ragsh) — the imperative RLM/CodeAct surface.
• Recursion and RLM — self-authoring and the recursive model.