diff --git a/README.md b/README.md index 6cbaa204..20ce568b 100644 --- a/README.md +++ b/README.md @@ -310,6 +310,16 @@ All configuration is controlled via environment variables in the `docker run` co | `AGENT_AUTO_CONTINUE_LIMIT` | `1000` | Max consecutive auto-continue attempts before circuit breaker triggers | | `AGENT_NODE_TIMEOUT` | `600000` | Superstep timeout in milliseconds (default 10 minutes) | +**Optional — Process (Sub-Agent):** + +| Variable | Default | Description | +| ------------------------------------- | ---------- | ---------------------------------------------- | +| `PROCESS_SUBAGENT_TIMEOUT` | `600000` | Sub-agent process timeout in milliseconds | +| `PROCESS_SUBAGENT_MAX_CONCURRENT` | `4` | Max concurrent sub-agent processes | +| `PROCESS_SUBAGENT_SESSION_MODE` | `isolated` | Session isolation mode (`isolated`, `forked`, `shared`) | +| `PROCESS_SUBAGENT_DEFAULT_STRATEGY` | `parallel` | Default fan-out strategy (`parallel`, `sequential`) | +| `PROCESS_SUBAGENT_DEFAULT_ON_ERROR` | `continue` | Default error handling strategy (`continue`, `fail-fast`) | + **Optional — Persistence:** | Variable | Default | Description | @@ -403,7 +413,7 @@ Bundled LangChain tools gated by sandbox permissions: | **Code** | `executeCode` — code execution and analysis | | **Web** | `webSearch`, `web_extract` — outbound HTTP with timeout, URL allowlist filtering, multi-engine search backends | | **Media** | `image_generate` — image generation via fal.ai; `visionAnalyze` — vision/language analysis via OpenAI; `textToSpeech` — text-to-speech via OpenAI TTS | -| **Agents** | `mixtureOfAgents` — multi-agent orchestration | +| **Agents** | `mixtureOfAgents` — multi-agent orchestration; `subAgent` — spawn child-process agents with single execution and fan-out modes | | **Cron** | `cronJob` — cron job utilities | | **System** | `compactContext` — automatic conversation context compaction on LLM context-length errors (zero-permission, always registered) | @@ -553,6 +563,11 @@ Graceful shutdown flushes all buffered log entries to disk before process exit. | | `nodeTimeout` | `600000` | Superstep timeout in milliseconds (default 10 minutes) | | `lru` | `size` | `100` | Maximum number of cached LLM responses | | | `ttl` | `600000` | Cache entry TTL in milliseconds (10 minutes) | +| `process` | `subAgent.timeout` | `600000` | Sub-agent process timeout in milliseconds (default 10 minutes) | +| | `subAgent.maxConcurrent` | `4` | Max concurrent sub-agent processes | +| | `subAgent.sessionMode` | `isolated` | Session isolation mode (`isolated`, `forked`, `shared`) | +| | `subAgent.defaultStrategy` | `parallel` | Default fan-out strategy (`parallel`, `sequential`) | +| | `subAgent.defaultOnError` | `continue` | Default error handling strategy (`continue`, `fail-fast`) | | `persistence` | `mode` | `memory` | Storage backend (`memory`, `sqlite`) | | | `sqlite_path` | `memory/checkpoints.db` | SQLite checkpointer file path | diff --git a/docs/FLOWS.md b/docs/FLOWS.md index 04338906..08ec0394 100644 --- a/docs/FLOWS.md +++ b/docs/FLOWS.md @@ -21,6 +21,7 @@ Call chains and data flows for all primary code paths in the project, excluding - [File Tool Execution Flow](#file-tool-execution-flow) - [Terminal Tool Execution Flow](#terminal-tool-execution-flow) - [Web Tool Execution Flow](#web-tool-execution-flow) +- [Sub-Agent Tool Execution Flow](#sub-agent-tool-execution-flow) - [Sandbox Skill Execution](#sandbox-skill-execution) - [Memory Persistence Flow](#memory-persistence-flow) - [Context Loading](#context-loading) @@ -664,6 +665,63 @@ Multi-engine search backends (webSearch): └── CUSTOM_SEARCH_URL → custom endpoint ``` + +## Sub-Agent Tool Execution Flow + +**Entry:** `src/tools/subAgent.js` → `createSubAgentTool()` + +``` +subAgent tool (zero-permission, always registered): +├── validate input: delegation (required), context (optional), tasks (optional for fan-out) +├── if tasks provided (fan-out mode): +│ ├── for each task in tasks (bounded by maxConcurrent): +│ │ ├── spawn("sh", ["-c", `node index.js "${escapeShellArg(delegation)}" ||| "${escapeShellArg(context)}"`]) +│ │ ├── trackProcess(child, command) → { pid, child, status: "running", startTime } +│ │ ├── wait for completion or timeout (resolveTimeout: per-call > env > config) +│ │ └── parseSubAgentOutput(stdout) → { ok, result, error? } +│ │ └── Split on "# SubAgent" marker, parse JSON after marker +│ ├── if strategy === "sequential": wait for each to complete before next +│ ├── if strategy === "parallel": run up to maxConcurrent simultaneously +│ └── if onError === "fail-fast": abort remaining on first error +│ └── if onError === "continue": collect errors, return all results +├── else (single execution mode): +│ ├── spawn("sh", ["-c", `node index.js "${escapeShellArg(delegation)}" ||| "${escapeShellArg(context)}"`]) +│ ├── trackProcess(child, command) → { pid, child, status: "running", startTime } +│ ├── wait for completion or timeout +│ └── parseSubAgentOutput(stdout) → { ok, result, error? } +├── if returnParams provided: +│ └── filter result to only include specified keys +│ └── fallback to full text if not valid JSON +└── return { ok, result, error? } + +escapeShellArg(arg): +├── Replace backticks, dollar signs, single quotes, double quotes +├── Escape newlines, tabs, carriage returns +└── Wrap in double quotes for safe shell passing + +parseSubAgentOutput(stdout): +├── Split stdout on "# SubAgent" marker +├── Take content after marker +├── Try JSON.parse(content) +├── if valid JSON → { ok: true, result: parsed } +├── else → { ok: false, error: "Failed to parse sub-agent output" } + +resolveTimeout(options): +├── if options.timeout provided → options.timeout +├── else if MADZ_SUBAGENT_TIMEOUT env var → parseInt(env) +├── else → config.process.subAgent.timeout (default 600000) +``` + +**Process tracking:** Sub-agents share the `processTracker` Map from `terminal.js` for PID tracking and lifecycle management. Each sub-agent gets a unique PID that can be polled, waited on, or killed via the `process` tool. + +**Session isolation modes:** + +| Mode | Description | +|------|-------------| +| `isolated` | Fresh session, no parent context | +| `forked` | Forked from parent session with compaction | +| `shared` | Shared parent session context | + ## Sandbox Skill Execution **Entry:** `index.js` → `invokeSkill(skillName, input = {})` diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md index e83b6f54..be0d06d2 100644 --- a/docs/OVERVIEW.md +++ b/docs/OVERVIEW.md @@ -126,6 +126,40 @@ The agent runs: reason → call tool(s) → reason again → answer. Tool array --- +## Sub-Agent + +`src/tools/subAgent.js` — spawns child processes (`node index.js "PROMPT"`) to execute prompts as independent sub-agents. Supports single execution and fan-out (parallel/sequential) modes with configurable concurrency, timeout, and error handling. + +| File | Purpose | +|------|---------| +| `subAgent.js` | `createSubAgentTool()` — LangChain tool with marker-based stdout parsing; `parseSubAgentOutput()` — extracts structured results from sub-agent output; `escapeShellArg()` — handles quotes, backticks, dollar signs, newlines, tabs, carriage returns; `resolveTimeout()` — per-call > env var > config default priority | + +**Key features:** + +1. **Single execution mode** — Spawn one sub-agent with delegation + context, return structured result +2. **Fan-out mode** — Parallel/sequential task execution with configurable `maxConcurrent` limit +3. **Marker-based stdout parsing** — `# SubAgent` marker for result extraction (mirrors compaction tool) +4. **Response contract** — `{ ok, result, error? }` matching compaction tool pattern +5. **Process tracking** — Shared `processTracker` from terminal.js for PID tracking and lifecycle management +6. **Timeout resolution** — Per-call > env var > config default priority +7. **Parameter extraction** — Optional `returnParams` for JSON result filtering with fallback +8. **Session isolation modes** — `isolated` (fresh), `forked` (compaction), `shared` (parent) +9. **Shell escaping** — Handles quotes, backticks, dollar signs, newlines, tabs, carriage returns +10. **Error handling** — `continue` vs `fail-fast` strategies for fan-out batches + +**Configuration:** Sub-agent parameters are set via `config.process.subAgent`: + +| Key | Default | Description | +| --- | --- | --- | +| `process.subAgent.timeout` | `600000` | Sub-agent process timeout in milliseconds (default 10 minutes) | +| `process.subAgent.maxConcurrent` | `4` | Max concurrent sub-agent processes | +| `process.subAgent.sessionMode` | `isolated` | Session isolation mode (`isolated`, `forked`, `shared`) | +| `process.subAgent.defaultStrategy` | `parallel` | Default fan-out strategy (`parallel`, `sequential`) | +| `process.subAgent.defaultOnError` | `continue` | Default error handling strategy (`continue`, `fail-fast`) | + +--- + + ## Cache `src/cache/` — cache-aside LRU response cache for LLM API calls. diff --git a/docs/TUTORIAL.md b/docs/TUTORIAL.md index 81c4327d..0ae1d66b 100644 --- a/docs/TUTORIAL.md +++ b/docs/TUTORIAL.md @@ -262,6 +262,8 @@ Once inside the interactive terminal, use these commands: Changes to canonical memory require a `/new` command to refresh the current session context. +**Context compaction:** When conversations grow too long, `madz` automatically detects context-length errors and triggers a compaction routine. A tiered retention strategy preserves high-fidelity information: the system prompt and recent exchanges are kept intact, older exchanges are summarized, and the oldest messages are dropped. This happens transparently — you never need to start a new session or manually manage context. The `compactContext` tool is always available and can also be invoked directly by the agent. + ### Skills Skills are how you give `madz` new capabilities — a bit like a macro in Excel, but with more intention. You define a set of instructions, and `madz` follows them whenever a task matches. Skills let you package domain expertise, repeatable workflows, and specialized tools that `madz` can discover and invoke on demand. @@ -295,6 +297,8 @@ license: MIT Skills are stored in `skills/` and are version-controllable. Simple skills can be chained together into pipelines for complex multi-step processing, or composed by asking `madz` to coordinate between them. +**Built-in tools:** Beyond skills, `madz` ships with built-in tools for common tasks. The `subAgent` tool lets the agent spawn child-process agents to execute prompts as independent workers — supporting both single execution and fan-out modes (parallel or sequential) with configurable concurrency, timeout, and error handling. Other built-in tools include filesystem operations, terminal execution, search, memory management, and more. + --- ## ⚙️ Advanced Usage