Background Jobs

Background Jobs

Coyote can run long tool calls (e.g. builds, test suites, slow shell commands, slow MCP calls, etc.) as background jobs. A background job is a single whitelisted tool call running as a detached task: the model starts it with job__start, keeps working while it runs, gets a push notification when it finishes, and retrieves the result with job__collect.

Background jobs are available everywhere function calling is: plain REPL sessions, roles, agents, sub-agents, and graph LLM nodes.

Jobs vs. Sub-Agents

Jobs and sub-agents are complementary async systems:

	Background job	Sub-agent
What runs	ONE tool call (a process or an MCP invocation)	A full agent with its own LLM loop
Thinks?	No. It only produces output	Yes. Plans, calls tools, can spawn its own agents/jobs
Use for	"This single command takes minutes"	"This sub-task needs reasoning"
Budget	max_concurrent_jobs (default 5)	max_concurrent_agents (default 4)

The two systems compose one way only: agents (and graph LLM nodes) may start jobs, but a job can never start an agent, another job, or invoke any built-in tool. A job task owns only a frozen snapshot of what it needs to run its one tool call. There is no agent context inside a job for anything else to run against.

The Five Tools

Tool	Purpose
job__start	Run a whitelisted tool call in the background. Returns a job_<hex> id immediately.
job__check	Non-blocking status probe: status, elapsed time, and a tail of output captured so far. Never consumes the result.
job__collect	Block until the job finishes, then return its result and remove the job. The single retrieval verb. Results over 50k chars are tail-capped by default; full_result: true skips the cap, tail_lines: N keeps only the last N lines.
job__cancel	Kill the job's process group (SIGTERM, then SIGKILL after a 5s grace) and discard the handle. Returns partial output.
job__list	List registered jobs with status, elapsed time, and bytes of output captured.

job__start takes { tool, arguments }, where arguments is the same object the tool takes when called directly. It responds immediately:

{
  "status": "ok",
  "job_id": "job_a1b2c3d4",
  "tool": "execute_command",
  "message": "Running in background. Check with job__check, block with job__collect, cancel with job__cancel. You will receive a system_notifications entry on completion. Jobs do not survive coyote exiting."
}

Worked example: a long build

1. Start: The model backgrounds the slow command and keeps working:

   job__start --tool execute_command --arguments {"command": "cargo build --release"}
   → { "status": "ok", "job_id": "job_a1b2c3d4", ... }
   

2. Keep working: The model edits files, runs other tools, or answers questions. An occasional job__check --id job_a1b2c3d4 shows live progress (output_tail is the last chunk of output captured so far), without consuming anything.

3. Notification arrives: When the build finishes, the model's next tool result carries a system_notifications entry (see Push Notifications below) naming the exact collect command.

4. Collect: job__collect --id job_a1b2c3d4 returns the result (status, result, exit_code, elapsed time, final output tail) and removes the job. Collecting a still-running job simply blocks until it finishes. This is useful when the model has nothing else to do.

What Can Be Backgrounded

Backgroundable tools:

• External command tools: execute_command and any custom Bash/Python/JavaScript tool (see Function Calling), including agent-specific tools.
• mcp_invoke_*: MCP server tool invocations (see MCP Servers).

Everything else is rejected with a teaching error explaining what to do instead:

Attempted tool	Error
agent__*, job__*	'<tool>' is already asynchronous — call it directly. Agents may start jobs, but jobs never start agents or other jobs.
user__*	'<tool>' is interactive and must run in-turn — a background job cannot touch the terminal. Call it directly.
todo__*, memory__*, skill__*, rag__*	'<tool>' mutates agent/session state and must run in-turn. Call it directly.
fs_*, ast_grep	'<tool>' is fast — invoke it directly instead of backgrounding it.
mcp_search_*, mcp_describe_*, mcp_read_*, mcp_prompt_*	'<tool>' is a sub-second call; invoke it directly.

Each rejection ends with: Backgroundable tools: external command tools (e.g. execute_command) and mcp_invoke_* calls.

Two more gates always apply:

• Context availability: job__start can only background tools that were actually declared to the model in the current request. A tool filtered out by a role/session/agent/graph-node enabled_tools list is rejected with '<tool>' is not enabled in this context — job__start can only background tools declared to you in this request. Use the exact name of a tool from your current catalog. Backgrounding is never a way around tool filters.
• Capacity: At the concurrency limit, job__start rejects with At capacity: N/M jobs running. Collect or cancel one first.

And visibility follows capability: the job__* tools are only declared where they can do something. A context whose declared tools include nothing backgroundable (e.g. a graph llm node with tools: []) sees no job__* declarations at all. One carve-out: while a context still owns registered jobs (say a job was started and the tool it used was then disabled mid-session), the lifecycle verbs (job__check/collect/cancel/list) stay declared until the registry drains, only job__start disappears. A running job can never become unreachable.

Other errors you may see: No job '<id>' is registered — it may have already been collected or cancelled. job__list shows active jobs. (unknown/consumed id), and cross-kind teaching errors — '<id>' is a spawned agent, not a background job — use agent__check / agent__collect / agent__cancel (and the inverse from the agent__* tools).

Push Notifications

When a job (or a spawned agent; completion notifications cover both) finishes, coyote merges a system_notifications entry onto the last tool result of the model's next tool batch:

{
  "output": "...the tool's own result...",
  "system_notifications": [
    {
      "event": "job_completed",
      "id": "job_a1b2c3d4",
      "tool_or_agent": "execute_command",
      "status": "success",
      "next_action": "job__collect --id job_a1b2c3d4 for output"
    }
  ],
  "notification_instruction": "Background tasks have finished; collect each result with its next_action command."
}

event is one of job_completed, job_failed, agent_completed, agent_failed. Notifications for jobs that were already collected or cancelled are dropped. The model is never pointed at a dead id.

The turn-end guardrail

If the model tries to end its turn with running or finished-but-uncollected background tasks, coyote injects a system reminder instead of ending the turn:

[SYSTEM GUARDRAIL] You attempted to end your turn with 2 unreclaimed background task(s).

Still running (1):
- job_a1b2c3d4 (job): call `job__collect` (blocks until done, returns output) or `job__cancel` (discards)
...
Completed but UNCOLLECTED — collect NOW (1):
- `job__collect --id job_ffee0011`

After 3 reminders without action, coyote cancels the remaining tasks and discards any uncollected results, then lets the turn end. In practice the model collects on the first reminder; the guardrail exists so results are never silently abandoned.

Configuration

# config.yaml (global)
max_concurrent_jobs: 5      # default: 5; 0 disables background jobs entirely

# agents/<name>/config.yaml (per-agent override)
max_concurrent_jobs: 2      # this agent gets its own budget; omit to inherit the global value

# agents/<name>/graph.yaml (graph-agent override, agent-level — next to model/temperature)
max_concurrent_jobs: 2      # budget for the whole graph run; omit to inherit the global value

• Resolution is agent-override → global → default 5, exactly like max_tool_result_chars.
• Also settable via the COYOTE_MAX_CONCURRENT_JOBS environment variable (see Environment Variables).
• 0 disables the feature for that context: the job__* tools and their prompt instructions are simply never offered to the model. It's as if the feature doesn't exist.
• Function calling is required. With a model or configuration that doesn't support function calling, jobs are off for the same reason: no tool declarations ever reach the model.
• Every job budget is per-context: a sub-agent's jobs register with the sub-agent, not its parent, and each agent resolves its own max_concurrent_jobs. Cancelling an agent also cancels its jobs.
• There is deliberately no per-node budget in graph agents: every llm node (including parallel branches in the same super-step) draws from the graph run's one shared pool, so the budget belongs to the run as a whole. (Job ownership is still node-local; see the fine print below.)

Behavioral Fine Print

• Snapshot semantics. A job runs against a snapshot of the config, environment, and PATH taken at job__start. Changes made afterwards don't affect a running job. A job only produces output; it never mutates session state.
• No persistence. Jobs die with the coyote process. Quitting the REPL kills every job's process group; there is no reattach-after-restart. Switching agents (.agent) also cancels running jobs. Background work belongs to the context that started it.
• Two output channels. While running, everything the tool produces (its output file (sampled ~every 300 ms) plus raw stdout+stderr) streams into a bounded ring buffer (last 64 KiB). That's the output_tail that job__check shows, with output_bytes_captured counting everything ever written and tail_truncated flagging a clipped tail. The result returned by job__collect is separate (the tool's actual output), capped tail-first at the last 50,000 characters by default. Build failures land at the tail, so that's the end that's kept. The truncation header says exactly what was kept and what to do next time. Collecting is consume-once, so decide before collecting (job__check's output_bytes_captured shows the size): pass tail_lines: N to keep only the last N lines, or full_result: true to skip the cap and return everything (the session-wide max_tool_result_chars limit still applies). For very large outputs, prefer having the command write to a file and paging it with fs_read.
• Timeouts. Process jobs honor COYOTE_TOOL_TIMEOUT (default 1800s, 0 = unlimited), resolved at job__start; on expiry the process group is killed and collect reports the timeout as a tool error. MCP jobs have no timeout; cancel a hung one with job__cancel.
• Polling ergonomics. job__check (and job__list) never trip coyote's tool-call loop detector, so a model can legitimately poll. As a nudge against busy-waiting, once several consecutive job__check calls return an unchanged status and output, the result gains a hint telling the model to stop polling and rely on the completion notification instead; the counter resets the moment anything changes.
• REPL surfaces. When enabled, the job__* tools appear in .info tools, but they're built-in infrastructure. .list tools and .tool enable/.tool disable deliberately exclude them, so they can't be individually toggled (.tool enable job__start errors with "Unknown tool"). Availability is governed solely by max_concurrent_jobs and function-calling support.
• Graph LLM nodes. Jobs are node-local: the graph llm node that starts a job must collect or cancel it before the node ends. The turn-end guardrail enforces this on clean paths (an uncollected job burns node iterations and can fail the node at its iteration limit), and anything still registered when the node exits, on any path, including errors and timeouts, is cancelled with its result discarded. There is no handing a job to a downstream node. Parallel branches only ever see (and are nagged about) their own jobs. A node whose tools: whitelist declares nothing backgroundable sees no job__* tools at all.
• Escalations still surface. If a job__collect would block while child agents have pending escalations, it returns early with the escalation summary instead of deadlocking, and tells the model to reply first and collect again.