How to orchestrate “sub-agents” with Codex CLI (queue/worker pattern) in parallel? MCPs fit?

[JeffSkynird]

1. Goal
I’d like to run multiple sub-agents with Codex CLI that collaborate via a queue (producer/consumer). A supervisor agent splitting work and N worker agents processing tasks sequentially or in parallel, with shared state (files/folders).

2. What I understand so far

• With Codex CLI we can already script step-by-step tasks.
• We could simulate cooperation by sharing a workspace (folder) and a task queue (file/JSON/SQLite), then driving Codex via shell commands in the same or different terminal.
• Or using the same terminal to create multiple tasks for codex in queue.
• I’ve read about MCP (Model Context Protocol) as a way to create a sub agent pattern in parallel mode.

3. Questions

1. Is there an official pattern (or example) to orchestrate multiple Codex CLI sessions as “sub-agents” (supervisor + workers)?
2. Recommended way to share state safely across queue tasks (locks, idempotency, conflict resolution)?

What I’m thinking (proposal)

Planner-first pattern (one Codex call plans, N workers execute)

Flow

1. Planner (single Codex CLI run) → outputs a machine-readable task list (JSONL/JSON) with:

   • id, role, cwd, message, context_files, explicit rules/allowlist (what the worker may touch), and optional deps for ordering.

2. Validator checks/normalizes the planner’s JSON (schema).

3. Workers run in parallel (Bash, RabbitMQ, or MCP), each consuming one task and applying the rules.

---

1) Task schema (JSON)

{
  "id": "string",
  "role": "builder | tester | research | <custom>",
  "cwd": "string",                    // base folder
  "message": "string",                // brief instruction for the worker
  "context_files": ["string"],        // files to read only
  "rules": {
    "write_root": "outputs/<id>/",    // write-only sandbox
    "allow_write_globs": ["outputs/<id>/**"],
    "forbid_write_globs": ["**/*.lock","runs/**"],
    "shell_allowed": true
  },
  "deps": ["id-optional"]            // tasks that must complete first
}

---

2) Planner prompt (Codex CLI)

codex chat \
  --cwd workspace \
  --system "You are the Planner. Produce a minimal task plan for parallel sub-agents.
Output ONLY valid JSON array (no prose). Follow the schema. 
Rules:
- Prefer 3–6 tasks. 
- Include per-task 'rules.write_root' (outputs/<id>/).
- Choose context_files precisely; do not bloat.
- Use dependencies only when necessary." \
  --message "Goal: Implement feature X with tests and brief research notes.
Repo layout: 
- specs/A.md
- src/
- tests/
- notes/brief.md
Constraints:
- Builders write only under outputs/<task_id>/.
- Testers may run commands but must not write outside outputs/<task_id>/.
- Researchers are read-only; produce a summary in outputs/<task_id>/research.md." \
  --approve all \
  --save plan.jsonl > tasks.json

Example planner output (tasks.json)

[
  {
    "id": "t1",
    "role": "research",
    "cwd": "workspace",
    "message": "Summarize alternatives based on notes/brief.md and specs/A.md.",
    "context_files": ["notes/brief.md","specs/A.md"],
    "rules": {
      "write_root": "outputs/t1/",
      "allow_write_globs": ["outputs/t1/**"],
      "forbid_write_globs": ["runs/**"],
      "shell_allowed": false
    }
  },
  {
    "id": "t2",
    "role": "builder",
    "cwd": "workspace",
    "message": "Implement module A in src/, guided by specs/A.md. Produce draft in outputs/t2/.",
    "context_files": ["specs/A.md","src/"],
    "rules": {
      "write_root": "outputs/t2/",
      "allow_write_globs": ["outputs/t2/**"],
      "forbid_write_globs": ["**/*.lock","runs/**"],
      "shell_allowed": true
    },
    "deps": ["t1"]
  },
  {
    "id": "t3",
    "role": "tester",
    "cwd": "workspace",
    "message": "Write and run tests for module A. Report results.",
    "context_files": ["tests/","outputs/t2/"],
    "rules": {
      "write_root": "outputs/t3/",
      "allow_write_globs": ["outputs/t3/**"],
      "forbid_write_globs": ["runs/**"],
      "shell_allowed": true
    },
    "deps": ["t2"]
  }
]

3) Validate the plan (Node, quick)

4A) Feed workers (Bash, parallel) — with rules applied

# run_parallel_from_plan.sh
set -euo pipefail
mkdir -p runs logs outputs

run_task() {
  id="$1" role="$2" cwd="$3" msg="$4" ctx="$5" write_root="$6"
  SANDBOX="runs/$id"
  mkdir -p "$SANDBOX" "$write_root" "logs"
  rsync -a --exclude 'outputs' --exclude 'runs' "$cwd/" "$SANDBOX/"

  # Build context.md from context_files
  tmpctx="$(mktemp -t ctx.XXXX).md"; echo "## Context $id ($role)" > "$tmpctx"
  echo "$ctx" | jq -r '.[]?' | while read -r f; do
    [ -e "$SANDBOX/$f" ] && { echo -e "\n---\n# $f\n" >> "$tmpctx"; cat "$SANDBOX/$f" >> "$tmpctx"; }
  done

  codex chat \
    --cwd "$SANDBOX" \
    --system "You are the $role sub-agent.
Rules:
- Treat repo as read-only.
- Write outputs only under ../$write_root
- Do not modify anything outside ../$write_root" \
    --message "Use context at $tmpctx. Task ID: $id. $msg" \
    --approve all \
    --save "logs/$role-$id.jsonl"
}

export -f run_task
jq -rc '.[] | [.id,.role,.cwd,.message,(.context_files // []), .rules.write_root] | @tsv' tasks.json \
| xargs -I{} -P 4 bash -lc 'IFS=$'\''\t'\'' read -r id role cwd msg ctx write_root <<< "{}"; run_task "$id" "$role" "$cwd" "$msg" "$ctx" "$write_root"'

4B) Feed workers (RabbitMQ) — same idea, plus leases/retries

[hoodrichpirobo]

There are two separate patterns here, and I would not mix them too much.

For interactive, human-supervised parallel work, Codex now has first-class subagent workflows. In the CLI/app you can explicitly ask it to spawn agents, for example:

Review this branch with parallel subagents. Spawn one agent for security risks, one for test gaps, and one for maintainability. Wait for all three, then summarize the findings with file references.

Codex handles spawning, waiting, routing follow-ups, and consolidating results. You can also define custom agents under .codex/agents/ or ~/.codex/agents/, and tune limits with [agents] settings such as agents.max_threads, agents.max_depth, and agents.job_max_runtime_seconds.

For a true queue/worker system, I would still build the queue outside Codex and run Codex as a worker, not try to make Codex itself be the queue manager.

The robust pattern is:

1. planner produces validated tasks
2. external scheduler claims/leases tasks
3. each task runs one isolated Codex worker
4. worker writes a patch/artifact/result
5. supervisor validates and merges results

Use codex exec for that, not an interactive TUI session. codex exec --json is the right interface for automation because it emits machine-readable JSONL events and exits cleanly.

Example shape:

codex exec \
  --cd "$WORKTREE" \
  --sandbox workspace-write \
  --ask-for-approval never \
  --json \
  -o "$OUT/result.md" \
  "$PROMPT"

The biggest design rule: do not let multiple workers edit the same working tree at the same time.

For write-heavy work, use one of these instead:

• one git worktree per task
• one branch per task
• one copy/sandbox per task
• workers produce patches under outputs/<task_id>/
• supervisor applies patches sequentially and runs tests

For read-heavy work, parallel subagents are much safer: exploration, triage, test-log analysis, dependency review, docs summarization, security review, etc.

For shared state, keep it boring and transactional:

• SQLite/Postgres/Redis/RabbitMQ is better than a model-edited JSON file
• tasks should have queued | running | done | failed states
• use leases with expiry, not permanent locks
• include attempt_count, worker_id, started_at, finished_at
• make every task idempotent
• workers should write only to their own output directory
• supervisor owns merges into the real repo

MCP fits, but as an integration boundary, not as the scheduler by itself.

Good MCP uses here:

• expose claim_task, finish_task, fail_task, write_artifact
• expose repo metadata or issue tracker state
• expose a queue service to Codex
• run Codex itself as an MCP server from another orchestrator, such as the Agents SDK

Bad MCP use here:

• expecting MCP alone to provide locking, retries, conflict resolution, or process supervision
• letting several agents mutate the same files through MCP without an external transaction/merge strategy

So I would choose based on the goal:

• Human-driven parallel review: use Codex subagents directly.
• Automated batch workers: use external queue + codex exec.
• Larger orchestrated agent system: use Codex MCP server + Agents SDK or your own scheduler.
• Shared mutable repo edits: isolate workers and merge patches later.

Your proposed planner-first schema is basically the right direction. I would change two things:

1. make the scheduler enforce write_root and leases, not just prompt the model to obey them
2. make workers produce artifacts/patches, then have one supervisor apply and test them serially

That gives you parallelism without turning the repository into a shared mutable conflict zone.