tokencontrol

Dependency-aware parallel task runner and portfolio auditor for AI coding agents with multi-provider fallback and cost tracking.

Why This Exists

AI coding agents (Claude Code, Codex, z.ai) are powerful but suffer from three efficiency problems when used interactively:

Context pollution — each task invocation pollutes the conversation with irrelevant file reads. After 10 tasks across 6 repos, the context window is full of code from other projects.
Context drift — as history grows, the agent loses focus. By tool call 50+, it re-reads files and makes decisions based on stale context.
Token bleed — tokens spent on planning and copy-pasting between tasks. Each round-trip costs planning tokens that produce no code.

Tokencontrol eliminates all three: define tasks once in JSON, execute in parallel across multiple LLM providers, track results with forgeaware. The interactive session stays clean for architecture and review.

What This Is

Reads a JSON task file with dependency declarations
Builds a DAG and executes tasks in topological order with configurable parallelism
Portfolio scanner — 26 checks across 6 categories audit repos for structural, security, and quality issues
Scan-to-task pipeline — scan --format tasks generates agent-ready task files with detailed prompts
Runner fallback cascade — if codex rate-limits, falls to z.ai, then claude, with tier-based filtering
Seven runner backends — codex, claude, gemini, opencode, cline, qwen, script
Multi-file glob — --tasks 'pattern*.json' loads and merges multiple task files
Worktree isolation — --parallel-repo enables true same-repo parallelism via git worktrees
Secret scanning — pre-dispatch scan excludes unsafe runners from repos with secrets
Auto-commit — commits changes that agents leave unstaged
Runner graylist — auto-detects false positives and excludes low-quality runners
Task state tracking — persistent state prevents duplicate runs across sessions
Live TUI — real-time task status with --tui full|minimal|off|auto
Sentinel loop — continuous daemon for autonomous scan-run cycles
Post-run hooks — auto-import results to forgeaware for cost tracking
Produces JSON reports for post-run analysis and rerun of failures

What This Is NOT

Not a CI/CD system — designed for AI agent orchestration, not build pipelines
Not Airflow — orchestrates LLM coding agents, not ETL data jobs
Not a process supervisor — runs once, exits when done
Not a build system — delegates builds to Makefiles in target repos
Does not provide its own AI — orchestrates external runners (Codex, z.ai, Claude)

Quick Start

Install

# Homebrew
brew install ppiankov/tap/tokencontrol

# Go install
go install github.com/ppiankov/tokencontrol/cmd/tokencontrol@latest

Configure

Create .tokencontrol.yml in your repos directory:

repos_dir: /path/to/repos
workers: 6
fail_fast: true
max_runtime: 30m

default_runner: codex
default_fallbacks:
  - zai

runners:
  codex:
    type: codex
  zai:
    type: codex
    env:
      OPENAI_API_KEY: "env:ZAI_API_KEY"
      OPENAI_BASE_URL: "https://api.zai.zhipu.ai/v1"

post_run: /path/to/forgeaware/scripts/forge-import-run.sh $TOKENCONTROL_RUN_DIR

Generate and Run

# Generate task file from work orders
tokencontrol generate --repos-dir ~/dev/repos --config .tokencontrol.yml

# Preview execution plan
tokencontrol run --dry-run --tasks tokencontrol-tasks.json --repos-dir ~/dev/repos

# Execute with live TUI
tokencontrol run --tasks tokencontrol-tasks.json --repos-dir ~/dev/repos --config .tokencontrol.yml --workers 6

Example Dry-Run Output

tokencontrol — 3 tasks, 4 workers

Execution plan (dry-run):

  1. [P1] fix-auth-validation — Fix input validation in auth handler
     difficulty: simple (score: 3)
     repo: acme/backend
     prompt: The login endpoint at /api/auth/login does not validate email format before querying...

  2. [P2] add-rate-limiting — Add rate limiting to public API endpoints (after fix-auth-validation)
     difficulty: medium (score: 8)
     repo: acme/backend
     prompt: Add rate limiting middleware to all /api/public/ endpoints...

  3. [P3] update-readme — Update API documentation for auth changes (after fix-auth-validation, add-rate-limiting)
     difficulty: simple (score: 2)
     repo: acme/docs
     prompt: Update the API documentation in README.md to reflect the new email validation...

Example Run Summary

--- Summary ---
Total: 3  Completed: 2  Failed: 0  Skipped: 0  Auto-committed: 1  Duration: 2m45s
Tokens: 12.3K tokens in / 6.8K tokens out (19.1K tokens total)

See examples/task-files/ for sample task files.

Workflow

┌─────────────┐    ┌─────────────┐     ┌──────────────┐     ┌──────────────┐     ┌─────────────┐
│    scan     │───▶│  generate   │────▶│   audit      │────▶│    run       │────▶│  review     │
│             │    │             │     │              │     │              │     │             │
│26 checks    │    │parse WOs    │     │remove done   │     │DAG schedule  │     │status report│
│6 categories │    │inject config│     │narrow partial│     │runner cascade│     │forgeaware   │
│task output  │    │merge files  │     │validate      │     │live TUI      │     │rerun failed │
└─────────────┘    └─────────────┘     └──────────────┘     └──────────────┘     └─────────────┘

Step 1: Scan — audit all repos for structural, security, and quality issues. Optionally generate task files for autonomous fixing.

tokencontrol scan --repos-dir ~/dev/repos --format tasks --output scan-tasks.json

Step 2: Generate — scan repos for docs/work-orders.md, extract pending WOs, inject runner profiles from .tokencontrol.yml.

tokencontrol generate --repos-dir ~/dev/repos --config .tokencontrol.yml

Step 3: Run — execute tasks in parallel with dependency ordering and live TUI. Supports glob patterns for multi-file loading.

tokencontrol run --tasks 'tokencontrol-*.json' --repos-dir ~/dev/repos --config .tokencontrol.yml --workers 6

Step 4: Review — inspect results, verify repos, rerun failures.

tokencontrol status --run-dir .tokencontrol/<latest>
tokencontrol rerun --run-dir .tokencontrol/<latest>    # failed/skipped only

Runner Cascade

When a runner fails or is rate-limited, tokencontrol automatically tries the next provider in the cascade:

gemini (primary) ──fail──▶ codex (fallback 1) ──fail──▶ claude (fallback 2)

Seven built-in runner types: codex, claude, gemini, opencode, cline, qwen, script.

Configure per-task or globally:

{
  "default_runner": "gemini",
  "default_fallbacks": ["codex", "claude"],
  "runners": {
    "gemini": { "type": "gemini" },
    "codex": { "type": "codex" },
    "zai": {
      "type": "codex",
      "env": {
        "OPENAI_API_KEY": "env:ZAI_API_KEY",
        "OPENAI_BASE_URL": "https://api.zai.zhipu.ai/v1"
      }
    }
  }
}

The TUI shows which runner completed each task:

✓ COMPLETED  repo-WO01  Add unit tests           3m20s  [codex]
✓ COMPLETED  repo-WO02  Add SARIF output         4m10s  [via zai]    ← fell back from codex
✗ FAILED     repo-WO03  Add TUI                  0s     (tried codex→zai→claude)

Assign tasks to specific runners for parallel provider utilization:

{ "id": "repo-WO01", "runner": "codex", ... }
{ "id": "repo-WO02", "runner": "zai", ... }

Forgeaware Integration

Tokencontrol auto-imports run results to forgeaware via the post_run hook. After each run:

forge-import-run.sh reads report.json from the run directory
Extracts per-task: repo, result (pass/fail), token count, wall time, runner used
Logs to ~/.forgeaware/metrics.log for aggregation and dashboards

This creates an accountability pipeline: every AI agent task is tracked — which provider, how many tokens, how long, pass or fail.

Usage

`tokencontrol run`

Flag	Default	Description
`--tasks FILE`	`tokencontrol.json`	Path to tasks JSON file (supports glob patterns like `'tokencontrol-*.json'`)
`--workers N`	`4`	Max parallel runner processes
`--repos-dir DIR`	`.`	Base directory containing repos
`--config FILE`	`.tokencontrol.yml`	Settings file (workers, runners, post_run)
`--filter PATTERN`		Only run tasks matching ID glob
`--dry-run`	`false`	Show execution plan without running
`--verify`	`false`	Run `make test && make lint` per repo after completion
`--max-runtime DUR`	`30m`	Per-task timeout duration
`--idle-timeout DUR`	`5m`	Kill task after no stdout for this duration (only stdout events reset the timer; stderr does not)
`--fail-fast`	`false`	Stop spawning new tasks on first failure
`--tui MODE`	`auto`	Display mode: `full` (interactive TUI), `minimal` (live status), `off` (no live display), `auto` (detect TTY)
`--allow-free`	`false`	Include free-tier runners in fallback cascade
`--retry`	`false`	Re-execute failed and interrupted tasks (skips completed)
`--codex-quota-remaining N`	`0`	Remaining Codex token budget; `0` disables quota preflight
`--codex-quota-reserve N`	`0`	Tokens to reserve (not spend) from remaining budget
`--codex-quota-safety X`	`1.3`	Safety multiplier on estimated Codex tokens
`--codex-quota-enforce`	`true`	Block run when preflight predicts quota shortfall
`--codex-quota-lookback N`	`20`	Number of recent run reports to use for token history
`--no-auto-commit`	`false`	Disable post-task auto-commit
`--parallel-repo`	`false`	Enable worktree-based parallel execution for same-repo tasks

`tokencontrol scan`

Audit all repos for structural, security, and quality issues. Runs 26 filesystem-based checks across 6 categories: structure, go, python, security, ci, quality.

Flag	Default	Description
`--repos-dir DIR`	`.`	Base directory containing repos
`--format FMT`	`text`	Output format: `text` (human-readable), `json` (machine), `tasks` (tokencontrol task file)
`--filter-repo NAME`		Scan only this repo
`--severity LEVEL`		Minimum severity: `critical`, `warning`, `info`
`--check CATEGORIES`		Check categories to run (comma-separated: structure,go,python,security,ci,quality)
`--owner ORG`	(inferred)	GitHub owner for task format output
`--runner NAME`	`codex`	Default runner for task format output
`--output FILE`	(stdout)	Write output to file instead of stdout

The tasks format generates agent-ready prompts with file paths, code patterns, verification commands, and constraints — designed to be immediately runnable via tokencontrol run without editing.

# text summary
tokencontrol scan --repos-dir ~/dev/repos

# generate task file for autonomous fixing
tokencontrol scan --repos-dir ~/dev/repos --format tasks --output scan-tasks.json

# then run the fixes in parallel
tokencontrol run --tasks scan-tasks.json --repos-dir ~/dev/repos --workers 6

`tokencontrol generate`

Flag	Default	Description
`--repos-dir DIR`	`.`	Base directory to scan for repos
`--config FILE`	`.tokencontrol.yml`	Inject runner profiles from config
`--output FILE`	`<repos-dir>/tokencontrol-tasks.json`	Output task file path
`--owner ORG`	(inferred from git)	GitHub owner/org for repo slugs
`--runner NAME`	`codex`	Default runner for generated tasks
`--filter-repo NAME`		Only scan this repo

`tokencontrol rerun`

Rerun failed and skipped tasks from a previous run. Preserves runner profiles and settings.

tokencontrol rerun --run-dir .tokencontrol/20260217-143750

`tokencontrol status`

tokencontrol status --run-dir .tokencontrol/20260217-143750

`tokencontrol version`

tokencontrol version
# tokencontrol 0.15.1 (commit: 61c49dd, built: 2026-03-02T15:14:00Z, go: go1.25.7)

`tokencontrol watch`

Monitor a running tokencontrol session in real-time (top-like TUI).

tokencontrol watch                                      # auto-detect latest run
tokencontrol watch --run-dir .tokencontrol/20260217-143750   # specific run

`tokencontrol verify`

Proofcheck a completed run: detect false positives, verify tests and lint pass.

Flag	Default	Description
`--run-dir DIR`	(required)	Run directory to verify
`--repos-dir DIR`	`.`	Base directory containing repos
`--mark-done`	`false`	Mark verified tasks as done in state tracker

tokencontrol verify --run-dir .tokencontrol/20260217-143750 --repos-dir ~/dev/repos

`tokencontrol validate`

Validate a task file without running anything. Checks JSON structure, dependency cycles, repo references, and runner profiles.

tokencontrol validate --tasks tokencontrol-tasks.json --repos-dir ~/dev/repos

`tokencontrol unlock`

Remove a stale repo lock file. Use when a task crashes with a lock held, preventing other tasks from accessing the repo.

tokencontrol unlock --repos-dir ~/dev/repos --repo org/my-repo

`tokencontrol graylist`

Manage the runner quality graylist. Graylisted runners are excluded from fallback cascade positions (but still used when explicitly assigned as primary runner).

tokencontrol graylist list                              # show all graylisted runners
tokencontrol graylist add deepseek --model deepseek-chat --reason "low quality"
tokencontrol graylist remove deepseek --model deepseek-chat
tokencontrol graylist clear                             # remove all entries

`tokencontrol state`

Manage persistent task state. State tracking prevents duplicate runs across sessions.

tokencontrol state list                                 # show task states
tokencontrol state reset task-id                        # reset a specific task
tokencontrol state clear                                # clear all state

`tokencontrol sentinel loop`

Continuous daemon: scan repos, deduplicate completed tasks, run, cooldown, repeat.

tokencontrol sentinel loop --repos-dir ~/dev/repos --cooldown 30m --workers 6

`tokencontrol doctor`

Health check: verify runners are installed, config is valid, dependencies are available.

tokencontrol doctor

`tokencontrol init`

Initialize a new .tokencontrol.yml config and optional task file scaffold.

tokencontrol init                              # interactive setup
tokencontrol init --repos-dir ~/dev/repos      # specify repos directory

`tokencontrol pr`

Create GitHub PRs from completed worktree tasks. Pushes worktree branches and creates PRs via gh.

Flag	Default	Description
`--run-dir DIR`	(required)	Run directory with report.json
`--repos-dir DIR`	`.`	Base directory containing repos
`--dry-run`	`false`	Show what PRs would be created
`--draft`	`false`	Create draft PRs

tokencontrol pr --run-dir .tokencontrol/latest --repos-dir ~/dev/repos
tokencontrol pr --run-dir .tokencontrol/latest --dry-run              # preview

`tokencontrol ingest`

Import external run results into forgeaware.

tokencontrol ingest --run-dir .tokencontrol/20260217-143750

Task Spec

{
  "description": "Sprint 42 work orders",
  "default_runner": "codex",
  "default_fallbacks": ["zai"],
  "runners": {
    "codex": { "type": "codex" },
    "zai": { "type": "codex", "env": { "OPENAI_API_KEY": "env:ZAI_API_KEY" } }
  },
  "tasks": [
    {
      "id": "repo1-WO01",
      "repo": "org/repo1",
      "priority": 1,
      "title": "Add unit tests",
      "prompt": "Add unit tests for all internal packages...",
      "runner": "codex"
    },
    {
      "id": "repo1-WO02",
      "repo": "org/repo1",
      "priority": 1,
      "depends_on": ["repo1-WO01"],
      "title": "Add integration tests",
      "prompt": "Add integration tests that depend on..."
    }
  ]
}

Field	Required	Description
`id`	Yes	Unique task identifier
`repo`	Yes	Repository in `owner/name` format
`priority`	Yes	Execution priority (1=high, 2=medium, 3=low, 99=last)
`title`	Yes	Short description
`prompt`	Yes	Full prompt for the AI agent
`depends_on`	No	IDs of tasks that must complete first (string or array)
`runner`	No	Runner backend (default: from config or `codex`)
`fallbacks`	No	Runner profiles to try on failure/rate-limit
`difficulty`	No	Task difficulty: `simple`, `medium`, `complex` (auto-scored at generate time)
`score`	No	Numeric difficulty score (auto-scored at generate time)

Task file top-level fields:

Field	Description
`default_runner`	Runner for tasks without explicit `runner` field
`default_fallbacks`	Fallback cascade applied to tasks without `fallbacks`
`runners`	Named runner profiles with type, model, env overrides
`parallel_repo`	Enable worktree isolation for same-repo tasks
`merge_back`	Auto-merge worktree branch back to main (default: true, FF-only)
`review`	Auto-review config: `enabled`, `runner`, `fallback_only`

Signal Handling

Ctrl+C sends SIGINT to tokencontrol, which:

Cancels all running tasks via process group kill (entire process tree, not just the runner)
Skips pending tasks that haven't started
Writes a partial report.json with results collected so far
Releases all repo locks

In-flight tasks are killed immediately. The report is always written, even on interrupt.

Exit Codes

Code	Meaning
`0`	All tasks completed successfully
`1`	One or more tasks failed
`4`	All tasks hit API rate limits

Secret Scanning

Before dispatching tasks, tokencontrol scans each repo for secrets using pastewatch-cli. Repos with detected secrets are flagged, and unsafe runners (those without structural secret protection hooks) are excluded from the fallback cascade for those repos. Runners with built-in safety hooks (claude, cline) are always allowed.

After task completion, output files are scanned and any detected secrets are redacted in place.

Runner Graylist

Tokencontrol auto-detects false positives — tasks that complete with exit code 0 but produce no real work (no git commits, no output events). Runners that produce false positives are automatically graylisted with their specific model.

Graylisting is model-aware: graylisting deepseek:deepseek-chat does not block deepseek:deepseek-reasoner. Graylisted runners are excluded from fallback cascade positions but still used when explicitly assigned as primary runner.

The graylist persists across runs at ~/.tokencontrol/graylist.json. Manage manually with tokencontrol graylist.

Architecture

cmd/tokencontrol/main.go        -- CLI entry point (exit codes: 0, 1, 4)
internal/
  cli/
    run.go                  -- run command: DAG scheduling, worktree/lock, TUI, post-run hook
    cascade.go              -- runner cascade, fallback filtering (graylist, free, secret, tier)
    generate.go             -- generate command: scan repos, inject runner profiles
    scan.go                 -- scan command: portfolio auditor
    rerun.go                -- rerun command: retry failed tasks with preserved config
    status.go               -- status command: auto-detects latest run dir
    graylist.go             -- graylist CLI subcommands (list, add, remove, clear)
    state_cmd.go            -- state CLI subcommands (list, reset, clear)
    doctor.go               -- doctor command: runner, config, dependency checks
    init.go                 -- init command: scaffold .tokencontrol.yml and task file
    pr.go                   -- pr command: create GitHub PRs from completed worktree tasks
    root.go                 -- Cobra root, version vars, global flags
  config/
    settings.go             -- .tokencontrol.yml loading, runner profile config
    loader.go               -- task file loading, glob resolution, multi-file merge
  task/
    model.go                -- Task, TaskFile, TaskResult, RunReport, RunnerProfileConfig
    graph.go                -- Dependency DAG, topological sort (Kahn's algorithm)
    scheduler.go            -- Worker pool with dependency-aware scheduling
    scorer.go               -- Task difficulty scoring, runner tier defaults
  runner/
    runner.go               -- Runner interface and registry
    codex.go                -- Codex exec backend (JSONL parsing, env resolution)
    claude.go               -- Claude Code CLI backend (stream-json parsing)
    gemini.go               -- Gemini CLI backend (stream-json parsing, headless mode)
    opencode.go             -- OpenCode CLI backend (JSON output, multi-provider)
    cline.go                -- Cline CLI backend (headless, structural safety hooks)
    qwen.go                 -- Qwen Code CLI backend (stream-json, model override)
    lock.go                 -- Per-repo file locking with wait-and-retry
    worktree.go             -- Git worktree isolation for same-repo parallelism
    blacklist.go            -- Runner blacklist with TTL for rate-limited providers
    graylist.go             -- Model-aware runner graylist with persistence
    prescan.go              -- Pre-dispatch secret scan (pastewatch-cli)
    autocommit.go           -- Post-task auto-commit with deterministic messages
    usage.go                -- Token usage accumulation helper
    event.go                -- Shared event types (usage, items)
    validate.go             -- Model pre-validation (OpenCode config parsing)
    health.go               -- Connectivity error detection (TLS/DNS/connection)
    profile.go              -- Runner profile resolution (env: prefix → os.Getenv)
  scan/
    scanner.go              -- Scan() entry point: walk repos, run checks, sort findings
    checker.go              -- Checker interface, AllCheckers() registry (26 checks)
    checks.go               -- Check implementations + promptBuilder for autonomous prompts
    finding.go              -- Finding, Severity, TaskPrompt() (prompt vs suggestion)
    repo.go                 -- RepoInfo, DetectRepo(), language detection
    format.go               -- TextFormatter, JSONFormatter, TaskFormatter
  reporter/
    tui.go                  -- Bubbletea interactive TUI (full mode)
    live.go                 -- ANSI live status (minimal mode)
    text.go                 -- Text reporter with cascade attempt display
    json.go                 -- JSON report writer
  state/
    state.go                -- Persistent task state tracker (.tokencontrol/state.json)
    filter.go               -- Task filtering based on state (skip completed, --retry)
  sentinel/
    loop.go                 -- Sentinel loop daemon (scan → dedup → run → cooldown)
    tracker.go              -- CompletionTracker (dedup across cycles)
    state.go                -- SentinelState (thread-safe shared state)
    tui.go                  -- Mission Control TUI (3-tab Bubbletea dashboard)
  generate/
    workorder.go            -- Work-order parser
    generate.go             -- Task file generator with difficulty scoring
  ingest/
    ingest.go               -- Forgeaware result import

CI/CD Integration

See the CI Integration Guide for GitHub Actions workflows and setup instructions. Example workflows are in examples/github-actions/.

Known Limitations

No remote execution — runs processes locally
No pre-flight quota check — if a runner rate-limits, fallback cascade handles it
Worktree merge-back is fast-forward only — divergent branches require manual resolution

Roadmap

License

BSL 1.1

Name		Name	Last commit message	Last commit date
Latest commit History 134 Commits
.github/workflows		.github/workflows
cmd/tokencontrol		cmd/tokencontrol
docs		docs
examples		examples
internal		internal
.gitignore		.gitignore
.golangci.yml		.golangci.yml
.goreleaser.yml		.goreleaser.yml
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile.goreleaser		Dockerfile.goreleaser
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

tokencontrol

Why This Exists

What This Is

What This Is NOT

Quick Start

Install

Configure

Generate and Run

Example Dry-Run Output

Example Run Summary

Workflow

Runner Cascade

Forgeaware Integration

Usage

tokencontrol run

tokencontrol scan

tokencontrol generate

tokencontrol rerun

tokencontrol status

tokencontrol version

tokencontrol watch

tokencontrol verify

tokencontrol validate

tokencontrol unlock

tokencontrol graylist

tokencontrol state

tokencontrol sentinel loop

tokencontrol doctor

tokencontrol init

tokencontrol pr

tokencontrol ingest

Task Spec

Signal Handling

Exit Codes

Secret Scanning

Runner Graylist

Architecture

CI/CD Integration

Known Limitations

Roadmap

License

About

Topics

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 43

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

`tokencontrol run`

`tokencontrol scan`

`tokencontrol generate`

`tokencontrol rerun`

`tokencontrol status`

`tokencontrol version`

`tokencontrol watch`

`tokencontrol verify`

`tokencontrol validate`

`tokencontrol unlock`

`tokencontrol graylist`

`tokencontrol state`

`tokencontrol sentinel loop`

`tokencontrol doctor`

`tokencontrol init`

`tokencontrol pr`

`tokencontrol ingest`

Packages