ToolBudgetBench

A benchmark and demo harness for deciding which tools an LLM agent should see.

When an agent has 20, 50, 100, or 200 MCP-style tools, the harness has to make a hard choice: expose every schema and spend context, or expose fewer tools and risk hiding the right one. ToolBudgetBench makes that tradeoff measurable.

PYTHONPATH=src python3 -m toolbudgetbench demo-disclosure \
  --query "Find the ordinary non-admin CRM contact record for Acme."

That command runs a local progressive-disclosure demo: first a compact tool index, then a small set of expanded schemas, then a selected tool and token savings report.

Current Headline Results

These are deterministic-proxy results. They are useful for reproducible harness comparison, but they are not a claim that the same ranking already holds for frontier LLMs.

Setting	Strategy	Success	Gold Recall	Policy Violations	Avg Schema Tokens
Synthetic 200 tools, five seeds	adaptive_context_portfolio	0.727	0.825	0.020	443.6
Synthetic 200 tools, five seeds	sequence_aware_routing	0.650	0.913	0.000	657.7
Synthetic 200 tools, five seeds	compressed_description_top_k	0.397	0.739	0.262	452.2
Synthetic 200 tools, five seeds	all_tools	0.265	1.000	0.365	22460.4
Realistic 200-tool slice	adaptive_context_portfolio	0.469	0.938	0.000	370.0
Realistic 200-tool slice	all_tools	0.375	1.000	0.188	17550.0

What You Get

• A deterministic MCP-style benchmark generator for tool registries and tasks.
• A realistic MCP-style fixture with 48 tools, 32 tasks, toolsets, servers, sibling tools, multi-step tasks, argument checks, confirmation-gated writes, risk/permission labels, and execution-style checks.
• All 32 realistic tasks backed by a local stateful execution environment for checking final task outcomes, not only tool names.
• A deterministic 200-tool realistic expansion with 152 additional enterprise-style distractor tools for large-registry pressure tests.
• Nineteen exposure strategies, including all-tools, top-k retrieval, category-first routing, dynamic-k, negative filtering, compression, progressive disclosure, risk-aware toolset routing, plan-guided context routing, contrastive sibling routing, sequence-aware routing, and an adaptive context portfolio.
• ToolLint diagnostics for weak tool descriptions, missing constraints, semantic overlap, prompt-injection-like text, and schema budget waste.
• Policy diagnostics for high-risk or permission-requiring tools, including policy_violation_rate and permission_exposure_rate.
• Ground-truth provenance for synthetic_generator, human_verified, execution_verified, and future public benchmark adapters.
• OpenAI-compatible model evaluation while keeping deterministic proxy runs as the default for CI and reproducibility.
• Paper-ready source files and a lightweight installable Codex skill in skills/toolbudgetbench/.

Why This Matters For Harness Engineering

Tool calling is not only a model problem. The harness decides what tool context the model sees before it acts. A good harness must balance:

• coverage: is the right tool visible?
• confusion: are too many similar tools visible?
• cost: how many schema/prompt tokens were spent?
• documentation quality: are tool descriptions good enough to retrieve and use?
• disclosure timing: should the model see an index first and schemas later?
• permission pressure: did the harness expose or select high-risk tools for an ordinary request?

ToolBudgetBench is built to evaluate those harness decisions directly.

Quick Start

This repo has no required runtime dependencies outside the Python standard library.

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
make test
make demo-disclosure

Install the companion Codex skill from this checkout:

mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills"
cp -R skills/toolbudgetbench "${CODEX_HOME:-$HOME/.codex}/skills/"

Then ask Codex to use $toolbudgetbench when evaluating a local MCP-style tool registry. The skill is intentionally thin: it points Codex at this repo's CLI instead of copying benchmark logic into the skill.

Run a benchmark:

PYTHONPATH=src python3 -m toolbudgetbench run \
  --tasks 120 \
  --total-tools 200 \
  --tool-counts 20 50 100 200 \
  --seed 13 \
  --output artifacts/final/runs.jsonl \
  --report artifacts/final/summary.md \
  --artifacts-dir artifacts/final

Build a grouped report from existing JSONL:

PYTHONPATH=src python3 -m toolbudgetbench report \
  --input artifacts/final/runs.jsonl \
  --output artifacts/final/by_gold_source.md \
  --group-by gold_source

Validate non-synthetic fixtures:

make validate-fixtures

Generate human-readable fixture provenance audits:

make audit-fixtures

Convert a local BFCL-like, MCP-like, or named public-benchmark snapshot into a ToolBudgetBench fixture without vendoring external data:

PYTHONPATH=src python3 -m toolbudgetbench import-adapter \
  --adapter mcp_atlas \
  --input /path/to/local_snapshot.json \
  --output artifacts/public_adapter/mcp_atlas_fixture.json \
  --target-tools 200 \
  --source-url https://example.org/local-snapshot-source \
  --source-version commit-or-snapshot-date \
  --license-name reviewed-license \
  --redistribution-allowed no \
  --snapshot-notes "local review complete; do not vendor raw data"

Supported adapter names are bfcl, api_bank, tau_bench, mcp, mcp_atlas, mcp_bench, livemcpbench, and mcpmark. Aliases such as api-bank and tau-bench also work. The named adapters use small local JSON normalizers, but they preserve distinct gold_source values such as public_benchmark_mcp_atlas, public_benchmark_api_bank, and public_benchmark_tau_bench.

--target-tools keeps the imported public-benchmark tools and tasks, then adds generated distractor tools so the same snapshot can be evaluated as a tool-budget derived slice. The generated tools are marked separately and are not claimed as part of the public benchmark. These runs should be reported as BFCL/API-Bank/tau-bench/MCP-derived tool-budget slices, not as official leaderboard scores. The generated fixture records the input file sha256 plus source URL, version, license, redistribution status, and notes so later results can be audited.

Smoke-test the adapter path:

make adapter-smoke

Run a larger local public-derived adapter suite:

make public-adapter-suite

This builds BFCL-like, API-Bank-like, MCP-Atlas-like, and tau-bench-like synthetic local snapshots, converts them through the public adapter layer, expands each fixture to 200 tools, and runs the same exposure strategies. It is a reproducible pipeline check for public-derived budget slices, not an official public benchmark result. Reports include both gold_recall for the canonical source label and acceptable_recall for any complete accepted tool path.

Run the realistic MCP-style smoke slice:

make realistic-run

Run the expanded 200-tool realistic registry slice:

make realistic-run-200

Group an existing report by provenance, difficulty, or required toolset:

PYTHONPATH=src python3 -m toolbudgetbench report \
  --input artifacts/realistic/runs.jsonl \
  --output artifacts/realistic/by_required_toolset.md \
  --group-by required_toolset

Exposure Strategies

Strategy	Purpose
all_tools	Expose every full schema.
toolset_only	Select likely toolsets first, then expose tools inside them.
semantic_top_k	Retrieve top-k tools with dependency-free TF-IDF.
linted_top_k	Normalize descriptions with ToolLint, then retrieve.
category_first_routing	Select likely categories, then retrieve within them.
negative_filtered_top_k	Filter obvious non-matches using narrow negative-guidance cues.
dynamic_k	Use fewer or more tools based on retrieval confidence.
compressed_description_top_k	Expose compact schema/description surrogates.
progressive_disclosure	Expose a compact index, then expand candidate schemas.
progressive_disclosure_repair	Add a bounded repair round for low-confidence or missing-toolset cases.
budgeted_toolset_risk_routing	Route by toolset, schema cues, action cues, and risk metadata.
plan_guided_context_routing	Decompose the query into steps and expose step-annotated schemas.
hybrid_plan_compress_routing	Add compressed semantic fallback to plan-guided routing.
contrastive_sibling_routing	Prefer ordinary siblings and suppress unrequested admin/bulk/external/public variants.
sequence_aware_routing	Mark ordered multi-step candidates so first-step tools rank before later-step tools.
adaptive_context_portfolio	Select compressed, contrastive, sequence-aware, risk-aware, or plan-guided exposure from query features.
oracle_toolset	Upper bound for perfect toolset routing.
oracle_top_k	Upper bound that always includes gold tools.
random_top_k	Lower-bound control.

Optional LLM Evaluation

Deterministic proxy runs are the default. To run a small OpenAI-compatible smoke test, set an API key and model:

export OPENAI_API_KEY=...
export TOOLBUDGETBENCH_MODEL=gpt-4.1-mini
make llm-smoke

You can also point at any Chat Completions-compatible endpoint:

PYTHONPATH=src python3 -m toolbudgetbench run \
  --agent openai_compatible \
  --model gpt-4.1-mini \
  --openai-base-url https://api.openai.com/v1 \
  --tasks 30 \
  --tool-counts 50 200 \
  --strategies all_tools semantic_top_k progressive_disclosure dynamic_k

LLM runs write the same metrics as proxy runs, plus agent_type, model_name, estimated_cost, and usage metadata.

For experiments that should survive API interruptions, use incremental logging:

python3 scripts/run_llm_incremental.py \
  --fixture data/realistic_mcp_fixture.json \
  --tasks 6 \
  --total-tools 48 \
  --tool-counts 48 \
  --strategies all_tools semantic_top_k progressive_disclosure progressive_disclosure_repair compressed_description_top_k \
  --model gpt-4.1-mini \
  --base-url https://api.openai.com/v1 \
  --wire-api responses \
  --output-dir artifacts/llm_incremental

For public releases, keep model-backed run logs and raw API traces local unless they have been reviewed for cost, provenance, and disclosure risk.

Artifacts

The CLI writes one JSONL record per (task, tool_count, strategy), a Markdown summary, and research artifacts:

• aggregate.csv
• failure_breakdown.csv
• pareto_frontier.csv
• confidence_intervals.csv
• paired_comparisons.csv
• task_kind_breakdown.csv
• difficulty_breakdown.csv
• required_toolset_breakdown.csv
• repair_diagnostics.csv
• execution_diagnostics.csv
• gold_source_breakdown.csv
• lint_summary.csv
• lint_failure_associations.csv
• confusions.csv
• run_config.json
• run_log.md

Run the multi-seed suite:

make experiment-suite

Build the local arXiv source bundle:

make arxiv-package

Ground Truth

The main benchmark uses controlled synthetic gold labels so registry size, smell rate, and exposure policies can be swept reproducibly. To avoid overclaiming, the repo also includes:

• data/human_verified_fixture.json: 30 hand-authored tasks.
• data/execution_verified_fixture.json: toy tasks backed by a local executable environment.
• data/realistic_mcp_fixture.json: hand-authored realistic MCP-style smoke slice with 48 tools, 32 tasks, toolsets, servers, sibling tools, multi-step tasks, argument checks, 4 multi-path tasks, 11 forbidden-tool annotations, 22 confirmation-gated tasks, 10 permission-requiring tools, and 32 local execution-verified realistic tasks.
• data/realistic_mcp_fixture_200.json: deterministic expansion that preserves the 48 hand-authored tools and adds 152 programmatic distractor tools for 100/200-tool exposure-pressure tests. The expanded registry has 32 permission-requiring tools.
• src/toolbudgetbench/adapters.py: public benchmark adapter paths for BFCL, API-Bank, tau-bench, generic MCP, MCP-Atlas, MCP-Bench, LiveMCPBench, and MCPMark local snapshots, with no vendored external data.

The paper treats synthetic, human-verified, execution-verified, and public-adapter ground truth separately.

Project Status

This is a benchmark-first research artifact with a lightweight demo harness. It is not a production MCP router. Current limitations: semi-synthetic main tasks, approximate token counting, simple lexical retrieval, heuristic ToolLint, and optional rather than default LLM evaluation. The realistic slice models a lightweight task-level confirmation gate and risk labels, not a complete authorization system.

Useful public entry points:

• skills/toolbudgetbench/SKILL.md
• paper/main.pdf
• paper/main.tex