Integration Guide

Integration Guide: Benchmarking Compitum with RouterBench

Quick-Scan Infobox

Aspect	Details
Target User	Developers tuning routers, educators validating for curricula; Python/CLI comfort assumed.
Core Goal	Seamlessly eval Compitum's routing against RouterBench baselines—measure utility, stability, and emergent patterns.
Time to First Eval	20 mins: Env setup → Pretrain → Run → Scan reports.
Key Analogy	Like a skill-check arena: Pit your router against benchmarks, log the stats (energies, violations) for iterative grinds.
Artifacts Path	Outputs in data/eval_results/ (JSONs) and reports/ (plots/summaries).

Core Principles Layer (Technical Backbone)

RouterBench integration tests Compitum's bounded optimization end-to-end: Pre-trains calibrated predictors on historical traces (e.g., 5-shot prompts for few-shot evals), then runs evaluate_routers to simulate query batches under constraints ( A x \leq b ), scoring on free-energy minima, drift bounds, and shadow sensitivities. The --local flag stubs external models (offline routing), while --data-path loads pickled benchmarks (e.g., math/logic tasks). PYTHONPATH exports src for import resolution; artifacts include per-run certs (JSONs with signals) and aggregated reports (e.g., mean E, violation rates <1%).

• Setup: Isolated venv pulls RouterBench deps (e.g., scikit-learn for predictors, pandas for data)—avoids clashing with Compitum's core install.
• Pre-Train: Fits CalibratedPredictor on traces: ( \hat{q} = f(\text{features}; \theta) ), minimizing MSE over utility components; saves to models/predictors.pkl.
• Eval Run: Invokes QP solver per batch: Select argmin E, assert invariants (e.g., feasible fraction >95%). Config YAML tunes weights (alpha=0.4) and data slices.

Why integrate? Validates observer bounds: Benchmarks as quadratic queries, ensuring low entropy without collapse—mechanistic proof over subjective vibes.

Code Dive: Custom Eval Config Snippet (Collapsible for Tinkerers)

Runnable notebook version (GitHub-rendered)

Runnable notebook version (GitHub-rendered)

Pro Tip: Post-run: jq '. | select(.violation_rate > 0.01)' data/eval_results/*.json—flags constraint tweaks.

Approachable Layer (Relatable Hooks for Educators)

Benchmarking here is like grading a unit test: Setup the "answer key" (pre-train on samples), run the "exam" (eval batches), and review scores (reports on energy descent). For classrooms, it's validation gold—route student prompts through benchmarks to confirm "coherent paths" emerge, using shadows to justify "extra credit" (looser bounds for creativity). No gatekept metrics: Reports plot drift like a heartbeat, keeping the loop alive and auditable.

• Classroom Tie-In: Eval a debate task set—high coherence? Your router's spotting theme clusters; low? Amp beta_s in config for evidence weight.
• Analogy Echo: Pre-train as calibrating scales: Ensures fair weights before the weigh-in, preventing skewed selections.
• Common Hurdle: Path mismatches? Use rel paths (e.g., $(pwd)/src)—keeps it portable across machines.

Quick-Reference Toolkit (Eval Workflow)

Step	Command	Platform Notes	What Happens	Check For
1. Env & Deps	python -m venv .venv-routerbench && source .venv-routerbench/bin/activate && pip install -r src/routerbench/requirements.txt	Win: .venv-routerbench\Scripts\activate.	Isolates RB tools (sklearn, etc.).	which python → venv path.
2. Pre-Train	PYTHONPATH=$(pwd)/src python -m scripts.pretrain_predictors	Win: set PYTHONPATH=%cd%\src && .venv-routerbench\Scripts\python.exe -m ....	Fits predictors on traces; saves PKL.	ls models/ → predictors.pkl.
3. Run Eval	PYTHONPATH=$(pwd)/src python -m routerbench.evaluate_routers --config data/routerbench/evaluate_routers.yaml --local --data-path routerbench_5shot.pkl	Win: Adapt paths/backslashes.	Batches routes; scores invariants.	ls data/eval_results/ → JSONs.

Troubleshooting Table (Collapsible: Bench Snags)

Issue	Symptom	Fix	Analogy
Import Error	"No module routerbench".	Verify PYTHONPATH; echo $PYTHONPATH.	Wrong map—retrace to src.
Pre-Train Fail	MSE NaNs.	Check trace data; add clipping in script.	Unbalanced weights—recalibrate.
Eval Timeout	Hangs on batches.	--n-runs 10; lower D=16 in config.	Marathon vs. sprint—shorten laps.
Empty Reports	No artifacts.	--output-dir explicit/; check perms.	Silent oven—set the timer.

Next Steps: Your Loop

• Run a Mini-Eval: Tweak YAML for 10 runs—spot violation spikes?
• Visualize Outputs: Load reports/summary.csv in pandas; plot E distributions.
• Extend: Add custom tasks (e.g., edu prompts) to data/routerbench/; PR benchmarks.

See also: Cookbook for scripting evals; repo README's RouterBench section; Verification for invariant ties; Configuration for weight overrides.