Introducing SmartRouter: inspectable model routing and LLM testing tasks

[StepenkoAnatoli]

I'm building SmartRouter, a model-routing policy and local supervision toolkit for coding agents.

The question behind it is simple: before an agent picks a cheaper model or delegates a task, how do we check that the model is allowed to do the work, can do it, has a way to verify the result, and fits the remaining budget?

SmartRouter makes those checks explicit. It checks permissions, capabilities, verification, affordability, and economics in that order. Delegated work gets a bounded brief and acceptance checks. Authorized tasks run autonomously by default; human checkpoints are enabled only when explicitly requested.

Start here: README and visual overview.

You can try the local policy checks without an API key or a provider call:

git clone https://github.com/StepenkoAnatoli/SmartRouter.git
cd SmartRouter
python tools/validate_skills.py --strict
python tests/smart_router_decisions.py

Current limits: live-provider routing is experimental. There are open accounting and recovery gaps, and the existing model comparison is a small policy pilot. It does not establish a production model ranking or measured cost savings. The host application still owns permissions, model calls, billing, and final verification.

Help test different LLMs

I've opened focused tasks with acceptance criteria:

• Add unseen reservation-accounting fixtures — a good first issue; no model calls needed.
• Repeat GPT comparisons on a frozen unseen task set.
• Compare Claude profiles with the GPT benchmark.
• Prepare a Gemini adapter and offline evaluation plan.
• Benchmark a local or open-weight LLM.
• Capture usage and latency accurately.

See CONTRIBUTING.md for setup and checks. Pick an issue and leave a comment describing the part you want to take on. For live model tests, agree on the protocol and spend allowance first; offline fixtures are a useful place to begin.

I'd especially like feedback from people building coding-agent hosts: which routing decision or failure case would you need to inspect before trusting an integration?

If you find the project helpful, please consider giving it a star so you can find it again.

This announcement was drafted with AI assistance from the repository documentation.