Skip to content

fix(router): use the LLM to pick from the full roster on low-confidence routes (v0.292.5) - #537

Merged
vikasprogrammer merged 1 commit into
mainfrom
feat/router-llm-pick
Aug 3, 2026
Merged

fix(router): use the LLM to pick from the full roster on low-confidence routes (v0.292.5)#537
vikasprogrammer merged 1 commit into
mainfrom
feat/router-llm-pick

Conversation

@vikasprogrammer

Copy link
Copy Markdown
Owner

Fixes the "wrong list" behind Cockpit's "Route to an agent instead" — and the underlying routing weakness it exposed.

Root cause

Routing "build a feature" returned app-packager, marketer, newsletter-writernot engineer. Without an embedder, the keyword scorer matches word overlap, and "build a feature" doesn't lexically hit engineer's description ("writes code, debugs, ships fixes"). So it disambiguated to the wrong few. The LLM step existed but only re-ranked the near-tie shortlist — it never reconsidered the full fleet, so it couldn't surface engineer.

Fix

On any low-confidence keyword result (disambiguate/none), if an LLM is configured, it now picks the single best-fit agent from every agent's description (llmTieBreakllmPick, over the full roster). A confident keyword route is left alone (no LLM cost); UNSURE keeps the keyword decision; no LLM configured → unchanged keyword-only behavior.

Result — the reported case and its variants now resolve correctly:

  • "build a feature" → engineer
  • "which agent can help me build a feature" → engineer
  • "spam and abusive content" → trust-safety via keyword, 0 LLM calls (confident routes skip the LLM)

This lifts routing quality across Cockpit, Discord, and Slack (all go through chooseAgent), now that a Claude key is configured.

Verification

  • typecheck + build clean; test:governance159/159 + tier-A 18/18 + capability 18/18
  • Stubbed-model e2e: keyword-weak → full-roster LLM pick → engineer (1 call); confident keyword → trust-safety (0 calls); no key → keyword-only unchanged

After merge

Deploy, then in Cockpit hit "Route to an agent instead" on that ask answer (or just type "build a feature") → it should route to engineer, not the marketing agents.

src/edge/router.ts.

🤖 Generated with Claude Code

…ce routes (v0.292.5)

Without an embedder the keyword scorer is weak — a task's words rarely match an
agent's description verbatim ("build a feature" doesn't lexically hit engineer),
so it disambiguated to the wrong few (app-packager/marketer/newsletter). The LLM
step now, on any low-confidence keyword result (disambiguate/none), picks the
single best-fit agent from EVERY agent's description instead of only re-ranking a
near-tie shortlist. So "build a feature" / "which agent can help me build a
feature" → engineer. Confident keyword routes are left alone (no LLM cost);
UNSURE keeps the keyword decision; no LLM configured → unchanged.

Fixes the wrong list behind Cockpit's "Route to an agent instead" and lifts
routing quality across Cockpit/Discord/Slack.

Verified: typecheck + build clean; test:governance 159/159 + tier-A + capability;
stubbed-model e2e (keyword-weak → full-roster LLM pick → engineer; confident
keyword → 0 LLM calls; no key → keyword-only unchanged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vikasprogrammer
vikasprogrammer merged commit 1be739d into main Aug 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant