v0.2.2 — confidence threshold + low-confidence fallback

Confidence threshold + low-confidence fallback in the routing dispatcher. Backward compatible at the API surface — RouteResponse schema is unchanged; only a new value for path_taken is possible.

What's new

CONFIDENCE_THRESHOLD (env, default 0.65)

When IntentClassifier.classify() returns a confidence below the threshold, the dispatcher skips the four normal routing arms and lands on a transparent fallback that:

• Reports the model's best-guess intent and the actual (low) confidence.
• Sets path_taken = "low_confidence_fallback".
• Puts the would-have-been path into the trace, so the operator can see what the router would have done.
• Returns a user-facing message explaining the uncertainty — no fake answer.
• Makes no LLM call. The fallback is intentionally cheap; an LLM-driven clarifying-question step is the next natural extension.

$ curl -X POST $URL/route -d '{"input":"Tell me about the requirements"}'
{
  "intent": "chitchat",
  "confidence": 0.493,
  "path_taken": "low_confidence_fallback",
  "answer": "I couldn't confidently classify your request — the most likely intent was 'chitchat' with confidence 0.493, below the threshold of 0.65. Try rephrasing with a clearer cue.",
  "trace": [
    "low confidence: intent='chitchat' confidence=0.493 threshold=0.65",
    "would have routed to: chitchat:direct_llm",
    "no LLM call made; future: ask clarifying question or escalate to LLM router"
  ]
}

Empirical motivation

The 0.65 default isn't arbitrary. The generalization probe committed at router/results/generalization_test.txt (20 fresh sentences, never seen during training) shows two clean regimes:

• Clear inputs (a human would know the intent at a glance): confidence 0.81–0.94.
• Ambiguous inputs (deliberately stripped of class markers): confidence 0.39–0.59.

0.65 falls cleanly in the gap and produces no false negatives on the clear set and no false positives on the ambiguous set.

Defensive parsing

• CONFIDENCE_THRESHOLD parses as float; on ValueError falls back to the default rather than crashing the service.
• Values outside [0, 1] are clamped — a typo like 9.0 can't accidentally turn off the gate.

Tests

Eight new test cases in backend/tests/test_api.py:

• test_high_confidence_routes_through_normal_path
• test_low_confidence_falls_back_without_calling_llm — asserts the provider factory is never called in the fallback (no token spend)
• test_low_confidence_trace_includes_attempted_intent_and_confidence
• test_low_confidence_answer_does_not_pretend_to_have_classified
• test_explicit_threshold_overrides_default
• test_env_var_configures_threshold
• test_env_var_invalid_value_falls_back_to_default
• test_env_var_out_of_range_is_clamped

82 backend tests total now (was 74).

Live verification

URL=https://agent-router-909428365094.us-central1.run.app

# High confidence — normal route
curl -X POST $URL/route -H 'Content-Type: application/json' \
  -d '{"input":"What is the capital of France?"}' | jq .path_taken
# "simple_qa:direct_llm"

# Low confidence — fallback path, no LLM call (~300ms warm)
curl -X POST $URL/route -H 'Content-Type: application/json' \
  -d '{"input":"Tell me about the requirements"}' | jq .path_taken
# "low_confidence_fallback"

Cloud Run deploy

gcloud run deploy agent-router \
  --image us-central1-docker.pkg.dev/research-agent-498415/agent-router/agent-router:v0.2.2 \
  --service-account=agent-router-runtime@research-agent-498415.iam.gserviceaccount.com \
  --set-env-vars='^@^...@CONFIDENCE_THRESHOLD=0.65' \
  ...

Not in this release

• The fallback does not call any LLM to ask a clarifying question. That's the next natural extension and is left as a marker in the code (Dispatcher._low_confidence_fallback docstring).
• The trained DistilBERT model used by the live service is still the v0.1.0 GitHub Release artifact (MPS-trained on the local laptop). The Colab-trained CUDA model (the one that produced the perfect confusion matrix in router/results/) hasn't been published as a release yet — confidence values in production may differ slightly from the numbers in the README's generalization probe until the model in the release is updated.