Problem
Supplying a custom classifier rubric is only half of what a deployment needs. The rubric governs
what the judge is asked; the route still parses one fixed reply shape:
recommended_route, p_solve, confidence, abstain, capability_boundary, primary_rule, crux
with strict: true, additionalProperties: false, and a deny_unknown_fields struct behind it.
A rubric written for a specific domain naturally asks a different question. Instead of "how likely
is the efficient tier to solve this?" it asks "which tier should handle this?" and gets an answer:
{"route": "weak", "confidence": 0.9, "abstain": false}
That reply cannot be parsed today. Every turn yields no verdict and falls open to the capable tier —
silently, since a missing verdict is a normal, handled condition. The route appears to work while
doing no routing at all.
Proposal
verdict_format = "tier_named" on the llm_classifier route.
The two formats express the same decision. A named tier is a solve probability with the
threshold already applied, so no new routing policy is needed — only a reader that maps the name
onto the existing one.
| Names |
Selects |
weak, local, simple, medium |
weak target |
strong, cloud, complex, reasoning |
strong target |
Read from route or recommended_tier, whichever is present. Case-insensitive.
Behaviour deliberately shared with the probability format, because these are properties of the
routing contract rather than of the reply shape:
- an explicit
abstain routes to the fail-open tier and outranks a named tier;
- an unrecognized name is not a decision — reported ambiguous, so a caller holding a pinned tier
keeps it rather than being re-tiered on a vocabulary mismatch;
- a verdict under
min_confidence is likewise not a decision.
Extra fields are tolerated, not rejected: rubrics commonly report diagnostic signals alongside
the decision, and refusing a verdict for carrying more information than we read would be perverse.
No response schema is attached in this mode. The supplied rubric states its own output shape, and
constraining it to the packaged schema would contradict the rubric's own instructions.
Scope
9 files, +507/−13. New crates/libsy/src/algorithms/util/tier_verdict.rs with 8 unit tests; the
classifier judge becomes generic over its verdict type so one request builder serves both contracts.
Validation
| Gate |
Result |
cargo test --workspace |
652 passed (baseline 640 + 12 across this branch and ) |
cargo clippy --workspace --all-targets -- -D warnings |
clean |
cargo fmt --check |
clean |
uv run pytest tests/ |
unchanged |
End-to-end tests drive all four vocabularies through a live route and assert the selected target,
plus a test that no response schema is sent in this mode.
Problem
Supplying a custom classifier rubric is only half of what a deployment needs. The rubric governs
what the judge is asked; the route still parses one fixed reply shape:
with
strict: true,additionalProperties: false, and adeny_unknown_fieldsstruct behind it.A rubric written for a specific domain naturally asks a different question. Instead of "how likely
is the efficient tier to solve this?" it asks "which tier should handle this?" and gets an answer:
{"route": "weak", "confidence": 0.9, "abstain": false}That reply cannot be parsed today. Every turn yields no verdict and falls open to the capable tier —
silently, since a missing verdict is a normal, handled condition. The route appears to work while
doing no routing at all.
Proposal
verdict_format = "tier_named"on thellm_classifierroute.The two formats express the same decision. A named tier is a solve probability with the
threshold already applied, so no new routing policy is needed — only a reader that maps the name
onto the existing one.
weak,local,simple,mediumstrong,cloud,complex,reasoningRead from
routeorrecommended_tier, whichever is present. Case-insensitive.Behaviour deliberately shared with the probability format, because these are properties of the
routing contract rather than of the reply shape:
abstainroutes to the fail-open tier and outranks a named tier;keeps it rather than being re-tiered on a vocabulary mismatch;
min_confidenceis likewise not a decision.Extra fields are tolerated, not rejected: rubrics commonly report diagnostic signals alongside
the decision, and refusing a verdict for carrying more information than we read would be perverse.
No response schema is attached in this mode. The supplied rubric states its own output shape, and
constraining it to the packaged schema would contradict the rubric's own instructions.
Scope
9 files, +507/−13. New
crates/libsy/src/algorithms/util/tier_verdict.rswith 8 unit tests; theclassifier judge becomes generic over its verdict type so one request builder serves both contracts.
Validation
cargo test --workspacecargo clippy --workspace --all-targets -- -D warningscargo fmt --checkuv run pytest tests/End-to-end tests drive all four vocabularies through a live route and assert the selected target,
plus a test that no response schema is sent in this mode.