Skip to content

feat(libsy): read a directly named tier as a classifier verdict #281

Description

@gburachas

Problem

Supplying a custom classifier rubric is only half of what a deployment needs. The rubric governs
what the judge is asked; the route still parses one fixed reply shape:

recommended_route, p_solve, confidence, abstain, capability_boundary, primary_rule, crux

with strict: true, additionalProperties: false, and a deny_unknown_fields struct behind it.

A rubric written for a specific domain naturally asks a different question. Instead of "how likely
is the efficient tier to solve this?" it asks "which tier should handle this?" and gets an answer:

{"route": "weak", "confidence": 0.9, "abstain": false}

That reply cannot be parsed today. Every turn yields no verdict and falls open to the capable tier —
silently, since a missing verdict is a normal, handled condition. The route appears to work while
doing no routing at all.

Proposal

verdict_format = "tier_named" on the llm_classifier route.

The two formats express the same decision. A named tier is a solve probability with the
threshold already applied, so no new routing policy is needed — only a reader that maps the name
onto the existing one.

Names Selects
weak, local, simple, medium weak target
strong, cloud, complex, reasoning strong target

Read from route or recommended_tier, whichever is present. Case-insensitive.

Behaviour deliberately shared with the probability format, because these are properties of the
routing contract rather than of the reply shape:

  • an explicit abstain routes to the fail-open tier and outranks a named tier;
  • an unrecognized name is not a decision — reported ambiguous, so a caller holding a pinned tier
    keeps it rather than being re-tiered on a vocabulary mismatch;
  • a verdict under min_confidence is likewise not a decision.

Extra fields are tolerated, not rejected: rubrics commonly report diagnostic signals alongside
the decision, and refusing a verdict for carrying more information than we read would be perverse.

No response schema is attached in this mode. The supplied rubric states its own output shape, and
constraining it to the packaged schema would contradict the rubric's own instructions.

Scope

9 files, +507/−13. New crates/libsy/src/algorithms/util/tier_verdict.rs with 8 unit tests; the
classifier judge becomes generic over its verdict type so one request builder serves both contracts.

Validation

Gate Result
cargo test --workspace 652 passed (baseline 640 + 12 across this branch and )
cargo clippy --workspace --all-targets -- -D warnings clean
cargo fmt --check clean
uv run pytest tests/ unchanged

End-to-end tests drive all four vocabularies through a live route and assert the selected target,
plus a test that no response schema is sent in this mode.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions