Skip to content

feat(libsy): re-decide the classifier tier on fresh user turns #347

Description

@gburachas

Problem

llm_classifier classifies a conversation once and, with session_affinity, holds that decision forever. Both halves are wrong for long agentic sessions:

  • Classify every turn and a tool chain switches tier mid-flight — the weak tier picks up a conversation the strong tier was three tool calls into, with no idea what it was doing.
  • Classify once (session_affinity) and the opening verdict goes stale. A session that opened "summarise this file" becomes a hard debugging session ten turns later and is still served by the tier chosen for the summary.

The unit that should drive a routing decision is not the request and not the session — it is the user turn. Tool results are the agent continuing work the user already asked for.

Proposal

A followup table that re-classifies when the user asks something new and pins that target across the tool-call turns in between:

[routes.assistant]
type = "llm_classifier"
followup = { refresh = "on_user_turn", recent_turn_window = 4 }
Key Default Meaning
refresh on_user_turn on_user_turn re-classifies on each new user message; pin_once holds the first decision
recent_turn_window unset Trailing turns the judge sees. At least 1
fail_open_tier capable Tier served when the judge yields nothing usable and nothing is pinned. Mode capability only
abstain_selector unset JSON Pointer to a boolean abstain flag in the verdict. Mode custom only

Available on modes capability and custom. Rejected alongside session_affinity, whose latch is first-wins and would prevent every refresh — silently turning on_user_turn into pin_once; use refresh = "pin_once" if that is what you want.

The deliberate asymmetry, which is the part worth reviewing:

  • an unusable verdict, or an unreachable judge, holds the pinned target. No evidence is not evidence of change. It falls open only when nothing is pinned.
  • in mode custom, an explicit abstain outranks the pin — the judge declining to decide is itself a decision. Mode capability has no equivalent, because its packaged rubric answers with a solve probability and carries no abstain field.
  • neither path retries the judge.

Relationship to #298 and #308

These overlap and the difference is intentional. #298's RoutingIdentity and #308's Harbor proxy_x_session_id normalisation both improve who a conversation is; this is about when its tier is re-decided. The pin lives in per-session State rather than the affinity map precisely because AffinityRouter's assignment is first-wins and process-local, so it can never be refreshed — which is the behaviour this issue exists to change.

Where a request carries no session id the pin has nothing to persist in, so it can optionally be keyed on a hash of the first user message, the same fallback session_affinity already offers.

Rebased onto current main and green. Depends on nothing else; sibling issue for the graded-verdict labels map filed separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions