feat: add routing-focused eval support

[googs1025]

Motivation

skill-up is good at evaluating whether an Agent completes a task correctly once a Skill is installed. For Skill development, another important question is whether the Skill should be triggered for a prompt at all.

Routing behavior matters because:

• a Skill should trigger for intended prompts
• it should not trigger for unrelated prompts
• in multi-Skill environments, one Skill should not over-trigger or shadow another Skill

Proposed capability

A routing case could look like this:

id: route-code-review-positive
tag: routing

input:
  prompt: "Review this pull request and identify risks"

routing_expect:
  must_include: [code-review]
  allow_extra: false
  max_skills: 1

A negative case could look like this:

id: route-code-review-negative
tag: routing

input:
  prompt: "Write a poem about winter"

routing_expect:
  must_not_include: [code-review]

Possible semantics:

• must_include: expected Skill names must be selected or loaded
• must_not_include: listed Skills must not be selected or loaded
• allow_extra: false: no other Skills should be triggered
• max_skills: upper bound for triggered Skills

Why this matters

Without routing evals, a Skill can pass positive task cases but still be harmful in practice:

• it may trigger too often
• it may respond outside its intended scope
• it may cause negative delta in multi-Skill setups

Initial small scope

To keep the first version manageable, this could start as:

1. model routing cases as normal cases with tag: routing
2. add a routing_expect schema
3. support only engines that expose triggered/loaded Skill information
4. return a clear skipped/unsupported result when routing info is unavailable

Open questions

1. Do current agent adapters expose enough information to reliably identify which Skills were triggered?
2. Should routing be a top-level suite type, or regular cases with tag: routing plus routing_expect?
3. Should the first version support only single-Skill routing, leaving multi-Skill routing for later?

[zpzjzj]

The routing-eval goal is valuable, but I think there is an observability question that must be resolved before adding routing_expect: how can skill-up reliably know that an engine actually selected, loaded, or applied a Skill?

These are different states:

installed -> selected -> loaded -> applied -> outcome changed

• installed: skill-up copied the Skill into the runtime. This is known, but it does not prove routing.
• selected: the engine's router decided the Skill was relevant.
• loaded: the engine read/invoked the complete Skill body.
• applied: the model followed the Skill instructions. This is normally internal and cannot be directly observed.
• outcome changed: a controlled with-Skill/without-Skill comparison shows a behavioral difference.

The current SessionResult has transcript and output data, but no normalized Skill activation events. Adapter observability is also inconsistent: an engine may expose a structured Skill tool call, another may expose only command/MCP calls, and another may provide only the final response. Looking at an output that resembles the Skill's behavior is not sufficient evidence, because the base model may produce the same answer without loading the Skill.

I suggest reversing the implementation order:

1. Prove an observation source first

Run a real canary for each candidate engine and determine whether it emits a stable, structured event containing the Skill identity, for example:

{
  "type": "skill.loaded",
  "name": "code-review",
  "status": "success"
}

A generic Skill tool call plus a successful tool result may also be usable. Monitoring a SKILL.md file read is weaker evidence: it proves that a file was opened, not that the complete Skill was selected or followed.

2. Normalize engine-native evidence

If at least one engine exposes reliable evidence, adapters could normalize it into something like:

type SkillActivation struct {
    Name   string
    Event  string // selected, loaded, invoked
    Turn   int
    Source string // engine_event, tool_call
}

The engine should also expose an explicit capability indicating whether Skill activation events are supported.

3. Reuse rule_based assertions

Rather than introducing a separate case evaluation path immediately, routing can remain a normal case and use deterministic judge rules:

tag: routing

judge:
  type: rule_based
  success:
    - skill_activated:
        name: code-review
  failure:
    - skill_activated:
        name: unrelated-skill

Aggregate constraints such as allow_extra or max_skills can be added after multi-Skill event reporting is proven. The tag should remain classification metadata rather than changing execution semantics.

4. Do not silently pass unsupported engines

When an engine does not expose reliable activation evidence, the result should be explicitly UNSUPPORTED or ERROR, not silently treated as PASS. Defaulting to SKIP can create false-green CI; skipping should require an explicit opt-in.

Even with a reliable load event, “applied” cannot be proven by that event alone. The strongest practical evidence is a combination of:

1. an engine-native Skill activation/load event,
2. behavior assertions showing the expected Skill-specific outcome, and
3. repeated with-Skill/without-Skill benchmark deltas to reduce model nondeterminism.

So my recommendation is to keep this issue, but make the first milestone an engine capability investigation and real canary. The final routing_expect or skill_activated schema should be designed only after we know which engines provide trustworthy evidence and what those events actually mean.

[googs1025]

Thanks for the detailed feedback. I agree that we should resolve the observability problem before introducing routing_expect.

The distinction between installed -> selected -> loaded -> applied -> outcome changed makes sense. In particular, installed Skills are not enough evidence for routing, and matching the final output is also too weak because the base model may produce the same behavior without loading the Skill.

I think we can split this into two milestones:

1. Capability investigation / canary

   • Run a small routing canary for each candidate engine.
   • Check whether the engine exposes stable evidence for Skill activation, such as selected, loaded, or tool-call based invocation.
   • Document what each engine can and cannot reliably report.

2. Normalized activation evidence

   • If at least one engine exposes reliable evidence, add a normalized adapter-level structure, for example:

   type SkillActivation struct {
       Name   string
       Event  string // selected, loaded, invoked
       Turn   int
       Source string // engine_event, tool_call
   }

   • Add an explicit engine capability flag for Skill activation observability.
   • For unsupported engines, return UNSUPPORTED or ERROR rather than silently passing.

For the first assertion API, reusing rule_based judge sounds like the right direction:

tag: routing

judge:
  type: rule_based
  success:
    - skill_activated:
        name: code-review
  failure:
    - skill_activated:
        name: unrelated-skill

We can leave aggregate constraints like allow_extra and max_skills for a later milestone after multi-Skill activation reporting is proven.

Would you prefer the first PR to focus only on the canary/capability investigation and documentation, or should it also introduce the normalized SkillActivation type and capability flag for adapters that can support it?