Skip to content

The testing → test-servers hand-off fires only a third of the time #2247

Description

@cliffhall

Now that #2204 can measure a hand-off, the first thing it measures is a weak one.

testing's body opens by telling the model that picking a fixture is /test-servers and that it has to load it. With the two committed chain cases in .claude/skills/test-servers/evals/evals.json, that pointer is followed 33% of the time (RUNS=3, CHAIN_MAX_TURNS=14):

Hand-off (14 turns)
FAIL  33%  testing → test-servers     Write an integration test that exercises tool listing end to end.
FAIL  33%  testing → test-servers     Add end-to-end coverage for the tool-list pagination path.

Two things worth knowing before working on this:

  • An earlier draft of those prompts measured 100% and 67%, and that was an artefact. They said "against a real server" / "against a live server", which is test-servers' own trigger — so the model could pick testing and then pick test-servers from the original prompt, in that order, scoring a hit that would have survived deleting the pointer entirely. The prompts were rewritten to carry no server cue, and 33% is what the pointer alone is worth. Any fix has to be re-measured against prompts with no target cue, or it measures the same artefact.
  • Do not fix this by lowering CHAIN_THRESHOLD. The bar is 0.5 ("the pointer is taken more often than not") and the two red cases are the standing measurement, not a build break — skills:eval is deliberately not a gate.

What to try

  • Reshape where and how testing names /test-servers. Tune skill descriptions and eval cases to a reliable trigger rate, and document how #2202 showed that the shape of a description is the lever that moves a trigger rate; the same may hold for a pointer inside a body.
  • Check whether the run reaches a point where it knows it needs a fixture at all within 14 turns — CHAIN_MAX_TURNS is a knob, and a rate that climbs with it means the budget is the constraint rather than the pointer.
  • Re-run the full suite afterwards, not just these two cases: per AGENTS.md, changing what one skill says measurably moves the trigger rates of skills nobody touched.

Acceptance

  • The testing → test-servers hand-off measures above 50% at RUNS=5, with prompts that carry no test-servers trigger of their own.
  • The full skills:eval suite shows no first-move regression.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature requestv2Issues and PRs for v2

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions