You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Now that #2204 can measure a hand-off, the first thing it measures is a weak one.
testing's body opens by telling the model that picking a fixture is /test-servers and that it has to load it. With the two committed chain cases in .claude/skills/test-servers/evals/evals.json, that pointer is followed 33% of the time (RUNS=3, CHAIN_MAX_TURNS=14):
Hand-off (14 turns)
FAIL 33% testing → test-servers Write an integration test that exercises tool listing end to end.
FAIL 33% testing → test-servers Add end-to-end coverage for the tool-list pagination path.
Two things worth knowing before working on this:
An earlier draft of those prompts measured 100% and 67%, and that was an artefact. They said "against a real server" / "against a live server", which is test-servers' own trigger — so the model could pick testing and then pick test-servers from the original prompt, in that order, scoring a hit that would have survived deleting the pointer entirely. The prompts were rewritten to carry no server cue, and 33% is what the pointer alone is worth. Any fix has to be re-measured against prompts with no target cue, or it measures the same artefact.
Do not fix this by lowering CHAIN_THRESHOLD. The bar is 0.5 ("the pointer is taken more often than not") and the two red cases are the standing measurement, not a build break — skills:eval is deliberately not a gate.
Check whether the run reaches a point where it knows it needs a fixture at all within 14 turns — CHAIN_MAX_TURNS is a knob, and a rate that climbs with it means the budget is the constraint rather than the pointer.
Re-run the full suite afterwards, not just these two cases: per AGENTS.md, changing what one skill says measurably moves the trigger rates of skills nobody touched.
Acceptance
The testing → test-servers hand-off measures above 50% at RUNS=5, with prompts that carry no test-servers trigger of their own.
The full skills:eval suite shows no first-move regression.
Now that #2204 can measure a hand-off, the first thing it measures is a weak one.
testing's body opens by telling the model that picking a fixture is/test-serversand that it has to load it. With the two committed chain cases in.claude/skills/test-servers/evals/evals.json, that pointer is followed 33% of the time (RUNS=3,CHAIN_MAX_TURNS=14):Two things worth knowing before working on this:
test-servers' own trigger — so the model could picktestingand then picktest-serversfrom the original prompt, in that order, scoring a hit that would have survived deleting the pointer entirely. The prompts were rewritten to carry no server cue, and 33% is what the pointer alone is worth. Any fix has to be re-measured against prompts with no target cue, or it measures the same artefact.CHAIN_THRESHOLD. The bar is 0.5 ("the pointer is taken more often than not") and the two red cases are the standing measurement, not a build break —skills:evalis deliberately not a gate.What to try
testingnames/test-servers. Tune skill descriptions and eval cases to a reliable trigger rate, and document how #2202 showed that the shape of a description is the lever that moves a trigger rate; the same may hold for a pointer inside a body.CHAIN_MAX_TURNSis a knob, and a rate that climbs with it means the budget is the constraint rather than the pointer.Acceptance
testing → test-servershand-off measures above 50% atRUNS=5, with prompts that carry notest-serverstrigger of their own.skills:evalsuite shows no first-move regression.