skill-quality: measure and improve description-driven auto-invocation (eval-based)

[kyle-sexton]

Motivation

The frontmatter-alignment interview (#3524) settled that discoverability rides almost entirely on description content: auto-invocation is the model matching a request against the listing text, so description quality is the fleet's highest-leverage unmeasured surface. The fleet currently passes all static checks (zero skills over the 1,536 cap, trigger phrases enforced by check-skill.sh), but nothing measures whether the descriptions actually win the invocations they should.

One question is already closed, by citation

Packed description vs when_to_use split: nothing to measure. The official frontmatter reference states when_to_use is "Appended to description in the skill listing and counts toward the 1,536-character cap", and the harness joins them with a literal 3-char " - " (encoded in check-skill.sh's combined-length check and audit_skill_visibility.py's ListingConfig.joiner_chars). Identical content split differently produces byte-near-identical model input, so an A/B on the field split has no effect to detect. The fleet keeps the packed pattern; this issue is about the content, not the field layout.

Proposed work

1. Instrument. Model-graded invocation probes using machinery the repo already owns: claude plugin eval fixtures and/or headless claude -p runs, where each probe is a realistic user request and the grade is whether the intended skill is selected (and no competitor is). Start with a small probe set over the skills with the weakest invocation evidence (claude-ops:audit-skill-visibility's starvation report is the ranking input).
2. Baseline, then treatment. Grade current descriptions; then rewrite candidates per the platform best-practices levers (key use case first, specific vocabulary, third person, what AND when) under check-skill.sh's trigger-preservation constraint; re-grade.
3. Candidates parked from the sweep interview:
   • paths adoption for genuinely file-scoped skills (a precision lever; wrong globs suppress, so adopt only with probe coverage).
   • The two near-cap descriptions (claude-ops:audit-performance at 1,526 and claude-ops:audit-skill-visibility at 1,518 of 1,536): trim only if probes show tail content matters.
   • when_to_use convergence for the 3 outlier skills using it (cosmetic; only if free).
4. Exit criterion. A repeatable probe suite in CI-or-on-demand form, plus either measured improvements applied fleet-wide or a recorded finding that current descriptions already saturate the probes.

Constraints

• Trigger-phrase preservation is a hard gate (check-skill.sh fails dropped triggers vs base).
• Aggregate listing budget policy stays owned by claude-ops:audit-skill-visibility; this issue measures per-skill description quality only.

[kyle-sexton]

This was generated by AI during triage.

Merged from #3613 (duplicate)

#3613 arrived from different provenance (deferred vertical V4 of the playground-article corpus effort, docs/topics/playground-integration/PLAN.md) but asks for the same deliverable this issue does: build an eval harness once, rewrite skill descriptions under check-skill.sh's trigger-preservation gate, and show a measured trigger-rate improvement plus a rerunnable method. Closing it as a duplicate of this issue, which is the earlier and better-specified of the two: it names the ranking input (claude-ops:audit-skill-visibility's starvation report) and closes the packed-description versus when_to_use sub-question by citation rather than leaving it to be measured. Carrying over what #3613 had:

Methodology detail, for step 1's probe design. The optimizing-descriptions methodology from the same upstream site is directly reusable and more specific than "a small probe set":

• Roughly 20 labeled eval queries per skill, positive and negative, so a probe set measures false triggers as well as missed ones.
• Explicit trigger-rate thresholds rather than an eyeballed comparison.
• Train/validation splits, so a description is not overfitted to the specific phrasings it was tuned against. This is the part most likely to be skipped and the part that decides whether a measured improvement generalizes.
• An iteration loop that synthesizes failures into the next round of probes.

Cost framing #3613 carried, which motivates prioritization. The Agent Skills client-implementation guidance puts the catalog cost at roughly 50 to 100 tokens per skill at session start, paid whether or not the skill fires. This fleet's own native-surfaces evidence records a consumer-style listing measured at about 13x the documented 8,000-character default across 153 listing-eligible skills, so silently dropped descriptions are the normal case in a fleet this size, not an edge case. That is the argument for picking the highest-traffic or worst-triggering skills first, which matches this issue's starvation-report ranking.

Boundary preserved. This issue's constraint stands unchanged: aggregate listing-budget policy remains owned by claude-ops:audit-skill-visibility, and the work here measures and improves per-skill description quality only. #3613's budget framing is motivation for ordering the work, not a mandate to shorten descriptions here; per this issue's step 3, the two near-cap descriptions get trimmed only if probes show the tail content does not matter.

---

Generated by Claude Code