Scope
Managed-skill authoring/curation review of current master at b9204dcf7de8faac4ef05af367a546aa29c32e9f.
This is distinct from #839, which owns evidence-budget starvation before the skill-writer backend runs, and from #888, which owns duplicated/prose-shape plugin tests. This issue owns the authoring objective and validation contract once a skill proposal exists.
Evidence
The managed-skill model currently derives host discovery text mechanically in crates/tracedecay-automation/src/managed_skill_model.rs::managed_skill_description: if the draft summary does not already begin with a Use-when form, it prefixes Use when ... and truncates it to the frontmatter budget.
Repo-local authoring guidance in .codex/skills/writing-agent-managed-skills/SKILL.md and its Claude mirror explicitly requires discovery text to start with Use when.
The bundled-skill contract in tests/agent_suite/shared_skill_contract_test.rs further codifies source shape rather than selection quality:
- minimum description length: 50 chars;
- maximum: 320 chars / 45 words;
- description must start with
Use or contain . Use ;
- exact punctuation/shape requirements.
Meanwhile the skill writer already has richer evidence available: session evidence, skill usage, stale-skill recommendations, overlap candidates, and underused tool-family signals. The existing evals/agent_adoption harness can grade first useful action, outcome, efficiency, and ablation conditions on neutral prompts.
Problem
A skill description is router metadata. Use when can be a good writing pattern, but it is not a correctness property.
Mechanically prefixing a human summary can turn a vague or over-broad summary into syntactically valid but badly routed metadata. Minimum character counts can actively encourage trigger stuffing. A source-shape gate cannot detect:
- a skill that fires on almost every task;
- two skills whose descriptions are confusable;
- a skill that never gets selected;
- a skill that is selected when no skill is needed;
- a skill that duplicates tool descriptions or generic model competence;
- an underused tool family where the correct fix is better tool metadata/hints rather than another skill.
The curator should be able to conclude no new skill is warranted, patch/merge/archive an existing one, or move detailed material into references.
Required design
- Stop deriving routing descriptions by blindly prefixing the draft summary.
- Make routing/discovery text an intentional proposal field, or derive it from explicit trigger evidence rather than the human-facing summary.
- Keep only actual host syntax/size limits as hard validators. Remove arbitrary minimum length and exact
Use when phrase requirements as correctness gates.
- Retain concise trigger-first writing as guidance where it empirically routes well, not as a mandatory string pattern.
- Have the writer evaluate existing overlapping skills before creating a new one and prefer patch/merge/archive when the workflow is already covered.
- Permit a
no_skill_needed outcome when evidence reflects one-off behavior, generic reasoning, a tool-description problem, a hint problem, or insufficient repeated evidence.
- For underused tool-family recommendations, distinguish among: improve tool description, improve hint/routing metadata, patch an existing skill, create a skill, or take no action. Do not map missed usage mechanically to more skill text.
- Preserve explicit approval/deployment and provenance semantics for managed skills.
Behavioral validation
Extend the existing agent-adoption/eval machinery rather than introducing a second evaluator.
For each new or materially changed managed skill, generated validation should contain:
- positive prompts that should route to it;
- near-neighbor prompts that should route to a different skill;
- negative prompts where no skill should be invoked;
- task-outcome checks after selection, not merely invocation detection.
Measure routing precision/recall, over-trigger rate, first useful action, and task outcome. A description rewrite is an improvement only if those metrics improve or preserve a justified tradeoff.
Skills with repeated patches/views but no successful uses should be eligible for archive/merge or routing redesign, not indefinite wording churn.
Test cleanup relationship
#888 should remove source-shape/prose tests made obsolete by this behavioral contract. Keep parser/frontmatter interoperability and real host limits, but do not replace removed Use when/length assertions with another equivalent string lint.
No live skill-writer or agent eval runs were performed for this audit.
Scope
Managed-skill authoring/curation review of current
masteratb9204dcf7de8faac4ef05af367a546aa29c32e9f.This is distinct from #839, which owns evidence-budget starvation before the skill-writer backend runs, and from #888, which owns duplicated/prose-shape plugin tests. This issue owns the authoring objective and validation contract once a skill proposal exists.
Evidence
The managed-skill model currently derives host discovery text mechanically in
crates/tracedecay-automation/src/managed_skill_model.rs::managed_skill_description: if the draft summary does not already begin with a Use-when form, it prefixesUse when ...and truncates it to the frontmatter budget.Repo-local authoring guidance in
.codex/skills/writing-agent-managed-skills/SKILL.mdand its Claude mirror explicitly requires discovery text to start withUse when.The bundled-skill contract in
tests/agent_suite/shared_skill_contract_test.rsfurther codifies source shape rather than selection quality:Useor contain. Use;Meanwhile the skill writer already has richer evidence available: session evidence, skill usage, stale-skill recommendations, overlap candidates, and underused tool-family signals. The existing
evals/agent_adoptionharness can grade first useful action, outcome, efficiency, and ablation conditions on neutral prompts.Problem
A skill description is router metadata.
Use whencan be a good writing pattern, but it is not a correctness property.Mechanically prefixing a human summary can turn a vague or over-broad summary into syntactically valid but badly routed metadata. Minimum character counts can actively encourage trigger stuffing. A source-shape gate cannot detect:
The curator should be able to conclude no new skill is warranted, patch/merge/archive an existing one, or move detailed material into references.
Required design
Use whenphrase requirements as correctness gates.no_skill_neededoutcome when evidence reflects one-off behavior, generic reasoning, a tool-description problem, a hint problem, or insufficient repeated evidence.Behavioral validation
Extend the existing agent-adoption/eval machinery rather than introducing a second evaluator.
For each new or materially changed managed skill, generated validation should contain:
Measure routing precision/recall, over-trigger rate, first useful action, and task outcome. A description rewrite is an improvement only if those metrics improve or preserve a justified tradeoff.
Skills with repeated patches/views but no successful uses should be eligible for archive/merge or routing redesign, not indefinite wording churn.
Test cleanup relationship
#888 should remove source-shape/prose tests made obsolete by this behavioral contract. Keep parser/frontmatter interoperability and real host limits, but do not replace removed
Use when/length assertions with another equivalent string lint.No live skill-writer or agent eval runs were performed for this audit.