You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Field note: what the Tier 1 Discoverability score reads, and a question about Tier 3
#490
Thank you for building SkillEvaluator. This is a short field note from one trial, and a question.
I ran Tier 1 only (quality-check, and tier1 --no-llm) on one of my own public skills, TheColliery/CoalFace at v0.15.3, using SkillEvaluator v0.8.0. I have not run Tier 2 or Tier 3 (no Docker on that machine), so nothing below is a judgement of them.
What I saw. The skill scored Discoverability 90 of 100. Reading _check_discoverability in quality_score.py, that sub-score looks at the description's length (50-150 recommended), a few word lists (trigger words such as "use" and "when", vague words, first and second person, broad words without a "not for" phrase), a ## Purpose heading, and the shape of the skill name. docs/tier1-validation.mdx describes it as "description quality, trigger wording, purpose, scope, and naming". So it is a static reading of the text, which is what a no-LLM Tier 1 can do.
What I check by hand. I also ask a different question about a skill: when the text is needed, does an agent actually follow it? In practice that means a control that is shown to fail on the broken text before I trust a pass, a round that another rule had already overridden counts as void rather than as a pass, and every abnormal run (a skip, a crash, a not-run) is named in the record instead of folded into an average. None of that is visible to a document-shape score, and I would not expect it to be.
Question. Is Tier 3 (the with-skill and no-skill arms and Skill Lift) the intended answer to "does the agent pick the skill up and follow it", with the Tier 1 Discoverability number meant only as a cheap early signal? And is there a recommended way to read the two side by side, so that a high Tier 1 Discoverability is not taken as a statement about routing? From the docs the no-skill baseline looks like part of what I describe, but I have not run it, so I may be missing how it handles a control that must fail first or a void round.
I am not proposing a change. If the answer is already in the docs, a pointer would be enough.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Thank you for building SkillEvaluator. This is a short field note from one trial, and a question.
I ran Tier 1 only (
quality-check, andtier1 --no-llm) on one of my own public skills,TheColliery/CoalFaceatv0.15.3, using SkillEvaluatorv0.8.0. I have not run Tier 2 or Tier 3 (no Docker on that machine), so nothing below is a judgement of them.What I saw. The skill scored Discoverability 90 of 100. Reading
_check_discoverabilityinquality_score.py, that sub-score looks at the description's length (50-150 recommended), a few word lists (trigger words such as "use" and "when", vague words, first and second person, broad words without a "not for" phrase), a## Purposeheading, and the shape of the skill name.docs/tier1-validation.mdxdescribes it as "description quality, trigger wording, purpose, scope, and naming". So it is a static reading of the text, which is what a no-LLM Tier 1 can do.What I check by hand. I also ask a different question about a skill: when the text is needed, does an agent actually follow it? In practice that means a control that is shown to fail on the broken text before I trust a pass, a round that another rule had already overridden counts as void rather than as a pass, and every abnormal run (a skip, a crash, a not-run) is named in the record instead of folded into an average. None of that is visible to a document-shape score, and I would not expect it to be.
Question. Is Tier 3 (the with-skill and no-skill arms and Skill Lift) the intended answer to "does the agent pick the skill up and follow it", with the Tier 1 Discoverability number meant only as a cheap early signal? And is there a recommended way to read the two side by side, so that a high Tier 1 Discoverability is not taken as a statement about routing? From the docs the no-skill baseline looks like part of what I describe, but I have not run it, so I may be missing how it handles a control that must fail first or a void round.
I am not proposing a change. If the answer is already in the docs, a pointer would be enough.
All reactions