Repository navigation
Show & tell: a one-command audit for Agent Skills (12 rules + a context budget) #754
yunloong-Y
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Your agent didn't get dumber. Its context window got eaten.
I've sent a handful of PRs here, so I'm posting this in Show and tell to ask for a critique rather than making a launch announcement. It's a small standalone side tool, not a LearnHarness change — there is no PR attached to this.
A SKILL.md is "just Markdown", so nothing compiles it, nothing lints it, and nothing tells you what it costs. Meanwhile every skill you install gets injected into the context window. I kept running into the same three problems:
references/file is 2,000 lines of always-on prose.So I built skillscope, an offline auditor for Agent Skills:
100 − Σ(error 14 / warn 7 / info 2). A credential leak caps the grade at D regardless of score.--budget 8000and it tells you how many tokens are left and which skills to trim first.--fail-under 70exits non-zero.It has zero third-party dependencies, makes zero network calls, and needs no account or token. It is both a CLI and a Skill (the repo's own
SKILL.mdpasses its own audit at A/100).demoneeds no setup and ships deliberately bad sample skills. The output I get today:Honest limits, because this is a measurement tool: the token count is a heuristic estimate, not tokenizer output — do not use it for billing. The score is rule-based deduction, not a model's judgment. It reads local files only; it does not fetch or invent star counts. If you think a rule is wrong, that is exactly the kind of issue I want.
Across upstreams I've sent 128 PRs with 47 merged (verifiable via GitHub search), mostly in the "read the contributing guide first, one PR per problem" school. Bilingual README, MIT, offline forever.
What would help most: which rule is missing, whether you'd actually gate CI on a score, and what a fair "context budget" default looks like for your setup.
All reactions