Skip to content

Development

m4bwav edited this page Oct 11, 2026 · 1 revision

Layout

Path What it is
skills/science-study-evaluator/SKILL.md the procedure the agent follows
skills/science-study-evaluator/references/ methodology.md (the checklist), fetch-routes.md, explaining.md
skills/science-study-evaluator/scripts/paper_meta.py the metadata script
skills/science-study-evaluator/*.md, evergreen.json research log, changelog, lessons, test runs, setup and state
tests/test_paper_meta.py offline unit tests
evals/ eval cases in claude plugin eval format
ai-docs/ handoff, decisions and notes from work sessions

Unit tests

They run offline: each database answer is an invented fixture served by a fake fetcher. From the repository root:

$ python3 tests/test_paper_meta.py 2>&1 | tail -n 1
OK (skipped=1)
$ echo $?
0

That is 15 tests passing and one live smoke test skipped; SSE_LIVE=1 runs it against the real APIs. CI runs the suite on Windows, macOS and Linux with Python 3.10 and 3.14 on every push, and the live test every Monday and on demand.

Evals

The cases check that the skill fires on the right prompts, stays quiet on decoys (editing your own manuscript, a t-test on your data), and, given a DOI, runs paper_meta.py and opens with a verdict. Results are in TESTS.md; 6 of 6 passed on 2026-10-11.

claude plugin eval . --ablation none --no-publish -j 5 --tag trigger --trust-plugin
claude plugin eval . --ablation none --no-publish -j 1 --tag bash --allow-tools Bash Write Edit WebFetch WebSearch --trust-plugin

The second needs a Bash grant and network, so on Windows it runs under WSL2.

The refresh

The skill is evergreen. evergreen.json holds the schedule (tier moderate, about 30 days). A refresh runs the searches in RESEARCH.md, logs each finding as an R- entry with its sources, and edits methodology.md only for a change that adds or retires a check. It bumps the methodology's version line and logs a C- entry with the reason. MAINTENANCE.md has the procedure for machines without the evergreen plugin.

Releasing

A release bumps the version in .claude-plugin/plugin.json, plugin.json, VERSION in paper_meta.py, metadata.version in SKILL.md and version in evergreen.json together, tags vX.Y.Z, and publishes a GitHub release with the changelog entry as notes.

This wiki

Every output on these pages comes from a verification program run against the released tag. The program and its saved output are in the repository's ai-docs/notes/, so the next release can rerun it and compare.

Clone this wiki locally