Repository navigation
How An Appraisal Works
The skill's SKILL.md gives the agent four steps. The detailed checklist is references/methodology.md, version 2026-10-01.1 in release 0.1.0.
The agent gets from the article to the paper and runs paper_meta.py on it. Then it checks four things:
- The citation is real: authors, journal, year and DOI match an actual paper. Content farms often invent author lists while linking a real study, and that alone makes the verdict Fabricated / misattributed.
- The status today, not on the day the article was written: published, retracted or superseded. Economics working papers can take years to reach a journal, and the author's homepage is often the only place that says so.
- The recency the article claims. An outlet calling a 15-month-old paper "new" has made up the freshness, and a gap that long often means a later paper from the same sample that the article ignores.
- A reddit link is never fetched. An r/science title is the outlet's headline word for word, so the agent searches for it and appraises the outlet's article.
Six checks run on every review, whatever version the checklist is at:
- Read the paper itself, never the article's summary of it: the n per group, effect sizes with confidence intervals, what was really measured, the limitations, funding and conflicts.
- Check interests independently. Affiliations are compared against the conflict-of-interest statement, and the agent asks who is promoting the study and what they sell.
- Work out the spin delta: the paper's own hedged conclusion next to the headline. Each hop is graded on its own (abstract, press release, author interview, outlet), because the overreach is sometimes the author's.
- Look behind the exposure's name. For a score, index or composite, get the items. A "bullies earn more" study whose "bully" score was one item of fourteen is the case that added this check.
- For an uncontrolled before-and-after study, find the hold-out outcome: a measure that should not respond. If it moved anyway, so much of every other change could be practice or placebo.
- For an observational link between an exposure and an outcome, look for sibling comparisons and negative-control exposures (the same exposure at a time it cannot act), which test whether the link comes from who gets exposed.
The methodology adds branches for randomised trials (CONSORT 2025, RoB 2), non-randomised studies (ROBINS-I v2), equivalence claims, qualitative interviews (COREQ), regression papers, umbrella reviews (AMSTAR-2, overlap), and paper-mill and retraction screening.
The review opens with the verdict and a one-sentence reason, then shows what the study shows next to what the article claims, with real numbers. It translates the effect into something you can feel, such as cases per 1,000 people, and labels that as its own arithmetic. Then come the three to six flags that matter, the bottom line and the sources. Every technical term is explained the first time it appears; Configuration shows how to tell it who reads the reviews.
The review is saved as a Markdown file and one line is added to a verdict log; Configuration has the paths. Before writing, the agent reads the last lines of the log, because a repeat outlet or a recycled sample is worth a sentence.
| Verdict | Meaning |
|---|---|
| Solid | sound design for the claim, decent venue, meaningful effects, faithful reporting |
| Real but overhyped | legitimate study, distorted framing in the coverage or in the paper's own abstract and the author's interviews |
| Weak | real but underpowered, confounded, or measured through a proxy beyond its claims |
| Junk / predatory | the venue or the methods disqualify it |
| Fabricated / misattributed | the citation doesn't check out, or the paper has integrity flags |
This wiki describes science-study-evaluator 0.1.0 and was last updated on 2026-10-11. The skill and script are MIT licensed. Report problems in the issues.
Using it
The releases
Contributing
Elsewhere