Faithful summaries for Claude — 9 enforceable rules, benchmarked. We indexed 16,682 published skills: 460 mention summarizing, 174 are genuine summarizers — and every single one optimizes structure and brevity (sections, action items, tone). Zero state a checkable rule about the relationship between the summary and its source: nothing added, numbers verbatim, hedges intact, attribution preserved. Meanwhile academia has measured exactly these failures for a decade (FactCC, FRANK, SummaC, AggreFact, FaithBench) — and no skill imports any of it. This skill is that missing transfer: hallucination-benchmark error types turned into generation-time rules.
Built and bench-tested by SkillProof, the tested Claude skills directory.
A summary is lossy compression of someone else's claims; the only legitimate loss is omission — and even omission has rules. The nine rules, each auditable against the source:
- No claim without a source sentence — untraceable sentences get cut; needed context is marked
[context, not in source]. - Numbers copied, never re-derived — value + unit + referent verbatim; no rounding, averaging, or silent arithmetic.
- Preserve the hedge — "may reduce risk" never becomes "reduces risk" (and "confirmed" never softens to "might").
- Preserve attribution — "the CEO claims margins improved" ≠ "margins improved"; no fake consensus in transcripts.
- Declared compression contract — every summary opens with
~4,800 words → ~200 words · importance order. - Omission rules — decisions, deadlines, risks, and the "but" that reverses a headline never get cut for length; notable cuts are listed.
- Contradictions flagged, never silently resolved — two conflicting figures stay two figures, in a visible Flagged block.
- Quote anchors on load-bearing claims — the 2–4 claims the reader will act on carry a verbatim span or exact pointer.
- Claim-by-claim self-audit before delivering.
Output contract: compression line → summary → optional Flagged block (contradictions, ambiguities, out-of-source context, notable omissions). The Flagged block only appears when non-empty.
git clone https://github.com/Skillproofdev/summary-discipline ~/.claude/skills/summary-discipline
# restart Claude Code — triggers on "summarize", "tl;dr", "key points of", meeting digests, long-doc compressionOne command — the repo IS the skill.
Subset run — 6 of 11 documents. These numbers are from a pre-registered 6-document subset (D02 cardiotide trial, D04 quarterly review, D05 incident postmortem, D07 launch transcript, D09 sleep-research abstracts, D11 planted-contradiction memo), one per stress category, chosen before any runs. The remaining 5 documents are not yet run; treat this as a directional read, not the full-corpus result. Baseline vs skill, same model, same target length, each output audited claim-by-claim against a frozen ground-truth sheet.
| Metric (6-doc totals) | Baseline | Skill |
|---|---|---|
| Added claims | 0 | 0 |
| Distorted numbers | 0 | 0 |
| Hedge-drops (certainty upgrades) | 1 | 0 |
| Attribution losses | 0 | 0 |
| Critical omissions | 10 | 2 |
| Total error events | 12 | 2 |
| Contradictions flagged (D11, of 3 planted) | 0 | 3 |
| Mean compression ratio | ~4.5× | ~2.6× |
On the planted-contradiction document the baseline silently resolved all three numeric conflicts (stated one budget, one cutover date, one vendor headcount as fact, never signalling the memo disagreed with itself); the skill flagged all three with both values intact. The skill also preserved 8 caveats the baseline dropped while keeping the headline — e.g. a research abstract's "null result should not be interpreted as evidence of safety" and a postmortem's alert-noise trade-off.
Honest caveats. On added claims, distorted numbers, and attribution losses both arms scored
zero — at these lengths the baseline model is already disciplined there, so the skill didn't beat
it. The lone baseline hedge-drop is borderline. And the skill's omission/contradiction gains are
partly confounded by length: its total outputs run ~1.7× longer (it exceeded the word target on
all six documents, once by running the body itself long and citing fidelity), because the Flagged
block is appended after a near-target summary body. The cleanest result is contradiction flagging,
which is a behavior independent of word budget. Full 11-document runs are needed before
generalizing. Method and per-document scores: bench/results/verdict.md.
Same protocol as our other skills (methodology): identical prompts, one baseline agent vs one agent that reads this SKILL.md first, claim-by-claim audit of both outputs against the source.
Corpus — 10–12 source documents, chosen so each stresses a different rule:
- 3 news articles (attribution-heavy: quotes, officials, disputed claims)
- 2–3 technical/business reports (number-dense: revenue, percentages, dated figures)
- 2–3 meeting transcripts (multi-speaker, disagreements, decisions + action items)
- 2 research-abstract chains (hedge-dense: "may", "preliminary", stated limitations)
- 1 document with an internal contradiction (planted or naturally occurring), to test flagging
Pre-registration. Before any summary is generated, we record for each document: (a) its key-fact list — the 5–10 facts a competent reader must not lose (decisions, headline numbers with referents, reversing caveats); (b) every hedged claim (sentence + hedge word); (c) every attributed claim (claim + speaker); (d) every number (value, unit, referent). Auditing happens against this pre-registered sheet, so "critical omission" is not defined after seeing the output.
Runs. For each document: one baseline summary (plain "summarize this" prompt) and one skill summary (same prompt, skill loaded), same model, same target length.
Metrics — each summary is split into atomic claims and audited against the source:
| Metric | Definition |
|---|---|
| Added-claim count | Summary claims with no supporting source sentence (unmarked context counts; marked [context, not in source] does not) |
| Distorted-number count | Figures that differ from the source in value, unit, or referent, or were silently rounded/averaged/converted |
| Hedge-drops | Pre-registered hedged claims rendered without their modality (certainty upgrades); certainty downgrades counted separately |
| Attribution losses | Pre-registered attributed claims rendered in the document's own voice, or two speakers merged into one position |
| Critical-omission count | Pre-registered key facts absent from the summary (a headline kept without its reversing caveat = the caveat is omitted) |
| Contradiction handling | For the contradiction document: flagged / silently resolved / omitted |
| Compression ratio | source words ÷ summary words — reported so faithfulness gains can't hide behind longer summaries |
Per-document scores plus corpus totals for both agents; negative results published as-is, like every SkillProof benchmark.
"Summarize", "tl;dr", "key points of", "what does this doc say", meeting-notes digests, long-document/thread/transcript compression, executive summaries. Explicitly excluded: creative rewriting, translation, tone changes, expanding text, writing new content.
We test other people's skills for a living (public methodology). Our own skills get the same treatment — measured benchmarks, negative results included.
More skills from us: skillproof-skills index · Free tools for skill authors: SKILL.md validator, token calculator, Rules⇄SKILL.md converter.
MIT — use it, fork it, ship it.