Report Defect Audit

Report Defect Audit (2026-10-24)

A hand audit of ten finished DeepDelve reports, read the way a scientist would read them: every cited claim checked against the saved source, every part of the question checked against the report. The goal was a map of what actually goes wrong, so that later changes can be justified by a measured defect instead of a hunch. No code or default was changed because of this audit.

Status: data collection and labelling are finished. A human review of a 20-item sample (10 random severity 3 findings, 10 random severity 1 or 2) was done by the project owner, who agreed with all 20; it was one reader, without per-item notes. Treat the numbers as provisional all the same: the 8-report validation set below has not been human-reviewed.

Why this was done

Earlier work measured the cost of the completion checks (minutes per check, how often a first draft fails) and compared the review pass against no review pass on injected faults. That tells us what the machinery costs, not what is wrong with the reports. A project whose output is read by scientists needs the second answer first.

Method

The protocol was written before any audited report was read (eval/audit_runs/PROTOCOL.md in the main repo).

• Sample. Three saved reports with their sources (EU AI Act developments, Finland and Estonia digital services, one Raft versus Paxos run from September) plus seven reports written on 2026-10-24 by the current code from saved evidence: psilocybin for depression, GLP-1 drugs for obesity, LSM trees versus B-trees, Raft versus Paxos, congestion pricing, the 1918 influenza pandemic, remote work. One run each, no reruns, no picking. Only these three queries had saved sources among the older runs, which is why seven reports were generated.
• Units. Every cited claim against its cited source; every part of the query against the report; every statement of a gap against the body; every source carrying a key claim.
• Defect types. Fixed list before reading: claim not supported (C1), wrong citation (C2), overstated or generalised (C3), mismatched comparison (C4), internal contradiction (C5), outdated status presented as current (C6), question part unanswered (Q1), relevant saved material left out (Q2), weak source carrying a key claim (S1), padding (S2), format or meta defect (F1), gap disclosure wrong (G1, G2).
• Severity. 3 means a scientist acting on it would be misled. 2 weakens trust or needs rework. 1 is cosmetic.
• Stage. Evidence (E), writing (W), gap and judging logic (J), harness or scorer artefact (H).
• Second look. A blind second labeller (gpt-oss:20b) re-judged 40 random claim and cited-source pairs.

Results

Ten reports, 99 defects, 21 of severity 3. Every one of the ten reports contains at least one severity 3 defect (interval 72 to 100 percent).

Defect type	Defects	Severity 3	Reports affected (of 10)	95% interval
Claim not supported (C1)	16	7	9	60 to 98%
Overstated or generalised (C3)	16	0	9	60 to 98%
Internal contradiction or unreconciled conflict (C5)	9	3	8	49 to 94%
Weak source carrying a key claim (S1)	10	1	8	49 to 94%
Format, meta or scorer artefact (F1)	12	1	8	49 to 94%
Wrong citation (C2)	12	0	6	31 to 83%
Question part unanswered (Q1)	6	3	6	31 to 83%
Relevant saved material left out (Q2)	6	3	5	24 to 76%
Mismatched comparison (C4)	6	1	5	24 to 76%
Outdated status (C6)	3	1	3	11 to 60%
Gap disclosure wrong (G2)	2	1	2	6 to 51%
Padding (S2)	1	0	1	2 to 40%

Severity 3 examples, described without copying source text:

• A figure read backwards: the source says congestion pricing would cut interstate costs to $57 billion from $79 billion; the report says it would lower costs by $57 billion.
• A dose built from an unrelated number: a drug dose in the GLP-1 report matches only a "5.4 million patients" statistic in the sources.
• A claim opposite to its own cited source, twice: one report says the EU AI Office enforces high-risk system rules where its source says national authorities do; another says a database "incorrectly claims" to use Paxos where the source says it does.
• A conclusion built on mismatched editions: Finland's 2025 index score set against Estonia's 2022 score, with different averages, to say one country lags, while two saved EU reports rate Finland's digital public services near the top.
• Stale status while newer material sat in the saved sources: Phase 3 psilocybin results "expected in summer 2024" while a July 2026 Phase 3 result was saved and unused.
• Question parts silently dropped: the mechanism half of the psilocybin question, the consequences half of the 1918 influenza question, the production-systems half of the LSM question.
• The harness's own gap section contradicting the body: it said no source states phase 3 weight-loss results directly beside six reported phase 3 results.
• A report with no inline citations at all, passed as clean.
• A test-preparation answer key cited as an "analysis".
• One study counted three times, as evidence for and against.

What went well: one report (remote work) hedged every vendor figure with its provenance and said plainly that the evidence was mixed. It was the best calibrated report of the ten, and it was built on only 8 findings.

Where the problems arise

Almost all defects were attributed to the writing stage. That is partly by construction, because seven of the ten runs reuse saved evidence, so retrieval failures are under-sampled. The one clear evidence-stage case: for the remote work query the pipeline fetched the fewest pages of any of 32 saved evidence sets (39, against a median of 88) and kept 15 quotes (median 43), so the writer received 8 findings from 7 sources while research papers sat unused in the saved pages. The keep rate itself is not the signal: 1.3% is ordinary, since across the 32 sets it runs from 0.4% to 1.7% (median 1.1%). A low absolute count of kept quotes and fetched pages is what marks a thin evidence set. That report was also the best calibrated of the ten and had the fewest defects, so thin evidence did not by itself produce more errors here.

What the existing checks did

Of the four check firings verified by hand, one was a real catch (a wrong citation in the GLP-1 report, not one of its severity 3 defects) and three were false alarms: a source that did support the claim (1918 influenza), a figure compared with an unrelated microdosing page that held the run for about 30 minutes (psilocybin), and a speed gain compared with a pollution reduction (congestion pricing). The severity 3 defects above were not flagged by any check. The scorer itself had blind spots: it reported "0 of 26 sources cited" for a report with 14 numbered sources because of the source list format, and it counted a source as cited when it was only listed, so a body without inline citations scored as clean. This is a sample of four firings, not a rate.

Second labeller

On 40 random claim and cited-source pairs the blind second labeller agreed with my first labels on 28 (kappa 0.40, against 0.85 on an earlier 79-pair set). Re-reading the 12 disagreements against the full source text: my first labels were wrong on 8 (too lenient on interpretive add-ons such as "greatest to date" or a review "concluding" something it reported as mixed), the second labeller was wrong on 4 (three strict false negatives and one missed backwards figure). Against that adjudicated truth my labels were right on 32 of 40 and the second labeller on 36 of 40. The adjudication was done by the first labeller, so it is not independent. The practical reading is that the defect counts are probably lower bounds.

Limits

• One labeller plus one machine check; a 20-item human review agreed with every sampled finding, but it was one reader (the project owner) and covers about a fifth of the findings.
• Ten reports; three of the queries are policy or technology rather than science.
• Seven reports are one run each from saved evidence, so run-to-run variance is not measured.
• Retrieval and fetching failures are under-sampled by design.
• The three older reports come from earlier code; one extra September Raft report was left unaudited for the same reason.
• Counting error is real: my own tallies drifted during the audit and were corrected from the label files, which are the authority.

Candidate fixes, recorded as hypotheses only

None of these is built. Each needs a labelled set and a pre-registered A/B before any default changes.

1. Check that a cited figure sits next to the claim's subject in the cited source, not merely somewhere in the saved sources, and flag a relation that reverses it (to versus by).
2. Check the gap section against the body, so a statement that no source exists is not printed next to a result that answers it.
3. Make the scorer count in-text citations, not entries in the source list, and accept both source list formats.
4. Compare the dates in a report with the newest dated saved source (outdated status).
5. Check that every part of the query maps to a report section.
6. Flag the same metric from different editions or years side by side.
7. Make the cross-source contradiction check require the same quantity and subject, since its false alarms cost the most time.
8. Warn when the evidence stage keeps few quotes or fetches few pages in absolute terms (the keep rate is uninformative). Weak: the one thin set produced the best calibrated report, so this may not be worth building.

Follow-up (same day)

• Two false-alarm repairs merged into the working branch. The cross-source contradiction check read a decimal percentage such as 29.1% as a bare decimal and compared it with unrelated numbers; the claim-support check treated "Pacific Island" and "Pacific Islands" as different terms. Each has a test that fails without the fix. On the ten audited reports the first check no longer fires and the second keeps its three real catches and loses its false alarm.
• Cheap validation instead of a new batch. A planned batch of ten new reports (6 to 10 hours of GPU) was cancelled. Eight existing reports were drawn by a fixed rule and labelled before any detector ran on them: 59 defects, 14 of severity 3, and 7 of the 8 reports have at least one severity 3 defect (first set: 10 of 10).
• Detector results on the new reports. The figure-in-context detector fired 16 times with no false hit, but all hits came from one class, citation numbers that do not match the source list, so it stays advisory. The citation-structure rule for many citations on one sentence scored 0.67 and was dropped. The weak-source list was precise (4 of 4) but found only 2 of 4 reports with a weak-source defect. A new false alarm class was found in the claim-support check (a sentence beginning "Using Raft ..." is read as a two-word term) and is not fixed.
• A frequent defect with no detector yet: comparisons on different measures were a severity 3 defect in 5 of the 8 new reports (for example a country's internet penetration set beside the other's share of services online, or the OECD index set beside the UN index under one name).
• Advisory hints. evaluate_run prints the detector findings under the scorecard as hints, never as flags.
• Reader notes in every report (mandatory). Every finished research report now ends with a marked "Reader notes (automatic, unverified)" block. It lists up to five hints from the three best-measured detectors (a figure that is not in its cited source but is in another page that was read, a body with no inline citations, a date that has already passed), or says that no hint was raised and that this does not mean the report is free of errors. There is no setting to turn it off. The block is added once after the last completion check, removed again before any check on a resumed run, contains no links or numbered lines so the citation parsers read the report unchanged, and never fails a run. It was shown on real reports before merging: on the GLP-1 report it names the invented dose and three wrong-source figures but also lists one correct dose beside them, which is why it is worded as hints.
• Third false-alarm repair. A bold label such as "indirectly:" was fused with the next word into a made-up term ("Raft Kubernetes") that no source contains, which flagged a supported claim; it now counts as a sentence break.

Files

Protocol, worksheet and label tools, label files and the tabulation script are under eval/audit_runs/ in the main repo. Counts in this page come from tabulate.py over the label files.