Comprehension Diagnostic

Comprehension Diagnostic: Reading versus Writing in Local Models

Run 2026-10-24 to 2026-10-25. Status: diagnostic, not a model verdict under the Model Evaluation Standard. Scope: English only.

Summary

When the source material is a real, full-text research paper, does a locally served language model fail at reading it or at writing a report from what it read? Thirteen model setups, run on a single 16 GB GPU through Ollama, were tested on ten open-access papers with 101 hand-keyed questions and ten hand-built, true-by-construction findings files. Reading was not the dominant failure: by the pre-registered rule (more than 20 percent wrong or invented answers), no setup was a reading bottleneck, although two lost 9 to 19 answers to empty output. Writing was the weak stage: no setup fell below the 10 percent defective-unit threshold, and the lowest observed rates were 10.3 percent (gemma4:12b at its largest context) and 10.9 percent (ornith-1.5:9b). A 9B model (ornith-1.5:9b) was the strongest overall on both stages. Reading and writing quality were only weakly related across models, and longer, more evaluative reports carried more defects. A model purpose-built for cited report writing fabricated about 39 percent of its content under the same strict-evidence conditions.

1. Question and scope

The pipeline separates research (retrieval and note taking) from report writing. Earlier audits attributed most defects in finished reports to the writing stage, but could not separate a reader that misunderstands a source from a writer that distorts a correct understanding. This diagnostic isolates the two: reading is tested with the whole paper in context and no pipeline around it; writing is tested from perfect input, so any defect belongs to the writer. All papers and prompts are English. The intended users work in many languages, so results here do not transfer to other languages without a separate check.

2. Method

2.1 Instruments

• Corpus. Ten open-access (CC BY) papers from 2026, five biomedical and five from other fields, selected by a fixed rule. Two reached the model degraded by PDF conversion and were kept deliberately.
• Questions. 101 questions with answer keys written before any model was run: numeric recall (N), checks of a stated claim against the paper (C), stated limitations (L), direction and attribution (D), and questions the paper cannot answer (U). The C items include premises that the paper does not support (for example, an effect-size ratio the authors never claim) and internal inconsistencies in the papers themselves.
• Reading task. The whole paper in context, one question at a time, graded by one labeller against the key as correct, partly correct, wrong, correctly unanswerable, invented, or empty.
• Writing task. One call that receives a findings file built from the keys (true by construction) and the pipeline's own Builder instructions. Every claim-bearing unit (sentence, bullet or table row) is labelled for defects using the audit taxonomy (unsupported, overstated, wrong figure, invented source, format artefact) with severity 1 to 3. The unit rule is identical for every model.
• Decision rules fixed in advance. Reading is a bottleneck above 20 percent wrong or invented answers; writing is a bottleneck above 10 percent defective units.

2.2 Models and serving

All models ran on Ollama 0.40.9 on a 16 GB card, one at a time. Models too large for that card were excluded. Thinking controls were tested on the raw API before each run (Section 3.3). The context window was the largest value that loaded fully on the GPU, with the exceptions described in Section 3.2.

2.3 Run conditions

Ten setups ran under one shared condition chosen for comparability: temperature 0.2, fixed seed, a 2000-token reading cap and a 4000-token writing cap. These are not the models' recommended settings, so those rows are lower bounds. The remaining setups used settings from the model's own documentation:

Setup	Sampling and caps	Source
Tongyi-DeepResearch-30B	temperature 0.6, top_p 0.95, presence penalty 1.1, 10,000 tokens	official repository inference code (the model card gives none)
AstaBrief 8B	temperature 0.7, top_p 0.95, 4096 tokens, its own training prompt and chat template	model card
gpt-oss rerun	named reasoning level medium, temperature 1.0, top_p 1.0, top_k 0, 16,384 tokens	the vendor's repository README (the model card gives none)
Marco rerun (stopped)	temperature 0.7, top_p 0.95, 16,384 tokens	model card

3. Results

3.1 Reading and writing

"Right" counts correct answers plus correct refusals on unanswerable questions. "No answer" is empty output or a cap hit without a final answer. Intervals are Wilson 95 percent intervals on the 101 questions. Severity 3 denotes a wrong or invented figure, source or claim.

Setup	Conditions	Right of 101 (95% interval)	Partly	Wrong	Invented	No answer	s/answer	Defective writing units	Severity 3	Reports citing inline
ornith-1.5:9b	think off, T 0.2, 64K	95 (88 to 97%)	4	2	0	0	17	10.9% (30/274)	1	10/10
MiMo-V2.6-Distill-Qwen-9B	think off, T 0.2, 262144	94 (86 to 97%)	5	2	0	0	26	17.0% (63/370)	0	9/10
Tongyi-DeepResearch-30B (IQ3_XS)	no off switch, recommended sampling, 49152	94 (86 to 97%)	4	3	0	0	19	20.2% (52/257)	0	2/10
gemma4:12b	think off, T 0.2, 64K	91 (83 to 95%)	6	4	0	0	8.1	11.7% (25/213) (10.3% at 262144)	0	10/10
gpt-oss 20B, rerun	level medium, T 1.0, cap 16384, 64K	90 (82 to 94%)	5	5	1	0	24	20.5% (79/386)	5	10/10
gpt-oss 20B, level low	T 0.2, cap 2000, 64K	85 (76 to 90%)	6	8	2	0	8.5	20.3% (71/349)	0	9/10
granite4.2:8b	think off, T 0.2, 64K	84 (75 to 89%)	12	5	0	0	19	29.0% (82/283)	1	3/10
Ling-3.0-tiny	think off, T 0.2, 64K	82 (72 to 88%)	10	8	1	0	2.8	19.0% (56/295)	2	10/10
Bolt-E4B	think off, T 0.2, 64K	82 (72 to 88%)	13	6	0	0	8.6	11.6% (16/138)	0	10/10
gemma4:e4b	think off, T 0.2, 64K	77 (67 to 83%)	13	11	0	0	9.0	13.3% (29/218)	1	10/10
gpt-oss 20B, original default	think=false (no instruction), T 0.2, cap 2000, 64K	74 (64 to 81%)	3	5	0	19	24	15.4% (55/356)	0	7/10
Marco-DeepResearch-8B	no off switch, T 0.2, cap 2000, 64K	67 (57 to 75%)	9	11	5	9	30	15.9% (47/295)	0	0/10

AstaBrief 8B, run in its native prompt format, is reported separately because it is not a reading model: about 39 percent of its writing units were defective (251 of 650, estimated by section rather than sentence), including 46 severity 3 defects, and every report carried inline citations, many of them fabricated. In the project's Builder prompt it produced empty or non-report output for four of ten papers. Its reading was not graded.

Marco's rerun at its recommended settings was stopped at 50 of 101 answers because of repetition loops (Section 4, finding 4); only its first run appears above.

3.2 Effect of context size

Every prompt in the study is below 19,300 tokens, so a larger window adds no input. Four models were rerun at the largest context that loaded fully on the card: Ling-3.0-tiny (131072), granite4.2:8b (98304), ornith-1.5:9b (262144) and gemma4:e4b (131072). All four produced identical text to their 64K runs. gemma4:12b at 262144 produced different text for 98 of 101 answers and all ten reports with unchanged reading totals (91 right) and a writing rate of 10.3 percent against 11.7 percent. A control run of the same model at 64K reproduced the original run exactly, so the change is caused by the larger window and not by sampling noise. gpt-oss (13 GB of weights) was not rerun at larger context; Bolt-E4B was already at its native maximum.

3.3 Thinking controls

Model	Behaviour
gemma4 family, Granite 4.2, Bolt-E4B, ornith, Ling, MiMo	think=false is an effective off. Bolt-E4B needs the gemma4 renderer and parser; MiMo's Modelfile contains a stop string that returns a single empty token when thinking is on.
gpt-oss	No off mode. Boolean false sends no reasoning instruction, so the model runs at its default; a named level (low, medium, high) must be sent.
Marco, Tongyi	No off switch. The template always opens a think block, so think=false only moves the reasoning into the visible answer.
AstaBrief 8B	In its own template and prompt it does not think and the flag changes nothing (outputs byte-identical). Ollama's library-matched template forces a think block that the model was not trained on, and had to be replaced.

4. Findings

1. Reading was not the main bottleneck. No setup exceeded 20 percent wrong or invented answers. Two lost answers to empty output instead: the original gpt-oss default (19) and Marco (9).
2. Writing was the main bottleneck. No setup was under 10 percent defective units. The strongest reader-writer combination was ornith-1.5:9b (95 right, 10.9 percent, one severity 3 defect). A second model of the same architecture family (MiMo) read equally well (94) but wrote at 17.0 percent, so writing quality tracked the individual model's training more than its size or reading skill.
3. Reasoning models read well and write loosely. MiMo, Tongyi and the gpt-oss rerun, which reason or elaborate at length, read 90 to 94 right but wrote at 17 to 21 percent, with long reports, evaluative filler, and in gpt-oss's case invented figures in tables (five severity 3 defects). This is an observation on a small number of models, not an established mechanism.
4. Models without an off switch need a reasoning-aware pipeline. Tongyi finishes cleanly under a 10,000-token cap, but its reports retain the reasoning and eight of ten omit inline citations. Marco enters a repetition loop (an endless run of closing think tags) on some questions; four such answers consumed 41 of the first 69 minutes of its rerun, so enlarging the cap does not rescue it.
5. The gpt-oss deficit was a settings problem. Sending an explicit reasoning level with the vendor's sampling and a larger cap removed all 19 empty answers and raised reading from 74 to 90 right. Writing did not improve.
6. A retrieval-trained report writer is unsafe under strict evidence. AstaBrief was trained to cite supplied quotations and to add background from memory. Given ten finding-sized references, it filled the gap with fabricated citation keys, a fabricated URL on every citation of one report, invented authors and an invented reference list, and effect sizes for outcomes the findings never state.
7. Context size did not change grades. See Section 3.2.
8. The same items fail across models. Accepting a premise the paper does not support (an effect-size ratio claimed twice as large), significance inferred from overlapping intervals, a stated correlation bound that the data exceed, a percentage that does not match its table, a text-versus-table discrepancy on quantization, a self-contradictory ablation ranking, and the ordering of ring entropy against diffraction-peak order. These are the items to monitor in any agent run.

5. Threats to validity

• One labeller; labelling after the first model sets was not blind. The answer keys were written before any run, and one disputed key (the ring-entropy ordering) was rechecked against the source and confirmed.
• Single runs, one seed per condition, 101 questions and ten reports per setup; the reading intervals in Section 3.1 span roughly 8 to 10 points, so adjacent rows are not distinguishable.
• Conditions are heterogeneous: ten setups share a non-recommended sampling and a small reading cap, which penalises reasoning-aloud models and understates them.
• AstaBrief's writing labels are counted by section for tractability.
• Perfect input isolates the writer but removes the noise a real pipeline supplies; ten papers from English-language open-access journals are not representative of all disciplines.
• The diagnostic does not test retrieval, multi-document synthesis or non-English text.

6. Candidate interventions for the writer role (hypotheses, not built)

None of these has been implemented. Each requires a labelled comparison and the project's A/B discipline before any default changes. Items marked structural do not depend on English wording and are the safer choices for a multilingual audience; items marked English-specific should be kept in swappable configuration.

#	Intervention	Defect class targeted	Language
1	Render facts, figures, citations and gap statements with code from structured findings; the model writes only connective prose and the evidence paragraph	invented figures and sources, leaked pipeline phrasing	structural (fixed labels need a string table)
2	Constrain citation tokens to the supplied identifier set with structured output or a grammar	fabricated citation keys and URLs	structural
3	Revise the task prompt to request facts, caveats and gaps, permit evaluative statements only when a caveat supports them, forbid new figures, names and acronym expansions, and set a length budget	evaluative filler, wrong expansions, length	English-specific
4	Two-step writing: list the finding identifier behind each sentence, check the identifiers mechanically, then write	unsupported claims	structural
5	Write section by section with only that section's findings in context	gap filling from memory	structural
6	Separate reader and writer roles (for example a strong reader extracts, a low-defect writer writes), evaluated under the Model Evaluation Standard	stage mismatch	model-dependent
7	Sample several drafts and select the one with the fewest figure and citation mismatches	residual invented figures	structural
8	Strip reasoning blocks and wrapper tags from saved reports	format artefacts	structural
9	Test lower writer temperature	invented figures (suggestive only: confounded with reasoning level)	structural

Fine-tuning remains deferred by project decision. A sensible first experiment replays the existing ten findings files through items 1, 3 and 6 with the current harness.

7. Errata and reproducibility

• Erratum (2026-10-25). An earlier version of the results table listed ornith-1.5:9b as 74 right. That counted correct answers only; with the 21 correct refusals the grade file gives 95. The table above is generated from the grade files.
• Files. Per-model grades (grades_R_*.json) and defect labels (labels_W_*.json) are in eval/comprehension/, with the run scripts (sampling, context and token caps are set by DD_* environment variables) and the protocol. Model outputs and paper text stay local because they quote licensed text; the AstaBrief training prompt (CC BY-NC) is also kept local.