Literature Review Writer Role

Literature Review: The Writer Role

Part 6 of the Literature Review. Written 2026-10-25 as the academic and implementation background for redesigning the report-writing stage after the Comprehension Diagnostic, which found that writing, not reading, is the main bottleneck of the pipeline.

1. Question and method

Question. How can a report writer built on small local models be made to state only what the evidence supports, cite it correctly, and let a reader verify it quickly, without relying on the writer to police itself?

Method. Candidate papers were found by search, downloaded into the gitignored papers/ folder, and read by me from the extracted full text, not from search summaries. Each entry states what was read. The review uses the project's convention: ✅ means the whole paper was read from the extracted text, main body and every appendix, so that a checkmark implies totality. Two things were not read for any entry: reference lists, and figure images (annotation-interface screenshots, bar charts), for which only captions and surrounding text were available. A first pass on 2026-10-25 covered only the main text; the appendices were read in a second, complete pass on the same day, and the entries below include what that pass added. Four downloaded papers were not read and are therefore not cited: the original QA-blueprint paper (arXiv:2207.00397), LongCite (2409.02897), XGrammar (2411.15100) and the survey of hallucination in natural language generation (2202.03629).

Two limits apply to the whole review. The studies use English text or mostly English text, and almost all use models far larger than the 8 to 12B models this project can serve, or fine-tune small encoder-decoder models, so effect sizes do not transfer directly to prompt-only use of local models.

2. Papers read

2.1 Gao, Yen, Yu and Chen (2023), "Enabling Large Language Models to Generate Text with Citations" (ALCE), EMNLP 2023, arXiv:2305.14627 ✅ (24 pp)

Read: all sections and appendices A to I (24 pages plus tables). Example tables 30 and 31 were read as text.

• Defines the task and metrics that most later work uses: each output statement cites at most three passages; citation recall is whether the concatenated cited passages entail the statement (judged by an NLI model), and citation precision flags citations that are irrelevant. Fluency (MAUVE), correctness and citation quality are reported together because each alone can be gamed: copying the top retrieved passage scores near-perfect on citation quality (99.4) but collapses fluency (20.8).
• The paper's headline: even the strongest systems lack complete citation support about half the time on ELI5.
• Citing after the fact is poor: writing closed-book and adding citations afterwards gave citation recall 47 percent lower than the vanilla method on ASQA.
• Sampling several outputs and keeping the one with the best automatic citation recall (RERANK) improved citation quality, and a human study confirmed it (ASQA citation recall 74.7 to 79.3, ELI5 50.8 to 59.7).
• Summaries or snippets of the passages raise correctness but lower citation quality (lossy compression). Putting more passages in context did not help ChatGPT beyond the top three. Instruction-tuned models cite far better than base models, and at least one demonstration helps.
• Limits the authors state: the NLI judge cannot detect partial support, and the datasets omit multi-hop and mathematical reasoning.
• Appendix findings. The automatic metrics were validated against people at moderate accuracy: citation recall 85.1 percent, citation precision 77.6 percent, and detection of irrelevant citations with only 66.1 percent precision (a high false-positive rate, because partial support is missed). The NLI judge was 80 percent accurate on sub-claims. Open models cite far worse than ChatGPT: on ASQA vanilla citation recall ranged from 6 to 51 percent for 7 to 13B models and was about 63 percent for the 70B models, against 73.6 for ChatGPT (reranking lifted Vicuna-13B from 51 to 72). A full instruction beat a short one (citation recall 73.6 against 69.6), one demonstration lifted recall from 69.3 to 74.6, and a fine-tuned fusion-in-decoder baseline trained on 4000+ examples lagged prompted ChatGPT on both correctness and citation quality. Reranking by citation recall cost about eight times the single-pass runtime in the later GenerationPrograms measurement. Most runs used three seeds; reranking, GPT-4 and the 16K model used one.
• Relevance: it supplies the evaluation grammar (recall, precision, plus fluency and coverage guards) and the evidence that post-hoc citation and prompt-only citation are weak, while best-of-N selection by a cheap verifier works.

2.2 Slobodkin, Hirsch, Cattan, Schuster and Dagan (2024), "Attribute First, then Generate: Locally-attributable Grounded Text Generation", ACL 2024, arXiv:2403.17104 ✅ (36 pp)

Read: the main text and appendices A to K (datasets, metric validation, implementation, prompts, examples, human study, latency, post-generation baseline). The annotation-interface screenshots were not inspected.

• Splits generation into content selection (verbatim source spans, validated by string matching), sentence planning (clusters of spans per sentence) and sentence-by-sentence generation conditioned on the cluster and the sentences already written. The selected spans become the citations, so attribution is a by-product, not a second task.
• Citations are about 45 times shorter than citing whole documents, and human verification time fell by roughly half. Attribution accuracy exceeded the ALCE-style baseline on long-form question answering; on multi-document summarization the baseline's document-level citations were more accurate but left twice as many sentences without any attribution. Prompted frontier models (Gemini) outperformed the fine-tuned small models. Conditioning on the preceding sentences made about 30 percent of sentences more cohesive.
• Limits: several model calls per output, so more computation and slower generation; sentence fusion occasionally added information beyond the selected spans, though most of the missing support sat in adjacent text.
• Appendix findings. Prompted-model cost is large: per instance, multi-document summarization took about 33 s for content selection plus 5 s for planning and 6 s for fusion, against 9 s for the ALCE-style baseline, and the iterated fusion step processed about 17,000 input tokens per instance; the fine-tuned content-selection step was slower still (69 s). Human attribution judgments had only fair agreement (Fleiss kappa 0.37). A post-generation variant (generate first, then retrieve support in the style of RARR) was clearly worse (attribution 67.8 on summarization and 50.2 on question answering, with 9.6 percent of answers unattributed). A full-sentence version of the attribution score shows that the strict span-level score understates support for the fine-tuned variants (64.4 against 80.1 on summarization), and in that table the ALCE-style baseline scored higher than Attribute First on summarization (88.7 against 79.5 for the in-context versions) but far lower on question answering (49.8 against 78.7 to 89.3). The constrained decoding used (forbidding n-grams absent from the source) applies only to the fine-tuned 447M encoder-decoder, not to prompted models.
• Relevance: the closest published match to an evidence-first writer. It also shows the cost: extra calls and a fusion step that can still add content.

2.3 Schuster, Lelkes, Sun, Gupta, Berant, Cohen and Metzler (2024), "SEMQA: Semi-Extractive Multi-Source Question Answering", arXiv:2311.04886 ✅ (19 pp)

Read: the main text and appendices A to E (writer task, rating study, dataset statistics, implementation, exemplar ablation, example prompts and outputs). The tool screenshots were not inspected.

• Answers interleave quoted spans copied verbatim from the sources, each marked with its source id, with free-text connectors that glue them into one passage. Because the quoted spans are literal, precision and recall can be scored by string matching, without a model judge.
• A user study found the semi-extractive answers more comprehensive, significantly more correct, and with more helpful attribution than ALCE-style cited abstractive answers, and they were verified more than twice as fast per word. Fluency was slightly lower, because gluing extracted spans coherently is hard; human-written references scored near-perfect fluency. A prompt using span-level quotes beat the same material converted to sentence-level citations.
• Limits: English questions and English Wikipedia sources only; the format is not bullet-proof against out-of-context quotation or wrong coreference; not suited to every question type.
• Appendix findings. Human answers were capped at 100 words and written by trained journalism graduates after three pilot rounds, one writer's overly extractive answers being sent back for rewriting; rater agreement was high (Krippendorff alpha 0.75, 100 outputs per model, three raters). The few-shot model used temperature 0 and choosing demonstrations by question similarity beat random demonstrations (task score 62.4 against 59.2), although random four-shot still beat nearest-neighbour one-shot. The appendix example shows the failure mode of the baseline plainly: the ALCE-style answer to an ambiguous question used one of four sources, while the semi-extractive answer covered all four.
• Relevance: direct support for rendering evidence verbatim with machine-checked provenance and letting the model write only the connectors, and a measured fluency cost to plan for.

2.4 Wan, Hirsch, Stengel-Eskin, Dagan and Bansal (2025), "GenerationPrograms: Fine-grained Attribution with Executable Programs", arXiv:2506.14580 ✅ (27 pp)

Read: the whole paper, including the conclusion and appendices A to E (open-model results, latency, correctness checks, program statistics, prompts).

• The model first writes a program of text operations (extract, paraphrase, compress, fuse) over numbered source sentences, then each operation is executed by a constrained model call whose output must be entailed by its inputs ("local attribution"). The program trace gives the citations.
• With GPT-4o, attribution F1 rose by 20.4 and 39.0 (document level, ASQA and LFQA) and 25.2 and 28.8 (sentence level) over ALCE; the paper writes these as percent, but the table values (for example LFQA 55.4 to 94.4) show they are absolute points. Correctness fell slightly (2 percent on ASQA, 7.1 percent on LFQA at document level). Every generated sentence carried a citation, whereas ALCE omitted citations from 25.8 and 40 percent of sentences. Re-sampling only the operations flagged as non-entailed (4.5 percent of operations on LFQA) improved attribution further and was cheaper than re-sampling whole outputs.
• Appendix findings. The result was repeated with an open model, Llama 3.3 70B, with the same direction (for example LFQA document-level attribution 62.1 to 91.3, ASQA correctness 51.8 to 46.1), and scoring those outputs with GPT-4o argues against self-preference; but 70B is far above what this project can serve, so it does not show that a small model can write valid programs. The correctness drop is largely stylistic: with a human-correlated LLM judge the difference vanished (87.5 against 86.7, p = 0.6) and in a 25-example human check programs scored higher (4.3 against 3.8), while outputs ran longer (about 141 against 87 words on LFQA). Only 1 invalid program tree occurred per dataset. Each sentence is generated independently with no cross-sentence smoothing, yet automatic fluency and coherence scores matched ALCE. Re-sampling flagged operations added about 1.8 s per example, whereas reranking whole outputs for ALCE took about eight times its single-pass time (3.7 s to 30 s). The attribution metric agreed with two human raters on 78.1 percent of 50 examples, and the judge for long inputs was GPT-4o.
• Relevance: the strongest published case for a plan that is executable and checkable per step, and for localised repair instead of whole-report regeneration.

2.5 Fierro, Amplayo, Huot, De Cao, Maynez, Narayan and Lapata (2024), "Learning to Plan and Generate Text with Citations", ACL 2024, arXiv:2404.03381 ✅ (21 pp)

Read: the main text and appendices A to H (question generation, answerability classifier, blueprint selection, grounding, human study, datasets, example outputs).

• Plans are sequences of questions that serve as a blueprint. Fine-tuned LongT5 models with a blueprint attributed better than the same model without one; the extractive blueprint (questions copied from the passages) was best, and its plans were more grounded. Without a plan, adding attribution slightly hurt summary quality. Attribution transferred zero-shot to ALCE and was competitive with prompted LLM pipelines, and a human study rated the blueprint model's citations most accurate.
• Limits: encoder-decoder models only, and mistakes remain out of domain.
• Appendix findings. The blueprint pipeline needs auxiliary models: two T5-3B question generators and a T5-11B answerability classifier. Removing, after the fact, blueprint questions the passages cannot answer raised attribution by 7.09 points without hurting other measures, which is evidence that a plan can be corrected mechanically before writing. Extractive blueprints were less grounded in the final summary than abstractive ones (80 against 96 percent at sentence level), even though they attributed better. In the human study, the share of fully supported sentences was 88.6 percent (ASQA) and 87.4 percent (ELI5) for the blueprint model, against 78.0 and 75.8 for vanilla ChatGPT and 57.9 and 37.6 for Vicuna-13B, with annotator agreement of 70, 93 and 89 percent on the three datasets. The authors also observe that the no-plan baseline drew on fewer passages than the plan-based models.
• Relevance: planning helps attribution when the plan is tied to the sources. It does not show that a prompted small model can plan for itself.

2.6 Grenander, Varia, Czarnowska, Vyas, Halder and Min (2025), "Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models", arXiv:2504.09071 ✅ (18 pp)

Read: the main text and appendices A to G (examples, plan-generation prompt, baseline prompts, evaluation rubric, compute, training details).

• Fine-tuning a 3.8B model (Phi-3.5-mini) with QA, coarse event, or combined plans gave no significant gain over no planning in quality or faithfulness on long narrative texts. A human study (five documents) found the models' own plans contained hallucinations at about the same rate as their summaries, so the summaries were as unfaithful as before.
• Replacing the model's plans with high-quality plans from a frontier model improved coverage (about 10 points) and faithfulness (about 6 points) in the human evaluation, but the model "follows the plan and then veers off into hallucinated generation", so faithfulness was only partly repaired.
• Limits: narrative fiction and transcripts, fine-tuned models under 4B, tiny human sample, automatic faithfulness metrics that disagreed with each other.
• Appendix findings. The plans used to train and to repair were written by a frontier model from the reference summaries, so the "external plan" condition is an upper bound that a deployed system will not have. Fine-tuning used LoRA for 100 epochs on 8 A100 GPUs for 4 to 24 hours per setting, with beam search (5 beams) and a ban on repeated 5-grams. The human evaluation was done by two of the authors, with coverage, faithfulness, conciseness and plan-grounding computed from atomic facts, and the Claude baseline was the best of 16 prompts tried.
• Relevance: a caution. A small writer's own plan is not trustworthy, and even a good external plan does not prevent drift during generation, so plans must come from verified evidence and the prose still needs checking.

2.7 Zhou, Zhang, Poon and Chen (2023), "Context-faithful Prompting for Large Language Models", Findings of EMNLP 2023, arXiv:2303.11315 ✅ (13 pp)

Read: the whole paper including the ethics note and the appendix on automatic prompt engineering.

• Recasting the context as a narrator's statement and asking for the narrator's opinion ("opinion-based prompts"), combined with an instruction and with counterfactual demonstrations, makes models rely on the supplied context more. On Natural Questions the memorization ratio of text-davinci-003 fell from 35.2 to 3.0 percent; LLaMA-2-7B-chat also improved on knowledge conflicts. For abstention the gains were reliable for the large model but not for smaller InstructGPT variants, which the authors attribute to weaker reading comprehension.
• Limits: reading comprehension and relation extraction, not long-form generation; the context is assumed reliable.
• Appendix findings. The instructions used were produced by automatic prompt engineering with a large model (text-davinci-003) for every model size, because the smaller models did not generate meaningful instructions themselves; the best instruction on each dataset is a one-line task description.
• Relevance: prompt-level levers exist, but this evidence is for short tasks and mostly large models.

2.8 Tam, Wu, Tsai, Lin, Lee and Chen (2024), "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models", EMNLP 2024 Industry, arXiv:2408.02442 ✅ (19 pp)

Read: the whole paper including limitations, appendices A to G (parser validation, additional models, per-dataset costs and tables, prompt variations).

• Strict output constraints (JSON mode, constrained decoding) lowered accuracy on reasoning tasks (GSM8K, last-letter, shuffled objects) and raised it on classification tasks; looser constraints gave better reasoning and lower variance. Parsing errors were not the main cause. In one case a model placed the answer key before the reasoning key and so skipped its own reasoning. Converting a free-text answer to the target format in a second step kept free-text performance; supplying the schema in the prompt hurt. A newer structured-output API still left free text slightly ahead on two of three reasoning sets.
• Limits: short reasoning and classification tasks with two-field schemas; no long-form writing.
• Appendix findings. The effect is strongly model- and task-dependent and noisy. Per-model tables show JSON above free text for Gemini-1.5-Flash on two reasoning tasks (last letter 77.0 against 65.4), a collapse for LLaMA-3-8B on GSM8K (74.7 free text to 48.9 JSON), a small drop for Gemma2-9B on GSM8K (86.5 to 84.2) but a large one on last letter (56.8 to 39.0), and large classification gains from structure for the two small models on the medical-diagnosis task (for example Gemma2-9B 22.9 to 53.0). Standard deviations across the nine prompt variations are often large (for example GPT-3.5 last letter with JSON, 25.2 plus or minus 29.1), so single-prompt comparisons are unreliable. How the final answer is extracted matters as much: a strict pattern match gave 10 to 83 percent accuracy where an LLM parser gave 52 to 86 (Mistral-7B 10.4 to 52.4), and the GPT-4-turbo parser agreed with humans on 97.7 percent of 300 responses. Token cost differed little between formats. The authors note they could not test larger models such as LLaMA 70B for cost reasons.
• Relevance: constrain narrowly (the citation field), keep evidence fields before claim fields, and do not wrap prose in a rigid schema. The project's own earlier reading of "The Constraint Tax" (see Verified Papers) points the same way for small models.

2.9 Du, Xu, Zhu, Wang and Mao (2025), "DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents", arXiv:2506.11763 ✅ (31 pp)

Read: the whole paper including limitations, appendices A to H (dimension definitions, judge selection, citation and agreement formulas, configuration, prompt templates).

• 100 expert-written research tasks (50 Chinese, 50 English) over 22 domains, weighted by the topic distribution of real queries. RACE scores report quality with a judge model, adaptive task-specific criteria and a reference report, on four dimensions (comprehensiveness, depth, instruction following, readability); FACT extracts statement-URL pairs and has a judge model decide whether the page supports the statement, reporting citation accuracy and effective citations per task.
• RACE agreed with human pairwise preferences more often than human annotators agreed with each other, in the authors' study of 50 Chinese tasks. Deep research agents differed little on RACE but widely on effective citations and on citation accuracy.
• Limits: the judge is a proprietary frontier model (Gemini 2.5 Pro) and the reference reports come from a commercial agent, so scores are relative; RACE measures quality, not faithfulness; FACT's support judgment is a binary model call, not a source-grounded check.
• Appendix findings. The FACT judge (Gemini 2.5 Flash) agreed with humans on 96 percent of 'support' and 92 percent of 'not support' decisions in a sample of 100 statement-source pairs. Human validation covered 50 tasks and 600 reports, costing 225 person-hours (about 1.5 hours per query per expert), and per-task correlations were computed on the 37 tasks where expert agreement was non-negative, so the headline agreement figures are partly conditional on consistent raters. Citation accuracy counts a task with no citable statements as zero. The compared systems were run in April and May 2025 with up to 36,000 output tokens, a 16,000-token reasoning budget and five search turns. The authors list small benchmark size, possible domain bias and limited human throughput as limitations.
• Relevance: a model for multi-axis, human-validated report evaluation, and an existing multilingual task set, but not a drop-in for local, source-grounded measurement.

2.10 Haussmann (2025), "The Challenge of Achieving Attributability in Multilingual Table-to-Text Generation with Question-Answer Blueprints", arXiv:2503.23204 ✅ (14 pp)

Read: the whole paper including appendices A to F (training details, the attributability metric, repetition penalty, blueprint translation, examples).

• Blueprints raised attributability for models fine-tuned on English data, but not in the multilingual setting (African languages). Reasons given: blueprints machine-translated from English introduced errors, models did not rely closely on their own blueprints, English blueprints led to mixed-language output, and the data were small. Generic metrics (BLEU, chrF) correlated poorly with human attributability, and the English-only FactKB metric could not be applied to other languages.
• Limits: tiny dataset, small mT5 models, table-to-text.
• Appendix findings. The attributability metric (STATA) is a regression fine-tuned on about 4,900 human-labelled rows, 74 percent of them negative. Translating a blueprint lost a technical term ("tercile" became "area" or "zone"), the sort of drift the author names as a cause of the failure. In the example outputs the blueprint was sometimes right (correct figure and year) while the verbalisation ignored it, or the verbalisation invented data, so a correct plan did not guarantee a faithful sentence. A repetition penalty of 1.2 was needed to stop degenerate repetition, although 1.4 scored higher.
• Relevance: planning scaffolds trained or translated for English do not carry over, and evaluation needs language-aware or structural checks.

3. Synthesis for DeepDelve

1. Attribution by construction beats attribution by instruction. Prompt-only citation is unreliable (ALCE) and post-hoc citation is worse (ALCE; the post-generation variant in Attribute First). Methods that fix the evidence before the prose (Attribute First, SEMQA, GenerationPrograms) give better attribution and faster human verification, at costs that are measurable: more calls and latency (several times the baseline for prompted Attribute First), and small correctness or fluency losses that are partly stylistic. The evidence for prompted versions comes from frontier or 70B models; only the fine-tuned variants were tested on small models.
2. The writer should not plan for itself. Small-model planning reproduces its own hallucinations (Grenander et al.), and even a good plan lets the model drift. Plans should be derived from verified evidence and checked mechanically. The stage-graph pipeline's evidence table, whose quotes are verified as literal source substrings, is exactly such a plan.
3. Constrain narrowly and cheaply, and measure on our own task. Restricting the citation identifier to the allowed set removes a class of error without wrapping prose in a rigid schema (Tam et al.), and evidence fields belong before claim fields. But the published effect of format restriction swings in sign by model and task and is noisy across prompts, including for 8 to 9B models, so the direction cannot be assumed for local models writing citations.
4. Verification is cheap enough to select with. Sampling a few drafts and choosing the one with the fewest unsupported or mismatched elements improved citation quality (ALCE), and localised repair of only the failing operation is cheaper than regenerating (GenerationPrograms).
5. Prompts help modestly. Faithfulness prompting (Zhou et al.) works but is shown for short tasks; it is a complement, not a fix.
6. Evaluate on several axes. Citation faithfulness alone is gameable by copying (ALCE), so any evaluation of a quote-based writer also needs coverage of the query's parts, fluency and verification time. RACE and FACT are useful models for this but depend on proprietary judges.
7. Multilingual design. Keep quotations verbatim in the source language, put only connectors in the reader's language, avoid translating planning scaffolds, and prefer structural checks (a figure not present in the evidence, an identifier outside the allowed set) to English-only metrics.

4. Implementation notes

• Ollama structured outputs. The format parameter accepts a JSON schema; the documentation advises lowering the temperature and also describing the schema in the prompt, and does not state how the schema is enforced. Enforcement, and behaviour with thinking models, are therefore untested and are listed under the hypotheses (Section 5).
• llama.cpp grammars. The grammar format (GBNF) can constrain output directly or be generated from a JSON-schema subset. The documentation lists limits that matter for citation constraints: unsupported keywords are skipped silently, nested references are broken, and optional-repeat patterns can make sampling very slow (bounded repeats are recommended). A schema must therefore be validated by testing, not assumed.
• Deterministic rendering needs no model library. Rendering a findings table and gap sentences from structured records is ordinary templating; the language-dependent part is the short set of fixed labels, which should live in one string table.

5. Hypotheses to test

Each requires a labelled comparison on the existing ten findings sets before any default changes.

• H1. Code-rendered facts, figures, citations and gap statements, with the model writing only connectors and the evidence paragraph, remove severity 3 defects (invented figures and sources) and the leaked pipeline phrasing, at some cost in fluency (SEMQA's measured trade).
• H2. Constraining the citation field to the allowed identifiers eliminates invalid citation ids for the models that currently fabricate them, without lowering other quality measures.
• H3. A task prompt that asks for facts, caveats and gaps, permits evaluation only from listed caveats, forbids new figures, names and acronym expansions, and sets a length budget lowers the evaluative-filler and wrong-expansion defects.
• H4. Writing section by section from only that section's findings, or selecting the best of a few drafts by a deterministic figure and citation check, lowers residual defects further.
• H5. A separate reader-extractor and writer pairing, using the strongest reader and the lowest-defect writer from the diagnostic, outperforms a single model on both stages.
• Outcome measures. Writer defect rate and severity 3 count (the diagnostic's labels), invalid-citation and invented-figure counts (deterministic), coverage of the query's parts, a fluency or readability rating, and the time a reader needs to verify a report.