Skip to content

v0.3.0 — deterministic quality gates and the section-rhetoric layer

Latest

Choose a tag to compare

@Ar9av Ar9av released this 21 Sep 17:19
· 2 commits to main since this release
36c3cc4

Four new deterministic gates and a rhetorical reference layer for the drafting agents. 72 stdlib-only regression tests, no new dependencies, and no change to the number of LLM calls in any step.

What's new

The rhetoric layer

  • skills/shared/section_rhetoric.md — the App. F.1 prompts specify what each section must contain and say nothing about how its paragraphs should be shaped, which is why generated sections come back content-complete and rhetorically flat. This supplies the missing layer: three Abstract templates selected by contribution count, the Introduction logic chain (a technical challenge is a limitation plus its cause), the Related Work topic-paragraph template, the Method module triad (design → motivation → technical advantage), the three questions Experiments must answer, and scope-vs-defect limitations for the Conclusion. Wired into Steps 3, 4, and 5.

Four gates that actually check the rules

  • reverse_outline.py (Step 5, Gate C) — strips a draft to one topic sentence per paragraph and flags no-topic-sentence, overlong, multi-pivot, orphan, citation-dump. The topic-sentence sequence becomes the reviewer's evidence for the Logical Flow axis, instead of its impression of the prose. Run-in bold headings are merged forward, which removed 29 false positives on the bundled example.
  • table_lint.py (Step 4) — booktabs rule violations as ERRORs (vertical rules, \hline, caption below the tabular, missing rules) and readability conventions as WARNs (numeric columns with no ↑/↓, decimal precision that changes down a column, six-word captions, tables after the Conclusion).
  • figure_lint.py (Step 2) — makes the plotting agent's stated hard rules checkable: caption coverage, resolution, aspect ratio, self-numbering captions, and — with --paper — whether the draft actually uses every figure that was rendered. PNG geometry is read from the IHDR/pHYs chunks directly, so no imaging library enters the dependency list.
  • claim_evidence_gate.py rewritten for real LaTeX — see below.

Claim-evidence map

--out-md emits a Value | Section | Claim | Evidence | Status table, ordered needs-evidence first, for direct use as a revision agenda. Every number in a draft is our result (present in experimental_log.md), someone else's result (the sentence carries a citation), or a fabrication.

Fixed

claim_evidence_gate.py extracted almost nothing from real LaTeX. On the example paper shipped in this repo it found 1 claim and reported PASS. Three compounding causes:

  1. \% was treated as a comment start, deleting the remainder of every line that stated a percentage.
  2. Patterns were written against plain text (3.2%, 2.13x) that never survives LaTeX — real drafts carry 3.2\%, 2.13$\times$, ${>}85\%$.
  3. \cite{...} was deleted before attribution detection ran, so the citation cues were dead code and numbers belonging to cited papers were reported as our own.

Plus: dedup keyed on the value alone, so match order decided classification; and baseline was an attribution cue, which filed the most common shape of a real result ("improves over the strongest baseline by 3.2%") as someone else's number.

The same paper now yields 52 claims — 28 traced to the log, 24 correctly attributed, 0 false positives — and a fabricated uncited result injected into it is caught (exit 1) where it previously passed.

Also fixed: examples/agentic-security-report had a figure (fig_attack_success_rates) that was rendered and cited by the paper but missing from captions.json — found by the new figure linter on its first run.

Changed

  • The reverse-outline gate is Gate C of the pre-refinement sequence; the research-brief gate becomes Gate D.
  • claim_evidence_gate.py now reports findings on drafts that previously passed vacuously. It remains a WARN gate — nothing halts — but Step 0 worklog entries will start carrying claim_evidence: WARN.

Attribution

The section templates, the reverse-outlining practice, the claim-evidence contract, and the table readability rules are adapted from Master-cai/Research-Paper-Writing-Skills (MIT), which distills Prof. Peng Sida's open paper-writing notes. The deterministic extraction, linting, and attribution logic are PaperOrchestra's.


Full changelog: v0.2.0...v0.3.0