Repository navigation
v4.40.0
Ten weeks of work since 4.39.0. The headline is check-references, which asks
of each reference whether the record its DOI or arXiv id names is the work it
cites, and runs without a workflow. Three factuality checkers were added (FT,
FL, FT2), and many of the fixes below came from running the pipeline on a real
course report.
Added — check-references: does each reference say what its record says?
REFERENCE_VERIFY used to ask whether a DOI resolves and whether an arXiv id has
an entry. A DOI that resolves to somebody else's paper passed, and that is the
commonest shape an invented reference takes: a real identifier under an
invented title or author list. reference_match.py now fetches the record
behind each identifier — doi.org's CSL JSON, which also carries Retraction
Watch notices in updated-by, and arXiv's Atom entry — and compares its title,
authors and year with what the reference says. A reference with no identifier is
searched for in Crossref and OpenAlex, and, when that settles nothing, its likely
title is looked up in OpenAlex's title index; a hit is taken as the work only
when its title appears in the reference nearly whole and its authors agree. Verdicts:
verified, mismatch, not_found, retracted, needs_review (found, but the
agreement is partial or cannot be checked) and unreachable.
Inside a run, mismatch, not_found and retracted fail the reference where a
resolving DOI used to pass it, and needs_review reaches
REFERENCE_REALITY_CHECK as needs_human_review. Search runs only with web
research enabled or flags["reference_search"] set. Outside a run,
report-workflow check-references FILE does the same for a finished .docx,
.md, .txt or .bib: one line per reference, --json for the records, exit
2 when any reference is refuted.
Measured on Badalova and Mayr's 104 hand-checked references
(10.5281/zenodo.21457492) beside the five checkers they ran: 30 of 33 problem
references caught, 15 of 71 correct ones left for review, none of the 71
accused. The three it misses are two wrong venues — one an arXiv preprint cited
as a JCDL 2025 paper — and a wrong first name under the right surname; neither
field is compared. Seven of the 15 are standards, software, datasets, model
cards and workshop pages that no bibliographic registry holds. The title-index
lookup moved nine correct references from needs_review to verified, and the
JCDL preprint with them: found without it, that reference was left for review
only because nothing was found. The 104 were also the development set.
scripts/run_reference_benchmark.py --check replays the recorded registry
answers in benchmarks/evidence/reference_match_2026-10-21/; --download and
--record rebuild them. The full-reference OpenAlex searches and the arXiv
entries in that archive were recorded about four hours before the rest, the
same day: by the second recording OpenAlex's daily budget could not cover them
again and arXiv answered 429. OpenAlex without a key allows about 100 searches a day per address, reset
at midnight UTC, and a reference with no identifier takes one or two; a free
key in OPENALEX_API_KEY raises that, and a spent budget is reported as
unreachable, not as a missing reference.
Measured — the tool arm re-records after a month in which it could not be recorded at all
e401849 added the FL checker three hours after e496f93 recorded the tool
arm, and FL hard-blocks three of that arm's own claims: the pain-point section
calls a table row labelled 1–2 「1–3 星區間」 three times. Every re-recording
since has failed at validate, CI's re-record job is red on the last pushed
commit for exactly that reason, and --check kept passing because it re-scores
the checked-in file rather than regenerating it. The Body SHA-256: line does
not catch this either: it proves nobody edited the fixture by hand, not that the
pipeline would still produce it. The three labels now name the row FL points at,
which is the one change to the author's prose — and it is the defect the
2026-08-14 archive recorded and deliberately left alone.
This is therefore the first recording to carry 80994fb, 4d501b8 and
2b5472a, which moved the derivations into published/calculation_record.docx
and replaced the [S1] markers with 「(本研究計算,表 N)」. Four numbers moved
and not one of them is about the writing: external_sources 12 → 0 (all twelve
were URLs and file names quoted inside the record that left, none a source
outside the data set), verifiable_numbers 137 → 167 with
verifiable_number_ratio 0.5269 → 0.5759 (thirty 表 N pointer digits, not one
further checkable figure — and it is what flips that dimension to a win against
the AI-direct arm), and paragraph_length_fitness 0.5333 → 0.4074 (the
twenty-one source-list entries took 18 fit paragraphs out of the denominator;
the authored paragraphs are unchanged at 22 fit, 31 short, 1 long). Two
dimensions are not measuring what their names say. That is in the archive as a
defect in the instrument, not banked as a result.
Re-judged blind, all three arms in one pass: hand 4/4/4, tool 4/4/4,
AI-direct 4/4/4; one judge scored the tool arm 3 on claim_strength and was
outvoted by the median. The stop condition is not met. An axis is won by
leading on more dimensions than you trail, and 0–0 is not a win — nothing was
lost, the rubric's top band simply no longer separates the three documents. The
same panel found the two defects that cost the AI-direct arm two points last
round and docked it nothing for them, so panel variance remains the only thing
this axis reliably measures. benchmarks/evidence/drone_market_2026-09-16/ is
the new archive and both earlier rounds stay as they were recorded. Declared:
the judging prompt told the judges the three documents need not be spread across
the scale, which earlier rounds may not have said, and it points the same way as
the ceiling that was hit.
Fixed — the report-quality arm looked for a blueprint key that no longer exists
f00f100 renamed requires_undermines to counter_evidence, and
scripts/run_report_quality_benchmark.py was the last place reading the old
name. Finding no counter-evidence section, the mechanical author handed all
eight claims to all four sections and wrote 32 claim paragraphs where it writes
20, which raised four of that benchmark's eight dimensions on an arm whose whole
job is to hold still. Repaired before the archive was written, together with the
undermines list it wrote, which nothing has read since that commit.
With the arm back in its intended shape, the citation work moves two numbers:
verifiable_number_ratio 0.9014 → 0.9143 and structured_paragraph_ratio
0.5714 → 0.6061. Both are two blocks leaving the delivered document — a bare
executive_summary section id that used to reach the page, and the 資料來源
heading over the [S1]–[S8] list — plus an appendix that is no longer
numbered as a chapter, so its 5 is no longer counted as a prose figure the
source does not contain. verifiable_numbers did not move at all, and the
harness still wins 6 of 8.
Fixed — the calculation record leaves the report, and the report's own appendix is the one a reader follows
The reviewer of the revised course report found its appendix unusable: the
entries the next section describes, printed as prose, carried program names
(g_cfp, bev_body), per-row switches (用 Ember 0、0、1、0、0) and the same
inputs and long formulas in every entry, and changing [S] to 「附錄 A」 had not
made them readable. The record now renders as a document of its own,
published/calculation_record.docx (計算附件 完整運算紀錄 / Calculation Record), with entries numbered C1, C2…, and the report keeps the appendix
its author writes. A derivation may declare explained_in, the section that
shows its calculation, and a non-text value is refused. A sentence citing a
computed figure points at the table the figure is placed in (本研究計算,表 5),
else at explained_in (附錄 A.3), else at its record entry (計算附件 C3);
a computed table's attribution points at explained_in or the entry, never at
itself. Merged labels read in order without a repeated prefix — 附錄 A.3、A.4、 表 5、表 6, 計算附件 C1–C3. The references come before an authored 附錄 / Appendix section, which is no
longer numbered as a chapter. A record input read from a wide sheet names the
unit in its column header and the publication the sheet declares, where it had
printed a bare number.
Fixed — a parameter table no longer runs to four pages
The same report's parameter table spread over four pages with a 說明 column so
narrow that a short clause took several lines. Part of that was the content — the
revision splits the table and moves conversion-only rows to its appendix — and
part was the widths: every wide column was capped alike, so parameter names
wrapped to make room for sources that wrapped anyway, and a unit header sized
at half its length printed 電池容 / 量 / (kWh). Each column now starts at
the narrowest width that breaks nothing it should not — a number, a unit, a
Chinese term of up to five characters — and the room left goes, a column at a
time, where it saves the most row height. A 註: / Note: paragraph under a
table's attribution stays on the page with the table.
Fixed — a computed figure says it was computed, and the appendix says how
A reviewer of the same Chinese course report found the body mixing author-year
citations with [S1]–[S24], the document closing on CSV file names and row
counts listed as if they were sources, and a percentage computed from Knobloch
et al.'s figures cited to Knobloch et al. as though they had stated it. A figure
the pipeline computed is now labelled in the sentence — (本研究計算,附錄 A3) /
(calculated in this study, Appendix A3) — a row of the author's own sheet is
本研究整理, and a figure computed from a declared publication names both. The
closing list became 附錄 A 計算與資料說明 / Appendix A. Calculations and Data:
each entry gives the result, the formula in × and ÷, and every input with its
unit, its row label and its source column, pointing at the entry of any input
computed earlier or any per-row input read from an earlier table. A per-row
input from the table's own sheet is given row by row — 用排碳係數 1、0、0、1、0
— where the entry used to name a column of 0s and 1s the reader never saw, unless
its values only repeat the row labels. A computed
table's attribution reads 來源:本研究計算(附錄 A14) instead of the worksheet's
name. Adjacent brackets holding a label merge into one.
Fixed — a table is shown at the precision each column needs, and holds together on the page
A break-even table printed 44,208.8 km beside 3.7 years. A formula column may
declare display_decimals (−2 prints 44,200; never finer than decimals) and
only the rendered column is rounded, half away from zero; the evidence keeps the
recorded figure. A formula reading an earlier formula now reads its unrounded
value: 658.3593 carried as 658.4 had moved a break-even distance by six
kilometres. A CSV source can be placed whole with [TABLE:<file stem>], pipe
tables get column widths from their content instead of equal widths — measured
at the template's 12 pt, where a 10.5 pt estimate left a header reading
電力係數 (g / CO2e/kWh / ), and letting a header take a third line before a
column of row labels wraps — rows no longer split inside a cell, a short table
stays on one page with its caption and attribution, and a figure keeps with its
caption.
Fixed — a line chart can mark its break-even points
A crossing at 44,208.8 km had no position on a chart whose labels were indices.
A line chart takes numeric positions in data.x and labelled data.markers,
each checked against the cited evidence like any plotted value, and an axis of
values past 10,000 prints thousands separators. A marker's label is printed on a
white box above the lines, so a later marker's dashed line no longer cuts through
it.
Fixed — minus signs and reference punctuation in a Chinese document
新細明體 and 微軟正黑體 have no U+2212: Word set "−25%" in the Chinese font,
borrowed the glyph from another face, and printed it wider than the "−14%"
beside it on a line opened up to fit it. A negative number's sign in a Chinese
document now renders as -; a minus between two terms of a formula, set in the
Latin face, and code keep theirs. A reference title or publisher that already
ends in ., ?, ! or their full-width forms takes no second mark — the list
printed "gasoline cars?.", "VDI e.V.." and "減碳足跡?。".
Fixed — the visual render check no longer calls a preview a pass
It rendered pages 1–3 of a 25-page report and reported passed because the
PNG files existed. It now renders every page and reports conversion, the pages
previewed out of how many, and whether a visual review was recorded for this
document in visual_review.json; the delivery summary says "visual review not
performed" until one covers every page.
Fixed — a Chinese cover is labelled in Chinese, and the title stays out of the table of contents
The cover of a Chinese report read "Student Id:", "Date:" and "Keywords:", and
the title — a Heading 1 — opened the table of contents. Known template fields,
keywords and correspondence take Chinese labels when the front matter is
Chinese, and the title paragraph takes the Title style. A title wider than the
page breaks after its colon instead of wherever the line ran out — the cover
printed "…二氧化" / "碳排放評估" — and the template field report still finds it.
Fixed — the Sources list, table attributions and undated citations read as written for a reader
Read page by page, the delivered Word file of a Chinese course report closed on
three pages of Sources-list entries quoting raw JSON rows ("“{“參數”: “汽油車保養”,
“數值”: “5”…}”") and formula internals cut off mid-token ("其中 heat = 7,586(lca");
the table computed from Knobloch et al.'s data was attributed to
"knobloch2020_table6_2015.csv (9 rows)"; and one sentence citing two undated U.S.
DOE pages printed four parentheticals side by side, because "n.d.-a" matched
neither the merge nor the duplicate-collapse shape. A row is now listed as its
fields, a computed figure as its result sentence marked 本報告計算 / Computed for
this report, and a long quote is cut at a clause boundary; a table computed over
a declared publication names the publication, and any computed table says it was
computed; suffixed undated citations merge like dated ones.
Fixed — registering evidence before the first stage no longer blocks it
register_derived_evidence rewrites the task briefs, and doing so re-labelled
prepare's own starter figure_plan.json as preserved_existing. The harness
then baselined that file as absent, so the first claim_matrix submission of
the run was a scope violation that no repair could clear — restoring it meant
deleting a file the pipeline wrote. The plan keeps its generated status while
the file on disk is still the one prepare produced.
Fixed — FE reads a continental source's 24,2 t and 90.000 km
The German VDI life-cycle factsheet writes its results as "24,2 t CO2äq" and
"bei 90.000 km". FE read them as 242 and 90.0, so a report restating the study's
90,000 km break-even could cite it only by printing "90.000 km" to readers who
take it for ninety. A text that commits to continental notation — it holds a
number that can only be a decimal comma and none that can only be English — is
now read in it, and "Kilometern" is the unit km.
Fixed — a structured draft can write a paragraph of more than one sentence
structured_drafts.json, the preferred new-draft path, compiled every sentence
into a paragraph of its own, so a report authored through it was a column of
one-sentence paragraphs — the shape the brief's own paragraph rule (context,
content, conclusion) rules out. A section may now group its sentences under
paragraphs: [{"sentences": [...]}]; each sentence keeps its claims, citations
and sentence-map row, Chinese sentences join without a space and English ones
with one, and a heading, table or figure marker is refused inside a
multi-sentence paragraph. The flat sentences list behaves as before, and
lint_artifacts reads both shapes.
Fixed — FE reads a source's notation: 6.058E-01, CO₂, and g CO2e/kWh
Three ways an official or scientific source prints a figure blocked the claim
that restated it correctly. A value in scientific notation — Taiwan's
carbon-footprint database prints 0.6058 as 6.058E-01 — was extracted and then
failed to parse, because the exponent was rewritten on its own as
float("E-01"). A subscript digit made a unit look absent: the claim's
kgCO2e/度 was searched for in kgCO₂e/度 without the NFKC fold the numbers get.
And the unit reader stopped at the space or at the digit of CO2, so "63 g CO2e/km"
was the unit g, "kgCO2e/度" was "kgCO", and a column headed "(g CO2e/kWh)" was
"gco2e/kwh" — three spellings of one unit that never matched. A gas named between
a mass and its denominator is now read past, so all three are g/km, kg/度 and
g/kWh; a different mass or a different denominator still fails.
Fixed — a PDF's sub- and superscripts stay on their line
pdfplumber groups characters into lines by their top edge, and a subscript set
5.8 pt below the letters beside it fell outside its 3 pt tolerance: the ICCT
white paper's "CO₂e/km" came out "CO e/km" with the 2 on a line of its own, 72
times in 38 pages. A small character touching a body-size character it overlaps
vertically is now placed on that character's line before the page text is
assembled; chart labels and other small text elsewhere are left where they are.
Broken occurrences in that report fell from 72 to 2.
Fixed — a chart's plotted values are checked against the evidence it cites
FIGURE_BUILD draws the numbers written into figure_plan.json, and nothing
compared them with the figure's source_evidence_ids: a claim had to find its
number in the ledger while the bar chart beside it could plot any value under a
real evidence id. FIGURE_PLAN_AUDIT now reads every plotted value — bars,
points, slices, error bars, table cells — and hard-blocks one that no cited
entry states (within the 1% FE allows prose), naming the value; an evidence id
the ledger does not hold is named too. Scientific notation (6.058E-01) and
signs are read, and data the pipeline built from its own recommendation is not
re-derived.
Fixed — intake no longer averages a column of mixed units, years or a constant
A parameter sheet listing a heat value in kcal/L beside an emission factor in
kg CO2/TJ and a mileage in km had its value column summarised ("平均數 4,695.72")
and averaged inside a table crossed by source; a column of publication years was
summarised as "最小值 2,025 … 平均數 2,025" and averaged in a cross table as
"年份 平均 2,015.00". A numeric column is summarised and crossed only when its
rows share one unit, it is not a year, and it varies. A year column is counted as
a category instead.
Fixed — formula evidence prints each operand with the places it was used with
The operand line of a formula exists to be redone, and it printed every value
above one to two places: gd = 2.29 beside a table computed from 2.2906, so the
stated arithmetic gave 158.0 where the cell said 158.1. Operands now print every
place they carry, up to six.
Fixed — a line number counts the lines an editor counts
Text and Markdown sources were split with str.splitlines, which also breaks on
form feed — and a PDF exported to text carries one at every page break. In a
government plan, text on line 908, 39 page breaks down, was cited as line 947, and a
"lines": "950-971" selection kept the wrong passage while silently dropping the
2030/2035/2040 targets it was chosen for. Lines now break on \n, \r\n and
\r only, as an editor, grep -n and a text-mode readlines() count them.
Fixed — a declared publication stays declared, and its own bibliography is not the report's
Two gaps in the source-citation work, found by the first full run with it. A
statistic the pipeline computes over a declared dataset — the prepare-time
summaries, a registered max, a ratio between two of them, a formula reading
only that table — dropped the citation and resolved to an [Sn] marker; it is
now cited as the dataset, while a join, or a formula mixing the dataset with the
author's own parameters, stays the author's computation. And a declared
publication's own references — 62 bibliography entries of a white paper, a news
page's navigation links — were read as sources of the report, so the reference
list of a 38-source report ran to 106 entries. A declared file is cited as
itself; what it cites is not extracted.
Fixed — a PDF table with a merged cell no longer stops prepare
pdfplumber reports a merged or blank cell as None and the block model holds
strings, so preparing from the ICCT white paper ended at SOURCE_PARSE with "110
validation errors for SourceContentBlock" — a hundred lines of table_data.0.1
naming neither the file nor the page. Cells are normalized to strings at the
parser, and a block the model still refuses is reported by file, page or line,
and the first offending field.
Added — a source can name the pages it is used for
The same report cites one table of an 86-page white paper and two articles of a
400-page regulation. Attaching them whole puts thousands of rows in front of the
author and the gates; hand-copying the passage into a new file loses the page
numbers a reader checks against and adds a transcription nobody verifies.
A source entry (and a --source-manifest object) now takes "pages": "80-86, 90"
for a PDF or "lines": "5504-5526" for a text file. Only blocks inside the span
are read, they keep their real locators, source_registry.json records how many
were kept, and a span that holds no text — or pages on a file without pages —
stops prepare naming the file and the numbers.
Fixed — text past a block's 2,000th character reaches the ledger
The PDF parser files a page as one block, and the evidence row kept its first
2,000 characters. On the ICCT white paper this report cites, 33 of 38 pages ran
past that and 38% of the text (42,392 characters) never reached the ledger —
including the headline "offset after about 17,000 km", at character 2,057 of
page 3, which FE then refused in every sentence stating it. A pdftotext dump with
no blank lines is one such block for the whole file. A long prose block is now
cut into rows under the limit, between lines and inside an over-long line between
sentences; each row keeps the page number and the lines it covers. Blocks that fit
keep exactly their old ids, and rows and tables are untouched.
Added — a calculation the report makes is computed here too
The same course report compared vehicles by arithmetic over cited parameters:
production plus battery size times an emission factor, plus a per-kilometre
figure times a mileage, and a break-even distance as one difference over
another. None of that is a statistic over rows, so none of it was
registrable; the calculation ran in a side script whose output entered as a
CSV no gate could check, which is the uncheckable number the ledger exists to
keep out.
register_derived_evidence now takes op: "formula". The expression is the
author's; every operand is read — one cell of a source (source, rows,
column, exactly one row), an earlier derivation, or one cell of an earlier
table — and the value is computed and recorded with each operand's value and
origin. With source, row_label and formulas the formulas run over every
row of a scenario table and register one table a [TABLE:] marker can place,
whose cells a later ratio or formula can read — a second formula table reads
the same-labelled row of the first when its table input names no row, so a
two-step scenario does not repeat the first step's expression. A bare number is refused as an
input (constants belong in the expression, where the reader sees them), an
undeclared name and an unused input are both refused as the trace of a typo,
and only + - * / ** and parentheses run.
Added — a source can say what it is a copy of
A course report on vehicle life-cycle emissions was driven through the pipeline
from Markdown transcriptions of an ICCT white paper, a government emission-factor
announcement and a journal article. Every figure traced to its row, and not one
sentence could be cited the way the reader expects: a .md source resolves to
an [Sn] marker and a Sources-list line naming the file, a PDF to a tag cut
from its file name, and the reference list carried neither. That default is the
right one — a file name does not say who published the content — but there was
no way for an author who does know to say so.
source_files entries now take citation (author, year, title,
publisher, container, volume, issue, pages, doi, url,
description, in_text, reference), and the CLI takes the same objects from
--source-manifest. A declared source is cited author-year — (Negri & Bieker, 2025), (Machala et al., 2025), (經濟部能源署,2026) — its rows count as
literature, and the reference list and .bib carry the declared record.
Adjacent parentheticals merge as APA writes them. Nothing is inferred: a missing
year prints n.d./無日期, an unknown field is refused, and an author list that
cannot be split into surnames (Tesla, Inc.) stops the run at prepare with a
request for in_text rather than printing a name nobody wrote.
The MCP start_report tool also gains author_block, affiliation_block,
keywords and template_fields, which the skill reference documented and only
the Python surface accepted.
Fixed — the BibTeX file no longer dates project files to the current year
publication_references.bib gave every file-derived source year = {<this year>} while the reference list for the same source printed (n.d.) — the
invented date the formatter's own rule forbids, in the one artifact a reader
imports wholesale. It now prints n.d..
Changed — the landing page carries the harsher benchmark too
README showed the two-arm report-quality comparison — harness wins 6 of 8,
loses 2, both reported rather than tuned away — and said nothing at all about
the three-arm one. That is the comparison that answers "better than a person,
better than an AI writing the report directly", and its answer is less
flattering: one axis still lost to the hand-written control, and the round was
decided by a two-point move in a document whose bytes did not change.
Showing only the kinder number is the same defect this repository has shipped
several releases to fix in its error messages, with the reader standing where
the author usually stands. Both READMEs now carry, at the same prominence as
the table and above it, what
benchmarks/evidence/drone_market_2026-08-14/summary.md already recorded: the
stop condition is met; it is met because a frozen arm scored 4/3/3 with this
panel and 4/4/4 with the previous one, so panel-to-panel variation is at least
a point per dimension against a winning margin of one point; the tool arm was
written a second time against the same task, the same rubric and the same
itemised deductions, so these are this task's ceiling; and the archive has no
held-out task, because building one means commissioning a second hand-written
control and a second AI-direct arm.
Both pages name the dated archive the numbers come from. The two-arm section is
unchanged — this adds a benchmark rather than replacing one.
README.zh-TW.md gained the same section with the same four qualifications;
the two pages were diffed assertion by assertion rather than read side by side.
Its stale "69 hand-designed cases" is now 73, matching the corpus after
e401849.
Added — a contract test pinning both READMEs to the three-arm archive
The guard that keeps docs/EVIDENCE.md and docs/DESIGN.md honest about the
adversarial corpus now has a sibling for the three-arm result. It recomputes
the per-axis win-loss records, the stop-condition boolean and the argument
medians from results.json and requires both READMEs to carry them.
The archive path is a pinned constant, not the newest matching directory: a
glob would silently change what the test compares the moment a new round is
recorded, and the stale README would then redden on some later, unrelated
commit. Pinned, swapping archives is a deliberate edit that forces both pages
to move in the same commit.
It compares recomputable values, not prose, so rewording a sentence cannot turn
it red. Its failure messages name both sides — what the page carries and what
the archive computes — and it was watched failing in both languages, on a wrong
axis record and on a flipped stop-condition, before landing.
Added — FL and FT2, the seventh and eighth factuality checkers
The first two hard blocks added in four batches. The previous three added
operations and visibility and no gates at all; these two each have a defect a
blind judge found, and each carries a case in the adversarial corpus so the
defect outlives the fixture it was found in.
FL — a band the prose names must be a band the cited table has. The report
e496f93 recorded calls a row labelled 1–2 「1–3 星區間 49 則評論」 three
times. The row is right, the 49 is right, and 1–3 stars is 68 reviews; FE
compares numbers and found nothing, so six checkers passed it and three
independent blind judges did not. It is the band-label repair of 0e0a40f one
layer up — the labels were corrected at the naming end and nothing was reading
prose that still quoted the old spelling.
Not auto-corrected, because 「1–3 星區間 49 則評論」 is ambiguous: the author may
mean the row, or may mean 1–3 stars, in which case the 49 is wrong too. Four
conditions, all required — a cited computed table, a band-shaped string in the
claim, absent from every cited table by label and by the membership line they
print, and sharing an endpoint with a band that exists. The endpoint condition
is what keeps 「6–8 個工作天」 out of a paragraph citing a price table. The cost
is the author who gets both ends wrong; that miss is accepted, because the hard
constraint here is no false blocks.
FT2 — a relation asserted between two columns. FT's fifth condition is a
pair of cells from one column, which is what fixes the column and the
direction; a sentence relating supply density to review depth names no single
column and FT stands down. Same arithmetic one dimension over, under the same
two-part rule, and additionally requiring the claim to name both columns.
Its motivating sentence does not trip it, and that is worth stating plainly:
over the seven price bands it cited, 14 of 20 ordered pairs agree with the
inverse relation it asserted and so do both ends. What was wrong with that
sentence was calling the relation "the strongest structural signal in this
data" — a claim about strength, not direction. Arithmetic cannot see that and
this project's boundary says a checker does not try. FT2 covers the shape FT
structurally cannot, not the sentence that revealed the shape.
Adversarial corpus: 69 cases to 73, recall 88.6% to 89.1%, false positives
0.0% to 0.0%. None of the 69 existing verdicts moved — which is the honest
report of these two checkers' coverage on prose-only evidence: zero. Both new
families (band_label_drift, cross_column_direction) need a computed grid in
the ledger, and the corpus had none until now.
Back-tested before landing: FL finds exactly the three known instances in the
tool arm and no fourth, and zero findings across sixteen recorded run
directories. Two false positives in the first version were repaired rather than
tolerated — 至 and 到 removed from the claim-side range detector, because
「也只有 1 至 2 件商品」 means one or two products, and categorical tables
excluded, because "1K+ bought in past month" contributes a bare endpoint 1
against which every "1-2" in a document shares an end.
Diagnosed — the term-coverage blocks, and why nothing changed
Claim terms not in evidence was reported as 18 instances across the eight
recorded runs. Counted as distinct (run, claim, term) it is five: one
HHI, one listing, and three USD, appearing 19 times across seven repeated
failure events. All five are one shape — a Latin technical token in an
otherwise-Chinese claim, where the cited evidence carries the same idea in
Chinese (美元) or not at all.
Nothing changed, and the reason is that the classification cannot be completed
from what is recorded. Every archived claim_matrix.json holds the repaired
claim, so the blocked text survives only inside the failure message. On five
instances reconstructed that way, loosening FE's comparison — including a
currency-notation alias, which is the narrowest available fix — is not a change
this evidence supports. The threshold was not touched.
Measured — the drone-market benchmark, re-recorded and re-judged
The tool arm was rewritten and all three axes were re-run.
benchmarks/evidence/drone_market_2026-08-14/ is the new archive; the previous
one is left as recorded, because the document it scored no longer exists.
The three deductions three blind judges converged on last round are gone. The
sentence claiming the category table and the price-band table shared no column
was false — both key on asin, and the same script had already joined them
twice. That cross tabulation is now computed and carries the argument: the
$200–500 band is 66.04% camera against 12.12% in $0–30, a gap of 53.92 points,
and the $800+ band is a third thing again, 32.08% agricultural. The claim that
denser supply meant thinner demand "across the whole table" was true of two
bands and false of two others; it now says so, names both counterexamples, and
claims only what the two ends support. And the executive summary's cut point
rests on three quantities from three files instead of two.
Scores, measured on the unchanged instrument before the layout axis was
extended: verifiable_numbers 90 → 137 (the hand-written control has 121),
tables 9 → 10, heading_informativeness 0.6667 → 0.8,
paragraph_length_fitness 0.5156 → 0.5333. Two went down and stay down:
counter_evidence_paragraphs 5 → 4, because the rewritten paragraph names two
counterexamples to its own generalisation using a word the keyword list does
not carry, and structured_paragraph_ratio 0.6875 → 0.6765.
Blind argument scores: tool 3/3/4 → 3/4/4, hand 4/4/4 → 4/4/4, AI-direct
4/4/4 → 4/3/3. The stop condition is met — and it is met because of that
last line, on a document whose bytes did not change. Three fresh judges found
an arithmetic error in the AI-direct arm's headline figure and a contaminated
population under its strongest counter-evidence; the defects are real and were
equally present last round, when the previous panel did not find them. What
that measures is the panel's variance, and this round's winning margin is one
point. Read the archive's opening section before its table.
Added — three delivery-layer dimensions on the layout axis
table_caption_ratio, table_provenance_ratio, table_size_fitness. The
first three layout rules read prose; the deliverable is a DOCX and nothing was
looking at the furniture around a table. Declared in the archive because the
author of the arm being measured chose them after reading all three documents:
two run 1.0 against 0.0 and are closer to "did the renderer run" than to "is
this better laid out", and the third is the one the pipeline loses.
Added — a re-record check, and a body hash on the arm that moves
CI now re-records the tool arm and diffs it against the checked-in fixture,
triggered only when the diff touches a path that reaches the deliverable. Not
"the script's file hash changed": a comment would turn that red, and what it
would measure is byte-identity of a source file rather than whether the fixture
moved.
The tool arm now carries a Body SHA-256: line like the other two. It is not
frozen — record_tool_arm.py rewrites it — so the constant is named
HASHED_ARMS, and what the hash catches there is an edit made by hand instead
of by a re-recording.
Every argument vote now records who cast it: model, the inputs it was shown,
and three flags that must all be false — same context as the author, saw the
pipeline code, saw the task prompt. The rubric tells a reader they can re-run
the votes themselves; without an identity record that sentence could not be
cashed. The check refuses votes that omit it.
Added — FT, the sixth factuality checker
A blind judge, given the three benchmark documents relabelled and the rubric and
nothing else, capped the delivered report's claim_strength on one sentence:
「銷量級距愈高的組別,價格中位數也愈高」, citing a fifteen-tier sales table and
quoting two of its rows. Over all fifteen tiers the relation is not there — the
three highest-volume groups carry three of the lowest prices. Both quoted
numbers were real and present in the cited evidence, so FA, FB, FE, FS and FD
all passed it. They compare text with text; none of them had ever read the grid.
FT reads the grid. When a claim quotes a pair of cells from one column of a
computed group table and its prose asserts a direction, it orders the groups —
band order for buckets, the leading magnitude for category labels like
10K+ bought in past month — and recomputes. A column is blocked when its
ordered group pairs and its two ends both go the other way, and the block prints
the real ordering so the repair is visible rather than guessed at.
Both halves of that test are load-bearing. A discordant majority on its own,
tried against one authored report, took out six honest claims over columns their
prose never generalised about; the tables were merely noisy. Requiring the ends
to disagree as well left exactly one finding: the defect it was built for.
It does not read what a sentence means. Ordering rows and counting concordant
pairs is the same arithmetic as expect, over rows the pipeline already owns —
the "no semantic layer" boundary in AGENTS.md still holds. Where the group axis
has no order the tool can recompute (brands, categories), FT stands down and
files nothing, rather than refusing a sentence it cannot check.
The guarantee moves accordingly: from "no fabricated numbers" to "no fabricated
numbers, and a direction drawn from a computed table is the direction it has".
Changed — the argument axis is no longer self-graded
Every argument vote in the archive is now cast by a separate judge given the
three documents relabelled and shuffled, the rubric and the CSVs, and told
nothing about how any arm was produced. Under self-scoring the arms read hand
3/3/4, tool 4/3/4, AI-direct 4/3/4. Blind, they read hand 4/4/4, tool 3/3/4,
AI-direct 4/4/4 — the tool arm inflated, the hand-written control deflated,
and the verdict on that axis reversed.
So the recorded verdict is now: the tool arm wins layout against both, wins
numeric against the AI-direct arm and draws it against the hand-written one,
and loses the argument axis to both. stop_condition_met: false. That is a
worse number than the release before it and a truer one.
Changed
- The tool arm's
f7is gone. Its replacement — accumulated reviews rise with
the sales tier, price does not — passed FT and was still wrong: three blind
judges independently found that the top three sales tiers are a pair of
binoculars and two memory cards. Every figure real, the relation real, the
population wrong. That is a domain judgement no checker should attempt, so it
is now limitationl5, naming the three rows and retiring the column. r1no longer offers 4.66★ and a 4.90% negative rate as two independent
corroborations; they are the same 102 reviews measured twice, and it says so.- The executive summary states the enter/where verdict instead of leaving it to
be inferred from the recommendations. verify_claimsruns FT too. There is no draft on that surface, so it reads the
claim text; a caller who supplies a computed table is exactly the caller who
can read a direction out of two of its rows.
Fixed — the first command of a session
start_report refused to begin until the caller recorded a decision for every
optional connector, including ones whose enable flag was off and which were
therefore not going to run. Silence about a switched-off connector is now taken
as what it is. A feature that is switched on without a recorded decision is
still refused, and so is one the user declined while its flag is true.