The benchmark grows: corrections, merges, a third lead, and a known gap

[WaylandYang]

Grows the temporal benchmark from #340 (25 questions, two waves) to 36 questions over three waves, and adds the two things a person does to a ledger that documents cannot: correct a start date, and merge two spellings of one name.

What the sheet now covers

scenario	what it measures
a three-step succession A→B→C	uniqueness closure firing twice on one chain
a corrected salary	closure on an attribute, each axis answering for itself
an employment that predates the document stating it	the world axis reads the stated start, not the arrival time
a deleted document	retraction: visible when the record axis is rewound, gone today (#268's tombstone)
a person corrects a start date (#311)	the world axis flips; the record axis rewound before the correction still shows the old interval
a person merges "L. Si" into "Li Si" (#337)	the world before the merge: the swallowed entity comes back with its own facts
a hire known only to the year	year precision: inside the year holds, outside does not
"no longer holds the title", date unknown	valid_to_precision = unknown — a known gap, see #345
three questions whose answer is nothing	before a project began, before a hire, after a retraction

Each wave now stamps its own moment (wave-1, wave-2, wave-3); within a wave the order is fixed — documents, deletions, corrections, merges, stamp — so a record-axis question can sit between any two human actions. --stamps re-scores an existing base without re-ingesting, which is how the numbers below were re-taken after the scorer fix.

known_gap: true questions are tallied on their own line and kept out of the headline. They pin behaviour we know is wrong so that the day it is fixed is visible; a headline that permanently carries a known number stops being read.

Numbers — 2026-09-05, dev @ d55a97e, DeepSeek-V3 + bge-m3

configuration	headline	world	record	known gaps
uniqueness axioms declared	36 / 36	22 / 22	14 / 14	0 / 1
ontology left to grow itself (--no-declare)	29 / 36	16 / 22	13 / 14	0 / 1

The gap is the same one #341 describes, now confirmed on the wider sheet: every miss in the control is a question that needs automatic closure — the four moments of the Aurora succession and the two of the salary correction. Nothing else moves.

Two results worth naming from the declared run: merge rewind and correction rewind are both clean — #337 and #311 hold on the real ingest path, not only in their database-backed tests. And the one question that fails is the one marked as a gap: a fact that ended on an unknown date is read as still holding at every at, because every world-axis filter tests valid_to IS NULL. Filed as #345 with the reads that share the pattern.

A scorer bug, recorded rather than smoothed over

The first version of the scorer took the names on every edge in the one-hop neighbourhood. A neighbourhood contains edges between neighbours — Li Si works for Meridian — which hold at the asked date and say nothing about the subject, so "Li Si led Aurora in 2023" was reported as a failure of the product. The server's at filter was right the whole time. The scorer now reads only edges touching the subject. A benchmark that reports correct behaviour as failure is worse than one that misses, which is why it is written down in the README.

Verified

Three full runs against a scratch deployment (its own database, its own data directory, a real model): the declared run at 36/36, the control at 29/36, and a re-score of the declared base through --stamps after the scorer fix. The --chat probe on the same base is below.

The chat probe — 24 / 37, no errors

Same base, same sheet, asked through Chat. The ledger run measures the ledger; this one measures whether the agent can ask. The thirteen misses are not scattered — they fall into three piles, and the first two are each one place in the code:

pile	misses	what it is
record axis	7	the tool surface has no as_of. entity_facts takes at only and search_chunks takes no time at all; #317 and #338 threaded as_of through the store and the HTTP API and left the agent at None. The model reaches for changes, the one tool that touches the record axis, and runs out of rounds narrating a plan. Filed as #347
attribute facts	6	fact_line never renders a literal value. A salary is object_value with no other_name, so the model is shown salary → ? (2023-06-01 → 2024-02-20) [90%] — interval and confidence arrive, the value does not — and reports the salary as unknown. Filed as #348
negation	1	the scorer matches substrings; "there is no evidence that Wang Wu advised…" contains the forbidden name. A known weakness of scoring free text, left as is
the agent skipped at	1	Zhou Qi's hire is year-2024; the model answered that he was employed in June 2023. This one is what the chat probe is for

Fix the first two and the chat number should sit around 35 / 37, which is the number that describes the product.

One number was retracted. The first run of this probe reported 8 / 37. The parser read .delta where the frame carries .text, so every reply was an empty string — and an empty string contains none of the forbidden names, which is how eight expect: null questions were scored as passes. The probe now records an empty reply as error and keeps errors out of the tally, and an HTTP failure is recorded as such rather than as a wrong answer. The lesson is written into the README: a chat probe has to prove it received a reply before it is allowed to produce a score.

🤖 Generated with Claude Code