Repository navigation
nullius v0.8.7
Two control runs, a literature outside AI, and the finding that the harness engages on the
prompt.
-
The control arm, run a fifth time, on a literature
three months old. The fourth run was on forty years of refereed work and the arms differed
in what a reader could verify, which is a real difference and a quiet one. This one asks
whether large reasoning models reason or pattern-match: preprint-dominated, contested, with a
popular account well ahead of the evidence. The difference gets loud. The unharnessed session
cites The Illusion of Thinking -- the paper the whole dispute is about -- through
arize.com/blog/, a monitoring vendor's marketing post about it, and five of its ten links
are not papers: that blog, a conference poster page, a company's own publication venue, a
benchmark product page and a Substack. No DOI anywhere in it. The harness arm holds the same
work resolved to NeurIPS, peer-reviewed, read tomethoddepth, quoted verbatim against
cached text. A blog post is not a work, socitecannot turn one into a citation. -
The independence arithmetic said yes for the first time. Across four earlier runs no claim
ever reachedestablished. Here one did, on two independent groups both measuring, and the
contested heart of the dispute is recorded ascontestedover the two papers that disagree. A
gate that only ever refuses is a gate nobody can calibrate. -
Nine of eighteen sources in that run are not peer reviewed and the ledger says so on every
one. It reaches the prose: "Lawsen (2025, preprint)", "Shojaee et al. (2025, NeurIPS)".
The unharnessed answer names no venue for any source at all. -
A slow index crashed
litinstead of being one index that did not answer.TimeoutError
is anOSErrorand is not aURLError, so a socket timeout arrived unwrapped and escaped
as a traceback, discarding the results the other indexes had already returned. This is the
same defect class control-2026-09-12 records against
audit, which was fixed at that one site and never swept for:http_json,http_text,
arxiv_searchandarxiv_lookupall had it. Found by using the tool, not by reading it.
Covered offline by patchingurlopento time out and asserting every helper converts rather
than raising. -
The control arm, run a fourth time, on a question
in the philosophy of language rather than in AI. Four cells this time, because the prompt
was varied as well as the arm, and the prompt is what decided the result. Asked only to
answer the question, a session inside a nullius project opened no unit, logged no search,
resolved nothing, and the stop gate allowed it -- correctly, since with no unit open there
is no unit state to refuse. Asked one sentence more (do the literature search properly),
the same setup ran 20 searches, resolved 21 references and wrote 19 notes. So three earlier
runs have been measuring the advice arm while calling it the tool arm, and whatmode certainbuys overmode adviceis still untested. -
The arm that scored best on the instrument scored best by writing nothing checkable. In
a nullius project with no unit open, the session had read the invariants, knew the rule that
refuses a credited surname with nothing resolved behind it, and satisfied it by crediting
nobody: zero identifiers, zero names, a clean row on every measureauditcomputes, and not
one source a reader can chase. The arm with no harness at all named five people you can
search for.auditcounts unbacked names and cannot count absent ones. -
auditwas undercounting the arm with no harness, in the direction that flatters this
repository. A draft carrying nine working links scored zero identifiers named. The
lookbehind that stops a DOI matching mid-token also stops every DOI inside a publisher URL,
and the one special case written for it covereddoi.org/-- the single URL shape a session
with no ledger does not write.pnas.org/doi/10.1073/...and
onlinelibrary.wiley.com/doi/10.1002/...read as nothing at all. Fixed, with the case in
both directions insmoke.sh, and the identifier counts for the unharnessed arms in runs
one to three should be read as floors. -
The gates were optional, in the one way nobody checks. Not by stepping around a gate --
by never entering the state a gate applies to.hook_stopopened with
if store.unit is None: return 0, so a session that wrote a research answer into a nullius
project and opened no unit passed every check there was, which is what the fourth control run
measured. A draft written here with no unit behind it now refuses the stop. Recorded at
write time on both paths, Write|Edit and the shell heredoc, because the measured session used
the heredoc. Two exits:nullius artifact <path>under an open unit, and every unit gate then
applies, or the newnullius scratch <path> "<why>", which says it was a thinking file and
puts that on the record. Refusing every unit-less session would be the wrong fix -- a
research folder is also where you answer unrelated questions -- so what is refused is the one
thing that is unmistakably the work. Both directions insmoke.sh, including that a unit-less
session which wrote nothing still finishes. -
And the gate was re-run against the cell that found the hole. Same prompt, word for word.
Under 0.8.6 the session opened no unit and the stop allowed it; under the gate it opened a
survey, declared what would close it, tracked the draft, closed on a locator and generated a
report -- and recorded, in the ledger rather than in the chat, that it had "zero query
vocabularies logged against a suggested three ... I wrote from what I already knew rather than
from a logged search". OpenAlex was rate limiting from this session's own earlier use, so
citecould not resolve and the research path itself was not exercised. What was exercised is
the gate, and the session stripped the author names it could not resolve rather than leave
them credited. -
The front page had been promising this for four releases. "states a session cannot finish
in" was the first sentence of the repository and was not true as written. It is now, for the
case it names, and the sentence says what it does not reach: research that produced no draft,
or a draft written outside the project. -
Score nothing until the arm's process has exited. The harness arm was read and audited
while it was still working, at 611 words with fourteen DOIs inline. It then cut itself to
355, moved the identifiers intonullius reportand left a 68-word provenance note in their
place, so every number from that first reading described a file that no longer existed -- and
the first draft of the write-up and the front page both carried them. Corrected against the
final artifacts, and the rule is now written down in the run, because nothing in
control.md said it and no earlier run had scored an arm mid-flight. -
A new scenario, and deliberately not an AI subject.
There are now eight scenarios, and the first seven test the tool against one literature's
habits: a canon fifteen years old, everything on arXiv, founding papers phrased in today's
words.
Whether the language you speak changes what you can see breaks all three. It exercises four
paths nothing else reached -- a field whose indexes exclude arXiv, a trade book the indexes
do not carry, a kept work closed by decision withfrontier, and the weakness
algorithms/graph.mdhad declared and never run: era proximity misjudges a field whose
canon genuinely is old. Berlin and Kay 1969 arrives at the highest multiplicity in the
walk and is flaggedancestor?. The flag firing is correct and the label is wrong, and the
scenario asserts the first while recording the second. -
Two runs of the same walk stopped for different reasons. Same three queries, same limit,
same depth, different two-works-per-vocabulary seed sets: one converged at hop 3 on 60%
saturation with 210 works reached, the other never converged and spent the budget at 475.
Saturation was not monotonic either -- 12, 38, 44, 7, 46, 32 -- so one hop's number is
not a curve.graph.mdfixed circularity between the walk and its seeds; nothing yet checks
whether the seeds are independent of each other, and that is now measured rather than
suspected. -
The front page shows two real sessions instead of a funnel. 0.8.6 settled on
coverage
output as the After column, which was honest and mismatched: the Before was a conclusion and
the After was a receipt, so the table replaced a claim with a log rather than with a better
claim. Both columns are now verbatim output from the fourth control run, both carry a
conclusion, and the left one is not a straw man -- it found a 2026 replication the harnessed
session missed. The example says so. -
The harness arm missed three recent works, and not for the reason it looked like. The
obvious reading was that a web search reaches this year and a bibliographic index lags.
Checked before releasing, and wrong: the 2025 scoping review it missed is in OpenAlex at
10.1017/langcog.2025.13, the 2026 replication at10.1073/pnas.2538139123, and the arm's
own 639 records include eighteen works from 2024 on. None of the three was ever retrieved
by any of its twenty vocabularies, every one of which names a paradigm, a population or a
mechanism. Recency is not one of the vocabularieslibrarian.mdasks for, nothing counts
it, and nothing refuses without it. Recorded as an open gap rather than patched, because the
fix is a gate and a gate wants a run behind it.