Skip to content

nullius v0.8.7

Choose a tag to compare

@github-actions github-actions released this 13 Sep 11:25
· 4 commits to main since this release

Two control runs, a literature outside AI, and the finding that the harness engages on the
prompt.

  • The control arm, run a fifth time, on a literature
    three months old.
    The fourth run was on forty years of refereed work and the arms differed
    in what a reader could verify, which is a real difference and a quiet one. This one asks
    whether large reasoning models reason or pattern-match: preprint-dominated, contested, with a
    popular account well ahead of the evidence. The difference gets loud. The unharnessed session
    cites The Illusion of Thinking -- the paper the whole dispute is about -- through
    arize.com/blog/, a monitoring vendor's marketing post about it, and five of its ten links
    are not papers
    : that blog, a conference poster page, a company's own publication venue, a
    benchmark product page and a Substack. No DOI anywhere in it. The harness arm holds the same
    work resolved to NeurIPS, peer-reviewed, read to method depth, quoted verbatim against
    cached text. A blog post is not a work, so cite cannot turn one into a citation.

  • The independence arithmetic said yes for the first time. Across four earlier runs no claim
    ever reached established. Here one did, on two independent groups both measuring, and the
    contested heart of the dispute is recorded as contested over the two papers that disagree. A
    gate that only ever refuses is a gate nobody can calibrate.

  • Nine of eighteen sources in that run are not peer reviewed and the ledger says so on every
    one.
    It reaches the prose: "Lawsen (2025, preprint)", "Shojaee et al. (2025, NeurIPS)".
    The unharnessed answer names no venue for any source at all.

  • A slow index crashed lit instead of being one index that did not answer. TimeoutError
    is an OSError and is not a URLError, so a socket timeout arrived unwrapped and escaped
    as a traceback, discarding the results the other indexes had already returned. This is the
    same defect class control-2026-09-12 records against
    audit, which was fixed at that one site and never swept for: http_json, http_text,
    arxiv_search and arxiv_lookup all had it. Found by using the tool, not by reading it.
    Covered offline by patching urlopen to time out and asserting every helper converts rather
    than raising.

  • The control arm, run a fourth time, on a question
    in the philosophy of language rather than in AI.
    Four cells this time, because the prompt
    was varied as well as the arm, and the prompt is what decided the result. Asked only to
    answer the question, a session inside a nullius project opened no unit, logged no search,
    resolved nothing, and the stop gate allowed it
    -- correctly, since with no unit open there
    is no unit state to refuse. Asked one sentence more (do the literature search properly),
    the same setup ran 20 searches, resolved 21 references and wrote 19 notes. So three earlier
    runs have been measuring the advice arm while calling it the tool arm, and what mode certain buys over mode advice is still untested.

  • The arm that scored best on the instrument scored best by writing nothing checkable. In
    a nullius project with no unit open, the session had read the invariants, knew the rule that
    refuses a credited surname with nothing resolved behind it, and satisfied it by crediting
    nobody: zero identifiers, zero names, a clean row on every measure audit computes, and not
    one source a reader can chase. The arm with no harness at all named five people you can
    search for. audit counts unbacked names and cannot count absent ones.

  • audit was undercounting the arm with no harness, in the direction that flatters this
    repository.
    A draft carrying nine working links scored zero identifiers named. The
    lookbehind that stops a DOI matching mid-token also stops every DOI inside a publisher URL,
    and the one special case written for it covered doi.org/ -- the single URL shape a session
    with no ledger does not write. pnas.org/doi/10.1073/... and
    onlinelibrary.wiley.com/doi/10.1002/... read as nothing at all. Fixed, with the case in
    both directions in smoke.sh, and the identifier counts for the unharnessed arms in runs
    one to three should be read as floors.

  • The gates were optional, in the one way nobody checks. Not by stepping around a gate --
    by never entering the state a gate applies to. hook_stop opened with
    if store.unit is None: return 0, so a session that wrote a research answer into a nullius
    project and opened no unit passed every check there was, which is what the fourth control run
    measured. A draft written here with no unit behind it now refuses the stop. Recorded at
    write time on both paths, Write|Edit and the shell heredoc, because the measured session used
    the heredoc. Two exits: nullius artifact <path> under an open unit, and every unit gate then
    applies, or the new nullius scratch <path> "<why>", which says it was a thinking file and
    puts that on the record. Refusing every unit-less session would be the wrong fix -- a
    research folder is also where you answer unrelated questions -- so what is refused is the one
    thing that is unmistakably the work. Both directions in smoke.sh, including that a unit-less
    session which wrote nothing still finishes.

  • And the gate was re-run against the cell that found the hole. Same prompt, word for word.
    Under 0.8.6 the session opened no unit and the stop allowed it; under the gate it opened a
    survey, declared what would close it, tracked the draft, closed on a locator and generated a
    report -- and recorded, in the ledger rather than in the chat, that it had "zero query
    vocabularies logged against a suggested three ... I wrote from what I already knew rather than
    from a logged search"
    . OpenAlex was rate limiting from this session's own earlier use, so
    cite could not resolve and the research path itself was not exercised. What was exercised is
    the gate, and the session stripped the author names it could not resolve rather than leave
    them credited.

  • The front page had been promising this for four releases. "states a session cannot finish
    in"
    was the first sentence of the repository and was not true as written. It is now, for the
    case it names, and the sentence says what it does not reach: research that produced no draft,
    or a draft written outside the project.

  • Score nothing until the arm's process has exited. The harness arm was read and audited
    while it was still working, at 611 words with fourteen DOIs inline. It then cut itself to
    355, moved the identifiers into nullius report and left a 68-word provenance note in their
    place, so every number from that first reading described a file that no longer existed -- and
    the first draft of the write-up and the front page both carried them. Corrected against the
    final artifacts, and the rule is now written down in the run, because nothing in
    control.md said it and no earlier run had scored an arm mid-flight.

  • A new scenario, and deliberately not an AI subject.
    There are now eight scenarios, and the first seven test the tool against one literature's
    habits: a canon fifteen years old, everything on arXiv, founding papers phrased in today's
    words.
    Whether the language you speak changes what you can see breaks all three. It exercises four
    paths nothing else reached -- a field whose indexes exclude arXiv, a trade book the indexes
    do not carry, a kept work closed by decision with frontier, and the weakness
    algorithms/graph.md had declared and never run: era proximity misjudges a field whose
    canon genuinely is old.
    Berlin and Kay 1969 arrives at the highest multiplicity in the
    walk and is flagged ancestor?. The flag firing is correct and the label is wrong, and the
    scenario asserts the first while recording the second.

  • Two runs of the same walk stopped for different reasons. Same three queries, same limit,
    same depth, different two-works-per-vocabulary seed sets: one converged at hop 3 on 60%
    saturation with 210 works reached, the other never converged and spent the budget at 475.
    Saturation was not monotonic either -- 12, 38, 44, 7, 46, 32 -- so one hop's number is
    not a curve. graph.md fixed circularity between the walk and its seeds; nothing yet checks
    whether the seeds are independent of each other, and that is now measured rather than
    suspected.

  • The front page shows two real sessions instead of a funnel. 0.8.6 settled on coverage
    output as the After column, which was honest and mismatched: the Before was a conclusion and
    the After was a receipt, so the table replaced a claim with a log rather than with a better
    claim. Both columns are now verbatim output from the fourth control run, both carry a
    conclusion, and the left one is not a straw man -- it found a 2026 replication the harnessed
    session missed. The example says so.

  • The harness arm missed three recent works, and not for the reason it looked like. The
    obvious reading was that a web search reaches this year and a bibliographic index lags.
    Checked before releasing, and wrong: the 2025 scoping review it missed is in OpenAlex at
    10.1017/langcog.2025.13, the 2026 replication at 10.1073/pnas.2538139123, and the arm's
    own 639 records include eighteen works from 2024 on. None of the three was ever retrieved
    by any of its twenty vocabularies, every one of which names a paradigm, a population or a
    mechanism. Recency is not one of the vocabularies librarian.md asks for, nothing counts
    it, and nothing refuses without it. Recorded as an open gap rather than patched, because the
    fix is a gate and a gate wants a run behind it.