Skip to content

v0.31.0-beta — Three fixes, and five defects it declares rather than hides

Pre-release
Pre-release

Choose a tag to compare

@joslat joslat released this 30 Aug 08:05
· 31 commits to main since this release
60d9a4f

Nine verticals, 470 questions. Three fixes, five instruments — and a changelog that names five defects this release does not fix.

Temporal recency could not rank anything

It scored 15/15 at V1, V8 and V9 — the only shape in the family on which no two systems could be told apart. It asked about the last three events in the chain, so gold was two adjacent links and the answer was a single transitive step over two sessions that named the asked events outright.

Guessability was ruled out before the construct was touched: the answer was first-named 6/15, middle 3/15, last 6/15, all at chance. The shape was sound and simply too easy on both halves.

It now asks about events spanning the chain.

before after
recency V9 15/15 6/15
vertical V9 42/50 30/50
headroom 0.16 0.40

Gold grows from 2 links to count − 1 and stays minimal: on A<B<C<D<E asked over {A, C, E}, dropping any intermediate link removes a transitive step the answer needs.

The whole vertical was redrawn. A seeded generator redraws everything downstream of any change to the draw sequence, so all 50 questions differ — not the 15 that were targeted. Question ids are stable across that redraw, which makes an id-keyed join succeed while comparing different questions. Temporal sha a6c10b3d…31d26e60…; controls reset.

Prospective: firing semantics are now required, not optional

The consuming project ran the corpus with ProspectiveFiring and ValidTime=Current both dark and scored 49/50. Every shape named the thing, which hands a similarity retriever the words of the session it needs while the harness supplies "today" and the corpus supplies the due date — in-context arithmetic no memory feature is needed for.

The new due-window shape names nothing: several reminders whose only distinguishing property is when each falls due, and an answer that is a set whose membership changes with the as-of instant.

before after
V8 interference cost 0.00 0.28 — the family's first real one
V9 34/50 23/50
headroom 0.32 0.54

Prospective sha 6ddd3e9b…a570b890…; also a full redraw, same id caveat.

Semantic gained a judge body — and the shared preamble did not change

Semantic shipped without a body deliberately, but that left it the only vertical with nothing to settle the preamble's abstained/missed line, which states the distinction as uncertainty versus denial and then illustrates abstained with "I have no record of that" — a denial.

Editing the preamble was tried, measured and reverted: it converted the family's canonical genuine-refusal cases to Missed across four verticals and dropped agreement 0.983 → 0.966. The rule that survives is a vertical body supersedes the shared preamble.

All 26 Semantic cases were enumerated before a line of the body was written, and 19 carry a declared route asserted independently of the outcome — so a template reaching the right label by the wrong reasoning fails rather than passing quietly. Semantic is now 26/26 in all three runs; family agreement 0.987 (0.991 / 0.987 / 0.996).

What this release admits about itself

Five defects ship declared and unfixed, in the changelog, with their directions stated. A release that carries wrong published numbers silently is the failure mode these corpora exist to catch.

V2 cannot judge closed-choice questions. Its reject line is an observed rate of 0.20; the chance floor is 0.50 at k=2. 71 of 470 questions enumerate their own alternatives, and 69 carry a V2 pass that records the reference model abstaining, not the corpus being non-inferable. A pure guesser is rejected with probability 0.989. Flattering direction.

The same floor sits under published arm counts. Chance-corrected: prospective V9 14/21 (0.67) → 0.33, temporal 24/35 (0.69) → 0.45, episodic 14/15 → 0.87. Headroom is a difference and largely cancels; the absolute counts do not.

Forgetting's coverage is diluted. Published mean_realised 0.670; over questions that have gold, 0.529, against a band floor of 0.50. Its 15 never-known probes are G=0 by design and realised_coverage returns a constant 1.0 for them, so 30% of the published mean measures nothing — and the echo calibration optimised that. Queued as a declared corpus revision.

Two shapes are out of band and were declared nowhere: episodic/list-order 0.275 (a reference retriever surfaces a quarter of its gold, so V9 there measures retrieval failure rather than reasoning) and forgetting/still-valid 0.467.

85.7% of the family by character is equalisation padding, uniform across all nine (82.4%–88.5%). Measured to be load-bearing, not waste — removing it puts 12 features over the 0.75 separability bar where 0 are now — so no redesign is proposed. Arithmetic additionally carries ledger voice (a bare common-noun subject carrying a value directly) at 77.5% of value-bearing sentences and 0.0% everywhere else.

Five instruments, none of which costs a model call

tool what it answers
validate_v2_chance_floor.py which passes a coin-flipper would also have earned
measure_signal_density.py what fraction of the text is content at all
measure_padding_value.py whether the scaffolding earns its keep
validate_coverage_population.py whether a published mean counts items it cannot measure
make_padding_free_variant.py controlled diagnostic cells for a consumer's extractor

Corpus verification

vertical sha256
arithmetic 2feda94be7e8c026 unmoved
bitemporal f5b384d7f0ff9c0f unmoved
conjunction c62ef477359749c1 unmoved
episodic 2c6000a6912e5ef0 unmoved
forgetting ba759097b9bd1fd3 unmoved
prospective a570b890a5b92c3b moved
semantic de0b1c22521198de unmoved
temporal 31d26e60fd21d00e moved
workingmemory 7e04e4cb1717cbbf unmoved

All nine: probes.status = run, probed_corpus_sha256 == shipped sha, reference deployment gpt-5.5. Judge prompt fingerprint b1d3f7216032ba27…, calibration 230 cases over 3 runs.

Independently verified read-only by the consuming project's pre-tag probe before this tag was cut.