Skip to content

3.0.1 — the paper reproduces exactly, and the instruments that measured it were wrong thirteen times

Latest

Choose a tag to compare

@vitkuz573 vitkuz573 released this 30 Sep 06:11
· 7 commits to main since this release

3.0.1

Seven packages on npm: @skillstate/core, opencode, mcp, cli, bench,
claude, codex. Node >= 20, ESM, MIT.

What this release is

A reimplementation of SKILL.state: Scalable Long-Horizon Agent Skills
— a fixed-schema execution state Σt read once per step and patched between
steps, so the prompt footprint is O(1) per step instead of replaying history.

The paper's claim reproduces exactly on every run measured, and the reason it
reproduces is visible: mean |A| moves only from 2,490 to 2,827 characters across
a 3.6x difference in task length, so the prefix sum collapses onto the closed form.

run files T Σ|Aₜ| eq. 8 (T+1)/2
30-files/p-1 30 45 113,270 23.00x 23.00
30-files/p-3 30 65 161,861 33.00x 33.00
60-files/p-1 60 164 463,655 82.50x 82.50

The part worth reading

FINDINGS.md is a document about a reimplementation whose
measurement instruments were wrong thirteen times — eight wrong in eight
different directions, four absent entirely, and one that was a citation nobody
followed. Two of them were tables in this project's own documentation that
re-derived from nothing.

The headline one: a 90-file run was read three times as three different defects
— a model losing track of its sum, a step ceiling, a network failure — and it was
a timeout 2400 in the project's own harness. The transcript spans 39.9 minutes
twice
, against a forty-minute cap, while the control arm at the same length
finished in 6.9. Exit code 0, empty stderr, plausible state file.

So the harness now records a run's own wall clock against the cap it was given
and reports at_timeout, and reads the transcript's last event for
ended_on_error. Verified on a live SIGTERM: 415.5 seconds of a 420-second cap,
flagged with nobody reading anything.

Every measurement is committed under measurements/, and
a test fails if this repository's prose disagrees with a record.

Where it loses, stated plainly

At 60 files, both arms to completion: the bounded arm is correct — 60/60 files,
final state equal to the truth, and it caught and repaired its own drift — and it
is 10.1x slower than the transcript-based control, and makes twelve more
requests. §3.3 counts characters; a session that patches on more turns lasts
longer. Both numbers are real and they point opposite ways.

The content ratio is 7.9x against a control that did its job properly, not the
46.9x this project quoted for a week — that one was measured against a control run
that kept 10 of 90 filenames in its state.

Verified

  • 1888 tests, 100% coverage on all four metrics, enforced as a build threshold
  • Paper citations resolve against a maintained section table, or carry the
    document they mean
  • The host exposes no agent loop (ctx.session.generate -> { text: string }), so
    the adapter is required rather than incidental

Full changelog: CHANGELOG.md.