Repository navigation
0.125.0: Run bundle v3 with live items in append-only files, loadRunBundle, faster repeated verify
humanish 0.125.0 keeps each live flush of a long study small, exports the run bundle loader, and
verifies a finished study faster. Run bundles are now humanish.run-bundle.v3: a running
participant's items so far live in an append-only file under live/, and each flush appends only
the items recorded since the last one. On a synthetic run of 24 participants working 8 hours, a
flush wrote about 0.23 MB in about 0.24 s, where 0.124.0 wrote up to 120 MB in 3.0 s. A finished
v3 bundle differs from v2 only in its schema name; 0.124.0 and earlier refuse it by name.
loadRunBundle(cwd, run) returns run.json with every trace and live file joined and checked, for
v1, v2 and v3 bundles. The five verifies of a finished study read each screenshot once: 10 s in
all for 1,203 screenshots, down from 35 s. A key in a study's title is redacted from the bundle,
and one in a persona id or desktop template fails at parse instead of blocking the run.
This is a minor release because the run bundle format changes:
| # | 0.124.0 | 0.125.0 | What to do |
|---|---|---|---|
| 1 | A run in progress wrote humanish.run-bundle.v2 run.json, whose running participants carry their items inline in streams[].liveActor.items (humanish.live-actor.v1). |
run.json is humanish.run-bundle.v3. A running participant's liveActor is humanish.live-actor.v2 and carries itemsFile: { path, generation, items, bytes, sha256 }, which indexes its append-only file live/<position>.<generation>.ndjson. A finished v3 bundle has no liveActor and no live files, and differs from v2 only in its schema name. (#1766) |
Accept humanish.run-bundle.v3 where you check the schema, and read participants' items through loadRunBundle(cwd, run) instead of liveActor.items in run.json. |
| 2 | npm 0.124.0 and earlier read every bundle humanish wrote. | They refuse a v3 run.json by name (run schema fails in verify, analysis refuses the source). 0.125.0 reads v1 and v2 bundles as they are. (#1766) |
Read v3 bundles with 0.125.0 or later. |
| 3 | humanish export --format bundle wrote its copy as v2. |
It writes v3 whatever the source version. (#1766) | Same as row 2 for a reader of exported copies. |
| 4 | A study whose actor.persona, participants[].persona or execution.desktop.template held a key, a token or a local path ran, and verify graded the run blocked. |
study check, run and migrate refuse it at parse, naming the field. (#1790) |
Use a plain persona file name, and set the template to its E2B name or id. |
Changed (breaking)
- Run bundles are now
humanish.run-bundle.v3, and a run in progress keeps each running
participant's items in an append-only live file. Each live flush appends only the items
recorded since the last one, tolive/<position>.<generation>.ndjson, and run.json's
liveActor(humanish.live-actor.v2) indexes the file withitemsFile(path, generation, item
count, byte count, sha256) in place ofitems. A file only grows. The writer starts a new
generation holding the participant's full list when an item already written was replaced or
removed, an append failed, or a sandbox id the run learned since occurs in the file, and removes
the earlier generation once run.json names the new one. On a synthetic run of 24 participants
working 8 hours, a flush wrote about 0.23 MB in about 0.24 s at 10%, 50% and 100% of the run,
where 0.124.0 wrote 12, 60 and 120 MB in 0.37, 1.7 and 3.0 s, and the process peaked at 0.5 GB,
down from 1.2 GB.humanish runremoves the live files after the final run.json, so a finished
bundle holds none; an interrupted run keeps them, indexed by its last run.json. Every reader
loads live items through the trace loader, which reads exactly the indexed bytes and refuses a
file whose size, sha256 or line count differs from its index. The live flush freezes each item
it records, so code that edits an item after recording it fails where it does. An in-progress
v3 run gradeslocal_onlywithRUN_NOT_FINISHED, as an in-progress v2 run did (#1766).
Added
- The package exports
loadRunBundle(cwd, run): run.json of a run id (orlatest) with every
participant's trace file and, while the run is in progress, live file joined and checked, for
v1, v2 and v3 bundles. A file that does not match its index throws an Error named
StreamTraceRefusedErrornaming the participant (#1785). - A run records one
live-items.rewrittenwarn event when a participant's live file starts a new
generation on two flushes in a row, since each such flush rewrites that participant's whole list. - Contributors can run
pnpm gatebefore pushing. It runspnpm release:checkwithCI=true, then
the site and Observer steps of CI's required jobs that the changed paths reach, and records a
pass keyed by the tree of HEAD. The opt-in pre-push hook (git config core.hooksPath .githooks)
refuses a push whose commit has no pass.pnpm packed:checkruns the post-publish check's rows
against a packed tarball or a published version.
Changed
- A finished live study verifies its run up to five times: for the Observer, before and during
analysis, and twice more to refresh the Observer. Each verify read and checked every screenshot
again. verify now keeps what one read of a screenshot showed for the rest of the command, and
reads the file again when its size, times or inode change, or when it changed less than 2 s
before it was read. It reads up to eight evidence files at once. On a synthetic study of 3
participants with 1,203 screenshots of 147 KB, the five verifies took 10 s in all, down from
35 s: about 5 s for the first and 0.9 to 2.1 s for each later one, which took about 7 s each
before.humanish verify, export and serve read every screenshot as before (#1780).
Fixes
- A key, a Bearer value or a sandbox URL in a study's
titleno longer makes its run fail
verification. The run review,run.jsonandobserver/observer-data.jsonrecord the title with
those redacted.study list,study showand the TUI still print it as written (#1790). - A study whose
actor.persona,participants[].personaorexecution.desktop.templateholds
text that verify flags as a secret or a local path, as written or encoded, now fails at parse
with a message that names the field. Before, the run recorded the value as written and verify
graded itblocked. A persona id names a persona file and a template names the image a run
launches on, so redacting either one would change what it names.humanish migraterefuses a
v2 file with such a persona id the same way (#1790).
Checked before release
- CI on this PR (sharded gate on both Node versions, site, Observer proofs, TUI, guest desktop, secret scan) is the gate of record.
pnpm gateon the release commit passed (release:check 254.5 s). - The post-publish check rehearsed on the release tarball (
pnpm packed:check): all 125 rows passed. The registry and unpkg rows run after publish. pnpm release:dogfood: a participant installed the tarball, raninitand the offline first-run study (share_ready, 16 of 16 checks, $0), and stopped at the credentials the gate withholds.- Live on the release tarball (hosted E2B, keys from the user store), about $2.30 estimated for the two runs that finished (the interrupted run recorded no cost summary):
- try-live with analysis: finished, analysis complete ($0.71 estimated), run.json 25 KB as
humanish.run-bundle.v3, nolive/folder after the final write, verify 16 of 16 (local_only, raw screenshots). - Two participants for 3 minutes on a public page, sampled every 10 s: each participant's live file grew by appends only (generation 1 throughout, 102 and 105 items, 18 KB and 20 KB at the last sample), run.json stayed between 9.8 KB and 18.8 KB, the final write removed
live/, and verify passed 16 of 16. - The same study interrupted with SIGINT at 75 s: both sandboxes reclaimed,
live/1.1.ndjsonandlive/2.1.ndjsonkept and indexed by the last run.json, verifylocal_onlywithRUN_NOT_FINISHED, andloadRunBundlefrom the installed package returned 35 and 18 items, matching each index. npm 0.124.0 refused the same run.json by name ("run.json declares humanish.run-bundle.v3; verify reads humanish.run-bundle.v2 and humanish.run-bundle.v1").
- try-live with analysis: finished, analysis complete ($0.71 estimated), run.json 25 KB as
- The live flush's writer, the trace loader and the contained append each passed Codex design and code review rounds; every finding is fixed with a test (#1798). The verify memo passed a Codex design round and a code round (#1797). The study field refusals passed a Codex review; its one finding is #1794 (#1795).
- The benchmark was not run: no change alters what a participant sees or what the analysis reads. The live flush now freezes each item it records, which the live runs above and every brain's fake-provider loop suite exercise.
Not verified
- A live run longer than a few minutes on v3, and the live Observer during one: the Observer feed
still sends every participant's items on each poll (#1766 part B). - A title holding a base64- or percent-encoded key still reaches the bundle, and verify still
grades that runblocked(#1794). - verify's memo on network or other coarse-timestamp filesystems.
- macOS and Windows.
Package
humanish@0.125.0: 992 files, shasum da397796239c463dfb913e2530b2669242385e62.