Skip to content

0.124.0: Small Observer files for long studies, faster verify, closing reports under zero data retention

Choose a tag to compare

@danielgwilson danielgwilson released this 12 Oct 09:51
· 154 commits to main since this release
35986b5

humanish 0.124.0 keeps a long study's Observer files small, checks screenshots about eight times
faster, and asks an openai-computer-use participant for its closing report at a scheduled end
under zero data retention. observer/observer-data.json is now humanish.observer-data.v2: the
run and each participant without their trace items and screenshot entries, which the Observer
page carries compressed. On a synthetic run of 24 participants working 8 hours, observer-data.json
fell from 155 MiB to 0.33 MiB and index.html from 94 MiB to 16 MiB, so verify reads both. A
reader of observer-data.json that needs a participant's items reads the run's traces. verify of a
synthetic 8-hour run with 8,895 real-size screenshots took 51 s, down from 282 s.

This is a minor release because the Observer's data file changes:

# 0.123.0 0.124.0 What to do
1 observer/observer-data.json (humanish.observer-data.v1) held every participant's trace items and every screenshot entry. It is humanish.observer-data.v2 and holds neither; the Observer page carries them compressed, and a served run answers observer/observer-feed.json with { data, items }. (#1779) A script that reads a participant's items from observer-data.json reads them from the run's trace (streams[].trace.path in run.json) or from observer-feed.json.
2 The Observer server served a saved observer-data.json while a run had no run.json. A saved v2 file that leaves participants' items out answers unavailable; a v1 file is served as before. (#1779) Nothing.

Changed (breaking)

  • observer/observer-data.json is now humanish.observer-data.v2: the run and each participant
    without their trace items and the screenshot entries of their artifact list, which stay in the
    run's traces. Each stream whose screenshot entries moved records how many in
    screenshotArtifacts. The Observer page carries the items gzip-compressed beside the data
    (<script id="observer-items" type="application/octet-stream" data-encoding="gzip">,
    humanish.observer-items.v1) and joins the two before it renders, so it still opens from disk
    and every view reads what it did. A served page polls observer/observer-feed.json, the data
    and items as { data, items }. On a synthetic run of 24 participants working 8 hours,
    observer-data.json fell from 155 MiB to 0.33 MiB and index.html from 94 MiB to 16 MiB; verify
    reads at most 32 MiB of a file, so the earlier files left such a run unable to grade
    share_ready (#1766).
    Migration: a reader of observer-data.json that needs a participant's items reads them from the
    run's trace (streams[].trace.path in run.json) or from a served run's
    observer/observer-feed.json. A file written by an earlier version still loads: the Observer
    server's saved-data fallback reads it, and humanish export exports a page saved by an earlier
    version with its items as before. The server answers a saved v2 file that leaves items out as
    unavailable, since it has no items to show. humanish export of a v2 page exports the current
    projection of the run and its traces, never the page's compressed copy, which verify cannot
    read as text.

Changed

  • verify checks each screenshot about eight times faster: 3.8 ms for a 1440x950 capture, down
    from 32.6 ms. It read every screenshot through a full decode, so a run of 24 participants
    working 8 hours, about 69,000 screenshots, took about 37 minutes to verify. A screenshot is now
    checked from its structure: every chunk's CRC, the IHDR fields, one run of image data, and a
    native inflate to exactly the bytes its rows need, each row starting with a valid filter type.
    A palette image is still decoded. On 8,883 retained screenshots the two checks agreed on every
    file and the total time fell from 258 s to 18 s; verify of a synthetic run of 3 participants
    working 8 hours (8,895 screenshots of 147 KB) took 51 s, down from 282 s. The new check also
    refuses a few files the decoder accepted: image data cut short, rows missing, image data split
    by another chunk, a second IHDR, and bytes after a palette image's zlib stream (#1780).

Fixes

  • A participant that reaches its scheduled end on openai-computer-use now writes its closing
    report, and the report's summary is its reason and its stream's final ui.state. The provider
    offered the closing report only on a conversation the server held, so a study under zero data
    retention, which carries the conversation on the client, skipped it and left "the participant's
    scheduled end ... was reached" as the participant's whole account. The report is now asked on
    the carried conversation too: its newest screens whole and its oldest turns summarized past the
    token budget, which the trace's conversation record counts. stopCause still says
    scheduled_end, and a participant whose report does not come back keeps the scheduled end as
    its reason. Claude as a local-agent participant model has no closing request, so
    study check and run now warn when its participants have a scheduled end (#1776).

Checked before release

  • CI on this PR (sharded gate on both Node versions, site, Observer proofs, TUI, guest desktop, secret scan) is the gate of record.
  • The post-publish check (handoffs kit, verify.sh), rehearsed on the release tarball: all 125 checks passed. The registry and unpkg rows run after publish.
  • Live on the release tarball (hosted E2B, keys from the user store):
    • try-live with analysis: finished, analysis complete, verify 16 of 16 (local_only, raw screenshots). observer-data.json is 26 KB with no participant items; a served Observer answered observer-feed.json with the participant's 48 items.
    • Two participants with a 3-minute scheduled end on the CLI's default (threaded) conversation: both stopCause: scheduled_end with a completed closing report, whose summary is each participant's reason and ui.state; verify 16 of 16. The run's index.html opened from disk in Chromium with no page errors and showed the participant's 14 frames, reasoning, steps and closing report from the compressed items.
    • On the branch before merge, through the library with zeroDataRetention: true: two participants with a scheduled end on a client-carried conversation wrote their closing reports (#1782).
  • verify of a synthetic 8-hour, 3-participant run with 8,895 real-size screenshots: 51 s, 282 s on 0.123.0. On 8,883 retained screenshots the new screenshot check and the old decode agreed on every file (#1781).
  • The benchmark was not run: no change alters what a participant sees or what the analysis reads, apart from the closing report at a scheduled end, which the live runs above cover.

Not verified

  • A closing report written after a client-carried conversation summarized its oldest turns; the live sessions were too short to summarize any.
  • A live run longer than a few minutes on 0.124.0, and the live Observer feed during one: the feed still sends every participant's items on each poll (#1766 step 3).
  • macOS and Windows.

Package

humanish@0.124.0: 988 files, shasum 92e7cf587cb0e3844a3204b0c89387f79d41eb90.