Skip to content

0.120.0: Analysis completes under Bun, runs past 16 participants print a bounded list, Observer counts participants by status

Choose a tag to compare

@danielgwilson danielgwilson released this 09 Oct 22:15
· 200 commits to main since this release
16d6064

humanish 0.120.0 completes an analysis run from a Bun process, prints a run of more than 16
participants in a bounded number of lines, and counts and filters the Observer's participants by
status and persona. Under Bun, undici is Bun's built-in module, whose Agent has no destroy(),
and the analysis provider's teardown after each request threw a TypeError that replaced the
request's result, so every automatic analysis from a Bun process ended unknown with
AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN and kept no receipt. A failing teardown is now ignored.
humanish watch and humanish run print a computer-use run of more than 16 participants as one
line counting them by status, up to 16 participants that did not pass, and a not listed: line.
Each participant line holds its closing message on one line cut at 160 characters, and a warning
printed the same for each participant prints once. The Observer's participant grid counts the
participants by status when they ended more than one way, and pressing a count filters the grid.
Its view options gain a Persona filter, a filter change returns to the first page, and Next page
and Previous page open the new page at its first participant.

This is a minor release because watch and run print a computer-use run of more than 16
participants in a new shape and cut each participant's closing message at 160 characters, which a
script reading their terminal output would notice. Their --json output is unchanged. The package
exports the same 13 values and 31 types as 0.119.0 and has the same file list, dependencies and
engines. It removes or renames no command, flag, study field or bundle field, and verify,
review and runs print the same for the same bundle. These outputs change for the same input:

# 0.119.0 0.120.0 What to do
1 watch and run printed one line per participant for a computer-use run of two or more participants. In the smoke, 20 participants (2 passed, 18 gave up) gave 144 stdout lines, 47 of them the participant list; in the consumer check, a dry run of 40 gave 99 stdout lines with 40 participant lines. Above 16 participants: "participants: 20 · 2 passed · 18 abandoned", the first 16 participants that did not pass, one line each, and "not listed: 4 participants (2 did not pass). The Observer shows each one." The smoke's run gave 96 stdout lines, 18 of them the list; the dry run of 40 gave 61, with "participants: 40 · 40 dry run, nothing ran live" and "not listed: 40 participants. The Observer shows each one." A run of 2 to 16 participants still lists each one. (#1744) Read each participant from --json (lanes[]), run.json or review. Match participants: and not listed: for the counts.
2 Each participant line of a run of 2 or more participants ended with the participant's whole closing message, line breaks included: 2 or 3 terminal lines per participant in the smoke. The closing message is on the participant's one line, whitespace collapsed, and cut at 160 characters with …: "participant buyer-01: abandoned (gave_up) · participant outcome · DID NOT REACH THE GOAL. The page only displays bakery information and today's loaf ... or oth…". A run of one participant prints its session: line whole, as before. (#1744) Read the whole message from --json (lanes[].session.reason), run.json or review.
3 watch and run printed every warning the result carried, so a warning added once per participant printed once per participant: 20 copies of "Screenshots are unblurred for local use ..." for 20 participants, 86 warning lines in all. Each distinct warning text prints once, on all four routes: 1 copy, 67 warning lines. Warnings whose text names a participant (window clipping and fill, slow DevTools, persona background) still print once per participant. --json keeps every copy. (#1744) Count warnings from --json's warnings.
4 An automatic analysis run from a Bun process (runStudy under Bun 1.3.11) ended state: unknown with AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN, analysisId: null and error.code: ANALYSIS_UNAVAILABLE, kept only start.json in analysis-attempts/<id>/, and recorded no cost, although its request was sent: the job ran 80 s. The analysis provider threw "TypeError: dispatcher.destroy is not a function" on a 200 and on an HTTP 500. It ends complete with analysis/<id>/analysis.json and receipt.json, as under Node: in the smoke, one participant's analysis was expected at $0.80 and billed $0.20 under Bun ($0.25 under Node). A provider failure under Bun is recorded as that failure (provider_http_error for an HTTP 500, with a fake fetch). (#1753, fixes #1752) Nothing. To analyse a run that 0.119.0 left unknown under Bun, run humanish analyze --run <id>; its dry run on 0.120.0 admits such a run at an expected $0.80 (one participant). Spend from those 0.119.0 requests is in no receipt.

Added

  • The Observer's participant grid counts the participants by status in a row under the run's
    tally, when they ended more than one way. Pressing a count shows only those participants, and
    pressing it again shows everyone. The view options gain a Persona filter when the participants
    have more than one persona. In the smoke, 20 participants showed "Reported complete 2" and "Gave
    up 18"; pressing "Gave up" showed 18 cards. (#1744, from #1737)

Changed

  • humanish watch and humanish run print a computer-use run of more than 16 participants in a
    fixed number of lines: one line counting the participants by status, up to 16 participants that
    did not pass, and how many it did not list. A run of 16 or fewer still lists every participant.
    Each participant's line holds its closing message on one line, cut at 160 characters, where it
    used to print the whole message. A warning repeated for each participant prints once (rows 1 to
    3). (#1744, from #1737)
  • In the Observer, a changed grid filter shows the first page of participants. (#1744, from #1737)

Fixes

  • An analysis run from a Bun process completes. Under Bun, undici is Bun's built-in module,
    whose Agent has no destroy(), and the provider's teardown after each request threw a
    TypeError that replaced the request's result. Every automatic analysis then ended unknown
    with AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN, even when the model answered, and kept no receipt for
    requests already sent. A failing teardown is now ignored (row 4). (#1753, fixes #1752)
  • In the Observer, Next page and Previous page under the participant grid now open the new page at
    its first participant. The grid kept its scroll position, so turning the page from the bottom of
    36 participants showed the end of the next page, about 10,000 px past its first participant on a
    phone. (#1744, from #1737)

Known gaps

  • An error raised after an analysis request is dispatched, other than the teardown, is still
    recorded as unknown with no receipt, so the spend of requests already sent is not recorded.
    #1752 suggested recording it as failed with the usage known; #1753 fixes only the teardown.
    (#1752)
  • No live run of 40 or 100 participants has been made. The smoke ran 20 on one Observer page;
    #1744 measured the grid, paging and watch at 40 and 100 with fixtures built from one recorded
    participant. (#1737, #1744)
  • Above 16 participants, run still prints the per-participant warnings whose text names the
    participant: in the smoke, 20 participants gave 67 warning lines, 20 of them persona-background
    warnings and 40 window clipping and fill warnings. The plan and progress on stderr print one or
    more lines per participant (191 stderr lines for 20). The shared-world formatter still lists every
    participant on its own line. (#1744, #1739)
  • watch prints the run's block and then the Observer's block ("humanish observer rendered"), and
    both carry the run-level warnings, so a dry run prints "This is a dry run: ..." twice. 0.119.0
    does the same.
  • Two of the Observer browser proofs wait without a time limit, so a stall hangs the CI step until
    its 5-minute timeout without naming the case or step. (#1750)
  • The Observer's spend.requests counts an analysis attempt as one request. A cohort analysis of
    more than 16 participants sends one request per cohort and a merge request, and its
    study-analysis.json reads "requests": 1. (#1743)
  • On a local-tree or clone subject, the first participant holds the others at the pipeline gate
    until its desktop is ready. Participants due before the gate opens start when it opens, late, and
    the plan's schedule does not count the gate. A failed first desktop skips every other
    participant. (#1741)
  • A participant waits for its scheduled start before it is skipped for the budget, so a run whose
    budget is crossed early idles until its last start, with no sandbox. run prints such a
    participant as "blocked · unknown ending" and leaves out the reason, which run.json holds.
    (#1745)
  • In-progress bundles and the Observer show a participant waiting for its start as running. The
    plan's peak and the provisioned shared world's app deadline count each session at its full budget
    and leave out each desktop's start-up time; with several waves on a provisioned shared world,
    that start-up comes out of the 10-minute teardown buffer. (#1745)
  • doctor --study of a study of more than 16 participants prints "ok post-run analysis" and does
    not say that the default $3 cap refuses its analysis; run and study check say so. (#1742)
  • A merged report can name the cohorts: the 0.119.0 smoke's 24-participant summary opens "Across
    both cohorts". The merge request's Analysis requesting line reads "0 evidence items, 0
    captures." (#1742)
  • run does not print the warning that names HUMANISH_E2B_MAX_CONCURRENT_SANDBOXES when the
    limit holds participants to waves; the run records it as a study.warning event, and the
    terminal shows only the plan line. (#1739)
  • run prints its analysis line from the declared participant count, so a --count override does
    not change it, and a --count above 64 with real email receiving still fails at run start.
    (#1739)
  • A held Enter in humanish tui can still confirm an armed action on its first key repeat. The
    400 ms floor ignores repeats that come sooner, and a system's initial repeat delay is often
    longer (500 ms by default in GNOME). (#1730)
  • A serve.start that bash cannot parse still waits out readyTimeoutMs. The scripted and
    shared-world routes give a failed build their generic failure code; the message carries bash's
    error. (#1734)
  • On a desktop with speech, heard speech reaches the participant only with a screenshot and the
    guest holds 8 utterances between two screenshots, so a long wait in a talking call can overflow
    it; a wait there stays 30 seconds unless the study raises actor.maxWaitMs. (#1724)
  • A host that sleeps during a hosted run is reported as provider stalls and failed participants.
    (#1740)
  • Two contained-file races from #1669 remain. They need a second local process, running as the
    same user, to swap a directory in the run between two filesystem calls: temporary-file cleanup
    still unlinks by path (item 1), and the reviewer-notes listing still lists notes/ by path and
    can follow a swapped directory (item 3). Storage validation and the scan still list each folder
    whole before the 10,000-entry cap applies, verify has no total read budget, and a legitimate
    sandbox-receipts.ndjson over 32 MiB is refused. (#1669)
  • Coding agents still do not tell their person about humanish tui. (#1701)
  • Path redaction in terminal transcripts still removes the text after a path up to the next space,
    such as a line break and the next header after cwd:. Nothing leaks; text a reviewer needs is
    lost. (#1702)
  • A hosted computer-use participant cannot type keysyms outside US printable ASCII (é, €) as
    keys; it has to type them as text. (#1713)
  • The routes still take E2B_DOMAIN from process.env on create and ignore one in
    RunStudyOptions.env. (#1717)
  • From #1646, after #1684: a 4-byte value inside a longer base64 run is not found, two further
    decodings at a marker edge are not read, RunSecrets.spans (the terminal route's per-chunk
    replacement) has no final check, and a value that is part of a marker nests markers. (#1646)
  • While a live run works, its study screen in humanish tui reads "1 run, none live".
  • Design findings cannot say that text or a click target is too small by measure. (#1638)
  • The impressions ask describes its kinds in visual terms, which can steer a participant toward
    visual critique whatever its persona. (#1654)
  • Nothing Mac-specific has run on a Mac: the Lima VM sizing and runtime setup --memory --cpus.

Package

humanish 0.120.0 packs to 2.5 MB, 7.9 MB unpacked, 972 files (npm pack on release/0.120.0 at
1122987, shasum ee45f9f5e398f9b9327ff3deecfeb09f117b724e), the same file list as 0.119.0. The
package exports only ., with the same 13 values and 31 types as 0.119.0 (pnpm api:proof). Its
dependencies, optional peer and engines are the same as 0.119.0's: Node 22.19 or newer, and
@e2b/desktop 2.3.2 or newer as an optional peer.

Checked before release

A live smoke on a tarball packed from main cf75cc6, in fresh projects under /tmp with a fresh HOME
and npm cache, DO_NOT_TRACK=1, and @e2b/desktop 2.4.0 beside humanish. CLI commands got keys
from the user key store through key discovery, with XDG_CONFIG_HOME pointing at it. The
runStudy scripts, which do no key discovery, got the same store through the runtime's own
--env-file (Bun 1.3.11 and Node 24.12.0) into that one process. No --dotenv was passed. The
store's OpenAI organization is non-ZDR, so every OpenAI participant ran threaded; its E2B key is
on a Pro plan. Controls ran the same study files on npm 0.119.0 before #1753 merged. Priced spend:
$2.62 for the smoke ($1.13 of it in three analyses; $0.60 in the npm 0.119.0 controls, whose Bun
analysis request was billed and not recorded), then $0.26 for the dogfood run and $5.87 for the
benchmark.

  • init --yes writes the same files as npm 0.119.0, and doctor (the concurrent-sandboxes row
    unset, at 100 and at lots), doctor --study try-live, keys, study check try-live and
    analyze --help print the same apart from the cwd.
  • runStudy from a Bun process (row 4): one participant on a static page with a declared analysis.
    On main under Bun 1.3.11 the participant ended goal_satisfied and the automatic analysis ended
    complete, with analysis.json and receipt.json written (expected $0.80, billed $0.20; 9
    cited evidence ids, all resolved). The same script under Node 24.12.0: complete (expected
    $0.80, billed $0.25). On npm 0.119.0 under Bun: unknown, AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN,
    only start.json, no cost recorded, after an 80-second job.
  • 20 participants with HUMANISH_E2B_MAX_CONCURRENT_SANDBOXES=100 and 2-minute sessions (rows 1
    to 3): 2 readers passed and 18 participants asked to order on a page without an order form gave
    up, on both versions; E2B listed the run's 20 sandboxes at once. run printed 96 stdout lines
    against npm 0.119.0's 144: the list was 18 lines against 47, the longest closing message 160
    characters, and the screenshots warning 1 copy against 20. The default analysis was refused
    before it started at the $3 cap and recorded skipped with AUTOMATIC_ANALYSIS_ADMISSION_REFUSED
    on both. In headless Chrome the candidate's Observer showed the status row "Reported complete 2",
    "Gave up 18"; pressing "Gave up" showed 18 cards and pressing it again 20; the Persona select
    listed both personas. npm 0.119.0's Observer of its own run had neither.
  • try-live, as init writes it: reached the goal in 10 participant turns, input growing every
    turn from 2,888 to 18,100 tokens, then 18,463 for the impressions request; cached input on each
    turn from the second was the previous turn's input minus 3; threaded on all 11 requests. $0.25
    run. Its automatic analysis, one request, was admitted at an expected $1.03 (admittedCostUsd
    $1.13, worst case $2.01, cap $3) and billed $0.67, with 2 findings and 4 design findings whose 29
    cited evidence ids all resolve. It gave 5 impressions (unclear, unclear, unfinished, liked,
    liked).
  • verify, review and runs (text and --json) on copies of the six live runs: all 38 outputs
    byte-identical on main and npm 0.119.0. A dry run of analyze --run on npm 0.119.0's unknown
    Bun run is admitted on main at an expected $0.80.
  • After every live run, reclaim --check exited 0 with clean, and verify exited 0 with
    local_only. Raw sandbox ids appear only in sandbox-receipts.ndjson. E2B listed 0 tagged
    sandboxes before each batch, and after the smoke 1 tagged sandbox from none of its runs (the
    dogfood ran then), and 0 after the benchmark. No key value from the store appears in any run
    or kit file, as written, base64 or hex (11,386 files).

On the release branch:

  • pnpm release:check on the release commit 1122987: exit 0 in 4.1 min, 515 test files passed
    and 10 skipped, 8,042 tests passed and 12 skipped; TUI 192 tests; lint 426 warnings at its cap of
    426; api:proof "13 values and 31 types match"; public-surface scan of 3,068 text files and 527
    binary assets; npm pack --dry-run 972 files with shasum
    ee45f9f5e398f9b9327ff3deecfeb09f117b724e. pnpm docs:generate there wrote 0.120.0 into cli.mdx.
    On the benchmark commit bd5a7d2, the release head: exit 0 in 3.9 min, the same test, lint and
    api:proof counts, a public-surface scan of 3,070 text files and 527 binary assets, and the same
    npm pack --dry-run shasum, so the tarball the checks below used is the head's package.
  • pnpm release:dogfood on 1122987 (the script run with Node, keys read from the user store by
    Node's --env-file into its environment, and a temporary HOME): "release:dogfood ok", verdict
    passed (goal_satisfied), no-spend satisfied, $0.26. The participant ran init, the free dry
    run, verify (16 of 16, share_ready) and review on the installed 0.120.0, and did not start a
    paid live run. verify on a copy of the run: share_ready, 16 of 16; reclaim --check clean.
    0.119.0's dogfood ended blocked (blocked_approval) after the same steps.
  • The post-publish check on the release tarball from 1122987 (TMPDIR=/tmp, HUMANISH_SPEC the
    tarball): 125 of 125 passed. Its 2 new rows cover watch --dry-run of 16 and 40 participants
    (#1744) and the analysis provider under Bun and Node with a fake fetch (#1753). The same script on
    npm 0.119.0 passed 123 and failed the 2 new rows. A first pass failed row 117 on both versions
    (124 and 123 of 125): it counted a warning repeated anywhere in watch's stdout, and watch
    prints the run's block and then the Observer's, each with the same two run-level warnings. The
    row now checks the run's block, where 0.120.0 dedupes; both passes are in post-publish/.
  • The consumer check: the 0.119.0 check's library and CLI consumer, plus 4 CLI lines for this
    release, on the release tarball from 1122987. On that tarball the consumer typechecks with 0
    errors, as on npm 0.119.0, and needs no migration. The library steps are the same on both
    versions, and of the 138 CLI lines from the 0.119.0 check, 136 are the same; the two that differ
    are the dry runs of 17 and 24 participants, which now print "participants: 17 · 17 dry run,
    nothing ran live" (row 1). watch --dry-run of 16 participants prints the same 51 lines; of 40,
    61 lines against 99 (row 1). The package's analysis provider with a fake fetch under Bun returns
    completed for a 200 and provider_http_error for a 500, where npm 0.119.0 throws the
    TypeError on both; under Node both versions return the same (row 4).
  • pnpm bench on 1122987 with its defaults (3 runs per arm, openai-computer-use, neutral
    mission), committed in bd5a7d2: report recall 12/15, analysis recall 11/15, 0 invented on either
    arm, $5.87 estimated. 0.119.0 scored 10/15 and 10/15 at $6.12 (11/15 each read by hand). The bench
    skipped the clean 3 analysis as skipped_budget, since spend stood at $5.87 and that analysis
    could have passed the $7 cap, so the analysis column covers 3 planted and 2 clean participants.
    D1 and D5 account for the rise: all three planted reports name each, where two did on 0.119.0.
    Read from the traces' click coordinates as #1749 does, every planted participant committed its
    first edit with Save (3 of 3 on both versions) and pressed Clear completed after checking a task
    (3 of 3, against 2 of 3 on 0.119.0). The D1 rise follows that reach; D5 was reached 3 of 3 on both
    and named 3 of 3 against 2 of 3. Planted 2's unresolved analysis finding "Edited task text
    reverted after Save and filter changes" describes D5, so analysis recall read by hand is 12/15.
    Three runs per arm cannot separate this rise from the noise #1749 measured (52/80 planted-report
    recall each for npm 0.114.0 and main d183d1a in one hour). The analysis cap was the dry run's
    admittedCostUsd, $1.21, against bills of $0.64 to $0.93. No planted participant typed more than
    28 characters in one action, so none met D2. Every participant ran threaded, ended on its own and
    gave 6 impressions.

Not verified

  • macOS and Windows. Every check ran on Linux x64 with Node 24.12 and Bun 1.3.11.
  • A live run on the published package: the live smoke ran on a main tarball, and the post-publish
    rows are offline dry runs, refusals, local runs and a fake fetch.
  • Bun versions other than 1.3.11 (#1752 saw 1.3.13), the humanish CLI run under Bun, and a Bun
    analysis of more than 16 participants: the smoke's Bun analysis sent one request, where the
    adopter's 34-participant run sent cohort requests and a merge. The fix is in the per-request
    teardown that every cohort and merge request runs.
  • A live analysis request that fails under Bun; the fake fetch covers an HTTP 500.
  • A live run of 40 or 100 participants, live paging in the Observer, and the shared-world,
    terminal and scripted routes' warning dedupe on a live run.
  • --json of watch and run compared between versions on a live run; #1744 says it is unchanged.
  • What the npm 0.119.0 control's Bun analysis request cost: 0.119.0 recorded no receipt.
  • analyze --run sent live on a run that 0.119.0 left unknown; only its dry run was tried.