Skip to content

0.119.0: Participants arrive over time, one analysis past 16 participants in cohorts, late starts skipped past the budget, study warnings redacted

Choose a tag to compare

@danielgwilson danielgwilson released this 09 Oct 07:20
· 197 commits to main since this release
1c2fc87

humanish 0.119.0 lets a computer-use or shared-world study start its participants over time, and
analyses a run of 17 to 100 participants in one report. participants[].startAfterMs starts a
participant up to 24 hours after the run starts its participants, and a count group's
startEveryMs spreads its members. Each participant's desktop is created at its start, the plan and
study check print the first and last start and the most participants at once, and run.json records
each participant's arrival. A participant whose start comes after the study crossed
caps.maxTotalUsd is skipped before its desktop is created. The analysis of more than 16
participants splits them into cohorts of at most 16, analyses each cohort with its own request, and
merges the cohort reports into the run's one report with one more request; 0.118.0 skipped the
automatic analysis of such a run. Admission, the plan's analysis line, study check and the
analyze --dry-run worst case count every request, and analyze --json gives their number as
admission.requests. At the default $3 cap the analysis of 17 or more participants is refused even
with no evidence, so it needs a higher review.analysis.maxCostUsd. A provisioned shared world gets
an app sandbox that lives until the last start or wave can end. A study warning recorded in the
run's bundle is redacted as a run failure is, so a hosted study whose participant instruction names
a sandbox URL other than the subject's verifies share_ready again, as on 0.111.2.

This is a minor release because it adds two study fields and a JSON field, changes how a run of
more than 16 participants is analysed, and changes outputs for the same input. A study that sets
startAfterMs or startEveryMs runs on 0.119.0, and npm 0.118.0 refuses it when read. A run of 17
to 100 participants, whose automatic analysis 0.118.0 skipped with
AUTOMATIC_ANALYSIS_PARTICIPANT_LIMIT, is analysed in cohorts when its cap admits it; at the
default $3 cap its default analysis is skipped with AUTOMATIC_ANALYSIS_ADMISSION_REFUSED.
AUTOMATIC_ANALYSIS_PARTICIPANT_LIMIT is no longer recorded, and study check --json no longer
carries analysis.skip; 0.118.0 added both. analyze --run on such a run sends three or more
requests, and its expected cost is about three times that of 0.118.0's partial analysis ($6.20
against $2.20 on the smoke's 24-participant copy). analyze --json adds
admission.requests. A participant whose start comes after the study budget is spent is recorded
blocked with no desktop, where it made one model request in 0.118.0. A provisioned shared world
whose waves outlast the sandbox ceiling is refused at plan time. A hosted study whose participant
instruction names a sandbox URL other than the subject's verifies share_ready again, where
0.113.0 to 0.118.0 graded it blocked and failed the run. The package exports the same 13
values and 31 types as 0.118.0; the exported RunBundle's participant records gain an optional
arrival. It removes or renames no command, flag, study field or bundle field. These outputs
change for the same input, which a script could notice:

# 0.118.0 0.119.0 What to do
1 A study that sets participants[].startAfterMs or startEveryMs was refused when read: "Unknown study field in participants[0]: startEveryMs. Known fields: ...", HUMANISH_STUDY_INVALID, exit 2. It plans and runs. study check adds a row, "- ok schedule: the first participant starts at +0s and the last at +3m 50s; at most 18 run at once when every session uses its 3m budget." (a checks[] entry named schedule in --json); the plan prints the same after schedule:; run.json records each participant's arrival (startAfterMs, and on a live run scheduledAt and startedAt). A start past 24 hours or not a whole number, startEveryMs without count, a start on a study of one participant, a group whose last member passes 24 hours and a later start for the external-public host are refused with HUMANISH_STUDY_INVALID, each in its own sentence. A study that declares no start prints and records the same as before. (#1745) A study that declares starts needs 0.119.0. npm 0.118.0 reads such a run's bundle with the same output, but not its study file, so its review of a run whose study sets review.analysis: false says "No analysis has run for this run."
2 A participant whose start came after the study crossed caps.maxTotalUsd (a later wave) had its desktop created, made one model request and stopped. It is skipped before its desktop is created: blocked in run.json and the result, with the summary "Late was skipped: study budget reached before this participant started: the run's estimated model spend $0.0021 crossed caps.maxTotalUsd=$0.001; no desktop was created.", and counted in the fan-out summary's skipped participants. In the smoke, two participants, the second due at +1 min: exit 2, "Fan-out run failed: 0/2 participants passed (1 skipped, 0 harness errors, 0 without engagement).", one sandbox. (#1745) Count a blocked participant whose summary says it was skipped as not run.
3 A live run of more than 16 participants recorded its automatic analysis as skipped with AUTOMATIC_ANALYSIS_PARTICIPANT_LIMIT ("analysis: not run, because the study has more than 16 participants ..."): exit 0 without a declared review.analysis, 2 with one. run, study check and the TUI printed "After live runs: automatic analysis will not run. This study has 24 participants, and automatic analysis reads at most 16. ..." One analysis covers every participant, in cohorts of at most 16 and one merge request. At the default $3 cap the line reads "refused before it starts for 17 participants in 2 cohort requests of at most 16 participants and one merge request even with no evidence (expected $3.15), since both its worst case and its expected cost plus a 10% margin are over $3", and a default analysis is skipped with AUTOMATIC_ANALYSIS_ADMISSION_REFUSED; a declared one is refused and the run exits 2. At a cap that admits it the line gives a range: "expected $5.14 to $9.25 for 24 participants in 2 cohort requests of at most 16 participants and one merge request ..." at $10. The smoke's 24-participant run at $10 was admitted at an expected $6.20 and billed $2.65. (#1742) Set review.analysis.maxCostUsd from the plan's range, or review.analysis: false. Match AUTOMATIC_ANALYSIS_ADMISSION_REFUSED; AUTOMATIC_ANALYSIS_PARTICIPANT_LIMIT is no longer recorded.
4 study check --json of more than 16 participants carried analysis.skip: "AUTOMATIC_ANALYSIS_PARTICIPANT_LIMIT". doctor --study printed "- note post-run analysis: Will not run: this study has 17 participants, more than automatic analysis reads. The participants still run." analysis.refusedFromUsd (3.150188 for 17 at $3) or analysis.expectedCostUsd, as for any study, and no skip. doctor prints the post-run analysis row a study of 16 gets. (#1742) Read refusedFromUsd or expectedCostUsd.
5 analyze --run on a run of more than 16 participants sent one request with the first 16 participants' evidence and saved a partial analysis listing the rest as omitted. On the smoke's 24-participant copy: "Worst case: $2.44, if the analyst writes its whole 32768-token output allowance, reasoning included.", expected $2.20, billed $1.36, 16 participant reviews. It sends one request per cohort and a merge request, and saves a complete analysis. On the same copy: "Expected cost: $6.20. Worst case: $7.07, if every request (2 cohort requests of at most 16 participants and one merge request) writes its whole 32768-token output allowance, reasoning included.", billed $2.79, 24 participant reviews. It prints an Analysis requesting line for each request; the merge's reads "0 evidence items, 0 captures." (#1742) Pass the dry run's worst case as --max-cost.
6 analyze --json's admission had no requests. admission.requests: 1 up to 16 participants, and the cohorts plus one past 16 (3 for 17 to 32). The other keys are unchanged. (#1742) Nothing; read it for the number of requests.
7 A stored analysis held at most 2,000 evidence items, 64 captures and 4 MiB. At most 6,400, 320 and 16 MiB, eight cohorts' packets. npm 0.118.0 reads the smoke's cohort analyses (351 evidence items, 24 captures) with the same output; one past its bounds was not tried. (#1742) Open a cohort analysis of more than 64 captures with 0.119.0.
8 A provisioned shared world that ran in waves planned with an app sandbox deadline of one session plus provisioning, which a later wave could outlast: 40 members at the default limit planned as "participants: 40, up to 19 at once". The app sandbox lives until the last start or wave can end, and a study that puts that past HUMANISH_E2B_MAX_SANDBOX_MINUTES is refused at plan time, dry runs too: "The participants use the app for up to 30m when every session uses its 10m budget, because 21 participants wait for a free slot. The subject sandbox, which serves the app until every participant ends, then needs a 70m deadline, ... set HUMANISH_E2B_MAX_SANDBOX_MINUTES to 70 or more.", HUMANISH_SHARED_WORLD_INVALID. (#1745) Set the setting to your plan's limit (Pro allows 1440), run more members at once, or lower execution.timeoutMs.
9 A study whose mission or participant instruction names a sandbox URL other than the subject's, in a line the script-like warning quotes, recorded the warning with the URL in events.ndjson, run.json and observer-data.json. verify graded the run blocked (PUBLIC_SAFETY_FINDINGS) and the run failed, dry runs too: "humanish run failed: Run bundle failed verification.", HUMANISH_COMPUTER_USE_FAILED, exit 2. This started in 0.113.0. The recorded warning reads [REDACTED_SECRET] in place of the URL, as the recorded mission already did, and the run verifies share_ready and exits 0. In the smoke, one live participant whose mission names https://3000-...e2b.app/inbox, with blurred screenshots: share_ready, 16 of 16, the URL in no run file; npm 0.118.0: exit 2, blocked. The terminal still prints the warning with the URL. (#1748, fixes #1747) Nothing; a study 0.113.0 to 0.118.0 failed this way runs again.

Added

  • A computer-use or shared-world study can start its participants over time:
    participants[].startAfterMs starts a participant that many milliseconds after the run starts
    its participants (up to 24 hours), and a group's startEveryMs spreads its members, member k at
    startAfterMs + (k - 1) * startEveryMs. Each participant's desktop is created at its start. One
    whose start comes while every slot is taken waits for the next free one. study check and the
    run's plan print the first and last start and the most participants at once, and run.json
    records each participant's arrival with its scheduled and actual start. A shared-world plan
    with a schedule prints its worst case in sandbox-minutes. A provisioned shared world's app
    sandbox serves until the last participant ends, and a study whose last start puts that past
    HUMANISH_E2B_MAX_SANDBOX_MINUTES is refused before anything is created. A rerun
    (--rerun-failed-from) starts its selected participants together (row 1). (#1745, from #1737)

Changed

  • A computer-use or shared-world participant whose start comes after the study crossed
    caps.maxTotalUsd, in a later wave or later in its schedule, is skipped before its desktop is
    created. It is recorded as blocked with a reason naming the budget and counted among the fan-out
    summary's skipped participants. It used to create its desktop, make one model request and stop
    (row 2). (#1745, from #1737)
  • One analysis of a run with more than 16 participants covers all of them, up to 128. They are
    split into cohorts of at most 16, each analysed by its own request under the full packet limits,
    and one more request merges the cohort reports into the run's one report. In 0.118.0 a live run
    of more than 16 participants recorded its automatic analysis as skipped with
    AUTOMATIC_ANALYSIS_PARTICIPANT_LIMIT, and analyze read the first 16 participants and listed
    the rest as omitted; that reason is no longer recorded. Admission, the expected cost, the worst
    case and the study check range cover every request. The plan's analysis line and the analyze --dry-run worst case name the cohort and merge requests, and admission.requests in --json
    counts them. The default $3 cap refuses the analysis of 17 participants even for a run that
    keeps no evidence, so a default analysis of more than 16 participants is skipped with
    AUTOMATIC_ANALYSIS_ADMISSION_REFUSED unless review.analysis.maxCostUsd admits it. analyze
    prints an Analysis requesting line for each request. A cohort or merge request that fails
    fails the attempt with no findings and a warning, and its usage counts every request sent. The
    humanish.study-analysis.v1 schema is unchanged; its stored limits grew to 6,400 evidence
    items, 320 captures and 16 MiB (rows 3 to 7). (#1742, from #1737)

Fixes

  • A study warning recorded in the run's bundle goes through the same redaction as a run failure.
    The warning that a participant instruction reads like a script quotes up to three of its lines,
    and a line that named a sandbox URL other than the subject's (a run inbox page, a second app)
    put that URL in events.ndjson, run.json and observer-data.json. verify then graded the
    bundle blocked and a hosted run failed with "Run bundle failed verification", where 0.111.2
    verified it share_ready. The terminal still prints the warning as written (row 9). (#1748,
    fixes #1747)
  • A provisioned shared-world study that runs in waves asks E2B for an app sandbox that lives until
    the last wave can end, and is refused when that passes HUMANISH_E2B_MAX_SANDBOX_MINUTES. Its
    app sandbox lived one session plus provisioning, which a second wave could outlast (row 8).
    (#1745, from #1737; the waves came in 0.118.0 with #1739)

Known gaps

  • The Observer and humanish watch at 40 and 100 participants are the last slice of #1737, in
    #1744, which is not in this release. (#1737, #1744)
  • The Observer's spend.requests counts an analysis attempt as one request. The smoke's
    24-participant run priced 3 requests, and its study-analysis.json reads "requests": 1. The
    bundle does not record how many requests an attempt sent; its receipt holds their summed usage.
    (#1743)
  • On a local-tree or clone subject, the first participant holds the others at the pipeline gate
    until its desktop is ready. Participants due before the gate opens start when it opens, late,
    and the plan's schedule does not count the gate: in the smoke, participants due at +10 s to
    +40 s all started at +44 s. A failed first desktop skips every other participant. (#1741)
  • A participant waits for its scheduled start before it is skipped for the budget, so a run whose
    budget is crossed early idles until its last start, with no sandbox. run prints such a
    participant as "blocked · unknown ending" and leaves out the reason, which run.json holds.
    (#1745)
  • In-progress bundles and the Observer show a participant waiting for its start as running. The
    plan's peak and the provisioned shared world's app deadline count each session at its full
    budget and leave out each desktop's start-up time; with several waves on a provisioned shared
    world, that start-up comes out of the 10-minute teardown buffer. (#1745)
  • doctor --study of a study of more than 16 participants prints "ok post-run analysis" and does
    not say that the default $3 cap refuses its analysis; run and study check say so. (#1742)
  • A merged report can name the cohorts: the smoke's 24-participant summary opens "Across both
    cohorts". The merge request's Analysis requesting line reads "0 evidence items, 0 captures."
    (#1742)
  • run does not print the warning that names HUMANISH_E2B_MAX_CONCURRENT_SANDBOXES when the
    limit holds participants to waves; the run records it as a study.warning event, and the
    terminal shows only the plan line. (#1739)
  • A study of many participants prints per-participant lines: 24 participants gave run 100
    warning lines, among them 24 persona-background warnings for a persona with no background.
    (#1739)
  • run prints its analysis line from the declared participant count, so a --count override does
    not change it, and a --count above 64 with real email receiving still fails at run start.
    (#1739)
  • A held Enter in humanish tui can still confirm an armed action on its first key repeat. The
    400 ms floor ignores repeats that come sooner, and a system's initial repeat delay is often
    longer (500 ms by default in GNOME). (#1730)
  • A serve.start that bash cannot parse still waits out readyTimeoutMs. The scripted and
    shared-world routes give a failed build their generic failure code; the message carries bash's
    error. (#1734)
  • On a desktop with speech, heard speech reaches the participant only with a screenshot and the
    guest holds 8 utterances between two screenshots, so a long wait in a talking call can overflow
    it; a wait there stays 30 seconds unless the study raises actor.maxWaitMs. (#1724)
  • A host that sleeps during a hosted run is reported as provider stalls and failed participants.
    (#1740)
  • Two contained-file races from #1669 remain. They need a second local process, running as the
    same user, to swap a directory in the run between two filesystem calls: temporary-file cleanup
    still unlinks by path (item 1), and the reviewer-notes listing still lists notes/ by path and
    can follow a swapped directory (item 3). Storage validation and the scan still list each folder
    whole before the 10,000-entry cap applies, verify has no total read budget, and a legitimate
    sandbox-receipts.ndjson over 32 MiB is refused. (#1669)
  • Coding agents still do not tell their person about humanish tui. (#1701)
  • Path redaction in terminal transcripts still removes the text after a path up to the next space,
    such as a line break and the next header after cwd:. Nothing leaks; text a reviewer needs is
    lost. (#1702)
  • A hosted computer-use participant cannot type keysyms outside US printable ASCII (é, €) as
    keys; it has to type them as text. (#1713)
  • The routes still take E2B_DOMAIN from process.env on create and ignore one in
    RunStudyOptions.env. (#1717)
  • From #1646, after #1684: a 4-byte value inside a longer base64 run is not found, two further
    decodings at a marker edge are not read, RunSecrets.spans (the terminal route's per-chunk
    replacement) has no final check, and a value that is part of a marker nests markers. (#1646)
  • While a live run works, its study screen in humanish tui reads "1 run, none live".
  • Design findings cannot say that text or a click target is too small by measure. (#1638)
  • The impressions ask describes its kinds in visual terms, which can steer a participant toward
    visual critique whatever its persona. (#1654)
  • Nothing Mac-specific has run on a Mac: the Lima VM sizing and runtime setup --memory --cpus.

Package

humanish 0.119.0 packs to 2.5 MB, 7.9 MB unpacked, 972 files (npm pack on release/0.119.0 at
558a295, shasum 9c884092b9d8dca9d1396e2699772752f617a691; the same file list as at 53dbf23).
Against 0.118.0 it adds dist/analysis/cohorts.js, dist/analysis/responses.js and
dist/study/arrivals.js, each with its .d.ts, and drops dist/study/automatic-analysis-plan.js
and .d.ts. The package exports only ., with the same 13 values and 31 types as 0.118.0 (pnpm api:proof); the exported RunBundle's participant records gain an optional arrival. Its
dependencies, optional peer and engines are the same as 0.118.0's: Node 22.19 or newer, and
@e2b/desktop 2.3.2 or newer as an optional peer.

Checked before release

A live smoke on a tarball packed from main 6405a7c, and its #1748 row on the release tarball from
558a295, in fresh projects under /tmp with a fresh HOME and npm cache, DO_NOT_TRACK=1, and
@e2b/desktop 2.4.0 beside humanish. Keys came from the user key store through key discovery, with
XDG_CONFIG_HOME pointing at it; no --dotenv was passed. The store's OpenAI organization is
non-ZDR, so every OpenAI participant ran threaded; its E2B key is on a Pro plan. Controls ran the
same study files or commands on npm 0.118.0, read copies of the main runs, or analysed a second copy
of the 0.118.0 smoke's 24-participant run. Priced spend: $8.54 for the smoke ($7.57 of it in four
analyses, $1.36 of those on npm 0.118.0; $0.02 in the npm 0.118.0 run of the #1748 row), then $0.22
for the dogfood run and $6.12 for the benchmark. The dogfood run, the benchmark and the consumer
check ran on the release commit 53dbf23, before main b382916 (#1748) was merged into the branch;
#1748 changes how study warnings are recorded in the bundle and nothing a participant receives or
does.

  • init --yes writes the same files as npm 0.118.0, and doctor (the concurrent-sandboxes row
    unset, at 100 and at lots), doctor --study try-live, keys, study check try-live and
    analyze --help print the same apart from the cwd line.
  • try-live, as init writes it: reached the goal in 11 participant turns, input growing every
    turn from 2,888 to 19,839 tokens, then 20,170 for the impressions request; cached input on each
    turn from the second was the previous turn's input minus 3; threaded on all 12 requests. $0.28
    run. Its automatic analysis, one request, was admitted at an expected $1.04 (admittedCostUsd
    $1.15, worst case $2.03, cap $3) and billed $0.78, with 4 findings and 4 design findings whose 30
    cited evidence ids all resolve. It gave 6 impressions (unclear, liked, unclear, unfinished,
    liked, liked).
  • 24 participants arriving 10 s apart (count: 24, startEveryMs: 10000) on a one-paragraph
    static page, HUMANISH_E2B_MAX_CONCURRENT_SANDBOXES=100, 3-minute sessions,
    review.analysis.maxCostUsd: 10 (rows 1 and 3). The plan read "the first participant starts at
    +0s and the last at +3m 50s; at most 18 run at once when every session uses its 3m budget."
    run.json recorded each arrival: participant 1 and participants 6 to 24 started within 55 ms of
    scheduledAt (19 of the 20 within 1 ms); participants 2 to 5, due at +10 s to +40 s, started
    at +44 s, when the first participant cleared the local-tree pipeline gate. Sandbox receipts are
    timed from the first start to 0.4 s after the last; E2B listed at most 7 of the run's sandboxes
    at once. 24 of 24 reached the goal; $0.62 run. The analysis line read "expected $5.14 to $9.25
    for 24 participants in 2 cohort requests of at most 16 participants and one merge request"; the
    automatic analysis was admitted at an expected $6.20 (admittedCostUsd $6.82, worst case $7.07,
    requests 3) and billed $2.65, 5 min 33 s. The report covers 24 of 24 participants with 24
    participant reviews, 2 findings, 2 design findings and 4 concern reviews; its 172 distinct cited
    evidence ids all resolve in the stored packet, and its 24 cited captures exist with their
    recorded sha256.
  • The #1748 row (row 9): one participant on the bakery page with blurred screenshots, a mission
    that names https://3000-isbxsmokewarning0119.e2b.app/inbox, and no analysis. On the release
    tarball from 558a295 the run exited 0 and verified share_ready, 16 of 16; its recorded warning
    and mission read [REDACTED_SECRET] in place of the URL, no run file holds it, and the terminal
    printed the warning with the URL. $0.04. On npm 0.118.0 the same study exited 2, "humanish run
    failed: Run bundle failed verification.", HUMANISH_COMPUTER_USE_FAILED, blocked with
    PUBLIC_SAFETY_FINDINGS, and the URL in run.json, events.ndjson and observer-data.json. $0.02.
    Dry runs of the same study, and of one whose participant instruction names such a URL, gave the
    same grades.
  • The budget skip (row 2): two participants, the second due at +60 s, caps.maxTotalUsd 0.001.
    The first crossed the budget on its first request ($0.0021). The second was recorded blocked
    with "no desktop was created", its arrival with scheduledAt and no startedAt; the run had
    one sandbox receipt, E2B listed one sandbox for it, and the run ended at the second's scheduled
    time. Exit 2, "0/2 participants passed (1 skipped, ...)". $0.006.
  • analyze --run on two copies of the 0.118.0 smoke's 24-participant run, with each version's
    dry-run worst case rounded up as --max-cost (row 5): npm 0.118.0 sent one request of 233
    evidence items and 16 captures and saved a partial analysis of 16 participants, 8 omitted
    (expected $2.20, billed $1.36). The candidate sent two cohort requests of 175 and 176 items with
    12 captures each and a merge request, and saved a complete analysis of 24 (expected $6.20, billed
    $2.79, 6 min 17 s); its 195 distinct cited ids all resolve and its 24 cited captures exist.
  • verify, review and runs (text and --json) on copies of the three live runs and the
    candidate-analysed 24-participant copy: byte-identical on main and npm 0.118.0, except review
    of the budget-skip run, whose study file npm 0.118.0 cannot parse (row 1). verify and review
    of the arrivals run with the release tarball from 558a295 print the same as with main's.
  • Dry runs and study check of the arrivals and budget-skip studies plan on main and are refused
    as unknown fields on npm 0.118.0. A provisioned shared world of 40 members and one whose second
    member starts 3 hours in are refused on main (row 8); npm 0.118.0 plans the 40.
  • After every live run, reclaim --check exited 0 with clean, and verify exited 0 with 16 of 16
    checks, local_only. Raw sandbox ids appear only in sandbox-receipts.ndjson. E2B listed 0
    tagged sandboxes after the smoke and after the #1748 pair (2 before, from other sessions' runs)
    and 0 after the benchmark. No key value from the store appears in any run or kit file, as written,
    base64 or hex (16,446 files).

On the release branch:

  • pnpm release:check on the release head 558a295, after the merge of main b382916 (#1748) and
    the benchmark note: exit 0 in 3.9 min, 513 test files passed and 10 skipped, 8,034 tests passed
    and 12 skipped (#1748 adds one); TUI 192 tests; lint 426 warnings at its cap of 426; api:proof "13
    values and 31 types match"; public-surface scan of 3,063 text files and 527 binary assets; npm pack --dry-run 972 files with shasum 9c884092b9d8dca9d1396e2699772752f617a691. On the benchmark
    commit cd8bf63 before the merge: exit 0 in 3.7 min, 3,062 text files, shasum
    b10a60b8c14494d4035e7bf353211a1001b21d93. On the release commit 53dbf23: exit 0 in 3.9 min, 512
    test files passed and 10 skipped, 8,033 tests passed and 12 skipped; TUI 192 tests; lint 426
    warnings at its cap of 426; api:proof "13 values and 31 types match"; public-surface scan of 3,060
    text files and 527 binary assets; npm pack --dry-run 972 files with shasum
    b10a60b8c14494d4035e7bf353211a1001b21d93. pnpm docs:generate there wrote 0.119.0 into cli.mdx;
    the merge leaves cli.mdx as it is. The first run on the benchmark commit, before an amend
    (f0ac0892), exited 1 in 11 s on prose.evidence.issue-refs: 103 (cap 102, over by 1): the new
    benchmark README paragraph named #1749. The amend states the finding without the issue number.
  • pnpm release:dogfood on 53dbf23, before the merge of #1748 (the script run with Node, keys read
    from the user store by Node's --env-file into its environment, and a temporary HOME):
    "release:dogfood ok", verdict blocked (blocked_approval), no-spend satisfied, $0.22. The
    participant ran init, the free four-participant dry run, verify ("all 16 evidence checks",
    share_ready) and review on the installed 0.119.0, and stopped at the keys and the local VM
    devices the gate withholds; the script reads that verdict as the expected stop. verify on a copy
    of the run: share_ready, 16 of 16; reclaim --check clean.
  • The post-publish check on the release tarball from 558a295 (TMPDIR=/tmp, HUMANISH_SPEC the
    tarball): 123 of 123 passed. Its 6 new rows cover the schedule from startAfterMs and
    startEveryMs in study check, the plan and run.json (#1745); four start refusals (#1745); a
    3-hour start planned on computer use and refused on a provisioned shared world (#1745); the
    17-participant analysis line at a $10 cap and analysis.refusedFromUsd at $3 (#1742);
    admission.requests 1 and 3 (#1742); and a dry run whose participant instruction names a
    non-subject e2b URL, which verifies share_ready with the URL in no run file and the warning on
    the terminal (#1748). Row 20bz now expects the 17-participant refusal line in place of 0.118.0's
    skip. The same script on npm 0.118.0 passed 116 and failed the 6 new rows and 20bz. An earlier
    pass of the 122 rows before #1748 passed 122 on the 53dbf23 tarball and 116 on npm 0.118.0.
  • The consumer check: the 0.118.0 check's library and CLI consumer, plus 14 CLI lines for this
    release, on the release tarball from 53dbf23, before the merge of #1748. On that tarball the
    consumer typechecks with 0 errors, as on npm 0.118.0, and
    needs no migration. The library steps are the same on both versions, and of the 124 CLI lines
    from the 0.118.0 check, 121 are the same; the three that differ are admission.requests in
    analyze --json (row 6), and the 17-participant study check line and doctor --study note
    (rows 3 and 4). The new lines differ as rows 1 and 3 to 8 say.
  • pnpm bench on 53dbf23, before the merge of #1748, with its defaults (3 runs per arm,
    openai-computer-use, neutral mission), committed in cd8bf63: report recall 10/15, analysis
    recall 10/15, 0 invented on either arm, $6.12 estimated. 0.118.0 scored 6/15 and 6/15 at $5.38. D1
    and D5 account for the rise: two planted reports and two planted analyses name each, where none
    did on 0.118.0; D3 and D4 stayed 3/3. Two unresolved planted lines describe D5 (planted 1's
    report, planted 3's analysis), so recall read by hand is 11/15 and 11/15. #1749 found no harness
    regression behind the decline from 0.114.0 to 0.118.0: a same-hour A/B of npm 0.114.0 and main
    d183d1a scored 52/80 planted-report recall each over 16 planted participants (25/40 and 27/40 in
    its first round), and three runs per arm cannot separate this rise from the same noise. The
    analysis cap was the dry run's admittedCostUsd, $1.25, against bills of $0.60 to $0.86; none was
    refused or skipped. No planted participant typed more than 27 characters in one action, so none
    met D2. Every participant ran threaded, ended on its own and gave 5 or 6 impressions.

Not verified

  • macOS and Windows. Every check ran on Linux x64 with Node 24.12.
  • A live run on the published package: the live smoke ran on a main tarball, and the post-publish
    rows are offline dry runs, refusals and local runs.
  • A participant waiting for a free slot on a live run: at 100 sandboxes nobody waited, and the
    smoke's late starts came from the pipeline gate. A live shared world with a declared schedule,
    external-public followers on a schedule, and a rerun of a scheduled study.
  • A start hours after the run starts: the smoke's latest start was +3m 50s.
  • The default-cap refusal of a live run of more than 16 participants
    (AUTOMATIC_ANALYSIS_ADMISSION_REFUSED) and its exit codes; tests cover them.
  • How many requests the automatic analysis sent: the bundle does not record it, and its usage is
    consistent with 3. analyze --run printed its 3.
  • A run of 40 or 100 participants analysed live (3 or more cohorts), a failed cohort or merge
    request live, and a Codex account analysis of more than 16 participants.
  • npm 0.118.0 reading a cohort analysis past its stored bounds (2,000 evidence items, 64
    captures, 4 MiB).
  • A live run on an E2B Hobby key.
  • The dogfood run, the consumer check and the benchmark on the final head 558a295: they ran on
    53dbf23, before the merge of #1748. The live smoke apart from the #1748 row ran on main
    6405a7c's tarball.
  • #1748 on a live run whose participant instruction, as opposed to its mission, names the URL: the
    smoke ran that case as a dry run.