Skip to content

0.127.0: Clock-end passes counted by what participants declared, a scaled wind-down, the concurrency cap in the plan

Choose a tag to compare

@danielgwilson danielgwilson released this 13 Oct 06:15
· 150 commits to main since this release
bf3421d

humanish 0.127.0 makes long scheduled studies report what happened. A participant that passes at
its scheduled end or session cap without declaring REACHED THE GOAL. is now counted as running
until that end (participants.ranUntilEnd), where 0.126.0 counted it as reaching the goal. The
default wind-down is a quarter of the window, at most 10 minutes, so a short window is no longer
told its end from its first turn. HUMANISH_CUA_MAX_CONCURRENCY now lowers the plan itself, on
shared-world studies too, and a participant whose scheduled end passes while it waits for a slot
is skipped before its desktop is created. The Observer keeps updating past 100,000 items for a
participant, and the bundle estimate is re-derived for run bundle v3.

This is a minor release because four behaviors change:

# 0.126.0 0.127.0 What to do
1 A participant that passed at its scheduled end or session cap counted in the review's participants.reachedGoal and was described as reaching the goal, whatever its closing report declared. Only a declared REACHED THE GOAL. counts in reachedGoal. Others count in participants.ranUntilEnd (present when above 0), and the summary says 1 ran until its session cap or 1 ran until its scheduled end. Older bundles keep their counts when re-read. (#1816) A script that reads reachedGoal for clock-end studies also reads ranUntilEnd.
2 With no windDownMs, a participant was told the time left for the last 10 minutes of its window. The last quarter of its window, at most 10 minutes. (#1816) Set windDownMs: 600000 to keep 10 minutes on a short window.
3 HUMANISH_CUA_MAX_CONCURRENCY lowered only the run's participant table, and did not apply to shared-world. It lowers the plan, study check and the run, before the E2B plan's limit, on shared-world too; a value under 2 refuses a shared-world study. Unset it or set it to 2 or more for shared-world studies.
4 A participant with endAfterMs that waited for a slot past its end created a desktop, gave a closing report at its first turn and passed. It is skipped before its desktop is created and recorded blocked with the reason; the plan warns, naming it. Let more participants run at once, or give it a later end.

Changed

  • A computer-use participant with endAfterMs or maxSessionMs and no windDownMs is told the time left for the last quarter of its window, at most 10 minutes. The default was a fixed 10 minutes, so a window of 10 minutes or less was told its end from its first turn. A 10-minute cap is now told from 7 minutes 30 seconds in, and a window of 40 minutes or more keeps 10 minutes. An explicit windDownMs is used as written. study check's windows row and the plan entry's windDownMs show the new default. (#1816)
  • The participants[].files refusal messages suggest fixtures/photo.png and Downloads/photo.png, and the study-file docs examples use neutral roles (support agents, customers, applicants). (#1822)
  • HUMANISH_CUA_MAX_CONCURRENCY now lowers the plan itself. study check's schedule row, the plan's late-start warnings and the run's schedule: line use the lowered count; before, only the run's participant table did, so study check could describe slots the run did not have. The plan applies it before the E2B plan's limit, so an execution.concurrency above that limit is no longer refused when the setting lowers it. A lowering adds a plan warning, as the E2B plan's limit does; when that limit lowers the count further, the plan carries its warning instead. The setting now applies to shared-world studies too, and a value under 2 refuses one, since a shared world runs at least two participants at once. (#1826)

Fixes

  • A served Observer page keeps updating once a participant passes 100,000 items. Past that count it refused every feed body and kept the last one it accepted, through the end of the run, so a later generation of a live file that scrubbed a sandbox id the run learned late never reached the page. A page opened after that count showed no run at all. The page no longer refuses a body for the length of a list. The server still reads a participant's trace or live file only up to 32 MiB; past that the page shows the participant's trace as unavailable, with a warning. (#1806)
  • A computer-use participant with endAfterMs that waits for a free slot until its end has passed is skipped before its desktop is created, and its record is blocked with the reason. Before, it created a desktop when a slot freed, was asked for its closing report at its first turn and passed. The plan gives such a participant no slot when it works out the schedule, so the participants after it start sooner in the plan, and it warns, naming each one it would skip. (#1826)
  • The bundle estimate that study check, humanish run and doctor --study print is re-derived for run bundle v3 from live and dry runs measured on 0.125.0. It checks run.json at about 225 bytes a model turn against the 16 MiB analysis reads, and each participant's trace against the same limit at 4.7 KB a turn, the busiest participant measured; the average was about 2 KB. It checks the Observer page against the 32 MiB verify reads, as the app, half of run.json and 0.12 of all the traces' bytes, and the file count against the public-safety scan. It no longer checks observer-data.json, which stays within 7 % of run.json. Five participants working 8 hours beside 25 short ones were estimated at a 30.2 MiB run.json and warned about; they are now estimated at 3.7 MiB. The warning starts at 17 participants working 8 hours for the Observer page, 30 for run.json, and a 10-hour window for one participant's trace. The bundle size row names the largest trace and the Observer page in place of the largest file. (#1819)
  • A computer-use participant that passes at a scheduled end or a session cap without declaring REACHED THE GOAL. is no longer counted or described as reaching the goal. Its closing report said DID NOT REACH THE GOAL. or declared nothing. The review counts it in a new participants.ranUntilEnd, present when above 0, and its summary says 1 ran until its session cap or 1 ran until its scheduled end. The participant's own summary says the same, and the Observer labels it Ran until session cap or Ran until scheduled end. It still passes. A participant that declared REACHED THE GOAL. still counts in reachedGoal. An older bundle keeps its reachedGoal count when re-read, and its completion line names the clock end. Analysis input still records goalSource: participant_report for such a participant, as 0.125.0 did, so an analysis saved then stays valid; its stopCause and declaredOutcome name the clock end. (#1816)
  • The declared outcome skips a > blockquote line. A closing report or last message whose first line quotes a page's REACHED THE GOAL. no longer declares it; the blocker scan already dropped blockquote lines. (#1816)

Checked before release

  • CI on this PR (sharded gate on both Node versions, site, Observer proofs, TUI, guest desktop, secret scan) is the gate of record. pnpm gate on the release commit passed.
  • The post-publish check rehearsed on the release tarball (pnpm packed:check): all 125 rows passed. The registry and unpkg rows run after publish.
  • pnpm release:dogfood: a participant installed the tarball, ran the offline first-run study (share_ready, 16 of 16 checks, $0) and stopped at the credentials the gate withholds, which counts as passing.
  • Live on the release tarball (hosted E2B, Chrome, keys from the user store), about $0.66 estimated across three runs:
    • try-live with analysis: finished, analysis complete with 3 findings and 3 design findings, verify 16 of 16 (local_only, raw screenshots).
    • HUMANISH_CUA_MAX_CONCURRENCY=1 with two participants on example.com. study check printed the plan's warnings that late would be skipped and that the 2 participants run 1 at a time, and its windows row says steady is told the time left from 1m 30s into its 2m cap. In the run, steady worked until its cap, declared REACHED THE GOAL. in its closing report and passed with stopCause: session_cap. late (endAfterMs: 60000) was skipped before any desktop was created: run.json records it blocked with the reason, and the run's only sandbox receipt is steady's. The run failed as 1 of 2 passed, as a skipped participant does. verify 16 of 16.
    • One participant held to endAfterMs: 120000 that read the page early and watched until its end: passed with stopCause: scheduled_end. Its closing report declared no outcome, so the review counted it in ranUntilEnd and said "1 ran until its scheduled end", with reachedGoal: 0. verify 16 of 16.
  • Each change had a Claude review; the Observer change had a Codex review with no findings. Every real finding is fixed with a test or listed below.
  • The benchmark was not run: no change alters what a participant is asked or what the analysis reads.

Not verified

  • The closing report at a clock end does not ask an OpenAI participant for its declared line, so a participant that reached its goal often lands in ranUntilEnd (#1827).
  • A skipped participant's run line says "unknown ending" and the summary counts it as taking part (#1828).
  • A live run past 100,000 items for one participant; analysis still refuses a run past 100,000 events or items (#1824).
  • HUMANISH_CUA_MAX_CONCURRENCY on a live shared-world run.
  • macOS and Windows.

Package

humanish@0.127.0: 1004 files, shasum cc9212c965f6e4a4711fc57396288dbb6ea77807.