Skip to content

0.121.0: One failed desktop fails alone, host sleep named as the cause

Choose a tag to compare

@danielgwilson danielgwilson released this 10 Oct 22:47
· 174 commits to main since this release
66acc55

humanish 0.121.0 keeps one participant's failed desktop from costing a whole hosted study, and
names a host that slept mid-run as the likely cause of the failures after it. A computer-use study
on an app-url subject no longer holds its participants behind the first one's desktop, and a
participant that fails before its session no longer stops the participants queued behind it; a
study whose subject is built in each sandbox still lets one participant set it up first, and the
next one takes over when the first never got a desktop. When the machine running a study sleeps,
the run records the suspension as a host.suspended event, and the run outcome, humanish run,
the review and the --json result say the failures of participants running through it are
likely its effect, while each participant's record keeps its own error. The review's participant
line lists interruption causes in a fixed order.

This is a minor release because outputs change for the same input:

# 0.120.1 0.121.0 What to do
1 In a hosted computer-use study, a participant whose desktop failed to start (or any participant that failed before its session) recorded every other participant blocked with "failed to provision its world (pipeline gate)" or a fail-fast skip. On an app-url subject, every other participant runs and the run reports one failure. On clone, local-tree and desktop-cli subjects, the next participant takes the first start when the first never got a desktop. A run where every start fails for one reason (a bad key, a quota) now reports each participant's failure where it reported one failure and the rest skipped. (#1759) Count failures by their codes; expect more failures and fewer skips.
2 A run whose host slept reported each participant's failure as its own. A host.suspended warn event; the run outcome, humanish run, the review and --json (hostSuspension) name the suspension as the likely cause of the failures of participants running through it. (#1760) Read hostSuspension before treating a burst of failures as product findings.
3 The review's participant line listed interruption causes in finish order. Causes are listed most first, then by name. (#1759) Do not match the order of causes in that line.

Fixes

  • The participant line of a run's review lists interruption causes in a fixed order, most first
    and then by name. It followed the order the participants finished in, so the same run could
    read differently (#1741).
  • A hosted computer-use study runs its other participants when one participant's desktop fails to
    start. The first participant held a gate on every subject, so when its desktop failed, every
    other participant was recorded blocked with "failed to provision its world (pipeline gate)",
    and an 11-participant app-url study whose first sandbox failed reported 10 skipped
    participants. An app-url participant only opens the app on its own desktop, so its startup
    failure is now its own and no gate holds the others back. A participant that fails before its
    session also no longer trips fail-fast, which skipped every participant queued behind a full set
    of slots or due later in a schedule. A clone, local-tree or desktop-cli subject, which
    every participant sets up the same way, keeps the gate: any participant that becomes ready
    releases it, a first participant whose sandbox was never created passes the first start to the
    next one, and a first participant whose desktop came up and then failed, as when the subject
    does not build, still blocks the rest (#1741).
  • A run whose host sleeps, such as a laptop closed on battery, says so. The run's status cadence
    is now also a heartbeat: a wall-clock gap of 30 s or more between two ticks is recorded as a
    host.suspended warn event with its start, end and length. On a computer-use run, the
    participants that failed in the harness or at their time limit while in flight across the
    suspension, and those skipped after such a failure, are named as its likely effect: the run
    outcome's first failure, the start of the review summary and a line under the humanish run
    header read "The host was suspended for 6m 5s at +1m; the 34 participant failures after it are
    likely its effect.", the review lists those participants in one gap line, and the result's
    hostSuspension holds the ids. A fan-out run's header then names the
    --rerun-failed-from command. A participant that stopped before the suspension, including one
    whose desktop failed to start, keeps its own failure. Each participant's own record and failure
    are unchanged. Before,
    a 6-minute sleep a minute into a 34-participant run read as 117 provider stalls, E2B deadlines
    and skips, none of which named the host (#1740).

Checked before release

  • pnpm release:check on the release commit: exit 0; 8,062 tests passed, 12 skipped; TUI 192; lint 426 at its cap; pack 974 files, shasum f4e9dc7f88930a9c7b7a1a132207b8056b03f08e.
  • #1759: 6 route tests through runStudyWith on fake desktops (an 11-participant app-url study whose first desktop fails runs the other 10; clone cases), two of which failed before the change; the fan-out suite passed 4 times in a row pinned to 2 cores after CI exposed two ordering assumptions.
  • #1760: 8 module tests and 4 route tests with an injected clock (a 6m 5s jump a minute in; a participant whose desktop failed before the jump keeps its own failure).
  • Live on the release tarball (keys from the user store, HUMANISH_E2B_MAX_CONCURRENT_SANDBOXES=100): a 4-participant hosted computer-use study on an app-url subject started all four desktops at once (Chrome DevTools answered at 4.7 to 10.6 s), all four passed, the automatic analysis completed, exit 0, verify 16 checks (local_only, raw screenshots). Run cost estimated $0.10 plus the analysis.

Not verified

  • A real host suspension on macOS or Linux. The clock behavior comes from reading the Node and libuv source; the 30 s threshold has no load measurement behind it.
  • A live hosted study whose first desktop fails to start; the tests stand in for it.
  • The benchmark was not run: neither change touches what a participant sees or what the analysis reads. The 0.120.0 benchmark stands.
  • macOS and Windows.

Package

humanish@0.121.0: 974 files.