Skip to content

0.144.0: Hosted studies run remotely by default

Choose a tag to compare

@danielgwilson danielgwilson released this 22 Oct 04:42
· 85 commits to main since this release
7b0b988

humanish 0.144.0 runs a hosted computer-use study remotely by default. A study on
e2b-desktop or modal-desktop with a public app-url runs its own process in a sandbox in your
E2B or Modal account, so it keeps going when this machine sleeps; the command follows it, serves
the live Observer here and copies the evidence back. Before the runner starts, the command names
the keys it sends. Pass --local for one run, or set execution.remote: false in a study, to keep
it on this machine.

Changed

  • A computer-use study on e2b-desktop or modal-desktop with a model API key and a public
    app-url subject now runs remotely by default, as --remote did: its own process runs in a
    sandbox on that provider, so it keeps going when this machine sleeps, and the command follows it,
    serves its live Observer here and copies it back. The command first names the keys the runner
    receives. To keep the study on this machine, pass --local for one run, or set
    execution.remote: false in the study for every run. --remote and execution.remote: true
    now require a remote run and refuse a study that cannot have one, with the reason. Any other
    hosted computer-use study still runs here and the command says why. A study this machine would
    refuse, such as one missing a key, is refused here before any runner exists. The run's JSON
    result names the provider and the forwarded keys under remote. When the terminal closes (SIGHUP)
    the run goes on remotely and the command exits 129 with HUMANISH_REMOTE_DETACHED, naming how to
    follow or stop it; a local run on SIGHUP still stops and reclaims. Dry runs, studies with email
    comms, terminal and shared-world studies, runs from a source checkout and runs started from the
    TUI stay on this machine.

Live checks

Before the default changed, --remote was run live on the published 0.142.0, and every check left 0 sandboxes on both providers:

  • Runs of more than an hour, two participants 65 minutes apart with default analysis: 69.5 minutes on Modal and 70 on E2B. Each was detached and then reconnected with observe --run, and passed with verify 16/16.
  • 20 Modal participants with recordings: passed in 139 s, with 57 MB copied back.
  • 10 E2B participants with --json: passed in 125 s.
  • SIGTERM: exit 143 after 8 s, sandboxes gone.

Not covered: terminal studies (they stay local), multi-GB bundles, recordings near the 128 MB cap, more than 20 participants.

A runaway runner during testing

While the new default was tested before release, a runner treated its own run as one to send remotely and started another runner, which did the same. 25 Modal runners started in about three minutes. None started a participant sandbox or made a model call; the estimated cost was at most $0.30, and Modal's bill for that hour was $0.305. All 25 were stopped, and each was confirmed by id. A runner now passes --local to its run and sets HUMANISH_INSIDE_RUNNER=1, so a run inside a runner always stays there; tests pin both. Nothing from that build reached a release.