Skip to content

0.149.0: Participants can run on Claude Haiku 5.5

Choose a tag to compare

@danielgwilson danielgwilson released this 26 Oct 10:18
· 26 commits to main since this release
f15eaa4

humanish 0.149.0 picks a computer-use participant's vendor from its model: with actor.type: computer-use, model: claude-haiku-5-5 runs on Anthropic and the gpt-* models on OpenAI, on
E2B, Modal and local VM desktops, in local and remote runs. Claude gets its whole computer
toolset, so the shared desktop translator gains triple click, button presses, held keys, pointer
reads and zoom, and local-agent participants get them too. A zoom's image is kept in the run
bundle. A Claude participant needs ANTHROPIC_API_KEY (humanish keys set anthropic), which a
remote E2B run keeps in E2B's egress proxy. openai-computer-use studies run as before.

Added

  • actor.type: computer-use runs each participant on the vendor of its actor.model:
    claude-haiku-5-5 on Anthropic with ANTHROPIC_API_KEY, the gpt-* models on OpenAI with
    OPENAI_API_KEY, and gpt-6.1-sol when the study names no model. It runs on E2B and Modal
    desktops, in local and remote runs, and in a local browser. actor.provider names the vendor of
    a model humanish does not list yet; that model runs unpriced, so a spend cap is refused.
  • A computer-use study is refused when it is read, before a run id exists, for a model
    humanish does not list and no actor.provider, an actor.provider that disagrees with the
    listed model, and a Claude participant's desktop over 2,000 px a side or 4,784 visual tokens.
  • humanish keys lists ANTHROPIC_API_KEY, doctor --study reports it for a study whose
    participants run on Anthropic, and a live run without it is refused before it starts with
    humanish keys set anthropic. A remote run on E2B keeps it in E2B's egress proxy, added to
    requests for api.anthropic.com as x-api-key; a Modal runner receives it as an environment
    variable.
  • actor.type: local-agent participants can propose every desktop action, including the six
    above. Codex's humanish_ui tool schema and Claude Code's reply shape both come from one
    schema. A read_pointer position comes back as text: in Codex's acknowledgment, and in Claude
    Code's next message. A zoom crop comes back as an image enlarged to fit the screenshot, as
    for Claude through the Anthropic provider: in Codex's tool result as zoomImages, and to Claude
    Code as an image file in its folder, which it opens with Read.
  • A computer-use participant's zooms and pointer reads are run evidence. The image a zoom gave the
    participant is kept as screenshots/zoom-NNN beside the turn screenshots, at the desktop's own
    pixels, and referenced from the zoom's ui_action with the zoomed region. It follows every
    screenshot rule: blurred under policies.redactScreenshots, lossless WebP otherwise, listed in
    the stream's screenshots and checked by humanish verify. A pointer read's ui_action records
    pointer, the position it read. The Observer shows a zoom's image on its activity row, and
    analysis offers it as a capture the analyst can cite.
  • claude-haiku-5-5 has a rate: $0.10 input, $0.01 cache read, $0.125 cache write and $0.50
    output per 1M tokens, 5x each for a request over 100,000 prompt tokens.
  • createAnthropicMessagesProvider builds a computer-use participant on Claude through the
    Anthropic Messages API and Claude's computer_toolset_20260801, on claude-haiku-5-5 at effort
    medium with a 16,000-token output limit by default. A library caller passes it to
    runComputerUseLoop, and a study reaches it with actor.type: computer-use (below). All 17 of Claude's computer actions run,
    zoom and cursor_position answered from what the desktop read, and a desktop over 2,000 px a
    side is refused.
  • ComputerUseAction has six more desktop actions, run on E2B, Modal and local VM desktops through
    the shared X11 translator: triple_click, mouse_down and mouse_up (a button, at the pointer
    or a point), hold_keys (a chord held for up to 30 seconds), read_pointer, and zoom (a fresh
    capture cropped to a region at the desktop's own pixels). The OpenAI provider never proposes
    them, so its requests are unchanged.
  • Keypresses and held keys accept the X keysym names a model writing xdotool syntax sends:
    Prior, Next, KP_Enter, KP_0 to KP_9, the keypad operators, ISO_Left_Tab, the sided
    modifiers (Control_L, Shift_R and the rest) and punctuation keysym names such as minus. The
    keypad's navigation names (KP_Home, KP_Page_Down and the rest) press the main keys. Before,
    each of these was rejected before dispatch.
  • A library ComputerUseProvider can set batchFailure: "halt": after a desktop command fails,
    the loop sends none of that batch's later actions and acknowledges each as not_dispatched.
  • A ComputerUseTurn can carry rejectedProposal: { reason } for a proposal the provider could
    not map to an action. The loop runs the turn's actions, acknowledges the proposal's position as
    not_dispatched, gives the participant the reason and continues the session.
  • A ComputerUseTurn can report interruption: "refused" with an optional refusalCategory. The
    session stops with stop cause provider_refused_prompt, keeps the reply's usage and records the
    category in its notice.
  • A trace's conversation.explicitReason can be stateless_api, for a provider whose API keeps
    no conversation between requests. Such a run gives no zero-data-retention warning.
  • ComputerUseExecutor.execute may resolve with a result, { kind: "pointer", x, y } for
    read_pointer or { kind: "frame", screenshot } for zoom. The loop passes it to the provider
    in that action's acknowledgement (previousExecution.actions[i].result). An executor that
    returns nothing is unchanged.
  • A ComputerUseProvider can set requireReportedUsageForSpendCap: true. Every session with a
    declared cap (maxUsd or a study budget) then runs as with the loop option of that name: a
    request that stalls or fails without a reply stops the session with stopCause: "usage_unreported" instead of being booked at a worst case and sent again. The Anthropic
    provider sets it, since that worst case cannot bound a request priced by prompt-size tiers.
  • A ComputerUseTurn can carry notices, which the trace records as warnings. A trace's
    conversation.requests[] can carry clearedInputTokens, the input the provider's server-side
    context editing cleared.

Changed

  • A Claude Code participant's reply uses the same action fields as Codex's tool: a scroll gives
    dx and dy, and a field may be omitted or null when unused. A proposal that does not match
    its action's fields is refused before any input, with the reason in the participant's next
    message, and the actions before it run. Before, it was dropped without a word.
  • Every ComputerUseProvider now receives previousExecution on each request after a turn that
    acted, with one acknowledgement per batch position. Before, only providers with
    requestPolicy: "fail_closed" received it. The OpenAI provider ignores it, so its requests are
    unchanged.
  • An OpenAI participant whose reply body is lost while it is read, after the request reached
    OpenAI, now has its usage marked incomplete and the request handled as lost in transit, as a
    dropped connection already was. Before, that request's usage went missing without a mark.
  • The default provider fetch passes response headers through, so a live OpenAI participant
    waits for a 429 or 5xx reply's Retry-After hint, up to 60 s, before it retries.
  • E2B, Modal and local VM desktops translate computer-use actions through one shared X11
    translator (src/substrates/x11/). Each sends the same native commands as before. The exported
    E2BDesktopSandbox type no longer declares the SDK's mouse and keyboard methods (leftClick,
    moveMouse, scroll, write, press, drag and the other clicks), since humanish sends input
    as xdotool commands; code that called them through this type should call the @e2b/desktop
    Sandbox directly.
  • A participant is given a zoom's image enlarged to fit its screenshot, with the aspect ratio kept
    (bilinear), as Claude's computer-use toolset documents for zoom. This covers Claude through
    the Anthropic provider and Codex and Claude Code on the local-agent route. Before, each got the
    crop at the desktop's own pixels, which Claude Haiku 5.5 found too small to read, and local
    agents enlarged the page with Ctrl+= instead. The run bundle keeps the crop at desktop pixels.
  • Analysis instructions say a zoom capture is the participant's closer look at a region and not a
    full screen. The added paragraph raises each analysis request's input by about 145 tokens.
  • openai-computer-use refuses a model the participant model table puts on another vendor, such
    as claude-haiku-5-5, and names actor.type: computer-use. Before, a capped study naming it
    passed every check, since the model has a rate, and failed on its first request. Any other model
    id still goes to OpenAI.
  • An effort level a listed participant model does not accept is refused when the study is read,
    for actor.reasoningEffort and for each participants[].reasoningEffort, on
    openai-computer-use too: none on the default gpt-6.1-sol no longer starts a run that fails
    on its first turn. A model humanish does not list is checked against its vendor's levels only.
  • actor.maxOutputTokens and actor.serviceTier apply to computer-use as to
    openai-computer-use. serviceTier: fast still needs a Fast mode rate, which only OpenAI
    models have.

Fixes

  • An OpenAI participant whose reply could not be written to HUMANISH_CUA_WIRE_CAPTURE_DIR keeps
    that reply's usage, for participant turns and closing requests. The trace records a "wire capture
    not written" warning. Before, the write error was raised as a provider failure and the billed
    reply's usage went unrecorded.