Repository navigation
0.149.0: Participants can run on Claude Haiku 5.5
humanish 0.149.0 picks a computer-use participant's vendor from its model: with actor.type: computer-use, model: claude-haiku-5-5 runs on Anthropic and the gpt-* models on OpenAI, on
E2B, Modal and local VM desktops, in local and remote runs. Claude gets its whole computer
toolset, so the shared desktop translator gains triple click, button presses, held keys, pointer
reads and zoom, and local-agent participants get them too. A zoom's image is kept in the run
bundle. A Claude participant needs ANTHROPIC_API_KEY (humanish keys set anthropic), which a
remote E2B run keeps in E2B's egress proxy. openai-computer-use studies run as before.
Added
actor.type: computer-useruns each participant on the vendor of itsactor.model:
claude-haiku-5-5on Anthropic withANTHROPIC_API_KEY, thegpt-*models on OpenAI with
OPENAI_API_KEY, andgpt-6.1-solwhen the study names no model. It runs on E2B and Modal
desktops, in local and remote runs, and in a local browser.actor.providernames the vendor of
a model humanish does not list yet; that model runs unpriced, so a spend cap is refused.- A
computer-usestudy is refused when it is read, before a run id exists, for a model
humanish does not list and noactor.provider, anactor.providerthat disagrees with the
listed model, and a Claude participant's desktop over 2,000 px a side or 4,784 visual tokens. humanish keyslistsANTHROPIC_API_KEY,doctor --studyreports it for a study whose
participants run on Anthropic, and a live run without it is refused before it starts with
humanish keys set anthropic. A remote run on E2B keeps it in E2B's egress proxy, added to
requests forapi.anthropic.comasx-api-key; a Modal runner receives it as an environment
variable.actor.type: local-agentparticipants can propose every desktop action, including the six
above. Codex'shumanish_uitool schema and Claude Code's reply shape both come from one
schema. Aread_pointerposition comes back as text: in Codex's acknowledgment, and in Claude
Code's next message. Azoomcrop comes back as an image enlarged to fit the screenshot, as
for Claude through the Anthropic provider: in Codex's tool result aszoomImages, and to Claude
Code as an image file in its folder, which it opens with Read.- A computer-use participant's zooms and pointer reads are run evidence. The image a zoom gave the
participant is kept asscreenshots/zoom-NNNbeside the turn screenshots, at the desktop's own
pixels, and referenced from the zoom'sui_actionwith the zoomedregion. It follows every
screenshot rule: blurred underpolicies.redactScreenshots, lossless WebP otherwise, listed in
the stream's screenshots and checked byhumanish verify. A pointer read'sui_actionrecords
pointer, the position it read. The Observer shows a zoom's image on its activity row, and
analysis offers it as a capture the analyst can cite. claude-haiku-5-5has a rate: $0.10 input, $0.01 cache read, $0.125 cache write and $0.50
output per 1M tokens, 5x each for a request over 100,000 prompt tokens.createAnthropicMessagesProviderbuilds a computer-use participant on Claude through the
Anthropic Messages API and Claude'scomputer_toolset_20260801, onclaude-haiku-5-5at effort
mediumwith a 16,000-token output limit by default. A library caller passes it to
runComputerUseLoop, and a study reaches it withactor.type: computer-use(below). All 17 of Claude's computer actions run,
zoomandcursor_positionanswered from what the desktop read, and a desktop over 2,000 px a
side is refused.ComputerUseActionhas six more desktop actions, run on E2B, Modal and local VM desktops through
the shared X11 translator:triple_click,mouse_downandmouse_up(a button, at the pointer
or a point),hold_keys(a chord held for up to 30 seconds),read_pointer, andzoom(a fresh
capture cropped to a region at the desktop's own pixels). The OpenAI provider never proposes
them, so its requests are unchanged.- Keypresses and held keys accept the X keysym names a model writing xdotool syntax sends:
Prior,Next,KP_Enter,KP_0toKP_9, the keypad operators,ISO_Left_Tab, the sided
modifiers (Control_L,Shift_Rand the rest) and punctuation keysym names such asminus. The
keypad's navigation names (KP_Home,KP_Page_Downand the rest) press the main keys. Before,
each of these was rejected before dispatch. - A library
ComputerUseProvidercan setbatchFailure: "halt": after a desktop command fails,
the loop sends none of that batch's later actions and acknowledges each asnot_dispatched. - A
ComputerUseTurncan carryrejectedProposal: { reason }for a proposal the provider could
not map to an action. The loop runs the turn's actions, acknowledges the proposal's position as
not_dispatched, gives the participant the reason and continues the session. - A
ComputerUseTurncan reportinterruption: "refused"with an optionalrefusalCategory. The
session stops with stop causeprovider_refused_prompt, keeps the reply's usage and records the
category in its notice. - A trace's
conversation.explicitReasoncan bestateless_api, for a provider whose API keeps
no conversation between requests. Such a run gives no zero-data-retention warning. ComputerUseExecutor.executemay resolve with a result,{ kind: "pointer", x, y }for
read_pointeror{ kind: "frame", screenshot }forzoom. The loop passes it to the provider
in that action's acknowledgement (previousExecution.actions[i].result). An executor that
returns nothing is unchanged.- A
ComputerUseProvidercan setrequireReportedUsageForSpendCap: true. Every session with a
declared cap (maxUsdor a study budget) then runs as with the loop option of that name: a
request that stalls or fails without a reply stops the session withstopCause: "usage_unreported"instead of being booked at a worst case and sent again. The Anthropic
provider sets it, since that worst case cannot bound a request priced by prompt-size tiers. - A
ComputerUseTurncan carrynotices, which the trace records as warnings. A trace's
conversation.requests[]can carryclearedInputTokens, the input the provider's server-side
context editing cleared.
Changed
- A Claude Code participant's reply uses the same action fields as Codex's tool: a scroll gives
dxanddy, and a field may be omitted or null when unused. A proposal that does not match
its action's fields is refused before any input, with the reason in the participant's next
message, and the actions before it run. Before, it was dropped without a word. - Every
ComputerUseProvidernow receivespreviousExecutionon each request after a turn that
acted, with one acknowledgement per batch position. Before, only providers with
requestPolicy: "fail_closed"received it. The OpenAI provider ignores it, so its requests are
unchanged. - An OpenAI participant whose reply body is lost while it is read, after the request reached
OpenAI, now has its usage marked incomplete and the request handled as lost in transit, as a
dropped connection already was. Before, that request's usage went missing without a mark. - The default provider fetch passes response headers through, so a live OpenAI participant
waits for a 429 or 5xx reply'sRetry-Afterhint, up to 60 s, before it retries. - E2B, Modal and local VM desktops translate computer-use actions through one shared X11
translator (src/substrates/x11/). Each sends the same native commands as before. The exported
E2BDesktopSandboxtype no longer declares the SDK's mouse and keyboard methods (leftClick,
moveMouse,scroll,write,press,dragand the other clicks), since humanish sends input
as xdotool commands; code that called them through this type should call the@e2b/desktop
Sandbox directly. - A participant is given a zoom's image enlarged to fit its screenshot, with the aspect ratio kept
(bilinear), as Claude's computer-use toolset documents forzoom. This covers Claude through
the Anthropic provider and Codex and Claude Code on the local-agent route. Before, each got the
crop at the desktop's own pixels, which Claude Haiku 5.5 found too small to read, and local
agents enlarged the page with Ctrl+= instead. The run bundle keeps the crop at desktop pixels. - Analysis instructions say a zoom capture is the participant's closer look at a region and not a
full screen. The added paragraph raises each analysis request's input by about 145 tokens. openai-computer-userefuses a model the participant model table puts on another vendor, such
asclaude-haiku-5-5, and namesactor.type: computer-use. Before, a capped study naming it
passed every check, since the model has a rate, and failed on its first request. Any other model
id still goes to OpenAI.- An effort level a listed participant model does not accept is refused when the study is read,
foractor.reasoningEffortand for eachparticipants[].reasoningEffort, on
openai-computer-usetoo:noneon the defaultgpt-6.1-solno longer starts a run that fails
on its first turn. A model humanish does not list is checked against its vendor's levels only. actor.maxOutputTokensandactor.serviceTierapply tocomputer-useas to
openai-computer-use.serviceTier: faststill needs a Fast mode rate, which only OpenAI
models have.
Fixes
- An OpenAI participant whose reply could not be written to
HUMANISH_CUA_WIRE_CAPTURE_DIRkeeps
that reply's usage, for participant turns and closing requests. The trace records a "wire capture
not written" warning. Before, the write error was raised as a provider failure and the billed
reply's usage went unrecorded.