Orca v0.5.11

Orca v0.5.11

Changes since v0.5.10:

• On macOS 14 or later, the interactive TUI's main session can see and
  operate other apps with the new computer tool: list and launch apps,
  observe a window's screenshot and accessibility elements, click, type, send
  keys and scroll. Each app needs your approval, entering text into a password
  field asks every time, and Esc stops the turn from anywhere.
• Computer use works in the background where it can. Otherwise it switches to
  the app briefly, restores the previous app and pointer, and says so. Command
  shortcuts and typing into sheets such as Save panels always take that path.
  [computer_use] foreground_fallback = false turns the switch off.
• launch_app finds an app by its English or display name even when it is
  not running (TextEdit on a Chinese system), and opens it without taking
  focus from the TUI.
• Live conversations keep only the newest three tool images. Tool image
  types are detected from file headers, and resuming a session does not load
  tool image data into the model's conversation.
• The Terminal-Bench adapter keeps the task instruction out of command lines,
  so a task's pkill -f cannot match that text and end the trial (#126). The
  evaluation probes and runbook were updated after the v0.5.10 round
  (#127–#129).

Changes

Computer use on macOS

The computer tool can list and launch apps, inspect windows and their
accessibility elements, click, type, send keys, and scroll. Use it to check a
UI or work in an app without a command-line interface. It requires macOS 14
or later and Accessibility and Screen Recording permissions for the terminal
running Orca. Input Monitoring is also needed if the driver reports missing
listening access when installing the Esc listener.

App approvals offer Allow once, Allow this app this session,
Always allow this app (saved), and Deny. The saved option writes a
computer allow rule with the app's exact bundle id as its pattern to the
user configuration. Password text entry asks separately each time, with only
Allow typing this once or Deny. This covers type and text-entering
key actions: printable keys alone or with Shift/Option, and Cmd+V (also
with Shift/Option). App grants and patterned allow rules, even one naming an
explicit #secure target, do not preapprove these calls. Only a whole-tool
allow rule for computer without a pattern does. Password managers in the
built-in list and apps declaring the finance category are blocked by default
unless an exact bundle-id allow
rule lifts the block; the host terminal remains blocked regardless of rules.
Listed terminals, IDEs, Finder, System Settings, and browsers carry risk
warnings. Deny rules take precedence over allow rules.

If password focus is detected only during a point action or from a stale
observation, Orca refuses unapproved input and sends no text. The result
instructs the agent to observe again and target the secure field by its element id
to request approval on retry. A call that resumes after its approval runs only
if it still targets the app and field that were approved; otherwise it is
refused and asks again.

type inserts literal text into the focused field and never moves focus. A
tab, newline or return is inserted only into a multi-line text area. For any
other field the call is refused and the agent is told to use key with
["tab"] or ["return"]. Other control characters and the macOS function-key
characters are refused as well. Typing that needs the foreground is refused
before anything is typed when the text cannot fit in the action's time budget,
so long text is split into several calls rather than typed in part. Fields an
action does not take, such as modifiers on key, are refused with a message
that shows the correct call.

For computer, plan mode permits only list_apps and observe.
list_apps needs no app approval; observe and app calls in full-auto
still require it unless an applicable rule or app grant already allows them.

Orca operates in the background where possible. launch_app starts apps or
opens URLs/files without requesting activation; an app may still choose to
activate itself. An app may be named by its bundle id, its English or display
name, or its .app folder name, also when it is not running. The approval and
the launch both use the resolved bundle id, so a name cannot reach a different
app than the one approved. Keyboard input into a sheet, such as a Save or Open
panel, or into a field another process owns, always uses the foreground, where
it reaches the focused field. Actions that need the foreground briefly switch to the
target app and restore the previous app and pointer, and the result reports
foreground use. A key containing Command
always takes this path: background Command shortcuts did not reach the app
in the real-Mac checks. [computer_use] foreground_fallback defaults to
true; setting it to false in the user configuration refuses those actions.
Project configuration cannot set it. From the first computer call until
the turn ends, Orca captures Esc system-wide to interrupt the running turn,
including when another app is in the foreground or a screenshot preview is
open.

Click the [N image(s) attached] line under a tool result to preview its
screenshot, and press q to close the preview. A truncated observation
means the accessibility tree reached its time limit and is partial; use
query to narrow a new observation or point targets from the latest
screenshot, then check the action result.

Computer use is limited to the interactive TUI's main session. Dragging,
direct field-value assignment, subagents, workflow children, ACP, and
orca exec are not supported. While a remote-control app such as UU Remote
draws its overlay over the screen, actions that need the foreground are
refused as covered. A whole-tool deny rule removes computer from
the model's available tools. The English and Chinese site docs describe
setup, approvals, risk levels, foreground switching, disabling the tool, and
privacy; the configuration pages and README link to them.

Tool images

• The live conversation retains data for only the newest three tool images,
  matching the request limit. Older observations keep their resource
  references and a note that newer observations superseded them; the TUI
  loads older previews from their resource files.
• PNG, JPEG, GIF, and WebP are detected from their file headers, overriding a
  mismatched declared type. Unsupported image data is dropped with a note.
• Deep compaction counts the tool images the conversation retains; superseded
  images beyond the newest three no longer count.
• Resuming a session keeps tool-image references and text without reading
  tool image data back into the model's conversation.
• Screenshots are sent to the provider with deepseek-flash (including its
  legacy names deepseek-v4-flash and deepseek-v4-flash-vision-exp) or auto.
  Other model selections get text and a note in place of tool images, even
  when a custom model supports user-attached images. Tool images are not sent
  to the separate vision model used to describe user attachments.

Terminal-Bench adapter

The adapters pass the task instruction through the ORCA_INSTRUCTION
environment variable, then copy and unset it before a shell builtin sends it
to Orca's stdin. Previously it was still present in the shell's command line:
a task's pkill -f could match a path in that instruction and end the trial.
Neither Orca nor its task commands inherit the variable. This fixes #126,
merged to main on October 12, 2026.

Evaluation

• docs/reports/2026-10-11-v0.5.10-evaluation.md records the v0.5.10 round.
  L1 was green in 25 of 26 runs. On Terminal-Bench 2.0 with deepseek-flash,
  the mean was 0.7273 (64/88), or 0.8068 (71/88) after the infrastructure
  re-runs; this is the first deepseek-flash baseline.
• scripts/eval/linux_shell_probe.py checks its two contracts separately. The
  shell check lends both images a CA bundle. The CA check requires a bare image
  to fail with the documented No CA certificates error without sending a
  request (#127).
• exit_code_probe.py and mcp_probe.py no longer attribute their failures to
  the unrelated, closed #85 (#128).
• scripts/eval/reap_harbor.sh removes the Compose projects that Harbor
  abandons on environment-start timeouts, including their networks. The
  evaluation runbook runs it before and after each round (#129).

Compatibility

• Configurations and sessions from v0.5.10 remain readable. The new
  foreground_fallback setting defaults to true when absent.
• Computer use is available on macOS 14 or later in the interactive TUI's
  main session. Linux, Windows, ACP, and headless execution do not gain the
  tool.
• Older tool screenshots remain available for TUI preview from stored
  resource files, but are not reloaded into resumed model conversations.

Verification

On the release commit, macOS 26.6.2 on Apple Silicon:

• Static checks and validators: all pass.
  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --locked -j 1: exit 0; no warning on lines this release adds.
  • The runtime-surface and Windows platform-boundary validators, and their tests.
  • scripts/release/verify-version-sync.mjs (0.5.11) and the version-sync, npm staging and publish-verification script tests.
  • npm --prefix site run build, npm --prefix site run check:seo and git diff --check.
• TUI library: cargo nextest run -p orca-tui --lib --locked --profile ci-serial --retries 0: 1813 passed.
• Workspace: cargo nextest run --workspace --all-targets --locked --profile ci --no-fail-fast --retries 0: 4309 of 4312 passed.
  • The 3 failures are MCP-startup timing tests.
  • They fail only when the test binary sits in this checkout under ~/Documents. The same binary copied to /tmp passes all three, and CI runs them green.
• Contracts: cargo test --test public_docs_contract --locked: 11 passed. python3 -m unittest discover -s terminal_bench -p 'test_*.py': 20 passed.
• Real-Mac computer use: cargo test -p orca-computer-use --test live_macos --locked -- --ignored --test-threads=1: 84 of 86 passed in one full run.
  • Both failures were the test's own accessibility read of an element that had just closed; Orca's assertions held. Both tests pass when run alone.
  • The Electron check needs an isolated copy of an Electron app. It passed in the earlier live session.
• Acceptance with deepseek-flash (--model auto): all six tasks from design section 7 passed — TextEdit, Calculator, Finder, a Safari page, a canvas click, and observing Cherry Studio. That took 57 computer calls and 52 approvals, all "Allow once". Section 10 of the phase 2 design records both acceptance runs.
• CI: Linux linux-full and Windows native-x64 and native-arm64 passed on PR #132. release.yml requires all three to pass for the tagged commit.

Upgrade

After v0.5.11 is published:

npm install -g @blade-ai/orca@0.5.11
orca --version