v0.3.16

Highlights

• Computer Use can be scored on a public benchmark. syscity bench run
  drives a remote desktop over OSWorld's HTTP desktop contract, bench serve
  steps one action at a time so the harness's own loop decides when the next
  observation is taken, and bench probe checks a VM's endpoints in seconds
  instead of three model rounds into a scored task. The OSWorld glue, the plan,
  the host runbook and a one-command KVM provisioning script land together.
• The desktop tools act on structure, not just pixels. Elements are
  addressed by snapshot id and by platform id, the action ladder escalates from
  accessibility to pixels to the keyboard only when a step is observed not to
  have taken effect, every step is verified, and a fourth rung acts inside the
  conversation's own browser page by role and accessible name.
• The x86_64 Linux CLI is built on Ubuntu 22.04. The release binary's glibc
  floor drops from 2.39 to 2.35, so it starts on every 22.04 machine — the
  benchmark host included — instead of only on 24.04.

Added

• syscity bench run: one Computer Use task against a remote desktop.
  OsworldDesktop implements ComputerAdapter over the VM's HTTP service
  (/screenshot, /accessibility, /screen_size, /execute), so the action
  ladder, element ids and effect verification run unmodified — they are what the
  benchmark is supposed to measure. The agent's tool set is only
  computer/screen_state/time/todo (shell, file, browser and web act on
  the host, not the VM) and pre-approved, because no human answers approval
  prompts; the result file is the schema the Python glue reads. A fake VM covers
  the cache-and-invalidate rule, AT-SPI parsing, stale ids and action codegen.

• syscity bench serve: the step protocol. syscity sends one pyautogui
  snippet over stdout, blocks until the harness has run it, and only then looks
  at the screen — the ordering OSWorld's own loop has. In step mode /execute
  is never called, and a test asserts it. Logging for bench subcommands now goes
  to stderr, because stdout is the protocol's channel; the test holds that line
  with the real binary.

• syscity bench probe: /screen_size, /screenshot, /accessibility and
  (with --execute) a read-only snippet through /execute, reported per
  endpoint in seconds. Failures name the endpoint and exit non-zero; an empty
  accessibility tree and a screenshot whose size disagrees with /screen_size
  are warnings, because neither stops a run but both change how the agent works.

• OSWorld glue and a runbook (bench/osworld/): run.py --mode replay|step joins OSWorld's evaluator to syscity's own observe/act loop,
  with --split/--domain/--ids/--resume and a per-run summary that keeps
  infrastructure errors out of the denominator. docs/computer-use-benchmark.md
  is the plan, docs/osworld-kvm-server.md the host runbook, and
  scripts/osworld-bench-setup.sh provisions a KVM host in one idempotent
  command (dependencies, Docker, OSWorld, the 12 GiB VM image, this release's
  binary and the bench/ glue, then the container self-check and probe). It
  downloads the published tarball and verifies its checksum, so the host needs
  no Rust toolchain; --from-source still builds instead.

• Element addressing and an action ladder (computer/element_index.rs,
  computer/session.rs): snapshots carry a monotonic id and every element a
  s{snapshot:08x}:{path} id; ids from an earlier snapshot are rejected as
  stale rather than remapped. Actions escalate ax_action → pixel →
  keyboard, and only an observed SuspectedNoop escalates — Unverifiable
  stops instead, so a click that could not be checked never fires twice.
  ComputerSession is now the one step engine (settle, three-state
  verification, failure accounting, rollback, reflection, sensitive-UI
  scanning); ComputerUseLoop delegates to it instead of keeping a copy.

• The ladder reaches the page, the role and the background. A fourth rung
  asks the conversation's own browser tab, by role and accessible name, through
  a fresh ARIA snapshot — no name match means no action, never a neighbouring
  control. Screenshots report the display geometry they were taken on and every
  coordinate-taking tool says which space it is in, so a click on a scaled or
  secondary display lands where the box was. ActionEffect::{Partial, Refused}
  separate a half-done step from one the platform declined; refusal is terminal.
  delivery: "background" acts through accessibility scripting without taking
  the screen, and refuses with the reason when only an input-layer mechanism
  exists.

• GAIA Level 1 benchmark. The official question_scorer is wired through the
  eval harness. The 53-task × 3-trial baseline was 70.4% (112/159); after the
  determinism, retrieval and OCR work below, a repeat run scores 78.0% (124/159),
  with tasks passing all three trials rising from 29/53 to 33/53 and zero-pass
  tasks falling from 8 to 3. Both runs are recorded in evals/ledger.md. See
  docs/benchmark.md §M2 for the method and the failure breakdown.

• web_search can read the pages it finds. fetch_top: N fetches the top N
  results in the same call (through web_fetch, so the SSRF guard applies), and
  depth: "advanced" asks the provider for full page text where supported
  (Tavily search_depth + include_raw_content). The result cap moves from 10
  to 20.

• image reads text out of images via the local RapidOCR models when the
  vision feature is on: the tool now returns recognized text in reading order
  alongside the existing metadata, instead of dimensions only. The OCR models are
  resolved on their own, without pulling the OmniParser detector.

• OCR decodes with the model's own dictionary. The recognition model emits
  class indices, so the english PP-OCRv3 dictionary is downloaded and pinned to
  the model; a class count that does not match it is an error, not a guess.
  Detection boxes expand by ratio·area/perimeter (the standard DBNet unclip)
  and long lines are no longer squeezed into a 1600 px recognition input, which
  is what made dense document screenshots read as fragments.

• Suite summaries report task-level outcomes: all trials passed / at least one
  / zero, with the zero-pass task ids listed. Trial-level rates hide flakiness.

• Eval runs inherit the caller's proxy through the new
  AgentConfig.tool_environment, so shell curl in an eval can reach sites that
  need it.

Changed

• The x86_64 Linux CLI is built on ubuntu-22.04 from the new linux-amd64
  feature profile, which pins the binary's glibc floor at 2.35. Built on
  ubuntu-latest (24.04) it required GLIBC_2.39 and could not start on Ubuntu
  22.04 at all — including the OSWorld benchmark host, whose runbook installs
  the released binary. The profile is default features plus cloud, minus
  vision; the arm64 Linux CLI and both macOS CLIs keep vision.

• Eval runs are deterministic and less budget-bound: temperature 0.0 (was
  0.7), 60 tool iterations (was 30) with a soft "wrap up" reminder at 70%, and a
  system prompt built from the registered tool list (it previously named five
  tools that were not registered).

• Eval runs state the answer contract explicitly. The appended prompt now
  says the FINAL ANSWER: line is read by an automatic checker, not a person:
  the value alone, no unit, no explanation, no parentheticals, lists in the
  requested order with repeats kept. On the GAIA run that motivated it, ~10
  failing trials were answers the model knew but wrapped in prose or units
  (22 years for 22, No — Earl cannot do it; … for No). Measured on the
  9-task subset drawn from those: 40.7% → 81.5% over 3 trials each, 7 tasks up,
  none down.

• web_fetch retries transient failures once and tells the model when a page
  looks JavaScript-rendered, instead of returning an empty body as if the page
  had nothing to say.

• MSRV is now Rust 1.97 (was 1.95), required by wasmtime 50. Cargo.toml,
  rust-toolchain.toml, the CI workflows, the CI Dockerfile, README.md and
  docs/build.md all move together — the MSRV job asserts they agree.

• wasmtime 48 → 50, zip 2 → 9 (locked at 9.1.2). zip 9's ZipFile::name()
  returns a Result<Cow<str>>; screen_recorder now propagates that error instead
  of comparing the name directly.

Security

• The zstd 0.13 advisories are gone (RUSTSEC-2026-0391/0398/0403). They came
  from wasmtime's cache feature, which syscity never configures — it pulls
  wasmtime-internal-cache → zstd 0.13, a line with no patched release. Building
  wasmtime with default-features = false and without cache leaves zstd 0.14
  (via zip 9.1.2) as the only copy.
• The three zip extract() waivers (RUSTSEC-2026-0367/0368/0369) are removed
  from deny.toml: zip 9 fixes them, and deny.toml's wasmtime@48.0.5
  duplicate-tree skip now names wasmtime@50.0.0.

Fixed

• The x86_64 Linux CLI links, and starts on 22.04. Moving the build to
  ubuntu-22.04 exposed that the ONNX Runtime vision links against — pyke's
  prebuilt for x86_64-unknown-linux-gnu — is compiled against glibc ≥ 2.38
  headers. On 22.04 the link fails outright (undefined symbol: __isoc23_strtoll, std::__cxx11::…::_M_replace_cold); on 24.04 it links but
  produces a GLIBC_2.39-only binary. The release therefore builds the
  linux-amd64 profile, which drops the ONNX-based OCR/UI detection on this one
  target. --features vision from source restores it on a 24.04+ host.
• The cargo cache is keyed by runner image. Swatinem/rust-cache keys on
  Linux-x64, so the 22.04 job could restore a target/ produced on 24.04.
  (It was not this run's failure — the fix landed and a cold build failed the
  same way — but the two jobs should not be able to share compiled objects.)