You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
Vision assist. bobby install configures a vision provider during onboarding, and bobby vision connect --provider {openai,ollama,mlx} writes the provider config. vision-proxy takes --upstream {openai,ollama,mlx}, a single --vision-base-url with per-upstream defaults, an optional --model, and --spawn-server to run the local vision server as a managed kill-on-drop child (--server-script or BOBBY_VISION_SERVER_SCRIPT, else auto-detected).
BOBBY-VISION/1 wire contract for propose and extract, with a canonical provider interface and local mlx-vlm, Ollama, and LM Studio backends. Responses are normalized across {x,y}, coordinate arrays, position objects, and bare action names, degrading to a valid click rather than failing.
Vision escalation corpus collection behind [vision] corpusDir: a JSONL sink recording raw action kinds, target index, and the outcome stage (visionFallback, visionRejectionFloor, visionActFailed) at each terminal escalation branch. element_at_point on managed Chromium reads the internal DOM channel, never the policy-gated evaluate_javascript path.
NVIDIA OpenShell host. bobby install --host openshell / bobby openshell install writes an openshell/ pack (MCP Streamable HTTP client config, protocol: mcp policy sample, merge-only policy-network.yaml, skill, README). bobby openshell provision|revoke --sandbox <id> mints or revokes one agent-scoped principal per sandbox and writes a 0600 injection env under the OS config dir. bobby init --emit openshell prints the MCP fragment.
bobby openshell list|status|rotate, with non-secret .status.json sidecars. bobby doctor reports openshell-pack and checks openshell-admin, openshell-companion, openshell-mcp-url, and openshell-sandboxes when a pack is present; it warns on ≥2 local sandboxes sharing one Firefox companion, on a non-loopback cleartext MCP URL or server.host, and when an older pack lacks the deny rules. Secrets root overridable via BOBBY_OPENSHELL_SECRETS_DIR.
a11y_snapshot accepts an optional target and returns just that subtree instead of the whole page, resolving through frame hops and reading the owning frame's AX tree. The DOM candidate collector roles forms, dialogs, main, nav, and labelled regions, so containers are resolvable as scope roots.
Firefox companion wait_for supports Text, Value, and Document conditions, matching Chromium. networkQuiet remains unsupported on Firefox.
Changed
The default explore phase advertises the standard working loop — click, click_and_wait_for_popup, type_text, control_action, upload_files, dialog, download_url — with full schemas, so a client pays no toolset_select and schema-discovery round trip before its first action. command_execute, evaluate_javascript, emulate, niche mutations, and the intent family stay phase-gated.
control_action targets require only role and accessibleName; ordinal, framePath, and shadowPath default, so an a11y or form snapshot target passes verbatim instead of being hand-expanded. Target role matching is case-insensitive on both engines, and an explicit ordinal: 0 matches an omitted ordinal.
control_actionselectOne/selectMany and select fills accept an option's visible label as well as its value (trimmed, case-insensitive fallback on both engines). Snapshots surface labels; verification compares the committed option values.
Intent resolution and a11y_snapshot descend one level into same-process iframes on managed Chromium: a main-frame target inside a frame resolves and acts, with each in-frame candidate stamped with a re-resolvable frame hop used when no explicit framePath was named. Capped at 8 frames per gather under a shared node budget; frames with no stable address (no id, test id, or src) are skipped.
Whole-page inspect after a mutating command reads the live DOM instead of refetching the URL over HTTP. Any non-read-only command taints the page and navigation clears it, so post-submit SPA state is visible rather than the app shell. Evidence carries executionPath.reason: pageMutated on the live read.
Page-scoped text waits (role: main|RootWebArea|… or css: body|html|:root) read live document.body.innerText via evaluate, with empty optional fields treated as absent, so async confirmations match what a whole-page inspect sees.
click_and_wait_for_popup defaults autoCheckpoint=true and accepts pinned commandId/attemptId, matching boundary click and intent_submit_and_verify. It registers window.open targets, and page_list syncs untracked page targets into the session (one browser per session), excluding chrome:// browser chrome, so authorization popups are drivable.
A plain click on an anchor with a download attribute routes through the armed download capture on managed Chromium: the file lands in the session's downloads with Download evidence instead of completing with nothing materialized.
workflow_start failures carry detail with the error code and message instead of a bare reason.
The advertised WaitCondition schema names every kind tag, required field, and enum instead of presenting an opaque object.
a11y_snapshot drops InlineTextBox leaves, which duplicated their StaticText parents' text and dominated the payload. Its description points at toolset_select for the phases hidden by the default explore phase.
OpenShell provision revokes any prior principal for the sandbox id before minting, uses a unique idempotency key per attempt, and rolls back the minted principal if writing the injection env fails. The default capability floor is the narrow openshell preset (--capabilities-preset agent for the full agent floor). The sample policy denies evaluate_javascript and job_* at the OpenShell proxy and raises MCP max_body_bytes to 262 KiB.
bobby://intents documents the framePath step shape with an example and the Firefox exact-CSS/test-id hop requirement.
The OpenShell and jobs CLI paths share one blocking /v1 HTTP client for bearer and interface headers.
Fixed
page_open on a session whose browser died invalidates that specific dead worker and retries once on a fresh one, instead of returning an opaque internal on every call. Concurrent recovery cannot discard another caller's healthy replacement, and a failed replacement launch no longer leaves a phantom registered page.
session_close no longer wedges on a dead browser: managed-Chromium teardown treats an already-gone browser (closed channel, canceled oneshot) as closed instead of failing the release, which previously left the session listed forever with every retry failing internal. Browser termination against an already-dead process is likewise tolerated.
Managed Chromium re-attaches dead page handles: after a renderer crash or target hiccup closes the handle's channel, the next command on that page re-attaches to the live target. A truly destroyed target unregisters the page so callers get a clean notFound instead of a dead handle. Page lifecycle state stays consistent across dispatch, and a wait whose target detaches mid-flight is accepted rather than surfaced as a driver error.
CDP oneshot canceled and dead-target loss map to targetDetached (retryable). Stale CDP node ids after a re-render map to targetNotFound with fresh-snapshot repair instead of a raw browserCommandFailed.
Boundary commands that fail with waitConditionTimedOut or verificationFailed stay failed for inspect-then-adjust instead of never-retry needsReconciliation. A command that fails before reaching the browser, on argument or target-resolution errors, reports a plain failed; reconciliation is reserved for effects that may have landed.
An intent post-state wait that times out after the boundary click landed is reported as a non-retryable verificationFailed stating the click landed, instead of a bare waitConditionTimedOut that invited a duplicate submit.
intent_submit_and_verify with a networkQuiet-only wait fails when [aria-invalid=true] markers remain, instead of reporting completed on a soft settle after a rejected submit.
intentaction_target preserves framePath and shadowPath from the intent target, so iframe submits no longer discard the frame hop. Document targeting falls back without losing a named intent target, and locateNotFound attaches the near-miss candidate set to the stuck report instead of an empty list.
Whole-page inspect over direct HTTP treats an empty-<body> SPA shell (title/meta chrome plus scripts) as javascriptRequired and falls back to the live browser instead of returning shell HTML. The shell probe no longer races client-side URL rewrites.
inspect denied by network policy (loopback page, non-http URL) degrades to the browser that already has the page open instead of failing a DOM read with networkPolicyDenied. download_url keeps the hard denial.
[http] accepts partial overrides: missing fields fall back to defaults instead of failing TOML parse, so a host that sets only allow_loopback no longer bricks MCP startup.
bobby doctor passes BOBBY_BROWSER_CONFIG into the MCP handshake child, so [mcp] startup_toolset and the rest of that file apply to tools/list.
networkPolicyDenied guidance names the loopback and private-destination cause and the http.allow_loopback / http.allow_private_network operator switches across the repair hint, the taxonomy, and the download_url description; for page-offered files it points at clicking the link.
upload_files policy errors name the resolved absolute roots and the gateway working directory that relative roots resolve against.
Empty-string target fields (css, role, accessibleName, …) are rejected as invalidRequest at resolution time on both engines, instead of polling unmatchable until a wait deadline.
Protocol-layer -32602 rejections carry error.data.repair like every other failure.
scripts/dev/firefox-start.sh launches Firefox directly and verifies it owns the process it started.