Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/plans/computer-use-refactor-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@
```
┌────────────────────────────────────────────────────────────────┐
│ L3 模式与配置面 │
│ · 单一开关面: ai.computer_use_enabled ⊇ browser_control │
│ · 双独立开关: ai.computer_use_enabled (桌面) 与 │
│ ai.browser_control_enabled (浏览器, 默认开) 互不牵连 │
│ · permission intents: computer_use + browser_control │
│ · 每回合工具组装 (仿 codex spec_plan): 按模型能力/平台/远程裁剪 │
│ · deny 表单一真源 (Rust 导出 + contract test 三端对齐) │
Expand Down Expand Up @@ -148,7 +149,7 @@

**决策 7:动作即观察。** 执行器在每个 mutating 动作后:settle 延迟(桌面固定/浏览器等 network idle)→ 自动截图或快照 diff → 打包进同一 result(仿 playwright-mcp Response 聚合器 + cua post-action screenshot)。配套截图 retention(只留最近 N 张,按块修剪保护 prompt cache)。

**决策 8:能力开关收敛为两级真实门控。** 删除装饰性 cargo feature 层;ControlHub 实现真实 `is_enabled()`(服从 `ai.computer_use_enabled` 或新增 `ai.browser_control_enabled`,按 DeliveryProfile/远程会话裁剪;新增 `browser_control` permission intent 进后端枚举与 `GlobalPermissionRulesDialog.tsx`(对应 C5)。
**决策 8:能力开关收敛为两个独立的真实门控。** 删除装饰性 cargo feature 层;桌面控制与浏览器控制是两个独立能力:`ai.computer_use_enabled` 只门控 ComputerUse 桌面工具(现状已如此);新增独立的 `ai.browser_control_enabled`(默认开)门控 ControlHub browser 域,关闭 computer use 不影响浏览器控制(产品决策确认,2026-07-26)。ControlHub 实现真实 `is_enabled()` 服从后者,并按 DeliveryProfile/远程会话裁剪;新增 `browser_control` permission intent 进后端枚举与 `GlobalPermissionRulesDialog.tsx`(对应 C5 的可管辖性诉求)。

---

Expand Down Expand Up @@ -201,7 +202,7 @@
### 阶段 6:配置/权限/Peer 面(~2 周)

- **文件**:`control_hub_tool.rs`(真实 `is_enabled`)、`GlobalPermissionRulesDialog.tsx` + 后端 intent 枚举(新增 `browser_control`)、`session-config.json` 文案修正、`SessionConfig.tsx`(拆 personalization/permissions 两组件、状态命令走传输适配层或标注本机/远端)、`peer-device-adapter.ts`/`peer_host_invoke.rs`/`cli/peer_host/deny.rs`(Rust 单一真源导出 + contract test,`browser_control_*` 补 deny)、`ChatInput.tsx`/`AgentsScene.tsx`(抽 `useComputerUseEnabled()` hook,slash 路径补门禁,门禁移后端 `get_available_modes`)、`agents.rs`/`agentVisibility.ts`(统一 ComputerUse 身份与命名)。
- 风险:低中。验证:deny 表 contract test 三端对齐;Peer 场景手测开关/权限弹窗归属;关闭 computer use 后确认 ControlHub browser 域同步禁用
- 风险:低中。验证:deny 表 contract test 三端对齐;Peer 场景手测开关/权限弹窗归属;关闭 computer use 后确认 ControlHub browser 域**不受影响**(两开关独立),关闭 browser_control 后确认 ControlHub browser 域禁用且 ComputerUse 不受影响

---

Expand Down
78 changes: 72 additions & 6 deletions src/apps/desktop/src/computer_use/desktop_host/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,45 @@ mod visual_grid_tests {
}
}

#[cfg(all(test, target_os = "windows"))]
mod windows_foreground_tests {
use super::*;

#[test]
fn foreground_app_reports_executable_separately_from_window_title() {
let app = DesktopComputerUseHost::windows_foreground_application(
"Search".to_string(),
4242,
Some("explorer.exe".to_string()),
);

assert_eq!(app.name.as_deref(), Some("Search"));
assert_eq!(app.process_name.as_deref(), Some("explorer.exe"));
assert_eq!(app.process_id, Some(4242));
}

#[test]
fn foreground_app_falls_back_to_title_only_when_process_lookup_fails() {
let app =
DesktopComputerUseHost::windows_foreground_application("Search".to_string(), 4242, None);

assert_eq!(app.name.as_deref(), Some("Search"));
assert_eq!(app.process_name, None);
}

#[test]
fn foreground_app_drops_empty_title_and_empty_executable() {
let app = DesktopComputerUseHost::windows_foreground_application(
String::new(),
0,
Some(String::new()),
);

assert_eq!(app.name, None);
assert_eq!(app.process_name, None);
}
}

/// Unified mutable session state for computer use — one mutex instead of five.
/// State transitions are applied centrally after each action (screenshot, pointer move, click, etc.).
#[derive(Debug)]
Expand All @@ -123,7 +162,7 @@ struct ComputerUseSessionMutableState {
navigation_focus: Option<ComputerUseNavFocus>,
/// Cached full-screen screenshot for fast consecutive crops.
screenshot_cache: Option<ScreenshotCacheEntry>,
/// After `screenshot`, block `pointer_move_rel` / `ComputerUseMouseStep` until an absolute move
/// After `screenshot`, block `pointer_move_rel` until an absolute move
/// from AX/OCR/globals (`mouse_move`, `move_to_text`, `click_element`) clears this.
block_vision_pixel_nudge_after_screenshot: bool,
/// After click / key / type / scroll / drag: recommend a **`screenshot`** to confirm UI state (Cowork verify).
Expand Down Expand Up @@ -394,6 +433,9 @@ end tell"#])
let bundle = parts.get(2).map(|x| x.trim()).filter(|x| !x.is_empty());
Some(ComputerUseForegroundApplication {
name: Some(name.to_string()),
// `name of p` from System Events is already the process name, not a
// window title, so it doubles as the process identity here.
process_name: Some(name.to_string()),
bundle_id: bundle.map(|b| b.to_string()),
process_id: Some(pid),
})
Expand Down Expand Up @@ -430,11 +472,12 @@ end tell"#])
} else {
String::new()
};
Some(ComputerUseForegroundApplication {
name: if title.is_empty() { None } else { Some(title) },
bundle_id: None,
process_id: Some(pid as i32),
})
let exe_basename = if pid == 0 {
None
} else {
crate::computer_use::windows_list_apps::exe_basename_for_pid(pid)
};
Some(Self::windows_foreground_application(title, pid, exe_basename))
};

ComputerUseSessionSnapshot {
Expand All @@ -444,6 +487,29 @@ end tell"#])
}
}

/// Build the Windows foreground-app identity from the window title and the
/// owning process's executable basename.
///
/// `name` stays the window title; `process_name` carries the *process*
/// identity. Callers that classify the frontmost app (browser detection)
/// must match on `process_name`: window titles collide with app names by
/// substring (a "Search" window contains "arc"). When the process cannot be
/// opened (access denied, exited), `process_name` is `None` and callers see
/// the pre-existing title-only shape.
#[cfg(target_os = "windows")]
fn windows_foreground_application(
title: String,
pid: u32,
exe_basename: Option<String>,
) -> ComputerUseForegroundApplication {
ComputerUseForegroundApplication {
name: if title.is_empty() { None } else { Some(title) },
process_name: exe_basename.filter(|s| !s.is_empty()),
bundle_id: None,
process_id: Some(pid as i32),
}
}

#[cfg(target_os = "linux")]
fn session_snapshot_linux() -> ComputerUseSessionSnapshot {
// Best-effort: no standard API across Wayland/X11 without extra deps.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ use enigo::{Axis, Button, Coordinate, Direction, Enigo, Key, Keyboard, Mouse, Se
use log::debug;
use std::time::Duration;

/// Relative nudges (`pointer_move_rel`, `ComputerUseMouseStep`) right after a model-driven screenshot are almost always wrong when deltas are guessed from the image; block until a trusted absolute move.
const VISION_PIXEL_NUDGE_AFTER_SCREENSHOT_MSG: &str = "Computer use refused: do not use `pointer_move_rel` or `ComputerUseMouseStep` immediately after a `screenshot` — nudging from the JPEG is inaccurate. First reposition with `move_to_text`, `click_element`, `locate` + `mouse_move` (`use_screen_coordinates`: true), or `mouse_move` using globals from tool JSON; then relative nudges are allowed if still needed.";
/// Relative nudges (`pointer_move_rel`) right after a model-driven screenshot are almost always wrong when deltas are guessed from the image; block until a trusted absolute move.
const VISION_PIXEL_NUDGE_AFTER_SCREENSHOT_MSG: &str = "Computer use refused: do not use `pointer_move_rel` immediately after a `screenshot` — nudging from the JPEG is inaccurate. First reposition with `move_to_text`, `click_element`, `locate` + `mouse_move` (`use_screen_coordinates`: true), or `mouse_move` using globals from tool JSON; then relative nudges are allowed if still needed.";

impl DesktopComputerUseHost {
pub(super) fn ensure_input_automation_allowed() -> BitFunResult<()> {
Expand Down Expand Up @@ -602,7 +602,7 @@ impl DesktopComputerUseHost {
.map_err(|e| BitFunError::tool(format!("lock: {}", e)))?;
let Some(map) = s.pointer_map else {
return Err(BitFunError::tool(
"Run action screenshot first: on macOS, pointer_move_relative / ComputerUseMouseStep convert pixel deltas using the last capture scale."
"Run action screenshot first: on macOS, `pointer_move_rel` converts pixel deltas using the last capture scale."
.to_string(),
));
};
Expand Down
2 changes: 1 addition & 1 deletion src/apps/desktop/src/computer_use/windows_list_apps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ unsafe extern "system" fn enum_windows_cb(hwnd: HWND, lparam: LPARAM) -> BOOL {
}

/// Resolve the full image path of `pid` and return its `.exe` basename.
fn exe_basename_for_pid(pid: u32) -> Option<String> {
pub(super) fn exe_basename_for_pid(pid: u32) -> Option<String> {
let handle = unsafe { OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, 0, pid) };
if handle.is_null() {
return None;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ hidden: true

Fast browser automation CLI for AI agents. Chrome/Chromium via CDP with accessibility-tree snapshots and compact `@eN` element refs.

Prefer BitFun's `ControlHub` browser domain when it is available; use this skill only when `ControlHub` is unavailable. The two stacks use separate browser instances, element refs, and login state, so do not mix them within one task.
`ControlHub`'s browser domain is BitFun's default path for ordinary web pages, so this skill is opt-in and you are reading it because it was invoked explicitly — proceed with agent-browser for this task. It is the right tool for Electron desktop apps, Slack workspace automation, and cloud browsers, which `ControlHub` does not cover. The two stacks use separate browser instances, element refs, and login state, so do not mix them within one task.

Install only after user approval: `npm i -g agent-browser@0.32.3 && agent-browser install`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ The user will primarily request you perform software engineering tasks. This inc
- When the user explicitly asks to complete work and review it carefully, finish the implementation first, then dispatch at most one independent read-only `CodeReview` Task. Do not fan out `CodeReview` into architecture, performance, security, product, or other invented dimensions: broader coverage belongs to the unified `/review` path, which selects bounded review lenses and owns cost confirmation. Do not launch review by default for every task.
- Treat reviewer output as adversarial evidence. The reviewer never fixes its own findings. Apply accepted fixes in the implementation agent, then request a fresh independent review only when the change or risk warrants it.
- When WebFetch reports a redirect, follow the redirect URL if it is relevant and safe for the user's request.
- For browser and web-page work, route in this order: (1) reading page content that does not require the user's login state: use WebFetch; (2) pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs); (3) non-Chromium browsers (Firefox/Safari) or native desktop apps: use `ComputerUse` desktop actions. Prefer `ControlHub` over browser-automation skills such as `agent-browser`; use those skills only when `ControlHub` is unavailable.
- For browser and web-page work, route in this order: (0) only opening or showing a URL for the user, with no page reading or interaction: use `ControlHub` with `domain: "browser"`, `action: "open_builtin"`; (1) reading page content that does not require the user's login state: use WebFetch; (2) pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs) — `connect` drives BitFun's managed browser profile, which is separate from the user's everyday browser; it persists cookies and logins across runs, so if the page shows a login wall, ask the user to sign in once in that window instead of retrying navigation or entering credentials yourself; (3) non-Chromium browsers (Firefox/Safari) or native desktop apps: use `ComputerUse` desktop actions only when `ComputerUse` appears in your current tool list; if it does not, tell the user the task needs the Computer Use mode (enabled via the Computer use setting) instead of guessing another path or calling an unavailable tool. `ControlHub` covers ordinary web pages; for scenarios it does not support — Electron desktop apps (VS Code/Slack/Discord/Figma), Slack workspace automation, cloud browsers — load `agent-browser` explicitly via `Skill(skill="agent-browser")`, since it is opt-in and not listed in your skills by default.
- When multiple tool calls are independent, run them in parallel. Keep dependent operations sequential, and never use placeholders or guess missing parameters.
- Use specialized tools for file reads, edits, searches, and deletions because they preserve workspace context and permissions. Use ExecCommand for commands that genuinely need a shell. Do not use shell commands only to communicate with the user.
- For security-sensitive tasks, support defensive analysis and remediation only. Refuse malicious code, exploit workflows, credential harvesting, or instructions that would facilitate abuse.
Expand Down
16 changes: 9 additions & 7 deletions src/crates/assembly/core/src/agentic/agents/prompts/claw_mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,20 @@ When a first-class tool exists for an action, use the tool directly instead of a

Use `ControlHub` for browser automation, terminal signalling, and routing/capability introspection only when it appears in your current tool list:

- `domain: "browser"` for websites and web apps in the user's real browser through CDP.
- `domain: "browser"` for websites and web apps in BitFun's managed browser profile through CDP.
- `domain: "terminal"` for signalling existing terminal sessions, such as interrupting or killing them.
- `domain: "meta"` for capability and route checks.

For browser and web-page work, route in this order:

1. Reading page content that does not require the user's login state: use `WebFetch`.
2. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs).
3. Non-Chromium browsers (Firefox/Safari) or native desktop apps: delegate to a `ComputerUse` session as described below.
1. Only opening, showing, previewing, or displaying a URL for the user (no page reading, no interaction): use `ControlHub` with `domain: "browser"`, `action: "open_builtin"`, `params: { url }`. The page renders in BitFun's built-in right-side browser panel. Do not delegate this to a `ComputerUse` sub-agent and do not call `connect`/`navigate` for it.
2. Reading page content that does not require the user's login state: use `WebFetch`.
3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). `connect` drives BitFun's managed browser profile, which is separate from the user's everyday browser; it persists cookies and logins across runs, so if the page shows a login wall, ask the user to sign in once in that window instead of retrying navigation or entering credentials yourself.
4. Non-Chromium browsers (Firefox/Safari) or native desktop apps: delegate to the `ComputerUse` sub-agent as described below.

Do not use `ControlHub` for local computer, operating-system, or desktop UI work. Desktop and system actions have moved to the dedicated `ComputerUse` tool/agent. This includes screenshots, OCR, mouse, keyboard, app state, app launching, opening files or URLs through the OS, clipboard access, OS facts, and local scripts.
Do not use `ControlHub` for local computer, operating-system, or desktop UI work. Desktop and system actions have moved to the dedicated `ComputerUse` tool/agent. This includes screenshots, OCR, mouse, keyboard, app state, app launching, opening local files and non-http(s) URLs through the OS, clipboard access, OS facts, and local scripts.

If the user asks you to operate or inspect the local computer, delegate the task to a `ComputerUse` session via SessionControl/SessionMessage only when both tools appear in your current tool list. Include the user's goal, target app/window/site, safety constraints, and expected verification in the handoff. If delegation is unavailable, explain that the task needs the Computer Use mode.
If the user asks you to operate or inspect the local computer, delegate the task via `Task` with the `ComputerUse` sub-agent, only when that sub-agent is listed among your available `Task` subagent types. Include the user's goal, target app/window/site, safety constraints, and expected verification in the handoff. If delegation is unavailable, explain that the task needs the Computer Use mode.

# Session Coordination

Expand All @@ -47,7 +48,8 @@ Choose the session type intentionally:
- `agentic` for implementation, debugging, and code changes.
- `Plan` for requirement clarification, scoping, and planning before coding.
- `Cowork` for research, documents, presentations, summaries, and other office-related work.
- `ComputerUse` for local computer/system/desktop operation and perception.

Local computer/desktop work is not a SessionControl session type; delegate it through `Task` with the `ComputerUse` sub-agent when that subagent type is available.

Operational rules:

Expand Down
Loading