Skip to content

v0.2.10 — GUI coordinate mapping + batch actions + screenshot masking

Choose a tag to compare

@dahai80 dahai80 released this 04 Sep 16:54
· 1 commit to main since this release
caf0a61

What's new

Three GUI features for fusion-osagent (Issues #38, #39, #40):

#38 — Coordinate-mapping (scale_factor)

GuiResult.scale_factor: f32 bridges logical points (GuiAction x/y + AXPosition) ↔ physical pixels (screenshot_width/height). Retina=2.0, non-Retina=1.0. Caller converts pixel = point * scale_factor.

#39 — Batch actions (gui_action_batch)

Sequential multi-action pipeline collecting per-step GuiResult. Non-transactional (single-step failure does not abort later steps).

results = executor.gui_action_batch([
    {"kind": "focus_app", "bundle_id": "com.apple.Terminal"},
    {"kind": "screenshot"},
])

#40 — Screenshot sensitive-region masking (mask_sensitive)

GuiAction::Screenshot gains mask_sensitive: bool (default false). When true, AX-secure text fields (kAXSecureTextFieldSubrole) are pixel-redacted to black in the PNG (Y-flipped, scale_factor-adjusted). Gated on Accessibility trust.

Safety

0 new unsafe blocks. +7 Rust tests, +9 Python tests.

Verification

526 Rust + 222 Python (7 skip TCC) green; clippy --all-targets -D warnings clean; fmt/ruff clean; maturin builds.

Closes #38, #39, #40.