Skip to content

feat: split screen intelligence from accessibility and add dedicated settings#50

Merged
senamakel merged 4 commits into
tinyhumansai:mainfrom
senamakel:feat/screen-intelligence
Mar 28, 2026
Merged

feat: split screen intelligence from accessibility and add dedicated settings#50
senamakel merged 4 commits into
tinyhumansai:mainfrom
senamakel:feat/screen-intelligence

Conversation

@senamakel
Copy link
Copy Markdown
Member

Summary

  • Introduced screen_intelligence as a first-class feature area while preserving existing openhuman.accessibility_* RPC compatibility.
  • Split Rust core module ownership so screen intelligence now lives under openhuman/screen_intelligence, with openhuman/accessibility as a compatibility re-export.
  • Added screen-intelligence settings update flow (openhuman.update_screen_intelligence_settings) and config fields for policy mode and allowlist/denylist behavior.
  • Kept accessibility and screen intelligence separate in settings UI by restoring Accessibility naming and adding a dedicated Screen Intelligence panel/route.
  • Wired screen captures to active-window policy filtering and memory persistence under screen_intelligence category.

Problem

  • The initial implementation mixed accessibility automation and screen intelligence into a single surfaced feature, creating product confusion and coupling.
  • Screen intelligence needed to become a configurable core feature with independent policy controls for capture targeting.
  • We needed this without breaking existing accessibility command paths already used by frontend and Tauri command bridges.

Solution

  • Created a dedicated openhuman::screen_intelligence module and moved implementation ownership there.
  • Kept backward compatibility by re-exporting from openhuman::accessibility and leaving openhuman.accessibility_* RPC methods in place.
  • Added and connected openhuman.update_screen_intelligence_settings to persist and apply runtime config updates.
  • Implemented policy controls (all_except_blacklist / whitelist_only) with allowlist + denylist matching.
  • Added a standalone Screen Intelligence settings panel and route, while restoring Accessibility panel/menu labeling and behavior.

Testing

  • yarn -s compile
  • cargo check --manifest-path src-tauri/Cargo.toml
  • Other checks run (list commands)
  • Commands: npx prettier --check ., npx eslint ., npx tsc --noEmit, cargo fmt --manifest-path src-tauri/Cargo.toml, cargo check --manifest-path src-tauri/Cargo.toml
  • Manual validation completed (list scenarios)

Impact

  • Runtime impact: desktop-focused screen intelligence controls and capture behavior now have their own feature boundary.
  • Compatibility: existing accessibility RPC command names remain functional; no immediate client migration required.
  • Architecture: clearer separation between accessibility permissions/session controls and screen intelligence policy/config ownership.
  • Data impact: screen-intelligence summaries are persisted under screen_intelligence memory category.

Breaking Changes

  • None
  • Yes (describe clearly, including migration steps)

Related

  • Issue(s): n/a
  • Follow-up PR(s)/TODOs: migrate remaining accessibility-labeled RPC names to explicit screen-intelligence names once clients are updated.

- Reformatted code in `core_server.rs` for better readability, including consistent indentation and line breaks.
- Enhanced the clarity of function calls and JSON handling by spreading parameters across multiple lines.
- Improved overall structure and maintainability of the dispatch and settings view response functions.
…IntelligenceConfig

- Renamed AccessibilityAutomationConfig to ScreenIntelligenceConfig across the codebase for improved clarity.
- Updated related structures and functions to utilize the new configuration, including default values and additional fields for enhanced functionality.
- Ensured backward compatibility by adjusting references in the AccessibilityEngine and related modules.
- Enhanced the AppContext structure to include window bounds, improving state management for UI components.
… functionality

- Added a new Screen Intelligence module to manage window capture policies, vision summaries, and memory ingestion.
- Implemented settings for enabling/disabling features, configuring capture policies, and managing allowlists/deny lists.
- Updated the accessibility panel to include Screen Intelligence options and integrated permission requests for screen recording and accessibility.
- Enhanced the core server to handle updates to Screen Intelligence settings and ensure proper state management across the application.
- Refactored related components and Redux state management to support the new features, improving user experience and functionality.
@senamakel senamakel merged commit a0c3089 into tinyhumansai:main Mar 28, 2026
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant