Skip to content

feat: accessibility automation v1 (core + tauri + settings)#47

Merged
senamakel merged 3 commits into
mainfrom
feat/accessbility
Mar 28, 2026
Merged

feat: accessibility automation v1 (core + tauri + settings)#47
senamakel merged 3 commits into
mainfrom
feat/accessbility

Conversation

@senamakel
Copy link
Copy Markdown
Member

Summary

  • Added a native rust-core accessibility automation subsystem (macOS-first) with ephemeral in-memory runtime state.
  • Implemented new core JSON-RPC methods under openhuman.accessibility_* for status, permissions, session lifecycle, capture, input actions, and autocomplete flows.
  • Exposed matching Tauri commands and frontend typed wrappers in tauriCommands.ts.
  • Added Redux accessibilitySlice for permissions/session/feature/error state and async lifecycle actions.
  • Added first-class Settings route/panel at /settings/accessibility and linked it in settings navigation/home.
  • Added Rust and frontend tests for lifecycle, validation, RPC envelopes, reducer behavior, and panel rendering.

Problem

  • The app lacked a dedicated, native accessibility automation layer for screen state monitoring, bounded input control, and predictive typing.
  • There was no unified core RPC/host/UI surface to manage consented sessions, permission checks, panic stop handling, and runtime status.

Solution

  • Implemented rust-core/src/openhuman/accessibility/mod.rs as a process-global engine with:
    • Hybrid capture scheduler (foreground context change + baseline cadence).
    • Session TTL enforcement and explicit consent-gated session start.
    • Denylist-based context blocking and input action validation.
    • In-memory-only capture/context buffers (no disk persistence by default).
    • Predictive autocomplete suggest/commit primitives.
  • Added AccessibilityAutomationConfig to core config schema/defaults and wired it into Config.
  • Extended rust-core/src/core_server.rs dispatch table with all openhuman.accessibility_* methods and response typing.
  • Added Tauri command proxies in src-tauri/src/commands/openhuman.rs, command registration in src-tauri/src/lib.rs, and typed bridge event emission (openhuman:accessibility) with host notifications on key events.
  • Added frontend command wrappers/types, Redux slice integration, and a dedicated settings UI panel.

Testing

  • yarn -s compile
  • cargo check --manifest-path src-tauri/Cargo.toml
  • Other checks run (list commands)
  • npx prettier --check .
  • npx eslint .
  • npx tsc --noEmit
  • cargo fmt --manifest-path src-tauri/Cargo.toml
  • npx vitest run --config test/vitest.config.ts src/store/__tests__/accessibilitySlice.test.ts src/components/settings/panels/__tests__/AccessibilityPanel.test.tsx
  • cargo test -p openhuman-core accessibility --manifest-path src-tauri/Cargo.toml
  • Manual validation completed (list scenarios)

Impact

  • Runtime/platform: Adds macOS-first accessibility automation surface in core; non-macOS paths report unsupported/limited permission status.
  • Security/safety: Explicit consent gate, bounded session TTL, denylist checks, and panic-stop action path.
  • Data handling: Capture/context state remains ephemeral in memory by default (no persisted frames).
  • Compatibility: Adds new config section (accessibility) with defaults; existing config loads remain backward compatible via serde defaults.

Breaking Changes

  • None
  • Yes (describe clearly, including migration steps)

Related

  • Issue(s): N/A
  • Follow-up PR(s)/TODOs:
    • Wire true OS-level global panic hotkey registration (current panic stop path is action-driven).
    • Expand native permission interrogation for Screen Recording/Input Monitoring beyond unknown fallback where feasible.

@senamakel senamakel marked this pull request as ready for review March 28, 2026 00:25
@senamakel senamakel merged commit 23c4e11 into main Mar 28, 2026
8 of 14 checks passed
@senamakel senamakel deleted the feat/accessbility branch March 28, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant