Skip to content

feat(simulator-management): add tools to toggle iOS Simulator software and hardware keyboard#347

Open
yjmeqt wants to merge 5 commits intogetsentry:mainfrom
yjmeqt:feat/toggle-keyboard
Open

feat(simulator-management): add tools to toggle iOS Simulator software and hardware keyboard#347
yjmeqt wants to merge 5 commits intogetsentry:mainfrom
yjmeqt:feat/toggle-keyboard

Conversation

@yjmeqt
Copy link
Copy Markdown

@yjmeqt yjmeqt commented Apr 19, 2026

Summary

Adds two MCP tools exposing the iOS Simulator's keyboard menu items, which have no direct xcrun simctl equivalent and were unreachable through the server.

  • toggle_software_keyboardSimulator → I/O → Keyboard → Toggle Software Keyboard (Cmd+K)
  • toggle_connect_hardware_keyboardSimulator → I/O → Keyboard → Connect Hardware Keyboard (Cmd+Shift+K)

Both delegate to a shared helper _keyboard_shortcut.ts that:

  1. Resolves the simulatorId to its device name via xcrun simctl list devices --json and verifies the simulator is booted.
  2. Brings Simulator.app forward with open -a Simulator.
  3. Raises the window for the target simulator via an AppleScript AXRaise on the window whose title contains the device name — errors out if no matching window is found (no silent fallthrough to the frontmost window).
  4. Sends the appropriate keystroke via a second osascript invocation.

Resolves #346.

Implementation notes

  • Why AppleScript, not defaults write ConnectHardwareKeyboard: the defaults route is a persistent global preference that only takes effect on simulator launch; it cannot toggle a running simulator and applies to all simulators simultaneously. The AppleScript approach mirrors the manual user action exactly, takes effect immediately, and leaves no persistent state — which matches what UI-automation consumers expect. Happy to revisit if maintainers prefer defaults.
  • Accessibility permission: the MCP host process needs Accessibility permission in System Settings for the osascript … tell process "Simulator" to keystroke … call to succeed. This is already a prerequisite for other UI automation tools in this repo; surfaced explicitly in both tool descriptions.
  • Workflow registration: both tools are added to manifests/workflows/simulator-management.yaml so they appear in the Simulator Management workflow group in TOOLS.md / TOOLS-CLI.md.
  • Docs: TOOLS.md and TOOLS-CLI.md regenerated via npm run docs:update; CHANGELOG.md gains a new ## [Unreleased] section with both entries linked to [Feature]: Add tools to toggle iOS Simulator software keyboard and hardware keyboard connection #346.

Test plan

  • npm run typecheck — passes
  • npm run lint — passes
  • npm run format:check — passes
  • npm run docs:check — passes
  • npm test — 144 files / 1457 tests pass (+3 files / +17 tests over baseline); covers success path, not-found, not-booted, open-app-failed, no-window-found, osascript-failed, and exception paths for both tools
  • Manual: on a booted iOS simulator with Accessibility permission granted, verify `toggle_software_keyboard` hides/shows the on-screen keyboard and `toggle_connect_hardware_keyboard` connects/disconnects the Mac keyboard input

yi-jiang-applovin and others added 5 commits April 19, 2026 17:09
Resolves a simulator UDID to its device name, verifies the simulator is
booted, focuses its Simulator.app window via AppleScript, and sends a
Cmd+K or Cmd+Shift+K keystroke. Consumed by the keyboard toggle tools
added in subsequent commits.

Refs getsentry#346
Exposes Simulator > I/O > Keyboard > Toggle Software Keyboard (Cmd+K)
as an MCP tool. Delegates to the shared keyboard shortcut helper.

Refs getsentry#346
Exposes Simulator > I/O > Keyboard > Connect Hardware Keyboard
(Cmd+Shift+K) as an MCP tool. Disconnecting makes the on-screen
keyboard appear for tap-based input during UI automation.

Refs getsentry#346
Regenerate TOOLS.md and TOOLS-CLI.md, register the new tools in the
simulator-management workflow, and add CHANGELOG entries for
toggle_software_keyboard and toggle_connect_hardware_keyboard.

Refs getsentry#346
Copy link
Copy Markdown
Contributor

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ca39643. Configure here.

}

function buildFocusScript(deviceName: string): string {
const safeName = deviceName.replace(/"/g, '\\"');
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incomplete AppleScript escaping misses backslash characters

Medium Severity

buildFocusScript only escapes double quotes in the device name but not backslashes. If a simulator name contains a backslash (e.g. Test\Device), the resulting AppleScript string will contain an unescaped \, which AppleScript interprets as an escape character — potentially corrupting the name match or causing a script error. Worse, a name like Device\" would produce Device\\" after quote escaping, which closes the AppleScript string prematurely. The codebase already has proper escaping in escapeSwiftStringLiteral in screenshot.ts that escapes backslashes before quotes, plus newlines/tabs — the same pattern is needed here.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit ca39643. Configure here.

'tell application "System Events"',
' tell process "Simulator"',
' set frontmost to true',
' set matchingWindows to (every window whose title contains "' + safeName + '")',
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substring window-title match can target wrong simulator

Medium Severity

buildFocusScript uses title contains for AppleScript window matching, which is a substring check. When a user has simulators with overlapping names (e.g. "iPhone 15" and "iPhone 15 Pro"), querying for "iPhone 15" matches both windows, and the tool silently raises and sends the keystroke to an arbitrary one. The codebase already identifies this exact pitfall in screenshot.ts (lines 72–86) and uses prefix + en-dash matching to avoid it. The new code lacks this mitigation, so users with commonly-named simulators can have the keyboard toggled on the wrong device.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit ca39643. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add tools to toggle iOS Simulator software keyboard and hardware keyboard connection

2 participants