Skip to content

v1.2.0

Choose a tag to compare

@JeanExtreme002 JeanExtreme002 released this 16 Nov 19:15
· 23 commits to main since this release
852712d

pip install --upgrade mcp-server-macos

Your assistant can now see what an app shows (its buttons, text fields, links, checkboxes, tabs and menus) as text, and use them by name: no screenshot to find what to click, and no coordinates to guess. A model without vision can operate apps too.

New

  • The ui toolset, over the Accessibility tree. Like the other toolsets, your assistant turns it on when a task needs it (enable_toolsets), or you list it with --toolsets.
    • Seeing: get_ui_elements lists what an app shows, each element with an id, its name, its value and where it is. find_ui_elements and wait_for_ui_element look for one by name or role, exact names first ("Save" before "Don't Save"). get_focused_ui_element tells where keystrokes go, get_ui_element_at what's at a point, and read_ui_element_text reads all the text an element holds, scrolled out of view included.
    • Using: press_ui_element, click_ui_element, set_ui_element_value, focus_ui_element and perform_ui_action act on an element by its id. list_menu_items gives an app's menus with their keyboard shortcuts, and click_menu_item chooses one by its path (File > Export > PDF…).
    • Cheap to read: a field left out is empty, so a button costs {id, role, name, x, y, width, height}, and get_ui_elements lists 300 elements at most unless asked for more. An element the app doesn't place has no position, rather than 0, 0. An id is never reused while the server runs; one from long ago asks for the list again.
    • Safe by default: every tool needs Accessibility, asked for as the windows tools do. Pressing, clicking, acting and choosing a menu item are marked open ended, since what a button does is the app's (send, delete, buy), so your client asks you first.

Changed

  • pymacos 1.40 or newer is required, for macos.ui. pip install --upgrade mcp-server-macos brings it.
  • The server's instructions mention what apps show among what it covers, so your assistant looks for these tools before taking a screenshot.

See the README for how to add the server to a client.