Skip to content

hs.axuielement Overview

Jump to bottom Edit New page
asmagill edited this page Oct 10, 2020 · 1 revision

A lot of Hammerspoon's ability to manipulate and query other applications comes from interacting with the accessibility objects that every macOS application support. For common actions, such as querying or changing window positions, triggering menu actions, etc. the details are wrapped up in functions and methods which provide a simple interface and hide most of the details from the user.

hs.axuielement is a Hammerspoon module that allows you to inspect and interact with these accessibility objects directly. While you can replicate a number of already built in functions, the real power of the module comes from being able to do things beyond the scope of the traditional modules.

With this document, I hope to provide an introduction to some of the things that are possible with this module. To make the most of this module, you need to think of it as a toolkit and not an end product just waiting for you to fill in a few pieces of data to get a desired result. This toolkit will let you investiage the accessibility structure of applications and build your own "end products" which can be used in other scripts. To take full advantage of this module, you should expect to do some experimentation first.

First things first

The Accessibility framework was initially created by Apple to aid developers of assitive technologies in the creation of tools to help users with disabilities. Any screen reader application, or other program that allow users that are hard of hearing, visually impaired, or otherwise restricted in some way to perform common actions within an application through an alternative to the traditional mouse and keybaord is using this framework to communicate with and control other applications.

Because of this, there is generally a focus on the user interface elements of an application and not spcifically on the actions possible by the application. User interface elements might be menus and their specific menu items, windows and the input fields or buttons within them, etc. By using the accessibility API, we are able to simulate the user interacting with these elements -- we can only tell the application to do something if there is currently a user interface object available that would allow the user to initiate the action.

Thankfully, much of this accessibility framework is embedded automatically in the common objects that are used when developers create an application -- for basic window manipulation, selecting of menu items, triggering of buttons, etc. the developers do not have to add any additional code -- the support for accessing these through the accessibility API is automatically provded unless explicitely overrideen or changed by the developer.

Because of this, the good news is that as long as the application is designed using the Cocoa frameworks and uses the Apple defined classes for user interface elements (or subclasses), the organization and existence of accessibility elements will usually follow some well defined patterns no matter what type of application you're trying to work with. The bad news is that for applications ported from other environments that don't directly map user interface elements to their Cocoa framework counterparts, our ability to control the application may be limited if the developers don't add their own accessibility support.

It should also be noted that developers can override this built in support, even if they do use the Cocoa frameworks (for example Password Managers will often restrict some accessibility functionality), so while this module allows you to manipulate most macOS applications, it is no more guaranteed to manipulate all of them than any other application providing assistive support.

Basic Organization of Accessibility Objects

Accssibility objects within a specific application follow a tree like structure with the application object itself being the root or initial node of the tree with branches terminating in the object representing specific user interface elements that the user interactis directly with (e.g. buttons, text fields, menu items, etc.).

Every accessibility object has attributes that describe the object. Many of these attributes are commonly found in all accessibility objects, but some will be specific to the type of accessibility object or even unique to the specific instance of the object. Common attributes that are almost always present include (but are not limited to):

  • AXChildren - a table array of 0 or more accessibility objects that descend from this one. For the application object, this will almost always contain one or more window objects and usually one or two menubar objects.
  • AXRole - a string specifying the role this object has within the application. A list of common roles (but not an exhastive list) can be found in the hs.axuielement.roles table.
  • AXSubrole - a string, or nil, which may further define the specifics of the role the object has. Not all elements define this, but it is common whith objects that may have many different appearance types (window and button objects, for example). A list of common subroles (but not an exhastive list) can be found in the hs.axuielement.subroles table.
  • AXFrame - a table containing position ans size keys (x, y, h, and w) specifying the location on the screen of this element.
  • AXParent - an accessibility object specifying the direct parent of this object. For the application object, this will be nil, but for all other objects, this will be the objects immediate ancestor. The hs.axuielement:path() method uses this attribute to determine the specific path from the application object to the object upon which the method is being performed including all intermediate objects in between.

There are many more attributes that may be present. A list of common attributes (but not an exhaustive list) can be found in the hs.axuielement.attributes table.

Naming conventions

It should be noted that by convention, all accessibility labels (attributes, actions, etc.) start with the prefix "AX". This is an Apple convention indicating that these labels belong to the accessibility portions of the Carbon AppKit framework, which provides the API which allows assitive applications, and Hammerspoon, to access the accessibility objects of other applications. It is uncommon, but possible, for an applications developer to provide attributes, actions, etc. that do not follow this convention, so when it doubt about what possibilities an object may offer, consult the hs.axuielement:actionNames(), hs.axuielement:attributeNames(), and hs.axuielement:parameterizedAttributeNames() methods to see what has been specifically defined for the given object.

Where to Next?

Home | Tutorials | Sample Configurations

Table of Contents

What's New?

Modules

Developer

Clone this wiki locally