Inspecting

Inspecting The User Interface

Inspecting the UI is the simplest thing that you can do with AXElements. There are many nooks and crannies that are interesting to talk about, so this document covers the core concepts that you will need to understand before you can begin discovering them yourself.

The AXAPI Abstraction

The Accessibility APIs (AXAPI) expose the system GUI via a series of trees (i.e. a forest). The trees in the forest are the various apps that are currently running, as well as a special tree that represents the whole system. The nodes in the tree are all AXAPI objects that have attributes and actions.

The root node of each tree is an AXAPI object representing the application. The application object's attributes contain some metadata about the application and offers some application wide actions. The most important attribute for an application, or any other AXAPI object are the children AXAPI objects, namely the menu bar and windows for the application.

The children attribute is what allows you to navigate downwards in the tree. Conversely, the parent attribute allows you to navigate upward in the tree. As you navigate a UI tree, you will find that some nodes represent literal objects such as buttons, while other nodes represent more conceptual objects such as a grouping or area. In actuality, the tree's structure is completely arbitrary based on what the app developer has decided to report to the AXAPIs; however, most apps will use the defaults provided by AppKit and "Just Work"™ the way you would expect.

As shown in the example above, a sample hierarchy would start with the application, which has a menu bar as one of its children and also a window. The focused application, Terminal, has a window that has a FullScreenButton as a child. The button has at least one action which is indicated by the boxed shape of the node. The windows in the application have other children as well but the tree becomes very large at that point and would not be easily displayed.

Not only is displaying a very large tree a problem, but navigating would also be a pain. The good news is that AXElements has the techniques and patterns for simplifying tree navigation that you get everywhere else, and maybe some new tricks. To get started, we need to get the root of a tree, an application object. There are several ways to get an application object, but the easiest way is to instantiate a new AX::Application object.

    app = AX::Application.new "Finder"

Accessibility Inspector

To quickly navigate through the UI tree, Apple has provided an Accessibility Inspector as part of their Developer Tools set. The inspector allows you to inspect part of the accessibilty hierarchy in real time with the mouse pointer. AXElements provides ways to quickly inspect a snapshot of a tree and display output as a graph or plain text, but real time updating is much easier with the inspector tool. It is worth playing around with the inspector to get a feel for what AXAPIs are capable of.

Attributes

Each AX object has attributes; buttons have a title, a position, a size, a pointer to the parent object, and a pointer to an array of children. The accessibility inspector will always show you the full list of attributes that an object has, in AXElements you can ask an object for its #attributes.

    puts app.attributes  # => [:role, :title, :children, :menu_bar, :windows, :main_window]

There are two ways to get the value of an attribute. The first way is to use #attribute and pass it the name of the attribute; but the better way to access attributes is to just treat the attribute name as a method on the object. For instance, the app object has a :main_window attribute and so it will also have a #main_window method.

    app.respond_to? :main_window  # => true
    app.main_window               # => #<AX::StandardWindow "AXElements.wiki" (0.0, 22.0) 4 children focused[✘]>

When you ask for the main window, you will get some kind of AX::Window object. In this case, it was an AX::StandardWindow, but it could have been a AX::FloatingWindow or some other type of window. You'll also notice that the description of the object is "AXElements.wiki" (0.0, 22.0) 4 children focused[✘], which is just some of the attribute values. The #inspect output for an AX object will try to include the values for only a few attributes that should give you a pretty good idea of what the object is; the screen position, (0.0, 22.0), is always included in #inspect output.

As a side note, if you have a hard time identifying something on screen, you can use the screen highlighter to identify the object. In standard AXElements, you can just use the highlight method.

    highlight app.main_window, timeout: 5

One last detail to note about attributes is predicate methods. As it is Ruby convention to append ? to methods which are predicates, AXElements will allow you to append ? to attribute names. As an example, the app has an :enhanced_user_interface attribute which returns true or false, so it would feel more Ruby-ish to append a ? to the method name like so:

if app.enhanced_user_interface?
  puts "Yup, it's enhanced."
else
  puts "What does it even mean to have an enhanced user interface?"
end

We can demonstrate how this all comes together with some more examples.

    window = app.main_window
    puts "The window's title is '#{window.title}'"

    button = window.children.find { |x| x.kind_of? AX::Button }

    if window == button.parent
      puts "Went in a circle."
    else
      puts "Something bad happened."
    end

The first line is familiar, we are just getting the main window for the app again. In the second line, we are printing out a string that tells us the name of the window. Easy peasy.

In the the third line, we get the children of the window and then search through them to find a button. While this is one way to navigate down the hierarchy, AXElements has simpler ways to navigate through the hierarchy, which is outlined in the Searching wiki.

The final lines in the example simply navigate back up in the tree to the parent element of the button and then check that the parent is the window. The parent of the button will be the same window mentioned earlier, and so you should see "Went in a circle." printed out.

Parameterized Attributes

If you've been using the accessibility inspector, you'll notice there is a section called "Parameterized Attributes" which is considered separate from regular attributes. These work just like regular attributes except that you need to provide a parameter to get the value. An example would be the :string_for_range parameterized attribute which clearly requires you to give a range, and would be akin to an array slice.

    title = app.main_window.title_ui_element
    title.static_text.string_for_range 0..5  # => "mrada"

As you can see, parameterized attributes are made available on an AX object in the same way as regular attributes. Just as with regular attributes, you can see which parameterized attributes are available by calling #parameterized_attributes on an AX object.

    app.parameterized_attributes    # => []
    title.parameterized_attributes  # => [:string_for_range, ...]

Setting Attributes

Some attributes are writable, such as the size of a window, or the value of a text area. The Accessibility Inspector will indicate writability by appending a (W) to the attribute name. You can ask an AX object if one of it's attributes are writable by calling #writable? and then set the attribute as you would expect in Ruby.

    window = app.main_window
    window.writable? :size  # => true
    window.writable? :position  # => true
    window.writable? :title   # => false

    window.size = [100, 100]
    sleep 1
    window.size = [500, 500]

More details about setting attributes is covered in the Acting wiki.

Reflection

AX objects support most, if not all, of the standard reflection methods that Ruby has to offer. Attributes and parameterized attributes will show up when you call #methods, and #respond_to? also works as expected.

However, in general, if you need to check if an object has an attribute before accessing the attribute, you may be doing something wrong, or should be using a lower level API such as #attribute.

A PSA About Attributes

The attributes that an object can have are quite arbitrary. Apple has documentation listing what you should expected each type of object to have, but developers can add new attributes if needed. It is also possible, though unlikely, to remove attributes.

Next Steps

Some attributes have actions which they can perform, such as a button press. You may wish to peruse the Acting wiki.

As mentioned above, searching is a very powerful feature that allows you to navigate the UI tree easily. You may wish to peruse the Searching wiki.