02. UIA tree
The UIAutomation framework holds the entire desktop with all the windows and controls inside a big tree consisting of elements. At the top is the root element: the Desktop itself. The Desktop elements direct children are all the windows (Notepad, Chrome, etc), then in turn the windows have child elements (toolbars, main content, buttons, etc) and so on.
The UIAutomation tree is not a fixed structure. For example if a window closes, its element with all its child elements are destroyed. When a webpage reloads or navigates to a new page, the tree for the browser window also updates with the changes. Sometimes parts of the tree are not available right away, but only constructed when the client needs them (for example hovers over the element with the mouse).
Accessing the whole tree is usually undesirable because it contains thousands of elements, and searching through it for our elements of interest might take a very long time. Usually we try to get a part of the tree, for example a window element.
There are multiple ways of getting elements from the tree, and here we go over some of the more important ones. To access the tree, we first need the UIA_Interface object. We can do that by calling the UIA_Interface() function which returns us exactly that. Usually we assign it to a variable for easier access: UIA := UIA_Interface(). Read more about this in the UIA interface section of this Wiki.
UIA_Interface.GetRootElement()
Calling UIA.GetRootElement will get us the root, or Desktop element. This contains the whole desktop. If there are multiple desktops, then only the currently visible one is accessible.
UIA_Interface.ElementFromHandle(hwnd, activateChromiumAccessibility=True)
ElementFromHandle takes two arguments:
- hwnd: this is the handle for the window of interest, and it is returned for example by the WinExist function. So to get the element for the Notepad window, we might do
notepadEl := UIA.ElementFromHandle(WinExist("ahk_exe notepad.exe")) - activateChromiumAccessibility: some windows are actually web applications based on Chromium, the same thing Chrome and Edge browsers are based upon. Sometimes they do not activate accessibility for UIA automatically, so the ElementFromHandle might need to send additional signals to the window. If activateChromiumAccessibility is set to True then it does that (which is the default value).
UIA_Interface.ElementFromPoint(x="", y="", activateChromiumAccessibility=True)
ElementFromPoint does what the name suggests: gets an element from screen coordinates. If no coordinates are provided, the current mouse position is used. activateChromiumAccessibility does the same as in ElementFromHandle.
NOTE: AutoHotkey MouseGetPos returns coordinates by default relative to the active window. Make sure to change that to screen coordinates with CoordMode, Mouse, Screen
UIA_Interface.SmallestElementFromPoint(x="", y="", activateChromiumAccessibility=True, windowEl="")
Gets ElementFromPoint and filters out the smallest child element that is under the point. If windowEl (element for the window under the point) is provided, then a deep search is performed for the smallest element, iterating over all the elements in the window to find the smallest one under the point (this might be very slow in large trees).
UIA_Interface.GetFocusedElement()
Retrieves the element that has the input focus, like a textbox or a button. The element might still have child elements as well.
UIA_Interface.ElementFromIAccessible(IAcc, childId=0)
Gets an element from a MSAA/Acc object (the predecessor for UIA). This only works with Acc objects where the window natively implements IAccessible, which excludes most Win32 windows (like Notepad, Windows Explorer etc). An example with Skype:
#include <Acc>
#include <UIA_Interface>
oAcc := Acc_Get("Object", "4", 0, "ahk_exe skype.exe")
MsgBox, % "AccName: " oAcc.accName(0) "`nObject type: " ComObjType(oAcc, "Name")
MsgBox, % UIA_Interface().ElementFromIAccessible(oAcc).Dump()
The use of this method is discouraged, since UIA usually has more reliable ways of getting elements available.
Moving from one element to another inside the tree can generally be done in two ways: either using Find methods from UIA_Element object, or walking the tree with a TreeWalker. In either case a new element (or array of elements) is returned, which in turn may contain more elements. For example we might FindFirst a content element, which in turn contains all the buttons, text etc. A more comprehensive explanation of moving in the tree and getting elements from a starting point element is provided in the Elements and TreeWalker sections in this Wiki.