08. Common pitfalls; tips and tricks
Preamble: using UIA can sometimes prove quite tricky, and many bugs or unforeseen erratic behaviours can be encountered. This is for multiple reasons:
- The userbase for UIA is quite niche and small, and the amount of people testing it is also small, so more bugs slip by.
- Originally UIA was created accessibility (people with vision or hearing problems), and accordingly some parts of UIA get more attention than others: patterns sometimes don't work as expected, methods like ElementFromHandle or TreeWalkers might not always give the expected results etc.
- UIA properties/methods sometimes need to be implemented by the program creators themselves, who might not pay much (or any) attention to accessibility. Most common controls (buttons, checkboxes etc) work fine, but specialized controls might not be accessible at all. Also some types of windows are sometimes very poorly accessible, for example Qt windows. If you encounter any weird bugs or issues, be sure to raise a issue on this GitHub page, or the AutoHotkey forums thread.
These days a lot of newer windows are actually not the classic Win32 windows most AutoHotkey functions were meant for (like ControlClick), but instead are browsers in disguise, using the Chromium platform to display their content. This includes Chrome, Edge, apps like Spotify, Skype, Teams, and many more. Chromium might not have accessibility enabled by default, which means that regular ElementFrom... methods won't work and the content of the app will not show. We can send Chromium the signal for accessibility by sending a special message to it (WM_GETOBJECT to Chrome_RenderWidgetHostHWND1), which activates UIA and makes the content accessible. UIA_Interface ElementFrom... methods (ElementFromHandle etc) do this by default since there are just so many Chromium/Electron apps, but for performance reasons this can be turned off by setting activateChromiumAccessibility to False.
Sometimes Chromium-based windows cannot be accessed like normal with ElementFromHandle and the content isn't accessible. In this case you will need to use ElementFromChromium:
#include <UIA_Interface>
UIA := UIA_Interface()
skypeEl := UIA.ElementFromChromium("ahk_exe skype.exe")
MsgBox, % skypeEl.DumpAll()
If accessing the window only once then ElementFromHandle/ElementFromPoint can be used as well, since on the first run the activateChromiumAccessibility will return the Chromium element:
#include <UIA_Interface>
UIA := UIA_Interface()
winEl := UIA.ElementFromHandle("ahk_exe skype.exe", skypeEl)
MsgBox, % winEl.DumpAll() ; will not show the content
MsgBox, % skypeEl.DumpAll() ; WILL show the content, but only on the first run (since UIA will be activated only once per run)
; The following will NOT work, since it would be ran for a second time. Use ElementFromChromium for multiple accesses
; winEl := UIA.ElementFromHandle("ahk_exe skype.exe", skypeEl)
; MsgBox, % skypeEl.DumpAll()
- If a window is minimized it is not accessible by UIA, so consider adding a check for that (with WinGet).
- Some programs hide their UIA tree when the window is not visible to save memory. Chromium applications in particular do this, so be careful to activate the window before using UIA calls.
For performance reasons, when a new UIA_Browser() is called, multiple browser-specific elements are pre-fetched to be used at a later time: URL bar, the main document element etc. Depending on the browser, this might take a relatively long time (for example in Opera, Firefox) and it is recommended to avoid creating this object multiple times if the window is not closed in the meantime. If none of the browser-specific methods (such as navigating) are used, then it might be better not to use UIA_Browser at all.
Usually an element can be found using Name, ControlType, AutomationId, ClassName, or a combination of these. For example, there might be multiple elements with the name "Username", but only one Edit field with that name, so FindFirstByNameAndType("Username", "Edit") can be used. In some cases though, the element really doesn't have anything to discern it by. In these cases there are a few options:
- First find a reference element, then move in the tree with FindByPath to the element of interest. For example, if a webpage has a Text element with the name "Username:" but the corresponding Edit field doesn't have a unique name, and if the Edit element is the next sibling for the Text element in the UIA tree, then use
element.FindFirstByNameAndType("Username:", "Text").FindByPath("+1"). - Find a container element inside the main element in which FindFirst will return your element of interest. For example a window might have many elements with the name "Username:", but if the element of interest is inside a container named "Login fields", then first find the container and then use FindFirst on that container (instead of the window element).
- Locate the element by path with FindByPath. This isn't recommended though, because if the structure of the window changes (for example with a software update, or the user changes it), then the path will also change.
UIA is generally slower than MSAA/Acc, but if properly used it should be faster than a human. The main advice here is to limit searches as much as possible, and avoid FindAll whenever possible.
- The bigger the tree, the slower the search: big windows with thousands of elements are slow to search through, so perhaps a branch of the tree could be searched instead?
- The bigger the search scope, the slower the search: try to use as small a scope as possible. The default for Find methods is TreeScope_Descendants=0x4 (the elements whole subtree), but perhaps TreeScope_Children=0x2 could be used instead?
- Use FindFirst instead of FindAll if possible, since FindAll will always have to go through the whole UIA tree, but FindFirst can finish when it finds a match (which could be much earlier).
- Use a smaller scope. The default is TreeScope_Descendants=0x4, but in some cases this might be very slow. For example if the first branch of the tree is huge but the second branch very small, and your element of interest is in the second branch, it might be better to first get that second branch with FindFirst TreeScope_Children=0x2 or TreeWalker, and then use FindFirst on that.
- If you plan on running multiple searches on the same tree or access properties multiple times, you might get a speed improvement by using caching.
- Sometimes you might have uses for the newer FindFirstWithOptions method, where you can specify the search direction with TreeTraversalOptions (so you could instead of using FindFirst create a FindLast method).
Using UIAViewer or Accessibility Insights make sure that the element is actually accessible by UIA, and check what properties, methods, and patterns it has available. If working with a Chromium app, make sure you set activateChromiumAccessibility to True and that the content is accessible with UIAViewer "Construct tree for Whole window" button (or UIATreeInspector), since elements are usually accessible with UIAViewers ElementFromPoint, but might not be with ElementFromHandle.
- Make sure with UIAViewer that you have the correct data for your search condition: for example when finding by name, "Value" is not the same as "Value " with a space at the end.
- Use element.Dump(), DumpAll(), or Highlight() methods to check if the element you got is the correct one
- If working with patterns, make sure that you actually got the pattern by checking for a property of the pattern.
If trying to use pattern methods (like InvokePattern Invoke()), then make sure the method actually works with the element using Accessibility Insights or UIAViewer. Sometimes the pattern shows as "available", but isn't properly implemented and doesn't work.