04. Elements
An element is a node from the whole UIA tree, that might itself have child nodes. An element can be a control like a Button, Edit, CheckBox, which can also have children elements: for example a Button element may have a child for an image or text inside the button. An element can also simply be a container for other elements, like a Pane or Table.
With newer UIAutomation versions, additional UIA_Element versions have been added to supplement the original UIA_Element with more properties and methods. This means that if you are running an older Windows, then all the newer versions might not be available (eg UIA_Element7). By default when an element is created, it is done with the highest available version. To check the version, use UIA_Element.__Version. To change the highest used version, modify UIA_Enum.UIA_CurrentVersion_Element value (by default is set to 7).
With elements we can mainly do three things:
- Get information about the element using properties: name, size, location, type etc.
- Do simple interactions such as focusing the element or clicking it (more complicated interactions are done using Patterns)
- Find new elements: using Find methods we can locate elements of interest from our starting point (such as getting all Window elements (open windows) from the root element), using Wait methods we can wait elements to appear.
Information about an element can be accessed through properties. There are two types of properties: element properties and control pattern properties.
The automation element properties consist of a common set of properties, such as Name, AcceleratorKey, and ClassName, that are exposed by all UIAutomation elements, regardless of the control type. Usually element properties do not change (eg the name stays the same).
Exists*, Value*, ProcessId, ControlType, LocalizedControlType, Name, AcceleratorKey, AccessKey, HasKeyboardFocus, IsKeyboardFocusable, IsEnabled, AutomationId, ClassName, HelpText, Culture, IsControlElement, IsContentElement, IsPassword, NativeWindowHandle, ItemType, Orientation, FrameworkId, IsRequiredForForm, ItemStatus, BoundingRectangle, LabeledBy, AriaRole, AriaProperties, IsDataValidForForm, ControllerFor, DescribedBy, FlowsTo, ProviderDescription
* Exists and Value are custom properties of UIA_Interface, that are not present in the original set of properties. Value uses the ValuePattern to get or set the current value (element.Value or element.Value := "new value"). Exists checks whether the element exists or not.
** Additional methods and properties might be available in higher UIA_Element versions (discussed in the end of this article)
Most of the properties are words or numbers, but some return objects:
- BoundingRectangle returns an object containing the left, top, right, and bottom boundaries of the element in screen coordinates. They can be accessed with
BoundingRectangle .l, BoundingRectangle.t, BoundingRectangle.r, BoundingRectangle.b. An easier way to get the coordinates is the GetCurrentPos method. - LabeledBy returns an element
- ControllerFor, DescribedBy, FlowsTo return an array of elements
Properties can be accessed with element.Property which is the same as element.CurrentProperty:
element := UIA.ElementFromHandle(WinExist("Notepad"))
MsgBox, % element.Name
For cached properties, the property name needs to be preceded by "Cached": element.CachedName
Control pattern properties are supported for elements which have a particular control pattern available. Each control pattern has a corresponding set of control pattern properties that the control must expose. For example, a control that supports the GridPattern should have the ColumnCount and RowCount properties, or ValuePattern has the Value property. Most control pattern properties are dynamic values (they might change with user interaction).
Control pattern properties are accessible with the elements GetCurrentPropertyValue or GetCachedPropertyValue methods. To use the GetCurrentPropertyValue, we need to provide the name or id of the property. These kinds of enumerations (a pair of a name and a corresponding id) are available in the UIA_Enum class.
To find the property ids, open the UIA_Interface.ahk file and search for module UIA_PropertyIds. There we see that for example UIA_NamePropertyId corresponds to the id 30005.
We have a few ways available to GetCurrentPropertyValue of a property:
-
element.RangeValueValue- although UIA originally intended these properties to be accessed with GetCurrentPropertyValue, UIA_Interface.ahk also allows accessing them like normal element properties -
element.GetCurrentPropertyValue("Name")- we can provide the property name to get the value -
element.GetCurrentPropertyValue(30005)- we can provide the id numeric value directly -
element.GetCurrentPropertyValue(UIA_Enum.UIA_NamePropertyId)- gets the id from the UIA_Enum class -
element.GetCurrentPropertyValue(UIA_Enum.UIA_PropertyId("Name"))- gets the id from the name using the UIA_Enum UIA_PropertyId method
RuntimeId, BoundingRectangle, ProcessId, ControlType, LocalizedControlType, Name, AcceleratorKey, AccessKey, HasKeyboardFocus, IsKeyboardFocusable, IsEnabled, AutomationId, ClassName, HelpText, ClickablePoint, Culture, IsControlElement, IsContentElement, LabeledBy, IsPassword, NativeWindowHandle, ItemType, IsOffscreen, Orientation, FrameworkId, IsRequiredForForm, ItemStatus, IsDockPatternAvailable, IsExpandCollapsePatternAvailable, IsGridItemPatternAvailable, IsGridPatternAvailable, IsInvokePatternAvailable, IsMultipleViewPatternAvailable, IsRangeValuePatternAvailable, IsScrollPatternAvailable, IsScrollItemPatternAvailable, IsSelectionItemPatternAvailable, IsSelectionPatternAvailable, IsTablePatternAvailable, IsTableItemPatternAvailable, IsTextPatternAvailable, IsTogglePatternAvailable, IsTransformPatternAvailable, IsValuePatternAvailable, IsWindowPatternAvailable, ValueValue, ValueIsReadOnly, RangeValueValue, RangeValueIsReadOnly, RangeValueMinimum, RangeValueMaximum, RangeValueLargeChange, RangeValueSmallChange, ScrollHorizontalScrollPercent, ScrollHorizontalViewSize, ScrollVerticalScrollPercent, ScrollVerticalViewSize, ScrollHorizontallyScrollable, ScrollVerticallyScrollable, SelectionSelection, SelectionCanSelectMultiple, SelectionIsSelectionRequired, GridRowCount, GridColumnCount, GridItemRow, GridItemColumn, GridItemRowSpan, GridItemColumnSpan, GridItemContainingGrid, DockDockPosition, ExpandCollapseExpandCollapseState, MultipleViewCurrentView, MultipleViewSupportedViews, WindowCanMaximize, WindowCanMinimize, WindowWindowVisualState, WindowWindowInteractionState, WindowIsModal, WindowIsTopmost, SelectionItemIsSelected, SelectionItemSelectionContainer, TableRowHeaders, TableColumnHeaders, TableRowOrColumnMajor, TableItemRowHeaderItems, TableItemColumnHeaderItems, ToggleToggleState, TransformCanMove, TransformCanResize, TransformCanRotate, IsLegacyIAccessiblePatternAvailable, LegacyIAccessibleChildId, LegacyIAccessibleName, LegacyIAccessibleValue, LegacyIAccessibleDescription, LegacyIAccessibleRole, LegacyIAccessibleState, LegacyIAccessibleHelp, LegacyIAccessibleKeyboardShortcut, LegacyIAccessibleSelection, LegacyIAccessibleDefaultAction, AriaRole, AriaProperties, IsDataValidForForm, ControllerFor, DescribedBy, FlowsTo, ProviderDescription, IsItemContainerPatternAvailable, IsVirtualizedItemPatternAvailable, IsSynchronizedInputPatternAvailable, OptimizeForVisualContent, IsObjectModelPatternAvailable, AnnotationAnnotationTypeId, AnnotationAnnotationTypeName, AnnotationAuthor, AnnotationDateTime, AnnotationTarget, IsAnnotationPatternAvailable, IsTextPattern2Available, StylesStyleId, StylesStyleName, StylesFillColor, StylesFillPatternStyle, StylesShape, StylesFillPatternColor, StylesExtendedProperties, IsStylesPatternAvailable, IsSpreadsheetPatternAvailable, SpreadsheetItemFormula, SpreadsheetItemAnnotationObjects, SpreadsheetItemAnnotationTypes, IsSpreadsheetItemPatternAvailable, Transform2CanZoom, IsTransformPattern2Available, LiveSetting, IsTextChildPatternAvailable, IsDragPatternAvailable, DragIsGrabbed, DragDropEffect, DragDropEffects, IsDropTargetPatternAvailable, DropTargetDropTargetEffect, DropTargetDropTargetEffects, DragGrabbedItems, Transform2ZoomLevel, Transform2ZoomMinimum, Transform2ZoomMaximum, FlowsFrom, IsTextEditPatternAvailable, IsPeripheral, IsCustomNavigationPatternAvailable, PositionInSet, SizeOfSet, Level, AnnotationTypes, AnnotationObjects, LandmarkType, LocalizedLandmarkType, FullDescription, FillColor, OutlineColor, FillType, VisualEffects, OutlineThickness, CenterPoint, Rotation, Size, IsSelectionPattern2Available, Selection2FirstSelectedItem, Selection2LastSelectedItem, Selection2CurrentSelectedItem, Selection2ItemCount, IsDialog
Every element has methods available to interact with the element (in limited ways, most interaction is done with patterns), to get properties, and to find next elements of interest.
The following are methods of interaction available for every element. More specific ways of interaction (eg. setting values, toggling etc) are done with patterns.
SetFocus() sets focus to the element. This usually also activates the window
SetValue(val, pattern="") sets the value of the element using ValuePattern or LegacyIAccessible by default. A specific pattern can be specified (eg RangeValuePattern).
Click(WhichButtonOrSleepTime="", ClickCountAndSleepTime=1, DownOrUp="", Relative="")
- Click the element using one of the available click-like methods (InvokePattern Invoke(), TogglePattern Toggle(), ExpandCollapsePattern Expand() or Collapse() (depending on the state of the element), SelectionItemPattern Select(), or LegacyIAccessible DoDefaultAction()), in which case ClickCount is ignored.
- If WhichButton is specified (for example "left", "right") then the AutoHotkey Click function will be used to click the center of the element.
- If WhichButton is a number, then Sleep will be called with that number. Eg Click(200) will sleep 200ms after clicking
- If ClickCountAndSleepTime is a number >=10, then Sleep will be called with that number. To click 10+ times and sleep after, specify "ClickCount SleepTime". Ex: Click("left", 200) will sleep 200ms after clicking. Ex: Click("left", "20 200") will left-click 20 times and then sleep 200ms.
- If Relative is "Rel" or "Relative" then X and Y coordinates are treated as offsets from the current mouse position. Otherwise it expects offset values for both X and Y (eg "-5 10" would offset X by -5 and Y by +10).
ControlClick(WinTitleOrSleepTime="", WinTextOrSleepTime="", WhichButton="", ClickCount="", Options="", ExcludeTitle="", ExcludeText="") ControlClicks the element after getting relative coordinates with GetClickablePointRelativeTo("window"). Specifying WinTitle makes the function faster, since it bypasses getting the Hwnd from the element. If WinTitle or WinText is a number, then Sleep will be called with that number of milliseconds. Ex: ControlClick(200) will sleep 200ms after clicking. Same for ControlClick("ahk_id 12345", 200)
These can be used to get next elements from the starting point element (see "UIA tree" section of this Wiki).
- To find one element (the first one), use the FindFirst methods. The most convenient ones are FindFirstByName, FindFirstByType, FindFirstByNameAndType and FindFirstBy. For example, to find the first element with the name "Skype" from the root element, we can simply do
UIA.GetRootElement().FindFirstByName("Skype"). FindFirstBy can be used to locate elements by any property, such as AutomationId:element.FindFirstBy("AutomationId=myautomationid")will locate the first element with AutomationId of "myautomationid". FindFirstBy can also be used to combine conditions:element.FindFirstBy("Name=Skype AND ControlType=Window"). - To find all the elements matching a condition, use the FindAll methods.
- To wait an element to exist, use the WaitElementExist methods WaitElementExistByName, WaitElementExistByType, WaitElementExistBy etc. If the element is found, then the method also returns the element, so it's possible to do
if (foundElement := element.WaitElementExistByName("Skype")) { .... - To wait an element not to exist, use WaitElementNotExist.
- To get an element relative to the position inside the UIA tree, use either TreeWalkers or FindByPath.
element.FindByPath("+1")returns the next sibling element,element.FindByPath("-2")the previous sibling from the previous sibling,element.FindByPath("2")the second child,element.FindByPath("P2")the parent of the parent.
FindFirst(c="", scope=0x4) FindAll(c="", scope=0x4) FindFirst retrieves the first child or descendant element that matches the specified condition, FindAll returns all matching elements. To learn more about conditions, check out the relevant section in this Wiki. scope must be one of TreeScope enumerations (default is TreeScope_Descendants := 0x4).
FindFirstBy(expr, scope=0x4, matchMode=3, caseSensitive=True) FindAllBy(expr, scope=0x4, matchMode=3, caseSensitive=True) Uses FindFirst or FindAll, but creates the condition from an expression.
expr:
Takes a value in the form of "PropertyId=matchvalue" to match a specific property with the value matchValue. PropertyId can be most properties from UIA_Enum.UIA_PropertyId method (for example Name, ControlType, AutomationId etc).
Example1: "Name=Username:" would use FindFirst with UIA_Enum.UIA_NamePropertyId matching the name "Username:"
Example2: "ControlType=Button would FindFirst using UIA_Enum.UIA_ControlTypePropertyId and matching for UIA_Enum.UIA_ButtonControlTypeId. Alternatively "ControlType=50000" can be used (direct value for UIA_ButtonControlTypeId which is 50000)
Criteria can be combined with AND, OR, &&, ||:
Example3: "Name=Username: AND ControlType=Button" would FindFirst an element with the name property of "Username:" and control type of button.
Flags can be modified for each individual condition by specifying FLAGS=n after the condition (and before and/or operator). 0=no flags; 1=ignore case (case insensitive matching); 2=match substring; 3=ignore case and match substring
If matchMode==3 or matching substrings is supported (Windows 10 17763 and above) and matchMode==2, then parentheses are supported.
Otherwise parenthesis are not supported, and criteria are evaluated left to right, so "a AND b OR c" would be evaluated as "(a and b) or c".
Negation can be specified with NOT:
Example4: "NOT ControlType=Edit" would return the first element that is not an edit element
scope:
Scope by default is UIA_TreeScope_Descendants.
matchMode:
If using Name PropertyId as a criteria, this follows the SetTitleMatchMode scheme:
1=name must must start with the specified name
2=can contain anywhere
3=exact match
RegEx=using regular expression. In this case the Name can't be empty.
caseSensitive:
If matching for a string, this will specify case-sensitivity.
FindFirstByName(name, scope=0x4, matchMode=3, caseSensitive=True) FindAllByName(name, scope=0x4, matchMode=3, caseSensitive=True) FindFirst/FindAll using UIA_NamePropertyId. "scope" is search scope, which can be any of UIA_Enum TreeScope values. "MatchMode" has same convention as window TitleMatchMode: 1=needs to start with the specified name, 2=can contain anywhere, 3=exact match, RegEx=regex match.
FindFirstByType(controlType, scope=0x4) FindAllByType(controlType, scope=0x4) FindFirst/FindAll using UIA_ControlTypeId. controlType can be the ControlTypeId numeric value, or in string form (eg "Button")
FindFirstByNameAndType(name, controlType, scope=0x4, matchMode=3, caseSensitive=True) FindAllByNameAndType(name, controlType, scope=0x4, matchMode=3, caseSensitive=True) FindFirst/FindAll using UIA_NamePropertyId and UIA_ControlTypeId. controlType can be the ControlTypeId numeric value, or in string form (eg "Button"). scope is search scope, which can be any of UIA_Enum TreeScope values. matchMode has same convention as window TitleMatchMode: 1=needs to start with the specified name, 2=can contain anywhere, 3=exact match, RegEx=regex match
FindByPath(searchPath="", c="") FindByPath gets an element by a relative "path" from a starting element.
- To get the nth child of the starting element, set searchPath to "n". To get a deeper node, separate the numbers with a ".": "2.1" will get the second childs first child. This kind of path can easily be got from the UIA_Element.DumpAll() method, which returns a path in the same style (this is like the Acc path but for UIA, they are not compatible!).
- To get the nth parent of the starting element, use "Pn": "P2" will get the parent of the parent.
- To get sibling elements, put a "+" or "-" in front of "n": +2 will get the next sibling from the next sibling (calling GetNextSiblingElement twice). Using this after "Pn" doesn't require a "." separator ("P2.-1" == "P2-1").
These conditions can also be combined: searchPath="P1-1.1.1.2" -> gets the parent element, then the previous sibling element of the parent, and then "1.1.2" gets the second child of the first childs first child.
c or condition argument can be used to only filter elements specified by the condition: UIA_Element.FindByPath("+2", UIA_Interface.CreateCondition("ControlType", "Button")) will only consider "Button" controls and gets the second sibling button.
WaitElementExist(expr, scope=0x4, matchMode=3, caseSensitive=True, timeOut=10000) WaitElementExistByName(name, scope=0x4, matchMode=3, caseSensitive=True, timeOut=10000) WaitElementExistByType(controlType, scope=0x4, timeOut=10000) WaitElementExistByNameAndType(name, controlType, scope=0x4, matchMode=3, caseSensitive=True, timeOut=10000) WaitElementExist, WaitElementExistByName, WaitElementExistByType, WaitElementExistByNameAndType call corresponding FindFirst functions (FindFirstBy, FindFirstByName etc) until the element is found, with a default timeout of 10 seconds.
WaitElementNotExist(expr, scope=0x4, matchMode=3, caseSensitive=True, timeOut=10000)
This method only uses an expression, so for instance to wait for an element with the name "MyElement" to disappear, use startingElement.WaitElementNotExist("Name=MyElement")
GetRuntimeId() gets an array containing an unique identifier for the current instance of the element. If the element is destroyed and created again, then this value will change.
GetCurrentPropertyValue(propertyId) retrieves the current value of a property for this element.
GetCurrentPattern(pattern) retrieves a UIA_Pattern object of the specified control pattern on this element. If a full pattern name is specified then that exact version will be used (eg "TextPattern" will return a UIA_TextPattern object), otherwise the highest version will be used (eg "Text" might return UIA_TextPattern2 if it is available). usedPattern will be set to the actual string used to look for the pattern (used mostly for debugging purposes).
GetClickablePoint() retrieves the physical screen coordinates of a point on the element that can be clicked. Not every element is clickable, and some elements may have mechanisms to imitate being "clickable" (for example in webpages Javascript can be used to make text clickable) but not have a clickable point available.
GetClickablePointRelativeTo(relativeTo="") is a wrapper for GetClickablePoint(), where additionally the coordinates are converted to relative coordinates. relativeTo can be window, screen or client, default is A_CoordModeMouse
GetParentHwnd() gets the parent window handle from the element.
GetCurrentPos(relativeTo="") returns an object containing the x, y coordinates and width and height for the element: {x:x coordinate, y:y coordinate, w:width, h:height}. relativeTo can be client, window or screen, default is A_CoordModeMouse.
Highlight(displayTime:=2000, color:="Red", d:=4) draws a rectangle around the elements borders, which can be used for debugging purposes.
Control patterns provide a way to interact with the element in a more specific way. For example, you use the InvokePattern for controls that can be invoked (such as buttons) and the ScrollPattern for controls that have scroll bars (such as list boxes, list views, or combo boxes). One element can have multiple patterns available, and child elements might not have the same patterns as the parent element.
More comprehensive information is available at Microsoft documentation.
Invoke, Selection, Value, RangeValue, Scroll, ScrollItem, ExpandCollapse, Grid, GridItem, MultipleView, Window, SelectionItem, Dock, Table, TableItem, Text, Toggle, Transform, LegacyIAccessible, ItemContainer
This isn't an exhaustive list of available patterns, a more complete list is available in UIA_Enum class UIA_PatternIds.
Patterns can be accessed in two ways:
GetCurrentPattern(pattern, ByRef usedPattern="") takes in the pattern in string or numeric form. For example element.GetCurrentPattern("Value") would return a new UIA_ValuePattern object, which can be used to access all ValuePattern methods and properties. Similarly we could provide UIA_Enum.UIA_ValuePatternId (which has the value 10002) as input and get the same result.
Some patterns have multiple versions available: TextPattern has UIA_TextPattern and UIA_TextPattern2, where the latter provides some extra methods along with all UIA_TextPattern methods. GetCurrentPattern by default returns the highest version available: element.GetCurrentPattern("Text") would return UIA_TextPattern2 if that is implemented by the element. To get a specific pattern version, include "Pattern" to the end: element.GetCurrentPattern("TextPattern") will get exactly UIA_TextPattern.
usedPattern is set to the pattern actually used (to know which version of the pattern was used).
Patterns can also be gotten in property style: element.ValuePattern will return the UIA_ValuePattern object. With this method we always need to specify the patterns version.
For more information about patterns, check out the Patterns section in this Wiki.
Properties
- OptimizeForVisualContent
- LiveSetting
- FlowsFrom : returns an array of elements
Methods
UIA_Element2 doesn't have additional methods
Properties
- IsPeripheral
Methods ShowContextMenu()
Properties
- PositionInSet
- SizeOfSet
- Level
- AnnotationTypes
- AnnotationObjects : array of elements
Methods
UIA_Element4 doesn't have additional methods.
Properties
- LandmarkType
- LocalizedLandmarkType
Methods
UIA_Element5 doesn't have additional methods.
Properties
- FullDescription
Methods
UIA_Element6 doesn't have additional methods.
Properties
UIA_Element7 doesn't have additional properties.
Methods
- FindFirstWithOptions(scope, c, traversalOptions=0, root=0) finds the first matching element in the specified order. traversalOptions must be one of TreeTraversalOptions enums. [optional] root is pointer to the element with which to begin the search.
- FindAllWithOptions(scope, c, traversalOptions=0, root=0)
- FindFirstWithOptionsBuildCache(scope, c, cacheRequest, traversalOptions=0, root=0)
- FindAllWithOptionsBuildCache(scope, c, cacheRequest, traversalOptions=0, root=0)
- GetCurrentMetadataValue(targetId, metadataId)