04. Elements
An element is a node from the whole UIA tree, that might itself have child nodes. An element can be a control like a Button, Edit, CheckBox, which can also have children elements: for example a Button element may have a child for an image or text inside the button. An element can also simply be a container for other elements, like a Pane or Table.
With elements we can mainly do three things:
- Get information about the element using properties: Name, Location, Type etc.
- Do simple interactions such as focusing the element or clicking it (more complicated interactions are done using Patterns)
- Find new elements: using FindElement methods we can locate elements of interest from our starting point (such as getting all Window elements (=open windows) from the root element), using WaitElement methods we can wait elements to appear (or disappear).
Information about an element can be accessed through properties. There are two types of properties: element properties and control pattern properties.
The automation element properties consist of a common set of properties, such as Name, AcceleratorKey, and ClassName, that are exposed by all UIAutomation elements, regardless of the control type. Usually element properties do not change (eg the name stays the same).
Location*, Length*, Parent*, Children*, Exists*, Value**, ProcessId, Type (short for ControlType), LocalizedControlType, Name, AcceleratorKey, AccessKey, HasKeyboardFocus, IsKeyboardFocusable, IsEnabled, AutomationId, ClassName, HelpText, Culture, IsControlElement, IsContentElement, IsPassword, NativeWindowHandle, ItemType, Orientation, FrameworkId, IsRequiredForForm, ItemStatus, BoundingRectangle, LabeledBy, AriaRole, AriaProperties, IsDataValidForForm, ControllerFor, DescribedBy, FlowsTo, ProviderDescription, OptimizeForVisualContent, LiveSetting, FlowsFrom, IsPeripheral, PositionInSet, SizeOfSet, Level, AnnotationTypes, AnnotationObjects, LandmarkType, LocalizedLandmarkType
* These are custom properties not in the original UIAutomation implementation. Location returns an object with the location of the element: {x, y, w, h}. Length returns the number of children the element has. Children returns an array of all direct children elements. Parent returns the parent element. Exists checks whether the element is still accessible.
** Most properties cannot be modifier, but Value supports setting the property: Element.Value returns the current value, Element.Value := "new value" sets the value.
Most of the properties are words or numbers, but some return objects:
- Location returns an object containing the x and y screen coordinates, width and height of the element, also the left, top, right, and bottom boundaries of the element (same as BoundingRectangle): {x, y, w, h} can be accessed as
Element.Location.x. To get relative coordinates, use GetPos method instead. - BoundingRectangle returns an object containing the left, top, right, and bottom boundaries of the element in screen coordinates. They can be accessed with
BoundingRectangle .l, BoundingRectangle.t, BoundingRectangle.r, BoundingRectangle.b. - LabeledBy returns an element
- ControllerFor, DescribedBy, FlowsTo return an array of elements
Properties can be accessed with Element.Property which is the same as Element.CurrentProperty:
element := UIA.ElementFromHandle("Notepad")
MsgBox Element.Name
For cached properties, the property name needs to be preceded by "Cached": Element.CachedName
Control pattern properties are supported for elements which have a particular control pattern available. Each control pattern has a corresponding set of control pattern properties that the control must expose. For example, a control that supports the GridPattern should have the ColumnCount and RowCount properties, or ValuePattern has the Value property. Most control pattern properties are dynamic values (they might change with user interaction).
Control pattern properties are accessible in the same way as normal properties: Element.ToggleState will return the ToggleState property value. Alternatively the GetPropertyValue method can be used with the corresponding property enumeration: Element.GetPropertyValue(UIA.Property.Name)
Control pattern properties are listed under the Patterns section of this Wiki, since they depend on which patterns are available.
Every element has methods available to interact with the element (in limited ways, most interaction is done with patterns), to get properties, and to find next elements of interest.
The following are methods of interaction available for every element. More specific ways of interaction (eg. setting values, toggling etc) are done with patterns.
SetFocus() sets focus to the element. This usually also activates the window
Click(WhichButton:="", ClickCount:=1, DownOrUp:="", Relative:="", NoActivate:=False)
Tries to click the element. The method depends on WhichButton variable: by default it is attempted to use any "click"-like methods, such as InvokePattern Invoke(), TogglePattern Toggle(), SelectionItemPattern Select().
- WhichButton : If WhichButton is left empty (default), then any "click"-like pattern methods will be used (Invoke(), Toggle(), Select() etc. If WhichButton is a number, then Sleep will be called afterwards with that number of milliseconds. Eg. Element.Click(200) will sleep 200ms after "clicking". If WhichButton is "left" or "right", then the native Click() will be used to move the cursor to the center of the element and perform a click.
- ClickCount : Is used if WhichButton isn't a number or left empty, that is if AHK Click() will be used. In this case if ClickCount is a number <10, then that number of clicks will be performed. If ClickCount is >=10, then Sleep will be called with that number of ms. Both ClickCount and sleep time can be combined by separating with a space. Eg. Element.Click("left", 1000) will sleep 1000ms after clicking. Element.Click("left", 2) will double-click the element Element.Click("left" "2 1000") will double-click the element and then sleep for 1000ms
- DownOrUp : If AHK Click is used, then this will either press the mouse down, or release it.
- Relative : If Relative is "Rel" or "Relative" then X and Y coordinates are treated as offsets from the current mouse position. Otherwise it expects offset values for both X and Y (eg "-5 10" would offset X by -5 and Y by +10 from the center of the element).
- NoActivate : If AHK Click is used, then this will determine whether the window is activated before clicking if the clickable point isn't visible on the screen. Default is no activating.
ControlClick(WhichButton:="left", ClickCount:=1, Options:="")
Uses ControlClick to click the element.
- WhichButton : determines which button to use to click (left, right, middle). If WhichButton is a number, then a Sleep will be called afterwards. Eg. ControlClick(200) will sleep 200ms after clicking.
- ClickCount : How many times to click. Default is 1.
- Options : Additional ControlClick Options (see AHK documentations).
These can be used to get next elements from the starting point element (see "UIA tree" section of this Wiki).
- To find one element, use the FindElement method with the appropriate condition (see "Conditions" in this Wiki). For example, to find the first element with the name "Skype" from the root element, we can simply do
UIA.GetRootElement().FindElement({Name:"Skype"}). - To find all the elements matching a condition, use the FindElements method.
- To wait an element to exist, use the WaitElement method. If the element is found, then the method also returns the element, so it's possible to do
if (foundElement := element.WaitElement({Name:"Skype"})) { .... - To wait the current element not to exist, use WaitNotExist.
- To get an element relative to the position inside the UIA tree, use either ElementFromPath, WalkTree, or TreeWalkers (see "TreeWalkers" section of this Wiki).
element.WalkTree("+1")returns the next sibling element,element.WalkTree("-2")the previous sibling from the previous sibling,element.WalkTree("2")the second child,element.WalkTree("P2")the parent of the parent.
FindElement(condition, scope:=4, index:=1, order:=0, startingElement:=0, cacheRequest:=0)
Retrieves the first child or descendant element that matches the specified condition. If no element is found, a TargetError is thrown.
- condition : The condition to filter with, or a callback function. The condition object additionally supports named parameters (eg
{Type:"Button", scope:"Children"}. If a callback function is provided, then a TreeWalker is used to traverse the tree which will be very slow: usually it's better to use callbacks with FindCachedElement(s). - scope : Optional TreeScope value: Element, Children, Family (Element+Children), Descendants, Subtree (=Element+Descendants). Default is Descendants.
- index : Looks for the n-th element matching the condition
- order: Optional. Custom tree navigation order, one of UIA.TreeTraversalOptions values [requires Windows 10 version 1703+]
- startingElement : Optional. Element with which to begin the search [requires Windows 10 version 1703+]
- cacheRequest : Optional. Cache request object
FindElements(condition, scope := 4, order:=0, startingElement:=0, cacheRequest:=0)
Returns all UI Automation elements that satisfy the specified condition. If no elements are found, an empty array is returned.
ElementExist(condition, scope:=4, index:=1, order:=0, startingElement:=0, cacheRequest:=0)
This is the same as FindElement, but if an element is not found then 0 is returned instead of throwing an error.
WaitElement(condition, timeout := -1, scope := 4, index := 1, order := 0, startingElement := 0, cacheRequest := 0)
Calls FindElement until the element is found, with an optional timeout value in milliseconds. Default is indefinite waiting.
WaitElementNotExist(condition, timeout := -1, scope := 4, index := 1, startingElement := 0, order := 0, cacheRequest := 0)
Wait an element to disappear: startingElement.WaitElementNotExist({Name:"TargetElement"})
ElementFromPath(path1[, path2, ...])
Tries to get an element from a path. If no element is found, an error is thrown.
Paths can be:
- Comma-separated numeric path that defines which path to travel down the tree. In addition to integer values, or TypeN which selects the nth occurrence of Type. Eg. Element.ElementFromPath("3,2") => selects Elements third childs second child Element.ElementFromPath("Button3,2") => selects Elements third child of type Button, then its second child
- UIA path copied from UIAViewer. Eg. Element.ElementFromPath("bAx3")
- A condition or conditions. In this case the provided conditions define the route of tree-traversal, by default with Scope Children. Eg. Element.ElementFromPath({Type:"Button"}, {Type:"List"}) => finds the first Button type child of Element, then the first List type child of that element
Same as ElementFromPath, but 0 is returned if no element is found.
WaitElementFromPath(path1[, path2, ..., timeout])
Wait element to appear at a path. The last argument may optionally be a timeout in milliseconds (default is indefinite wait).
Paths can be:
- Comma-separated numeric path that defines which path to travel down the tree. In addition to integer values, or TypeN which selects the nth occurrence of Type. Eg. Element.WaitElementFromPath("3,2", 2000) => waits for Elements third childs second child, timeout 2 seconds Element.WaitElementFromPath("Button3,2") => waits for Elements third child of type Button, then its second child
- UIA path copied from UIAViewer. Eg. Element.WaitElementFromPath("bAx3")
- A condition or conditions. In this case the provided conditions define the route of tree-traversal, by default with Scope Children. Eg. Element.WaitElementFromPath({Type:"Button"}, {Type:"List"}) => waits for the first Button type child of Element, then the first List type child of that element
WalkTree(searchPath, filterCondition?)
Traverses the tree purely using a TreeWalker that is created with an optional filter-condition (default is TrueCondition)
searchPath is a comma-separated path that defines the route of tree traversal:
n: gets the nth child (using GetFirstChildElement and GetNextSiblingElement)
+n: gets the nth next sibling
-n: gets the nth previous sibling
pn: gets the nth parent
"n": normalizes the element (returns the first parent or the element itself matching the filterCondition)
filterCondition can optionally be provided: a condition for tree traversal that selects only elements that match the condition.
Example: Element.ElementFromPath("p,+2,1") => gets the parent of Element, then the second sibling of the parent, then that siblings first child.
WaitNotExist(timeOut:=-1)
Waits for this element to not exist.
FindCachedElement(condition, scope:=4, index:=1, order:=0, startingElement:=0)
FindCachedElement can be used to find an element inside a cached tree, using only cached properties. Condition can also be a callback function. If no matching element is found, an error is thrown.
CachedElementElement(condition, scope:=4, index:=1, order:=0, startingElement:=0)
This is the same as FindCachedElement, but if no element is found then 0 is returned instead.
FindCachedElements(condition, scope:=4, order:=0, startingElement:=0)
Returns all UI Automation elements that satisfy the specified condition inside a cached tree, checking only cached properties. This is not a UIA native method: make sure the cached tree is reasonably small, otherwise the performance will suffer.
GetClickablePoint() retrieves the physical screen coordinates of a point on the element that can be clicked. Not every element is clickable, and some elements may have mechanisms to imitate being "clickable" (for example in webpages Javascript can be used to make text clickable) but not have a clickable point available.
GetWinId() gets the parent window handle from the element.
GetPos(relativeTo="") returns an object containing the x, y coordinates and width and height for the element: {x:x coordinate, y:y coordinate, w:width, h:height}. relativeTo can be client, window or screen, default is A_CoordModeMouse.
GetPropertyValue(propertyId) retrieves the current value of a property for this element. This is rarely used, since Element.Property does the same.
GetPattern(pattern) retrieves a Pattern object of the specified control pattern on this element. This is rarely used, since Element.SomePattern does the same (needs to end with the word "Pattern", such as Element.ValuePattern).
Highlight(showTime:=unset, color:="Red", d:=2) draws a rectangle around the elements borders, which can be used for debugging purposes.
showTime can be one of the following:
Unset - if highlighting exists then removes the highlighting, otherwise highlights for 2 seconds. This is the default value.
0 - Indefinite highlighting
Positive integer (eg 2000) - will highlight and pause for the specified amount of time in ms
Negative integer - will highlight for the specified amount of time in ms, but script execution will continue
"clear" - removes the highlighting unconditionally
Dump(scope:=1, delimiter:=" ", maxDepth:=-1)
Returns info about the element: ControlType, Name, Value, LocalizedControlType, AutomationId, AcceleratorKey.
- scope : Optional UIA.TreeScope value: Element, Children, Family, Descendants, Subtree (=Element+Descendants). Default is Element.
- delimiter : Optional delimiter to separate the returned properties. Default is space.
- maxDepth : Optional maximal depth of tree levels traversal. Default is unlimited.
DumpAll(delimiter:=" ", maxDepth:=-1)
Calls Dump on the whole Subtree starting from this element.