idb ui describe-all not showing all UI elements. Those which are child of an element with "role_description":"group" are missing

[himvarma]

Description

idb ui describe-all command is expected to return complete hierarchy of UI Accessibility elements, and should match with the output of WebDriverAgent in order to migrate to idb. But in actual, all those UI elements are missing from the query result which are under a parent with type->Group / role_description->group)

Reproduction

1. Use an app with complex UI hierarchy, where there is at least one element of type=Group and multiple child elements within it.
2. Open the app and that particular page with the condition mentioned in point 1
3. From terminal, run idb ui describe-all
4. Observe the output
5. Now run idb ui describe-point at various coordinates within the group element type
6. Also try to check UI hierarchy output using WebDriverAgent
7. UI elements returned in step 3 are missing all those child elements which are returned in step 5 and 6

Solution

Not Known

Additional Information

• idb revision: installed using command : pip3.6 install fb-idb

[dlongmuir]

All elements used to be shown, I have some output from 6 months ago (Sept 14, 2021) showing output from the bottom toolbar of the com.apple.mobilecal app; a couple of weeks later when I tried again (Sept 28, 2021) it wasn't working.

Steps to reproduce

running iPhone simulator
idb launch com.apple.mobilecal
idb ui describe-all | grep Inbox

expected results: Inbox to be found as an AXLabel
actual results: "Toolbar" AXGroup hides bottom UIElements

iPhone 12 Pro Simulator output for describe-point

idb ui describe-point 350 800 
{"AXFrame":"{{332, 771.66666666666663}, {42, 33}}","AXUniqueId":null,"frame":{"y":771.66666666666663,"x":332,"width":42,"height":33},"role_description":"button","AXLabel":"Inbox","content_required":false,"type":"Button","title":null,"help":null,"custom_actions":[],"AXValue":null,"enabled":true,"role":"AXButton","subrole":null}

[himvarma]

@dlongmuir Is there an older version of idb I can use to overcome this problem? This is a major issue, I hope it gets fixed soon because it blocks automation on many UI elements in any app.

[dlongmuir]

I'm not sure although it would be something to try. I am unclear if this is an issue with idb itself or if there were underlying Xcode or Simulator changes that caused the change, so trying an older version of idb would help isolate the cause.

[dlongmuir]

I'd be happy to try and investigate this if someone could give me a point in the right direction towards where the describe-all response is being generated so I could set a breakpoint and work backwards from there (or any other suggestions someone who is more familiar with the code might have)

[dlongmuir]

I've looked at this a bit and could someone from the dev team confirm that this isn't fixable because the NSAccessibilityElement that FBSimulatorControl gets back from the Apple simulator private APIs doesn't contain any accessibilityChildren? (for the "missing" elements)

[simon-gilmurray]

This is causing me issues when trying to use Maestro UITesting as my TabBar Buttons/elements are not available to query in IDB. I've tried explicitly setting isAccessibilityElement to true but this doesn't seem to help either.
I can query the elements using Sherlock Inspector and they work fine in XCUITest.

"attributes" : { "text" : "Tab Bar", "bounds" : "[0,761][390,844]" }, "children" : [ ], "clickable" : null } ],

[hagemon]

An alternative way for detecting UITabBarButton is using idb ui describe-point x y. But other group elements have not been confirmed.

In my project, idb ui describe-all --nested returns Tab Bar element (ignore some attributes here):

{
  "AXFrame": "{{0, 761}, {390, 83}}",
  "role_description": "group",
  "AXLabel": "Tab Bar",
  "type": "Group",
  "children": [],
}

Buttons are missing in child, but when using idb ui decribe-point 15 780 (an arbitrary point inside Tab Bar), I got UITabBarButton element (also ignore some attributes here):

{
  "AXFrame": "{{2, 762}, {126, 48}}",
  "role_description": "tab",
  "AXLabel": "Home",
  "type": "RadioButton",
  "AXValue": 1,
  "role": "AXRadioButton",
  "subrole": "AXTabButton"
}

Then I can programmatically detect all the buttons by code like:

tab_bar = idb_ui_describe_all().get_tab_bar()  # get attributes of TabBar
first_tab = idb_ui_describe_point(tab_bar.x+10, tab_bar.y+10)  # an arbitrary point
num_tabs = tab_bar.width // first_tab.width
for i in range(1, num_tabs):
    tab = idb_ui_describe_point(15+i*first_tab.width, tab_bar.y+10)

# ... do things with your tab

I don't know why describe-point can detect elements that do not exist in describe-all, while my Accessibility Inspector works fine to find these UITabBarButtons:

versions:

• fb-idb 1.1.7
• XCode 14.0
• Simulator 14.0

[dlongmuir]

@hagemon I'm doing something similar but it is really slow. I'm still optimistic this will get fixed at some point.

[hagemon]

I was taking an effort to deal this issue, and found that it may be restricted by AppKit APIs.

Concretely, with the latest release idb 1.1.8, I wrote an simple additional test case in FBSimulatorControlTests/Tests/Integration/FBSimulatorMediaCommandsTests.m as following:

- (void)testAccessibility
{
    FBSimulator *simulator = [self.control.set simulatorWithUDID:@"YOUR-DEVICE-UDID-HERE"];
    NSArray<NSDictionary<NSString *, id> *> * result = [[[FBSimulatorAccessibilityCommands commandsWithTarget:simulator] accessibilityElementsWithNestedFormat:TRUE] await:nil];
    NSLog(@"%@", result);
    NSLog(@"--------------");  // set a breakpoint here
}

The detailed operations:

1. Open target simulator and app under test.
2. Set breakpoint at the comment position above.
3. Run this test case.
4. Check output.

With the original code, elements are of class AXPMacPlatformElement(derived from NSAcessibilityElement), which returns children information using element.accessibilityChildren. But TabBar still returns empty children array. I investigated Apple Document and find accessbilityTabs property in NSAccessibilityElement, which is described as:

Use this property on a tab view element.

But after replacing accessibilityChildren with accessbilityTabs, I found the same result. I am wondering whether I used this property incorrectly.

Finally I check the difference between describe-all and describe-point, the latter one can return the tab bar elements by hitting through the tab bar (which mentioned in my previous reply). The only difference I found was the function called by self.translator (a object of class AXPTranslator)

- (FBFuture<NSArray<NSDictionary<NSString *, id> *> *> *)frontmostApplicationForSimulator:(FBSimulator *)simulator displayId:(unsigned int)displayId nestedFormat:(BOOL)nestedFormat
{
  return [FBFuture
    onQueue:simulator.workQueue resolveValue:^(NSError **error) {
      NSString *token = [self pushSimulator:simulator];

      // translation for all elements
      AXPTranslationObject *translation = [self.translator frontmostApplicationWithDisplayId:displayId bridgeDelegateToken:token];  

      translation.bridgeDelegateToken = token;
      AXPMacPlatformElement *element = [self.translator macPlatformElementFromTranslation:translation];
      element.translation.bridgeDelegateToken = token;
      NSArray<NSDictionary<NSString *, id> *> *formatted = [self.class recursiveDescriptionFromElement:element token:token nestedFormat:nestedFormat];
      [self popSimulator:token];
      return formatted;
    }];
}

- (FBFuture<NSDictionary<NSString *, id> *> *)objectAtPointForSimulator:(FBSimulator *)simulator displayId:(unsigned int)displayId atPoint:(CGPoint)point nestedFormat:(BOOL)nestedFormat
{
  return [FBFuture
    onQueue:simulator.workQueue resolveValue:^(NSError **error) {
      NSString *token = [self pushSimulator:simulator];

      // translation for element at point
      AXPTranslationObject *translation = [self.translator objectAtPoint:point displayId:displayId bridgeDelegateToken:token];  
      
      translation.bridgeDelegateToken = token;
      AXPMacPlatformElement *element = [self.translator macPlatformElementFromTranslation:translation];
      element.translation.bridgeDelegateToken = token;
      NSDictionary<NSString *, id> *formatted = [self.class formattedDescriptionOfElement:element token:token nestedFormat:nestedFormat];
      [self popSimulator:token];
      return formatted;
    }];
}

These two functions are declared in private headers from Apple, so I can't find out what happens inside. Hope someone can figure out these difference or if there is a proper way to use NSAccessibility properties.

[simon-gilmurray]

Is there anyone we can tag from the FB/IDB side to look at this?

[dlongmuir]

FYI, hope this saves someone some time - I just pulled latest and built idb_companion and this is still not returning the contents of the TabBar