Appium (and Inspector) cannot render elements on hybrid iOS Native/Web mobile app page

[lvconley]

Do I have the most recent component updates?

• I use the most recent available driver/plugin and server versions

Is the component officially supported by the Appium team?

• I have verified the component repository is present under the Appium organization in GitHub

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The application under test is the Walt Disney World mobile application on iOS. The issue is currently blocking over 1000 test cases. The issue involves the "Dine Reservation" flow. This is when a guest at Walt Disney World uses the app to query for available dining reservations amongst all of Disney World's restaurants on a given date for a party size under 10 people. The query returns a JavaScript Collection object with a JSON payload of about 145KB. When we run our automation, no elements that we need to search for can be seen by Appium, so all such tests fail. When the same flow is performed manually with Desktop Inspector, we can see the enclosing DOM markup for the container page, but the contents of the JSON file are not displayed. This list would include images of each restaurant returned by the server which have openings for a party size under 10 people on the given date, along with the restaurant name, their location, what their cuisine is like, what their ratings are, along with quite a lot of other relevant information. This information all displays fine in the app itself--both real and simulated devices, but Appium displays nothing related to the restaurant data returned by the server.

The issue happens regardless of whether we are in Native or Web View modes.

Now, if we go through the same query again but this time select a party size of 10 or more people, the object that comes back from the server is much smaller and Appium can find all required elements. Appium Inspector is also able to display all elements for the page. The issue may be related to this bug but I am unsure: #18085

Expected Behavior

The contents of the page returned by the server should be readable by Appium. Likewise, during manual testing with the Appium Inspector, all page elements returned by the server should be clickable and identifiable by Appium Inspector.

Minimal Reproducible Example

Here is a cleaned up JSON blob (I'm sure I've screwed up syntax while cleansing data) for a SINGLE restaurant that's included in the response by the server. But the full query will include, in this instance, about 71 restaurants! So as you will be able to see after looking at this cleaned up JSON blob (381 lines of data) for a SINGLE restaurant, that this amount of info has to be implemented for each of the 70+ restaurants (27,000+ lines) returned by the query: https://gist.github.com/lvconley/72217b0c801922a348689e9d6f3bdca1

Environment

• Operating system: iOS 16.2
• Appium server version: 2.0.1
• Appium driver(s) and their version(s): xcuitest@4.33.2
• Appium plugin(s) and their version(s): NA
• Node.js version (output of node --version): v16.19.0
• npm version (output of npm --version): 9.8.1
• Last component(s) version which did not exhibit the problem: NA dine flow is new to current release effort
• Platform and version under test: Walt Disney World mobile application for iOS 7.29
• Real device or emulator/simulator: Both real and simulator -- iPhone 14

Link to Appium Logs

No response

Further Information

No response

[jlipps]

This is expected. Since you're dealing with a webview, you should use the web context for element inspection. Web elements are not always "promoted" to native elements in Apple's accessibility logic.

There is very rudimentary inspection available in the Appium Inspector, but for webviews I typically recommend attaching desktop Safari to the device/simulator instead, so you have a full-fledged developer tools environment.

If you want to try Appium Inspector's support, you need to go into web/hybrid mode using the appropriate icon, then you should select the available web context as the active context.

[jlipps]

And the same is true when running automation code: you'll need to switch into the web context before running element queries and interactions for web content.

[lvconley]

@jlipps Hello! Thanks for the feedback! My mistake in not adding info about the issue happening whether we are in native mode or in web view mode. We do have the ability to switch context in code, and we do so as needed, but unfortunately it does no good.

[mykola-mokhnach]

XCUITest relies on Apple's XCTest framework, which queries the accessibility layer to gain the page tree.
I would try to check the app with the native Xcode's Accessibility Inspector as described in https://medium.com/yay-its-erica/ios-accessibility-the-accessibility-inspector-voiceover-swift-3-5f8e2bc50b20

If the inspector is unable to recognize elements then there is no way they could be recognized by XCUITest. In case the app under test itself properly exposes components to the accessibility then only Apple could fix such issues.

Some known issues that could be possible duplicates of this one:

• [IOS] Issue with loading huge data in Appium #13160
• Accessing page content when using deeply nested React Navigation navigators on iOS Using Appium #14825

[jlipps]

unfortunately it does no good

Meaning you don't see the context? Or you don't see any web elements in it?

[lvconley]

Hi @jlipps! Here's a screenshot of the Walt Disney World (WDW) iOS app with Appium Inspector in web view mode. In this screenshot, we've just queried for all restaurants at WDW available on a specified date for 2 people. This returned a JSON object with over 27,000+ lines of information on the 71 restaurants returned by the query. When we query for restaurants that can serve parties of 10 or more people on a given date (see other screenshots in Native and web view mode), the object is much smaller and we can see all elements.

 ets/13697495/9a45b031-24cf-4e32-ad52-167ff12568e3)

[lvconley]

[lvconley]

[mykola-mokhnach]

I don't observe any screenshots made from the webview context above. Being in web context the page source cannot contain native elements whose class names start with XCUI

[lvconley]

I don't observe any screenshots made from the webview context above. Being in web context the page source cannot contain native elements whose class names start with XCUI

Hmm. I put the Inspector in Web View mode and took the screenshot to show you what I'm seeing. Regardless of which mode we are in, when we get the large JSON blob (71+ restaurants) back from the server for any party size under 10, we cannot see any of these restaurant elements and so our tests fail. If we query for restaurants that can support reservations of 10 or more people on a given date, then the object that comes back is much much smaller and Appium/Inspector has no problem detecting and displaying the elements that we need to verify.

[jlipps]

The part about the app querying for JSON blobs is, I think, irrelevant to this point. Fundamentally what the app has in its native or web content is all that matters, not how it gets there. My guess is that what you're running into is the fact that you have a hybrid app, and iOS is promoting some of these elements to native elements for accessibility convenience. But when the number of elements is too large, it stops doing that. The solution would be to switch into the webview context.

The way to do this in the inspector is to first enable webview detection, which it looks like you have done. But there is also a web context dropdown within which you must actually choose/active the webview context. It's a 2-step process in other words (since there might be multiple webviews, so Appium cannot know which one you might want to switch into). Let me know if you have trouble finding the webview context switcher dropdown after looking for it, and I can hunt for a screenshot.

[jlipps]

here's a screenshot. notice the dropdown on the right. this shows up in the element inspector after you refresh when a webview is detected while in webview mode.

[jlipps]

if nothing shows up there, then you will want to see if you can connect to the webview with Desktop Safari. if you can't do that, then it's likely that the webview itself is not debuggable and you're out of luck. (Note that webviews in newer versions of iOS are not debuggable by default and require setting an inspectable flag on them, which must be done in the app source code).

[lvconley]

@jlipps Thanks for this but I'm not seeing a drop down regardless of which element I click on in the DOM tree. I'm using Appium Inspector 2023.8.1. I've seen a dropdown in the past in the rightmost column where the attributes of the selected element can be seen, but I'm not finding that dropdown anywhere right now, regardless of which mode I'm in.

[lvconley]

Just saw the screenshot. Looks like this page is not a web view then?

[mykola-mokhnach]

Just saw the screenshot. Looks like this page is not a web view then?

Is it a question or a confirmation?

I would say it makes sense for you to hire a professional consultant to better understand principles and limitations of iOS automation using XCUITest driver.

You could also try to ask questions at Appium forum as here we are mostly focused on actual server and driver issues.

[eglitise]

@lvconley since Appium Inspector 2023.8.1, the dropdown is shown directly in the top application header (see example video in appium-inspector/pull/1006). In your screenshots you have enabled hybrid mode but you have the orange warning icon, which means that no webview contexts were found.
I would suggest trying to connect to your app with Desktop Safari developer tools, like @jlipps said.

[jlipps]

Looks like this page is not a web view then

Well, it just means Appium can't detect any webviews. This could be for a number of reasons. Make sure have read the webview setup guide for the XCUITest driver.

The way to prove you have things configured correctly is to connect to the webview from Desktop Safari's developer tools. If you can't do that, then it's either not configured correctly or is not a webview.

Thanks for the updated instructions @eglitise, guess I had an older version on my system!

[lvconley]

So, I connected Safari and my simulator was detected, but as you can see, the dropdown on Safari shows that "No Inspectable Applications" are detected, even though my app is running, has focus, and the problematic page with the list of restaurants is displayed.

[jlipps]

Great, this clarifies things. It confirms that there is no inspectable webview in your app. If you believe your app does have a webview, then a developer needs to make it inspectable.

[lvconley]

@jlipps Thanks for all the tips and help! I'm not on the Dev team, so I'm unsure if this view is supposed to be a web view or not but it is a hybrid app and some views are definitely web views. On the page shown in the images above, on iOS, all elements can be seen by Appium at automation runtime when get back a list of 14 restaurants or less. The JSON here contains about 5000 lines of text. But when we query for a normal, common, typical, standard party size of two people--the server sends back a list of just about about every restaurant at Walt Disney World (seems to be around 71+-ish), and the JSON object contains over 27,000 lines of text and the elements are suddenly unseen by Appium with automation and unseen by Appium Inspector as well during manual testing. And on Android, on this same flow, WebDriver actually crashes hard and takes down our Walt Disney World mobile app (verified by support at SauceLabs) on both real devices and simulators. So, on Android, this specific issue is an absolute showstopper.

[jlipps]

My guess is still the same in that case:

My guess is that what you're running into is the fact that you have a hybrid app, and iOS is promoting some of these elements to native elements for accessibility convenience. But when the number of elements is too large, it stops doing that.

Your best bet for a next step is to ask the dev team to make the test version of the hybrid app with debuggable/inspectable webviews so that you can work in the web context.

[mykola-mokhnach]

Closed as third party issue

[lvconley]

I note that this issue has been closed, but do have a significant update to add. So, our developers enabled inspectability on the problematic webview, and we can indeed see all elements with the Safari dev console on this webview, but Appium Inspector CANNOT see any elements in the returned list of Walt Disney World restaurants (it can see elements outside of the element displaying the list, which is a massive JSON object), and behaves as initially reported above. Does this mean that this is in fact an Appium issue and not an Apple issue?

[KazuCocoa]

We cannot say anything since no appium server log, but it depends on the meaning. If you meant these elements were available in WEBVIEW context in Appium (not inspector), but cannot see anything in the Appium Inspector, then probably appium inspector can do something. So, could you clarify:

1. check if these WEBVIEWs are available in the latest Appium 2 and each driver
   • theoretically, if they are available in Safari WebInspector, safari's WEbView could see them
2. if could see, check it via appium-inspector. if not available in the same server log with 1, maybe the inspector could do something

Full Appium server log would help to take a look at possible issues

[jlipps]

It seems like there is still some confusion here about this whole JSON object thing. At the end of the day, Appium webview inspection can only see web elements. Whatever is in the HTML of the page. Just like a user. Users don't see JSON objects, although maybe the app parses that JSON object and turns it into web elements. So any discussion of JSON here is by definition irrelevant, however true it may be that the app receives a JSON blob in order to populate its data.

But if you are attaching the Inspector to the same webview as the one you are attaching Safari to, and not seeing the same data, then that is something we can investigate. But as Kazu said, to do that we'd need a full set of Appium logs.

[lvconley]

My apologies to you both for the delayed response here. My update above was factually incorrect. I had not switched to web view when I made that update. I was still in native mode view. When I do switch to web view, the elements can be seen, but Appium Inspector essentially becomes non functional. First, when in web view mode with Appium Inspector, the whole page takes about 4 minutes to load when we query for available restaurants where a party of two can make a reservation. The blob that comes back from the server varies in size depending on the list of restaurants that are returned. The properties sheet for every restaurant returned is 381 lines. On average, for a party of two on any given day at Walt Disney World, a list of 91 restaurants will be returned. That's 34,671 lines in the json that's returned. When this page is done loading in Appium Inspector, navigating the DOM tree by opening an element takes about 6 seconds per element click. And if the element you are trying to find is hundreds of elements deep, forget about finding it manually. So, we have an initiative in place to do a universal scroll approach, where the server sends back an initial list of 14 restaurants--which always display in Inspector, and can always be seen at automation runtime. And then as the user swipes down through the list of 14 restaurants, another fetch is made dynamically as they continue to scroll until the whole list is finally navigated by the user. So, we're pretty sure this issue will go away as soon as we've implemented this approach. Thanks to you and your team for all the great feedback and advice you've given us! It's been very useful!

[jlipps]

Ah ok, makes sense! Yes universal scroll will likely be much more performant for the user as well.

And for what it's worth, we don't really recommend the inspector for web inspection. It's typically a better experience to use Safari for that anyway.