WinAppDriver Unable to locate elements in WPF webview app

[vjacobjo]

Hi,
I am developing a python script that can automate a WPF webview app. The app basically runs a session off a Blazor Server on a WPF client. The App is able to locate certain parts of the of the App but it is unable to get the elements that are purely web based.

I understand that this could be a situation where as it is a hybrid app, I would need to change the context and inspect the elements in WebView. However, even when I try doing so, it encounters a failure:
selenium.common.exceptions.WebDriverException: Message: Command not recognized: GET: /session/21FC7F54-AFAC-4AB1-8B77-0868BA502C9A/contexts

I've attached the python script that I've used, along with the page source as well as the receipt from the WinAppDriver.

WinAppDriver_log.txt

page_source_info.txt

python_console.txt

If there is anything else I should provide, please do let me know.

[vjacobjo]

Versions of tools I've used:
Versions:

• WinAppDriver: v1.0.1
• Python: 3.9.7
• Appium-Python-Client:1.3.0
• Selenium: 3.141

[anunay1]

@vjacobjo : As this is a hybrid app, you will not be able to use WinappDriver. You can use the tool Playwright which has the capability to acquire a session with a already opened browser.

[liljohnak]

We do some testing using WebView2. Only some of the WebElements show up in WinAppDriver. Everything showes up when we use msedgedriver.exe. Also note, the Javascript exec functions for WinAppDriver work great for scrolling.

[anunay1]

@liljohnak Can you share the code for JS for scrolling with WinAppdriver?

[vjacobjo]

@anunay1 Thank you for the quick response.

I'm looking at the documentation and I assume that this was what you had meant: https://playwright.dev/python/docs/api/class-browsertype#browser-type-connect

For url, I'd like to use the exact same session that my webApp is using and not a new session. Would this mean setting up a debugging port on my webapp? How would I go about doing that?

@liljohnak I attempted to follow the document here: https://docs.microsoft.com/en-us/microsoft-edge/webview2/how-to/webdriver
It didn't work for me for two reasons:

• 1. I didn't have the the Edge option usewebview. I think this was because I was using Selenium 3 instead of Selenium 4. Using Selenium 4 would break my use of the Appium client.
• 2. I wanted to try to attach to the existing browser. This required me to enable a debugging port on my test application, which I had attempted to by writing the following code in the MainWindow.xaml.cs:
     var env = await CoreWebView2Environment.CreateAsync(null, null, new CoreWebView2EnvironmentOptions("--remote-debugging-port=9000")); await webView.EnsureCoreWebView2Async(env);
     Unfortunately I wasn't able to see the remote port available in chrome://inspect.

How did you go about using msedgedriver.exe and attaching it to your already launched webapp?

[anunay1]

Any good reason why you want to use the same session? Is the appium python client not working with selenium4?

[vjacobjo]

I'd like to test the application on the exact WebView instance, not a separate browser.

Initially, when I tried to install Selenium 4, I got a complaint from pip installer saying that its a breaking change to appium python client. I've since found out that the latest version of the Appium Python Client does use Selenium 4.

https://issueexplorer.com/issue/appium/python-client/602

Let me try this out.

Either way, I still don't know how to open a remote debugging port in my webapp to be able to connect to webApp with either MSEdge or playwright. I've just tried via command line here:

and while I got the app to run, I'm not able to see it on the remote debugging port:

[anunay1]

the universal default port is 9222, I have not tried it but it might work? or else you can pass --remote-debugging-port=9222 in arguments option for browserType.launchServer.

[vjacobjo]

I just tried it:

But still not seeing it in Chrome Inspect

*Note: the other devices are my chrome devices, but nothing with localhost:9222

I did also try running as adminstrator but the app doesn't run for some reason.

[anunay1]

I just created a project with Appium python client 2.1.2 and selenium version 4.1.0. Everything works fine. Can you please check at your end?

[vjacobjo]

I think I was able to as well. just now...

Any ideas on how I can enable the debugging port?

[anunay1]

It's an ugly way, but for testing purpose you can append the target in properties option of chrome -remote-debugging-port=9222 -- "%1" so that chrome is always launched with the remote debugging port.

[vjacobjo]

Like this?

It just opens the app.

Where exactly is the properties option in chrome? Also, I would ideally like to connect/ control to the webapp or the session which the WebApp has opened.

[liljohnak]

@vjacobjo we have a .bat file that creates that runs the C# tests and sets the debugging port.

set WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS ="--remote-debugging-port=9222

You can also add the above manually to your system environments

[liljohnak]

@vjacobjo Also when debugging use the edge browser to test using F12 at http://localhost:9222/

[anunay1]

Right click on chrome and click on properties:

NOTE: This will always open chrome with debugging port.

[vjacobjo]

@liljohnak so I added a batch file in the same directory as my exe that I would like to debug:

and then I ran the batch file.
Then I launched edge and navigated to localhost:9222 and pressed F12 to open the edge tools

I'm not able to see the app loaded on the browser

[vjacobjo]

@anunay1 okay I got myself to inspect the browser I am opening

However I'd like to be able to inspect the WebApp though. It's one step closer.

Maybe a little bit about the app. It has a Blazor Server running and then the WPF client that hosts it. I think somewhere in the stack, of the Dummy Configuration tool, I have to be able to set the remote debugging port there.

[vjacobjo]

okay I think I got it.

So I had to set the environment variable in the GUI there. I think the main mistake I made previously was that I didn't run the batch file that @liljohnak had suggested as an administrator so the environment variable wasn't set. When I add the environment variable in the GUI as shown here:

I am able to go into chrome://inspect, find my instance running in the webapp and then launch the dev tools to be able to control the webapp via the Chrome Dev tools

Now I think what I would need to do in my python script is launch the tool exactly like I've done before, but this time, I then launch msedge and connect with the url that connects to the specific session launched in the webapp.

[vjacobjo]

okay this is really dumb but now my code for automating desktops is now breaking.

I had upgraded the Appium-Python-Client to v2.1.2. And now when I run this simple code here:
element = self.driver.find_element(AppiumBy.NAME, "Clear") element.click()

it fails on click with the following error message:
File "c:\Projects\RandomSpikes\calculatortest.py", line 33, in test_initialize
element.click()
AttributeError: 'dict' object has no attribute 'click'

[anunay1]

element = self.driver.find_element(AppiumBy.NAME("Clear"))
element.click()

[vjacobjo]

^ That didn't work.
Turns out this is a bug in Selenium 4.
https://stackoverflow.com/questions/70295636/attributeerror-dict-object-has-no-attribute-click-error-with-find-element-b/70296487

I've got this fixed.

[vjacobjo]

Summary
Here's what needs to be done to automate BlazorWebView wpf application in python.

1. You will need to set the following system environment variable in your local machine:
   Variable name: WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS
   Variable values: --remote-debugging-port=9222

2. Install the following libraries:
   Appium-Python-Client==2.1.2
   selenium==4.1.0

3. If you haven't done so already, you should be installing the latest version of WinAppDriver

4. You would need to launch the application with WinAppDriver, but to be able to interact with the UI elements, you would have to use msEdgeDriver. Here is sample code to demonstrate this:
   `

    #note: you would need to start WinAppDriver.exe first
    desired_caps = {}
    desired_caps['ms:waitForAppLaunch'] = 5
    desired_caps["app"] = "C:/path/yourWPFapp.exe"
    startup_driver= self.get_driver(desired_caps)
    startup_driver.maximize_window()
    
    #Start Edge driver
    options = EdgeOptions()
    options.use_webview = True
    options.debugger_address = "localhost:9222"
    driver = WebDriver.Edge(executable_path="C:/path/msedgedriver.exe" ,options=options)
   

`

5. As a test, check to see if the driver returns the correct page source of the webApp:
   page_source = driver.page_source with open("page_source_info.txt", 'w') as f: f.write(page_source)

Special thanks to @anunay1 and @liljohnak for helping me out with this

[MagdelineNg]

@vjacobjo Hi, did you manage to use Selenium 4 with WinAppDriver in Python? I understand Selenium 4 is needed for the remote debugging port, but also has compatibility issues with WinAppDriver when automating the actions