UI automation

[agowa]

Support UI (and UI automation) the windows ecosystem sadly is often "UI first" when it comes to software design. And the powershell/command line interfaces are not available or not documented at all. It would be great for sysadmins to be able to encapsulate these applications in containers, so that they are no longer bound to a single host and can be put into a k8s cluster without requiring something developers and Ranorex and a lot of time.

[vrapolinario]

Thanks for opening this issue @agowa338.
Have you checked the Windows base container image? (Not Server Core or Nano Server)
This image does provide support for UI automation - the downside comes with the size of the image, but the scenario should work.

[agowa]

Ok, than it might be documentation. I was looking for this for some time, but only found stuff like this though:

• Sorry no. Microsoft specifically isn't supporting RDP or GUI's in containers, as far as I know. (...) but if you're trying to develop a desktop app, then containers aren't ideal.
• Microsoft made a change sometime after microsoft/windowsservercore:1709_KB4074588, breaking RDS.
• I’m looking forward to Windows apps being available with GUI; I think it’ll be really useful in my job.
• UI testing is explicitly mentioned in the post, however I (and according to the reactions on that post some other people as well) are unable to actually interact with the app through WinAppDriver or other UI automation tools. (...) and also that there is no mouse support

And this could also be related:

• If you are using Windows 10 or Server 2016 your keyboard and mouse will not work on Session 0. (...) Additionally, Microsoft has removed the Interactive Services Detection Service on Windows 10 Version 1803.

[ghost]

This issue has been open for 30 days with no updates.
@vrapolinario, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@vrapolinario, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@vrapolinario, please provide an update or close this issue.

[NikolayMatokh]

Still waiting....

[ghost]

This issue has been open for 30 days with no updates.
@vrapolinario, please provide an update or close this issue.

[slavanap]

Microsoft team, please provide an update.

[agowa]

Same. We need official way for UI automation.

From the replies this got I think some clarification is needed. I also think I now understand what @vrapolinario meant with there already being an official way. There is: https://docs.microsoft.com/en-us/dotnet/framework/ui-automation
However the API is quite "limited" in some while being quite capable in other situations.

That API is not able to fake input to notepad without the window being onscreen and having focus.
This is because notepad requires "System.Windows.Automation.TextPattern" to be used instead of "System.Windows.Automation.ValuePattern" and that one does not offer a set text method (more about why that matters in the example at https://docs.microsoft.com/de-de/dotnet/framework/ui-automation/add-content-to-a-text-box-using-ui-automation).
Therefore the only way to write into notepad automatically is using "System.Windows.Forms.SendKeys.SendWait". But if the focus suddenly changes, e.g. a popup opens, an error message is thrown, ... the input is provided to the wrong application...
Because of that, this is not a stable way to automate gui applications.

But there are also advantages of the current API. For example compared to AutoIt it allows to access elements within an WPF element within an dotnet application. And it doesn't require an additional runtime, powershell/dotnet is enough

Tl;Dr: The current state is better than I initially was aware of, but it still needs some polishing.

Example code for automating notepad:

[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationClient")
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationTypes")
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationProvider")
[void][System.Reflection.Assembly]::LoadWithPartialName("UIAutomationClientsideProviders")
[void][System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")

# According to the documentation this should be also called, but within powershell 7.1 it fails
# and within Windows Powershell (5.1) it is unstable and always throws an exception the first time it is called and succeeds the 2nd time...
# Also it doesn't seam to be needed for "just" automating notepad.
#$client = [System.Reflection.Assembly]::LoadWithPartialName("UIAutomationClientsideProviders")
#[Windows.Automation.ClientSettings]::RegisterClientSideProviderAssembly($client.GetName())

$autProc =  Start-Process notepad -PassThru
$rootElement = [Windows.Automation.AutomationElement]::RootElement
$condAUTProc = [Windows.Automation.PropertyCondition]::new([Windows.Automation.AutomationElement]::ProcessIdProperty, $autProc.Id)
$autElement = $rootElement.FindFirst([Windows.Automation.TreeScope]::Children, $condAUTProc)

# LocalizedControlTypeProperty is derived using "Accessibility Insights for Windows"
$documentElement = [Windows.Automation.PropertyCondition]::new([Windows.Automation.AutomationElement]::LocalizedControlTypeProperty, "document")
$document = $autElement.FindFirst([Windows.Automation.TreeScope]::Descendants, $documentElement)
$document.SetFocus()

[System.Windows.Forms.SendKeys]::SendWait("FooBar")

[vrapolinario]

Thanks @agowa338 for the follow up. Can I recommend you provide the feedback directly to the documentation above. It might be the case that the team who owns it has not seen feedback on it for Windows Containers.

@slavanap apologies for the delay. Does this helps with your case as well?

[slavanap]

Does
https://github.com/microsoft/WinAppDriver
work in containers? Because if not then the suggested approach requires manual walkthrough and tinkering of every step, and doesn't support steps recording.

[vrapolinario]

There's an issue on that repo: microsoft/WinAppDriver#1046 covering exactly that. While that tool appears not to work on Windows Containers as it depends on an active desktop session, there are alternatives in there.

I went ahead and opened an issue on their repo to add support for Windows Containers. Please remember this is a different team and we don't own their prioritization. We'll definitely work together if needed, though.

[slavanap]

The general issue with UI automation is that

• app installation,
• app startup sequence,
• app shutdown sequence
  do involve iteration with GUI, although these legacy apps run as a backend service, they require interactions with GUI to get launched.

Also, for such apps state monitoring like an ability to take a screenshot would be really useful.

[grochoge]

UI automation appears to not work at all when I try using windows:1909 and TestStack.White.

After opening notepad.exe and using the underlying System.Windows.Automation, I don't see any children under the root element (Desktop). IE, the following prints nothing:

using System.Windows.Automation;
...
var lWalker = new TreeWalker( Condition.TrueCondition );
var lElement = lWalker.GetFirstChild( AutomationElement.RootElement );
while ( lElement != null )
{
    Console.Out.WriteLine( $"{lElement.GetType().Name} {lElement.Current.Name} {lElement.Current.ProcessId} {lElement.Current.BoundingRectangle}" );
    lElement = lWalker.GetNextSibling( lElement );
}

[grochoge]

The Get-Process Notepad | ? MainWindowTitle -ne "" | Select MainWindowTitle Powershell command mentioned as working with windows:1809 in another thread no longer works with windows:1909.

What we could really use is a way to run multiple copies of Windows Sandbox as part of our CI. Why can't we use that as a Docker base? That also doesn't require a 6GB download/17GB image.

[agowa]

Thanks @agowa338 for the follow up. Can I recommend you provide the feedback directly to the documentation above. It might be the case that the team who owns it has not seen feedback on it for Windows Containers.

The team apparently doesn't accept feedback. There isn't "provide feedback" at the bottom of the page as other docs pages have...

[vrapolinario]

Than can I recommend you provide the feedback directly at their repo? Here's the URL: https://github.com/microsoft/dotnet.

[ghost]

This issue has been open for 30 days with no updates.
@vrapolinario, please provide an update or close this issue.

[slavanap]

Is it right that UI automation of legacy apps for Windows Containers infrastructure is not a priority for dev team right now?

[vrapolinario]

We made some progress on this with the Windows image that has a larger set of APIs available. Other than that, it seems the request is for tools that perform automation of UI tests need to support its own deployment on Windows containers. This unfortunately is out of our scope on the platform.

[grochoge]

@vrapolinario There appears to be negative progress on the Windows image that had a larger set of APIs available. From reading here, things that used to work in 1809 containers (like getting the main window title) no longer work in 1909 containers.

What we could really use is containers based on Windows Sandbox. But Microsoft puts artificial restrictions on that (along with the artificial licensing restrictions on normal Windows sessions) that makes it very very difficult to perform Windows Desktop UI automated testing.

[vrapolinario]

Adding @weijuans-msft for visibility.
Can't speak for the Sandbox option. We do want to address this scenario and we're looking into it, but no timeline at this point.
We'll update this issue once we have more to share.

[slavanap]

@vrapolinario re large set of APIs, is it now possible to take a screenshot of a running app in a Windows Container? If not, when will it be available?

[vrapolinario]

I'm not aware of a PowerShell option (or cmd for that matter) to take a screenshot, so I'm assuming it's not possible.

[vrapolinario]

Folks, after some internal investigation I'm getting the conclusion that we don't have a better answer at this point on UI automation, other than the APIs we provide on the Windows image. Since this seems to be a hot topic, I wanted to comment here and let you know that this is something we're taking in consideration as we evaluate new capabilities for Windows containers. Thanks for your input so far.

We'll close this issue for now, but please rest assured the inputs from this thread will feed our investments on upcoming releases.

[nchandra-seg]

Thanks for this. Please let us know if there is any roadmap

[david-terk]

With the release of this new image, are there any changes with respect to this topic?

https://techcommunity.microsoft.com/t5/containers/announcing-a-new-windows-server-container-image-preview/ba-p/2304897

[ghost]

update?

[pradeipp]

Hi @vrapolinario, any update on this? it's been 6+ months since you were last active in this thread.

[cwilhit]

Hey everyone, the Windows containers product team is still monitoring this issue. I am reopening the issue to keep for tracking the feature request, but there are no new updates to provide--it is still sitting on our backlog. If/when we do have an update, we will post it here or you can monitor the Windows container blog.

[ghost]

This issue has been open for 90 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[slavanap]

Up

[weijuans-msft]

At this time, we don't have any plan to support. Close it for now. Can reopen if this is changed.

[slavanap]

@weijuans-msft, with all the respect it's not completed, it is failed.

[nick-wellinghoff]

Can you offer an explanation as to why this use case is not supported? Its literally one of the most useful features of containers. UI testing is a giant chore using VMs. Clearly all the code to make it work already exists (e.g. windows sandbox) and this is a totally artificial software imposed limit.