WinUI 3 Tests fail to run after upgrade to Windows App SDK 1.1

[Marv51]

Description

After upgrading my test project to Windows App SDK 1.1 the execution of tests fails.

Steps to reproduce

I followed https://devblogs.microsoft.com/ifdef-windows/winui-desktop-unit-tests/

UITestMethod attributed tests, fail to run after I upgrade to Windows App SDK 1.1
with error message:

Exception thrown while executing test. If using extension of TestMethodAttribute then please contact vendor. 

Error message: 

UITestMethodAttribute.DispatcherQueue should not be null. 
To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization., 

Stack trace:    
at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.Execute(ITestMethod testMethod)
at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(TestMethodInfo testMethodInfo)

The exact same project work with Microsoft.WindowsAppSDK 1.0.3.

I tried to add [assembly: WinUITestTarget(typeof(App))] but that doesn't work either.

The DispatcherQueue should get set in the OnLaunched method of my App class:
UITestMethodAttribute.DispatcherQueue = m_window.DispatcherQueue;

Expected behavior

Test should get executed.

Actual behavior

Running tests fails with message above. No window ever pops up and the App class seems to be not created.

Environment

WinAppSDK 1.1.0
VS 17.2.3
Microsoft.TestPlatform.TestHost 17.2.0
MSTest.TestFramework 2.2.10
MSTest.TestAdapter 2.2.10

.csProj (click to expand)

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>WinExe</OutputType>
    <TargetFramework>net6.0-windows10.0.22000.0</TargetFramework>
    <TargetPlatformMinVersion>10.0.18363.0</TargetPlatformMinVersion>
    <RootNamespace>Telani.UITests</RootNamespace>
    <ApplicationManifest>app.manifest</ApplicationManifest>
    <Platforms>x86;x64;arm64</Platforms>
    <RuntimeIdentifiers>win10-x86;win10-x64;win10-arm64</RuntimeIdentifiers>
	<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>
	<ImplicitUsings>enable</ImplicitUsings>
    <UseWinUI>true</UseWinUI>
  </PropertyGroup>
  <ItemGroup>
	  <ProjectCapability Include="TestContainer" />
  </ItemGroup>
  <ItemGroup>
    <AppxManifest Include="Package.appxmanifest">
      <SubType>Designer</SubType>
    </AppxManifest>
  </ItemGroup>
  <ItemGroup>
    <Content Include="Images\LockScreenLogo.scale-200.png" />
    <Content Include="Images\SplashScreen.scale-200.png" />
    <Content Include="Images\Square150x150Logo.scale-200.png" />
    <Content Include="Images\Square44x44Logo.scale-200.png" />
    <Content Include="Images\Square44x44Logo.targetsize-24_altform-unplated.png" />
    <Content Include="Images\StoreLogo.png" />
    <Content Include="Images\Wide310x150Logo.scale-200.png" />
  </ItemGroup>

  <ItemGroup>
	<PackageReference Include="Microsoft.WindowsAppSDK" Version="1.0.3" />
    <Manifest Include="$(ApplicationManifest)" />
  </ItemGroup>

    <ItemGroup>
	    <PackageReference Include="MSTest.TestAdapter">
		    <Version>2.2.10</Version>
	    </PackageReference>
	    <PackageReference Include="MSTest.TestFramework">
		    <Version>2.2.10</Version>
	    </PackageReference>
	    <PackageReference Include="Microsoft.TestPlatform.TestHost">
		    <Version>17.2.0</Version>
		    <ExcludeAssets>build</ExcludeAssets>
	    </PackageReference>
    </ItemGroup>
</Project>

App.xaml.cs (click to expand)

using Microsoft.UI.Xaml;
using Microsoft.UI.Xaml.Controls;
using Microsoft.UI.Xaml.Controls.Primitives;
using Microsoft.UI.Xaml.Data;
using Microsoft.UI.Xaml.Input;
using Microsoft.UI.Xaml.Media;
using Microsoft.UI.Xaml.Navigation;
using Microsoft.UI.Xaml.Shapes;
using Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer;
using System.Runtime.InteropServices.WindowsRuntime;
using Telani.UITests;
using Windows.ApplicationModel;
using Windows.ApplicationModel.Activation;
using Windows.Foundation;
using Windows.Foundation.Collections;

[assembly: System.Runtime.Versioning.SupportedOSPlatform("windows10.0.18363.0")]
//[assembly: WinUITestTarget(typeof(App))]

namespace Telani.UITests;

/// <summary>
/// Provides application-specific behavior to supplement the default Application class.
/// </summary>
public partial class App : Application
{
    /// <summary>
    /// Initializes the singleton application object.  This is the first line of authored code
    /// executed, and as such is the logical equivalent of main() or WinMain().
    /// </summary>
    public App()
    {
        this.InitializeComponent();
    }

    /// <summary>
    /// Invoked when the application is launched normally by the end user.  Other entry points
    /// will be used such as when the application is launched to open a specific file.
    /// </summary>
    /// <param name="args">Details about the launch request and process.</param>
    protected override void OnLaunched(Microsoft.UI.Xaml.LaunchActivatedEventArgs args)
    {
        Microsoft.VisualStudio.TestPlatform.TestExecutor.UnitTestClient.CreateDefaultUI();

        m_window = new MainWindow();

        // Ensure the current window is active
        m_window.Activate();

        UITestMethodAttribute.DispatcherQueue = m_window.DispatcherQueue;

        // Replace back with e.Arguments when https://github.com/microsoft/microsoft-ui-xaml/issues/3368 is fixed
        Microsoft.VisualStudio.TestPlatform.TestExecutor.UnitTestClient.Run(Environment.CommandLine);
    }

    private Window m_window;
}

Tests.cs (click to expand)

using Microsoft.UI.Xaml.Controls;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer;
using System.Text;

namespace Telani.UITests;

[TestClass]
public class QuickTest
{

    [TestMethod]
    public void TestMethod1()
    {
        Assert.AreEqual(0, 0);
    }

    [UITestMethod]
    public void TestMethod2()
    {
        var grid = new Grid();
        Assert.AreEqual(0, grid.ActualWidth);
    }
}

AB#1643611

[Evangelink]

@Haplois Isn't this something you handled recently?

[marcomarin89]

@Marv51 I'm experiencing the same issue

[andrewittorio]

Hi all,

I have the same problem too. In my case it seems that the App class is not instantiated when running the tests.
In accordance with this article https://docs.microsoft.com/en-us/windows/communitytoolkit/mvvm/ioc
the App class is where I setup my services. But when the tests run the GetService always returns null.

For this reason I cannot update my projects to 1.1.3.
Do you have any suggestions?

Thank you,

[mikebmcl]

It appears that when switching from 1.0.X to 1.1.X, the call to the Win32 API function GetCurrentPackageFullName via

testfx/src/Adapter/PlatformServices.WinUI/AppModel.cs

Line 29 in 9694a42

result = Interop.GetCurrentPackageFullName(ref packageFullNameLength, null);

switches from succeeding (i.e. returning an insufficient buffer error) to failing with APPMODEL_ERROR_NO_PACKAGE. As a result, when Test Explorer loads the assembly, in 1.0.X builds it finds IsAppContainer: True but in 1.1.X IsAppContainer: False.

I'm not sure why it's doing this, but it does indeed not have any packaging information after the switch. In a 1.0.X build, a test marked [TestMethod] that does not do anything related to WinUI will launch the app (even though it doesn't need to do anything on the UI thread) and pass or fail as intended. In 1.1.X, it will not launch the app but it will still pass or fail as intended. With 1.1.X, when using WinUITestTargetAttribute to identify your Application-derived class, any test marked [UITestMethod] will not report any results in Test Explorer when you try to run it. Instead, if you look at the Tests section in the Output window, you will see that when it tried to call Application.Start, it failed with a COMException saying that the class is not registered (HResult 0x80040154).

Without WinUITestTargetAttribute, a [UITestMethod] test will fail at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.Execute with "UITestMethodAttribute.DispatcherQueue should not be null. To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization." despite having the code to set it as described in earlier comments.

Also, just as a debugging note, if you switch a project from WindowsAppSDK 1.0.X to 1.1.X, you will probably need to do a rebuild to see the errors; for me if I just switched and did a regular build, it would still launch the app and appear to work correctly (leading to a false hope that I had figured out a work-around).

I tried using a local build of MSTest (using the rel-2.2.10 branch) and changing it to use WindowsAppSDK 1.1.0 (and 1.1.3) instead of 1.0.0, but it did not change the results in any way.

The code at

testfx/src/TestFramework/Extension.WinUI/UITestMethodAttribute.cs

Line 148 in 9694a42

var id = applicationType.Assembly.GetType("Microsoft.WindowsAppSDK.Runtime.Identity");

that attempts to initialize the SDK fails because the type specified, Microsoft.WindowsAppSDK.Runtime.Identity, is not found in either 1.0.X or 1.1.X. It just returns null so no instance of a type is ever created to trigger SDK initialization. I was unable to find a good replacement to test that with.

My test project is essentially the same as the original bug report.

[mikebmcl]

Some updated information. I was able to partially workaround this by turning the app into an unpackaged app in order to be able to initialize the WindowsAppSDK and then start the app, as described below. For reasons I put at the end, I don't feel that this workaround is a viable long-term solution, but it might help some people for now. Note: I used the MSTest 2.2.10 packages and WinApp SDK 1.1.3.

Here is what I did:

I changed my test project to be explicitly unpackaged and to not automatically load WinAppSDK by adding
<WindowsPackageType>None</WindowsPackageType>
<WindowsAppSdkBootstrapInitialize>false</WindowsAppSdkBootstrapInitialize>
to the project's first property group. I did not need to delete or change any manifest or packaging files. (Note: These changes need to be removed or commented out if you wish to run the application outside of the testing environment since the initialization of the WindowsAppSDK happens outside of the app itself.)

I then created a local copy of UITestMethodAttribute and modified it to perform the bootstrap initialization of the WindowsAppSDK (see the attached WinAppSDK11UITestMethodAttribute.cs file).

I set the entry point using WinUITestTargetAttribute because WinAppSDK11UITestMethodAttribute will not work unless you do this.

Lastly, I changed my [UITestMethod]-decorated methods to use [WinAppSDK11UITestMethod].

I then ran the tests and the application launched properly and was able to run UI tests such as verifying an element's existence using its AutomationId.

Unfortunately, this is not a good workaround for any long-term purposes.

If you have anything in your App (UI or otherwise) that directly or indirectly requires a package identity, this will not work for you unless you went through the effort of creating a sparse package just for UI testing purposes. Outside of this workaround, the apps are being built into MSIX packages and thus automatically have identity without that.

You need to make sure you download and run the appropriate WindowsAppSDK runtime installer (or setup your testing app to do bootstrapping to install it). It's important to do this because you probably won't have the Microsoft.WinAppRuntime.DDLM packages installed. They are only necessary for unpackaged apps since they are responsible for "pinning" the runtime that your app is using to make sure it isn't upgraded or otherwise changed while your app is running. That package together with the Bootstrap class that the workaround makes use of takes care of that. Without it you would need to create your own solution for that potential problem. (That problem doesn't exist when your app is packaged.)

Lastly, you need to make sure to update the versioning data in WinAppSDK11UITestMethodAttribute to make sure it will find a version of WindowsAppSDK that is acceptable and is installed on the computer you will be doing testing on. The data in there now is for version 1.1.3 and needs to be manually changed to support other versions since the app is unpackaged and thus does not have a programmatically detectable version of the SDK associated with it.

Regardless, it is a workaround that may be helpful to some people until this problem is resolved so I figured I would add it here.

To me its main value is showing that it is possible to use MSTest to test apps that use WindowsAppSDK 1.1 (since it will properly launch the app and run tests against the UI as long as you go through with all of the steps needed to have it be unpackaged so you can get the runtime to actually initialize and start your app). The root problem is that (for some reason I haven't discovered), Windows reports DLLs from an app packaged with SDK 1.1 as not having package identity whereas with SDK 1.0 they are reported as having package identity. MSTest relies on that identification of having package identity to properly launch WinUI packaged apps for testing and so if you update an app to SDK 1.1 (without any other changes), testing stops working.

WinAppSDK11UITestMethodAttribute.zip
.

[kalizurawski]

I am also having this issue as mikebmcl is describing. Unfortunately that workaround does not work for me because it creates issues with the SDK. I hope this gets fixed soon because I have no choice besides using 1.1.0 so I can't downgrade to 1.0.x to gain use of the UITestMethod.

[mikebmcl]

From what I can tell, MSTest.TestFramework still has bugs relevant to this (described below) but the problem that starts everything off does not originate with MSTest.TestAdapter. Its ITestDiscoverer, MSTestDiscoverer, never finds the test container for the WinUI app (regardless of SDK version) and thus also finds none of its tests. Instead, it's found by "test container discoverer executor://projectoutputcontainerdiscoverer/v1" which comes from Microsoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscoverer. I cannot find that source for that type on GitHub or anywhere else so I'm guessing that it is an internal part of Visual Studio that isn't public.

I'm attaching a repro project. In addition to Debug and Release configurations, it also has Test10 and Test11 configurations. When using x86, those will expose the tests and swap between WindowsAppSDK 1.0.4 (Test10) and 1.1.3 (Test11). In any other configuration the SDK is 1.1.3. The PackageReference to the correct SDK for the configuration is included using conditional ItemGroup markup in the .csproj file. Because of this, Solution Explorer will not always show the correct value for the SDK package version that is currently being used. It doesn't seem to expect that different versions of a nuget package will be conditionally included. But as long as you do a Rebuild after switching configurations before attempting to run the tests, the proper SDK package will be used.

The Tests.MainWindowContentIsStackPanel test method in the repro will still cause unexpected behavior (either a failure or a crash) in MSTest.TestFramework (testhost) when trying to run it using Test11 (SDK 1.1.3) so this is still an MSTest bug.

If [assembly:WinUITestTarget(typeof(MSTestWinAppSDK11BugRepro.App))] in App.xaml.cs in the repro is commented out, then Tests.MainWindowContentIsStackPanel (the test that uses [UITestAttribute]) will fail due to an exception with error message "UITestMethodAttribute.DispatcherQueue should not be null" despite the value being assigned properly in App.OnLaunched since the test host never attempts to create an instance of the app.

If [assembly:WinUITestTarget(typeof(MSTestWinAppSDK11BugRepro.App))] in App.xaml.cs in the repro is not commented out, then Tests.MainWindowContentIsStackPanel (the test that uses [UITestAttribute]) will crash the test host due to a COMException "Class not registered (0x80040154 (REGDB_E_CLASSNOTREG))", which aborts the active test run (thus not running any of the remaining tests). Specifically it results from "[Error] The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.TypeInitializationException: The type initializer for 'WinRT.ActivationFactory1' threw an exception." which occurs when an instance of the app is attempted to be launched since MSTest.TestFramework knows about the WinUI app due to the WinUITestTargetAttribute` being set.

Ultimately, Microsoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscoverer detects and reports "IsAppContainer: False" when using SDK 1.1 but (correctly) detects and reports "IsAppContainer: True" when using SDK 1.0. Why its behavior differs and what all results from it is something I'd need to be able to see its source to tell. Some change to the WindowsAppSDK between 1.0 and 1.1 is ultimately the source of the problem. Since I was able to get UI tests to work using SDK 1.1 when using a customized version of UITestMethodAttribute that runs the app as unpackaged as described in my earlier post, it's probably not something unfixable. It's just that for whatever reason, that discoverer's behavior difference results in the Win32 API function GetCurrentPackageFullName finding a package name using SDK 1.0 but not finding one using SDK 1.1.

This was all tested using Windows 11 Pro 10.0.22622 Build 22622 with Visual Studio 2022 Community Version 17.3 Preview 5.0.

MSTestWinAppSDK11BugRepro.zip

Hopefully you can file an internal ticket about the Microsoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscoverer problem to bring it to the attention of the people who maintain that. If it is a problem with internal code in Visual Studio, perhaps there's a chance it can be fixed before 17.3 (Stable) is released.

[Evangelink]

Just a small message to let you know that we have seen this ticket and we will handle as fast as we can (we are a little short staffed with holidays and some other high priority tasks we need to handle).

[mikebmcl]

Just as a note, this might be the same problem as or closely related to this open issue in the WindowsAppSDK:
microsoft/WindowsAppSDK#2768

[Haplois]

Probably not, we invoked the ModuleInitializer in test test project by instantiating an auto generated class, it seems to be removed. So when we call into WinUI methods, since those are not initialized yet it fails. (from my short investigation, the actual problem might be different.)

[mikebmcl]

I've found a better workaround (though still not optimal). It's better because you can keep your app packaged, but it is not optimal because you need to do the testing from a console rather than in Visual Studio and there are some potential pain points associated with that (described below).

If you follow the instructions for "UWP Example" using VSTest.Console from the command line making sure to change /Platform:x64 to Platform:x86, then the tests will execute correctly for an app using WindowsAppSDK 1.1 (and the new 1.2 preview as well). You don't need to change anything about your app. Just build it in an x86 configuration as you normally would for when doing unit testing. Also, don't worry about the name being "UWP Example" in the documentation link nor about the output from the command saying MAUI = false and WinUI = false. As far as I can tell, the actual command line just configures the utility so that it will properly temporarily package/deploy the application (regardless of whether the package container would be a UWP Appx or a WindowsAppSDK MSIX) then run it and connect to it properly in order to run the unit tests.

Note that if you do try to run an x64 version, the app will launch but then no tests are run and it just sits there until you close the app and use Ctrl-C to terminate vstest.console.exe from the console. I don't think x64 testing was supported originally though, so I don't think this is actually a change. I also didn't spend any time trying to get it to work; I just ran it to see what would happen and that's what happened.

The downsides are that you need to run it from the command line rather than directly in Visual Studio, and will need to either specify the test methods directly using the /Tests:SomeTestMethod,AnotherTestMethod switch (see the above link for more details) or use Test Filtering if you only want to run some of your tests.

Test Filtering is a lot more practical if you have a lot of tests but note that some attributes, such as OwnerAttribute and TestPropertyAttribute, are not supported in filtering (the link lists the attributes that are supported). You cannot use a Playlist file. There's also no switch that will let you run only the tests that previously failed. So you may need to write some powershell scripts or batch files if you want features such as parsing a log file to extract the names of the test methods that failed and then only run those. And you may need to add some TestCategoryAttribute annotations to your methods (you can add multiple to the same method) to support all of your desired filtering scenarios.

---

I ended up coming across this because the using Trace as the Logging Level in the Test settings in VS, I noticed that WinAppSDK 1.0 apps were getting a string value for Package in the TestRunCriteriaWithTests class (VSTest) which was the MyProject.build.appxrecipe file when running the tests whereas with SDK 1.1+, Package didn't appear in the output for "TestRequestHandler.OnMessageReceived: received message: (TestExecution.StartWithTests)" (presumably because it was null for the reason stated in the documentation of the Package property).

I think this all ties back to ProjectOutputContainerDiscoverer determining that SDK 1.1+ projects are not app containers. It seems to do that by calling the IVsHierarchy.GetProperty method using VSHPROPID_AppContainer and VSConstants.VSITEMID.Root and interpreting the result as a bool.

The description for VSHPROPID_AppContainer is: "When set, indicates the target framework moniker to which the project system should retarget upon its first opportunity. The project is responsible for ensuring the applicability of the framework that is returned. This property is optional." I think that with SDK 1.0 something was happening such that this was getting set to retarget to UAP,Version=v10.0 and something changed such that SDK 1.1+ no longer causes that to be set. To test that I'd need to write a Visual Studio Extension. Even then, I don't know of a way to debug what Test Explorer itself is doing so I wouldn't know for sure that this was the reason that SDK 1.1 projects don't get a Package value and thus don't do the temporary deployment allowing them to be run as packaged apps so that MSTest's interactions with them work properly.

The more I dig into it, and I'm not sure I'm going to dig any further now that I have a workaround I can use, the more I'm convinced that it's a VSTest issue. One that can't be resolved for Test Explorer except by fixing it in the internal version that Test Explorer uses since VSTest has been built-in rather than an extension since Visual Studio 2019. If my guess about the discoverer determining that SDK 1.1 projects are not app containers are right, then it actually would come from some change to WindowsAppSDK that appeared in 1.1. Regardless, there are a lot more changes in the data of the Message that TestRequestHandler.OnMessagedReceived got for the StartWithTests message when using SDK 1.0 versus SDK 1.1 all of which seem related to seeing the project as packaged app versus as a .NET dll that has tests, respectively.

My main purpose was to share the new workaround. I figured I'd also note down these insights I gained in the process of finding it in case they are of some help in finding and resolving the problem!

[mikebmcl]

Edit: Setting <WindowsSdkPackageVersion> is not required. I ran a test after setting <WindowsAppContainer> and it seems that despite doing a rebuild of the project, Test Explorer was using some cached data. Unloading and reloading the project (and presumably closing and reopening the Solution) is required to get Test Explorer to clear its cache after you've added that property. I'm attaching a new version of the fixed repro that also includes more information about these three properties in its Readme.md file.

---

I solved the problem. What happened was that WindowsAppSDK stopped defining certain MSBuild properties beginning in version 1.1 that it defined in 1.0. There are also certain properties that changed. In total, there are four properties that need to be added to your .csproj now rather than the one property that the article mentions. Once these are added, if you followed the rest of the directions in the article you will be able to use MSTest with Test Explorer in Visual Studio (the same as before). I'm attaching the fixed version of the repro project I posted earlier. But here are the key properties that need to be in a <PropertyGroup>, it's probably best to put them in the first one but I don't think it matters:

<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>
(You probably have this already if you followed the article but I'm putting it here just in case.)

<EnableMsixTooling>true</EnableMsixTooling>
(The Msix Tooling is no longer in preview as of SDK 1.1 so this is the new property to use, but there is no harm in keeping the previous Preview version and it might even be needed for some corner cases.)

<WindowsAppContainer>true</WindowsAppContainer>
(WindowsAppSDK 1.0 defined this but it stopped defining it in 1.1+ so you need to define it now. There's no harm in defining it in a 1.0 project.)

The last one, <WindowsSdkPackageVersion> depends on what your <TargetFramework> property is set to. The value below assumes a project with <TargetFramework>net6.0-windows10.0.19041.0</TargetFramework>

<WindowsSdkPackageVersion>10.0.19041.26</WindowsSdkPackageVersion>

The key to getting a correct value for that is that the build number (19041 here) should match the build number of your target framework. Also, the version number, including the revision number (26 here), must exactly match a Microsoft.Windows.SDK.NET.Ref version otherwise you will get an error. You do not need to add that package to your project. You will most likely get an error if you try. The reason for the link to NuGet is just because it is a convenient source of proper version numbers. Also, it's recommended that you use the most recent revision but you can use earlier revisions if you have some reason to (e.g. version 10.0.19041.24).

Making sure you have those four three properties correctly defined in your project's .csproj file solves the problem. I'm not aware of any side effects , though you may want to take care to keep the WindowsSdkPackageVersion up-to-date unless you need to maintain a specific version.

The problem with the entire test run crashing if there's a COMException when trying to start the app still should probably be fixed, but I think that's an issue with VSTest. Either way, the above changes avoid that because the app will launch correctly and testing will work the same as was described in the original article!

(Note: This is the updated version mentioned in the edit note at the top.)
MSTestWinAppSDK11BugSolved.zip

[Evangelink]

Really sorry for the super long delay folks...

@mikebmcl I remember that the 1st time I tried your repro I was able to see the issue but I cannot repro anymore. We tried to create a new repro but all our tries failed. Would you mind sharing if you are still able to reproduce the issue? If so what are the conditions (same project listed above? what steps are you doing? Would you mind also sharing the result of (Get-AppxPackage *windowsapprun*).PackageFullName (it should display the list of installed WinUI runtimes).

I have pinged @mikebmcl but obviously we are interested in all your feedback about this issue.

[mikebmcl]

@Evangelink I do still encounter it when using MSTestWinAppSDK11BugRepro above with Test11|x86 configuration.

(Get-AppxPackage *windowsapprun*).PackageFullName
Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x64__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x86__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.2-experimental1_2000.572.710.0_x86__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x64__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x86__8wekyb3d8bbwe

When using MSTestWinAppSDK11BugSolved above, which includes
<WindowsAppContainer>true</WindowsAppContainer>
in the first <PropertyGroup> for the project, then the issue disappears. What I determined was that Microsoft.WindowsAppSDK versions 1.0.X defined this property but versions 1.1.X do not. MSTestDiscoverer seems to rely on that property being set to determine whether or not the project is an Appx container and thus whether it should run it using the test host code path that's meant for WinUI or the default test host.

I found that sometimes VSTest will not pickup changes in the repro after changing the solution configuration just by doing a Rebuild. Instead I needed to do a Clean first followed by a Build or Rebuild. I think it's because of the conditional package inclusion in the .csproj that enables testing against WindowsAppSDK 1.0.X and 1.1.X versions.

Tested with Visual Studio 2022 Community 17.3.6 on Windows 11 Pro 10.0.22623 Build 22623.

[Evangelink]

Thank you! I will try to create a clean VM and see if I can repro because locally I can't :c

(Get-AppxPackage windowsapprun).PackageFullName Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x64__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x86__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.2-experimental1_2000.572.710.0_x86__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x64__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x86__8wekyb3d8bbwe

My runtimes are similar to yours (I don't have 1.2):

Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x86__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x64__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x64__8wekyb3d8bbwe
Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x86__8wekyb3d8bbwe

I found that sometimes VSTest will not pickup changes in the repro after changing the solution configuration just by doing a Rebuild. Instead I needed to do a Clean first followed by a Build or Rebuild. I think it's because of the conditional package inclusion in the .csproj that enables testing against WindowsAppSDK 1.0.X and 1.1.X versions.

I have seen similar behavior. I am also ensuring that I uninstall app between builds (I am not super familiar with WinUI so I don't know if it could be messing up).

[Evangelink]

I have managed to reproduce the issue consistently! I have tweaked a little your csproj (see below) so that I can test the various combinations of x86, x64, Test10 and Test11.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>WinExe</OutputType>
    <TargetFramework>net6.0-windows10.0.19041.0</TargetFramework>
    <TargetPlatformMinVersion>10.0.17763.0</TargetPlatformMinVersion>
    <RootNamespace>MSTestWinAppSDK11BugRepro</RootNamespace>
    <ApplicationManifest>app.manifest</ApplicationManifest>
    <Platforms>x86;x64;arm64</Platforms>
    <RuntimeIdentifiers>win10-x86;win10-x64;win10-arm64</RuntimeIdentifiers>
    <PublishProfile>win10-$(Platform).pubxml</PublishProfile>
    <UseWinUI>true</UseWinUI>
    <Configurations>Debug;Release;Test10;Test11</Configurations>
	  
	<!-- 
	  For Test10 (x86 and x64), only the following property is required.
	  If property is not present, the tests won't run. 
	  I couldn't test arm64:
	  DEP3308: Deployment target 'Local Machine' does not support projects targetting ARM64 platform. Supported platforms: X86,X64.
	-->
	<!--<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>-->
	  
    <!--
      For Test11 (x86 and x64), a combination of WindowsAppContainer and
	  EnableMsixTooling OR EnablePreviewMsixTooling is required.
	  I couldn't test arm64:
	  DEP3308: Deployment target 'Local Machine' does not support projects targetting ARM64 platform. Supported platforms: X86,X64.
	-->
	<EnableMsixTooling>true</EnableMsixTooling>
	<!--<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>-->
    <WindowsAppContainer>true</WindowsAppContainer>
  </PropertyGroup>

  <ItemGroup>
    <Content Include="Assets\SplashScreen.scale-200.png" />
    <Content Include="Assets\LockScreenLogo.scale-200.png" />
    <Content Include="Assets\Square150x150Logo.scale-200.png" />
    <Content Include="Assets\Square44x44Logo.scale-200.png" />
    <Content Include="Assets\Square44x44Logo.targetsize-24_altform-unplated.png" />
    <Content Include="Assets\StoreLogo.png" />
    <Content Include="Assets\Wide310x150Logo.scale-200.png" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.TestPlatform.TestHost" Version="17.3.0">
      <ExcludeAssets>build</ExcludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.Windows.SDK.BuildTools" Version="10.0.22621.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.10" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.10" />
    <Manifest Include="$(ApplicationManifest)" />
  </ItemGroup>
  <ItemGroup>
    <ProjectCapability Include="TestContainer" />
  </ItemGroup>

  <ItemGroup Condition="'$(Configuration)'=='Test10'">
    <PackageReference Include="Microsoft.WindowsAppSDK" Version="1.0.4" />
  </ItemGroup>

  <ItemGroup Condition="'$(Configuration)'!='Test10'">
    <PackageReference Include="Microsoft.WindowsAppSDK" Version="1.1.4" />
  </ItemGroup>

  <!-- 
    Defining the "Msix" ProjectCapability here allows the Single-project MSIX Packaging
    Tools extension to be activated for this project even if the Windows App SDK Nuget
    package has not yet been restored.
  -->
  <ItemGroup Condition="'$(DisableMsixProjectCapabilityAddedByProject)'!='true' and '$(EnableMsixTooling)'=='true'">
    <ProjectCapability Include="Msix" />
  </ItemGroup>

  <PropertyGroup>
    <DefineConstants>TESTRUNNER</DefineConstants>
  </PropertyGroup>

  <!-- 
    Defining the "HasPackageAndPublishMenuAddedByProject" property here allows the Solution 
    Explorer "Package and Publish" context menu entry to be enabled for this project even if 
    the Windows App SDK Nuget package has not yet been restored.
  -->
  <PropertyGroup Condition="'$(DisableHasPackageAndPublishMenuAddedByProject)'!='true' and '$(EnableMsixTooling)'=='true'">
    <HasPackageAndPublishMenu>true</HasPackageAndPublishMenu>
  </PropertyGroup>
</Project>

For Test10 (x86 and x64), the only way to have the test working (for me) was to add <EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>.

For Test11 (x86 and x64), the only way to have the test working (for me) was to have either

<EnableMsixTooling>true</EnableMsixTooling>
<WindowsAppContainer>true</WindowsAppContainer>

or

<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>
<WindowsAppContainer>true</WindowsAppContainer>

Could you confirm this is what you have on your side?

Note that for Test11, I had to uninstall app and restart VS between changes to be sure of what was really working or not. Simple update/rebuild after a successful run were often resulting in the new setup to also work while if I uninstall and restart VS it was not working anymore.

I will start back discussion with WinUI team now that I have a consistent reproduceable scenario.

[mikebmcl]

That essentially matches what I have, yes. I define both

<EnableMsixTooling>true</EnableMsixTooling>

and

<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>

for all configurations.

WindowsAppSDK 1.0 does not reference EnableMsixTooling since the feature was still in preview then so including it does nothing. WindowsAppSDK 1.1 added EnableMsixTooling. SDK 1.1 checks for EnablePreviewMsixTooling or EnableMsixTooling and builds and runs correctly with either or both defined. I expect that in some future version WindowsAppSDK will probably deprecate and remove EnablePreviewMsixTooling. Since SDK 1.1 did not generate any warnings or errors when both are defined I included both for all configurations in case EnablePreviewMsixTooling is removed in the future.

I too can only get Test11 to work when
<WindowsAppContainer>true</WindowsAppContainer>
is included along with either or both of EnablePreviewMsixTooling and EnableMsixTooling.

I found that doing a Clean followed by Rebuild would display the correct behavior when switching to or from Test11. I did not need to explicitly uninstall the app or restart VS, but that just saves a bit of time.

[Evangelink]

Just a quick message to say that we (with WinUI team) are still investigating this issue.

[Evangelink]

Hey there! We identified a root cause of this error and filled a Dev Community bug for a fix. We also identified some improvements on WinUI and MSTest sides and we also discussed a way to improve our testing matrix to reduce likelihood of such issue in the future.

Thanks again for your patience.

[Evangelink]

I will move forward by closing this issue as the fix won't be done in MSTest. As I said already, thank you all for your patience and the big investigation effort you did (special mention to @mikebmcl).

[khoabui1412]

I still meet the error: UITestMethodAttribute.DispatcherQueue should not be null. in unpackaged mode.
My project config:

<PropertyGroup>
    <OutputType>WinExe</OutputType>
    <TargetFramework>net6.0-windows10.0.19041.0</TargetFramework>
    <TargetPlatformMinVersion>10.0.17763.0</TargetPlatformMinVersion>
    <RootNamespace>UnitTest</RootNamespace>
    <ApplicationManifest>app.manifest</ApplicationManifest>
    <Platforms>x86;x64;arm64</Platforms>
    <RuntimeIdentifiers>win10-x86;win10-x64;win10-arm64</RuntimeIdentifiers>
    <PublishProfile>win10-$(Platform).pubxml</PublishProfile>
    <UseWinUI>true</UseWinUI>
    <EnableMsixTooling>true</EnableMsixTooling>
    <WindowsAppSDKSelfContained>True</WindowsAppSDKSelfContained>
    <WindowsPackageType>None</WindowsPackageType>
    <EnableMsixTooling>true</EnableMsixTooling>
  </PropertyGroup>

  <ItemGroup>
	<ProjectCapability Include="TestContainer" />
    <Content Include="Assets\SplashScreen.scale-200.png" />
    <Content Include="Assets\LockScreenLogo.scale-200.png" />
    <Content Include="Assets\Square150x150Logo.scale-200.png" />
    <Content Include="Assets\Square44x44Logo.scale-200.png" />
    <Content Include="Assets\Square44x44Logo.targetsize-24_altform-unplated.png" />
    <Content Include="Assets\StoreLogo.png" />
    <Content Include="Assets\Wide310x150Logo.scale-200.png" />
  </ItemGroup>

  <ItemGroup>
	  <PackageReference Include="Microsoft.TestPlatform.TestHost" Version="17.5.0" />
	  <PackageReference Include="Microsoft.Windows.SDK.BuildTools" Version="10.0.22621.755" />
	  <PackageReference Include="Microsoft.WindowsAppSDK" Version="1.2.230313.1" />
	  <PackageReference Include="coverlet.collector" Version="3.2.0">
		  <PrivateAssets>all</PrivateAssets>
		  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
	  </PackageReference>
	  <PackageReference Include="MSTest.TestAdapter" Version="3.0.2" />
	  <PackageReference Include="MSTest.TestFramework" Version="3.0.2" />
    <Manifest Include="$(ApplicationManifest)" />
  </ItemGroup>

[Evangelink]

@khoabui1412 Are any workaround mentionned in this comment #1127 (comment) solving the issue for you or it is still broken with the workarounds?

[khoabui1412]

@Evangelink, it's still broken, it's ok if I set app as packaged app, but my case I want to run in unpackaged mode.

[Evangelink]

Hi @khoabui1412,

Could you please let me know if you have the issue only in VS or also on command line (calling vstest.console.exe)?
Would you mind providing me with a repro project? I am not super experienced with WinUI so having your exact issue would help me with investigations.

[khoabui1412]

@Evangelink , it also has same error when calling by vstest.console.exe.
Here're sample project:
UnpackagedWinUI3.zip

[Evangelink]

@khoabui1412 I have moved to a different error (no longer on MSTest side) by setting [assembly: WinUITestTarget(typeof(UnitTest.App))] in the project.

The active test run was aborted. Reason: Test host process crashed

Test Run Aborted with error System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable() in /_/src/Microsoft.TestPlatform.CommunicationUtilities/LengthPrefixCommunicationChannel.cs:line 90
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/TcpClientExtensions.cs:line 53
   --- End of inner exception stack trace ---.

Could you try on your end and confirm you see the same issue?

[khoabui1412]

@Evangelink,
Iam not sure how to set that, I try to set that in obj folder in UnitTest.AssemblyInfo.cs file and It still meets the error about MSTest as before.

[BarcoScott]

Hey all.

In order to be able to include a QA gate for merging PRs in my github repo for my WinUI application, I need to create a github action to run my unit tests, but I can't seem to make and tests that require UITestMethod work from the command line, which brought me to this thread.

Has anyone gotten this to work with vstest.console.exe? If so, can you provide an example. (For extra credit, do you have a github workflow file that uses it?)

Here's what I've done so far in trying to get this to work.

Starting with the MSTestWinAppSDK11BugSolved example above, I removed the #IF/ENDIF statements and the conditional elements from the .csproj. Then I upgraded all the packages to latest as of today, 7/7/23, just as I'm using in my own code:

While it runs fine from the latest Visual Studio 2022 Enterprise via the Test Explorer, it fails from the command line as shown here (I edited down the path information, but it's otherwise a straight copy/paste):

PS C:\...\MSTestWinAppSDK11BugSolved> vstest.console /Platform:x86 C:\...\MSTestWinAppSDK11BugSolved\MSTestWinAppSDK11BugSolved\MSTestWinAppSDK11BugRepro\bin\x86\Debug\net6.0-windows10.0.19041.0\win10-x86\MSTestWinAppSDK11BugRepro.dll
Microsoft (R) Test Execution Command Line Tool Version 17.6.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed ShouldAlwaysSucceed [2 ms]
  Failed IsPackaged [116 ms]
  Error Message:
   Assert.IsTrue failed. PackagingData.IsPackagedProcess returned false when true was expected.
  Stack Trace:
     at MSTestWinAppSDK11BugRepro.Tests.IsPackaged() in C:\...\MSTestWinAppSDK11BugSolved\MSTestWinAppSDK11BugSolved\MSTestWinAppSDK11BugRepro\Tests.cs:line 25

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.TypeInitializationException: The type initializer for 'WinRT.ActivationFactory`1' threw an exception.
 ---> System.Runtime.InteropServices.COMException (0x80040154): Class not registered (0x80040154 (REGDB_E_CLASSNOTREG))
   at WinRT.BaseActivationFactory..ctor(String typeNamespace, String typeFullName)
   at WinRT.ActivationFactory`1..ctor()
   at WinRT.ActivationFactory`1..cctor()
   --- End of inner exception stack trace ---
   at WinRT.ActivationFactory`1.As(Guid iid)
   at Microsoft.UI.Xaml.Application.Make___objRef_global__Microsoft_UI_Xaml_IApplicationStatics()
   at Microsoft.UI.Xaml.Application.get__objRef_global__Microsoft_UI_Xaml_IApplicationStatics()
   at Microsoft.UI.Xaml.Application.Start(ApplicationInitializationCallback callback)
   at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.<>c__DisplayClass12_0.<InitializeApplication>b__1()
   at System.Threading.Thread.StartHelper.Callback(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Thread.StartCallback()


Test Run Aborted with error System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

Thinking this might be a "PackagingData" issue, I replaced the tests with something simple, e.g. a test that just Asserts that the return from "new BitmapImage()" is non-null, and I get nearly the same behavior, but without the Assert failure message.

I'm hoping I'm just doing something wrong that you can point out to me, perhaps in my command line, or that I need to build something differently, include a different attribute, etc. rather than making me wait for a bug fix.

And, again, a plea to anyone who can share a working example of running UITestMethod-decorated tests from the command line and/or from a github action (I'd even take an NUnit or Xunit example).

(Let me know if I should create a new thread for this issue instead.)

Thanks in advance.

[BarcoScott]

HA! After I hit send, it occurred to me that maybe I didn't follow the vstest.console instructions correctly. I went back and reread the UWP instructions, pointing at the appxrecipe file instead of the DLL (and included the other parameters in the example) and... it worked!

Sorry for the noise. I hope this serves to point someone else in the right direction.

[Evangelink]

Hi @BarcoScott,

Thanks for the information. It would have been better to create a different issue for your question as there is no link with the current issue. Also, vstest.console.exe related questions should be posted in https://github.com/microsoft/vstest repository.

[Evangelink]

@Evangelink , it also has same error when calling by vstest.console.exe. Here're sample project: UnpackagedWinUI3.zip

Bumping all versions to latest, I see the following behaviors:

1. WindowsAppSDKSelfContained and WindowsPackageType commented out, test execution is green
2. <WindowsAppSDKSelfContained>True</WindowsAppSDKSelfContained> and <WindowsPackageType>None</WindowsPackageType>, test execution happens but fails to access the UI dispatcher
3. Other combinations are giving some inconsistent states

I am having a deeper look at why the dispatcher is not found and how to fix it. For the cases of 3, I would wait to get some strong request from users as it seems less reproducible.

[Evangelink]

Using this test project: MSTestWinUI.zip and this test matrix

WindowsAppSDKSelfContained | Not set 			   | True                  | False
WindowsPackageType		   | Not set | None | MSIX | Not set | None | MSIX | Not set | None | MSIX
Result with VSTest         | OK      | KO   | OK   | OK      | KO   | OK   | OK		 | KO   | OK

I end up with only WindowsPackageType set to None being invalid. In such case, the issue is

File name: 'C:\src\temp\MSTestWinUI\UnitTest\bin\x64\Debug\net6.0-windows10.0.19041.0\win10-x64\AppxManifest.xml'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
   at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
   at System.Xml.XmlTextReaderImpl.FinishInitUriString()
   at System.Xml.XmlReaderSettings.CreateReader(String inputUri, XmlParserContext inputContext)
   at System.Xml.Linq.XDocument.Load(String uri, LoadOptions options)
   at Microsoft.VisualStudio.UwpTestHostRuntimeProvider.AppxManifestFile.ReadTargetPlatformVersions()
   at Microsoft.VisualStudio.UwpTestHostRuntimeProvider.AppxManifestFile.get_TargetPlatformVersion()
   at Microsoft.VisualStudio.UwpTestHostRuntimeProvider.Deployer.Microsoft.VisualStudio.UwpTestHostRuntimeProvider.IDeployer.get_TargetPlatformVersion()
   at Microsoft.VisualStudio.UwpTestHostRuntimeProvider.UwpTestHostManager.<LaunchHostAsync>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.UwpTestHostRuntimeProvider.UwpTestHostManager.<Microsoft-VisualStudio-TestPlatform-ObjectModel-Host-ITestRuntimeProvider-LaunchTestHostAsync>d__28.MoveNext()<---

which is an error outside of MSTest. I have reached out to the team and will post some update here.

[Evangelink]

I have checked with internal teams and unpackaged apps is not a supported scenario at the moment. I'll move forward by closing this ticket and opening a new one (see #2784) to add support for this feature.