Add support for NativeAOT

[nohwnd]

Originally posted by @dotMorten in: https://github.com/nohwnd/mstest-runner-examples/issues/1

Summary

One of my biggest griefs with MSTest is I can't test if my stuff works right in an AoT build.
Try building your project with dotnet publish -c Release -p PublishAot=true -r win-x64 and run it, and you'll get loads of build warnings, and a runtime crash.

Now I understand why it currently doesn't work (too much reflection and datacontract serializers), but solving this would be huge for ensuring AoT compatibility.

It would likely require a little code generator for each TestMethod attribute instead of relying on reflection discovery, and using the new System.Text.Json serializers.

AB#1950768

Tasks

Beta Give feedback

1. Add basic support for native AOT #2293
   Area: Native AOT Type: Feature +2
   
2. Add support for AssemblyInitialize/AssemblyCleanup #2754
   Area: Native AOT Type: Feature sprint +3
   
3. Add support for ClassInitialize/ClassCleanup #2755
   Area: Native AOT Type: Feature sprint +3
   
4. Add support for TestInitialize/TestCleanup #2756
   Area: Native AOT Type: Feature sprint +3
   

Blog post: https://devblogs.microsoft.com/dotnet/testing-your-native-aot-dotnet-apps/

[nohwnd]

The new MSTest runner which lives in https://github.com/microsoft/testfx/tree/main/src/Platform/Microsoft.Testing.Platform is internally fully NativeAOT compatible, and can already do this. It inserts itself into the executable you building, which makes it work nicely with dotnet publish and it's various options. Including trimming and PublishAOT.

What is missing is a test framework that can do such thing. As you describe MSTest relies internally on reflection. But we have an alternative implementation which is now shipped in Microsoft.Testing.Framework, and we want to eventually use the same approach for mstest.

As an output you will get an application that can run tests, and has no additional dependencies.

I've added an example here, of how you can achieve nativeaot test running with the existing pieces right now. For actual integration of this into MSTest we don't have a public plan yet.

https://github.com/nohwnd/mstest-runner-examples/blob/main/NativeAOTProject1/README.md

(this example is outdated and the repo is removed, refer to the mstest example below in this thread. )

[testplatform-bot]

✅ Successfully linked to Azure Boards work item(s):

• User Story 1950768

[Evangelink]

This ticket will be used as epic so removing it from 3.3 miletsone.

[nohwnd]

Here is super early preview of the nativeAot for MSTest.

We've created a whole new engine and a source generator that are published as MSTest.Engine and MSTest.SourceGeneration nuget packages on our preview feed (https://aka.ms/mstest/preview) under our Microsoft Testing Platform Tools license.

Both the engine and source generator are in alpha, so please don't consider any of the public APIs final.

Here is an example of project that can run tests in native aot. Putting it together it quite manual, but we are working on improving this.

How to use?

Create a new mstest or console project. Set <OutputType>exe</OutputType> and <PublishAot>true</PublishAot>. Reference the listed packages, and add test application setup to Main method.

Run dotnet publish -r win-x64 to publish the application.

Run the application from the publish folder.

<!-- file TestProject1.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <OutputType>exe</OutputType>
    <PublishAot>true</PublishAot>
  </PropertyGroup>

  <ItemGroup>
    <!-- From preview feed. -->
    <!-- 
      Experimental MSTest Engine & source generator, 
      close sourced, licensed the same as our extensions 
      with Microsoft Testing Platform Tools license.
    -->
    <PackageReference Include="MSTest.Engine" Version="1.0.0-alpha.24151.3" />
    <PackageReference Include="MSTest.SourceGeneration" Version="1.0.0-alpha.24151.3" />
    
    <!-- From nuget.org. -->
    <PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="1.0.2" />
    <PackageReference Include="MSTest.TestFramework" Version="3.2.2" />
    <PackageReference Include="MSTest.Analyzers" Version="3.2.2" />
    
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\ClassLibrary1\ClassLibrary1.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
  </ItemGroup>

</Project>

// file Program.cs
using Microsoft.Testing.Extensions;
using Microsoft.Testing.Framework;
using Microsoft.Testing.Platform.Builder;
namespace TestProject1
{
    internal static class Program
    {
        public static async Task<int> Main(string[] args)
        {
            var builder = await TestApplication.CreateBuilderAsync(args);
            // Registers TestFramework, with tree of test nodes 
            // that are generated into your project by source generator.
            builder.AddTestFramework(new SourceGeneratedTestNodesBuilder());
            builder.AddTrxReportProvider();
            var app = await builder.BuildAsync();
            return await app.RunAsync();
        }
    }
}

// file UnitTest1.cs
using ClassLibrary1;

namespace TestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            Assert.AreEqual(3, new Class1().Add(1, 2));
        }

        [TestMethod]
        [DataRow(1, 2)]
        public void TestMethod2(int left, int right)
        {
            Assert.AreEqual(3, new Class1().Add(left, right));
        }

        [TestMethod]
        [DynamicData(nameof(Data))]
        public void TestMethod3(int left, int right)
        {
            Assert.AreEqual(3, new Class1().Add(left, right));
        }

        [TestMethod]
        [DynamicData(nameof(Users))]
        public void TestMethod4(User u)
        {
            Assert.AreEqual(3, new Class1().Add(u.Left, u.Right));
        }

        public static IEnumerable<object[]> Data { get; } =
        [
            [1, 2],
            [-3, 6],
        ];

        public static IEnumerable<object[]> Users { get; } =
        [
            [new User { Left = 1, Right = 2 }],
            [new User { Left = -3, Right = 6}],
        ];
    }

    public class User
    {
        public required int Left { get; init; }
        public required int Right { get; init; }
    }
}

source: mstest-native.zip

What is working?

• Simple test TestMethod
• TestMethod with DataRow
• TestMethod with DynamicData

What is not working?

• Setups, teardowns, parallelization options, and anything else that you can think of that is not in the working list is probably not working.

• Classes that are static, or partial get silently skipped over.

Let us know which features you miss the most so we can prioritize.

[SGStino]

This works great when running the published executable, but is it supposted to work with the visual studio integration at the moment?

Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Could not find testhost

I'm not sure how visual studio runs the selected tests, but since it is actually discovering them, my guess is that it's not through compiled code, but rather source analysis and invoking methods dynamically in the target assembly?

[Evangelink]

It's only working/supported when using MSTest runner.

[tastratton]

So glad you are adding AOT test capability!

I've been able to create both AOT and JIT test projects by writing AOT tests first and then linking the .cs files from the AOT project into JIT MSTEST projects. 

Is there any way to tell the Test Explorer in Visual Studio to just completely ignore the AOT tests?

the errors Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Could not find testhost
in the output window when Test Explorer is used are a mild nuisance.