Ease integration with dotnet test by providing EXEs for both sides of the test runner handshake

[plioi]

We have an opportunity to greatly simplify integrating with dotnet-test, without having to alter dotnet.exe's own test command implementation, if we look to the design of the excellent TestDriven Visual Studio plugin as a guide. Here, I sum up the current means of integration, how TestDriven accomplishes the same, and the benefits of that alternative.

Please let me know if I'm missing something!

Integration Today

To implement the dotnet test integration for an existing test framework, I first looked to how xUnit and NUnit have implemented theirs (see dotnet-test-xunit and dotnet-test-nunit). These match the expectations outlined in .NET Core CLI test communication protocol. To meet these requirements, though, both frameworks had to implement a lot of the same things in service to the interprocess handshake before even getting to their own actual test running code, and I'm about to do the same.

For instance, each test framework has to provide an exe rather than a dll, whose main method has to parse dotnet test's own command line arguments in order to initiate the Design Time JSON messaging handshakes or to initiate the alternate console running mode. Both have to reference JSON.NET in order to parse and serialize the payloads going back and forth. They both have to conditionally fire up and maintain an AppDomain, when running on a platform that has AppDomains, to ensure proper handling of assembly loading/unloading and reading from the correct app.config file. They both have to manage what folder is the Current Directory during execution. Also, because they are a full process and have to own parsing of all the command line arguments, this architecture precludes making dotnet test unambiguously support the "--" separator found in other dotnet commands (see #6488).

Integrating with TestDriven

Contrast this design with that of the TestDriven plugin by @jcansdale. That plugin operates under a similar architecture with interprocess communication and plugins provided by an arbitrary test framework. However, all of the processes involved are provided by TestDriven. Each framework need only provide a library which interacts with the process through a few trivial interfaces. Here's the entirety of my integration with TestDriven, for instance: TdNetRunner.cs, TestDrivenListener.cs. The ITestRunner and ITestListener interfaces are owned and provided by TestDriven, and the rest is purely about the specific test framework in question.

With the TestDriven architecture, the individual test framework doesn't have to care about parsing command line arguments, managing the Current Directory, managing the AppDomain on platforms that have such a thing (including such nightmares as discovering the need to manage MarshalByRefObject lifetimes), interprocess communication, and the like. They merely provide an implementation of ITestRunner and report results through ITestListener. If TestDriven didn't do all of that internally, each framework would have to care about those things just to work with TestDriven.

Applying TestDriven's Success to dotnet test

This brings us back to dotnet test and the exe that each test framework provides today. Imagine that the developers of the NUnit and xUnit integrations refactored their implementations with TestDriven as a guide, separating the test framework specific concerns from the dotnet test interprocess communciation concerns, resulting in an exe devoted to dotnet test and a library that implements a simple interface. The exes would be identical, at which point that exe might as well be provided as a part of the dotnet CLI tooling.

The new interfaces could accept the existing JSON payload types already being passed back and forth between processes. This is just one level of indirection away from that communication.

If the new exe parsed the dotnet test command line arguments, passing those parsed values as simple POCOs to a framework's intergration DLL, and if it also passed an args array for everything after "--", we'd also satisfy #6488 and avoid colliding with each framework's own argument naming/style.

If the new exe, when running for a platform that has AppDomains, performed the same AppDomain management that TestDriven and each framework have had to implement again and again, each framework could be simplified internally a great deal, with the added bonus of making the move to cross-platform .NET Core support far easier in each case.

Summary

If both executables in the .NET Core CLI test communication protocol were provided by the donet CLI tooling, and the integration point worked by inspecting a far-simpler framework-specific DLL which communicates through simple interfaces, then .NET test frameworks would be able to integrate with dotnet test far more easily without having to reinvent the wheel and rediscover/fix common bugs, improving adoption of .NET CLI and providing a more consistent experience across test frameworks.

[bradwilson]

How is this different than what you did with RockSteady?

[bradwilson]

How are you going to support the custom command line options we support today? Being a console test runner means needing support for command line filtering, reporting, configuration, etc.

[plioi]

I'm not familiar with RockSteady.

I think I address custom command line args above. If the proposed exe parses the args passed to its own Main method for its own needs, it could pass them along (or possibly just those after "--") into the given test framework's plugin as one of the inputs. Where TestDriven integration involves xUnit, for instance, implementing ITestRunner which accepts AssemblyInfo/MethodInfo from TestDriven into xUnit, the analog to ITestRunner could have arbitrary args[ ] as one of the interface methods' parameters.

[bradwilson]

RockSteady is the test runner in Visual Studio. Compare dotnet-test-xunit with https://github.com/xunit/visualstudio.xunit

[BrainCrumbz]

We're currently inspecting same projects mentioned by @plioi , plus also his own and brand new dotnet-test-mspec, and thinking about implementing the same techniques for NSpec.

We'll probably go the same way: create custom dotnet-test runner, figure out what command line options it can "receive" from dotnet-test in console mode, same thing in design-time mode, see/copy how others are processing command line options, add some kind of controller/façade within framework library assembly, see/copy how others are communicating across assemblies (that is, an implicit API buried in XML strings), and so on.

[BrainCrumbz]

Not sure how much this is known. It seems that might help, at least for command line parsing: https://msdn.microsoft.com/magazine/mt763239

[plioi]

Now that dotnet test integration is changing dramatically with the move back to msbuild-based tooling, this request is probably a moot point and can be closed. It's still a shame that test frameworks have any need to care about AppDomains when the runner (msbuild now) could do that internally.