VsTest invokes the wrong ITestExecutor method when the user chooses to Run All

[plioi]

Description

Test adapters implement ITestExecutor, which has 2 overloads for the RunTests entry point method.

void RunTests(IEnumerable<string> sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
void RunTests(IEnumerable<TestCase> tests, IRunContext runContext, IFrameworkHandle frameworkHandle)

The first is intended for "Run All", while the second is intended for when the user has selected a few specific tests to run.

At some point, VsTest stopped calling the first method for Run All, and treats all runs as if the user was selecting few tests to run. In the case of the user choosing to Run All, this amounts to the tests parameter containing an item for every single test in the assembly. This means that each test framework has to cross reference that list against their own internal reflection work in order to run tests.

This is at best a memory leak, and at worst a performance hit within each test framework, as a test project grows in size.

This also had the side effect of breaking NUnit: nunit/nunit3-vs-adapter#658

Environment

Visual Studio Community 2019 v 16.5.4

[nohwnd]

@drognanar is this related to the jsonrpc overload problem you were solving recently?

[drognanar]

this is related to the change how we trigger tests for run all

tagging @jfleisher as he's been looking into nunit/nunit3-vs-adapter#658

[nohwnd]

This is by design now, we are working on making discovery faster and faster, and moving vstest.console in-process, so we would end up with just one serialization. TE sends all testcases so then can run only tests that were actually discovered, and not all. But maybe there is more nuance to it.

[plioi]

That doesn't make sense. A test framework still has to do its own concept of discovery/vetting against the giant unbounded list, to only run things actually discoverable. Waiting for TE to do all that irrelevant prework when a simple "run all" signal would have sufficed is only slowing things down at every step.