Parallelizing tests through agents

[matkoch]

Parallelizing tests is usually done by passing only a set of all test assemblies, or filtering them by namespace. I propose a different approach invoked by:

TestFx.exe  --assemblies A.dll B.dll --agent-count 3 --agent-instance 1
TestFx.exe  --assemblies A.dll B.dll --agent-count 3 --agent-instance 2
TestFx.exe  --assemblies A.dll B.dll --agent-count 3 --agent-instance 3

Every invocation gets the full list of assemblies. Internally, all tests get sorted by name, optionally randomized using an external seed.

The execution is then defined by taking every (#AgentInstance - 1) + 3 * n test from all tests.

[matkoch]

Problem

Test suites with excessive setup/cleanup code. Separating them, will drastically increase execution time.

Possible solutions

• Only include tests without test suite setup.
• Opt-out by attribute