Hash/Partition based Test Filter

[Plasma]

Suggestion

Implement a Hash/Partition based Test Filter, so that tests match the filter based on a hash and modulo comparison. This is separate and different to parallel execution attribute.

This would allow a subset of tests to be ran (and others excluded) based on a partition number, in order to partition the running of the same (large, slow) test suite across multiple machines by selectively filtering to a subset of tests based on a partition count and number.

Use case

Using GitHub Actions to run nunit tests, it would be great to use the matrix option in GitHub Actions to run multiple copies of the test process across multiple machines at once in order to speed up test runs for large test suites easily.

A filter parameter that allowed me to specify the number of partitions (eg, 10) and the partition number to filter tests for (eg, 3) would make it very easy to partition the running of tests in GitHub Actions and other runners where sub-jobs are supported.

Usage example

--filter <partition/max partitions>

Algorithm may be something like IsMatch == sha1(fully qualified test name) % max partitions == partition

Contribution

I'm happy to contribute a PR if this sounds like a reasonable contribution.

[OsirisTerje]

Sounds like a good idea to me. @nunit/framework-team ?

[stevenaw]

I think it's a neat idea. Filtering by namespace could be another option here, but that requires knowledge of the target test assembly and it's more difficult to generalize. This is a bit more elegant.

We have a bit of a known issue where dynamic values in a ValueSource can change between discovery and execution (#3894) which could complicate this if we were to use a hashed test name. However we partition should ideally be unchanging in all scenarios. Perhaps test id is a better option? It's not 100% deterministic (#2771) but at least it's less dynamic and should be good enough for this case

[Plasma]

Thank you for the feedback; perhaps to simplify could we consider just treating all permutations (TestCase) of the same fully-qualified test name as the same (for partitioning purposes)? The nice thing about just indicating the partition number and total count is that we don't need to document/configure/list namespaces (complicated) to just run a subset of (arbitrary) tests.

A future improvement could be to internally switch to deterministic test-IDs (at a test permutation level / TestCase too) when that is possible to improve the partitioning, which can be an implementation detail.

[stevenaw]

@Plasma I think that's a great idea. I was close to suggesting that myself initially 😁
@nunit/framework-team Thoughts or concerns with this approach?

[Plasma]

I've added #4392 as an initial prototype for feedback.

I had trouble testing this outside of the nunit project, I couldn't get nunit from source to work with dotnet test on a test project (any pointers appreciated) as it always complaints no tests were found, even with the Test SDK and adapter from nuget installed.

[OsirisTerje]

@Plasma Which version of the adapter are you using? Since you're branching off main, you need the 4.5 version which is in alpha.

[Plasma]

Thanks @OsirisTerje for the tip, I've re-based the PR under v3.13-dev and was able to test locally.

I've also made nunit/nunit-console#1345 to reflect the change needed to recognize the partition filter token using the console / engine.

I was able to get the partition where filter working for the nunit console runner, however adding support via the dotnet test command appears to need extra work.

Usage example:

# Without any partitioning
nunit3-netcore-console ConsoleApp4.dll --explore

ConsoleApp.Tests.Test1
ConsoleApp.Tests.Test2
ConsoleApp.Tests.Test3
ConsoleApp.Tests.Test4

# Partitioning in use, where we see the tests assigned to partition 1, 2 and 3 of 3 total:
nunit3-netcore-console ConsoleApp4.dll --explore --where "partition == 1/3"
Test Filters
    Where: partition = 1/3

ConsoleApp.Tests.Test4

nunit3-netcore-console ConsoleApp4.dll --explore --where "partition == 2/3"
Test Filters
    Where: partition = 2/3

ConsoleApp.Tests.Test2
ConsoleApp.Tests.Test3

nunit3-netcore-console ConsoleApp4.dll --explore --where "partition == 3/3"
Test Files
    .\bin\Debug\net6.0\ConsoleApp.dll

Test Filters
    Where: partition = 3/3

ConsoleApp.Tests.Test1

# We can also see that partition 1/1 will match all tests
nunit3-netcore-console ConsoleApp4.dll --explore --where "partition == 1/1"

Test Filters
    Where: partition = 1/1

ConsoleApp.Tests.Test1
ConsoleApp.Tests.Test2
ConsoleApp.Tests.Test3
ConsoleApp.Tests.Test4

# And partition 123/1000 is (unlikely!) to match any tests:
nunit3-netcore-console ConsoleApp4.dll --explore --where "partition == 123/1000"

Test Filters
    Where: partition = 123/1000

Example test fixture source:

namespace ConsoleApp
{
    [TestFixture]
    class Tests
    {
        [Test]
        public void Test1()
        {

        }

        [Test]
        public void Test2()
        {

        }

        [Test]
        public void Test3()
        {

        }

        [Test]
        public void Test4()
        {

        }
    }
}