Regex --where class with negative lookahead doesn't filter tests if SetUpFixture is used

[k15tfu]

Hi!

It turned out that =~ operator with a negative lookahead doesn't filter test classes if SetUpFixture is used in this namespace:

using NUnit.Framework;

namespace nunit_classfilter_csharp
{
    [SetUpFixture]
    public class MySetUpClass  // <-- but OK without this fixture
    {
    }

    [TestFixture]
    public class EnabledTest
    {
        [Test]
        public void Test()
        {
        }
    }

    [TestFixture]
    public class DisabledFooTest
    {
        [Test]
        public static void Test()
        {
        }
    }

    [TestFixture]
    public class DisabledBarTest
    {
        [Test]
        public void Test()
        {
        }
    }
}

and run with:

$ mono ./tools/nunit3-console.exe .../bin/Debug/nunit_classfilter_csharp.dll --where "class=~/^nunit_classfilter_csharp\\\\.(?!(DisabledFooTest|DisabledBarTest)\\\\b)/" --noresult --process:InProcess
NUnit Console 3.15.0 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Sunday, April 24, 2022 6:23:18 PM

Runtime Environment
   OS Version: Linux 4.15.0.176
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    /home/user/p/test/nunit_classfilter_csharp/bin/Debug/nunit_classfilter_csharp.dll

Test Filters
    Where: class=~/^nunit_classfilter_csharp\\.(?!(DisabledFooTest|DisabledBarTest)\\b)/


Run Settings
    ProcessModel: InProcess
    DisposeRunners: True
    WorkDirectory: /home/user/p/test/nunit_classfilter_csharp
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.8
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: net-4.8
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Passed
  Test Count: 3, Passed: 3, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2022-04-24 16:23:18Z
    End time: 2022-04-24 16:23:18Z
    Duration: 0.306 seconds

but without this MySetUpClass fixture everything is fine and it runs the same set of tests as with

--where "class!~/^nunit_classfilter_csharp\\\\.(DisabledFooTest|DisabledBarTest)\\\\b/"

, or

--where "class=~/^nunit_classfilter_csharp\\\\.EnabledTest\\\\b/"

options.

NUnit Console 3.15.0
NUnit 3.13.3

[CharliePoole]

Summary of how filtering works...

1. The console runner takes the expression you pass as a where argument and creates a string representing your filter.
2. That filter is passed to the engine.
3. The engine passes the filter to the NUnit framework.
4. The NUnit framework sees that a regular expression is involved and creates a C# regular expression, which it applies.

Note that item 4 implies that the filter is evaluated by the platform under which you tests are run. I'm not sure if all platforms support negative lookahead, so that will require checking.

Since no part of NUnit actually interprets the regex, there are two possibilities.

1. The .NET class library is not handling the regex correctly.
2. NUnit is using the filter results incorrectly.

Figuring out which of the two applies will take some debugging.

[k15tfu]

@CharliePoole Thanks for the clarification, given that it works w/o SetUpFixture, I guess it's not about negative lookahead support in the .NET class library. BTW I checked that on Windows under .NET Framework and on Linux using Mono.

[CharliePoole]

I'm wondering if the filter is being used to exclude the setup fixture, which would also keep the test fixtures from running. I'll try to track it down further to determine if it gets sent properly to the framework. If it does, then I'll transfer this issue there.