Parameterized fixture with Explicit attribute can not be run when selected by name

[CharliePoole]

Given code like this...

    [TestFixtureSource(nameof(GetParallelSuites)), Explicit]
    public class ParallelExecutionTests
    {
        ...
    }

Running the console with a filter --where test==Namespace.Of.ParallelExecutionTests causes no tests to run. I think that's because we are running the suite that includes all the instances of the fixture but the attribute is applied to each fixture.

AFAIR this works correctly in the case of methods, so we should look at how we do it there.

Workaround for the above, BTW, is to use --where test=~Namespace.Of.ParallelExecutionTests.

[j4nw]

I'd like to look into this

[rprouse]

@j4nw the issue is yours. I am going to assign myself to mentor you on the issue if you have any issues. Please keep us up to date on your progress and let me know if you need any help.

[j4nw]

@rprouse just letting you know that I haven't forgotten about the issue, I haven't been able to find the time these past few days. Thank you

[rprouse]

@j4nw I am putting this issue back up for grabs. Feel free to submit a PR if you come back to it.

[Poimen]

Hi - I came across this bug.

After some digging yesterday, I traced up to the point of running the PerformWork function in CompositeWorkItem.cs

nunit/src/NUnitFramework/framework/Internal/Execution/CompositeWorkItem.cs

Line 85 in 46f8b2a

if (Test.RunState == RunState.Explicit && !Filter.IsExplicitMatch(Test))

When a test fixture is parametric and the where condition is specified, Filter above is the FullNameFilter. This then attempts to compare the parametric fullname to the specified fullname.

As a way of an example (that is very silly):

    [TestFixtureSource(nameof(GetParallelSuites)), Explicit]
    public class SomeTestFixtureForExplicitIssue
    {
        public SomeTestFixtureForExplicitIssue(string word, int num)
        {
        }

        [Test]
        public void Test1()
        {
            Assert.That(1, Is.EqualTo(1));
        }

        static object[] GetParallelSuites = {
            new object[] { "Question", 1 },
            new object[] { "Answer", 42 }
        };
    }

The full name for one of these test fixtures becomes ClassLibrary1.SomeTestFixtureForExplicitIssue("Answer",42) due to it being parametric.

Hence when you run --where test=ClassLibrary1.SomeTestFixtureForExplicitIssue the match on the full name fails as it attempts to match the full name by direct comparison through the ValueMatchFilter:

nunit/src/NUnitFramework/framework/Internal/Filters/ValueMatchFilter.cs

Line 66 in 46f8b2a

return ExpectedValue == input;

The reason using --where test=~ClassLibrary1.SomeTestFixtureForExplicitIssue works is that it detects the ~ and uses the RegEx code path in the ValueMatchFilter to match ClassLibrary1.SomeTestFixtureForExplicitIssue and matches against the full name starting with ClassLibrary1.SomeTestFixtureForExplicitIssue rather than the direct match:

nunit/src/NUnitFramework/framework/Internal/Filters/ValueMatchFilter.cs

Line 64 in 46f8b2a

return input != null && new Regex(ExpectedValue).IsMatch(input);

This leads to a couple of ways in solving the issue (that I could think of):

1. This is the way it is supposed to work and explicit fixtures should be specified (fix the parser to accept parametric fixtures because you can't do --where test=ClassLibrary1.SomeTestFixtureForExplicitIssue("Answer",42))
2. Change the FullnameFilter to do a StartsWith check rather than a == - i.e.
   

   nunit/src/NUnitFramework/framework/Internal/Filters/FullNameFilter.cs

   Line 45 in 46f8b2a

   return Match(test.FullName);

   
   would need to change not to use ValueMatchFilter directly.
3. Introduce a partial name filter matcher that would do point 2 above

I could also be over simplifying the whole process as well.

Hence I was wondering what your take on this would be?

Thanks.

[CharliePoole]

Been a while since I submitted this but I believe what I was suggesting in the initial comment is that Explicit is being applied to each fixture rather than to the suite containing the two fixtures. This may not be consistent with how we handle the same situation for test methods. Someone would have to check.

In your example, I believe the suite is correctly selected and executed but since it contains only Explicit fixtures, nothing further is run.

IOW I don't think it's a problem with filters but with how Explicit is applied.