Exception when using test filters from .runsettings or --filter argument from dotnet test

[thomasbiel]

NUnit 4.0.1
NUnit3TestAdapter 4.5.0
dotnet 6.0 / windows 11
running tests from command line via "dotnet test"

when using a test filter like
(cat != UsesNetwork) and (cat != LongRunning)
in .runsettings file or via --filter argument for dotnet this exception occurs:

Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in [...].dll
An exception occurred in the driver while counting test cases.
at NUnit.Engine.Runners.DirectTestRunner.CountTestCases(TestFilter filter)
at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.VisualStudio.TestAdapter.NUnitEngine.NUnitEngineAdapter.Run(ITestEventListener listener, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\NUnitEngineAdapter.cs:line 108
at NUnit.VisualStudio.TestAdapter.Execution.Run(TestFilter filter, DiscoveryConverter discovery, NUnit3TestExecutor nUnit3TestExecutor) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\Execution.cs:line 51
at NUnit.VisualStudio.TestAdapter.VsTestExecution.Run(TestFilter filter, DiscoveryConverter discovery, NUnit3TestExecutor nUnit3TestExecutor) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\Execution.cs:line 154
at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 295 InnerException: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out or range of valida values (Parameter 'index') Actual value was 0. at NUnit.Framework.Interfaces.TNode.NodeList.ThrowArgumentOutOfRangeException(Int32 index) at NUnit.Framework.Interfaces.TNode.get_FirstChild() at NUnit.Framework.Internal.TestFilter.FromXml(TNode node) at NUnit.Framework.Internal.TestFilter.GetChildNodeFilters(TNode node) at NUnit.Framework.Internal.TestFilter.FromXml(TNode node) at NUnit.Framework.Internal.TestFilter.FromXml(String xmlText) at NUnit.Framework.Api.FrameworkController.CountTests(String filter) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Span1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at NUnit.Engine.Drivers.NUnitNetStandardDriver.ExecuteMethod(MethodInfo method, Object[] args)
at NUnit.Engine.Drivers.NUnitNetStandardDriver.ExecuteMethod(String methodName, Object[] args)
at NUnit.Engine.Drivers.NUnitNetStandardDriver.CountTestCases(String filter)
at NUnit.Engine.Runners.DirectTestRunner.CountTestCases(TestFilter filter)

[OsirisTerje]

The filter terms dont look right. Can you state the complete dotnet test command you're using?

A normal filter should look like:

dotnet test --filter TestCategory!=CategoryA&TestCategory!=CategoryB

Based on your filter command above, it may be that you're using a NUnit.Console Where statement, which also is supported by dotnet test, but then NOT using the --filter command, but the NUnit Where property instead. See Where syntax and the blog post here

For short, using the dotnet test, it would then look like:

dotnet test -- NUnit.Where="cat != CategoryA and cat != CategoryB"

PS: Note the space after --

PS2: If you add in the --filter it will override the Where, so dont do that.

---

If still not working, can you please upload a repro project?

You can a) Add it as a PR to https://github.com/nunit/nunit3-vs-adapter.issues, add it as a separate folder named IssueXXX where the XXX is the issue number, or b) Add it as a zipped attachment here , or c) Add it to your own repo and just post the link.

Make it as small as you can, it should compile "as is".

[thomasbiel]

I got the filter mixed up, I started with --filter TestCategory!=CategoryA&TestCategory!=CategoryB but that didn't work, so I read the documentation and tried the NUnit-specific version -- NUnit.Where="cat != CategoryA and cat != CategoryB", also w/o luck - regardless of adding the filter via CLI or in .runsettings. I will try to boil down a sample for you.

[thomasbiel]

Gotcha - everything works fine until there is a test marked with [Explicit] along with non-marked tests in a fixture.
NUnitFilterSample.zip
just run the test.ps1 und NUnitFilterSample\Product

[OsirisTerje]

What happens when there is an Explicit there?

[thomasbiel]

Then you get the mentioned exception, otherwise the testrun completes

[OsirisTerje]

Thanks @thomasbiel . I can repro this. I made the sample you added a bit simpler, and it seems that it crashes with only the single test class with 2 tests, one explicit, and one not.

Also, it crashes when the filter has two categories, but not with only one.
And the Explicit triggers it, remove that and it works.

I added dump files, and the filter line that is sent to the framework is:

<filter><and><not><cat>Sample</cat></not><not><cat>Stress</cat></not></and><not><prop name='Explicit'>true</prop></not></filter>

I have added two cmd files, one crash.cmd and one works.cmd.

The stack trace indicates the crash happens in the framework, so I will move this issue there.

The repro can be found here: https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue1146/NUnitFilterSample/src

[thomasbiel]

Is the filter XML you dumped as expected? I would like to contribute if I can, maybe you can give a hint where to start.

[OsirisTerje]

Yes, the filter is as expected.

You have to debug the framework. You can either use NUnitLite as the runner, and set up the same filter there, or you can create a new sln with both the framework and the adapter and set the adapter for debugging. The command for that is here: https://docs.nunit.org/articles/vs-test-adapter/Debugging.html

The call chain will also pass through the engine, but you can ignore that one. When it stops in the adapter, then you set further breakpoints in the framework.

[thomasbiel]

first analysis shows that TNode.FromXml produces an invalid output for the XML you dumped:

which leads to a follow-up error since NotFilter does not expect to contain an empty node list.
Are you sure the XML should look like (1) and not like (2)?

[BrightLight]

We tried to update from NUnit 3.16.0 3.14.0 to 4.0.1 today and encountered the same problem.

everything works fine until there is a test marked with [Explicit] along with non-marked tests in a fixture.

The exception happens also if there is a whole fixture that is marked as [Explicit].
I checked our project for use of [Explicit] and found two locations. One was on a test method within within a fixture (which had other, non-explicit tests). The other location was on a fixture itself.

[TestFixture]
public sealed class Foo
{
  [Test]
  public void Test1()  { }

  [Test]
  [Explicit]
  public void Test2()  { }
}

[Explicit]
public sealed class Bar
{
  [Test]
  public void Test1()  { }
}

I had to remove both to get rid of the exception.

[OsirisTerje]

@BrightLight 3.16 ? There is no NUNit 3.16. Do you mean the console ?