Run only specific tests when using dotnet test?

[timbussmann]

This official docs page documents how to selectively run tests using the --filter argument when using dotnet test. It shows what filters are supported by mstest and xunit but I haven't found any information regarding supported filters for nunit. Do you have any more information on how to run only specific tests when using dotnet test?

[PierreRoudaut]

It is actually doable to use dotnet --filter with nunit. Here's a summary on what I found on this article

using NUnit.Framework;
using System.ComponentModel;

namespace nunit_exercise
{
  public class Tests
  {
    [Test]
    [Category("Foo")]
    public void Test1()
    {
      Assert.Pass();
    }

    [Test]
    public void MyTest2()
    {
      Assert.Pass();
    }
    [Test]
    [Category("Foo")]
    public void MyTest()
    {
      Assert.Pass();
    }
  }
}

• dotnet test --filter Name=Test1 (will run Test1 only)
• dotnet test --filter TestCategory=Foo (will run Test1 and MyTest )
• dotnet test --filter FullyQualifiedName~nunit_exercise.Tests (will run all tests)
• dotnet test --filter FullyQualifiedName~nunit_exercise.Tests.My (will run MyTest and MyTest2)

Those commands can of course be used in combinaison of conditional operators | and &

Cheers !

[CharliePoole]

The filters supported are the standard VS filters and are the same for all frameworks that allow them. NUnit does allow those filters although they work according to VS and not NUnit semantics. The main difference is that Explicit tests are not recognized as requiring special handling.

At one point we had an issue regarding possibly permitting use of NUnit's own filtering syntax (Test Selection Language) but that has never been implemented.

[harleydk]

And

• dotnet test --filter TestCategory!=IntegrationTests

will run all categories besides integration-tests. HTH.

[OsirisTerje]

You can now use the NUnit filtering syntax (from version 3.16), see release notes for details,

The MS Docs also now contain information for NUnit, so that issue is also resolved.

[molszews]

@OsirisTerje will v3.16.1 be published on nuget feed?

[OsirisTerje]

@molszews Yes, ETA is Saturday

[OsirisTerje]

Also see details in this blogpost

[nicomartinezdev]

Not sure this is worthy of its own thread, so posting the question here in case I'm missing something really basic:
Is it possible to include and exclude test categories in the same filter?
i.e:

dotnet test --filter "TestCategory=Category1&TestCategory!=Category2"

This syntax is not doing what I expect, which would be to run all tests of "Category1" that are not part of "Category2". Instead I'm getting:
TestCategory!=Category2: command not found

Thanks for taking a look ;)

[CharliePoole]

You need to escape the & on the command line in whatever way your shell requires. I can't be more specific, not knowing the platform you're using.

[nicomartinezdev]

You need to escape the & on the command line in whatever way your shell requires. I can't be more specific, not knowing the platform you're using.

This is currently failing in Azure DevOps running on a Linux agent (it works locally in Windows and MacOS)

[nicomartinezdev]

Thanks @CharliePoole! I escaped it and now it works as expected.
In case someone else ever encounters this:

dotnet test --filter "TestCategory=Category1\&TestCategory!=Category2"

This did the trick

[fescobar]

You can run multiple TestCategories running this:

dotnet test --filter "TestCategory=Category1|TestCategory==Category2"

[Puvvada-Ananth]

What is the way to run multiple include and multiple exclude in a single run for example my requirement is that

dotnet test --filter TestCategory=category1&TestCategory!=category2&category3 vice versa
dotnet test --filter TestCategory=category1&category2&TestCategory!=category3
Is it making any sense or do we have a different approach to achieve this ? I am trying to implement this in a CI tool and using linux platform to run it.
Thanks in advance.

[OsirisTerje]

@Puvvada-Ananth The 1st line looks like the correct one, the second will not work. But, when you run these, do they do what you expect?
You can also have a look at the NUnit Where syntax, which can be used for the same purpose. See the docs, and the Tips&Traps section.

[Puvvada-Ananth]

Hi @OsirisTerje
The first method is also failing with following error
No test matches the given testcase filter

[OsirisTerje]

Just tried it here, and the syntax above just works. If you get the messages you show, that means your filter doesn't match any tests.
If you're on Linux, then check if you need to escape the | and & operators , like &.

Also check your exact syntax:

dotnet test --filter "TestCategory=category1&TestCategory!=category2"

And always

<property><operator><value>

, so don't add any multiple values in any other form.

I see you have an &category3 on the first one, that doesn't work.

PS: Did you try the where syntax?

[Puvvada-Ananth]

@OsirisTerje
yes this is working,

dotnet test --filter "TestCategory=category1&TestCategory!=category2"

What if i wanted to use &category3 and etc., ?

[OsirisTerje]

dotnet test --filter "TestCategory=category1&TestCategory!=category2&TestCategory!=category3"

[bewinter]

Sorry to add another comment to this closed issue, but this seems to be the best matching place of all the resources I found so far.

dotnet test --filter "TestCategory=category1&TestCategory!=category2&TestCategory!=category3"

@OsirisTerje: Referring to your example call, I'm having serious problems with a rather current TestAdapter version 4.4.2 and an extremely similar call. I've seen some comments in the 4.x.x ReleaseNotes about some filter functionality refactoring but I couldn't see any clear warning that this would require users to rewrite all existing filter strings (only of a specific format?) in their code. Do you know whether this should work in >=v4.0.0? In my tests, it only worked until 3.17.0...

In my case, a test method with attribute Category("LiveTest") runs (unexpectedly!) with a call like this:

dotnet test tests/Foo.Tests.csproj --configuration Debug --filter 'TestCategory\!=LiveTest'

(same without the escaping backslash)

But is does not run, i.e. skip the test (as expected!) with this call:

dotnet test tests/Foo.Tests.csproj --configuration Debug --filter 'TestCategory=LiveTestXXXXX'

I tried lots of variations of the syntax, also with boolean operators between multiple category checks (which would be the actual production use case), but it seems there's something wrong with the negation. Can anyone who also found this issue during research confirm this behaviour? May this be a bug and should I open a real issue?

[OsirisTerje]

@bewinter Can you upload your test project, just to save some time here.

@OsirisTerje: Referring to your example call, I'm having serious problems with a rather current TestAdapter version 4.4.2 and an extremely similar call. I've seen some comments in the 4.x.x ReleaseNotes about some filter functionality refactoring but I couldn't see any clear warning that this would require users to rewrite all existing filter strings (only of a specific format?) in their code. Do you know whether this should work in >=v4.0.0? In my tests, it only worked until 3.17.0...

Not sure I understand what your ask is here. Users don't need to rewrite anything in their code afaik. Can you point to what you noticed?

[OsirisTerje]

I used the repro at https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue425 . It has some methods with cat Foo and one with Bar, one with both and one with none. I ran the following
dotnet test --filter "TestCategory=Foo" --logger "Console;verbosity=Normal"
The result as expected:

Then I changed to a negative filter:
dotnet test --filter "TestCategory!=Foo" --logger "Console;verbosity=Normal"
and got as expected the other two tests

The only difference I see is that you show single quotes, and not double quotes. But that can be just writing here in markdown.

So, check the repro. If your code differs and give different results, please upload a repro for it.