Test filter on a SetUpFixture class without namespace run the whole assembly

[Paciv]

With a class like this in the assembly (no namespace):
[SetUpFixture]class AssemblyInitialize { }

running the command:
dotnet test TestAssembly.dll -- NUnit.Where='test=AssemblyInitialize'
Run all the tests in the assembly, instead of... nothing ?

It would be nice if it could only launch the tests in the specified class.

[stevenaw]

@Paciv I haven't personally been able to check this out yet, but there may be a typo in your command (NUni.Where vs NUnit.Where). Assuming that AssemblyInitialize also contains tests, have you also tried using class=AssemblyInitialize instead of test=AssemblyInitialize ?

If it helps, there's a bit more info on the various supported syntaxes here: https://docs.nunit.org/articles/nunit/running-tests/Test-Selection-Language.html

[Paciv]

I fixed the typo in the problem description, thanks for pointing out (there was no typo when trying the commands).

Our SetUpFixture classes contain no tests, only OneTimeSetUp and OneTimeTearDown

The same behavior is observed for every filter type used:
dotnet test TestAssembly.dll -- NUnit.Where='test=AssemblyInitialize'
dotnet test TestAssembly.dll -- NUnit.Where='class=AssemblyInitialize'
dotnet test TestAssembly.dll -- NUnit.Where='AssemblyInitialize'
dotnet test TestAssembly.dll -- NUnit.Testlist='testlist1' (where testlist1 is a text file with a single line containing AssemblyInitialize)

I recognize that executing the command itself is a bit strange in the first place, but it is part of a more complex automatized process using test lists to split tests.

I still think that this is not the expected behavior though, and that would help to have it "fixed" so it only runs the tests from the specified target (which is none in this case), instead of the whole assembly.

[stevenaw]

Thanks @Paciv
Without digging deeper, I'm wondering if you're hitting this behaviour (from https://docs.nunit.org/articles/nunit/writing-tests/attributes/setupfixture.html):

A SetUpFixture outside of any namespace provides SetUp and TearDown for the entire assembly.

While it's a bit different in this case as you're also trying to target only that class. Out of curiousity, do you see the same behaviour if you put the setup in a dedicated namespace? Could you also confirm the version of the .NET and NUnit frameworks you're targeting, as well as if there's some tests in the assembly which contains AssemblyInitialize?

FWIW, there's been some discussions in the past (#3420, #1713) over exactly how to handle these sort of "setup" methods though I don't think it's been considered how it could interact with filtering in this case.

If the dedicated namespace doesn't work for you and you need an immediate resolution, something else you could also consider if a ModuleInitializer could help with assembly-level initialization if you are targeting a runtime which supports it.

[Paciv]

Our AssemblyInitialize classes have been created based on behavior described in the documentation, to make sur the OneTimeSetup is properly executed before any test of the assembly.

The problematic behavior is not observed when the SetupFixture class is in a namespace, but doing that would defy the previous point.

Setup is net6 (or net7) sdk targeting net48 and net6.0 with NUnit 3.13

[stevenaw]

Can you help me understand a bit more about why you need to target this setup class specifically? Based on how you've described it, with no namespace and decorated with a [SetupFixture] and methods for [OneTimeSetup] and [OneTimeTearDown]:

[SetUpFixture]
class AssemblyInitialize
{
  [OneTimeSetUp]
  void A() { /* assembly-level setup here */ }

  [OneTimeTearDown]
  void B() { /* assembly-level teardown here */ }
}

Then AssemblyInitialize.A() should be automatically run once-only per execution of a test assembly, before any desired tests. Similarly, AssemblyInitialize.B() will be run once-only per execution of a test assembly after all desired tests. Assuming that AssemblyInitialize lives inside of TestAssembly.dll, are you trying to only have AssemblyInitialize.A() only run the first time you target tests in the assembly and for nunit to remember that across multiple executions (for example, like below)?

dotnet test TestAssembly.dll -- NUnit.Testlist='testlist1'
dotnet test TestAssembly.dll -- NUnit.Testlist='testlist2'

[stevenaw]

I'm beginning to wonder if you're perhaps looking for the process-wide "global setup / teardown" functionality described in #4106

[mikkelbu]

As far as I have understod SetUpFixture are also considered a kind of test (at least they are part of the test hierarchy like a parent), so the following code

[SetUpFixture]
class AssemblyInitialize
{
    [OneTimeSetUp]
    public void RunBeforeAnyTests()
    {
        //Console.WriteLine("RunBeforeAnyTests");
    }
}

namespace SomeNamespace
{
    [TestFixture]
    public class SomeClass
    {
        [Test]
        public void SomeTest()
        {
            Assert.Pass();
        }
    }
}

is "represented" by the following XML. So even if we don't consider AssemblyInitialize a traditional test it is still possibly to select it and run it.

  <test-suite type="Assembly" id="1-1003" name="MyDll.dll" fullname="RETRACTED\MyDll.dll" runstate="Runnable" testcasecount="1" result="Passed" start-time="2022-12-22T20:34:59.1292051Z" end-time="2022-12-22T20:34:59.3422224Z" duration="0.209488" total="1" passed="1" failed="0" warnings="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.13.3.0" clr-version="MyCLRVersion" os-version="MyOS" platform="Win32NT" cwd="SomeConsole" machine-name="MyMachine" user="MyUser" user-domain="SomeDomain" culture="en-US" uiculture="en-US" os-architecture="x64" />
    <settings>
...
    </settings>
    <properties>
...
    </properties>
    <test-suite type="SetUpFixture" id="1-1000" name="[default namespace]" fullname="AssemblyInitialize" classname="AssemblyInitialize" runstate="Runnable" testcasecount="1" result="Passed" start-time="2022-12-22T20:34:59.1651085Z" end-time="2022-12-22T20:34:59.3422224Z" duration="0.176958" total="1" passed="1" failed="0" warnings="0" inconclusive="0" skipped="0" asserts="0">
      <test-suite type="TestSuite" id="1-1004" name="SomeNamespace" fullname="SomeNamespace" runstate="Runnable" testcasecount="1" result="Passed" start-time="2022-12-22T20:34:59.1740869Z" end-time="2022-12-22T20:34:59.3401908Z" duration="0.166807" total="1" passed="1" failed="0" warnings="0" inconclusive="0" skipped="0" asserts="0">
        <test-suite type="TestFixture" id="1-1001" name="SomeClass" fullname="SomeNamespace.SomeClass" classname="SomeNamespace.SomeClass" runstate="Runnable" testcasecount="1" result="Passed" start-time="2022-12-22T20:34:59.1740869Z" end-time="2022-12-22T20:34:59.3401908Z" duration="0.166254" total="1" passed="1" failed="0" warnings="0" inconclusive="0" skipped="0" asserts="0">
          <test-case id="1-1002" name="SomeTest" fullname="SomeNamespace.SomeClass.SomeTest" methodname="SomeTest" classname="SomeNamespace.SomeClass" runstate="Runnable" seed="1537707564" result="Passed" start-time="2022-12-22T20:34:59.1740869Z" end-time="2022-12-22T20:34:59.3352756Z" duration="0.161250" asserts="0" />
        </test-suite>
      </test-suite>
    </test-suite>
  </test-suite>

I'll see if I can find some more documentation on this, but I guess most of this is in comments to issues.

[Paciv]

@stevenaw
No, the per assembly / namespace setup is the right tool for what we want to achieve

Like I said earlier, this test command is part of a global test strategy. The easiest way to resolve my problem would be to avoid running tests on this filter, though it raises other problems on our side, for which I won't go into details here.

The point of this issue is that running tests with a filter on a SetupFixture decorated class with no namespace run all the tests in the assembly, where I would have expected no tests to run at all.

As @mikkelbu points out, in the test result XML hierarchy, the SetupFixture suite is englobing all the tests under the SetupFixture class namespace (all the assembly if the class has no namespace).
From this is might looks like a deliberate choice to have the all the tests in the assembly launched when asking to run the namespace-less SetupFixture decorated class.
But if that's the expected behavior, we could expect that when the SetupFixture class has a namespace, it launches all the tests in the namespace, which isn't the case.

I am not at work at the moment so I can't confirm, but I remember having no tests run at all when trying with a SetupFixture class in a namespace. Although when I tested again from where I am writing, it actually launches all the tests in the assembly too (NUnit 3.13.3, target net5.0). Both behaviors are unexpected to me.

[OsirisTerje]

@Paciv As pointed out the SetupFixture encapsulates all tests in either an assembly - if placed at that level, or in a namespace - if placed at that level.

That means that asking for tests using the SetupFixture will run all tests in assembly or namespace respectively.

The reason your repro didn't show the same for the namespace might be because the test name was without the namespace perhaps.

I've added a repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4268

and if you run it like this for where the setupfixture is in a namespace:

dotnet test -- NUnit.Where="test=Issue4268.CommonInitialize" NUnit.DumpXmlTestResults=true

and like this when it is global (assembly level)

dotnet test -- NUnit.Where="test=CommonInitialize" NUnit.DumpXmlTestResults=true

You will see that it runs the test in both cases.

The namespace XML is:

<test-suite type='SetUpFixture' id='0-1006' name='Issue4268' fullname='Issue4268.CommonInitialize' runstate='Runnable' testcasecount='1'>
         <test-suite type='TestFixture' id='0-1007' name='SomeTests' fullname='Issue4268.SomeTests' runstate='Runnable' testcasecount='1'>
            <test-case id='0-1002' name='TestIt' fullname='Issue4268.SomeTests.TestIt' methodname='TestIt' classname='Issue4268.SomeTests' runstate='Runnable' seed='738487075' />
         </test-suite>
      </test-suite>

So this is by design.

We should however clarify this in the documentation.