SetUpFixture not run

[provegard]

Description

I'm trying to migrate a test project from .NET Classic 4.7.2 to .NET Core 2.2. I have a class marked with SetUpFixture and inside it a OneTimeSetUp method. It is not run at all.

Versions

NUnit version: 3.12.0
NUnit3TestAdapter version: 3.15.0
Visual Studio: 2017 Professional, 15.9.15
.NET platform: Core 2.2

Repro

using NUnit.Framework;

namespace Tests
{
    public class Tests
    {
        [Test]
        public void Test1()
        {
            Assert.True(SUF.HasRun);
        }
    }
}

[SetUpFixture]
public class SUF
{
    public static bool HasRun;

    [OneTimeSetUp]
    public void BeforeEverything()
    {
        HasRun = true;
    }
}

[mikkelbu]

I'm on the phone, so I cannot test the example now, but are the testfixture and SetUpFixture class in the same namespace? (this is not the case in the example above).

From https://github.com/nunit/docs/wiki/SetUpFixture-Attribute

This is the attribute that marks a class that contains the one-time setup or teardown methods for all the test fixtures under a given namespace.

[provegard]

No they are not, but:

• Moving the SetUpFixture class into the Tests namespace has no effect; it's still not run.
• The code works under .NET 4.7.2 (Visual Studio + ReSharper, no NUnit3TestAdapter installed).
• The page you linked also says:

A SetUpFixture outside of any namespace provides SetUp and TearDown for the entire assembly.

[mikkelbu]

• The page you linked also says:

A SetUpFixture outside of any namespace provides SetUp and TearDown for the entire assembly.

Sorry, I got the test class and setupfixture class swapped around.

I've looked a bit into this, and I don't think that the problem is related to .NET framework vs. .NET core, but is due to the prefix filtering that has been implemented in NUnit for performance (and has been added to the adapter in 3.15). If I go back to the adapter 3.14 then it works on my machine.

I tried to run the test using both the console and the adapter and noticed that they found a different number of classes/fixtures when I executed all the tests. So my guess is that the prefiltering makes NUnit skip the class that is marked with the SetUpFixture.

From the console

InternalTrace: Initializing at level Debug
21:40:10.695 Debug [ 5] DefaultTestAssemblyBuilder: Loading PATH\bin\Debug\net45\SetUpFixture.dll in AppDomain domain-
21:40:10.714 Debug [ 5] DefaultTestAssemblyBuilder: Examining assembly for test fixtures
21:40:10.717 Debug [ 5] DefaultTestAssemblyBuilder: Found 2 classes to examine
21:40:10.839 Debug [ 5] DefaultTestAssemblyBuilder: Found 2 fixtures with 1 test cases
...

From the test adapter

InternalTrace: Initializing at level Debug
21:40:02.096 Debug [12] DefaultTestAssemblyBuilder: Loading PATH\bin\Debug\net45\SetUpFixture.dll in AppDomain domain-2a110627-SetUpFixture.dll
21:40:02.105 Debug [12] DefaultTestAssemblyBuilder: Examining assembly for test fixtures
21:40:02.106 Debug [12] DefaultTestAssemblyBuilder: Found 1 classes to examine
21:40:02.139 Debug [12] DefaultTestAssemblyBuilder: Found 1 fixtures with 1 test cases
...

[provegard]

You're right, I tested with 3.14.0 and it works as expected. Thank you!

[OsirisTerje]

@mikkelbu The nunit console should have the same prefiltereing. Does that work? If so, then there must be something in the code for the adapter. The prefiltering should only affect tests, not setup fixtures. They are not (afaik) not part of the list from the test explorer.
I was afraid of this happening.
If we can't find out quickly what's causing this, i'll it back with a 3.16 version

[OsirisTerje]

We also need a test included which exposes this bug.

[mikkelbu]

@OsirisTerje I'm quite sure that only nunitlite has implemented the prefiltering and nothing has been done in the console (there is an open issue in nunit/nunit-console#438, but that is all). So perhaps this is an issue in the framework and not in the adapter, but I have not looked at the prefiltering code, so this is just my guess.

[OsirisTerje]

@mikkelbu In 3.15 there also was some updates of dependencies. I wonder if there is something there that causes this. There should be no reason whatsoever that this works with .net 4. and not with .net core. It sounds like some engine or framework issue.
I also see from the dump file that the setupfixture is included there.

[mikkelbu]

@OsirisTerje This has nothing to do with .NET framework vs. .NET Core. It is also failing with .NET 4. I just tried to use the prefiltering with nunitlite-runner.exe and that also fails on the example above (if I e.g. gives it the prefilter "Tests" then SUF is not added, as it does not satisfy the filter). So I say that is an framework issue that the prefiltering is also affecting SetUpFixtures.

I'll move the issue to the NUnit project, but I don't have much time at the moment to examine this in more detail.

Ps. SetUpFixture was just my project name.

[OsirisTerje]

@mikkelbu If I run All Tests from test explorer, that is the prefilter should not be in use, and I use 3.15, then it works for .net fw 4.7, and it doesnt for .net core.

[mikkelbu]

@OsirisTerje I can still make it fail also on .NET Framework 4.7 (sometimes I had to do a clean build, or to use git clean, to get consistent results when switching between FW and Core and between different versions of nuget packages).

I've inserted some extra logging in the framework code, and in discovery there is no prefilter, but when I run All tests a filter is used. I'll try to see what kind of prefilter, but I'll probably first have time to do it Tuesday night.

[mikkelbu]

@OsirisTerje Which version of VS are you using. On VS 2019 (version 16.2.3) it fails on .NET Framework 4.7.2, but the same project passes on 15.9.15, so perhaps VS has change how they call the adapter for "All tests" in VS 2019?

[OsirisTerje]

@mikkelbu I am on travel now, so only got my laptop. On my work machine I tested it on VS 2019 16.2.x (dont remember the rev no), and it worked on 4.7 and not on .net core. Got to check with the PG if there has been any changes in the way they call the adapter.
I've looked through the changes on 3.15, and I can't see any assembly changes that are really off.
Which version of NUnit are you testing on? I used the 3.12 latest version.

[ohenriksson]

I have the exact same symptoms in a .Net 4.7.1 project.
All tests are in the same namespace.
[SetUp] is executed but not [OneTimeSetup]
VS 2019(16.2.3)

I've reverted back to the 3.14 adapter.

[OsirisTerje]

@ohenriksson @provegard Version 3.15.1 should now work for you. The prefilter which caused this is turned off by default.

[ohenriksson]

@ohenriksson @provegard Version 3.15.1 should now work for you. The prefilter which caused this is turned off by default.

Thank you! I've upgraded without problems

[provegard]

@OsirisTerje apologies for the late reply. I have ugpraded as well and everything works.