New DiscoveryMode doesn't play nicely with TestFixtureSource - Missing GenericFixture

[afscrome]

VS 2022 / dotnet sdk 6.0.100
.Net Core 3.1 & 5.0
NUnit 3.13.2
Nunit3TestAdapter 4.1.0

After upgrading Nunit3TestAdapter from 3.17.0 to 4.1.0, a number of my tests disappeared from the Visual Studio test runner. If I run through dotnet test I get a Not a TestFixture, SetUpFixture, ParameterizedFixture or TestSuite, but GenericFixture exception logged as a warning (see full stack trace below).

I've narrowed this down to being caused by TestFixtureSource - the error can be reproduced with the code sample below.

This seems similar to #869 but I'm still encountering the same exception after upgrading to 4.1. The workaround of changing DiscoveryMode to Legacy in a run settings file does successfully work around this problem.

Repro code:

    [TestFixtureSource(typeof(FixtureSources), nameof(FixtureSources.Types))]
    public class SomeTest<T>
    {
        [Test]
        public void Foo()
        {
            Assert.Pass();
        }
    }

    public static class FixtureSources
    {
        public static Type[] Types =
        {
            typeof(object)
        };
    }

Exception:

Exception NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryException,    Exception thrown executing tests in C:\git\AsosCore\ApiAuthentication\src\Authentication.Web.Host.Tests\bin\Debug\netcoreapp3.1\Authentication.Web.Host.Tests.dll
Not a TestFixture, SetUpFixture, ParameterizedFixture or TestSuite, but GenericFixture
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractTestFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 258
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 219
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 198
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 198
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 198
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 198
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ConvertXml(NUnitResults discovery) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 186
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.Convert(NUnitResults discoveryResults, String assemblyPath) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 140
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping`2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 280

[OsirisTerje]

@afscrome
Thanks!
I believe this is in the same category. I'll add your repro code to the test suite, and get a fix out with 4.1.
I believe this is in the Generic type of tests, and it's not unexpected. But I see some generic tests comes out just as parametrized, so good to have a repro that actually throws that error.

[PatrickLeGarrec]

I use the NUnit Ordering Library.

VS 2022
.Net Core 6.0
NUnit 3.13.2
Nunit3TestAdapter 4.1.0

KO with VS
KO with dotnet test

C:>dotnet test C:\Repositories.Net.Net6\Steprunner_Framework\Fixtures\bin\Debug\net6.0-windows\Fixtures.dll --filter "FullyQualifiedName~Ordered.RootOrderedTestFixture"
Outil en ligne de commande d'exécution de tests Microsoft (R), version 17.0.0
Copyright (c) Microsoft Corporation. Tous droits réservés.

Démarrage de l'exécution de tests, patientez...
Au total, 1 fichiers de test ont correspondu au modèle spécifié.

Exception NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryException, Exception thrown executing tests in C:\Repositories.Net.Net6\Steprunner_Framework\Fixtures\bin\Debug\net6.0-windows\Fixtures.dll
Expected ParameterizedMethod, Theory or GenericMethod, but was TestFixture
at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractParameterizedMethodsAndTheories(NUnitDiscoveryTestFixture tf, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 299
at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 214
at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractTestFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 267
at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 229
at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ConvertXml(NUnitResults discovery) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 185
at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.Convert(NUnitResults discoveryResults, String assemblyPath) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 137
at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 279 Aucun test ne correspond au filtre testcase donné FullyQualifiedName~Ordered.RootOrderedTestFixture` dans C:\Repositories.Net.Net6\Steprunner_Framework\Fixtures\bin\Debug\net6.0-windows\Fixtures.dll

=> Expected ParameterizedMethod, Theory or GenericMethod, but was TestFixture
This is not a TestFixtureSource issue but TestFixture, very similar...

[OsirisTerje]

@PatrickLeGarrec Not sure what the NUnit Ordering library is. But, is this the same situation as reported above? If it is different, please upload a small repro :-)

[PatrickLeGarrec]

herited from

NUnit Ordering library

dotnet test C:\Repositories.Net.Net6\Steprunner_Framework\Fixtures\bin\Debug\net6.0-windows\Fixtures.dll --filter "FullyQualifiedName~Ordered.RootOrderedTestFixture"

Hope that helps

This library is so great that we could not imagine to abandon it ;)
https://github.com/Sebazzz/NUnitTestOrdering

[OsirisTerje]

@PatrickLeGarrec Ok. This looks different from the first reporters issue. Can you provide a very small repro of this case? Attach it here or add link to own repo. I believe it should also go into a separate issue, this is not the same as this one.

[PatrickLeGarrec]

yes, in few days.. i need to quickly rebuild a sample, this one is a business case with many dependencies i cannot provide.

I m pretty sure it is a similar issue due to discover

[OrderedTestGlobalSetUpFixture]
[SetUpFixture]

and

[OrderedTestFixture]

[OsirisTerje]

This issue is about missing Generic Fixture. Yours look like it isn't about that :-)
But awesome if you can get that repro in a few days time :-)

[OsirisTerje]

@afscrome I got the fix in. Your repro works with this change (#918) It would be nice if you can try the enclosed alpha version and verify it works in your scenario.
NUnit3TestAdapter.4.2.0-alpha.1.zip

[PatrickLeGarrec]

here is

https://github.com/PatrickLeGarrec/NUnitFixtureOrdering

[OsirisTerje]

@afscrome Version 4.2.0-beta.1 released now on nuget.org https://www.nuget.org/packages/NUnit3TestAdapter/4.2.0-beta.1 . It also contains an engine upgrade, which is the reason for the beta. Would appreciate confirmation that it still works for you.

[afscrome]

@OsirisTerje 4.2.0-beta.1 works perfectly now. Thank you very much.