NullReferenceException when [TestFixtureSource] refers to data in a generic class.

[yaakov-h]

NUnit output:

Executing: C:\Tools\NUnit3\bin\nunit3-console.exe
NUnit Console Runner 3.0.5813 
Copyright (C) 2015 Charlie Poole

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    ValveKeyValue/ValveKeyValue.Test/bin/Release/ValveKeyValue.Test.dll


Errors and Failures

1) Invalid : ValveKeyValue.Test.ArrayTestCase<TEnumerable>
An exception was thrown while loading the test.
System.NullReferenceException: Object reference not set to an instance of an object.
   at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.BuildFrom(ITypeInfo typeInfo, ITestFixtureData testFixtureData)
   at NUnit.Framework.TestFixtureSourceAttribute.<BuildFrom>d__17.MoveNext()
   at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo)

2) Invalid : ValveKeyValue.Test.TopLevelEnumerableTestCase<TEnumerable>
An exception was thrown while loading the test.
System.NullReferenceException: Object reference not set to an instance of an object.
   at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.BuildFrom(ITypeInfo typeInfo, ITestFixtureData testFixtureData)
   at NUnit.Framework.TestFixtureSourceAttribute.<BuildFrom>d__17.MoveNext()
   at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo)

Run Settings
    WorkDirectory: C:\projects\valvekeyvalue
    NumberOfTestWorkers: 2

Test Run Summary
    Overall result: Failed
   Tests run: 85, Passed: 85, Errors: 0, Failures: 0, Inconclusive: 0
     Not run: 0, Invalid: 0, Ignored: 0, Explicit: 0, Skipped: 0
  Start time: 2016-03-13 09:36:40Z
    End time: 2016-03-13 09:36:41Z
    Duration: 1.241 seconds

Full build process output is available at https://ci.appveyor.com/project/xPaw/valvekeyvalue/build/1.0.0.20-AppVeyor#L60

Seemingly, nunit3-console returns a status code of 0. I'd expect in a failure case such as this, it would return non-zero.

Code that causes this issue:

[TestFixtureSource] here: https://github.com/SteamDatabase/ValveKeyValue/blob/dc20dcde8d4efddfcc0ed7730fd693563401300a/ValveKeyValue/ValveKeyValue.Test/Text/ArrayTestCase.cs#L7
Points to this collection here: https://github.com/SteamDatabase/ValveKeyValue/blob/dc20dcde8d4efddfcc0ed7730fd693563401300a/ValveKeyValue/ValveKeyValue.Test/TestFixtureSources.cs#L9

[ChrisMaddock]

I've created a minimum example of this error - the issue appears to be that TestFixtureSource doesn't work with generic classes. The below reproduces the crash with NUnit 3.2.

    [TestFixtureSource("SupportedEnumerableTypesForDeserialization")]
    public class ArrayTestCase<T>
    {
        private string _aString;

        public ArrayTestCase(string aString)
        {
            _aString = aString;
        }

        [Test]
        public void Test()
        {
            Assert.That(_aString, Is.EqualTo("Something"));
        }

        public static IEnumerable<string> SupportedEnumerableTypesForDeserialization
        {
            get
            {
                yield return "TestCase";
            }
       }
    }

Additionally verified that the console returns zero on this crash - as it's at test-suite level, and only invalid assemblies, or individual tests are counted towards the return code. That seems like a separate issue - I'll create another.

[CharliePoole]

@ChrisMaddock It's usually best if somebody manages to actually review, categorize and prioritize an issue before we work on it. As a committer, you're able to do that yourself. It's a small thing, but it allows us all the opportunity to come back and say whether we agree or disagree with how you are suggesting to handle it. Additionally, if you then decide you want to work on the issue, please assign it to yourself, so that everybody knows.

In this case, it's probably no harm since it seems like your review and fix all came together this morning anyway. We can all comment on the issue and the fix simultaneously. :-)

[CharliePoole]

The return code issue is something I thought we had already dealt with. I'll have to search.

[CharliePoole]

I changed the title again. The problem isn't using [TestFixtureSource] in a generic class, it's when the data referred to lives in the same or probably even a different generic class.

I'd expect this to occur if the data source were an instance member of a class implementing IEnumerable. In that case, NUnit has to create an instance of the object and would not have enough information to do so, since no Type parameter is provided anywhere.

However, your example uses static data, so I find it a bit surprising.

[ChrisMaddock]

@CharliePoole

In this case, it's probably no harm since it seems like your review and fix all came together this morning anyway.

A little confused - I haven't done any work to fix this issue? This is unrelated to #1377, if that was your thinking? Was just aiming to first create a smaller reproducible example here, apologies if I've overstepped somewhere!

The problem isn't using [TestFixtureSource] in a generic class, it's when the data referred to lives in the same or probably even a different generic class.

In @yaakov-h's original project, it's the test fixture class which is generic, not the class where the data lives. I've inadvertently lost that distinction in my example however, by trying to oversimplify.

[CharliePoole]

@ChrisMaddock Au contraire! If anything, you understepped - if that's a word. I was taking it that you didn't feel "authorized" to label things as bugs, assign priorities, etc. and I wanted you to know that you are and should absolutely do those things the first time you start looking at an issue. Issues with no labelling have either not yet been looked at or were too puzzling to categorize.

I initially thought you also had a PR on this, but that was for another issue, so never mind that.

OK, so do we have confirmation of this bug in some form or another? If not, you should either confirm it or label and leave it for somebody else. If you want to work on it, assign it to yourself. As a new guy, you'll probably get a bunch of advice from us once you put your name on something, but you'll just have to live with that. :-)

[CharliePoole]

This is potentially tricky in that there could be some things that can never work along with others that really should be made to work. For example, if NUnit is expected to instantiate some generic class without knowing what Type parameters to use... well, that's just not gonna happen!

[ChrisMaddock]

@CharliePoole - Ok, no problem. Have generally left such things to yourself or @rprouse - as I'm not sure how you assign priorities/milestones, but can cope with marking bugs as bugs. :-)

This is potentially tricky in that there could be some things that can never work along with others that really should be made to work.

I imagined it would be of that ilk once I narrowed it down. Hopefully there should be scope to at least catch the error however!