TestCaseSource that contains Exception with InnerException - not running tests

[konrad-gora]

After updating NUnit 3 to NUnit 4 (4.1.0) my test cases with InnerException are not run. I'm using TestAdapter 4.5.0

[FixtureLifeCycle(LifeCycle.InstancePerTestCase)]
public class Tests1
{
    [TestCaseSource(nameof(GetSomeExceptionCases))]
    public void NoMethod_GivenException_ShouldPassTest(Exception testCase) => testCase.Should().NotBeOfType<string>();

    public static TestCaseData[] GetSomeExceptionCases() =>
    [
        new TestCaseData(new ArgumentNullException()),
        new TestCaseData(new ArgumentNullException("message", new Exception())),
    ];
}

When I downgrade NUnit to 3.14.0, both tests are passing.

[stevenaw]

@konrad-gora thanks for the report.
We've noticed some EOL issues in test case display names. Does the name of the passing test case remain on one line?

[stevenaw]

This could be related to #4584

[manfred-brands]

@konrad-gora What is the package that implements your Should method? Is it calling into NUnit Assert. In that case you likely need a new version that is NUnit 4 compatible.

Can you replace your test with:

public void NoMethod_GivenException_ShouldPassTest(Exception testCase) => Assert.That(testCase, Is.Not.InstanceOf<string>());

Then switch NUnit versions.

[konrad-gora]

@konrad-gora thanks for the report. We've noticed some EOL issues in test case display names. Does the name of the passing test case remain on one line?

Yes, passing test have one line name.
When I change the name of the test with InnerException (filling in property TestName), the test passes.

[konrad-gora]

@konrad-gora What is the package that implements your Should method? Is it calling into NUnit Assert. In that case you likely need a new version that is NUnit 4 compatible.

Can you replace your test with:

public void NoMethod_GivenException_ShouldPassTest(Exception testCase) => Assert.That(testCase, Is.Not.InstanceOf<string>());

Then switch NUnit versions.

Should() is from FluentAssertion. I changed the test like you suggested and it works in the same way (on both, NUnit 3 and NUnit 4). This assertion is just to have something inside.
When I was trying to debug it on NUnit 4, I put a breakpoint at the start of the test method and I wasn't hit.

[manfred-brands]

Thanks @konrad-gora. We can exclude this then.

To see if it is related to the name, can you set the Name property on the TestCaseData to something simple.

[manfred-brands]

To see if it is related to the name, can you set the Name property on the TestCaseData to something simple.

This seems to work:

new TestCaseData(new ArgumentNullException()).SetName("First"),
new TestCaseData(new ArgumentNullException("message", new Exception())).SetName("Second"),

[manfred-brands]

@stevenaw DisplayName.GetValueString calls (indirectly) Exception.ToString() which result in a stack trace, including a \r\n:

System.ArgumentNullException: message ---> System.Exception: Some Exception Message\r\n --- End of inner exception stack trace ---

Adding a case for Exception and formatting this like: "ArgumentNullException: message, Exception: Some Exception Message" works.

This still depends on the user not passing "\r\n" as part of the exception message.

[konrad-gora]

To see if it is related to the name, can you set the Name property on the TestCaseData to something simple.

This seems to work:

new TestCaseData(new ArgumentNullException()).SetName("First"),
new TestCaseData(new ArgumentNullException("message", new Exception())).SetName("Second"),

Yeah, I tested it with providing names and tests passed.