Missing output with failed test stack traces for Assert.Multiple

[gps7]

Hi,
I am missing test output with stack traces for failed tests if Assert.Multiple is used. For running example test fails but output not showing all required information about failures inside of Multiple assert scope but only single stack trace to the test itself:

[Test]
public void AssertMultiple()
{
Assert.Multiple(() =>
{
ValidateInteger();
Assert.That("a", Is.EqualTo("b"), "String");
ValidateInteger();
});
}

private void ValidateInteger() => Assert.That(1, Is.EqualTo(2), "Integer");
Running Nunit console with fallowing command line provides empty TEST_OUTPUT.log file:
"nunit3-console.exe" MyTests.dll --test=MyTests.TestClass.AssertMultiple --out=TEST_OUTPUT.log

I've expect to see not single stack trace but 3 stack traces for 3 failures in single Assert.Multiple request which leads to failed assert. I've already checked the Nunit-console application and it provides all this information correctly.

I am using Framework 4.8 with TestCentric 2.0.0-alpha4 version.

This is a fallow up issue after creating issue #4129 nunit/nunit#4129 at Nunit and issue #1192 nunit/nunit-console#1192 at Nunit-Console. Please refer to the lined items for more details if needed.

[CharliePoole]

I'm confirming this by code inspection. The GUI uses the historic Errors and Failures display, which was part of the old NUnit V2 GUI. At that time, there was no such thing as multiple asserts, warnings, etc. So, like other third-party runners for NUnit tests, it cannot handle the newer features.

This is a big deal both in priority and difficulty to fix. As an illustration, the ResultNode class, which is used to represent test results, doesn't even have fields for multiple messages and stack traces. I've made it a 2.0 issue because I think the fix would be too difficult in the 1.x environment and we are already up to alpha 6 in the 2.0 releases.

I'll need to plan a series of small changes to build up to this, while maintaining compatibility with older versions of NUnit, so hang on while I think about it. :-)

[CharliePoole]

Steps:

• Update Result.Node to include collection of Assertion records, the "new" way of tracking test results.
• Add a test with Assert.Multiple and multiple failures to mock-assembly.dll for use in testing the UI.
• Populate Errors and Failures list control using assertions
• Populate Errors and Failures StackTrace using assertions

[CharliePoole]

@gps7 Based on my experiments, I think you should be able to see the correct results by selecting the failing test in the tree and using the Test Properties tab. This remains an issue, however, because that tab doesn't give access to the source code based display of the stack.

I have given some thought about eliminating the Errors and Failures tab, but I can't figure out a way to integrate everything into the Properties tab. For now, I'd like to get Errors and Failures working properly.

[CharliePoole]

For now, I'm keeping the existing UI. The list of failed tests will become a list of failed assertions so that a test with 3 failures will have 3 entries in the list. I think this is simpler than having the user first select the test, then select the assertion.

[CharliePoole]

Surprisingly, this was easier than I though. So far I have avoided delving into the internals of the stack display control. The fix is available from the myget feed as version 2.0.0-dev00165

[gps7]

@CharliePoole thanks for fixing, Glad to hear that it was easier than originally thought.

[CharliePoole]

🎉 This issue has been resolved in version 2.0.0-alpha7 🎉

The release is available on:

• GitHub Release
• NuGet Package
• Chocolatey Package