Missing stack trace when exception occurs during OneTimeSetUp

[ericnewton76]

Missing the stack trace for the error if it bubbles up from a OneTimeSetUp method.

[ericnewton76]

I have forked vs-adapter and I'm looking through the code, and this type of scenario isn't tested.

https://github.com/ericnewton76/nunit3-vs-adapter/commit/09901cd4d26afbb4aa6dc02f28ed79c5abb9556c
I added a class to mock-assembly project (at the end of MockAssembly.cs) but i'm not sure how to run the tests against this new mock assembly.

Additionally I wonder if nunit itself has the stack trace for the exception during OneTimeSetUp and if the vs-adapter is displaying all that it has.

[ericnewton76]

Ok, attempted to run build script, because I see it includes test running, but MSBuild giving me some kind of VS2017 sxs installation problem, so dont install VS2017 yet. The VS developers are lamenting the fact that VS2015 cant build VS2017 at the moment. I'm guessing this is related to my problem. with getting msbuild to build successfully. github/VisualStudio#655 (comment)

[OsirisTerje]

I can repro this, and I see we don't add the OneTimeSetup crash information to the output. The information comes from the TestFixture node, and the site info is "Child".
@CharliePoole @rprouse : I see the Setup and Teardown have its own site names, but OneTimeSetup and OneTimeTearDown are not listed there. And I can't see any of these site names in the output result file. I only see Child and Parent.

[CharliePoole]

One time setup is the parent setup. It's difficult to know what to do here since VS has no place to report suite results but only test cases. We could report the suite result on each test case but that's potentially a lot of duplication. I compromised by reporting only the message but the stack trace could be reported as well.

[ericnewton76]

How does nunit handle an exception in onetimesetup? Does it basically mark the whole test fixture as invalid, or does it keep retrying onetimesetup for each test in the fixture (Which to me would makes most sense)

[rprouse]

It marks the whole fixture as failed, it doesn't retry. We assume that your setup code is deterministic and will fail consistently.

[OsirisTerje]

Is the output I see in the console output (testresult.xml) identical to what the adapter gets as a response?

[CharliePoole]

Yes, the adapter gets the entire XML result in pieces as each test completes. (It could also get the completed result but it doesn't need it since the info is given to VS as it arrives.) The result of a test suite is pretty much ignored, since it's not associated with any of the tests that have been discovered by VS. You can see that happen here: https://github.com/nunit/nunit3-vs-adapter/blob/master/src/NUnitTestAdapter/NUnitEventListener.cs#L148

The result of a test case is handled in the method right above that one. It records what is contained in the test case result.

[OsirisTerje]

Thanks! And what is the real meaning of the term/attribute "site" ? It can take values like Child and Parent, but also SetUp and TearDown and the like ?

[CharliePoole]

In the framework, it's an enum. We use the enum names as attribute values. The default is "Test" which is what usually happens. The code in the adapter only logs a message for setup and teardown suite failures, since those don't get reflected anywhere else.

Note that the one time setup error message was being shown in the original issue description. That indicates it must be coming in the test case failure. You want to examine whether that's happening with the stack trace too. I suspect not.

[CharliePoole]

BTW these attributes should all be documented here: https://github.com/nunit/docs/wiki/Test-Result-XML-Format

[carlin-q-scott]

So if I want a OneTimeSetup that gives me sufficient detail to figure out the cause of the failure, should I do something like this?

static bool oneTimeSetupRan;

[SetUp]
public void OneTimeSetup()
{
    if (!oneTimeSetupRan)
    {
         throw new NotImplementedException();
         oneTimeSetupRan = true;
    }
}

[CharliePoole]

@carlin-q-scott That would definitely simulate the one-time setup for a fixture. It's a pity to have to go to that trouble though.

[CharliePoole]

@nunit/vs-extensions-team What do we want to do here? Is it simply a limitation of the VS interface? Should we try to report the stack trace for each error? That would require our remembering the error info for each one time setup failure and reporting it for the corresponding test cases. Non-trivial, especially in the presence of multiple fixtures running in parallel.

[blorger]

This is not just limitation of VS interface. NUnit simply does not provide call stack information. Problem remains if you execute tests with NUnit console, no call stack on console output or in result file.
Why not simply change content of a message. Provide entire exception description rather than just exception message. This way there will be no need to debug a test or modify it (move one time setup code inside test, or wrap each setup method in try/catch block to log exception ...) just to get information where failure occurs.

[OsirisTerje]

If I have an exception in the OneTimeSetup, for each test it is reported, like this:
OneTimeSetUp: ClassLibrary11.WhateverException : Exception of type 'ClassLibrary11.WhateverException' was thrown.

The stack trace is coming with the suite finished event a bit later, but then the test results have already been sent.

I think it would be better just to return the full exception stacktrace with each failing tests. It would only slow down failing tests, so that should be no big matter. And then no changed code is needed for the adapter.

@rprouse / @CharliePoole If you agree, can we add an issue on NUnit for this, or move this to NUnit ?

[CharliePoole]

If we want to make it look a certain way for VS, then I think we should make it look that way in the adapter. NUnit itself attaches the error to the test that failed, which was the fixture. That's a pretty fundamental thing about how NUnit looks at tests: everything is a test - test cases, methods, classes, namespaces and the assembly. VS looks at the tests as a list of cases. Since the VS view is simpler, it should be possible to make things appear that way if it's really desired.

[OsirisTerje]

Yes, we can do this in the adapter, but NUnit is reporting this per test case, just with less information. It has split the information of the exception between the test case and the fixture, with the test case reporting the exception and the message, and the fixture reporting the same plus the stacktrace.

[CharliePoole]

Add the issue to NUnit in that case, so we can discuss it there. I think it should be a new issue, since up to now this has been about how the adapter displays what it gets from NUnit, rather than changing what NUnit delivers.

We have a similar problem in the console runner, because the console runner historically only counted test cases and only displayed test case failures and errors. Over time, we started displaying more info. But the console has access to the entire XML, so modifying the output was easier.

[OsirisTerje]

Issue moved to nunit/nunit #2466 via ZenHub