ExceptionHelper.GetExceptionMessage(Exception ex) should tolerate exceptions from exceptions

[MaceWindu]

We have a test which fails with buggy exception (ex.Message throws exception). When nunit tries to get exception details for test outcome messages, it leads to test runner crash, which results in test run stop (still visual studio test runner shows progress animation, but no test executed anymore).

Stack trace of original exception:

   at System.Collections.ArrayList.get_Item(Int32 index)
   at Sybase.Data.AseClient.AseErrorCollection.get_Message()
   at Sybase.Data.AseClient.AseException.get_Message()
   at NUnit.Framework.Internal.ExceptionHelper.GetExceptionMessage(Exception ex)
   at NUnit.Framework.Internal.ExceptionHelper.BuildMessage(Exception exception, Boolean excludeExceptionNames)
   at NUnit.Framework.Internal.TestResult.ExceptionResult..ctor(Exception ex, FailureSite site)
   at NUnit.Framework.Internal.TestResult.RecordException(Exception ex)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.Dispatch(WorkItem work)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.Dispatch(WorkItem work)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.Dispatch(WorkItem work)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.Dispatch(WorkItem work)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.Dispatch(WorkItem work)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.Dispatch(WorkItem work)
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.RunChildren()
   at NUnit.Framework.Internal.Execution.CompositeWorkItem.PerformWork()
   at NUnit.Framework.Internal.Execution.WorkItem.RunOnCurrentThread()
   at NUnit.Framework.Internal.Execution.WorkItem.Execute()
   at NUnit.Framework.Internal.Execution.SimpleWorkItemDispatcher.RunnerThreadProc()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Note that we cannot fix this exception class, because it is a third party library, which hardly be fixed by it's vendor.

Nunit version: 3.12.0 (used through nuget)

[CharliePoole]

So the Message property of the custom exception is apparently executing code? Wow!

So, for anything but a testing framework, I'd say "Too bad for them." But NUnit is supposed to work with bad code (whether yours or the vendors) and help you fix it, so I agree this should be handled. OTOH, I do not think we should go any further in the exception chain of the Message than just reporting that an exception was thrown. Others?

[MaceWindu]

That solution will be fine. We just don't want test run to be aborted by test

[jnm2]

This is almost exactly the problem that I believe is causing the hang at #3295. In that case, .NET Framework has a bug (https://github.com/dotnet/coreclr/issues/19698) where calling Exception.StackTrace (or indirectly e.g. Exception.ToString()) throws AccessViolationException.

I was planning to work on that today, so I'll roll this into it.