Extra AssertionResult entries in TestResults

[CharliePoole]

There are a few situations where spurious AssertionResult entries are created in the result:

1. Assert.Throws or ThrowsConstraint when the code executed by the delegate argument includes asserts. The second-level asserts may leave failure and warning entries in the result that is being used for the top level test.

2. Executing code that causes an AssertionException and then catches it. This is done in some of our NUnit tests and it's a "trick" used by some users. The failure represented by the exception is added to the result before we can catch it.

3. Any Assert that takes a delegate could theoretically do the same thing if there were Asserts in the code executed. However, this is something of a pathological situation, since there isn't much reason to use Asserts in such delegates.

To resolve this, all such "second level" code should be executed in an isolated context and should apply to a separate result, which is ignored by the top-level code.

[CharliePoole]

To document the problem, here is a list of all TestFixtures with cases that produce spurious AssertionResult entries with a Status of "Failure" even though the test itself passes.

NUnit.Framework.Assertions.ArrayEqualsFailureMessageFixture
NUnit.Framework.Assertions.AssertFailTests
NUnit.Framework.Assertions.AssertEqualsTests
NUnit.Framework.Assertions.AssertPolarityTests
NUnit.Framework.Assertions.AssertThrowsTests
NUnit.Framework.Assertions.AssertZeroTests
NUnit.Framework.Assertions.AssertThrowsAsyncTests
NUnit.Framework.Assertions.AssertThatTests
NUnit.Framework.Assertions.AsyncThrowsTests
NUnit.Framework.Assertions.ConditionAssertTests
NUnit.Framework.Assertions.DirectoryAssertTests
NUnit.Framework.Assertions.CollectionAssertTest
NUnit.Framework.Assertions.GreaterFixture
NUnit.Framework.Assertions.GreaterEqualFixture
NUnit.Framework.Assertions.LessEqualFixture
NUnit.Framework.Assertions.LessFixture
NUnit.Framework.Assertions.FileAssertTests
NUnit.Framework.Assertions.ListContentsTests
NUnit.Framework.Assertions.NotSameFixture
NUnit.Framework.Assertions.NotEqualFixture
NUnit.Framework.Assertions.SameFixture
NUnit.Framework.Assertions.TypeAssertTests
NUnit.Framework.Assertions.StringAssertTests
NUnit.Framework.Attributes.MaxTimeTests
NUnit.Framework.Constraints.AllItemsConstraintTests
NUnit.Framework.Constraints.CollectionContainsConstraintTests
NUnit.Framework.Constraints.CollectionEqualsTests
NUnit.Framework.Constraints.CollectionOrderedConstraintTests
NUnit.Framework.Constraints.EqualConstraintTests
NUnit.Framework.Constraints.EqualConstraintTests+DictionaryEquality
NUnit.Framework.Constraints.EqualConstraintTests+FloatingPointEquality
NUnit.Framework.Constraints.EqualTests
NUnit.Framework.Constraints.ExactCountConstraintTests
NUnit.Framework.Constraints.NotConstraintTests
NUnit.Framework.Constraints.NumericsTests
NUnit.Framework.Constraints.RangeTests
NUnit.Framework.Constraints.AsyncDelayedConstraintTests
NUnit.Framework.Constraints.DelayedConstraintTests
NUnit.Framework.Tests.Constraints.DictionaryContainsKeyConstraintTests
NUnit.Framework.Tests.Constraints.DictionaryContainsValueConstraintTests

There are quite a lot of them in our tests, because we often need to test failure conditions. In general, three different techniques are used:

1. Assert.Throws

var ex = Assert.Throws<AssertionException>(() => Assert.AreEqual(2+2, 5));
Assert.That(ex.Message, Is.EqualTo(expectedMessage));

2. ThrowsConstraint

Assert.That(() => Assert.AreEqual(2+2, 5),
Throws<AssertionException>.With.Message.EqualTo(expectedMessage));

3. Try / Catch

AssertionException caughtException = null;
try
{
    Assert.AreEqual(2+2, 5));
}
catch(AssertionException ex)
{
    caughtException = ex;
}

Assert.That(ex, Is.NotNull.And.Message.EqualTo(expectedMessage));

In each case the Assert.AreEqual expression fails, adding a failed AssertionResult to the test result. Even though the outer logic in each case causes the test to be marked as a success, the failed result remains and is written to the XML file.

To resolve this, both Assert.Throws and ThrowsConstraint need to be modified to run the provided delegate or lambda in an isolated TestExecutionContext, producing a result which is discarded.