Cannot catch AssertionException

[carlin-q-scott]

Executing an NUnit assertion within a try block results in it not being captured by the catch block and it fails the test.

The following test fails due to the Assert.False(true) and does not throw the NotImplementedException:

using System;
using NUnit.Framework;

namespace ApiTests
{
    [TestFixture]
    public class Class1
    {

        [Test]
        public void ATest()
        {
            try
            {
                Assert.False(true);
            }
            catch (Exception e)
            {
                throw new NotImplementedException("whoops", e);
            }
            Assert.True(true);
        }
    }
}

While looking through #2007 I found this workaround:

using System;
using NUnit.Framework;
using NUnit.Framework.Internal;

namespace ApiTests
{
    [TestFixture]
    public class Class1
    {
        [Test]
        public void ATest()
        {
            using (new TestExecutionContext.IsolatedContext())
            {
                try
                {
                    Assert.IsTrue(false);
                }
                catch (Exception e)
                {
                    throw new NotImplementedException("whoops", e);
                }
            }
            Assert.True(true);
        }
    }
}

[CharliePoole]

The most fundamental question about this code is "Why do you want to do this?" As it stands, the code is only an example but the only reason I can see for catching AssertionException is if you are testing NUnit itself.

In fact, in the code given, the exception will be thrown and you should be able to trace it in the debugger. However, catching the exception will not prevent the test from failing as you appear to expect.

I'll keep this open for now in case you have more info to add but it doesn't look like a bug to me so far, just a mismatch between what NUnit does internally and what you expected.

[carlin-q-scott]

My use case is basically the same as the one stated in #2007. I am performing actions and then polling for a state transition in my SUT, so I wrote a RetryUntil function that hides the assertion failures until a timeout is reached or they pass. The fact that NUnit is preventing the capturing of the AssertionException is confusing and a fundamental departure from all unit testing frameworks. Presumably, it's well documented somewhere that I didn't notice.

[JustinRChou]

Could you use the delayed constraint?

[CharliePoole]

NUnit has always considered the use of an AssertionException as an implementation detail. The same goes for IgnoreException, SuccessExeption and InconclusiveException. It's certainly true that it's an implementation detail used in common with many (but not all) other test frameworks. Over the years, when I've seen folks catching an AssertionException, either in an online forum or at one of my client shops, I've told them "Don't do that because there is no guarantee the implementation will remain the same."

As explained in more detail in #2007, NUnit no longer even throws an assertion exception upon every assertion failure. In your case it does and I believe you are incorrect in thinking it is not being thrown. However, distinct from earlier releases, the exception is used only to terminate execution, not to report the failure. The failure is reported before the exception is thrown.

We can certainly talk about ways to improve NUnit to make it easier to do what you are trying to do, but going back to the old implementation that relies on an exception is really not an option. That's because NUnit now needs to be able to report a failure without terminating execution, for example when you use Multiple Asserts or Warning messages.

One way to handle at least some situations is to use the DelayedConstraint. You would have to check to see if it meets your needs.

I suggest you write some pseudocode that you wish you could use and we'll talk about ways to make that happen.

[carlin-q-scott]

The DelayedContraint works for us except we're using FluentAssertions for most of our assertions and sometimes we wrap them in Assert.Multiple. So I think it would be confusing for my team to have two different formats for asserting with polling. One when they're using FluentAssertions and another when using NUnit assertions.

My example code does not throw an exception of type NotImplementedException, which it should if the assertion is throwing an exception. What NUnit seems to be doing now is collecting the assertion failures and throwing them after the method exits, unless I wrap my code in an IsolatedContext as shown in my workaround.

My workaround seems to be working for me but maybe I'm not noticing a side-effect of it? I could also write an extension for FluentAssertions that mimics the DelayConstraint extension methods NUnit provides to make it easier for switching between NUnit constraint assertions and FluentAssertions.

I really don't think there's anything the NUnit team should do to address my concerns except maybe document it, and this issue report itself may be sufficient in that regard.

[CharliePoole]

You might take a look at the XML generated for your test. I suspect you'll find something like

<assertions>
    <assertion>...</assertion>
    <assertion>...</assertion>
</assertions>

where the first assertion element is for the failure and the second is for the assertion thrown. This won't matter to you if you are using a legacy runner that doesn't read that information but if you use a runner that understands the new elements, you will see both the failure and the error reported.

I do see what you are missing from NUnit... we give you the ability to repeat a test until it succeeds using [Retry] as well as the ability to repeat a single assertion using .After. We don't currently give you the ability to repeat a series of assertions until they all pass.

[CharliePoole]

See issue #2043 and the proposed solution. I'm keeping several issues open for now, even though they may be duplicates, because each gives a slightly different use case.

[carlin-q-scott]

@CharliePoole Being able to retry a batch of assertions after performing an action is exactly what I want that's missing from NUnit. FluentAssertions still has some nice features that are missing from the NUnit Constraints but I'm perfectly happy to continue using that library.

[CharliePoole]

@carlin-q-scott What if we simply added a wait period to the existing RetryAttribute? That is if the test failed, we would wait a given amount of time between retries. That would allow re-running the whole test, but not a range of asserts within the test, but may be helpful in some cases.

[Retry(5, Delay=1000)]

For a batch of asserts, we would need to invent a syntax.

[carlin-q-scott]

Most of our use cases involve performing an action, waiting and then performing another action. So retrying the test is not desirable.

[CharliePoole]

In that case you may want to stick with your current approach but substitute Assert.Throws or Assert.Catch for the try/catch block.

[CharliePoole]

@carlin-q-scott That works well with the current DelayedConstraint. The precise use would depend on how you determine success of each action, but for example, if the actions are wrapped as boolean functions...

Assert.That(() => PerformFirstAction(), Is.True.After(3).Seconds.PollEvery(500).Milliseconds;
Assert.That(() => PerformSecondAction(), Is.True.After(1).Second;

What we don't have is the ability to repeat the entire sequence.

[carlin-q-scott]

My issue with using the DelayedConstraint is that I cannot use it with Assert.Multiple so when we need to assert multiple things after performing the action I'd like to maintain the details and clarity provided by multiple specific assertions rather than assert that the series of series of conditions are true.

How would I use Assert.Throws or Assert.Catch to make a decision to continue execution if not thrown. It seems like those assertions would fail my test if the program state ever passed the state assertions for continuing execution.

Maybe presenting the tests for my RetryUntil method would help clarify some things:

using System;
using ApiTests;
using FluentAssertions;
using NUnit.Framework;

namespace UnitTests
{
    [TestFixture]
    [Category("Unit")]
    public class TestHelpersTests
    {
        [Test]
        public void RetryUntil_FailsAfterTimeoutReached()
        {
            Action action = () => TestHelpers.RetryUntil(() => false, Assert.IsTrue, TimeSpan.Zero);

            action.ShouldThrow<TimeoutException>("because it should have failed after the timeout was reached");
        }

        [Test]
        public void RetryUntil_ShouldEventuallySucceedAfterWaiting()
        {
            var retries = 0;
            Action action = () => TestHelpers.RetryUntil(
                () => retries++ == 1,
                Assert.IsTrue,
                1.Seconds());
            
            action.ShouldTakeCloseTo(TestHelpers.RetryInterval).And().ShouldNotThrow("because it should have passed once the request passed the assertion");
            retries.Should().Be(2, "because there should have been 2 retries before success");
        }
    }
}

The second test fails if I use Assert.Catch.

[CharliePoole]

The approach I suggested would work where the sequence is

Action
Wait
Assert
Assert
Assert

or even if there are multiple waits for each individual assert.

I'll look at your code more closely later today.

[CharliePoole]

@carlin-q-scott Although the discussion ended up ranging a bit wider, the original problem of the try / catch not working was the primary fix in the 3.6.1 release. I'm closing this, but if you find that there is still a problem, we can re-open it.