3.6 Multiple assertion backwards compatibility

[AndyBan]

We use nunit along with selenium to check that elements are present on a web page etc. Our framework uses exception capturing loops for certain cases such as interacting with browsers which isn't always the most reliable thing. This allows a bit of time for elements to appear which they are created by Javascript however if things don't appear after a short time then the nunit assertion exception is not captured and bubbles up to the runner and is reported on.

We have found since upgrading to 3.6 that the failed assertions are getting recorded due to the new multiple assertion feature even though the same assertion later passes. I think exceptions should not be recorded unless you are inside a multiple assertion block, would you agree?

A potential fix is to move this earlier in the ReportFailure method:
if (TestExecutionContext.CurrentContext.MultipleAssertLevel == 0)
throw new AssertionException(writer.ToString());

in src/NUnitFramework/framework/Assert.cs

[CharliePoole]

The future direction for NUnit is that all assertion failures will be recorded. For that reason, we will need to solve your problem in some other way than you suggest. Can you provide a bit more info (code or pseudo-code?) about how you are ignoring some failures and re-running the tests?

[stevenebutler]

We are using the same technique in our selenium based framework with the same issues on nunit 3.6.1. We have reverted to using nunit 3.4.1 until we find a good solution for use with nunit 3.6.1.

I will try to outline what our framework does to synchronise on pages here.

We define a web page class that encapsulate actions on the page, as well as means to identify items on screen, including information sections, web page controls etc. This page also has a standard way of waiting for a web page to load. It uses a SeleniumExtensions wait.Until() construct to periodically poll an AssertPageShowing() method in the derived page class.

The AssertPageShowing() method simply does things like:

AssertPageTitle("My page title");  // Convenience method that calls Assert.That() on the title of the browser
Assert.That(ControlIdentifier.Displayed, "Important control that signals the page has finished loading is present and able to be interacted with");
Assert.That(OtherControl.Displayed, "etc");

As the page is loading dynamically, the wait.Until() will continually trigger nunit assertions that we ignore during the timeout period. Once the wait.Until() completes, we call the assertion function a final time to ensure that it did actually load the page successfully, before returning the page to the calling method to allow interactions to proceed.

We want the assertions to be ignored while the wait.Until() hasn't expired, but the assertions should be logged once the timeout has expired (and thus register a test failure only at that time). The new multiple assertions feature in 3.6.1 is failing our tests even when we would have expected them to pass.

Note that we originally did not use assertions in the AssertPageShowing() method, and instead had an IsPageShowing() method that returned true or false. The problem with this approach was we had no idea what caused the test to fail if there are multiple items that needed to be present, or in particular states, for us to consider the page load complete. By using the assertions, we had instant feedback in the test logs as to why the page load failed.

If there is another approach that would work better with nunit 3.6.1 I would consider changing our design, but otherwise, as per the original issue post, I'd really appreciate a way to stop assertions registering while we're waiting to see if the assertions will stop failing.

[JustinRChou]

Would it be possible to use a DelayedConstraint?

[CharliePoole]

The only issue with a DelayedConstraint is that it applies to a single assertion. However, if he can deal with repeating each assertion until it passes, rather than all assertions, I think that would work.

@stevenebutler Could you do something like this?

AssertThat(Page.Title, Is.EqualTo("Page Title").After(...).PollEvery(...));
Assert.That(ControlIdentifier.Displayed).After(...).PollEvery(...));
Assert.That(OtherControl.Displayed).After(...).PollEvery(...));

Alternatively, wrap them all into a single method call and use After on an assertion that calls that method.

[stevenebutler]

That might work, but up until now I didn't even realise the DelayedAssertion existed, so it'll take me some time to experiment.

[JustinRChou]

I was experimenting using the DelayedConstraint by changing a string in a thread. Had to use a StringBuilder.

        [Test]
        public void DelayTest()
        {
            Thread th = new Thread(()=> {
                Thread.Sleep(2000);
                sb.Append("Added");
            });
            th.Start();
            Assert.That(sb, Is.EqualTo(new StringBuilder("StartAdded")).After(5000));
            th.Join();
        }

But I got

Expected: <StartAdded> after 5000 millisecond delay
  But was:  <StartAdded>

The Assert fails which is probably failing on the reference. I thought EqualTo didn't compare the reference? Or maybe it's still a threading issue.

[CharliePoole]

NUnit has no special equality processing for StringBuilder so the StringBuilder.Equals override is used. Equality for a StringBuilder is defined as having the same content, capacity and max capacity. IOW, it's not very useful to compare StringBuilders.

I think this would work, however

Assert.That(() => sb.ToString(), Is.EqualTo(new StringBuilder("StartAdded")).After(5000));

[JustinRChou]

That indeed worked.

[CharliePoole]

@AndyBan Did you want this open or closed? 😄

[AndyBan]

I was a bit early on reopening it, a bit of further investigation and I found things are ok :)

Whilst the above works for us it is a little bit verbose would be my only comment. We also configure our waits globally as we found it wasn't very maintainable to configure them on a per instance basis. I've yet to confirm with my team on the best solution but its looking like we'll handle the waiting for the answer before calling through to your assert as selenium throws exceptions which wouldn't be handled by your interval assert which I think is correct behaviour.

[stevenebutler]

@stevenebutler Could you do something like this?

AssertThat(Page.Title, Is.EqualTo("Page Title").After(...).PollEvery(...)); Assert.That(ControlIdentifier.Displayed).After(...).PollEvery(...)); Assert.That(OtherControl.Displayed).After(...).PollEvery(...));

I don't think this approach would work for me, since we want the .After() to apply to the group of tests as a whole.

Alternatively, wrap them all into a single method call and use After on an assertion that calls that method.

This seems like the way to go, but I'm not sure how to express it in nunit.

Assert.That(() => AssertPageShowing().After(...).PollEvery(...)));

doesn't work, since AssertPageShowing isn't a constraint. Would this approach require providing some form of constraint interface to the method?

I had misread the problem I was having though. It turns out I don't get the polling failures logged in total unless the overall test fails anyway. If the polling eventually succeeds, my log is not polluted with the retry attempts, only if the test failed anyway.

[JustinRChou]

I wonder if you could do your checks in another method that returns a boolean and do something like
Assert.That(TestOnPage(), Is.EqualTo(true).After(timeout));