Replacing ThrowsAsync with a composable async alternative

[jnm2]

Today's Assert.ThrowsAsync and related methods do sync-over-async. This means they convert an async operation into a synchronous one. In other words, they block until the task is complete and return void or the return value directly, rather than returning an awaitable type themselves such as Task ThrowsAsync or Task<Exception> CatchAsync.

What does the problem look like?

public class Assert
{                Problem: not async Task
                    ⬇
    public static void ThrowsAsync(AsyncTestDelegate code) { … }

                Problem: no async Task ThatAsync
                    ⬇
    public static void That(ActualValueDelegate<Task> code) { … }
}

      Problem: not async Task
[Test]   ⬇
public void TestMethod()
{
  Problem: no `await`
    ⬇
    Assert.ThrowsAsync(async () =>
    {
        await Task.Delay(10_000); // Fiddle while Rome burns, why don’t ya 😇
    });

  Problem: no `await`
    ⬇
    Assert.That(async () => await …, Throws.InstanceOf<FooException>());
}

Why is this a problem?

Waiting synchronously for an async operation is a antipattern with real-world consequences. The consequences range from losing scalability (if you're doing async right, there is no thread) to outright deadlocks and other logic errors.

Well, in 3.11, I fixed the deadlock issue if you instantiate Windows Forms or WPF controls in your tests. So what then, why not keep doing sync-over-async in Assert?

• First, deadlocks are still a threat. I only fixed deadlocks on recognized message-pumping SynchronizationContexts, and even that hasn't been stress-tested for reentry with assert delegates nested inside assert delegates. We shouldn't have to go to the length of testing sync-over-async-over-sync-over-async; our APIs shouldn't be allowing it like this.

• Second, without a good reason, it goes in the face of hard-learned good practices with async/await. The async-await paradigm is complex enough for those who are still becoming familiar with it that we should not be requiring them to code their tests around this antipattern. Assert.*Async is doing nothing which justifies an exception. It's one thing for NUnit as a framework to do sync-over-async with the entry point, the test method itself—this is exactly like C# 7.1's async Task Main—but it's quite another thing for NUnit as a library, in Assert.*Async, to be doing sync-over-async as well.

What does the solution look like?

I've had proposals for this nagging me for a year and a half now, and none of them seem like slam dunks.

If we just switch Assert.ThrowsAsync from void to async Task we silently break some people's code. And if we switch Assert.CatchAsync from Exception to Task<Exception>, we cause any code using it to stop compiling. The second instance might be considered good pain, but the first option (silent breakage) is absolutely unacceptable.

➡ This means likely the best course is deprecation warnings for all Assert.*Async methods.
(Assuming we don't ship a special analyzer in the NUnit package to turn the silent errors into compiler errors and provide a "fix all.")

If we do deprecation warnings, these are our options as I see them:

• Pick new names for the async methods. (I have no ideas for names, though.)
• Introduce AssertAsync.Throws and AssertAsync.That where all the methods return Task or Task<>.

@rprouse I would like to fix this in 3.11 as part of the theme, if that seems good to you.

[rprouse]

I would like to get more @nunit/framework-and-engine-team members to weigh in here too. I have always found it odd that we do sync-over-async and don't require people to await async asserts. The original code was written before I joined the team and I think much of it was inherited from 2.6. I always thought that our async asserts would be much simpler if they were async all the way down. I'm not sure, but I think that the original decision might have been to support async void test methods which we no longer support.

Backing out of this mistake is a thorny issue though. We don't want to break existing code. Even deprecation warnings are a pain. Modern C# code has async everywhere and most of the test I write are async. Introducing that many warnings into peoples' unit tests won't be popular.

That said, I like the AssertAsync or maybe AsyncAssert idea. I don't really like the idea of finding new names for our current async asserts.

I would appreciate if other contributors and the NUnit community as a whole could weigh in on this. I think it is a great idea in principle, but I would like to do it in a way that will cause the least pain for people.

[CharliePoole]

@rprouse Throws.Async came in with 3.2. We all had our doubts back when it was added in PR #1142. In fact you made comments about being uncomfortable with sync over async back then. The contributor, @thomaslevesque , initially made it return Task<Exception> so it had to be awaited. It may be useful to review all the discussion that led to its being changed to be synchronous in the final commit.

I think our attitude toward asserts has been formed by our attitude toward async tests. In general, we have to run tests synchronously, even if they are async, in order to be able to capture the result. We applied the same principle to asserts, I think with good reason. Our contract has always been that execution terminates when an assertion fails. Therefore, we had to wait for the assertion to complete in order to know if we should continue or not. Of course, we now have multiple asserts and warnings, so execution doesn't always terminate on failure.

I think we probably should think about true async assertions, but I don't think it can be done cleanly in the current architecture. We would have to put as much thought and design work into it as we did into parallel execution of tests.

[jnm2]

AsyncAssert is a really good suggestion.

I have an idea for what it would look like. If I put up a spike, it might be something to talk about.

We would still run the entry points sync-over-async. Test methods, setups, etc. We'd just provide a new true-async assertion API. The framework architecture stays the same but we gain a new library API. What I'm thinking of might in theory even be able to be done as an outside library extending NUnit.

[jnm2]

We could also consider not providing a deprecation warning and shipping an analyzer with a suggestion, or even leave it to XML docs and release notes.

@JohanLarsson Have you had experience writing a fix-all to update API usage, changing Assert.Throws to await AsyncAssert.Throws and changing the containing method to async Task? I know it can be done in theory.

[jnm2]

Just to be clear: I don't think it would be a bad idea to make the framework's worker infrastructure true-async, but I'm not sure what we'd gain and I don't want to jeopardize our standing in Johan's benchmarks. 😊

[JohanLarsson]

@jnm2 Yes, fix all is a bit of a pain using using the built in stuff in Roslyn but there are ways to make it work.

I'm mixed about async asserts as they add some noise to tests.

[rprouse]

@JohanLarsson just curious, are your images for Rider benchmarks swapped for NUnit and xUnit? It seems odd that they would be opposite on each IDE.

[JohanLarsson]

@rprouse I did not run the Rider benchmarks, was contributed in a PR, so not sure.
The repo is just a quick and dirty hack to get some data to confirm a feeling I had.

[jnm2]

Ran into a deadlock today in a test with a custom synchronization context because Assert.Multiple(AsyncTestDelegate) blocks and returns void rather than returning Task and running asynchronously. It needs to be handled the same as the above:
await AsyncAssert.Multiple(async () => { … });

[jnm2]

I started exploring this in https://github.com/nunit/nunit/compare/jnm2/async_assert. Many of the things I want to do will be easier once I finish my work on non-Task awaitables.

An early question: what is our policy on adding methods to public interfaces, such as Task<ConstraintResult> IContraint.ApplyToAsync(object)? If we add that method, we cause a source-breaking change for anyone who implemented IConstraint without deriving from the base Constraint class. It's also a binary-breaking change if a library is compiled for NUnit 3.10 and is used in a project which updates to 3.11.

If we instead introduce IAsyncConstraint : IConstraint which declares that new method, do we also introduce IResolveAsyncConstraint with IAsyncConstraint Resolve(), and deal with breaking changes for people using the public OperatorStack class by replacing uses of IConstraint with IAsyncConstraint?
Or do we use a runtime cast like this?

[mikebeaton]

If we just switch Assert.ThrowsAsync from void to async Task we silently break some people's code.

ThrowsAsync returns Exception (not void), just like CatchAsync

Sorry, I am almost certainly missing something ridiculous to be making this comment. But if I'm not, doesn't that mean we get breaking, 'good pain' by changing the signature, without needing AssertAsync?

EDIT: I think I've got it - sorry! Even though ThrowsAsync and CatchAsync have the same signature (I'm sure!), the problem is that anybody using the result of either would notice if there was a change from returning Exception to returning Task<Exception>... but that anybody just making the assertion that a throw happens without using the Exception result wouldn't notice that their code is still compiling, but now silently always succeeding (since it's now being returned an un-executed Task and doing nothing with it). But this would still generate a warning, right? (So, potentially, 'good pain'.)

My reason for asking - as a member of the NUnit community - also willing to get involved and create PRs if it will help: It's noticeably more fiddly to write async tests due to ThrowsAsync not being async so, all other things being equal (or even nearly equal), it would be very nice if it could just be changed to being async. Especially if that throws up something - at least warnings - and to fix them all the user needs to do is globally replace Assert.ThrowsAsync with await Assert.ThrowsAsync - that's do-able for a library user, right? And then also to fix up any tests of which the only async part was ThrowsAsync, and which have therefore just become async when they weren't before, to be async Task instead of void, as they arguably 'should' have been all along. But any of those won't even compile until you do that. So this is a one-off change to any project, for the end user - a major version number bump change to Nunit, definitely, but it looks like a do-able, one-off, global search-and-replace type change with visible warnings, for the end user?

[jnm2]

That sounds good for a v4. @rprouse, what do you think about doing something sooner than v4 like AsyncAssert.Throws and removing or obsoleting it in v4? Or can we wait and rework in v4? We'll definitely need to ship NUnit Analyzers by default with an analyzer that errors if Assert.ThrowsAsync changes meaning.

[jnm2]

Because of Assert.That(async () => await SomethingAsync(), Throws.TypeOf<FooException>()) needing to be awaitable, v4 is certainly the easiest way to go because we can change public interfaces.

Even Assert.That(async () => { await Task.Delay(1000); return 4; }, Is.EqualTo(4)) would need to be an awaitable call.

I think we should do Assert.ThatAsync as the most obvious awaitable version, and throw ArgumentException if you use Assert.That when anything async is going on with a message to use Assert.ThatAsync. How does that sound?

[rprouse]

Is this causing pain to enough people to make big changes now? I expect that if we change the behaviour of Assert.That to throw an ArgumentException now, it would be a lot of work for people with a lot of async in their tests to go through and fix them all. It isn't a straight search and replace, and it is a runtime failure, not a compile-time failure.

I think we're in a bit of a corner here and I don't know what the right answer is. We can't change Assert.That to be awaitable because it would break too many people. I also don't think that most people would appreciate why we are doing it because our current async tests work fine 99% of the time.

So, I'm okay with introducing true async asserts with new Assert.ThatAsync calls, but I'd be uncomfortable throwing exceptions if people try to use our current Assert.That. I even think that might be a big enough breaking change that I'd be reluctant to do it in a major upgrade. Most people don't read the breaking changes on major upgrades or aren't able to do the work needed.

I would like to see analyzers gentle pushing people to better alternatives.

[Dreamescaper]

It's quite easy to create analyzer with warning (or error) when Assert.That is used with awaitable type.

Simple code fix to use await Assert.ThatAsync and make test method async if needed should be easy to implement as well. Might be harder with cases when assert is present in separate non-test method, not sure.

[Daniel-Svensson]

There are similiar problems with Assert.Multiple which we would like to use but are quite hesitant to use is currently is.

For Multiple maybe we can have a simple solution with `Assert.MultipleAsync' which returns Task, but for Throw that is not a good option.

[Daniel-Svensson]

I just saw a PR with a proposed fix #3577

[Temtaime]

So... We should have async methods anyway.
What solution is prefered ?
Should we create Assert.Throws overloads with Func argument?
I'd go with a breaking change proposed in #3577

[Daniel-Svensson]

I think Assert.ThatAsync as suggested by @rprouse seems lika a reasonable appoach.

In order to support ThrowsAsync and Multiple or similar what do you think about
a AsyncAssert or AssertAsync class which would allow adding Task async alternatives to Throws, Multiple (and maybe event That) without introducing breaking changes?

[trampster]

I just ran into this issue Assert.ThrowsAsync should return a Task so it is awaitable.

[uecasm]

I'm sad that Assert.ThatAsync and Assert.MultipleAsync (and something for Throws) still don't exist yet in $CURRENT_YEAR.

I'm currently hitting a deadlock due to #2917 not posting to a custom synchronization context, which could be avoided if it did just let the await go all the way down.

[OsirisTerje]

@jnm2 @Daniel-Svensson @uecasm @trampster @Dreamescaper @mikebeaton @JohanLarsson Can you guys confirm that https://www.myget.org/feed/nunit/package/nuget/NUnit/4.0.0-dev-07733 is working for you?

[mikebeaton]

@jnm2 @Daniel-Svensson @uecasm @trampster @Dreamescaper @mikebeaton @JohanLarsson Can you guys confirm that https://www.myget.org/feed/nunit/package/nuget/NUnit/4.0.0-dev-07733 is working for you?

Hi - thanks! I am using NUnit3TestAdapter with Visual Studio - I cannot immediately see an NUnit4TestAdapter to use for this test - should there be?

[OsirisTerje]

@mikebeaton The NUnit3TestAdapter works with NUnit version 4. We have a bit of name and version mangling here. If we could have changed the name it would be without the number 3 there, it doesn't mean anything anymore. And, if you guys find something that says it doesn't anymore, we need to fix that.

[mikebeaton]

Okay, with NUnit3TestAdapter 4.5.0-alpha.4 from NuGet.org my existing tests are running as is with NUnit 4.0.0-dev-07798.

Btw the version of NUnit3TestAdapter which I can find in the same feed (https://www.myget.org/F/nunit/api/v3/index.json) which contains NUnit 4.0.0-dev-07798 is NUnit3TestAdapter 4.4.0-dev-02338, which does not work with NUnit 4.0.0-dev-07798.

The description of NUnit3TestAdapter 4.5.0-alpha.4 still mentions NUnit 3:

On to the actual issue(!): I gather that even though there has been a major version change 3 to 4, the decision was eventually taken not been to make a breaking change to ThrowsAsync, but rather to provide an alternative? a) I do think that's a shame, but hey; and b) Is there a quick link to the documentation of the new alternative? (i.e. specifically what to code to make a truly async equivalent of ThrowsAsync)?

EDIT: Ré point a), see #2843 (comment) above. :-)

[mikebeaton]

I am hoping to replace:

            var ex = Assert.ThrowsAsync<InvalidOperationException>(async () => {
                await SomeAsyncMethod(arg1, ..., argn);
            });

            Assert.That(ex.Message, Is.EqualTo(ExpectedMessage));

I got as far as:

            await Assert.ThatAsync(async () =>
            {
                await SomeAsyncMethod(arg1, ..., argn);
            }, Throws.InstanceOf(typeof(InvalidOperationException)));

That works, but doesn't include an equivalent of the last part of my test (i.e. I'm not sure how to access the actual thrown exception, if I want to do more tests on it).

It also seems that it might be much nicer to have a typed variant of InstanceOf(...), so Throws.InstanceOf<InvalidOperationException>(), but I may be missing an existing way to do something like this.

[uecasm]

I wouldn't call it a "decision" as such; I'm just a random mook and this was my first nunit PR 😁

There aren't currently any docs as such other than the test cases in the PR itself, but it's pretty self-explanatory.

await Assert.MultipleAsync(async () =>
{
    await Assert.ThatAsync(async () => await BreakSomething(), Throws.InvalidOperationException);
    await Assert.ThatAsync(() => MethodReturningTaskInt("test"), Is.EqualTo(42));
    Assert.That(SynchronousMethod(), Is.EqualTo("also works"));
});

You can check exception messages with Throws.InvalidOperationException.With.Message.Contains("xxx") etc.

Also, Throws.InstanceOf<InvalidOperationException>() already exists; the property above is just a short-hand for it.

[mikebeaton]

Ah, I'm glad there is Throws.InstanceOf<T>(). 👍

So I'm now at:

            await Assert.ThatAsync(async () => {
                await SomeAsyncMethod(arg1, ..., argn);
                },
                Throws.InstanceOf<InvalidOperationException>()
                    .With
                    .Message
                    .EqualTo(ExpectedMessage)
                );

Which is indeed equivalent to my original test.

Might the existence of Throws.InvalidOperationException (not used in my code just above, but you mentioned it) be a bit misleading? It makes it appear at first as if there might not be a way to check for Throws.ArbitraryException, since InvalidOperationException in that invocation isn't the exception type at all, but an NUnit-defined value with the same name. I wonder if it's a good idea to have that?

Also, how do I check an arbitrary property of the thrown exception (not just Message) and, more generally, do an arbitrary further test on the exception object?

Thanks.

[uecasm]

https://docs.nunit.org/articles/nunit/writing-tests/constraints/ThrowsConstraint.html

It supports most of the constraint expression syntax, so you can check other arbitrary properties by using Property<string>(nameof(Class.Prop)).EqualTo("y") etc.

If you find yourself writing the same complex constraints repeatedly, you can make your own custom constraint to shorten the syntax.

You're not really supposed to need to directly obtain the result value or exception with the constraint-based asserts; only test them via the constraints. But if you really want to recover the exception object directly, you can just put the method call into a try-catch and not use ThatAsync at all if you prefer.

[mikebeaton]

@uecasm - okay, thanks for that.

This clearly goes outside your contribution, assuming those are all pre-existing bits of NUnit that I just haven't used, but is there no .Where or .All (EDIT: or maybe I really mean .Select or .SelectThat) or similar which can be chained to apply an arbitrary lambda expression to (OR: to select a part of?) the exception which was thrown?

[mikebeaton]

@OsirisTerje - The summary is, you can definitely change things around to get truly async, but otherwise like-for-like tests, for the (admittedly fairly simple) things I was currently doing with ThrowsAsync, thanks to this PR.

If I can offer my opinion, it is that ThatAsync is a great improvement and equivalent of That; but also that it is a mistake to leave ThrowsAsync as is. Personally I would rather go through the pain of seeing a bunch of warnings for all my existing ThrowsAsync uses (cf my earlier comments above #2843 (comment)) because those warnings would be taking me where I need to go, to fix my async exception tests to be how I always wanted them to be in the first place. And I believe it's a mistake not to take this opportunity to sort out the confusion caused by having a non-awaitable method which deals with async stuff and has a name ending ...Async. It should be converted to its direct, truly async equivalent (a breaking change; but one which will, at least, generate (wanted) warnings), alongside this undoubted improvement of ThatAsync.

[uecasm]

This clearly goes outside your contribution, assuming those are all pre-existing bits of NUnit that I just haven't used, but is there no .Where or .All (EDIT: or maybe I really mean .Select or .SelectThat) or similar which can be chained to apply an arbitrary lambda expression to (OR: to select a part of?) the exception which was thrown?

You can use Matches(Predicate<T>) to compare against an arbitrary lambda. Though making a custom constraint is usually better if it's not a one-off since you can produce more useful failure messages.

Using autocomplete in your IDE of choice is usually the best way to discover the available constraint expressions. Not all of them appear to be documented on the website, but they do have autocomplete docs.

[mikebeaton]

Using autocomplete in your IDE of choice is usually the best way to discover the available constraint expressions. Not all of them appear to be documented on the website, but they do have autocomplete docs.

Yes, I do know that. It was autocomplete that made me think there wasn't an InstanceOf<T> - user error, I guess. And knowing that Matches is the right thing to use is slightly more than just finding it in autocomplete, but thanks for the info.

[CharliePoole]

The constraints themselves are pretty well documented but the static methods used to instantiate them are unfortunately not.

Remember you can always use Assert.That(actual, new SomeKindOfConstraint(...));

[mikebeaton]

Thank you. Not at computer, but I'd got as far as wondering whether there's something like:

            await Assert.ThatAsync(async () => {
                await SomeAsyncMethod(arg1, ..., argn);
                },
                Throws.InstanceOf<InvalidOperationException>()
                    .With
                    .SelectThat(e => e.Message)
                    .EqualTo(ExpectedMessage)
                );

Something like that should be possible with the strongly typed InstanceOf<T>, I guess?

[uecasm]

I already told you that one. The existing syntax is even shorter:

await Assert.ThatAsync(async () => await SomeAsyncMethod(arg1, ..., argn),
    Throws.InstanceOf<WhateverCustomException>()
        .With.Message.EqualTo(ExpectedMessage));

Or for InvalidOperationException specifically, you can just use Throws.InvalidOperationException.With.Message.EqualTo(ExpectedMessage).

[uecasm]

The constraints themselves are pretty well documented but the static methods used to instantiate them are unfortunately not.

There aren't any website docs for the PredicateConstraint...

[mikebeaton]

I already told you that one. The existing syntax is even shorter:

await Assert.ThatAsync(async () => await SomeAsyncMethod(arg1, ..., argn),
    Throws.InstanceOf<WhateverCustomException>()
        .With.Message.EqualTo(ExpectedMessage));

Or for InvalidOperationException specifically, you can just use Throws.InvalidOperationException.With.Message.EqualTo(ExpectedMessage).

No you misunderstand my question. You've told me the special case method for accessing Message, and the less special case method for accessing a given Property. Thank you. I'm now asking if there is an even more general way to map the With object, in this case the exception, using an arbitrary function, to a value (which would then be further automatically wrapped ready to apply the That tests to it).

[CharliePoole]

There aren't any website docs for the PredicateConstraint...

Probably lost in one as docs were ported (2 or three times) to new platforms. Should be a documentation issue.

[uecasm]

No you misunderstand my question. You've told me the special case method for accessing Message, and the less special case method for accessing a given Property. Thank you. I'm now asking if there is an even more general way to map the With object, in this case the exception, using an arbitrary function, to a value (which would then be further automatically wrapped ready to apply the That tests to it).

Other than Property itself (and various collection-related accessors), I don't think so. Typically you'd use Matches or a custom constraint for that sort of thing, or capture the value external to the assertion so that you can perform multiple assertions on the same value (without just using And combinators).