New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use Assert.Multiple with async code #2348

Closed
james-uffindell-granta opened this Issue Aug 4, 2017 · 0 comments

Comments

Projects
None yet
2 participants
@james-uffindell-granta

james-uffindell-granta commented Aug 4, 2017

Passing an async lambda into Assert.Multiple compiles fine with no warnings but always passes instantaneously, even if the code inside the lambda should fail.

Sample test code:

namespace app
{
    [TestFixture]
    public class Tests
    {
        [Test]
        public void TestMultipleSync()
        {
            Assert.Multiple(() =>
            {
                Thread.Sleep(TimeSpan.FromSeconds(5));
                Assert.Fail();
            });
        }

        [Test]
        public void TestMultipleAsync()
        {
            Assert.Multiple(async () =>
            {
                await Task.Delay(TimeSpan.FromSeconds(5));
                Assert.Fail();
            });
        }
    }
}

I would intuitively expect both these tests to take 5 seconds to run before then failing. However, only TestMultipleSync fails; TestMultipleAsync passes successfully in less than a second. Output from console runner:

PS C:\dev\scratch\test> .\packages\NUnit.ConsoleRunner.3.7.0\tools\nunit3-console.exe .\app\bin\Debug\app.exe
NUnit Console Runner 3.7.0
Copyright (c) 2017 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.15063.0
  CLR Version: 4.0.30319.42000

Test Files
    .\app\bin\Debug\app.exe


Errors, Failures and Warnings

1) Failed : app.Tests.TestMultipleSync
at app.Tests.<>c.<TestMultipleSync>b__0_0() in C:\dev\scratch\test\app\Tests.cs:line 17
at NUnit.Framework.Assert.Multiple(TestDelegate testDelegate)
at app.Tests.TestMultipleSync() in C:\dev\scratch\test\app\Tests.cs:line 14

Run Settings
    DisposeRunners: True
    WorkDirectory: C:\dev\scratch\test
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.5
    ImageRequiresX86: True
    RunAsX86: True
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 2, Passed: 1, Failed: 1, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 1, Errors: 0, Invalid: 0
  Start time: 2017-08-04 10:23:02Z
    End time: 2017-08-04 10:23:08Z
    Duration: 5.504 seconds

I guess the lambda is being interpreted as an "async void" rather than an "async Task" and so its results end up ignored? There doesn't seem to be any obvious other way of doing an async multiple assert though - most of the other Assert methods that take a TestDelegate have an equivalent AsyncTestDelegate method (Catch/CatchAsync, DoesNotThrow/DoesNotThrowAsync), but there's no MultipleAsync method.

There's also nothing on the documentation page for Multiple Asserts (https://github.com/nunit/docs/wiki/Multiple-Asserts) which calls out that you can't use it with async code - on the contrary, it says the assert block "may contain any arbitrary code".

Using NUnit 3.7.1, console runner 3.7.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment