Request for Assert.Multiple

[BanksySan]

Is there any support for porting NUnit's Assert.Multiple() functionality to xUnit?

I've found it a very nice syntax to use and a lot quicker than writing custom constraints.

[bradwilson]

I have no idea what this means. Please re-open with a specific description of the requested feature.

[rizanzaky]

In nUnit now you can do something called multiple assertion for one testMethod which is, as I know is a system adopted to recognize all assertion failures when you have several assertions in one method.

Multiple Asserts: nUnit

In other words, normally when you have multiple assertions in a method, if one assertion fails, only that failure is displayed and nothing about the rest. But what this feature enables is, enables assertion failure reporting for all assertions in one method.

Here is an example,

The result shows only the first failure, but any other failures or successes are not displayed,

Hope I put it clearly enough to get this request reopened 😅

I believe the link provided will help you understand the request 😊

Thank you!!!

[marcind]

The XUnit Assert class works differently than the one in NUnit. XUnit will throw an exception on an assertion failure, whereas NUnit reports an error to the test execution context.

[rizanzaky]

Oh it could be so, I don't know. But still it would be really handy to have this feature found in nUnit, in xUnit as well. So that's why this request is made 😇. Hope it's a good requirement to have 🤔

[bradwilson]

You could write it yourself in about 10 lines of code: take a params Action[] for each assert, collect all the thrown exceptions, then throw a new composite failure assertion exception.

[PerthCharern]

I guess I can, but wouldn't it be nice for this to be a part of the framework?

NUnit repo has had a long thread discussing the benefits/implementation of this before: nunit/nunit#391

The syntax they ended up using is also very simple to understand: https://github.com/nunit/docs/wiki/Multiple-Asserts

[bradwilson]

There are plenty of things we could do, but for many reasons we choose not to. This reasons include: (a) implementation and maintenance cost; (b) competing priorities; (c) philosophical objections to features.

In this case, you're running afoul of (c). We do not believe that a unit testing framework should ever run more than one failing assertion.

[bfis108137]

I know this is an old topic but I disagree that you should never have more than one assertion. We are testing our website using selenium and say we open a certain page. We might make 20 assertions for various html elements on that page and we want to see all of the results. Now we obviously could write a separate test for each element but keep in mind that the page needs to be opened at each test. So now we are opening the same page 20 times. Or we could open the page at the beginning of the first test but then we are depending on the order of the tests which is bound to be changed by some programmer who replaces me in 2 years.

[mateusz-dobrowolny]

@bfis108137 Absolutely agree.

Especially with "QA high level tests" like Selenium, it is so high cost (of time) to reach a place in the application, that there is a need to merge assertions. And there is a risk that the more I assert, the higher probability of failure.

I pretty well understand the "backend developer" point of view why the unit tests shold make only a single assertion, but there are other people using the frameworks, and some use cases require mutliple assertions.

[cicorias]

anyone comes across this, an approach that worked for me, although exact types (see the comments on derived types)

https://stackoverflow.com/a/47302498/140618

var exceptions = new List<Type>()
            {
                typeof(NullReferenceException),
                typeof(KeyNotFoundException),
                typeof(InvalidOperationException),
            };

var ex = await Assert.ThrowsAnyAsync<Exception>(() => foo.Bar());
Assert.Contains(ex.GetType(), exceptions);

[dhelper]

Actually there's a good "unit testing" reason we should have multiple assertions in a single test - because we actually testing one thing, think about a message object returned from the code under test, that message has several properties and values, but I'm actually testing one message - one result, from time to time you will find a test that has a single outcome that manifest itself as multiple values - http result + status is one such instance.
and yes I might be able to use the class equals but sometimes I cannot since it's not my class and sometimes I do not want to compare everything, I need a more file grained solution.

• And yes, you can easily write you own Assert.All (and I have), but as a unit testing framework - you should provide guidelines and best practices and I think this is one of them.

I understand you do not want to provide a new capability that would enable developers to write bad unit tests, and I have seen this featured abused in NUnit and JUnit but I think the reason we need it is the original sin of all JUnit based frameworks in .NET and Java - they throw exception on error, this is a mistake that was never corrected and one more pain point for developers starting with unit testing, I think that this alone merits at least thinking bout adding this simple solution.

[bradwilson]

@dhelper wrote:

Actually there's a good "unit testing" reason we should have multiple assertions in a single test

You mis-attribute what I said. I didn't say you shouldn't run one assertion per test. This is what I actually said:

We do not believe that a unit testing framework should ever run more than one failing assertion.

(Emphasis changed to highlight what I think is the key difference between what you're saying and what I'm saying.) It's fine to write multiple assertions to test one logical thing, but I disagree that continuing to run most assertions when one has failed is something that is generally the right behavior in a unit testing framework. Pretty much every argument I've ever drilled down on with people who want multiple failures per test weren't actually writing unit tests; hence, why I believe this is a feature that can and should belong outside of xUnit.net itself.

If you disagree philosophically with what I just wrote, then that's fine. But I've already considered your point, and I'm not swayed.

[Melandel]

We do not believe that a unit testing framework should ever run more than one failing assertion.

I too misunderstood this sentence.

Given that one does not control when an assertion will fail,

running no more than one failing assertion

is easily interpreted as

running only one assertion

because that would be the only way to guarantee that you never have more than one failing assertion.

So, thank you for clarifying!

Can you confirm my new understanding of the sentence ? (right below)

We believe that if a unit test breaks, the first assertion failure should be largely sufficient to pinpoint something went wrong, and which code was responsible for that

[bradwilson]

Yes, that's an accurate representation.

Here's a trivial example of why it usually makes sense to just fail fast:

// ...
Assert.NotNull(someValue);
Assert.Equal(42, someValue.SomeProperty);
// ...

The second assert would fail with a NullReferenceException. In fact, in v3, you can't write the second assert without writing the first assert if you have #nullable enable turned on, and someValue is potentially null. We have decorated the assertions with nullable annotations:

https://github.com/xunit/assert.xunit/blob/352c42472d8aa446bde7457242e2d7eccdf7a058/NullAsserts.cs#L23

[mauriciogracia]

Is kind of sad to see how this request for a feature became the above set of interactions were the position of the person that could to do the change is just "do it yourself I wont do it for philosophical reasons" and those reason end up being just point of views.

It seems like @bradwilson rather have each person implement their own code than providing and out-of-box implementation that shows the way he thinks it should work. (besides "throw new NotImplementedException();" of course)

[MovGP0]

When I get an entity from an method under test, I want to validate all it's properties at once, instead of writing a separate test per property.

Still, this counts as only one assertion, because I validate a single entity.

One way around that in xUnit to use a custom Validator that throws a single Exception with all the validation errors. You could use FluentValidation for that.

Another is to use a dedicated assertion framework. In example using FluentAssertion with an AssertionScope.

[bradwilson]

This is shipping as part of v3, in two forms.

Assert.Multiple (for multiple individual asserts):

xunit/src/xunit.v3.assert.tests/Asserts/MultipleAssertsTests.cs

Lines 38 to 80 in caa2cb4

	[Fact]
	public void MultipleAssert_Success_DoesNotThrow()
	{
	var ex = Record.Exception(() =>
	Assert.Multiple(
	() => Assert.True(true),
	() => Assert.False(false)
	)
	);
	Assert.Null(ex);
	}
	[Fact]
	public void MultipleAssert_SingleFailure_ThrowsFailingAssert()
	{
	var ex = Record.Exception(() =>
	Assert.Multiple(
	() => Assert.True(true),
	() => Assert.False(true)
	)
	);
	Assert.IsType<FalseException>(ex);
	}
	[Fact]
	public void MultipleAssert_MultipleFailures_ThrowsMultipleException()
	{
	var ex = Record.Exception(() =>
	Assert.Multiple(
	() => Assert.True(false),
	() => Assert.False(true)
	)
	);
	var multiEx = Assert.IsType<MultipleException>(ex);
	Assert.Collection(
	multiEx.InnerExceptions,
	innerEx => Assert.IsType<TrueException>(innerEx),
	innerEx => Assert.IsType<FalseException>(innerEx)
	);
	}

Assert.Equivalent (for easier record comparisons; simplest example of using anonymous types to compare to structured types):

xunit/src/xunit.v3.assert.tests/Asserts/EquivalenceAssertsTests.cs

Lines 340 to 367 in caa2cb4

	public class MixedComplexAndAnonymousTypes
	{
	[Fact]
	public void Success()
	{
	var expected = new { Shallow = new { Value1 = 42, Value2 = "Hello, world!" }, Value3 = 21.12m };
	var actual = new DeepClass { Value3 = 21.12m, Shallow = new ShallowClass { Value1 = 42, Value2 = "Hello, world!" } };
	Assert.Equivalent(expected, actual);
	}
	[Fact]
	public void Failure()
	{
	var expected = new { Shallow = new { Value1 = 42, Value2 = "Hello, world" }, Value3 = 21.12m };
	var actual = new DeepClass { Value3 = 21.12m, Shallow = new ShallowClass { Value1 = 42, Value2 = "Hello, world!" } };
	var ex = Record.Exception(() => Assert.Equivalent(expected, actual));
	Assert.IsType<EquivalentException>(ex);
	Assert.Equal(
	"Assert.Equivalent() Failure: Mismatched value on member 'Shallow.Value2'" + Environment.NewLine +
	"Expected: Hello, world" + Environment.NewLine +
	"Actual: Hello, world!",
	ex.Message
	);
	}
	}