CollectionException is less helpful in 2.5.x and later than it was in 2.4.x

[Jabbl]

In previous versions, 2.4.x and below, the inner exception was provided to the collection exception when asserting, thus providing a full stack trace to the method call that threw.
In the current version, the Collection exception only contains the message of the inner exception.

If there are multiple similar asserts in the collection assert, the test has to be debugged to identify which exact line failed.

Example:

[Fact]
public void Test_Asserts()
{
    List<Dummy> list = new() {
        new Dummy(true, true, true, true, true, true),
        new Dummy(true, true, true, false, true, true)
    };

    Assert.Collection(list,
        item =>
        {
            Assert.True(item.Value1);
            Assert.True(item.Value2);
            Assert.True(item.Value3);
            Assert.True(item.Value4);
            Assert.True(item.Value5);
            Assert.True(item.Value6);
        },
        item =>
        {
            Assert.True(item.Value1);
            Assert.True(item.Value2);
            Assert.True(item.Value3);
            Assert.True(item.Value4);
            Assert.True(item.Value5);
            Assert.True(item.Value6);
        });
}
record Dummy(bool Value1, bool Value2, bool Value3, bool Value4, bool Value5, bool Value6);

In 2.5.0 and up, this test will cause the following exception, where line 15 is the Assert.Collection line:

  Message: 
   Assert.Collection() Failure: Item comparison failure
                                                                                                                    ↓ (pos 1)
   Collection: [Dummy { Value1 = True, Value2 = True, Value3 = True, Value4 = True, Value5 = True, Value6 = True }, Dummy { Value1 = True, Value2 = True, Value3 = True, Value4 = False, Value5 = True, Value6 = True }]
   Error:      Assert.True() Failure
            Expected: True
            Actual:   False

  Stack Trace: 
    XUnitExamples.Test_Asserts() line 15
    RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

In xUnit 2.4.2, the full stack trace of the exception was included in the message, where line 30 is the inner assert that failed:

  Message: 
    Assert.Collection() Failure
    Collection: [Dummy { Value1 = True, Value2 = True, Value3 = True, Value4 = True, Value5 = True, Value6 = True }, Dummy { Value1 = True, Value2 = True, Value3 = True, Value4 = False, Value5 = True, Value6 = True }]
    Error during comparison of item at index 1
    Inner exception: Assert.True() Failure
            Expected: True
            Actual:   False

  Stack Trace: 
    <>c.<Test_Asserts>b__1_1(Dummy item) line 30
    XUnitExamples.Test_Asserts() line 15
    RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

When the classes being tested are more complex and the data being asserted is less uniform, the 2.4.x exception is much more helpful, as it specifies the exact line of the assert that failed.

[bradwilson]

That is a weird and unexpected behavior, as I can't think of any reason it should happen. I'll have to dig in and see what's going on.

[bradwilson]

Figured it out. The formatting of the error in the new message leaves out the stack trace from the error. The older assertion library didn't format the error, so it's just one big stack trace.

With a simplified test:

[Fact]
public void TestMethod()
{
	var list = new[] { true, false };

	Assert.Collection(
		list,
		x => Assert.True(x),
		x => Assert.True(x)
	);
}

Here's the output from 2.4.2: (includes the lambda on line 15)

Here's the output from 2.5.0: (does not include the lambda)

Here's the output from 2.6.7-pre.8: (includes the lambda) https://xunit.net/docs/using-ci-builds

The inner stack trace is now printed below the error message, which includes the failure line of the lambda. It looks slightly different because of the way stack traces are filtered by the console runner, which doesn't see the inner stack trace as an actual stack trace; it's part of the message.

VSTest (aka dotnet test) looks different as well, but in a different way:

I experimented with a few different ways to display the information, and although the filename format is different across runners, this still seemed like the cleanest way to present the information.