ContainEquivalentOf on a DBNull.Value behave differently between Debug and Release

[mikef907]

Description

Different results are being reported when using ContainEquivalentOf against an instance of SqlParameter with a DBNull.Value. When I run this in Debug build the test passes, when I switch to Release build it fails.

Complete minimal example reproducing the issue

Please consider the following

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Data.SqlClient;
using FluentAssertions;

namespace UnitTestProject1
{
    public class Foo
    {
        public string Bar => "FOOBAR";
        public List<SqlParameter> Parameters => new List<SqlParameter> {
            new SqlParameter("@inBar", DBNull.Value),
            new SqlParameter("@inSettings", Bar),
        };

    }

    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            var sut = new Foo();

            sut.Parameters.Should().ContainEquivalentOf(new SqlParameter("@inBar", DBNull.Value));

            if (!string.IsNullOrEmpty(sut.Bar))
                sut.Parameters.Should().ContainEquivalentOf(new SqlParameter("@inSettings", sut.Bar));
        }
    }
}

Expected behavior:

Debug and Release build should either both pass or fail, I'm not sure if the intention is to fail this or if it should pass, it would be nice if it did pass since it is consistent with the way parameter verification is done elsewhere.

Actual behavior:

Debug build passes while Release build fails

Versions

• FluentAssertions 6.3.0
• .NET Framework 8.0 and 4.7.2

Additional Information

Please note that if I switch the order of the assertions and run the if statement before the DBNull assertion this will pass in both builds.

[dennisdoomen]

I can reproduce it, but frankly, I'm flabbergasted...

[dennisdoomen]

So for me, this fails in release builds:

var sut = new Foo();

sut.Parameters.Should().ContainEquivalentOf(new SqlParameter("@inSettings", "FOOBAR"));

If you try to run it through the debugger, it succeeds.
If you enable FA tracing using WithTracing, it succeeds.

But this also succeeds.

var sut = new Foo();

sut.Parameters.Should().ContainEquivalentOf(new SqlParameter("@inSettings", "FOOBAR"));

Console.WriteLine("blah");

I've also tried to add this particular test to the Fluent Assertions codebase, but then it always succeeds.

So something fishy happens in release builds that affects the SqlParameter. The Fluent Assertions binaries are always build in release mode, so that should not affect it. I have a suspicion that it has something to do with the fact that SqlParameter is a MarshalByRefObject and behaves like a proxy.

The workaround is to assert only the relevant properties have a certain value, like this.

sut.Parameters.Should().ContainEquivalentOf(new
{
    ParameterName = "@inSettings",
    SqlValue = new SqlString("FOOBAR")
});

Either way, I have no clue where this is coming from and how to fix it. I have never seen this in the 20 years I've been coding in .NET.

[jnyrup]

[TestMethod]
public void Fails()
{
    var parameters = new[] { null, "B" };

    parameters.Should().ContainEquivalentOf("B");
}

	debug	release
net48	OK	FAIL
net5	OK	OK

If I change CallerIdentifier.DetermineCallerIdentity to simply returning null both TFMs work in passes in release mode.

[dennisdoomen]

Hmm. Did you try adding some lines? Or attaching the debugger in release mode?

[jnyrup]

StringEqualityEquivalencyStep.ValidateAgainstNulls invokes FailWith with "Expected }}} to be {0}{reason}, but found {1}."
That later throws a FormatException due to }}}.

}}} comes CallerIdentifier.GetSourceCodeStatementFrom that tries to extract it from the source file.

int expectedLineNumber = frame.GetFileLineNumber(); returns 16 as the expected line number.
In the file below that is the line below ContainEquivalentOf.

using FluentAssertions;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace MSTestV2.Specs
{
    [TestClass]
    public class FrameworkSpecs
    {
        [TestMethod]
        public void Fails()
        {
            var parameters = new[] { null, "B" };

            parameters.Should().ContainEquivalentOf("B");
        }
    }
}

So multiple things are going on.

• Line numbers in release mode are unreliable.
• CallerStatementBuilder does an approximative parsing, in other words there are no guarantees on the output.
• FailWith uses the output from above as input to string.Format which is insecure to do.

Perhaps we shouldn't use DetermineCallerIdentity at all in release mode?

After a five minutes googling here's a method that is labelled with "is not perfectly reliable".

private static bool IsDebug()
{
    return Assembly
        .GetEntryAssembly()
        .GetCustomAttributes(typeof(DebuggableAttribute), false)
        .Cast<DebuggableAttribute>()
        .Any(e => e.IsJITOptimizerDisabled);
}