`EqualConstraint` fails with missing assembly references on .NET Framework since NUnit 4.2

[emwl]

Our Renovate bot pushed the update to NUnit 4.2 earlier this morning, and caused a bunch of tests to fail that worked fine with 4.1.

They either fail with

System.IO.FileNotFoundException : Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
   at NUnit.Framework.Constraints.Comparers.StreamsComparer.Equal(Object x, Object y, Tolerance& tolerance, ComparisonState state, NUnitEqualityComparer equalityComparer)
   at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance, ComparisonState state)
   at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance)
   at NUnit.Framework.Constraints.EqualConstraint.ApplyTo[TActual](TActual actual)
   at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, NUnitString message, String actualExpression, String constraintExpression)
   at Project.Tests.Namespace.SomeTests.TestMethod(String data, Boolean expected) in E:\BUILD01A\_work\121\s\Project\Project.Tests\SomeTests.cs:line 129

or

System.IO.FileNotFoundException
  HResult=0x80070002
  Message=Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
  Source=nunit.framework
  StackTrace:
   at NUnit.Framework.Constraints.Comparers.StreamsComparer.Equal(Object x, Object y, Tolerance& tolerance, ComparisonState state, NUnitEqualityComparer equalityComparer)
   at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance, ComparisonState state)
   at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance)
   at NUnit.Framework.Constraints.EqualConstraint.ApplyTo[TActual](TActual actual)
   at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, NUnitString message, String actualExpression, String constraintExpression)
   at Project.Tests.Namespace.SomeTests.TestMethod(String data, Boolean expected) in C:\_git\Project\Project.Tests\SomeTests.cs:line 128

Difference is running on our CI (which has Visual Studio Build Tools 2022 17.6 and uses vstest.console.exe through Azure DevOps Server) vs. running on a local dev machine (which has Visual Studio 2022 17.11 and uses either CodeRush Test Runner or the built-in Test Explorer.) There doesn't appear to be any update to NUnit3TestAdapter, with 4.6.0 being the most recent one.
Same results when running dotnet test in the project directory.

That particular test project is .NET Framework 4.8 with an SDK-style project format (so barely 20 lines of project file with the minimum necessary not provided by various .targets files) while the assembly being tested is .NET Standard 2.0.

The common thing between all failing tests is that they call Is.EqualTo with a parameter or use a Has.Count.EqualTo constraint against a collection:

[Test]
public void TestCollectionCount()
{
    var utility = new UtilityClass("something\nto\nparse");
    Assert.That(utility.Lines, Has.Count.EqualTo(3)); // throws
    //Assert.That(utility.Lines.Count(), Is.EqualTo(3)); // also throws
    //ClassicAssert.AreEqual(3, utility.Lines.Count()); // also throws
}
[TestCase(null, false)]
[TestCase("anything", true)]
public void TestCollectionEmpty(string input, bool expected)
{
    var utility = new UtilityClass(input);
    Assert.That(utility.HasLines, Is.EqualTo(expected)); // throws
}

// with a simplified (and somewhat pointless) test class:
class UtilityClass
{
    public UtilityClass(string? lines)
    {
        Lines = lines?.Split('\n').ToList() ?? [];
        HasLines = Lines.Count > 0;
    }
    public List<string> Lines { get; }
    public bool HasLines { get; }
}

All of those asserts were converted with NUnit.Analyzers before upgrading to 4.x, they used to be Assert.IsTrue/Assert.IsFalse or Assert.AreEqual.

Replacing the EqualConstraint with something else (for example expected ? Is.True : Is.False rather than using Is.EqualTo(expected)) lets the boolean tests pass; but no alternative seems to work for the count tests.
What's even more surprising: Other Is.EqualTo() (using strings for example) work just fine.

[emwl]

#4735/#4737 seem like good candidates for causing this. The projects that fail don't use System.Buffers or System.Memory; while a few of our projects still work (which do reference those assemblies, either directly or indirectly.)

A viable workaround is adding a reference to System.Memory to the test project; but is this really what should be happening here?
Feels like the NuGet spec file should do that instead (or whatever the most appropriate location is.)

[OsirisTerje]

@emwl Thanks a lot for this!

How does a csproj for the repro above look? GOT IT!
DId I read you right, that it works with sdk 17.11 and not 17.6, or not with any of them ? Don't work with any of these.

@manfred-brands Can you chime in here ?

UPDATE:
Repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4794

[emwl]

No, it doesn't work with either; but the error message is different (one complains about System.Buffers while the other complains about System.Memory.)

The actual .csproj looks like this (with a few things anonymized/removed,) but this should do the trick:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net48</TargetFramework>
    <OutputType>Library</OutputType>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
    <AppendTargetFrameworkToOutputPath>false</AppendTargetFrameworkToOutputPath>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>
  <ItemGroup>
    <ProjectReference Include="..\OtherProject\OtherProject.csproj" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="NUnit" Version="4.2.0" />
    <PackageReference Include="NUnit.Analyzers" Version="4.3.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="NUnit3TestAdapter" Version="4.6.0" />
    <PackageReference Include="RhinoMocks" Version="3.6.1" />
    <!-- uncomment to fix test failures caused by nunit#4794 -->
    <!--<PackageReference Include="System.Memory" Version="4.5.5"/>-->
  </ItemGroup>
</Project>

You don't really need the ProjectReference for this (I think,) but main point is that nothing in that chain pulls in System.Memory. Uncommenting the reference makes the tests work correctly.

I just checked with a few of the other projects, it does work in a test project targetting .NET 6.0 or .NET 8.0; so I believe this is limited to .NET Framework.
And it might be missing the System.Memory entry in here:

nunit/nuget/framework/nunit.nuspec

Lines 34 to 37 in 3f1eb04

	<group targetFramework="net462">
	<dependency id="System.Threading.Tasks.Extensions" version="4.5.4" exclude="Build,Analyzers" />
	<dependency id="System.ValueTuple" version="4.5.0" exclude="Build,Analyzers" />
	</group>

(I don't have any experience authoring NuGet Packages though, so this is only a guess.)

[OsirisTerje]

Thanks!

Repro here: https://github.com/nunit/nunit.issues/tree/main/Issue4794

[manfred-brands]

This was introduced with #4668 where a dependency on System.Memory was added.
We also need to add this dependency to the nunit.nuspec file.

[OsirisTerje]

Also System.Buffers ?

[emwl]

Also System.Buffers ?

Good question. I tested System.Buffers in my first workaround, and it didn't change anything. Adding System.Memory (with or without System.Buffers) did the trick. Memory does have Buffers as transitive dependency, so it is pulled in anyways. Not sure if this needs an explicit reference or not.

[OsirisTerje]

I updated the repro project with a target for net8 too, just to show it works there.

[manfred-brands]

Also System.Buffers ?

No. System.Memory depends already on System.Buffers: