nunit Framework tests do not run with "dotnet test" nor inside VS2019 (Windows)

[FrankRay78]

Expected: as per the NUnit documentation here, I should be able to run the unit tests from the command line using dotnet test

Actual: No tests are ever found

Environment
Dell XPS 15, 32BG ram
Running clrver outputs

Versions installed on the machine:
v2.0.50727
v4.0.30319

.Net 5.0 SDK has been installed manually

Steps to reproduce

1. Open VS2019 and clone the nunit repo into a clean directory
2. Open windows PowerShell and navigate to the directory
3. Run build --target=Test --configuration=Release
4. NuGet packages restored, all dlls are built, all unit tests pass (for all framework targets)

5. Navigate to the bin\Release\net5.0 directory
6. Run dotnet test nunit.framework.tests.dll --framework:net5.0

7. Open the solution in VS2019 and set the test target architecture to x64

8. Save all and close VS2019
9. Repeat steps 3, 5 and 6 above

10. Specify the path to the TestAdapters located here:

11. Run the following command dotnet test nunit.framework.tests.dll /TestAdapterPath:C:\Users\frank\Documents\Professional\Source\nunit-VS2019-2\bin\Release\net5.0 --framework:net5.0

12. Open the solution in VS2019 and upgrade to the latest TestAdapter:

13. Repeat build step 3 above.
14. Check the output has the updated TestAdapter

15. Run the following command dotnet test nunit.framework.tests.dll /TestAdapterPath:C:\Users\frank\Documents\Professional\Source\nunit-VS2019-2\bin\Release\net5.0 --framework:net5.0

nb. I've tried the above steps for the net4.5 build output to the same effect

Commentary
I believe the inability to find / execute the tests using dotnet test may explain why I cannot get unit tests to execute within Visual Studio 2019 (with or without Resharper) on Windows and also JetBrains Rider (on either Windows 10 and Debian Linux) - see attached log output
VS2019 test output log.txt

Related Issues
#3008
#2973

Related Discussions
#3861

Additional Information

I ran the following command from the bin\Release\net5.0 directory, dotnet test nunit.framework.tests.dll --list-tests --framework:net5.0 --diag dotnet_test_log.txt --logger "console;verbosity=detailed"

The following logs were generated:
dotnet_test_log.host.21-06-01_20-09-24_22867_6.txt and dotnet_test_log.txt

[manfred-brands]

I found some time to look into this today.
The tests are all skipped because the adapter doesn't find a supported nunit.framework.

The NUnit3DriverFactory.IsSupportedTestFramework explicitly tests on version 3 of the framework:

return NUNIT_FRAMEWORK.Equals(reference.Name, StringComparison.OrdinalIgnoreCase) && reference.Version.Major == 3;

Note that this class is in the nunit-console repository.

The current master branch has version 4.

Locally if have tried a version with the one-token change (>= 3) and it now does find tests, both with dotnet test and inside Visual Studio`.

Note that this involves changes in 3 repositories:

• nunit-console (change == 3 into >= 3),
• nunit-vs-adapter (reference newer version of nunit.engine)
• nunit (reference newer version of nunit3testadapter)

But they not all pass!
Only 2 errors in net45, but 70 in net5.0.
There are several errors like the below:

  Failed LoadWithPreFilter("NUnit.Tests.Assemblies","NUnit.Tests.Singletons","NUnit.Tests.FixtureWithTestCases") [2 ms]
  Error Message:
     Exception has been thrown by the target of an invocation.
  ----> Delegates must be of the same type.
  Expected: Runnable
  But was:  NotRunnable

That failure is in when calling AssemblyLoadContext.LoadFromAssemblyPath.

Note that the above failures also occur in the v3.13-dev branch with the existing code (no changes).
So there is clearly a difference between the nunitlite runner part of nunit and the nunit.engine part of nunit-console.

Are they supposed to be compatible?

[OsirisTerje]

@manfred-brands

The NUnit3DriverFactory.IsSupportedTestFramework explicitly tests on version 3 of the framework:

return NUNIT_FRAMEWORK.Equals(reference.Name, StringComparison.OrdinalIgnoreCase) && reference.Version.Major == 3;

I am fixing this now for the embedded engine in the adapter, which is based off the 3.15.X branches. I will release a 4.5 alpha release of the adapter on nuget so it can be tested in the nunit framework repo.

[OsirisTerje]

@manfred-brands @stevenaw @rprouse
I have release version 4.5.0-alpha.3 on nuget which accepts NUnit 4, ref @manfred-brands comment above.
Trying that version on the framework repo, doesn't help much, as the same errors are still there, missing testhost.runtimeconfig.json, a file I don't think should be there at all.

Further, as can be seen in the image above, the net462 is not displayed, and there is a warning about that,

So it seems the net462 is not building properly, but no compiler warning or error about that.

And there is only one place where the adapter is added, and that is the framework.test project.

This Issue 3937 in the vstest repo is the only place I have found that refers to the same error messages, and the gist of it is that it looks like mixing framework DLLs in the same bin folder can cause this. Not sure which these can be, but I do see that all projects build to the same bin folder. It should be per framework, but that would not apply for dependencies.

[stevenaw]

Thanks for taking a look at this @OsirisTerje and experimenting with an alpha release of the adapter. I'll also try to take a second look at this when I am able. From my recollection this issue appears to only affect the main branch of the repo, but they've started to diverge enough such that there are now far more differences than simply the v3 / v4 assemblyinfo.

[OsirisTerje]

Do you know if the binary outputs have been changed?

Link to alpha: https://www.nuget.org/packages/NUnit3TestAdapter/4.5.0-alpha.3

[OsirisTerje]

1. Adding the https://www.nuget.org/packages/NUnit3TestAdapter/4.5.0-alpha.3 makes the tests reappear, and they run.
   

70 tests out of 5600 tests fails.

Not the framework462 though, those tests do not appear, but that must be something else.

However, the source code links from the tests in the test explorer to the source code doesn't work, it doesn't find the source code links.

2. Changing the outputpath from a common bin output, in Directory.Build.props to individual project bin outputs, restores the source links.
   

3. The failing tests seems all to be dependent upon DefaultTestAssemblyBuilder and NUnitTestAssemblyRunner, and these are failing.
   
   It fails during loading of the mockassembly:
   
   and the exception is thrown here:
   
   It complains about the Func_3 being mismatched.
   Note the red arrow, it fails at that point, so the underlying methods have not been called yet.
   The code that fails is from 2017, so something outside must have changed. @jnm2 - any comments here?

[OsirisTerje]

Exception details:

[stevenaw]

Thanks @OsirisTerje !
I'm not at a computer at the moment, but the main change I can recall between the branches aside from the version bump is that some older TFMs were dropped in main. There were some build changes to minimize the number of places we define the target frameworks by putting them in a Directory.build.props file, but I didn't think the build output directory itself changed

[OsirisTerje]

@stevenaw No, the build output directory have been like that since 2017. But I seem to remember that the last time I worked with the framework (a few months back though), these things did work. Not sure though.

[OsirisTerje]

There was a missing 462 in the alpha.3, so have uploaded a 4.5.0-alpha.4 which fixes the 462 tests. They actually run green too, it has a different way of loading assemblies, so it is not bothered by the other loading issue above.

[OsirisTerje]

@jnm2 Any idea why the loading of the assemblies using dynamic loading of the AssemblyLoadContext would fail?

[OsirisTerje]

Added System.RunTime.Loader nuget package to get hold of the AssemblyLoadContext, and it works. See latest in the PR.