Repository navigation
NUnit 4: dotnet test behaving differently locally vs on GH Actions (failing) #4571
Description
Started in #4569 - all those variations work fine in Windows 11 in addition to VS 17.8 + 17.9p1 with the NUnit 4 4.1.0-adev.1 package from MyGet:
PS D:\GitWorkspace\ILSpy\ILSpy.Tests> dotnet test
PS D:\GitWorkspace\ILSpy\ILSpy.Tests\bin\Debug\net8.0-windows> dotnet test ilspy.tests.dll
PS D:\GitWorkspace\ILSpy\ILSpy.Tests\bin\Debug\net8.0-windows> dotnet test --logger "junit;LogFileName=Debug.xml" --results-directory test-results ilspy.tests.dll
However, as part of GH Actions run (see PR icsharpcode/ILSpy#3132), it fails to find tests
Run dotnet test --logger "junit;LogFileName=Debug.xml" --results-directory test-results $env:Tests1 $env:Tests2 $env:Tests3
Microsoft (R) Test Execution Command Line Tool Version [17](https://github.com/icsharpcode/ILSpy/actions/runs/7030026378/job/19128743036#step:10:18).8.0 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 3 test files matched the specified pattern.
No test is available in D:\a\ILSpy\ILSpy\ICSharpCode.Decompiler.Tests\bin\Debug\net8.0-windows\win-x64\ICSharpCode.Decompiler.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
No test is available in D:\a\ILSpy\ILSpy\ILSpy.Tests\bin\Debug\net8.0-windows\ILSpy.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
No test is available in D:\a\ILSpy\ILSpy\ILSpy.BamlDecompiler.Tests\bin\Debug\net8.0-windows\win-x64\ILSpy.BamlDecompiler.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Results File: D:\a\ILSpy\ILSpy\test-results\Debug.xml
dotnet test --diag output file for such a run attached: dotnettestdiag-Debug.txt
What could be the difference of local Windows vs GH Windows runner? (and how to pinpoint)
Activity
christophwille commentedon Nov 29, 2023
Just a wild and stupid guess after having an Espresso - could it be that server OSes are not covered by #4569? After all, we are using a 2022 server image on GH.
OsirisTerje commentedon Nov 29, 2023
Can you try to add a new workflow, focusing just on the ILSpy.Tests and the dependencies of that of course, nothing else, and just an out-of-the-box action template, and a standard windows-10/11.
christophwille commentedon Nov 29, 2023
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
That's not happening I guess.
rgroenewoudt commentedon Nov 29, 2023
I just encountered a bug that finding tests doesn't work if TargetFramework contains windows: nunit/nunit3-vs-adapter#1141
This is perhaps the same issue.
christophwille commentedon Nov 29, 2023
That's #4564 - this is using the fix already, but with GH still not working.
manfred-brands commentedon Nov 29, 2023
@christophwille I need to check how Windows Server is detected and add support for it to NUnit as the existing Platform has no support for server OS. I'll try to get to it tomorrow.
manfred-brands commentedon Nov 30, 2023
Ok, the Windows runner reports as Windows 10 Server.
There was already a WindowsServer10 definition in NUnit Platform.
I have updated the OSPlatformTranslator to also add this Platform when appropriate.
christophwille commentedon Nov 30, 2023
Thanks; once merged & MyGet'd I can test it with our PR.
OsirisTerje commentedon Dec 1, 2023
@christophwille The -adev.2 was out yesterday on Myget. Can you verify it works?
christophwille commentedon Dec 1, 2023
Yes it worked.
OsirisTerje commentedon Dec 1, 2023
Thanks! Awesome. Then I'll try to get 4.0.1 out during the weekend.
OsirisTerje commentedon Dec 2, 2023
Version 4.0.1 is out on nuget, https://docs.nunit.org/articles/nunit/release-notes/framework.html#nunit-401---december-2-2023