Using nunit.console.exe (3.15.0) with .nunit input and --inprocess still doesn't work

[dschuermans]

Hey,

Either I'm extremely stupid, or using NUnit project files while defining --inprocess still doesn't work (although it is supposed to work, according to #386)
This is the first time ever I'm using NUnit though, and it's an existing code base and unit test setup.
I've only been moving dlls around and I'm now struggling with getting some tests to work that depend on a DLL thats no longer in its old location but else where.

Some (fictional) context

Let me first provide some context as to how the tests are being run and how the context looks like in which the tests are being run.
For sake of simplicity, I've simplified paths and filenames

My folder structure is as follows: (located in D:\sources\myproject\bin\release)

- NUnit
  - lib
  - tools
    - nunit3-console.exe
  - nunit-summary.exe
- subfolder
  - my.dll
- test.dll

I presume that the NUnit folder comes from a zip, since it's being copied into the build output folder by a post-build event on some project which contains the NUnit folder and has it checked into source control with all its contents

The subfolder/my.dll used to reside in the root, alongside test.dll and everything worked great.
The tests were being executed using the following command:

NUnit\tools\nunit3-console.exe test.dll /inprocess /work:"D:\sources\myproject\bin\release\UT_Results" /result:test.xml;format=nunit2 /out:TestResult_test.txt

The problem

Now with my.dll being moved into subfolder, I get a "System.IO.FileNotFoundException: Cannot load file or assembly" as expected.
I look around and stumbled upon the NUnit Project, which seem to have a solution for my problem, namely the binpath setting

So I created the following NUnit project by hand and placed it in the root folder, along side test.dll, named test.nunit:

<NUnitProject>
	<Settings activeconfig="Release" />
	<Config name="Release" binpath="subfolder">
		<assembly path="test.dll" />
	</Config>
</NUnitProject>

Then when attempting to run the test with the following command:
NUnit\tools\nunit3-console.exe test.nunit /inprocess /work:"D:\sources\myproject\bin\release\UT_Results" /result:test.xml;format=nunit2 /out:TestResult_test.txt

I get the following output:

NUnit Console 3.15.0 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Friday, 3 June 2022 14:25:19

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    test.nunit


Errors, Failures and Warnings

1) Invalid : D:\sources\myproject\bin\release\test.nunit
File type is not supported

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2022-06-03 12:25:20Z
    End time: 2022-06-03 12:25:20Z
    Duration: 0.033 seconds

System.InvalidOperationException : Result contains no assemblies.

--InvalidOperationException
Result contains no assemblies.
   at NUnit.Engine.Addins.NUnit2XmlResultWriter.InitializeXmlFile(XmlNode result)
   at NUnit.Engine.Addins.NUnit2XmlResultWriter.WriteXmlOutput(XmlNode result, XmlWriter xmlWriter)
   at NUnit.Engine.Addins.NUnit2XmlResultWriter.WriteResultFile(XmlNode result, TextWriter writer)
   at NUnit.Engine.Addins.NUnit2XmlResultWriter.WriteResultFile(XmlNode result, String outputPath)
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)

I'm not able to execute the test assembly without the /inprocess, as i'm greeted with other errors.
NUnit\tools\nunit3-console.exe test.dll /work:"D:\sources\myproject\UT_Results" /result:test.xml;format=nunit2 /out:TestResult_test.txt, which is the same command that used to work but without the /inprocess, will yield the following error:

Unhandled Exception: NUnit.Engine.NUnitEngineException: Remote test agent exited with non-zero exit code -532462766
   at NUnit.Engine.Services.TestAgency.OnAgentExit(Process process, Guid agentId)
   at NUnit.Engine.Services.TestAgency.<>c__DisplayClass13_0.<GetAgent>b__0(Object sender, EventArgs e)
   at System.Diagnostics.Process.OnExited()
   at System.Diagnostics.Process.RaiseOnExited()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)

Errors, Failures and Warnings

1) Error :
NUnit.Engine.NUnitEngineException : Unable to acquire remote process agent
--NUnitEngineException
Unable to acquire remote process agent
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunnerIfNeeded()
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

So I'm kind of starting to pull my hair out here.
I solved the issue with the dlls being moved for our executable by adding a probing element to its configuration file but I don't think such a thing is possible for the nunit3-console.exe?

I hope you can shed some light onto this issue or if it's an error on my end.

Thanks!

[CharliePoole]

This is a bit hard to answer, primarily due to "I presume that the NUnit folder comes from a zip, since it's being copied into the build output folder by a post-build event..." It sounds as if your team has set up some non-standard way of installing NUnit... apologies if that's not the case.

If that's what happening, it's certainly a valid approach, but not one I can help you with unless you can tell me precisely what is being done. Can you get info from whomever set this up? Alternatively, provide a full listing of all the files in each directory.

The sub-problem you are having with an NUnit Project is easy to answer, however. You cannot use an NUnit project without installing the NUnit project loader extension. Similarly you cannot use output format nunit2 without installing the NUnit V2 output writer extension, but I'm assuming that's installed if the command has worked in the past.

But what really counts is to describe precisely which NUnit files are where!

BTW, there's no evidence in your output that /inprocess is causing the problem.

[dschuermans]

Hey @CharliePoole,

I've asked the team responsible for the project, in the meanwhile I've generated the folder & file structure for the NUnit folder:

NUnit\lib
NUnit\tools
NUnit\nunit-summary.exe
NUnit\lib\net40
NUnit\lib\net40\nunit.framework.dll
NUnit\lib\net40\nunit.framework.xml
NUnit\tools\agents
NUnit\tools\net20
NUnit\tools\net20.engine.addins
NUnit\tools\nunit.console.nuget.addins
NUnit\tools\nunit.engine.api.dll
NUnit\tools\nunit.engine.api.xml
NUnit\tools\nunit.engine.core.dll
NUnit\tools\nunit.engine.dll
NUnit\tools\nunit3-console.exe
NUnit\tools\nunit3-console.exe.config
NUnit\tools\testcentric.engine.metadata.dll
NUnit\tools\agents\net40
NUnit\tools\agents\net40\nunit-agent-x86.exe
NUnit\tools\agents\net40\nunit-agent-x86.exe.config
NUnit\tools\agents\net40\nunit-agent.exe
NUnit\tools\agents\net40\nunit-agent.exe.config
NUnit\tools\agents\net40\nunit.agent.addins
NUnit\tools\agents\net40\nunit.engine.api.dll
NUnit\tools\agents\net40\nunit.engine.api.xml
NUnit\tools\agents\net40\nunit.engine.core.dll
NUnit\tools\agents\net40\testcentric.engine.metadata.dll
NUnit\tools\net20\nunit-v2-result-writer.dll

Hope this helps

[dschuermans]

@CharliePoole
Just got word that they are taken from an unpacked nuget package

[dschuermans]

@CharliePoole
As you mentioned, the NUnit Project loader extension seemed to have been the missing piece here.

Since we're not really using NuGet to install & maintain NUnit, I was wondering if the changes i've made are sufficient or if there's something I can do differently.

Right now I have:

• Created an extensions folder inside the NUnit\tools\ folder
• Modified the NUnit\tools\nunit.console.nuget.addins file and add a line at the bottom extensions/
• Copied the nunit-project-loader.dll from the extension nuget into the extensions folder

With these changes, I was able to utilize the NUnit Project file and the dll was correctly loaded from the subfolder 😄 🙏

One question comes to mind:
Is the nunit3-console.exe hardcoded to look for specific addins files?
Can I create a new addins file and put it in the NUnit\tools\ folder and have it picked up by nunit3-console.exe?

*Edit:
I went ahead and tried it anyway, it seems to work with a custom addins file too 😀

[dschuermans]

Going to close this, as my issue is solved.

Thanks for the swift response!

[CharliePoole]

Glad you have it fixed! Regarding the addins file, I think that it's generally a better practice to leave the addins file installed by NUnit untouched and add your own. However, in your case, you are not using the delivered addins file, which you copy, since it contains relative paths, which only work when both the engine and the extensions are installed via nuget. So, you may as well completely replace it using your own paths.