NUnit 3.4: NUnit.ConsoleRunner is not picking up NUnit.Extension.NUnitV2ResultWriter

[nalla]

When running NUnit.ConsoleRunner and NUnit.Extension.NUnitV2ResultWriter nuget packages in the following layout

.\tools\NUnit.ConsoleRunner\tools\nunit3-console.exe
.\tools\NUnit.Extension.NUnitV2ResultWriter\tools\nunit-v2-result-writer.dll

the call "C:\some\dir\tools\NUnit.ConsoleRunner\tools\nunit3-console.exe" "C:\some\dir\src\Tests\Test.Assembly.dll" "--work=C:\some\dir\tests" --full "--result=C:\some\dir\tests\TestResult.xml;format=nunit2" --labels=All --noheader succeeds when both packages have version 3.2.1 but fails if both packages have version 3.4.0. The error message is Unknown result format: nunit2

[CharliePoole]

Our current code assumes that your packages are installed by nuget using a particular format. Can you let us know how you did the installation? I'm concerned that we are failing to allow for some variation in how nuget works.

As a workaround, create a file of type .addins in the same directory as nunit.engine.dll with a single entry giving the relative path from that directory to nunit-v2-result-writer.dll.

[rprouse]

@nalla, are you using Cake build or another build tool to install the NUnit Console? That directory structure looks like the Cake build format.

[CharliePoole]

That would explain things! We are searching explicitly for a packages directory and for the name of a subfolder that starts with our package id.

[CharliePoole]

We should try to do this for 3.4.1. However, if it can't be done in short order we can just publicize the workaround.

[CharliePoole]

@nalla The 3.4.1 release will be put out tonight or tomorrow to deal with a different critical issue. If we had more information about how your packages were installed we might be able to deal with this issue as well. Without any info, the issue will wait for the next scheduled (3 months) or bugfix (not known) release.

@rprouse Have you installed NUnit via Cake on some project? Can you let me see the structure it creates?

[rprouse]

@CharliePoole I have used NUnit with Cake recently, but of course I can't find the project I did it on. Clearly I work on way too many codebases 😦

I am going to quickly update my nunit test project and will get back to you.

[CharliePoole]

It seems like we need some way for the engine to figure out that it is installed as part of a NuGet package. In that case, we can examine the old relative path ../../NUnit.Extension.*/**/tools/ to find extensions, without looking for the packages directory.

The trick is to know if we are part of a package, without checking for that directory! I'm thinking that I should add some sentinel file to the runner package, so it knows it's in a nuget package directory.

[rprouse]

@CharliePoole I added NUnit.ConsoleRunner to my test project using #tool "nuget:?package=NUnit.ConsoleRunner" in my build.cake file.

It installs to tools\NUnit.ConsoleRunner\tools\nunit3-console.exe as per the original report. I am going to add the NUnit 2 extension now and test.

[CharliePoole]

I'm wondering how the OP got the V2 result writer installed in parallel like that. Does Cake know about it?

[CharliePoole]

I suppose #tool will install any extension you point to, whether Cake knows about it or not.

[rprouse]

@CharliePoole #tool will only install tools it knows about, but the extensions can be added into the tools\packages.config file.

I cannot reproduce this. My tools\packages.config looks like this

<?xml version="1.0" encoding="utf-8"?>
<packages>
    <package id="Cake" version="0.13.0" />
  <package id="NUnit.Extension.NUnitV2Driver" version="3.4.0" />
  <package id="NUnit.Extension.NUnitV2ResultWriter" version="3.4.0" />
</packages>

Which produces a directory structure like that given in the report.

My cake test task looks like,

Task("Test-nunit-v2")
    .IsDependentOn("Build")
    .Does(() =>
{
    NUnit3("./nunit-v2/bin/" + configuration + "/nunit-v2.dll",
        new NUnit3Settings {
            Results = "TestResults-v2.xml",
            ResultFormat = "nunit2"
        });
});

And the result of running tests is the expected,

========================================
Test-nunit-v2
========================================
Executing task: Test-nunit-v2
NUnit Console Runner 3.4.0
Copyright (C) 2016 Charlie Poole

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.14366.0
  CLR Version: 4.0.30319.42000

Test Files
    C:/src/github/nunit-tests/nunit-v2/bin/Release/nunit-v2.dll


Test Run Summary
  Overall result: Passed
  Test Count: 5, Passed: 5, Failed: 0, Inconclusive: 0, Skipped: 0
  Start time: 2016-06-28 02:06:18Z
    End time: 2016-06-28 02:06:18Z
    Duration: 0.168 seconds

Results (nunit2) saved as C:/src/github/nunit-tests/TestResults-v2.xml
Finished executing task: Test-nunit-v2

Notice the nunit2 result format and this is a 2.6.4 test, so the v2 driver extension worked. NUnit was originally in my path, but I removed it, so that wasn't the reason it worked. I caught that when I didn't have the v2 driver listed 😄

Source code for this is at https://github.com/rprouse/nunit-tests

I am moving this out of the milestone and switching to confirm until we get more info. I believe @nalla is in Korea, so he probably won't be able to respond until tomorrow our time because of the time difference.

@nalla if you can provide more information on how to reproduce, we might get a fix in the 3.4.1 release since it may not happen tonight.

[CharliePoole]

Thanks for all that info. Saves me having to install it all for the first time since I haven't used the tool.

I don't think we need to precisely confirm this... I can confirm that there's no code in the engine that would ever find extensions installed in that configuration. The question is whether we should have such code or if this is something to leave to the user to create an addins file. The file we used in 3.2.1 could be copied into the Console.Runner directory and I'm pretty sure it would work.

Since it has been 14 hours since @nalla filed this I'm hoping he's up and ready to tell us more. :-)

[rprouse]

I was a bit surprised that the extensions were found with that directory structure, but how are the extensions found? How did it work? I had renamed the NUnit install directory to remove it from my path. Give me a bit, I am going to uninstall all versions and try again.

I don't like the concept of the user having to create an addin file. The various binaries in the tools directory shouldn't be checked into source control, so an addin file would be a pain. You could check in just that, but that would conflict with ignore files.

[rprouse]

@CharliePoole you said,

I can confirm that there's no code in the engine that would ever find extensions installed in that configuration

I have uninstalled every version of NUnit I have and can confirm that it is not in my path, but the v2 driver and result writers are working with Cake. Either Cake is adding to the path, or we have made a fortunate mistake 😄

It could be that something in Cake is making this possible and @nalla is running with some other tool with a similar NuGet layout.

[CharliePoole]

And these are not under a packages directory at all? Are there any addins files at play? The internal log will tell you where it's looking and what it finds.

As for user-edited addins files, we can look for them in the AppBase or the directory of the executable, which are probably the same.

[rprouse]

That is it, I had the console installed as a NuGet package in the solution, so it is in a packages directory. Cake walked up out of the tools directory and found the packages directory. Retesting...

[rprouse]

Now we are getting somewhere,

========================================
Test-nunit-v2
========================================
Executing task: Test-nunit-v2
NUnit Console Runner 3.4.0
Copyright (C) 2016 Charlie Poole

Unknown result format: nunit2
An error occured when executing task 'Test-nunit-v2'.
Error: NUnit3: Process returned an error (exit code -1).

This whole NuGet extensions thing is becoming a pain. I don't think walking up to packages was the right solution since you can override the name of the packages directory for nuget restore and Cake is installing NuGet packages with the nuget install command which allows you to strip out versions, change the output directory, etc.

Can't we just assume that from a tools directory we go up two directories and then all packages will be installed alongside that directory?

[CharliePoole]

That's essentially the logic that was in the wildcard path in nunit.nuget.addins. We can certainly emulate that approach in program logic or just put back the addins file - although I dislike depending on it
The other part is that we can't just trigger this from any directory named tools. I think we may want to do as I wrote earlier - add a file to the package that guarantees we are actually in such a directory.

I can get this finished tomorrow if we can wait a day or two to release.

[nalla]

Sorry for the late response. @rprouse It is indeed a cake build. Console runner is installed via #tool directive and the extension via packages.config in the ./tools directory.

I did this, because cake only installs packages containing an .exe via the #tool directive. But since I needed the extension, I added it to the config.

build.cake

#tool "nuget:?package=NUnit.ConsoleRunner

[...]

NUnit3( File( "./src/Test.Assembly/bin/debug/Test.Assembly.dll" ), new NUnit3Settings
  {
    NoHeader = true,
    Full = true,
    Labels = NUnit3Labels.All,
    Work = "./test-working-dir",
    Results = MakeAbsolute( File( "./test-working-dir/TestResult.xml" ) ),
    ResultFormat = "nunit2"
  } );

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Cake" version="0.13.0" />
  <package id="NUnit.Extension.NUnitV2ResultWriter" version="3.4.0" />
</packages>

Leading to the following directory structure

./tools/NUnit.ConsoleRunner
./tools/NUnit.Extension.NUnitV2ResultWriter
./tools/packages.config
./build.cake

What I can confirm is, that when I add the (from 3.2.1 removed) nunit.nuget.addins file back to the ./tools/NUnit.ConsoleRunner/tools directory. The test run succeeds.

BTW: Cake related it would be nice when adding the extension via the #addin directive that it will be picked up automatically. But using the #addin directive will result in the package being installed in the ./tools/addins/NUnit.Extension.NUnitV2ResultWriter directory or not being installed at all. At least on my machine was unable to install NUnit.Extension.NUnitV2ResultWriter with the #addin directive.

[rprouse]

@CharliePoole let's take the time to get this right. This issue is going to be less common and I think the Travis error is serious, but can wait a day.

The marker file is a good idea. What if we used existing files? If in tools directory and license exists in parent?

[CharliePoole]

Also, the nupkg exists in parent! Alternatively, I could go back to having nunit.nuget.addins.