Improved location of extensions

[CharliePoole]

@Chraneco commented on Mon Aug 15 2016

Hi!

We recently updated to NUnit 3.4.1 which unfortunately broke the integration with TeamCity. After some research I found out that this is because the TeamCity integration was moved to a separate NuGet package.

In our build pipeline we copy the contents of packages\NUnit.ConsoleRunner.3*\tools as part of a build artifact to a separate location, leaving behind the contents of all other packages (including NUnit.Extension.TeamCityEventListener.*).
I was hoping that it was enough to copy the "teamcity-event-listener" DLL into the folder next to "nunit3-console.exe", but it didn't help. That is because the NUnit engine only looks at the following hard-coded directories for extensions (listed in the nunit.engine.addins file):

../../NUnit.Extension.*/**/tools/     # nuget v2 layout
../../../NUnit.Extension.*/**/tools/  # nuget v3 layout

Unfortunately, this directory structure does not exist where we run our tests.

Making the following adjustment solves the problem for us

../../NUnit.Extension.*/**/tools/     # nuget v2 layout
../../../NUnit.Extension.*/**/tools/  # nuget v3 layout
teamcity-event-listener.dll

because we now copy the DLL into the directory where the console runner is put but we don't want to make this change every time we update NUnit. Is there a more generic solution for this? For example, letting the console runner always search in its own location for extension DLLs would be a possibility.

---

@NikolayPianikov commented on Mon Aug 15 2016

@Chraneco you could restore NUnit using this command line nuget.exe install NUnit.Runners -Version 3.4.1 -o packages and copy all installed packages

---

@Chraneco commented on Mon Aug 15 2016

@NikolayPianikov Thanks for your comment. At the point when we execute the unit tests we are completely outside of a Visual Studio project. The build artifacts contain only the necessary files and in a different directory structure. Restoring NuGet packages there is not possible and also not wanted.

---

@NikolayPianikov commented on Mon Aug 15 2016

Is it possible to update the .addins file in the script where you make your own NUnit tool directory?

---

@Chraneco commented on Mon Aug 15 2016

All the files and directories are put in place by TeamCity itself via Artifact dependencies.

---

@CharliePoole commented on Mon Aug 15 2016

@Chraneco Yes... we are discovering that the existing setup for locating extensions is not sufficient.

The main problem is that extensions are looked for relative to the directory containing the engine. The paths in the .addins file are tailored for an engine installed as an extension to find an engine installed as an extension to find other engines installed as an extension. In a case like yours, it would be handy to be able to locate addins relative to the VS project or the test assembly itself.

Since we don't already have an issue on this general topic, I changed the title of this issue accordingly. Things we may consider:

1. Adding additional roots where we look for .addins files
   • test assembly directory
   • global per-user location
   • global per-machine location
   • parent directory of the engine directory, recursively
2. Add an option to the command-line to specify extension paths to search
3. Convention-based approach, by which .addins files reference one another
   • Raises the question of which should be the true root

[davidjward30]

I'm running nunit under teamcity and everything worked fine until we ended up with 2 versions of nunit3 in the repository (3.2.1 and 3.4.1). This tends to happen as different product teams migrate at different times. This caused the nunit 3.2.1 tests to immediately fail:

ProjectLoader extension .nunit is already handled by another extension.

The extensions are checked in with the standard nuget layout

[CharliePoole]

@davidjward30 Your problem isn't really related to this issue, which is about how the engine actually finds extensions. It's definitely intentional that you can't install the same extension twice, as you might imagine. The code checks its own behavior, eliminating any duplication in the actual files it examines but if you have two different copies of an extension in different directories, it will try to use both of them.

Extension installation requires manual tuning if you do anything except use the default install. There is no reason why you can't use different versions of NUnit in different projects, because the nunit package is just the framework and has nothing to do with extensions. Having multiple versions of the engine and of extensions is another matter. You should never do that within a solution.

It would require much more detail about what you are doing to give any specific suggestions for how to change your setup. If you need that - if the above doesn't tell you what to do - please ask about it in then nunit-discuss list rather than continuing here.

[rprouse]

@davidjward30 to elaborate on what @CharliePoole said, it is fine to use two different versions of the NUnit framework in your solutions, but you should try to use the latest version of NUnit Console. Multiple versions should work, but if I remember right, we had a bug in 3.2 where it finds all extensions including duplicates. That has been fixed in 3.4.

Just update your Console runner to 3.4.1+ and use it to run your 3.2.1 tests.

[davidjward30]

@CharliePoole @rprouse Thanks for the advice; indeed I can run 3.4 and 3.2 tests with the 3.4.1 runner.

[CharliePoole]

Situation brought up in an email discussion by @NikolayPianikov ...

Just run these commands one by one:

nuget.exe install NUnit.Runners -Version 3.4.1 -o packages

nuget.exe install NUnit.Runners -Version 3.5 -o packages

nuget.exe install NUnit.Runners -Version 3.6 -o packages

After that run a NUnit console from one of directories packages\NUnit.ConsoleRunner.x.x.x

It is a general case I think. For simplicity, let's assume that they are backwards compatible J The question is which extensions will be used? Do we have any rules on that?

[sergiorykov]

Do you have strategy to choose compatible versions of extensions for runner?

nunit.nuget.addins:

../../NUnit.Extension.*/**/tools/     # nuget v2 layout
../../../NUnit.Extension.*/**/tools/  # nuget v3 layout

Eventually, after a couple of updates (without cleaning packages folder) you will get several versions of the same packages.

14.09.2016  18:44    <DIR>          NUnit.Console.3.0.1
21.09.2016  11:55    <DIR>          NUnit.Console.3.4.1
12.10.2016  15:42    <DIR>          NUnit.Console.3.5.0
21.09.2016  11:55    <DIR>          NUnit.ConsoleRunner.3.4.1
12.10.2016  15:42    <DIR>          NUnit.ConsoleRunner.3.5.0
21.09.2016  11:55    <DIR>          NUnit.Extension.NUnitProjectLoader.3.4.1
12.10.2016  15:42    <DIR>          NUnit.Extension.NUnitProjectLoader.3.5.0
21.09.2016  11:55    <DIR>          NUnit.Extension.NUnitV2Driver.3.4.1
12.10.2016  15:42    <DIR>          NUnit.Extension.NUnitV2Driver.3.5.0
21.09.2016  11:55    <DIR>          NUnit.Extension.NUnitV2ResultWriter.3.4.1
12.10.2016  15:42    <DIR>          NUnit.Extension.NUnitV2ResultWriter.3.5.0
21.09.2016  11:55    <DIR>          NUnit.Extension.TeamCityEventListener.1.0.1
12.10.2016  15:42    <DIR>          NUnit.Extension.TeamCityEventListener.1.0.2
21.09.2016  11:55    <DIR>          NUnit.Extension.VSProjectLoader.3.4.1
12.10.2016  15:42    <DIR>          NUnit.Extension.VSProjectLoader.3.5.0

[sergiorykov]

As for me to get clear understanding of migration process and how extensibility works will be enough:

1. in NUnit.Console add `tools\readme.txt'

• with suggestion to move path to NUnit.ConsoleRunner\tools
• link to github readme about extensibility and migration
• warning that you must add all dependent packages to packages.config if it was independent installation using command line nuget.exe

2. improve Readme.md

• migration plan from 3.0.1 and earlier (must be there - you broke backward compability in a minor release (3.0 -> 3.5))
• how extensibility works (all that magic)
• section about independent installation using 'nuget.exe' (user should explicitly specify all required packages) - it's limitation of nuget.exe - it willn't download all dependent packages. it's one of it's serious drawbacks

Hope this helps!

It's a point of view of a man who haven't seen all this functionality from the ground up as you saw it :)

[rprouse]

Eventually, after a couple of updates (without cleaning packages folder) you will get several versions of the same packages.

As of NUnit 3.2.1 (I think that is the version 😄 ), the engine automatically uses the latest version of each extension it finds and ignores older versions.

[CharliePoole]

@sergiorykov As Rob points out, we choose the "best" (newest) version of each extension package. There isn't a concept of compatibility wrt a particular engine version because extensions are required to be backward compatible. They can only specify a minimum engine version under which they work.

So with the directory example you posted, the engine would use version 1.0.2 of the teamcity extension and 3.5.0 of all the other extensions. The console / engine version would depend on which of the two available you ran.

We should have a test that verifies this, however.

I still have to look at the original 3.0 layout of files. If, as you say, all the files were actually contained in the package rather than handled as dependencies, we should document the change. I'm not sure, however, if the info really belongs in Readme.md, which seems to be getting a lot of work to do. :-) We do have pages and pages of docs where this could go, including the Breaking Changes page.

I'm still confused by the notion that nuget.exe won't download the dependent packages. I'm pretty sure that is what it does in our CI builds. Can you post a sample command line that demonstrates the problem?

[garuma]

Just to add another case to this issue: I would like to provide a framework driver extension to be found by Visual Studio NUnit test provider which, due to the current path limitations, means that a custom .addin file needs to be placed in the same directory than the engine DLL as it's the only thing I found would work from the current 3.6 (https://github.com/nunit/nunit-console/blob/release/3.6/src/NUnitEngine/nunit.engine/Services/ExtensionService.cs#L183).

This makes it quite impractical to distribute said extension since it requires manual copying from the user or re-packaging the original NUnit provider NuGet to include the extra extension.