Crash errors when mismatched copies of nunit.engine.api are encountered

[ChrisMaddock]

Since version 3.0, the nunit.engine.api assembly has been pinned at v3.0. Historically, we've made changes to the assembly that we considered to be backwards compatible, and subsequently released these without changing the version number.

We've since found out that certain runners can end up encountering multiple different builds of nunit.engine.api.dll all versioned 3.0.0, which has the potential to crash the runners under certain circumstances. This is particularly prevalent when the adapter is involved, as it copies the engine and associated assemblies into the test directory. Examples of this in #821, #800, #424, nunit/nunit#1348, dotnet/fsharp#2783, #6

In #802 we started incrementing the nunit.engine.api assembly version, however that broke the loading of extensions (#807). There's currently no clear solution to fixing both of these problems simultaneously - one probelm is caused by not incrementing the assembly version, and the other issue is caused when we do increment the assembly version.

Until this issue is fixed, we should try and avoid making any changes to the nunit.engine.api assembly. I think it remains to be seen how feasible that is.

Potential solutions

1. Fix loading of extensions referencing Engine API v3.0.0 #813 was an attempt at a workaround for this issue, by introducing a custom assembly resolver for the loading of extensions. This feels very risky and like opening a can of worms.

2. I think that if we required extension assemblies to be packaged alongside the version nunit.engine.api.dll they were built against, this would resolve the issue. Issue here: the NUnit-Org-maintained extensions haven't thus far been built in this way, so would all require new releases. I also expect many third-party extensions have been built using our own extensions as examples, and thus will hit the same problem.

3. Going forward, we agree to only change the nunit.engine.api assembly again at major versions. I personally think this is too restrictive to ongoing development - however does perhaps highlight that the nunit.engine.api assembly should be broken up in future major releases. (Ref: the proposed changes in v4 Console/Engine Platform support  #770)

[ChrisMaddock]

@CharliePoole has written a nice blog on the issue regarding conflict between different versions of the NUnit Console and NUnit VS adapter: http://charliepoole.org/technical/nunit-engine-version-conflicts-in-visual-studio.html