Allow to debug parallel tests

[OsirisTerje]

The adapter currently blocks parallel running when a debugger is attached. Debugging parallel tests are very confusing and hard. See Issue #483. The workaround has been to use the NUnit3-console for that, but we can add a setting that turns this blocking off so that it is up to the developer if they want to work with dotnet test (which is using the adapter) or use the console.

This issue came from a discussion in the NUnit Slack, and was raised by Alejandro Carrazzoni. The conversation is copied (by permission) for context below:

Alejandro Carrazzoni
2 hours ago
Hi! I have an issue when running nunit3-console. Running dotnet test works, but when I use nunit3-console I get an "Assembly could not be found" error. Can anyone help?
25 replies

terje
2 hours ago
Versions?

Alejandro Carrazzoni
2 hours ago
NUnit3-Console 3.15.4

terje
2 hours ago
Can you provide a small repro solution?

terje
2 hours ago
And, also show how you use it

Alejandro Carrazzoni
1 hour ago
I couldn't get it to reproduce in a small solution. The problem is that I have a transitive dependency that doesn't get copied to the output folder when building a project. However, that missing dll doesn't cause any issue when running dotnet test, only when running nunit3-console

terje
1 hour ago
dotnet test and the nunit3-console find their dependencies in different ways. You can try the 3.16.3 version, it has a slightly different way of locating assemblies. There are also pre-releases of the 4.X version, which again has changes here. Also, do you call them the same way? Or do you call the dotnet test from a higher folder in the hierarchy?

terje
1 hour ago
And, any reason why you would even use the nunit3-console when you can use dotnet test?

Alejandro Carrazzoni
1 hour ago
I want to use nunit3-console so I can debug multiple tests running in parallel. When I run using dotnet test nunit seems to detect I'm using a debugger and will run every test on a single thread. I found while googling that running with nunit3-console should work

terje
1 hour ago
Do you have the link to that information?
New

Alejandro Carrazzoni
1 hour ago
#483

#483 Debugging parallel tests
When reporting a bug, please provide the following information to speed up triage:
• Nunit 3.9.0, NUnit3TestAdapter 3.10
• Visual Studio 2017 Enterprise
• .Net Framework
Repro: https://gist.github.com/Dispersia/7a1d5a8488efcca23c68957a668480ab
^ Obviously not a real test, assumed if breakpointing on the assert that Total would be all, or the next breakpoint would have them all set already.
Attached a quick repro, and this might be solved already.
I see under:
https://groups.google.com/forum/#!topic/nunit-discuss/X3z2kMac04U
that this issue was created:
Show more
Comments
12
https://github.com/[nunit/nunit3-vs-adapter](https://github.com/nunit/nunit3-vs-adapter)|nunit/nunit3-vs-adapternunit/nunit3-vs-adapter | Mar 9th, 2018 | Added by GitHub
👍
1

Alejandro Carrazzoni
1 hour ago
I'm calling both dotnet test and nunit3-console from the repository root, and the assembly is in deeper in the hierarchy

terje
1 hour ago
Ok, but you point down to the assembly, right?

terje
1 hour ago
For nunti that is

Alejandro Carrazzoni
1 hour ago
Yes, I point down to the assembly

Alejandro Carrazzoni
1 hour ago
Using nunit-console version 3.16.3 fixed the issue. Thanks!

terje
1 hour ago
Awesome. It might have other issues, but as you doesnt seem to have dumped into those, you should be fine for your debugging session. I found the code piece Charlie was referring too in the issue you linked, and I will make a setting for the adapter so that you can turn that off, and thus debug with dotnet test too for these cases.

terje
1 hour ago
What you're asking for, to be able to debug parallel tests, are absolutely a valid scenario.

terje
1 hour ago
So thanks for raising this 🙂

Alejandro Carrazzoni
1 hour ago
I've ran into another issue. The test assembly has an App.config file, and we have a build step that creates a copy named testhost.dll.config. However when I use nunit3-console neither .config file is found

Alejandro Carrazzoni
42 minutes ago
Looks like it's using the nunit-agent.dll.config file in nunit's installation folder (edited)

terje
41 minutes ago
That might be correct, the nunit3-console will not use other config files than its own. The Testhost is the Microsoft test runner which invokes the nunit adapter and thus the engine and framework

terje
36 minutes ago
I am in the process of making a 4.5 release of the adapter, and this fix is so small I might be able to squeeze that one in. The 4.5 doesn't have much in it, it is just for supporting nunit4, so taking this one too is no big deal. So if you can hold off for a day or max two, you should be able to use standard dotnet test. You should avoid using nunit3-console in your build process if you can.

Alejandro Carrazzoni
34 minutes ago
OK thanks

terje
30 minutes ago
Are you ok with me copying this conversation into an issue in the adapter github repo?

Alejandro Carrazzoni
28 minutes ago
Sure it's OK (edited)