"Failed to start debugger: null" exception when debugging NUnit

[bernardbr]

Environment data

.NET Command Line Tools (1.0.1)

Product Information:
Version: 1.0.1
Commit SHA-1 hash: 005db40cd1

Runtime Environment:
OS Name: Windows
OS Version: 6.1.7601
OS Platform: Windows
RID: win7-x64
Base Path: C:\Program Files\dotnet\sdk\1.0.1

VS Code version:
Version 1.11.2
Commit 6eaebe3b9c70406d67c97779468c324a7a95db0e
Date 2017-04-13T08:03:11.395Z
Shell 1.4.6
Renderer 53.0.2785.143
Node 6.5.0

C# Extension version: 1.9.0

Steps to reproduce

Unzip the file and try debug!
UnitTest.zip

Expected behavior

Debugs run!

Actual behavior

"Failed to start debugger: null" exception

[DustinCampbell]

We've been specifically waiting until there are NUnit bits that we can support.

[DustinCampbell]

Once NUnit is ready we can add support in OmniSharp. Until then, it likely won't work.

[DustinCampbell]

I tried using this project and am getting the following output coming back from NUnit:

Error initializing RunSettings. Default settings will be used
System.NullReferenceException: Object reference not set to an instance of an object.
   at NUnit.VisualStudio.TestAdapter.AdapterSettings.Load(IDiscoveryContext context)
   at NUnit.VisualStudio.TestAdapter.NUnitTestAdapter.Initialize(IDiscoveryContext context, IMessageLogger messageLogger)
NUnit Adapter 4.0.0.0: Test discovery starting
Exception System.InvalidOperationException, Failed to save random seed.
Nullable object must have a value.
Exception System.NullReferenceException, Exception thrown discovering tests in c:\Projects\UnitTest\bin\Debug\netcoreapp1.1\TestNUnit.dll
Object reference not set to an instance of an object.
NUnit Adapter 4.0.0.0: Test discovery complete

[bernardbr]

Hello @DustinCampbell!
In my case, when I run dotnet test works fine:

Build started, please wait...
Build completed.

Test run for C:\Users\bernardo.dsn.erp\Desktop\UnitTest\bin\Debug\netcoreapp1.1\TestNUnit.dll(.NETCoreApp,Version=v1.1)
Ferramenta de Linha de Comando de Execução de Teste da Microsoft (R) Versão 15.0.0.0
Copyright (c) Microsoft Corporation. Todos os direitos reservados.

Iniciando execução de teste, espere...
NUnit Adapter 4.0.0.0: Test execution started
Running all tests in C:\Users\bernardo.dsn.erp\Desktop\UnitTest\bin\Debug\netcoreapp1.1\TestNUnit.dll
NUnit3TestExecutor converted 1 of 1 NUnit test cases
NUnit Adapter 4.0.0.0: Test execution complete

Total de testes: 1. Aprovados: 1. Com falha: 0. Ignorados: 0.
Execução de Teste Bem-sucedida.
Tempo de execução de teste: 1,7563 Segundos

But the NUnit Adapter is in "alpha" version yet.
I'll wait for the final version of NUnit to try debugging again!
Do you want me to close this issue?

[DustinCampbell]

@bernardbr: I did not run dotnet test. I clicked the 'run test' above the test method in VS Code. I retrieved the output in the instance of OmniSharp I was debugging.

[DustinCampbell]

I get the same output as you do when running dotnet test. However, in order to run and debug tests, OmniSharp is using dotnet vstest.

[bernardbr]

I did it too, and works fine:

Running test UnitTest.TestClass.TestMethod...
Test passed 

But before I ran dotnet restore and dotnet test at the prompt.

Note: I am using Windows

[DustinCampbell]

Sorry for not being clear @bernardbr: You'll get different output than I am because I've built from this PR: OmniSharp/omnisharp-roslyn#833, which increases output during test discovery.

[DustinCampbell]

I don't think the test is actually passing on your machine. If you debug omnisharp-vscode, you'll find that there aren't any test results. It looks like there's a bug in omnisharp-vscode that is writing "Test passed" if no test runs.

[bernardbr]

Ah ok, I understood!

Anyway, I think it would be better to close this problem and wait for a final version of NUnit. And then, try again!

What do you think?

[DustinCampbell]

I think the problem is related to this comment in the PR you linked:

vstest.console.exe

The .NET 4.5 tests can also be run with vstest.console.exe. The .NET Core tests are not running because they are not finding the dependent assemblies (nunit.framework.dll) which are not copied into the bin directory on build. I am not sure if this is expected behaviour. I would think most people would use dotnet test, but it needs looking into.

[bernardbr]

You're right!
I didn't seen this before!
I already mentioned @rprouse here. I will close this issue and wait for the final version!
Thanks for your attention and sorry for my bad english! =)

[DustinCampbell]

Not a problem @bernardbr. Thanks very for posting here. I'm anxious to get NUnit working. xUnit is in good shape, but NUnit is sorely missing.

[bernardbr]

I'm converting a NetFramework4's legacy code to netcore, and all unit tests are written with NUnit.
I really like NUnit.
And as far as I know, XUnit don't have "TestFixtureSetup" and "Setup" at the same time and do a migration to XUnit without this will be a problem to me...

[rprouse]

@DustinCampbell the adapter is working fine in Visual Studio right now, the only issue we are having is calling vstest.console directly from the command line. How does VSCode interface with the test adapter? I would assume that it is slightly different than Visual Studio.

Looking at our code for the callstack you have above, I looks to me that the IDiscoveryContext that is passed into the adapter does not have the RunSettings property set. We can work around that, but I wonder if it is intentional. It seems odd to pass in an interface with it's only property unset.

[DustinCampbell]

@rprouse: We don't actually interface with the test adapter directly. VS Code interfaces with the test adapter through OmniSharp by calling dotnet vstest and communicating over a socket. dotnet vstest runs vstest.console. (code)

After building a test project with dotnet build, we call dotnet with just a few arguments (code). It ends up looking something like this:

dotnet vstest  --Port:{port} --ParentProcessId:{parentProcessId}

We then communicate over the port passed to dotnet vstest using the Discovery and Execution Protocols.

We don't do anything fancy and it works for the xUnit test adapter.

I agree that it's weird that RunSettings not be set, but it appears to be how VS Test works (code). I guess this is just for any user settings that were specified by passing a --Settings flag? (Note that xUnit appears to be resilient to IDiscoveryContext.RunSettings being null (code). However, it's not clear to me that's the actual problem here)

[rprouse]

@DustinCampbell thanks for the info. Just knowing that OmniSharp uses dotnet vstest as opposed to dotnet test and the command line that you use will allow me to debug this and get it working. I hadn't been testing dotnet vstest, just dotnet test and vstest.console.exe directly. Too many ways to run tests through the same adapter? 😉

For those that are tracking this issue, the fix for this may have to wait until alpha2. I want to get alpha1 out as soon as possible to get more people working with it. I will do my best to iterate quickly though.

[DustinCampbell]

Also pinging @singhsarab in case he has any ideas.

[DustinCampbell]

Note; If I call dotnet vstest with the --Diag flag, I get different results from the CLI vs. OmniSharp:

• diag-from-cli.txt
• diag-from-omnisharp.txt

[DustinCampbell]

It feels like OmniSharp needs to be specifying default run settings, but that feels very error prone.

[DustinCampbell]

Maybe it's not so error prone. Simply passing in an empty run settings does the trick. E.g.:

SendMessage(MessageType.TestRunSelectedTestCasesDefaultHost,
    new
    {
        TestCases = testCases,
        RunSettings = @"<RunSettings />"
    });

@rprouse: It does indeed look like test adapters are expected to work in the face of null RunSettings. Any chance of getting that into the new release of the NUniTestAdapter?

[DustinCampbell]

I'm re-opening because there's a change I can make on our side to make this work. It's not a "correct" change, but it'll be progress.

[DustinCampbell]

OK. I've merged the fix in OmniSharp. Now to work on putting a new beta release together...

[rprouse]

@DustinCampbell you are correct that NUnit should handle null, the documentation clearly shows an example with a null value. We've just never had to deal with it before because Visual Studio always passed in a value, even if it was just <RunSettings />. We also never had documentation until VSTest was open sourced and always had to guess at the rules based on what was passed in.

I have fixed this in the NUnit adapter with nunit/nunit3-vs-adapter@d345c72 and the NuGet package 4.0.0-ci-00455-pr-313 and newer should work.