New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement VSTest support to enable running and debugging .NET Core tests in .csproj projects #1100

Closed
aochsner opened this Issue Dec 30, 2016 · 65 comments

Comments

Projects
None yet
@aochsner

aochsner commented Dec 30, 2016

I realize I'm on pretty bleeding edge (.NET Core 1.1.0, .NET CLI 1.0.0 preview 4 (csproj), etc) but wanted to raise this and see if I can help w/ anything.

Environment data

.NET Core version

Microsoft .NET Core Shared Framework Host

  Version  : 1.1.0
  Build    : 928f77c4bc3f49d892459992fb6e1d5542cb5e86

dotnet --info output:

.NET Command Line Tools (1.0.0-preview4-004233)

Product Information:
 Version:            1.0.0-preview4-004233
 Commit SHA-1 hash:  8cec61c6f7

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.11-x64
 Base Path:   /usr/local/share/dotnet/sdk/1.0.0-preview4-004233

VS Code version:
Version 1.8.1 (1.8.1)

C# Extension version:
1.6.2

Steps to reproduce

Create new xunit project
dotnet new -t xunittest

Update references to 1.1.0 in csproj:

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp1.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="**\*.cs" />
    <EmbeddedResource Include="**\*.resx" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NETCore.App" Version="1.1.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20161123-03" />
    <PackageReference Include="xunit" Version="2.2.0-beta4-build3444" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-beta4-build1194" />
  </ItemGroup>

</Project>

Open in VS Code and navigate to Test.cs.
Run Test via Omnisharp

Expected behavior

Test runs

Actual behavior

"Failed to run test because null" error

@vcsjones

This comment has been minimized.

Show comment
Hide comment
@vcsjones

vcsjones Jan 25, 2017

This is the only thing holding me back from adopting the preview4 tooling. While "run test" is a less severe of a problem (I can just do that from the command line...) "debug test" has the same problem, and I don't have a good work around for that.

vcsjones commented Jan 25, 2017

This is the only thing holding me back from adopting the preview4 tooling. While "run test" is a less severe of a problem (I can just do that from the command line...) "debug test" has the same problem, and I don't have a good work around for that.

@VladimirMakaev

This comment has been minimized.

Show comment
Hide comment
@VladimirMakaev

VladimirMakaev commented Jan 26, 2017

+1

@DustinCampbell DustinCampbell added the Bug label Jan 27, 2017

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Jan 27, 2017

Collaborator

Upon fixing the null reference (which was due to some ugly code that looked specifically for the presence of project.json), it looks like this might be quite seriously broken. Previously, OmniSharp just called dotnet test with --port and --parentProcessId flags to start communicating with the test framework. However, now that the .NET CLI has moved over to MSBuild (and VSTest), it's not clear whether this is possible any longer.

@piotrpMSFT, do you have any thoughts here? I'm guessing OmniSharp will need to commnicate a bit differently with VSTest using the details from https://github.com/microsoft/vstest. Do we just need to use dotnet vstest?

Collaborator

DustinCampbell commented Jan 27, 2017

Upon fixing the null reference (which was due to some ugly code that looked specifically for the presence of project.json), it looks like this might be quite seriously broken. Previously, OmniSharp just called dotnet test with --port and --parentProcessId flags to start communicating with the test framework. However, now that the .NET CLI has moved over to MSBuild (and VSTest), it's not clear whether this is possible any longer.

@piotrpMSFT, do you have any thoughts here? I'm guessing OmniSharp will need to commnicate a bit differently with VSTest using the details from https://github.com/microsoft/vstest. Do we just need to use dotnet vstest?

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Jan 27, 2017

Collaborator

OK. I'm making some progress with dotnet vstest using the spec at https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0007-Editors-API-Specification.md. 😄

Collaborator

DustinCampbell commented Jan 27, 2017

OK. I'm making some progress with dotnet vstest using the spec at https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0007-Editors-API-Specification.md. 😄

@jchannon

This comment has been minimized.

Show comment
Hide comment
@jchannon

jchannon Feb 3, 2017

Member

Keep in mind xUnit are looking to make their own command eg/dotnet xunit so not sure if that would change how tests are run/debugged in vscode @bradwilson

Member

jchannon commented Feb 3, 2017

Keep in mind xUnit are looking to make their own command eg/dotnet xunit so not sure if that would change how tests are run/debugged in vscode @bradwilson

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Feb 3, 2017

Collaborator

I don't think it would. I think we'd still want to go through the VSTest protocol. Otherwise, we'd be writing custom code for every test framework that's supported.

Collaborator

DustinCampbell commented Feb 3, 2017

I don't think it would. I think we'd still want to go through the VSTest protocol. Otherwise, we'd be writing custom code for every test framework that's supported.

@jchannon

This comment has been minimized.

Show comment
Hide comment
@jchannon

jchannon Feb 3, 2017

Member
Member

jchannon commented Feb 3, 2017

@bradwilson

This comment has been minimized.

Show comment
Hide comment
@bradwilson

bradwilson Feb 3, 2017

I think we'd still want to go through the VSTest protocol.

I agree, you should. dotnet xunit is about providing an equivalent to xunit.console.exe for .NET SDK applications. It is not intended to be automated by third parties.

bradwilson commented Feb 3, 2017

I think we'd still want to go through the VSTest protocol.

I agree, you should. dotnet xunit is about providing an equivalent to xunit.console.exe for .NET SDK applications. It is not intended to be automated by third parties.

@DustinCampbell DustinCampbell added Backlog and removed Bug labels Feb 6, 2017

@DustinCampbell DustinCampbell added this to the 1.8 milestone Feb 6, 2017

@DustinCampbell DustinCampbell changed the title from .NET Core 1.1.0 XUnit: Failed to run test because null to Running and debugging .NET Core tests is broken for .csproj projects Feb 6, 2017

@DustinCampbell DustinCampbell changed the title from Running and debugging .NET Core tests is broken for .csproj projects to Implement VSTest support to enable running and debugging .NET Core tests in .csproj projects Feb 6, 2017

@DustinCampbell DustinCampbell added the Test label Feb 7, 2017

@ErikSchierboom

This comment has been minimized.

Show comment
Hide comment
@ErikSchierboom

ErikSchierboom Feb 27, 2017

Will there be a .NET: Test command added to VS Code, similar to the .NET: Restore Packages command? That would be incredibly useful I feel.

ErikSchierboom commented Feb 27, 2017

Will there be a .NET: Test command added to VS Code, similar to the .NET: Restore Packages command? That would be incredibly useful I feel.

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Feb 27, 2017

Collaborator

Eventually. This just tracks getting the CodeLens indicators for "run test" and "debug test" working for .csproj.

Collaborator

DustinCampbell commented Feb 27, 2017

Eventually. This just tracks getting the CodeLens indicators for "run test" and "debug test" working for .csproj.

@ErikSchierboom

This comment has been minimized.

Show comment
Hide comment
@ErikSchierboom

ErikSchierboom Feb 28, 2017

Ah, I missed that. That would also be brilliant!

ErikSchierboom commented Feb 28, 2017

Ah, I missed that. That would also be brilliant!

@Jonathan34

This comment has been minimized.

Show comment
Hide comment
@Jonathan34

Jonathan34 Mar 7, 2017

ah well, another road blocker with xunit and vscode :(
hope the 1.8.0 is coming very soon to fully support csproj and xunit

Jonathan34 commented Mar 7, 2017

ah well, another road blocker with xunit and vscode :(
hope the 1.8.0 is coming very soon to fully support csproj and xunit

@Jonathan34

This comment has been minimized.

Show comment
Hide comment
@Jonathan34

Jonathan34 Mar 8, 2017

as a workaround: you can edit your tasks.json to add the test tasks you want, bind a shortcut like this:

[{
    "key": "shift+cmd+t",
    "command": "workbench.action.tasks.runTask",
    "when": "editorTextFocus"
}]

and select the test task.
if you want to run all the tests, you can create dependencies using the tasks.json v2.0.0 (cf https://code.visualstudio.com/updates/v1_10#_more-work-on-terminal-runner)

alternatively, switch to vs 2017 :)

Jonathan34 commented Mar 8, 2017

as a workaround: you can edit your tasks.json to add the test tasks you want, bind a shortcut like this:

[{
    "key": "shift+cmd+t",
    "command": "workbench.action.tasks.runTask",
    "when": "editorTextFocus"
}]

and select the test task.
if you want to run all the tests, you can create dependencies using the tasks.json v2.0.0 (cf https://code.visualstudio.com/updates/v1_10#_more-work-on-terminal-runner)

alternatively, switch to vs 2017 :)

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Mar 8, 2017

Collaborator

Great suggestion!

Collaborator

DustinCampbell commented Mar 8, 2017

Great suggestion!

@anass-b

This comment has been minimized.

Show comment
Hide comment
@anass-b

anass-b Mar 8, 2017

@Jonathan34 Could you please clarify more the procedure that you suggested ? Could you start a debugging session for a specific test method using VSCode's tasks.json/launch.json ?

anass-b commented Mar 8, 2017

@Jonathan34 Could you please clarify more the procedure that you suggested ? Could you start a debugging session for a specific test method using VSCode's tasks.json/launch.json ?

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 17, 2017

Collaborator

OK. This should be working in the latest beta: 1.9.0-beta3. Please try it out!

Please use the instructions for Installing Beta Releases.

Collaborator

DustinCampbell commented Apr 17, 2017

OK. This should be working in the latest beta: 1.9.0-beta3. Please try it out!

Please use the instructions for Installing Beta Releases.

@powellcj12

This comment has been minimized.

Show comment
Hide comment
@powellcj12

powellcj12 Apr 18, 2017

@DustinCampbell generally seems to be working for me on Mac (10.12.5 beta)!

When I initially installed the beta extension, I was still seeing various errors about not being able to attach the debugger, sometimes the test would never actually run, and one time even a message from VS Code that it was corrupt which linked me here. After reinstalling the app though (I'm running the 1.12.0-Insider Mac build), haven't run into any of those issues since.

powellcj12 commented Apr 18, 2017

@DustinCampbell generally seems to be working for me on Mac (10.12.5 beta)!

When I initially installed the beta extension, I was still seeing various errors about not being able to attach the debugger, sometimes the test would never actually run, and one time even a message from VS Code that it was corrupt which linked me here. After reinstalling the app though (I'm running the 1.12.0-Insider Mac build), haven't run into any of those issues since.

@achimschrepfer

This comment has been minimized.

Show comment
Hide comment
@achimschrepfer

achimschrepfer Apr 18, 2017

@DustinCampbell seems to work on Mac VSCode 1.11.2 and OmniSharp 1.9.0-beta3 - I had the same experience as @powellcj12 when I first started the debugger. But after then it worked like a charm (without reinstall).

achimschrepfer commented Apr 18, 2017

@DustinCampbell seems to work on Mac VSCode 1.11.2 and OmniSharp 1.9.0-beta3 - I had the same experience as @powellcj12 when I first started the debugger. But after then it worked like a charm (without reinstall).

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 18, 2017

Collaborator

@powellcj12, @achimschrepfer: Did you uninstall your previous version of C# and restart VS Code before installing the new beta? If not, that would explain why the debugger didn't work immediately (sometimes, VS Code likes to run an older version of an extension if it's still on disk).

Collaborator

DustinCampbell commented Apr 18, 2017

@powellcj12, @achimschrepfer: Did you uninstall your previous version of C# and restart VS Code before installing the new beta? If not, that would explain why the debugger didn't work immediately (sometimes, VS Code likes to run an older version of an extension if it's still on disk).

@powellcj12

This comment has been minimized.

Show comment
Hide comment
@powellcj12

powellcj12 Apr 18, 2017

Yes, I uninstalled the previous extension (which was the previous v1.9 beta), VS Code prompted me to reload (which I did), and then installed the new extension.

powellcj12 commented Apr 18, 2017

Yes, I uninstalled the previous extension (which was the previous v1.9 beta), VS Code prompted me to reload (which I did), and then installed the new extension.

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 18, 2017

Collaborator

OK. Just wanted to check.

Collaborator

DustinCampbell commented Apr 18, 2017

OK. Just wanted to check.

@richlander

This comment has been minimized.

Show comment
Hide comment
@richlander

richlander Apr 19, 2017

I wasn't able to get this to work on Windows. I installed the beta VSIX and uninstalled the prod one first. I found that the the extension was doing more than before but not the whole job and not reliably. It sometimes ran the test and sometimes gave me the error message that it currently does with the prod version. I can share my code with you if you'd like to try it (private repo).

richlander commented Apr 19, 2017

I wasn't able to get this to work on Windows. I installed the beta VSIX and uninstalled the prod one first. I found that the the extension was doing more than before but not the whole job and not reliably. It sometimes ran the test and sometimes gave me the error message that it currently does with the prod version. I can share my code with you if you'd like to try it (private repo).

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 19, 2017

Collaborator

Hey @richlander. any chance you'll be in the office tomorrow? If so, maybe we can connect I can take a look at your machine?

Collaborator

DustinCampbell commented Apr 19, 2017

Hey @richlander. any chance you'll be in the office tomorrow? If so, maybe we can connect I can take a look at your machine?

@achimschrepfer

This comment has been minimized.

Show comment
Hide comment
@achimschrepfer

achimschrepfer Apr 19, 2017

Yep, I also uninstalled the previous version of OmniSharp and restarted VSCode as well.

achimschrepfer commented Apr 19, 2017

Yep, I also uninstalled the previous version of OmniSharp and restarted VSCode as well.

@richlander

This comment has been minimized.

Show comment
Hide comment
@richlander

richlander Apr 19, 2017

Interesting. Just tried on my work PC (both "run" and "debug"). Works! Yeahh! Last night, I was using my machine at home. I'll try again tonight. Maybe it needs a harder kick.

Thanks for getting this working!

What do you think on performance? The VS experience (unscientifically) feels significantly faster for running single tests.

richlander commented Apr 19, 2017

Interesting. Just tried on my work PC (both "run" and "debug"). Works! Yeahh! Last night, I was using my machine at home. I'll try again tonight. Maybe it needs a harder kick.

Thanks for getting this working!

What do you think on performance? The VS experience (unscientifically) feels significantly faster for running single tests.

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 19, 2017

Collaborator

I'll try again tonight. Maybe it needs a harder kick.

Check your %UserProfile%.vscode\extensions folder. I've seen situations (though rarely) where VS Code leaves old extensions lying around and they still get loaded.

What do you think on performance? The VS experience (unscientifically) feels significantly faster for running single tests.

Baby steps @richlander. 😄 VS does a significantly amount of caching (and has an entire team devoted) to making the test experience good. Everytime we run a test, we need to shell out to the .NET CLI to build (with no fast up-to-date checks like VS), and shell out to the .NET CLI again to go through the "test discovery" phase and then do the actual running of the test once it's discovered. I have some ideas to improve this in the future, but getting it working is the high-order bit for 1.9.

Collaborator

DustinCampbell commented Apr 19, 2017

I'll try again tonight. Maybe it needs a harder kick.

Check your %UserProfile%.vscode\extensions folder. I've seen situations (though rarely) where VS Code leaves old extensions lying around and they still get loaded.

What do you think on performance? The VS experience (unscientifically) feels significantly faster for running single tests.

Baby steps @richlander. 😄 VS does a significantly amount of caching (and has an entire team devoted) to making the test experience good. Everytime we run a test, we need to shell out to the .NET CLI to build (with no fast up-to-date checks like VS), and shell out to the .NET CLI again to go through the "test discovery" phase and then do the actual running of the test once it's discovered. I have some ideas to improve this in the future, but getting it working is the high-order bit for 1.9.

@vcsjones

This comment has been minimized.

Show comment
Hide comment
@vcsjones

vcsjones Apr 19, 2017

For what its worth, this doesn't feel any slower to me than when this was working with project.json projects.

vcsjones commented Apr 19, 2017

For what its worth, this doesn't feel any slower to me than when this was working with project.json projects.

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 19, 2017

Collaborator

Thanks @vcsjones: It shouldn't, but I'd still like to make it better. 😄

Collaborator

DustinCampbell commented Apr 19, 2017

Thanks @vcsjones: It shouldn't, but I'd still like to make it better. 😄

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 20, 2017

Collaborator

OK. A release candidate for 1.9.0 is built and ready for testing. Once we feel good about this build, we'll push it to the marketplace (hopefully tomorrow -- we'll see! 😄)

Collaborator

DustinCampbell commented Apr 20, 2017

OK. A release candidate for 1.9.0 is built and ready for testing. Once we feel good about this build, we'll push it to the marketplace (hopefully tomorrow -- we'll see! 😄)

@fedoranimus

This comment has been minimized.

Show comment
Hide comment
@fedoranimus

fedoranimus Apr 24, 2017

I'm able to test now using the debugger and the launch tasks generated by VSCode.

However, when using the "debug test" option, I'm still seeing this issue.

Using 1.9.0 version of this plugin.

screenshot 2017-04-24 19 40 38

fedoranimus commented Apr 24, 2017

I'm able to test now using the debugger and the launch tasks generated by VSCode.

However, when using the "debug test" option, I'm still seeing this issue.

Using 1.9.0 version of this plugin.

screenshot 2017-04-24 19 40 38

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 24, 2017

Collaborator

Are you referring to the code lens items being out of order? If so, that's tracked with #936.

Collaborator

DustinCampbell commented Apr 24, 2017

Are you referring to the code lens items being out of order? If so, that's tracked with #936.

@fedoranimus

This comment has been minimized.

Show comment
Hide comment
@fedoranimus

fedoranimus Apr 25, 2017

No, while that is annoying, I'm referring to clicking the CodeLens "debug test" item and then receiving the message: "Failed to run test because null" as described in this issue originally.

The "run test" item works fine.

I'm running 1.9.0 of the omnisharp extension, 1.11.2 of VSCode.

fedoranimus commented Apr 25, 2017

No, while that is annoying, I'm referring to clicking the CodeLens "debug test" item and then receiving the message: "Failed to run test because null" as described in this issue originally.

The "run test" item works fine.

I'm running 1.9.0 of the omnisharp extension, 1.11.2 of VSCode.

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 25, 2017

Collaborator

Could you go ahead and file a new issue?

Collaborator

DustinCampbell commented Apr 25, 2017

Could you go ahead and file a new issue?

@Jonathan34

This comment has been minimized.

Show comment
Hide comment
@Jonathan34

Jonathan34 Apr 25, 2017

works fine with 1.9.0 and vscode 1.12.0 and .net core 1.1
thanks!

Jonathan34 commented Apr 25, 2017

works fine with 1.9.0 and vscode 1.12.0 and .net core 1.1
thanks!

@powellcj12

This comment has been minimized.

Show comment
Hide comment
@powellcj12

powellcj12 Apr 25, 2017

@fedoranimus I saw the same thing temporarily - can't seem to hit it any more, but my initial suspicion was that it was related to the specific .NET runtime I was targeting...

.NET Core 1.1 worked fine. I changed it to target .NET Standard 1.4 which resulted in some errors restoring packages, so I subsequently changed it to .NET Standard 1.5. At this point the 'debug' option in the extension wasn't working. I restored to .NET Core 1.1 and things started working again, but now everything works regardless of if I target .NET Core 1.0 or .NET Standard 1.4-1.6 so not really sure what might be happening. Still, might be worth checking the runtime you're targeting?

powellcj12 commented Apr 25, 2017

@fedoranimus I saw the same thing temporarily - can't seem to hit it any more, but my initial suspicion was that it was related to the specific .NET runtime I was targeting...

.NET Core 1.1 worked fine. I changed it to target .NET Standard 1.4 which resulted in some errors restoring packages, so I subsequently changed it to .NET Standard 1.5. At this point the 'debug' option in the extension wasn't working. I restored to .NET Core 1.1 and things started working again, but now everything works regardless of if I target .NET Core 1.0 or .NET Standard 1.4-1.6 so not really sure what might be happening. Still, might be worth checking the runtime you're targeting?

@fedoranimus

This comment has been minimized.

Show comment
Hide comment
@fedoranimus

fedoranimus Apr 25, 2017

Thanks @powellcj12 I will double check.

If the situation persists, I will open a new issue tonight @DustinCampbell.

fedoranimus commented Apr 25, 2017

Thanks @powellcj12 I will double check.

If the situation persists, I will open a new issue tonight @DustinCampbell.

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell
Collaborator

DustinCampbell commented Apr 25, 2017

Thanks @fedoranimus!

@fedoranimus

This comment has been minimized.

Show comment
Hide comment
@fedoranimus

fedoranimus Apr 25, 2017

#1428 Created to address the new issue

fedoranimus commented Apr 25, 2017

#1428 Created to address the new issue

@jmezach

This comment has been minimized.

Show comment
Hide comment
@jmezach

jmezach Apr 29, 2017

This isn't working yet for MSTest projects. Is that a separate issue, or should that work as well now?

jmezach commented Apr 29, 2017

This isn't working yet for MSTest projects. Is that a separate issue, or should that work as well now?

@DustinCampbell

This comment has been minimized.

Show comment
Hide comment
@DustinCampbell

DustinCampbell Apr 29, 2017

Collaborator

Separate issue please. Nothing has been done to support MSTest yet.

Collaborator

DustinCampbell commented Apr 29, 2017

Separate issue please. Nothing has been done to support MSTest yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment