New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support `dotnet test` in .NET CLI and .NET Core #1371

Closed
enricosada opened this Issue Mar 23, 2016 · 49 comments

Comments

Projects
None yet
9 participants
@enricosada
Copy link

enricosada commented Mar 23, 2016

We can already run nunit tests with dotnet run ( for example in this chessie commit ) with nunitlite using console app. And works ( forget dnxcore50 atm, should be netstandard )

But for test execution in .NET CLI ( https://github.com/dotnet/cli ), the command is dotnet test

Thedotnet test it's the one who should give support from ides ( ref dotnet/cli#1376 ) about discover/debug/etc

That's the current way forward, dnx is deprecated ( #927 should be closed ) and aspnet and dnx is moving to .NET CLI

A new library dotnet-test-nunit it's required for dotnet test

same as dotnet-test-xunit (ref https://github.com/xunit/coreclr.xunit/pull/1 )

the example project.json for an xunit test project (ref tests inside https://github.com/dotnet/cli ) is

{
    "version": "1.0.0-*",
    "dependencies": {
        "NETStandard.Library": "1.5.0-rc2-23911",
        "xunit": "2.1.0",
        "dotnet-test-xunit": "1.0.0-dev-91790-12"
     },
     "frameworks": {
         "netstandardapp1.5": { }
     },
     "testRunner": "xunit"
}

the testRunner property is read by dotnet test ( ref source ) and use dotnet-test-{testRunner}

/cc @piotrpMSFT @livarcocc for info because their work on .NET CLI and dotnet-test, dotnet-test-xunit because more info are nice 😄

I can help with that, if it's ok for you and @piotrpMSFT say go ( it's stable atm i think, but better be sure ) 😄

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Mar 23, 2016

We were waiting for .NET Core to settle down a bit before we moved forward with #927, that is why it hasn't been updated with a new title and information. I was planning on waiting for any announcements that may come out at //build before moving forward, but we would appreciate any help we can get. My plan is to start providing more complete support for more platforms with the 3.4 release this summer including .NET Core, UWP and Xamarin.

You have provided a well written summary here, so I am going to close #927 and use this issue for tracking going forward. If you want to help with this issue, then we can coordinate, but I would welcome your help.

The first step in this process was creating a PCL driver/agent that can load and execute tests without being tightly bound to a specific version of the NUnit Framework like NUnitLite and the current Xamarin runner. It is still in it's infancy, but the work in progress is in the nunit.portable.agent project.

Any help or advice that @piotrpMSFT or @livarcocc can give would also be appreciated 😄

@rprouse rprouse added this to the 3.4 milestone Mar 23, 2016

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Mar 23, 2016

@rprouse I have a bug on the CLI to write a better documentation regarding the interactions and requirements between dotnet test and the runner. I will get it done today or tomorrow and I will put a link to it here.

This is the bug tracking it: dotnet/cli#1803.

@rprouse rprouse modified the milestones: 3.4, 3.2.1 Apr 12, 2016

@ah-

This comment has been minimized.

Copy link

ah- commented Apr 23, 2016

How is this going? I would love to use this, and the dotnet cli ecosystem seems to be stabilizing.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Apr 23, 2016

I am making some progress, but I don't want to do too much until RC2 drops. I am getting sick of trying to keep up with the changes 😄

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented May 17, 2016

.NET Core RC2 has been released and new info is available. Links for reference.

@NickCraver

This comment has been minimized.

Copy link

NickCraver commented May 17, 2016

Can I help any here? This is a blocker to complete some of our ports like StackExchange.Redis.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented May 17, 2016

@NickCraver I could definitely use some help, that first read of the communication protocol didn't make things terribly clear 😄

First steps are deciding how we are going to approach this. We currently have the portable build that targets .NET Core, but it is limiting. I am considering creating a core build of the framework, but am unsure if I need to or should.

Then I need to look at what it takes to create a test extension for the dotnet command.

Lastly, I would like the nunit engine to be able to launch and communicate with a core nunit-agent so that nunit3-console can run .NET Core unit tests. I've been working on this, but still have a lot of work to do there.

How would you like to help?

@NickCraver

This comment has been minimized.

Copy link

NickCraver commented May 17, 2016

@rprouse I think you'll have to have a core build to possibly be used cross-platform, that's a critical feature.

I'm not sure how much time we can put into this vs. investing elsewhere though to be honest, given the amount of work left. For now, I need to get libraries out the door for many downstream people waiting to build their libraries and apps. I hate to say this, but given the lag here vs. what's available today, I think we'll have to finish porting everything over to xUnit, otherwise many hundreds to thousands of developers are waiting on a blocked chain of releases here.

I'll try and re-visit this after we're stable and time allows, but (and correct me if I'm wrong) at the moment we're talking about weeks of work left (going by your post) before I'd be able to use this to run what I hope is finished library ports ready to ship. From an author duty standpoint, I need to unblock people on me first, then help upstream as I can.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented May 18, 2016

@NickCraver rather than spend the effort porting to xUnit, it might be easier to create a command line NUnitLite runner to run your .NET Core tests. It isn't a perfect solution, but possibly the least amount of work. It is a bit out of date, but I blogged about how to do it at Testing .NET Core using NUnit 3.

I am doing it that way for a few personal projects and it is functional.

@TheRealPiotrP

This comment has been minimized.

Copy link

TheRealPiotrP commented May 18, 2016

I'd be happy to help with getting a proper dotnet-test-nunit build out there. I'm sure many folks would benefit. Much of the contract defined in the protocol is already implemented in a helper library. If we have a working xplat nunit runner, this should be little work.

@jskeet

This comment has been minimized.

Copy link

jskeet commented May 18, 2016

Would it help to break this into two phases - a sort of "quick and dirty, may have grotty hacks, but at least let's get something out there" release and then a more fully-fledged integration later?

(Just being about to build the tests with a TFM of netstandard1.3 would go a long way to unblocking me for Noda Time, for example.)

@roji

This comment has been minimized.

Copy link

roji commented May 18, 2016

@jskeet and @NickCraver you can use NUnit today via the NUnitlite "hack" as @rprouse suggested, I've been doing this for Npgsql for a while now and it works well. It's not the perfect solution but it should definitely unblock you guys.

@jskeet

This comment has been minimized.

Copy link

jskeet commented May 18, 2016

@roji: That worked with dnxcore50 (I've been using it with Noda Time for a while) - but using netstandard1.3, the dependencies on NUnit and NUnitLite are invalid.

@roji

This comment has been minimized.

Copy link

roji commented May 18, 2016

@jskeet my test project targets netcoreapp1.0 and depends on Nunit+NUnitLite 3.2.1. I have "imports" : [ "dotnet54" ] and it works like a charm...

@jskeet

This comment has been minimized.

Copy link

jskeet commented May 18, 2016

@roji: Thanks for that... fingers crossed...

@jskeet

This comment has been minimized.

Copy link

jskeet commented May 18, 2016

@roji: I'm still having trouble actually running the tests - could you link to where you're doing this in your test project, as an example to follow?

@roji

This comment has been minimized.

Copy link

roji commented May 18, 2016

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented May 18, 2016

@jskeet the imports should work for you too. RC2 changes a few things, so I will rewrite my blog post on how to test with NUnit 3 with RC2 in mind and post the code to GitHub. I will add a link here when it is done, hopefully this morning (EST).

@piotrpMSFT can you save me some time and point me to the helper library for creating dotnet-test-nunit? I can search, but I am chasing several things down at the moment so help would be appreciated. I'm sure you are busy too, so if you don't have it at hand, don't spend time on it, I will find it.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented May 26, 2016

I have added an initial PR for the dotnet-test-nunit runner at nunit/dotnet-test-nunit#1

It explores and runs tests, but still needs some work, cleaning up and testing. I will make the NuGet feed for the build artifacts more accessible if anyone wants to play with it.

If anyone has ideas on how I can debug the NuGet package when it gets loaded by Visual Studio, I could use some help. So far, it isn't running in Visual Studio, but that could be a setup problem.

@jskeet

This comment has been minimized.

Copy link

jskeet commented May 26, 2016

@rprouse: I'd be happy to see how Noda Time copes with it, if that would be useful. Let me know if there's anything in particular you'd like me to check out. Thanks for all your work!

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 1, 2016

It isn't ready for prime time yet, but I have NUnit .NET Core tests running at the console and in Visual Studio. There are still several issues to resolve before I get an alpha out, but we are now much closer as the hard work is done and it is down to the details.

I will post details on how to test with the CI build after I fix a few issues. I would appreciate it if people could kick the tires and help me find problems.

Here is a screenshot.

image

@rprouse rprouse self-assigned this Jun 1, 2016

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 4, 2016

@rprouse: Any progress on the CI build? Definitely keen to kick the tires; Noda Time uses a reasonable number of NUnit features, so it should be a good start, anyway. Really looking forward to being able to run tests in VS again. (Hope NCrunch and CodeRush for Roslyn start supporting it once it's out, too...)

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 5, 2016

@jskeet and others who are interested in kicking the tires and helping me test before I release an alpha, I have updated my demo repository on a dotnet-test-nunit branch to use the new dotnet-test-nunit runner.

Please report issues in the nunit/dotnet-test-nunit repo.

High level instructions are;

Update: dotnet-test-nunit is now available as an alpha on NuGet, select Include prereleases. You no longer need to update your nuget.config file.

dotnet-test-nunit is still under development, so you will need to add a NuGet.Config file to your solution to download NuGet packages from the NUnit CI NuGet feeds.

Your project.json in your test project should look like the following;

project.json

{
    "version": "1.0.0-*",

    "dependencies": {
        "NUnitWithDotNetCoreRC2": "1.0.0-*",
        "NETStandard.Library": "1.5.0-rc2-24027",
        "NUnit": "3.2.1",
        "dotnet-test-nunit": "3.4.0-alpha-1"
    },
    "testRunner": "nunit",

    "frameworks": {
        "netstandard1.5": {
            "imports": [
                "dnxcore50",
                "netcoreapp1.0",
                "portable-net45+win8"
            ]
        }
    },

    "runtimes": {
        "win10-x86": { },
        "win10-x64": { }
    }
}

The lines of interest here are the dependency on dotnet-test-nunit. Feel free to use the newest pre-release version that ends in -CI, that is latest from the master branch. Note that the NUnitWithDotNetCoreRC2 dependency is the project under test.

I have added "testRunner": "nunit" to specify NUnit 3 as the test adapter. I also had to add to the imports for both the test adapter and NUnit to resolve. Lastly, I had to add the runtimes. If anyone can explain why I need to do that, please let me know.

You can now run your tests using the Visual Studio Test Explorer, or by running dotnet test from the command line.

# Restore the NuGet packages
dotnet restore

# Run the unit tests in the current directory
dotnet test

# Run the unit tests in a different directory
dotnet test .\test\NUnitWithDotNetCoreRC2.Test\

Warning

As I said, this is still under development. dotnet-test-nunit version 3.3.0.39-CI listed above has a bug where it will throw an ArgumentException when it tries to save out the TestResult.xml file. That will be fixed shortly. Fixed in 3.3.0.49-CI.

Also note that the dotnet command line swallows blank lines and does not work with color. The NUnit test runner's output is in color, but you won't see it.

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 5, 2016

The "blank line swallowing" part is a known bug: dotnet/cli#2234

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 5, 2016

Thanks for the link @jskeet, it looks like color is also a known bug, dotnet/cli#1977

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 5, 2016

The ArgumentException has been fixed in 3.3.0.49-CI. I will update my instructions above with the fix. One other outstanding issue is that I am not currently supplying the line number for tests to the runner, so clicking on tests in the Visual Studio Test Explorer will not take you to the code.

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 5, 2016

Am I right in saying there's no command line argument support yet, like --where etc? (I fully understand that it's early days - just trying to check whether this is something I should be able to do.)

I've tried a slightly different project.json to you. I've included it below verbatim, but the important differences are:

  • I want to test the desktop runtime as well, so I've got two framework targets
  • I'm using netcoreapp1.0 instead of netstandard1.5
  • I've included the Microsoft.NETCore.App dependency, with "type"="platform"
  • Look ma, no runtime section. (Possibly due to some of the above changes...)
{
  "buildOptions": {
    "keyFile": "../../NodaTime Release.snk",
    "embed": {
      "include":  [   
        "TestData/*"
      ]
    }
  },

  "configurations": {
    "Debug": {
      "buildOptions": {
        "define": [ "DEBUG", "TRACE" ]
      }
    },
    "Release": {
      "buildOptions": {
        "define": [ "RELEASE", "TRACE" ],
        "optimize": true
      }
    }
  },

  "dependencies": {
    "NodaTime": { "target": "project" },
    "NodaTime.Testing": { "target": "project" },
    "NUnit": "3.2.1",
    "dotnet-test-nunit": "3.3.0.49-CI",
    "Microsoft.CSharp": "4.0.1-rc2-24027",
    "System.Dynamic.Runtime": "4.0.11-rc2-24027",
    "System.Reflection.Extensions": "4.0.1-rc2-24027",
    "System.Xml.XDocument": "4.0.11-rc2-24027"
  },

  "testRunner": "nunit",

  "frameworks": {
    "net451": {
      "frameworkAssemblies": {
        "System.Runtime": "",
        "System.Threading.Tasks": "",
        "System.Xml.Linq": ""
      }
    },
    "netcoreapp1.0": {
      "imports" : [ "dnxcore50", "netcoreapp1.0", "portable-net45+win8" ],
      "buildOptions": {
        "define": [ "PCL" ]
      },
      "dependencies": {
        "Microsoft.NETCore.App": { 
          "version": "1.0.0-rc2-3002702",
          "type": "platform"
        },
        "System.Console": "4.0.0-rc2-24027",
      }
    }
  }
}

Result:

Test Count: 15141, Passed: 15141, Failed: 0, Inconclusive: 0, Skipped: 0

Woot. (Using net451 I get a test count of 15646, which is what I'd expect.)

Next up: trying the same thing on a Linux machine (where your project.json presumably wouldn't work without modification as it doesn't mention Linux, but mine should, in theory; having said that, it's an Ubuntu 16.04 machine so its dotnet CLI is somewhat bodged together).

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 5, 2016

Hmm. Tests on Linux hadn't finished after 14 minutes of eating 100% CPU. Will need to look more closely at what's happening there.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 5, 2016

@jskeet

Am I right in saying there's no command line argument support yet, like --where etc?

Poorly tested, but I have wired up --where and most of the common command line parameters. I still have an issue, nunit/dotnet-test-nunit#4 to check that they are all wired up properly and work and add any that may be missing. So far, I haven't tested many, but I have used a few by adding them to the end of the dotnet command. I still need to figure out exactly how it works though. For example, the --debug command line option was working fine from the dotnet-test-nunit solution, but it throws dotnet-test Error: 0 : Microsoft.DotNet.Cli.Utils.CommandUnknownException: No executable found matching command "dotnet-test-" when I run it from my test solution. I also have no idea how to chain the help command line out to ours to show what is supported.

In the meantime, you can take a look at https://github.com/nunit/dotnet-test-nunit/blob/master/src/dotnet-test-nunit/CommandLineOptions.cs to see what command line options I think are supported 😄

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 6, 2016

Ah, yes - it looks like dotnet test doesn't have the same facility as dotnet run for separating arguments to the test framework from arguments to dotnet test itself. I'd been trying

dotnet test -- --help

and

dotnet test -- --where=cat!=Slow

Just using

dotnet test --where=cat!=Slow

works fine. Will file a feature request for that - it's nice to be able to be completely unambiguous.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 6, 2016

Clicking on tests and navigating to the code is now fixed in 3.3.0.60-CI.

There are still two high priority issues, then I am planning on doing an alpha release at which point I will close this issue and track everything in the new repository. The two issues are;

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 6, 2016

Just as an FYI, I ran the Noda Time tests again today on my Ubuntu box, and they completed after about 15 minutes. Looks like they're all running about 5-6x slower on my Linux i5 than on my Win10 i7. Not sure how much that's to do with CPU and how much is to do with JIT though - and it's definitely out of the scope of NUnit :)

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 7, 2016

@jskeet thanks for the update, that is good news. Did you need mono bindings for the Linux tests as indicated in nunit/dotnet-test-nunit#9?

For anyone who is testing, the latest good NuGet package is 3.3.0.69-CI, please test with that. There are now no outstanding issue that I think are serious enough to block a release. If none are reported this week, I will likely create an alpha release later this week or early next week. Once I do that, I will close this issue and future discussion/issues can go on in the dotnet-test-nunit repo.

Test now or forever hold your peace 😄

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 7, 2016

@rprouse: I haven't tried running it on Mono - only .NET Core on Linux. That worked fine with the exact project.json above.

Will give 69-CI a whirl...

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 8, 2016

One difference I've noticed between this and the console app runner is that the "dotnet test" version seems to include the time spent finding tests in the overall duration, whereas the "dotnet run" command I used to use with NUnitLite didn't. In Noda Time, this means if I run with --where=cat==Foo (there are no such tests in that category), "dotnet test" reports a duration of 14s where as "dotnet run" reports a duration of 0.01s - despite them both taking about the same real elapsed time.

Is this deliberate? I certainly don't mind the behaviour, but It may be worth noting it somewhere to avoid people thinking it's actually slower.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 8, 2016

@jskeet that is a good observation, it is indeed the case. Because we don't have the full NUnit engine to run the tests, the code is more simplistic, but I might be able to start the clock on the first test started event rather than wrapping the test run. I will enter an issue.

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 10, 2016

I have released an alpha version of dotnet-test-nunit on GitHub at https://www.nuget.org/packages/dotnet-test-nunit/3.4.0-alpha-1, so you no longer need to change your NuGet.config file and use CI releases.

Now that I have an alpha out, I am going to close this issue. Please help me test the alpha and report any issues in https://github.com/nunit/dotnet-test-nunit/issues. Final release is planned to go out at the end of the month with the NUnit 3.4 release. I will release updated alphas and betas depending on issues found.

@rprouse rprouse closed this Jun 10, 2016

@rprouse rprouse added the closed:done label Jun 10, 2016

@rprouse

This comment has been minimized.

Copy link
Member

rprouse commented Jun 10, 2016

I have updated the documentation in the readme for dotnet-test-nunit. I will keep that up to date.

https://github.com/nunit/dotnet-test-nunit

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 10, 2016

As I noted before, you don't need to specify the runtimes if you use "type": "platform" for the "Microsoft.NETCore.App" dependency. Given the xUnit docs, I think this is the preferred approach.

I currently have a pull request running through Travis - if that goes green, Noda Time will be depending on this, so that should give it some usage. Nicely done, sir...

@jskeet

This comment has been minimized.

Copy link

jskeet commented Jun 10, 2016

One downside I've noticed just now - this means I can no longer easily test with Mono on Linux. I believe that's not an NUnit issue, it's basically a manifestation of dotnet/cli#3073 but for NUnit. I'm still hopeful it might get fixed eventually; for the moment I'm going to disable my mono tests in Travis, with a bit of reluctance. (dotnet test is definitely the future.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment