Get codebase tests working with ReSharper and Rider #3008

jnm2 · 2018-09-07T03:09:45Z

Somewhere along the line we stopped being able to run tests on ReSharper. Test Explorer is the only way to run IDE tests right now. Rider is another IDE I want to get working.

DaiMichael · 2018-09-23T10:51:13Z

@jnm2 - Interestingly... this seems to be working since I upgraded to point to the latest NUnitVSTestAdapter (v3.10.0)

I don't have Rider to test that but I'm able to select a framework and debug in R#.

jnm2 · 2018-09-23T22:53:57Z

Good! I tried in Rider 2018.2.3:

22 GetArgsFromFiles tests fail in nunitlite.tests on each .NET Framework build with assertion failures like this one:

NUnitLite.Tests.CommandLineTests.GetArgsFromFiles("--arg1 @file1.txt --arg2 @file2.txt","file1.txt:--filearg1 --filearg2,file2.txt:--filearg3","--arg1","--filearg1","--filearg2","--arg2","--filearg3")

  Expected is <System.String[5]>, actual is <System.String[2]>
  Values differ at index [1]
  Expected string length 10 but was 6. Strings differ at index 2.
  Expected: "--filearg1"
  But was:  "--arg2"
  -------------^

nunit.framework.tests run painfully slowly in Rider. It looks like it might even take an hour. It's reporting 69 failures but I haven't figured out how to see them; when I filter to errors, everything says 'Running' or 'Pending' and I can't filter those out.

Sebazzz · 2018-10-05T17:08:28Z

ReSharper uses its own way of detecting NUnit tests. Might that be the issue?

asendimitrov · 2018-10-17T11:18:00Z

Not sure if this is meant to be raised as a separate issue, but my team recently discovered we can't run tests via ReSharper too (obviously not NUnit codebase ones). Running a test after rebuilding the solution works, but every other attempt afterwards fails in ReSharper with test being marked as inconclusive. Looking at the error output, it seemed that ReSharper is trying to find the tests .dll in its own directory rather than the default one (bin\Debug). We tried the suggestion by @DaiMichael but that didn't solve it.

It turned out to be an issue with NUnit 3.11.0. We're currently using NUnit 3.10.1 which works fine with ReSharper.

jnm2 · 2018-10-17T18:55:36Z

@asendimitrov Last I heard, ReSharper relies on the VSTest adapter. Do you have the latest NUnit3TestAdapter package referenced in your test csproj? If not, would you please open a new issue with a repro? I know that ReSharper works with NUnit 3.11.0 in many projects, so something unique is happening with your project.

asendimitrov · 2018-10-18T08:43:23Z

Hi @jnm2 , thanks for you response.

I don't think we've ever needed to use NUnit3TestAdapter in any of our solutions. We have solutions using SpecFlow (which might be one reason) but we also have ones which use pure NUnit tests and they run fine without NUnit3TestAdapter.

Nevertheless, I've added the latest NUnit3TestAdapter (v 3.10.0) in a solution using SpecFlow and a solution not using it, both of these have NUnit 3.11.0 installed. Both have the same issue marking tests as inconclusive on every run after the first.

I don't think NUnit3TestAdapter affects R# in any way unfortunately. If you have other suggestions, I'm happy to keep digging (kinda want to make sure it's on my end before opening a separate issue).

P.S. The R# error message I mentioned in my previous post is the following:
2018.10.18 11:31:35.403 ERROR Cannot run tests: File not found: C:\Users\asend\AppData\Local\JetBrains\Installations\ReSharperPlatformVs15_c21f0cb1\ExampleTests.dll

jnm2 · 2018-10-18T11:22:25Z

Without seeing the repro projects, no ideas come to mind except that you might have a VSIX NUnit adapter overriding the project-level adapter?

asendimitrov · 2018-10-19T14:02:50Z

Nope, I don't have anything installed. I just tried with a brand new solution:

using NUnit.Framework;

namespace NunitGitRepro
{
    public class Class1
    {
        [Test]
        public void Test()
        {
            Assert.Pass("Test passed");
        }
    }
}

This is my packages.config:

<packages>
  <package id="NUnit" version="3.11.0" targetFramework="net461" />
</packages>

After the 1st run, the test is marked as inconclusive with the error i mentioned above.

VS version: 15.8.7
ReSharper version: 2017.3.3

jnm2 · 2018-10-19T15:51:22Z

And if you install NUnit3TestAdapter in your packages.config (current version 3.10.0), ReSharper doesn't start working? (They've told us they use this test adapter.)

mikkelbu · 2018-10-19T23:21:28Z

As far as I can tell Reshaper/Rider bundles the runners and do not make usage of the adapter

From https://www.jetbrains.com/help/rider/Reference__Options__Tools__Unit_Testing__NUnit.html

Enable NUnit support
Depending on these settings, JetBrains Rider will use bundled NUnit test runners: v. 2.6.4 and/or v. 3.6.1 correspondingly.

and
https://www.jetbrains.com/help/resharper/Third_Party_Software_Ultimate.html

NUnit 2.6.4 runtime for ReSharper test runner | 2.6.408
...
NUnit 3.6 runtime for ReSharper test runner | 3.6.3

@asendimitrov What if you set "Custom NUnit Installation" in the Resharper NUnit settings to your installation of the console does that change anything? (the reference notes that for this "JetBrains Rider 2018.2 supports NUnit 3.10.1 and above.")

asendimitrov · 2018-10-22T08:28:17Z

Sorry @jnm2 , I should've really shared that as a result in the first place... But yes installing Nunit3TestAdapter doesn't do anything to help in this case:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="NUnit" version="3.11.0" targetFramework="net461" />
  <package id="NUnit3TestAdapter" version="3.10.0" targetFramework="net461" />
</packages>

@mikkelbu That didn't help either, although I think that even if it did it wouldn't be a solution to the issue as this is about ReSharper running the tests without needing anything additional (as it it does with NUnit 3.10.1)

jnm2 · 2018-10-24T19:35:18Z

@mikkelbu Ah, looks like they only use our adapter for .NET Core tests: nunit/nunit3-vs-adapter#487 (comment), nunit/nunit3-vs-adapter#487 (comment). Thanks!

jnm2 · 2018-10-24T19:46:15Z

@asendimitrov I'm using VS 15.8.7 and ReSharper 2018.2.3. I think I followed your steps exactly and I got this:

Here's the entire project I created: NunitGitRepro.zip

Does it fail for you? If so, I'd try updating ReSharper or go to JetBrains for support since I can't think of anything else. If not, can you make it fail and send it back?

mikkelbu · 2018-10-25T09:36:51Z

@asendimitrov @jnm2 I also tried to repoduce the problem, and I was successful VS 15.8.5 and ReSharper 2017.3.5. I tried to look into the errors and warnings and also got "Cannot run tests: File not found" in the resharper GUI.

From https://resharper-support.jetbrains.com/hc/en-us/community/posts/360000436150-PipeException-The-pipe-endpoint-net-pipe-machine-TestExecutor-port-could-not-be-found-on-your-local-machine-while-debugging-mstest-projectI got the inspiration to run devenv /ReSharper.LogFile C:\resharper_log.txt /ReSharper.LogLevel Verbose, this gave me a log with several exceptions thrown, but I've not been able to use this information to anything (yet).

I also tried some of the suggestions mentioned in the thread, but that did not solve the problem.

I'll see if I have time to look more into it tonight (and e.g. try out the NunitGitRepro.zip)

Additional links:

asendimitrov · 2018-10-25T12:07:04Z

@jnm2 I'm getting the same issue with your project. Here's mine if you wanna give it a try:
asendimitrov_nunitgitrepro.zip

Only difference is the existence of NUnit3TestAdapter as you advised.

@mikkelbu I've done some research on the jetbrains site as well, seems like there may be different reasons to why ReSharper behaves like that. By raising this here I just wanted to make sure it's not an NUnit issue itself that causes this, as the older framework version actually works fine which was a smell to me.

Thanks both for looking into this!

rprouse · 2018-12-02T21:53:57Z

@NikolayPianikov can you help with this issue or know someone who can?

NikolayPianikov · 2018-12-03T08:52:38Z

@rprouse I asked for comments

FrankRay78 · 2021-05-31T16:03:31Z

Although nearly 2 years later, I'm still getting this same issue.

VS2019 with .Net 5 installed.
Cloned the nunit repo as of 31 May 2021.
Can build the solution both in the IDE and also from the command line.

Right click on the AssertEqualsTests class --> "Run Unit Tests". Result:

However, open windows terminal and navigate to "bin\Debug\net5.0".
Run "nunitlite-runner.exe nunit.framework.tests.dll". Result:

jnm2 added is:build pri:normal labels Sep 7, 2018

jnm2 mentioned this issue Sep 7, 2018

Write up recommended way to test and link it from the readme #2973

Closed

jwg4 mentioned this issue Nov 8, 2018

Rerunning tests in Resharper without rebuilding gives Inconclusive #3086

Closed

FrankRay78 mentioned this issue Jun 1, 2021

nunit Framework tests do not run with "dotnet test" nor inside VS2019 (Windows) #3867

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get codebase tests working with ReSharper and Rider #3008

Get codebase tests working with ReSharper and Rider #3008

jnm2 commented Sep 7, 2018

DaiMichael commented Sep 23, 2018

jnm2 commented Sep 23, 2018 •

edited

Loading

Sebazzz commented Oct 5, 2018

asendimitrov commented Oct 17, 2018 •

edited

Loading

jnm2 commented Oct 17, 2018

asendimitrov commented Oct 18, 2018 •

edited

Loading

jnm2 commented Oct 18, 2018

asendimitrov commented Oct 19, 2018 •

edited

Loading

jnm2 commented Oct 19, 2018 •

edited

Loading

mikkelbu commented Oct 19, 2018

asendimitrov commented Oct 22, 2018 •

edited

Loading

jnm2 commented Oct 24, 2018

jnm2 commented Oct 24, 2018 •

edited

Loading

mikkelbu commented Oct 25, 2018

asendimitrov commented Oct 25, 2018 •

edited

Loading

rprouse commented Dec 2, 2018

NikolayPianikov commented Dec 3, 2018

FrankRay78 commented May 31, 2021

Get codebase tests working with ReSharper and Rider #3008

Get codebase tests working with ReSharper and Rider #3008

Comments

jnm2 commented Sep 7, 2018

DaiMichael commented Sep 23, 2018

jnm2 commented Sep 23, 2018 • edited Loading

Sebazzz commented Oct 5, 2018

asendimitrov commented Oct 17, 2018 • edited Loading

jnm2 commented Oct 17, 2018

asendimitrov commented Oct 18, 2018 • edited Loading

jnm2 commented Oct 18, 2018

asendimitrov commented Oct 19, 2018 • edited Loading

jnm2 commented Oct 19, 2018 • edited Loading

mikkelbu commented Oct 19, 2018

asendimitrov commented Oct 22, 2018 • edited Loading

jnm2 commented Oct 24, 2018

jnm2 commented Oct 24, 2018 • edited Loading

mikkelbu commented Oct 25, 2018

asendimitrov commented Oct 25, 2018 • edited Loading

rprouse commented Dec 2, 2018

NikolayPianikov commented Dec 3, 2018

FrankRay78 commented May 31, 2021

jnm2 commented Sep 23, 2018 •

edited

Loading

asendimitrov commented Oct 17, 2018 •

edited

Loading

asendimitrov commented Oct 18, 2018 •

edited

Loading

asendimitrov commented Oct 19, 2018 •

edited

Loading

jnm2 commented Oct 19, 2018 •

edited

Loading

asendimitrov commented Oct 22, 2018 •

edited

Loading

jnm2 commented Oct 24, 2018 •

edited

Loading

asendimitrov commented Oct 25, 2018 •

edited

Loading