Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotnet xunit fails for .NET Core 2.0.3 with .NET Core SDK 2.0.3 #1573

Closed
axelheer opened this issue Nov 19, 2017 · 30 comments
Closed

dotnet xunit fails for .NET Core 2.0.3 with .NET Core SDK 2.0.3 #1573

axelheer opened this issue Nov 19, 2017 · 30 comments

Comments

@axelheer
Copy link

@axelheer axelheer commented Nov 19, 2017

Since upgrading to the .NET Core SDK 2.0.3 dotnet xunit is unable to resolve the target framework.

C:\...\xunit.integration\dotnet-xunit> dotnet xunit -framework netcoreapp2.0
Detecting target frameworks in v2x_MultiTarget_DotNetSdk.csproj...
Building for framework netcoreapp2.0...
  v2x_MultiTarget_DotNetSdk -> C:\...\xunit.integration\dotnet-xunit\bin\Debug\netcoreapp2.0\v2x_MultiTarget_DotNetSdk.dll
Running .NET Core 2.0 tests for framework netcoreapp2.0...
The specified framework version '2.0' could not be parsed
The specified framework 'Microsoft.NETCore.App', version '2.0' was not found.
  - Check application dependencies and target a framework version installed at:
      \
  - Alternatively, install the framework version '2.0'.

Specifying the exact framework version helps:

C:\...\xunit.integration\dotnet-xunit> dotnet xunit -framework netcoreapp2.0 -fxversion 2.0.3
Detecting target frameworks in v2x_MultiTarget_DotNetSdk.csproj...
Building for framework netcoreapp2.0...
  v2x_MultiTarget_DotNetSdk -> C:\...\xunit.integration\dotnet-xunit\bin\Debug\netcoreapp2.0\v2x_MultiTarget_DotNetSdk.dll
Running .NET Core 2.0.3 tests for framework netcoreapp2.0...
xUnit.net Console Runner (64-bit .NET Core 4.6.25815.02)
  Discovering: v2x_MultiTarget_DotNetSdk
  Discovered:  v2x_MultiTarget_DotNetSdk
  Starting:    v2x_MultiTarget_DotNetSdk
    BasicTests.Theory(value: null) [FAIL]
      Assert.NotNull() Failure
      Stack Trace:
        C:\...\xunit.integration\dotnet-xunit\BasicTests.cs(23,0): at BasicTests.Theory(Object value)
    BasicTests.Failing [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        C:\...\xunit.integration\dotnet-xunit\BasicTests.cs(14,0): at BasicTests.Failing()
  Finished:    v2x_MultiTarget_DotNetSdk
=== TEST EXECUTION SUMMARY ===
   v2x_MultiTarget_DotNetSdk  Total: 5, Errors: 0, Failed: 2, Skipped: 0, Time: 0.156s

Maybe because the updated DefaultItems.targets file doesn't specify the "full" version number...

@bradwilson

This comment has been minimized.

Copy link
Member

@bradwilson bradwilson commented Nov 19, 2017

This is not a bug.

Your code is being built against 2.0.0 even though you don't have it installed (that's legal, since compilation happens against NuGet packages, not against the installed runtime). We run against exactly what you're compiled against, on purpose. If you want to compile against 2.0.0, you need to install 2.0.0. If you want to compile against 2.0.3, then set the property:

<RuntimeFrameworkVersion>2.0.3</RuntimeFrameworkVersion>

You could reasonably ask why the .NET Core team shipped 2.0.3 while leaving the compiler still targeting 2.0.0 even though you don't have it installed. That's a question for them, not us.

@bradwilson bradwilson closed this Nov 19, 2017
@bradwilson

This comment has been minimized.

Copy link
Member

@bradwilson bradwilson commented Nov 19, 2017

For those who are looking to find out what runtimes you have installed, they are under the shared/Microsoft.NETCore.App folder where you installed dotnet.

Windows:

Windows

Linux / OS X / WSL:

Linux

@nguerrera

This comment has been minimized.

Copy link
Contributor

@nguerrera nguerrera commented Nov 20, 2017

You could reasonably ask why the .NET Core team shipped 2.0.3 while leaving the compiler still targeting 2.0.0 even though you don't have it installed. That's a question for them, not us.

One of them here. ;)

A framework-dependent app compiled against 2.0.0 will automatically roll forward to run on latest 2.0.x instead. The default runtime version of 2.0.0 for tfm netcoreapp2.0 allows your 2.0 app to run in hosted environments that may or may not have latest patch applied. In the past, we bumped the default versions and people were broken when they recompiled and republished their app to a hoster that hadn't patched yet.

Ideally, dotnet xunit would get the same roll forward behavior as normal app activation via runtimeconfig.json. I'm guessing you're using --fx-version, which will not roll forward in the same way. We've encountered the same issue with dotnet ef. Cc @livarcocc who was working with the EF folks on addressing that.

@bradwilson

This comment has been minimized.

Copy link
Member

@bradwilson bradwilson commented Nov 20, 2017

Ideally, dotnet xunit would get the same roll forward behavior as normal app activation via runtimeconfig.json.

Actually, I don't believe that xUnit should have this behavior. As stated above, I believe it's preferable to run against exactly what you're compiled against (or exactly what you request, via command line); otherwise you're not sure whether your code runs correctly against the target runtime.

It would be reasonable for a developer to want to test against multiple target frameworks (which is why we allow them to specify it), and we wouldn't want a silent roll-forward to give a developer a false sense of security about the actual compatibility of the code they've written.

@llehn

This comment has been minimized.

Copy link

@llehn llehn commented Nov 20, 2017

I think Brad is right, and I think the way the runtime rolls the version forward for apps is a good thing too. However this creates a developer experience problem. Having to pass a non-constant version string (non constant in a way it will change to 2.0.4 and so on...) is a deal breaker for scripting. On most dev machines on my team the logic that xunit runs agains the same version as it was compiled is handled implicitly: there is only one version installed. I think xunit users would appreciate if we could have a nicely working solution (instead of a "workaround"-feeling).

@adamralph

This comment has been minimized.

Copy link
Contributor

@adamralph adamralph commented Nov 20, 2017

It appears that I have 2.0 installed:

image

image

But I still get:

image

This problem started after I "upgraded" VS to 15.4.4 yesterday (I'm afraid I don't know what version I had before).

@bording

This comment has been minimized.

Copy link

@bording bording commented Nov 20, 2017

@adamralph You're seeing that because of a CLI bug: dotnet/cli#7901 (comment)

However, the suggested workaround that doesn't lock you into a specific runtime version doesn't work for xunit, and @bradwilson has said here that he doesn't want it to work.

While I can see his reasoning, I also think that there should be a way to opt-in to the roll forward behavior if we want it.

@bradwilson

This comment has been minimized.

Copy link
Member

@bradwilson bradwilson commented Nov 20, 2017

I am 100% against fixing Microsoft's bug with a work around that would re-enable unpredictability. The fact that a user asking for 2.0 instead of 2.0.0 and getting a 100% failure rate seems like broken behavior on their end, because they are the ones who forced 2.0 instead of 2.0.0 on the user in the first place without fully grasping that they would be breaking 3rd party software.

I will add a hard-coded workaround to convert x.y into x.y.0 but I want to be clear that Microsoft inflicted this bug on you (and me), not xUnit.net.

@livarcocc

This comment has been minimized.

Copy link

@livarcocc livarcocc commented Nov 20, 2017

If you don't want to be rolled forward, which seems to be the case here and I was not aware of this before. Then yeah, using --fx-version like xunit is doing is the right approach. And, doing what EF is doing does not address the issue, as you would get rolled forward.

@bradwilson we didn't decide to change it to 2.0 without considering third parties. It is a bug that it says 2.0 instead of 2.0.0 and we are addressing it for the next release of the .NET Core SDK (2.2.0).

@bradwilson

This comment has been minimized.

Copy link
Member

@bradwilson bradwilson commented Nov 20, 2017

@livarcocc I'm still forced to fix it on the assumption that 2.0.3 will be used for quite a while.

@adamralph

This comment has been minimized.

Copy link
Contributor

@adamralph adamralph commented Nov 21, 2017

Thanks to everyone for the info. For now, I've worked around it by adding

<RuntimeFrameworkVersion>2.0.0</RuntimeFrameworkVersion>
thatfrankdev pushed a commit to spektrummedia/Spk.Common that referenced this issue Nov 30, 2017
Francis Cote
As advised in this thread : xunit/xunit#1573
onovotny added a commit to dotnet/reactive that referenced this issue Nov 30, 2017
304NotModified added a commit to NLog/NLog.Web that referenced this issue Dec 1, 2017
Fix xUnit
304NotModified added a commit to NLog/NLog.Web that referenced this issue Dec 1, 2017
304NotModified added a commit to NLog/NLog.Web that referenced this issue Dec 1, 2017
ap0llo added a commit to ap0llo/synctool that referenced this issue Dec 6, 2017
Use 'dotnet test' instead of 'dotnet xunit' for test execution
to work around dotnet cli bug.
See xunit/xunit#1573
@CumpsD

This comment has been minimized.

Copy link

@CumpsD CumpsD commented Dec 8, 2017

Just got bitten by this again, had completely forgotten about it, until I remembered to add <RuntimeFrameworkVersion>2.0.3</RuntimeFrameworkVersion> to our test .csproj, I'm sure I'll get bitten by it again once we move to a next .NET Core version and forget all about that csproj setting :)

@skalinets

This comment has been minimized.

Copy link

@skalinets skalinets commented Feb 13, 2018

Seems like this fix dotnet/reactive@cd17630 is the best one

@dataspun

This comment has been minimized.

Copy link

@dataspun dataspun commented Feb 18, 2018

@skalinets in VS Code adding <RuntimeFrameworkVersion>2.0.0</RuntimeFrameworkVersion> with your referenced fix works like a charm

@hnafar

This comment has been minimized.

Copy link

@hnafar hnafar commented Feb 26, 2018

When building inside a container (i.e. only one version is installed) in order to avoid hard-coding the runtime version in the project file or scripts, I just get the version at runtime:

dotnet xunit -fxversion $(ls /usr/share/dotnet/shared/Microsoft.NETCore.App/)

equivalent windows folder is C:\Program Files\dotnet\shared\Microsoft.NETCore.App

@hughbiquitous

This comment has been minimized.

Copy link

@hughbiquitous hughbiquitous commented Mar 5, 2018

Inspired by @hnafar, I'm using dotnet --info with some sed & awk mischief to solve the problem:

dotnet xunit -fxversion $(dotnet --info | sed -n '/Microsoft .NET Core Shared Framework Host/,$p' | grep Version | awk '{print $3}')
@AArnott

This comment has been minimized.

Copy link
Contributor

@AArnott AArnott commented Mar 10, 2018

@nguerrera Where can we get .NET Core 1.0.5 runtime nowadays? All the old links seem to have been removed from the distributing web sites.

@AArnott

This comment has been minimized.

Copy link
Contributor

@AArnott AArnott commented Mar 10, 2018

FWIW, it feels like there ought to be a command like dotnet install runtime -v 1.0.5 or something. If I know I have some dotnet runtime, it ought to be able to acquire a different version for me, ala NPM. Then our test automation could simply ensure that we have all the runtimes installed that we expect to test against rather than delicately depending on the test agent having the runtime already installed.

@nguerrera

This comment has been minimized.

Copy link
Contributor

@nguerrera nguerrera commented Mar 11, 2018

Re FWIW, it's on the backlog and something we definitely hope to do.

@nguerrera

This comment has been minimized.

Copy link
Contributor

@nguerrera nguerrera commented Mar 11, 2018

You can also use the dotnet-install script to install specific versions. See https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-install-script

@AArnott

This comment has been minimized.

Copy link
Contributor

@AArnott AArnott commented Mar 11, 2018

@bradwilson, isn't this part of the error message a bug?

The specified framework 'Microsoft.NETCore.App', version '2.0' was not found.
  - Check application dependencies and target a framework version installed at:
      \
  - Alternatively, install the framework version '2.0'.

Surely "installed at \" isn't helpful, is it?

@bradwilson

This comment has been minimized.

Copy link
Member

@bradwilson bradwilson commented Mar 11, 2018

That's not our error message. That comes from dotnet.exe.

@nguerrera

This comment has been minimized.

Copy link
Contributor

@nguerrera nguerrera commented Mar 11, 2018

It does look like a bug in the host. File at https://github.com/dotnet/core-setup

@AArnott

This comment has been minimized.

Copy link
Contributor

@AArnott AArnott commented Mar 11, 2018

Thanks. Filed dotnet/core-setup#3815

MichaelSimons added a commit to dotnet/dotnet-docker that referenced this issue Apr 2, 2018
MichaelSimons added a commit to MichaelSimons/dotnet-docker that referenced this issue Apr 2, 2018
MichaelSimons added a commit to MichaelSimons/dotnet-docker that referenced this issue Apr 2, 2018
JanLenoch pushed a commit to Kentico/kontent-management-sdk-net that referenced this issue Apr 13, 2018
@alsami

This comment has been minimized.

Copy link

@alsami alsami commented Apr 14, 2018

Hey folks,

what If I am having a netcoreapp2.0 project but CLI and runtime Version 2.1.4 installed?
I can't set the runtime version to 2.1.4. Setting it to 2.0.0 also doesn't work.

On windows this didn't seem to be a problem but on linux (ubuntu 16.04) it does. Any help would be appreciated.

@bording

This comment has been minimized.

Copy link

@bording bording commented Apr 14, 2018

@Birdperson90 2.14 isn't a runtime version, it's an SDK version. If you have SDK 2.1.4 installed, then you have runtime 2.0.5 installed, and that's the version you need to use in the RuntimeFrameworkVersion property.

See https://github.com/dotnet/core/blob/master/release-notes/2.0/README.md to see a list of recent SDK/runtime pairs.

FYI It looks like that README hasn't been updated for the 102, 103, and 104 SDKs, but they all include the 2.0.6 runtime.

@alsami

This comment has been minimized.

Copy link

@alsami alsami commented Apr 15, 2018

@bording thanks for the clarification! It worked :)

@bloritsch

This comment has been minimized.

Copy link

@bloritsch bloritsch commented Aug 22, 2018

So it looks like the dotnet xunit command line requires System.Text.Encoding.CodePages as a dependency. I had to add that to my test package to get the tests to run properly--in addition to specifying a specific framework version.

@adamralph

This comment has been minimized.

Copy link
Contributor

@adamralph adamralph commented Aug 23, 2018

@bloritsch in case you weren't aware, dotnet xunit was discontinued in the 2.4 release:

Unfortunately, this release also removes the dotnet xunit runner, as the stability of the runner was never perfect, and it suffered from many assembly-loading related issues. Users from .NET Core can continue to use VSTest (either inside Visual Studio, or via dotnet test).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.