Dependency on System.Runtime.InteropServices.RuntimeInformation causes runtime error for apps targeting net451 on *nix systems (i.e. running on Mono)

[JunTaoLuo]

Scenario

1. Running a net451 app depending on System.Runtime.InteropServices.RuntimeInformation on OSX/Ubuntu will produce the following exception at runtime:

 dotnet -v run -f net451                                                                                                                                                                               1 ↵
Telemetry is: Enabled
Project RuntimeServiceOnMono (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Running /usr/local/bin/mono --debug /Users/jtluo/Documents/workspace/juntaoluo/tp/RuntimeServiceOnMono/bin/Debug/net451/osx.10.11-x64/RuntimeServiceOnMono.exe
Process ID: 29130

Unhandled Exception:
System.DllNotFoundException: System.Native
  at (wrapper managed-to-native) Interop/Sys:GetUnixNamePrivate ()
  at Interop+Sys.GetUnixName () [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform (OSPlatform osPlatform) [0x00000] in <filename unknown>:0
  at ConsoleApplication.Program.Main (System.String[] args) [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.DllNotFoundException: System.Native
  at (wrapper managed-to-native) Interop/Sys:GetUnixNamePrivate ()
  at Interop+Sys.GetUnixName () [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform (OSPlatform osPlatform) [0x00000] in <filename unknown>:0
  at ConsoleApplication.Program.Main (System.String[] args) [0x00000] in <filename unknown>:0

Example

See repro at https://github.com/JunTaoLuo/RuntimeServiceOnMono. To run the sample, run dotnet restore and dotnet run -f net451

I understand Mono is not a scenario that's actively being developed for but we should not be causing a exception like this especially since we intend to use the InteropServices APIs in often used ASP.NET packages like Logging.

Currently this means that most of our samples and apps cannot run on Mono (full CLR on *nix).

Environment

dotnet --version:

Microsoft .NET Core Shared Framework Host

  Version  : 1.0.1-rc3-004312-00
  Build    : 1db6c07638a70a621b312e78d4dc9fb31a530f2f

mono -V:

Mono JIT compiler version 4.0.5 ((detached/1d8d582 Tue Oct 20 15:15:33 EDT 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS:           normal
    SIGSEGV:       altstack
    Notification:  kqueue
    Architecture:  x86
    Disabled:      none
    Misc:          softdebug
    LLVM:          yes(3.6.0svn-mono-(detached/a173357)
    GC:            sgen

cc @BrennanConroy @muratg @Eilon

[joshfree]

Version : 1.0.1-rc3-004312-00

@JunTaoLuo could you upgrade to CoreFX rc3-24128-00 / Microsoft.NetCore.App 1.0.0-rc3-004338 / CLI 1.0.0-preview2-002911 and report back?

[mellinoe]

I suspect this will still be happening on the latest packages. BTW, it looks like the issue is with System.Runtime.InteropServices.RuntimeInformation, rather than System.Runtime.InteropServices. We have a net45 implementation for Windows, but not for Unix. Unix probably always picks up the netstandard1.1 version, which won't necessarily work on Mono. I guess it would work if System.Native was deployed with the app, but that probably won't happen by default.

[muratg]

cc @Eilon @DamianEdwards @davidfowl

[ericstj]

Updated the title and assignee. This isn't for Interop, it's RuntimeInformation. I would expect some different results using the latest packages, since those will probably result in the desktop implementation being chosen.

Not much we can do in packaging here since Mono pretends to be desktop and doesn't use a RID.

We could make the desktop library try to p-inoke to some *nix APIs (directly, without shim) if it didn't find the Windows ones. Not sure how folks feel about that.

[JunTaoLuo]

I still see the same error with the latest packages. System.Runtime.InteropServices.RuntimeInformation/4.0.0-rc3-24128-00. CLI version: 1.0.0-preview2-002913. Note that the repro only targets net451 and does not depend on Microsoft.NetCore.App.

[joshfree]

@ellismg is going to take a look

[ellismg]

https://github.com/dotnet/cli/issues/3350 is a CLI problem that is causing a unix asset to be deployed instead of the Windows one. Even if we deploy the correct asset, we still end up with p/invokes into kernel32 (which won't work) but we'd at least be able to provide some light-up mono implementation via late-binding.

[sunsided]

btw., same on mono 4.5.2.372 on Ubuntu 14.04, targeting net452 with 1.0.0 packages from NuGet. I see it on the console logger though (.AddConsole()).

Is there some workaround we can use?

[borgdylan]

I confirm that by using a separately built System.Native, I could get the RuntimeInformation class to work on Mono on both x86 and x64 Ubuntu 16.04. It is a matter of shipping those libraries as source/binaries for the architectures that mono supports.

[borgdylan]

@ericstj The .NET CLI includes a RuntimeInformation class that detects Windows vs. *nix cases and P/Invokes straight thru to the native syscalls. I confirm that, that code cross-compiles and works fine unmodified on Linux/Mono: https://github.com/dotnet/cli/tree/rel/1.0.0/src/Microsoft.DotNet.InternalAbstractions/Native

[ericstj]

@borgdylan that's already understood. As @ellismg mentioned we need to decide how we want to do that since mono represents itself as .NET desktop WRT to NuGet.

[borgdylan]

The way the CLI internal abstractions work would be best. That code works ok irrespective of CLR and treats operating systems directly. The alternative would be to make mono use parts of System.Native and friends as part of its codebase so that APIs new in corefx and not present in mono would just work. That would open up prospects of using mono's core parts i.e its CLR as a CoreCLR stand-in on platforms where the CoreCLR does not run in order to run apps that target corefx. Example platforms are Linux x86, LInux on MIPS, Linux on PowerPC.

[borgdylan]

The latter case would need cooperation from the Xamarin team.

[ericstj]

code works ok irrespective of CLR

Yep that's what we're thinking to do for this particular case. We'll make the desktop implementation of this contract "work" on Mono and give some meaningful results without relying on p-invokes to a native shim we cannot deploy. I suspect we'll do some reflection lightup to detect it's mono then call some mono APIs that provide similar information. That's what @ellismg meant by late binding.

I don't think we want to have the native shims become something that's part of Mono. They are too brittle.

[borgdylan]

@ericstj The reflection based search for Mono.Runtime where finding it means Mono and not finding it means .NET CLR has been done for years without issues. I consider this as the Litmus test for Mono.

[ericstj]

Yep, the other thing we need is an implementation of this for Mono, ideally without any PInvokes. Perhaps we can translate mono's Environment.OSVersion if that's reliable.

[borgdylan]

First of all due to how full framework behaves, Mono will not distinguish between Linux and Mac in the Environment APIs. Both will be reported as Unix.

[borgdylan]

A per before, you can distinguish 32 and 64 bit but not whether it is ARM/Intel/MIPS/PowerPC.

[borgdylan]

The OS version will be the Linux kernel version. (in the Linux case)

[borgdylan]

Therefore this cannot be done using pure Mono/.NET without P/Invokes. There is a reason why System.Native was used for CoreCLR. The only way to make this API run perfect is to have native implementation libraries on all three CLRs or to have all three CLRs ship a runtime info file or use an RID.

[borgdylan]

The code in: https://github.com/dotnet/cli/blob/rel/1.0.0/src/Microsoft.DotNet.InternalAbstractions/Native/PlatformApis.cs solves part of the problem but requires IO and P/Invokes.

[borgdylan]

The code in: https://github.com/dotnet/cli/blob/rel/1.0.0/src/Microsoft.DotNet.InternalAbstractions/RuntimeEnvironment.cs reiterates the fact that you cannot distinguish Intel/ARM/MIPS/PowerPc. Mono supports a host of architectures and you cannot get it right without the System.Native port to all those architectures.

[borgdylan]

@ericstj Do remember that Mono means Xamarin which means MSFT. I do see a world where .NET and mono teams collaborate to have a subset of System.Native be present in Mono and be compatible with the CoreCLR version.

[ericstj]

Do remember that Mono means Xamarin which means MSFT.

@borgdylan I'm not arguing here, nor do I disagree with anything that's been said around mono support. I am sorry if that's how this is coming off, but that's the last thing on my mind. I want to support mono and I want to do so with minimal coupling. I'm just stating the constraints we have and discussing possible design choices.

you cannot get it right without the System.Native port to all those architectures.

Shims don't solve the processor architecture problem, they just push it to NuGet. The shims hardcode the architecture value and NuGet uses RID to give us the shim with the right hardcoded value. I'd like to keep the goal of not having mono projects require a RID.

Which I understand is why you are suggesting:

System.Native be present in Mono

/cc @stephentoub @jkotas @davidfowl
The shims are not public surface area. They version quite frequently and are required to do so in order to support new API. If we put them in mono it means mono pins itself to a specific shim version that must be target-able by upstack packages. We don't have any way to represent that today. It means we'd have to 'know' that net46 means a mono version with shim version X and if a new implementation required version X+1 of that shim it couldn't be supported on net46 due to mono's pinning, even though that has nothing to do with desktop framework. I really don't want to go down this rabbit hole if it can be avoided.

The main reason we have shims was to avoid many forks of the IL per-distro/version, initially we were doing all the translation in IL and directly PInvoking to the *nix OS. I suspect there is a way to do so here, we just need someone more knowledgeable in *nix than myself to do a deep dive. I think @ellismg already has an idea here so I'd like to wait for his feedback, especially since this issue is assigned to him, before we go further down the path of trying to design this.

[borgdylan]

I'll try once I arrive at work. It's 7:30 am here.

[borgdylan]

So the PR got accepted. I will make a new branch with the standalone build based on the PR and @jkotas 's feedback on the current version.

[borgdylan]

@jkotas @Tragetaschen @ericstj The new standalone stuff will be done here: https://github.com/borgdylan/corefx/commits/sysnative_standalone2

[borgdylan]

With my latest round of testing I can conclude that the library built by the standalone and normal build systems lead to the same behaviour of the S.R.IS.RI API under mono.

[zgramana]

For anyone looking for a workaround (on macOS/Darwin at least), here are some notes on getting things working on OS X (and in theory they should work on Ubuntu with little or no modification, but I haven't tested):

1. Find your loadable native images (.dylib on macOS/Darwin, .so on Linux). The location depends on what your dev/packaging platform is.

   1. If you're building and running on the same platform, you can likely find them at /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0.
   2. If you're cross-compiling, or want to be more generalized, you can find them for each installed platform using this command:

   ls ~/.nuget/packages/runtime.*.*.*.runtime.native.System/1.0.1*/runtimes/*/native
   

2. Make your native image files loadable. The loader function, dlopen, is going to probe the filesystem for a file prefixed with lib. Even if it knows where to find your images, without the prefix, dlopen will not find them. You have to give dlopen a little help by choosing one of methods below.

   Note: If you want to watch this process happen, just prefix the dotnet command like this:

     MONO_LOG_LEVEL=debug dotnet run
   

   You'll see all the probing going one, like:

   Mono: DllImport error loading library '/path/to/your/src/project/bin/Debug/net451/osx.10.11-x64/libSystem.Native.dylib': 'dlopen(/path/to/your/src/project/bin/Debug/net451/osx.10.11-x64/libSystem.Native.dylib, 9): image not found'.
   

   1. Rename/Copy/Symlink from System.*.{dylib|so} to libSystem.*.{dylib|so}.
   2. Map the expected name to the actual name. This adds an rpath that lets dlopen see lib{FOO} even though the filesystem only has {FOO}:

   sudo install_name_tool -add_rpath /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0/libSystem.Native.dylib /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0/System.Native.dylib
   

   Notes: On Linux, looks like you'd probably use patchelf instead. For either option, you can automate this process for all your files with a little awk fu:

   sudo ls /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0/System*.dylib | awk '{split($1,arr,"/");name=arr[length(arr)];i=index($1, name);path=substr($1, 0, i - 1);newName=path "lib" name;cmd="sudo install_name_tool -add_rpath " newName " "  $1;cmd | getline; print}'
   

3. [Only if you chose 2.i] Put your native images in dlopen's search path. You have a few choices, and depends on what your ultimate goals are.

   1. Copy the respective libSystem.Native.{dylib|so} files from Step 2 to bin/{config}/{framework}/{platform}/libSystem.Native.{dylib|so}. Nothing fancy/brute force. Do this for each project. If someone does a clean clone of a repo, or you're doing stuff in CI, you should really add a pre-compile task to get this done. Be mindful of any tooling/automation that might periodically rm -Rf bin/Debug/*.
   2. Set LD_LIBRARY_PATH to one or more paths containing your native images. Normally on macOS/Darwin this would be DYLD_LIBRARY_PATH and it's ilk (see man DYLD_LIBRARY_PATH for the details), but seems Mono fine with either one. I don't know if they ever plan to change that, so YMMV.

Tip: Make sure you are running mono in 64-bit mode, otherwise it won't be able to load the native images. Since there isn't a way to pass this flag via dotnet CLI, you use the ENV variable instead. This command will add it to your shell startup script:

   echo export MONO_ENV_OPTIONS=--arch=64 >> ~/.zshrc` # or '~.bashrc' if you use bash)

Notes

Regardless of your choice in Step 3, only libuv.{dylib|so} will make into the published folder.
For files like libSystem.Native you'll have to copy that into the publish folder yourself...unless you already installed the .NET Core on the machine you're deploying too. In that case, you could wrap your executable in a simple one-line shell script that looked something like: LD_LIBRARY_PATH=/usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.0.0 mono bin/Debug/net451/osx.10.11-x64/my-app.exe. If you were to symlink Microsoft.NETCore.App/Current to Microsoft.NETCore.App/1.0.0 you could reasonably easily update without touching your script.
I'm not entirely sure what the best practice/intent is concerning /usr/local/share/dotnet paths versus ~/.nuget paths.

When the above doesn't work, I've also found it helpful to run brew update && brew upgrade and if you get errors and warnings, run brew doctor to see what needs to get cleaned up. You might also need to run brew prune to fix broken symlinks in /usr/local before brew upgrade will succeed.

[sunsided]

Btw., for the case of the missing System.Native library, aspnet/KestrelHttpServer#963 (comment) provides a workaround. I managed to get net4xx targets working with that on mono 4.7 using the 3131 tooling.

[AlexGhiondea]

@jkotas @ericstj @Priya91 what are the next steps for this?

[DustinCampbell]

What's the plan to address this? We just discovered that this package slipped into the desktop C# compiler causing it to no longer run on Mono.

cc @jaredpar and @agocke

[jkotas]

@DustinCampbell Desktop != Mono. The desktop C# compiler package is meant to run on desktop. If you run app build for desktop on Mono, your mileage will vary. That's by design.

[agocke]

@jkotas I thought the point of "netstandard" was that it would allow people to build different runtimes targeting the same standard.

If Mono is an implementation of "net46" how can it not be expected to be compatible with netstandard libraries compiled against "net46"?

[davidfowl]

That's true but this package should be made to work on mono. @emgarten do we have a mono TFM as yet? If not, we can just do some runtime light up.

[jkotas]

mono TFM

Yes, mono TFM is the proper "correct by construction" solution for this problem.

[DustinCampbell]

agreed -- for many things. The Microsoft.Build.Runtime package would benefit greatly from that TFM as well

[agocke]

Maybe for some things. But that abandons the notion that Mono is an alternate implementation of some/most of net46 -- which is its entire purpose today.

If you abandon that design, you are leaving behind almost all of Mono's consumers.

[emgarten]

do we have a mono TFM as yet?

there is no mono TFM in NuGet currently

[jkotas]

Mono has played catch up game to make the cross-platform subset of desktop APIs work. If you want to take advantage of it for your app, you have to be careful to only depend on what it has caught up with. It has not caught up with System.Runtime.InteropServices.RuntimeInformation yet. There is a proposed PR to fix that: mono/mono#4234. Once it goes through, the particular problem with S.R.InteropServices.RuntimeInformation will be solved.

[akoeplinger]

@jkotas

It has not caught up with System.Runtime.InteropServices.RuntimeInformation yet.

That's not true, we've been shipping an implementation for that in Mono 4.6. The issue is that the System.Runtime.InteropServices.RuntimeInformation nuget package contains an implementation in the lib\net45 folder which will be copied to the app directory and since it calls into Windows APIs it'll crash on other platforms.

Mono prefers loading an assembly from the app directory, that's why the GAC version is not getting used. The PR you mentioned above works around that problem by blacklisting assemblies which we know to be broken on Mono, which is hardly a perfect solution. We're discussing the larger issue at https://github.com/dotnet/corefx/issues/15112 and should probably continue the discussion there.

[jkotas]

Agree. I should have been more specific - Mono has not caught up yet with how System.Runtime.InteropServices.RuntimeInformation and similar OOB packages are deployed in desktop apps.

[davidfowl]

@akoeplinger what version of mono is going to fix this?

[akoeplinger]

@davidfowl we shipped a "fix" with Mono 4.8 which essentially blacklists some specific assemblies+versions like SRI.RuntimeInformation: https://github.com/mono/mono/blob/mono-4.8.0-branch/mono/metadata/image.c#L1035-L1091

A more future-proof solution is currently being discussed in dotnet/corefx#15112

[Daniel15]

For what it's worth, even with Mono 5.4 I still have to symlink /usr/local/bin/libSystem.Native.so to System.Native.so in order to get ASP.NET Core 2.0 sites to load, otherwise I get the System.DllNotFoundException: System.Native exception.