csproj: document how to properly pack platform-specific native assemblies

[yatli]

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): dotnet.exe

dotnet.exe --version (if appropriate): 2.1.4

OS version (i.e. win10 v1607 (14393.321)): win10 v1709 (16299.248)

Description

We've got rid and tfm specific native assemblies in a package but until very recently I haven't find enough documentation about how to pack them and let the runtime pick the appropriate native library automatically.

Relevant links:

• New csproj additions -- I find the IncludeAssets value "Native" which says "native assemblies" will be copied over. So where are they?
• Nuget MSBuild targets -- traditional folders like build/, tools/, lib/, etc.
• Fragmented information about project.json, mentioning runtimes/win-x64 and so on -- , but how am I supposed to know that this convention is carried over to the new csproj?

I have to look into the Nuget code repo, and here: https://github.com/NuGet/NuGet.Client/blob/023fe7670796a8986bbfdc520029e4cf0a6bbfda/src/NuGet.Core/NuGet.Packaging/ContentModel/ManagedCodeConventions.cs#L452

That's it, now I know it's runtimes/{rid}/native/{any?}. Searching for a concrete example on the internet:
I get issues, not documentation.
Searching for "nuget pack native assemblies" did not work either..
It turns out, the correct information is located here: - Bingo!
However, this page is advertised as "Supporting multiple .NET framework versions", which is really orthogonal to what I want. Multi-targeting is about tfm, and platform-specific stuff are about rid -- I have never found a single page that connects all these dots together.

So I suggest documenting such behavior at the following docs sites:

1. "Additions to the csproj format" -- https://docs.microsoft.com/en-us/dotnet/core/tools/csproj
2. "Package creation workflow" -- https://docs.microsoft.com/en-us/nuget/create-packages/overview-and-workflow -- there's multi-targeting page, and a native packages page, and I think it's better to create a new page "platform-specific native libraries" to avoid confusion.
3. Also, document that runtimes/{rid}/native does not work with netfx -- you have to place native dlls in the lib/ folder (risky, could cause msbuild warnings/errors), or embed the dlls and release them at runtime.

[veikkoeeva]

Also relevant is #3931 and #4837 (comment).

[PatoBeltran]

@karann-msft can you please take a look at this issue?

[PatoBeltran]

@nkolev92 maybe you can provide more context on how this can be solved

[veikkoeeva]

@PatoBeltran To me the greatest problem was the information that is scattered. I also came quickly around the documentation @yatli refers to, but it didn't spark the lightbulb immediately. Partially because I didn't first understand there's a difference with the new reference method and how it's done before that and that one really needs a .target file. It might be useful to point out these distinctions explicitly (and that there's old documentation). Maybe also give instructions to look at obj folder and how do all the elements fit into the whole system. This includes such things as the managed libraries can be found in the global Nuget index and don't need to be copied over, but calling native files, or putting content files/executables/P/Invoke assets (it's maybe important to enumerate hese explicitly for the sake to reduce cognitive stress) one needs to pick them up with a .target file. For reference, #3931 (comment) is the SO issue I opened and linked various things.

[yatli]

@PatoBeltran just like @veikkoeeva mentioned, the information is too fragmented and it is very hard to pull the pieces together. You see, even though I have looked directly at the source code and completely understand how to proceed, I tried to reverse engineer a valid search query for the answer that I already know, and I failed.

[ektrah]

3. Also, document that runtimes/{rid}/native does not work with netfx -- you have to place native dlls in the lib/ folder (risky, could cause msbuild warnings/errors), or embed the dlls and release them at runtime.

runtimes/{rid}/native does work with .NET Framework — under very narrow, undocumented conditions. It's a mystery.

Any kind of official information on this would be highly welcome.

[veikkoeeva]

Cross-referencing #5910. Basically the same issue.

[karann-msft]

cc: @anangaur @kraigb

[Apollo3zehn]

After some testing, it seems that for corefx the RID fallback mechanism works as expected (e.g. win7-x86 -> win-x86 -> win -> any). But for netfx you need to specify the exact same RID in the .csproj that appears in the nuget package under runtimes/{rid}/native. Otherwise the native .dll's are not copied to the output directory. The same happens if no <RuntimeIdentifier>...<RuntimeIdentifier> is specificed at all.

Instead, without RID specified, corefx copies the whole runtimes/* folder and the application finds the correct libraries during runtime. I would expect the same behavior for netfx. But I think this is not possible because there is no deps.json which would point to the available files.

[deinok]

Hi, I'm trying to include MRAA to a NetStandard project.

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">

	<PropertyGroup>
		<TargetFrameworks>netstandard1.1;netstandard2.0</TargetFrameworks>
	</PropertyGroup>

	<ItemGroup>
		<Content CopyToOutputDirectory="PreserveNewest" Include="runtimes/linux-arm/native/libmraa.so" Link="libmraa.so" Pack="true" PackagePath="runtimes/linux-arm/native/libmraa.so" />
		<Content CopyToOutputDirectory="PreserveNewest" Include="runtimes/linux-arm/native/libmraa.so.2" Link="libmraa.so.2" Pack="true" PackagePath="runtimes/linux-arm/native/libmraa.so.2" />
		<Content CopyToOutputDirectory="PreserveNewest" Include="runtimes/linux-arm/native/libmraa.so.2.0.0" Link="libmraa.so.2.0.0" Pack="true" PackagePath="runtimes/linux-arm/native/libmraa.so.2.0.0"/>
		<Content CopyToOutputDirectory="PreserveNewest" Include="runtimes/linux-x64/native/libmraa.so" Link="libmraa.so" Pack="true" PackagePath="runtimes/linux-x64/native/libmraa.so" />
		<Content CopyToOutputDirectory="PreserveNewest" Include="runtimes/linux-x64/native/libmraa.so.2" Link="libmraa.so.2" Pack="true" PackagePath="runtimes/linux-x64/native/libmraa.so.2" />
		<Content CopyToOutputDirectory="PreserveNewest" Include="runtimes/linux-x64/native/libmraa.so.2.0.0" Link="libmraa.so.2.0.0" Pack="true" PackagePath="runtimes/linux-x64/native/libmraa.so.2.0.0"/>
	</ItemGroup>

</Project>

And I'm using this: [DllImport("libmraa")]

If libmraa is installed, all the code works well, but i want to create a Package that include the precompiled mraa and the SO doesn't have libmraa installed it says: File not found.
Did I left something?

[yatli]

@deinok you don't need to copy the assemblies to the output folder. and, maybe try [DllImport("mraa")]?

launching the program with: LD_DEBUG=all dotnet run ... would also help to diagnostic the problem.
also, make sure you're packing the dependencies of mraa -- missing dependencies result in a "file not found" error in the main library, which is a bit confusing to start with.

[deinok]

I will try the LD_DEBUG and try to figure what is happening and report back

[deinok]

Okey, problem was:
libmraa.so system link to libmraa.so.2
libmraa.so.2 system link to libmraa.so.2.0.0

Using [DllImport("mraa")]
Deleting libmraa.so and libmraa.so.2 and changing the name of libmraa.so.2.0.0 to libmraa.so works.
So, seems like system links dont work when including native code to a package

[yatli]

@deinok maybe too late but I still want to share the info -- the DllImport name guessing heuristic doesn't like multiple dots in a filename.

Say you have "foo.dll" -- [DllImport("foo")] -- OK!
Say you have "foo.C.dll" -- [DllImport("foo.C")] -- BAD! (Trying to find foo.C literally)

[rossng]

This is sorely needed. I'm trying to figure out how to get C++/CLI projects to work smoothly with PackageReference and it's a nightmare. We have some existing nupkgs where native assemblies are included in the lib/native folder. PackageReference doesn't pull these into the output folder, even though the documentation seems to suggest that they would be by default. I ended up having to remove build from the PrivateAssets, which makes no sense at all to me. It used to work fine with packages.config.

How should I be packaging native and C++/CLI assemblies? How am I meant to reference them from other projects? What if there is a chain of package dependencies sitting between the package containing the native assembly and the consuming project? Are we meant to build 'fat' packages, or is this even supported? What happens in situations where your projects are using a mixture of PackageReference and packages.config (the real world!)? None of these things are documented anywhere.

[Apollo3zehn]

Just for reference, a working solution for simple multi-platform support is shown here. With <GeneratePackageOnBuild>true</GeneratePackageOnBuild>, the native libs are included in the package that is generated each build.

The libraries are then simply invoked via DllImport, with EcShared.NATIVE_DLL_NAME defined here. The platform specific dll names are generated automatically, i.e. although the DLL name is defined as "soem_wrapper", the runtime searches for libsoem_wrapper.so on Linux and soem_wrapper.dll on Windows with the restriction that multiple dots in the name are not working and that I don't know if netfx is already behaving the same as corefx.

@rossng, I guess once one of the packages containing the native libs is referenced in your project (e.g. via <PackageReference>), you can simply use the DllImport attribute as shown in the referenced link (to hopefully answer the chaining question).

Regarding the actual problem: From your post it is not clear if you are simply building the project or if you are publishing it. If you are building it, no libs are copied but referenced from the central Nuget folder (%userprofile%\.nuget\packages). If you are publishing it, depending on your chosen target platform, the right native assemblies are copied to the output/publishing folder (at least on corefx).

[rossng]

Do you know how this works with C++/CLI? I suppose this isn't technically a native lib question, though it's similar in the sense that you are bundling some non-C# assemblies into a package - let me know if it's better to ask elsewhere.

The reason I'm asking this is that I have some (non-shareable) packages which include both native DLLs and C++/CLI assemblies. I want to be able to use PackageReference when depending on these packages, but at the moment the default behaviour doesn't seem to pull in the DLLs, and I don't know why. Maybe it's because they're packaged wrongly in the first place, maybe the PackageReference has to include some extra incantation. The core problem is that none of it is documented - so, even if I do get it to work, I have no idea if it's because that's the supported way or if I was just lucky.

There's some automagic that allows you to use classes defined in a C++/CLI project just by adding a ProjectReference and copying the C++/CLI output DLL to the output directory. So I have a vcxproj called CLRPackage and a csproj called CLRPackageWrapper which contains something like this:

<ItemGroup>
  <ProjectReference Include="..\CLRPackage\CLRPackage.vcxproj">
  </ProjectReference>
</ItemGroup>

<ItemGroup>
  <None Include="..\x64\Debug\CLRPackage.dll">
    <PackagePath>native/</PackagePath>
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    <Pack>true</Pack>
  </None>
</ItemGroup>

I'm then publishing this wrapper project to a local repository. Unfortunately the the ProjectReference gets converted to a package reference and this ends up in the wrapper nuspec:

<group targetFramework=".NETFramework4.6.1">
  <dependency id="CLRPackage" version="1.0.0" exclude="Build,Analyzers" />
</group>

But CLRPackage doesn't exist - I'm only publishing the wrapper. This makes it impossible to install the NuGet package. I have no idea whether I'm supposed to publish the vcxproj and wrapper separately or whether there's some flag to disable this behaviour.

My choice of native/ for the PackagePath is arbitrary - I guess this is wrong. It's unclear to me whether a C++/CLI assembly should be in native/, lib/native/, runtimes/ or somewhere else entirely. Because I can't install the package, I haven't yet been able to test which works. Again, I can't find any documentation for this.

[Apollo3zehn]

How are you publishing the C# project? Are you using something like dotnet publish -Configuration=Release ... or are you using the Visual Studio publishing button? (I experienced differences between both).

Or are you simply generating a nuget package on build (which is not 'publishing')? I am a little confused because I don't know where in a publish step a nuspec file is created. Only if you create a nuget package, a nuspec file is generated within the package itself. Or do you mean the nuspec file in the obj folder which is generated with <GeneratePackageOnBuild>true</GeneratePackageOnBuild>?

Referenced projects and their files are put into the publish folder only during a real publish process (e.g. using dotnet publish). If you are simply generating a package (not publishing), project references are converted to package references. This has the big advantage that during development depending projects can be referenced directly within the same solution. And later, when the packages are generated, the project references are converted into package references. So when a user installs the main package, the depending package is installed as well.

The alternative would be to not use project references at all and only use package references during development. But this is no fun if you update the depending projects: You need to upload package, wait for it to be published (15 mins or so), then install the new version in the main project. Also this prevents usage of nuget in combination with CI pipelines like AppVeyor because the packages would get different version numbers.

But back to your problem: I have no idea how to get your CLRPackage files into the CLRPackageWrapper package. My understanding of the nuget system is that either you dotnet publish your main project, which is typically done for executable projects, and all files end up in a publish folder. Or you create two packages (CLRPackage and CLRPackageWrapper) and upload both.

Edit: Like I did in the project referenced in my previous post. I have a SOEM.PInvoke project that simply P/Invokes into a native DLL and I have an EtherCAT.NET project that depends on SOEM.PInvoke. Both packages are uploaded simultaneously to NuGet (see the dependency chain here).

[rossng]

Apologies, yes - I mean generating a NuGet package on build, and the nuspec inside the generated nupkg. I don't use dotnet publish at all, so I'm not familiar with it.

I'll experiment with packaging the C++/CLI and the C# wrapper separately.

[yatli]

I thought I have graduated from this issue...
However, to publish (for example, a self-contained application) is different from to build a nuget package.
When I use dotnet publish --self-contained, the <Content><PackagePath>...</PackagePath></Content> property is ignored and it won't be copied over.
The workaround is to make a separate project to hold the native binaries and then reference the generated nuget package (!)

[Wes-Kuegler]

This question is still plaguing me. Is there any good documentation on the topic?

[kblok]

@Wes-Kuegler I subscribed to this issue to get updates. I wouldn't consider a doc, but I made some progress shipping platform-specific binaries on PlaywrightSharp. I shared what I learned here https://www.hardkoded.com/blog/playwright-sharp-monthly-dec-2020

[bgavrilMS]

Folks, do you have any updates on this? There are some solutions here and there, but they mostly focus on later versions of .NET. We are an SDK (Microsoft.Identity.Client) and need support for all .NET Fx 4.6.1+ .NET Core 3.1 + and NetStandard. The NET Fx seems problematic.

[BrannonKing]

To add what I learned this week:

It's possible to target net47/8 and netstandard2.0 with native DLLs in a single project without using a nuspec file. I did have to include a targets file for the net47 portion. The csproj file contains this line to achieve that:

<None Include="NLoptNet.targets" Pack="true" PackagePath="build/net47">...

in addition to all the native dlls being copied to the runtimes folder. Notice the use of "build/" there, which thing seems to be undocumented. And the targets file includes lines that look like this:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup Condition="$([MSBuild]::IsOSPlatform('Windows'))">
    <None Include="$(MSBuildThisFileDirectory)..\..\runtimes\win-x64\native\nlopt.dll" Condition="'$(Platform)'!='x86'">
      <Link>nlopt_x64.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
...

Hence, there is only a single copy of each DLL embedded in the package. I'm unsure about the use of MSBuildThisFileDirectory and wondering if there is a better way to get the folder where the package is installed.

I don't have a good solution for unit testing the package project, but this kind of code in a static constructor can help on the far end:

AssemblyLoadContext.Default.ResolvingUnmanagedDll += (assembly, name) =>
{
	var rid = RuntimeInformation.RuntimeIdentifier;
	rid = Regex.Replace(rid, @"\d+-", @"-");  // drop version specificity
	var ext = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? "dll" : "so";
	NativeLibrary.TryLoad($"runtimes/{rid}/native/{name}.{ext}", out var handle);
	return handle;
};

I'm not sure what the equivalent of that is in a pre-net6 world. It looks like .net6 can load DLLs with alternative names, which would be really useful in the ol' NetFramework 4.x.

[flier268]

For .NET 8
editing code from:

<ItemGroup>
    <None Update="A.dll">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
  </ItemGroup>

to:

<ItemGroup>
    <Content Include="A.dll">
        <Pack>true</Pack>
        <PackagePath>lib\$(TargetFramework)</PackagePath>
    </Content>
</ItemGroup>  

[zivkan]

We now have some docs for this: https://learn.microsoft.com/nuget/create-packages/native-files-in-net-packages

Given the title of this issue, and the first post, I'm closing this issue as complete.

@flier268 your suggestion doesn't work for native libraries. If you put A.dll in lib/$(TargetFramework), then NuGet will tell the .NET SDK to pass it to the C#/VB/F# compiler as a managed assembly, and then the compiler will complain that it can't find .NET metadata in the file. Plus NuGet only selects *.dll files from lib/<tfm>/ directories, so it won't work for Linux or Mac (or Android, iOS, etc) libraries.

Please see the docs linked for more info.