New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helper class for dealing with native shared libraries and function pointers #17135

Closed
mellinoe opened this Issue Mar 15, 2017 · 115 comments

Comments

Projects
None yet
@mellinoe
Contributor

mellinoe commented Mar 15, 2017

Approved Proposal

From #17135 (comment)

public sealed class NativeLibrary
{
    public static bool TryLoad(string name, Assembly caller, DllImportSearchPaths paths, out NativeLibrary result);

    public IntPtr Handle { get; }

    public bool TryGetDelegate<TDelegate>(string name, out TDelegate result) where TDelegate : class;
    public bool TryGetDelegate<TDelegate>(string name, bool exactSpelling, out TDelegate result) where TDelegate : class;
    public bool TryGetSymbolAddress(string symbolName, out IntPtr result);
}

Original proposal

Note: Proposal updated based on discussion: #17135 (comment)

Background

Many popular C-callable shared libraries exist which do not have a consistent name across platforms. Examples include:

  • SDL2
  • OpenAL
  • Vulkan
  • Many more (the above are just ones I've used personally)

CoreCLR does not support any notion of "remappable" PInvokes, a la DllImport in Mono. The name of the shared library must be encoded directly in the DllImport attribute, which makes it impossible to create a single wrapper assembly which works on many platforms. This imposes an unnecessary development and deployment burden on library developers who could otherwise ship a single, simple DLL. Many third-party libraries which rely on PInvoke's only work on Mono platforms because of the DllMap feature. These libraries do not work on .NET Core at all, even if they are otherwise completely compatible with its profile.

The alternative to using [DllImport] is to write logic which manually opens the shared library, discovers function pointers, and converts them to managed delegates. In a sense, this very similar to what we are doing with our "Portable Linux" version of the runtime and native shims. However, this code can be complicated, tedious, and error-prone. This proposal is for a small helper library, with optional cooperation by the runtime, to make this scenarion easier.

Proposed API

namespace System.Runtime.InteropServices
{
    // Exposes functionality for loading native shared libraries and function pointers.
    public class NativeLibrary : IDisposable
    {
        // The default search paths
        public static IEnumerable<string> NativeLibrarySearchDirectories { get; }

        // The operating system handle of the loaded library.
        public IntPtr Handle { get; }

        // Constructs a new NativeLibrary using the platform's default library loader.
        public NativeLibrary(string name);
        public NativeLibrary(string name, DllImportSearchPath paths);
        public static NativeLibrary Open(string name, DllImportSearchPath paths, bool throwOnError);

        // Loads a native function pointer by name.
        public IntPtr LoadFunction(string name);

        // Loads a function whose signature matches the given delegate type's signature.
        // This is roughly equivalent to calling LoadFunction + Marshal.GetDelegateForFunctionPointer
        public T LoadDelegate<T>(string name);

        // Lookup a symbol (not a function) from the specified dynamic library
        public IntPtr SymbolAddress(string name);

        // Frees the native library. Function pointers retrieved from this library will be void.
        public void Dispose();
    }
}

Usage

public static class Sdl2
{
    private static NativeLibrary s_sdl2NativeLib = new NativeLibrary(GetSdl2LibraryName());
    // Determine library name at runtime. This could be influenced by a DllMap-like policy if desired.
    private static string GetSdl2LibraryName();

    private delegate IntPtr SDL_CreateWindow_d(string title, int x, int y, int w, int h, int flags);
    private static SDL_CreateWindow_d SDL_CreateWindow_ptr = s_sdl2NativeLib.LoadFunctionPointer<SDL_CreateWindow_d>("SDL_CreateWindow");
    public static IntPtr CreateWindow(string title, int x, int y, int w, int h, uint flags)
        => SDL_CreateWindow_ptr(title, x, y, w, h, flags);
}

public static void Main()
{
    IntPtr window = Sdl2.CreateWindow("WindowTitle", 100, 100, 960, 540, 0);
}

Open Design Questions

Probing Path Logic

CoreCLR applies particular probing logic when discovering shared libraries listed in a DllImport. Ideally, calling new NativeLibrary("lib") would follow the same probing logic as [DllImport("lib")], so that PInvoke's can be easily converted. This could potentially be solved with some internal or public runtime helper function exposed from System.Private.CoreLib. Alternatively, the constructor for NativeLibrary` could include another parameter which controlled probing logic, allowing someone to plug their own logic in.

Generic delegate types

The simplest way to convert from a native function pointer to a clean managed delegate is through the use of Marshal.GetDelegateForFunctionPointer<T>(IntPtr). Unfortunately, this cannot be used with generic delegates, e.g. Func<string, int, int, int, int, int, IntPtr> in the above example. Given that many native function signatures include pointer types, which cannot be used as generic type arguments, other than as IntPtr, this is not a major problem. However, many cases would benefit from being able to use Action and Func types in the LoadFunction<T> method. We could probably allow this using Reflection.Emit, but it would most likely be ugly, complicated, and slower than if custom delegate types were used. If Marshal.GetDelegateForFunctionPointer<T>(IntPtr) accepted generic delegate types, this feature would greatly benefit.

Disposability and lifetime tracking

The constructor of NativeLibrary involves opening an operating system handle to a native shared library, via LoadLibrary, dlopen, etc. I have proposed that NativeLibrary be disposable, which would involve calling FreeLibrary, dlclose, etc. Is this desirable or useful?

A related issue is how the runtime currently tracks and manages these handles. Should handles opened by NativeLibrary be coordinated and tracked in the same way that other handles (via PInvoke, etc.) are?

Prototype

I have a prototype version of this library implemented here: https://github.com/mellinoe/nativelibraryloader, and I have successfully used the pattern described here in a few projects.

@yizhang82 @janvorli @danmosemsft @conniey

@danmosemsft

This comment has been minimized.

Member

danmosemsft commented Mar 15, 2017

You've eliminated the option of mimicing Mono's solution (DllMaps, as you mention)?
Would this API ultimately work for Mono, making DllMap unnecessary?

@akoeplinger

This comment has been minimized.

Member

akoeplinger commented Mar 15, 2017

Related discussion (which you're probably aware of): dotnet/coreclr#930

@mellinoe

This comment has been minimized.

Contributor

mellinoe commented Mar 15, 2017

@danmosemsft See the link that @akoeplinger shared.

@akoeplinger

This comment has been minimized.

Member

akoeplinger commented Mar 15, 2017

@mellinoe since my thread is pretty old by now, do you know what the current status of MCG (which was proposed as the solution by .NET runtime folks) is?

@mellinoe

This comment has been minimized.

Contributor

mellinoe commented Mar 15, 2017

I don't know. I'd be interested to hear an update from @yizhang82 about it.

@karelz karelz added api-needs-work and removed enhancement labels Mar 15, 2017

@IllidanS4

This comment has been minimized.

IllidanS4 commented Apr 15, 2017

Nobody uses unsafe pointers in extern methods, only IntPtr or managed equivalents. Usage of generic delegates should be allowed in GetDelegateForFunctionPointer, and frankly I don't see what difficulties would implementing it have. On the other hand, I think that the "LoadFunction" doesn't allow for as much marshalling options as the DllImport system can. What if you could specify "instance extern" methods on types inheriting from NativeLibrary, which would be automatically imported from the library with the specified marshalling options?

@mellinoe

This comment has been minimized.

Contributor

mellinoe commented Apr 18, 2017

Nobody uses unsafe pointers in extern methods, only IntPtr or managed equivalents.

It is very common to use unsafe pointers in extern methods, in my experience. I certainly do it in a lot of my libraries, and I've seen it in many libraries from others, as well. Wrapping things into IntPtr's is unnecessarily verbose when you're already doing fundamentally unsafe things. Now, it's also very common to "wrap" the PInvokes into methods will fully-safe signatures, but those call the unsafe version in turn.

What if you could specify "instance extern" methods on types inheriting from NativeLibrary, which would be automatically imported from the library with the specified marshalling options?

That sounds like it would involve a lot of external machinery (IL rewriting), or actual C# language support. Certainly an interesting idea, but probably outside of scope here.

@yizhang82 Do you have any feedback for this proposal?

@akoeplinger

This comment has been minimized.

Member

akoeplinger commented Apr 18, 2017

Another problem with the proposed API is that calling dlopen() with arbitrary native libraries is prohibited on iOS and recent Androids so this wouldn't work there (i.e. unusable by Xamarin).

@mellinoe

This comment has been minimized.

Contributor

mellinoe commented Apr 18, 2017

Another problem with the proposed API is that calling dlopen() with arbitrary native libraries is prohibited on iOS and recent Androids so this wouldn't work there (i.e. unusable by Xamarin).

That is a good point, and it may also be the case for UWP applications.

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented Nov 12, 2017

Just my 2 cents on this:

  • On operating systems where you cannot load a native library, this could throw a PlatformNotSupportedException, right? (Although I believe that at least on iOS, you can load native libraries which are part of your application bundle.)
  • I like the idea to keep a dll-map like functionality separate from this API. It makes sense to me that this is a lowel-level API which is close to dlopen/LoadLibrary where you can pass an explicit, full path to the library you want to load. Nothing prevents you from implementing dllmap-like functionality in a separate API, which you can use to resolve the path to the library you want to load
  • I'd suggest adding a static property NativeLibrarySearchDirectories which exposes AppContext.GetData("NATIVE_DLL_SEARCH_DIRECTORIES"), hence:
namespace System.Runtime.InteropServices
{
    // Exposes functionality for loading native shared libraries and function pointers.
    public class NativeLibrary : IDisposable
    {
+       // The default search paths
+       public string[] NativeLibrarySearchDirectories { get; }

        // The operating system handle of the loaded library.
        public IntPtr Handle { get; }

        // Constructs a new NativeLibrary using the platform's default library loader.
        public NativeLibrary(string name);

        // Loads a native function pointer by name.
        public IntPtr LoadFunction(string name);

        // Loads a function whose signature matches the given delegate type's signature.
        // This is roughly equivalent to calling LoadFunction + Marshal.GetDelegateForFunctionPointer
        public T LoadDelegate<T>(string name);

        // Frees the native library. Function pointers retrieved from this library will be void.
        public void Dispose();
    }
}
@migueldeicaza

This comment has been minimized.

Member

migueldeicaza commented Nov 13, 2017

This wrapper looks good, additionally, I would add the following API, that would lookup a symbol (not a function) from the specified dynamic library:

IntPtr SymbolAddress (string name);

This would be useful to access global variables and other global symbols in the loaded library.

@tmds

This comment has been minimized.

Member

tmds commented Nov 13, 2017

@mellinoe can this be used to perform lookups using dlvsym instead of dlsym on Linux? dlvsym is a Linux specific mechanism that allows symbols to be versioned. It allows a library to provide different versions of the same symbol. The dlsym lookup (that is used in coreclr) always gives you the oldest version of the symbol. See dotnet/coreclr#8721 for more info.

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented Nov 13, 2017

@tdms Looking at https://linux.die.net/man/3/dlvsym and https://www.gnu.org/software/gnulib/manual/html_node/dlvsym.html, dlvsym is a glibc-extension. This means it's not available on non-glibc Linux, BSD, macOS and Windows (GetProcAddress doesn't seem to take a version parameter, either).

I'm wondering whether it makes sense to add support for a glibc-extension in the core API?

@tmds

This comment has been minimized.

Member

tmds commented Nov 13, 2017

I'm wondering whether it makes sense to add support for a glibc-extension in the core API?

I'm not requesting dlvsym to be baked in .NET Core. I tried this with dotnet/coreclr#8721 (+PR) and it was not accepted.
I got feedback this should be done via some extension mechanism. I believe this proposal could fit that use-case.

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented Nov 13, 2017

@tmds Well with this API you should be able to do something like this, right:

using(var dl = new NativeLibrary("dl"))
{
    var dlvsym = dl.LoadDelegate<dlvsym_delegate>("dlvsym");
}

I do hope that the NativeLibrary API will make it into .NET core, though ;-)

@tmds

This comment has been minimized.

Member

tmds commented Nov 13, 2017

I think in this proposal, I'm looking to override LoadFunction. So e.g. LoadFunction("foo@VERS_1.1") calls dlvsym(Handle, "foo", "VERS1_1");.

@tmds

This comment has been minimized.

Member

tmds commented Nov 15, 2017

The constructor of NativeLibrary involves opening an operating system handle to a native shared library, via LoadLibrary, dlopen, etc. I have proposed that NativeLibrary be disposable, which would involve calling FreeLibrary, dlclose, etc. Is this desirable or useful?

What happens when someone uses a delegate after disposing the library?

A related issue is how the runtime currently tracks and manages these handles. Should handles opened by NativeLibrary be coordinated and tracked in the same way that other handles (via PInvoke, etc.) are?

Are libraries used via PInvoke ever closed? When?

@ghost

This comment has been minimized.

ghost commented Nov 20, 2017

What happens when someone uses a delegate after disposing the library?

simple ObjectDisposedException?

@jkotas jkotas modified the milestones: Future, 2.1.0 Dec 5, 2017

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented Dec 5, 2017

Adding the area owners @russellhadley @luqun @shrah to this thread.

One of the issues we're trying to solve is how managed code can load native libraries in a cross-platform way.

For example, System.Drawing.Common depends on libgdiplus which ships in various forms in various distros. So the code probes for a couple of files and loads the correctly library using dlopen.

One of the limitations we've hit is that dlopen also lives in separate libraries - libdl.so on most Unixes but sometimes also libdl.so.2 (CentOS) or libc (FreeBSD).

I've tried to address that in #25134 by adding dlopen to the PAL but the consensus on that issue was that System.Drawing.Common cannot call the PAL directly and any wrapper around dlopen needs to be exposed in a separate API which is part of corefx.

Hence this issue 😄 .

Can you give your feedback on this API proposal?

@jkotas

This comment has been minimized.

Member

jkotas commented Dec 9, 2017

We may need a constructor overload that takes DllImportSearchPath to control the probing paths.

@GrabYourPitchforks

This comment has been minimized.

Member

GrabYourPitchforks commented Apr 17, 2018

@chmorgan the next steps are for us to finish the existing PR for this feature when the checkin window opens back up.

@dotMorten

This comment has been minimized.

dotMorten commented Apr 17, 2018

What about the fact that on iOS callbacks must be static? That's where all my interop code diverges the most. For us to be able to create truly reusable interop code, these sort of differences needs to be addressed too.
Second when running on .NET Framework as AnyCPU we need to be able to point to two different DLLs (or 3 if we get ARM64 support) based on which architecture the app is running under.

@ghost

This comment has been minimized.

@dotMorten

This comment has been minimized.

dotMorten commented Apr 17, 2018

@kasper3 Good point. I really like the \runtime\[TFM]-[arch]\native\ approach that UWP uses. The other platforms need to support this as well - currently we have to create .targets files and either embed or xcopy deploy the dynamic libraries.

@Ruslan-B

This comment has been minimized.

Ruslan-B commented Apr 17, 2018

@kasper3, @dotMorten no offence, but I guess warping the native to OS packaging systems is a bit of topic, as in case you are the owner of the native library - you can maintain a library naming in a dotnet friendly way. So you need this functionality only in case you are not the owner of the original library(s) which is exposing C-API. To summarize this is needed to cover discrepancies between different OS and habits for naming these libraries.

@dotMorten

This comment has been minimized.

dotMorten commented Apr 17, 2018

@Ruslan-B I fail to understand the relevance to whether you own the original library or not.

The way you deploy your native libs via NuGet is quite different on each platform.
On Android you have to embed you .so files as <EmbeddedNativeLibrary/> or <AndroidNativeLibrary/> depending on whether its embedded in a library or an app (why they didn't make it one build action is beyond me). On UWP and WPF you have to xcopy deploy the DLL, same on iOS but there you also have to define several MtouchExtraArgs.

Having the \runtimes\ folder part be part of the nuget story across all platforms would simplify and unify how you get the native libs deployed in your app. While I agree that part of the discussion is probably more appropriate in the NuGet repo, it's completely valid in the context of unifying how we use native code on the various platforms.

@Ruslan-B

This comment has been minimized.

Ruslan-B commented Apr 17, 2018

@dotMorten in this case I would suggest to reread the original proposal.

@dotMorten

This comment has been minimized.

dotMorten commented Apr 17, 2018

@Ruslan-B Read the entire thing, and you'll find the discussion has expanded quite beyond that.

@Ruslan-B

This comment has been minimized.

Ruslan-B commented Apr 17, 2018

@dotMorten true, however, I think the only reason we need this, is to handle discrepancies between different OS with stable habits of naming things.

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented Apr 17, 2018

This issue is an API proposal issue. It concerns an API for loading native libraries and using the functions they expose. It was created because we need to load (lib)gdiplus for System.Drawing across platforms.

How you package these native libraries with your app (and whether you should package them at all or use a different acquisition mechanism) is related but not strictly in scope for the API proposal.

I don't know about iOS and Android, but the .NET Core SDK supports the runtimes folder structure and that works very well for the projects and I'm involved in - YMMV.

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented May 23, 2018

@karelz @joshfree This API didn't make it for 2.1, any way we can track it for 2.2.0 (it's currently marked as Future)?

@karelz karelz modified the milestones: Future, 2.2.0 May 23, 2018

@karelz

This comment has been minimized.

Member

karelz commented May 23, 2018

Currently there is not much difference between Future and 2.2. I moved it to 2.2, but it depends if we get to agreement and the result passes code reviews (there has been quite a few people with strong opinions).

@xoofx

This comment has been minimized.

Member

xoofx commented May 23, 2018

Currently there is not much difference between Future and 2.2. I moved it to 2.2, but it depends if we get to agreement and the result passes code reviews (there has been quite a few people with strong opinions).

@karelz How should we proceed from there? Based on my previous pseudo-proposal should I open a new issue?

@karelz

This comment has been minimized.

Member

karelz commented May 23, 2018

I will defer to @GrabYourPitchforks who worked on it and has PR ready (#17135 (comment)), he should be able to suggest best next steps (once he is back from vacation - one more week).

@xoofx

This comment has been minimized.

Member

xoofx commented May 23, 2018

What about the fact that on iOS callbacks must be static? That's where all my interop code diverges the most. For us to be able to create truly reusable interop code, these sort of differences needs to be addressed too.

@dotMorten btw, what do you mean by static? Static linking required by iOS with mono? Isn't dlopen being supported starting from iOS8+? (or you need to support older iOS?)

@joshfree

This comment has been minimized.

Member

joshfree commented May 23, 2018

I will defer to @GrabYourPitchforks who worked on it

Actually we should defer this to @jeffschwMSFT and the Interop team, who are interested in driving this feature area forward in .NET Core. @jeffschwMSFT could you share your team's current thoughts on direction / shape?

@jeffschwMSFT

This comment has been minimized.

Member

jeffschwMSFT commented May 23, 2018

@qmfrederik we are taking a step back and considering the broader scenario (exploring a dllmap like mechanism). Right now we don't have a concrete design, but once we do we will share and update this thread.

@qmfrederik

This comment has been minimized.

Collaborator

qmfrederik commented May 23, 2018

So, based on all the feedback, perhaps we should split the issue in two.

The core gist of this is that we want a way to call dlopen/LoadLibrary and dlsym/GetProcAddress; most of us can take it from there. The API @xoofx proposed as well as the original proposal for this issue seem fine.

Making cross-platform library loading (i.e. dllmap & friends) would be orthogonal to that, no?

@ghost

This comment has been minimized.

ghost commented May 23, 2018

Currently for 2.2, there is one inconsistency left in DllImport tracked by dotnet/coreclr#17604.
This approved API is definitely not related to DllImport enhancement with DllMap like mechanism (which is actually tracked by @akoeplinger's dotnet/coreclr#930). So we would love to see @GrabYourPitchforks's PR dotnet/coreclr#16409 getting resurrected.

@xoofx

This comment has been minimized.

Member

xoofx commented May 23, 2018

Making cross-platform library loading (i.e. dllmap & friends) would be orthogonal to that, no?

Yeah, I agree, maybe not completely orthogonal, but it is more an implementation detail or an enhancement to the resolution. It should not change the API shape, nor the default behavior if a dllmap file is not present (which is good). Dllmap is a super nice feature/addition and it should be considered, but maybe we should split first an implementation without, which was almost done it seems, and then proceed on the full dllmap story (which will take likely several months of study+shape+implem+tests...etc.)

@swaroop-sridhar

This comment has been minimized.

Contributor

swaroop-sridhar commented Oct 15, 2018

Closing this issue, in favor of #32015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment