dynamic loading

[MartinNowak]

This still lacks thorough testing so there are likely some bugs but it supports
almost all dynamic loading scenarios, including loading shared libraries in different threads.

depends on #587, #589, #590, #591 and #592

[MartinNowak]

module main;
import core.runtime, core.thread;

void main() {
    auto lib = Runtime.loadLibrary("./liba.so");
    auto thr = new Thread({
        auto lib = Runtime.loadLibrary("./liba.so");
        Runtime.unloadLibrary(lib);
    });
    thr.start();
    thr.join();
    Runtime.unloadLibrary(lib);
}

module liba;
import std.stdio;

shared static this() { writeln("shared static this()"); }
shared static ~this() { writeln("shared static ~this()"); }
static this() { writeln("static this()"); }
static ~this() { writeln("static ~this()"); }

output

shared static this()                     
static this()
static this()
static ~this()
static ~this()
shared static ~this()

[WalterBright]

Does it work for loading a D dll from C code?

[WalterBright]

The problem with calling dlclose() here is that the libdl.a needs to be linked in to the D program to access the dlxxxx() functions. This is annoying for normal D programs. The solution is to move the functions that call the dlxxxx() functions into a separate module, hence a D program won't need to link in libdl.a unless it actually tries to dynamically load a DLL.

Alternatively, libdl.so could itself be dynamically loaded as required.

[MartinNowak]

Right, I fixed this in b44e958.
Now only the shared druntime actually implements rt_loadLibrary/rt_unloadLibrary and some other functions. Calling Runtime.loadLibrary/Runtime.unloadLibrary when linking against a static druntime will lead to a linker error.

[MartinNowak]

Does it work for loading a D dll from C code?

Yes, that will work too.
The remaining shortcomings are thread local initialization of dependent libraries and the missing tests.
I'm currently working on this, but I don't know if I have enough time left to finish it within this week.

[WalterBright]

Calling Runtime.loadLibrary/Runtime.unloadLibrary when linking against a static druntime will lead to a linker error.

But what if you statically link with libphobos2.so, and then want to dynamically load a DLL?

[MartinNowak]

But what if you statically link with libphobos2.so, and then want to dynamically load a DLL?

You can see it in posix.mak, libdruntimeso.a is build with DLL support but libdruntime.a is build without,
i.e. libphobos2.so does support loading DLLs but libphobos2.a does not.

[MartinNowak]

I updated and finished the low-level implementation.
This now supports almost all shared library scenarios merely automatically.
It is also tested pretty thoroughly.
Here are the main gotchas for experimenting.

There is an extra layer of thread local reference accounting over the global one in the runtime linker which has some implications on how to use shared libraries.

• Loading libraries should only be done through Runtime.loadLibrary/Runtime.unloadLibrary
  (or rt_loadLibrary/rt_unloadLibrary for C), so that the thread local accounting doesn't
  miss anything.
  The recommended pattern to load a D library from a pure C program is to dlopen
  phobos/druntime first and to use rt_loadLibrary/rt_unloadLibrary hereafter.
• Loading a library only initializes it and it's linked dependencies for the current
  thread.
• Threads where a library is not initialize will not see the library's modules, nor the EH
  tables and the GC will miss to scan the library's TLS memory.
• A spawned thread (core.thread.Thread) will initialize the same libraries as it's parent
  thread. There is an implicit dlopen call for every dynamically loaded library so to
  prevent the parent thread from unloading. When a thread finishes it will drop/unload all
  remaining references to loaded libraries.
  The high-level implementation of the shared libraries support will provide a method to
  iterate over all loaded libraries, thus it will be possible to explicitly unload unused
  inherited libraries.
• The function thread_attachThis won't yet work correctly with shared libraries. The Thread doesn't know which libraries to initialize. (see Issue 10986).

Finding the dependencies of a library is somewhat expensive so it is only performed for D libraries and non-recursively.

• Because all dependencies are loaded before the library itself we do know the recursive dependencies.
  In case of loading a C library that depends on D libraries or for
  D->C->D dependency chains we will miss the recursive dependency.
• This might possibly lead to uninitialized threads when loading such a library in more than one thread.

I will try to fix any arising bugs ASAP but I don't have much time left currently.
Please tag bugzilla entries with the dll keyword.

Work on the high-level interface will start around the end of next week.

[jpf91]

The recommended pattern to load a D library from a pure C program is to dlopen
phobos/druntime first and to use rt_loadLibrary/rt_unloadLibrary hereafter.

Is it still possible to build a plugin for a C application which only uses dlopen?

[jpf91]

Should we do some explicit error handling (throwing OutofMemoryError, etc)?

[MartinNowak]

done

[MartinNowak]

It was and will remain possible to do that with the static phobos, given that the plugins don't interact (ODR issues).
Furthermore it should work correctly with the shared phobos, given that it doesn't involve multiple threads. We should add a test case for this.
There are some limitations to what is possible with multi-threading.

[MartinNowak]

We should add a test case for this.

And done.

[WalterBright]

Awesome!

[dnadlinger]

Please, Martin, whatever you do, please be sure to archive your explanations from this and other comment threads in a suitable form, even if it's only copying and pasting them to a Wiki page.

[dnadlinger]

I guess a comment should be added that the function is a template on purpose to avoid having to pull in the symbol from the druntime lib. It might not be so obvious for new users/contributors.

[MartinNowak]

Please, Martin, whatever you do, please be sure to archive your explanations from this and other comment threads in a suitable form, even if it's only copying and pasting them to a Wiki page.

I reposted this in the newsgroup but we need better documentation/articles/tutorials.
This preliminary and informal post is targeting people who are keen to experiment with the feature, but we shouldn't grow it into learning material.

[Geod24]

This PR introduced a regression (linker error on OSX): https://issues.dlang.org/show_bug.cgi?id=20587