WIP: shared library support

[jpf91]

@ibuclaw:
This implements shared library support for GDC. I initially hoped to find a cross-platform (runtime linker independent) solution, but plugin loading in D can't be done without special linker support, so I just followed the DMD implementation.

Notable changes: DMD emits a call to _d_dso_registry into every object file and uses COMDAT groups to make sure only one call is in the final executable. I'm not sure if we can do this with GCC: Although I can easily make the function COMDAT, I don't know if it's possible to mark the DECL_STATIC_CONSTRUCTOR as COMDAT. So I ended up with one function per shared library but many calls to this function as every object file places one link to this function into the constructors array.
So instead of generating this _d_dso_registry call in the compiler I wrote a small __sharedlib.c helper which does the call. It must be linked once into every shared library and the main application as well. It's being located by gcc like the crt0.o files with a 'startfile'-spec.
I actually wonder if we should rewrite d-spec.cc to use spec strings? But I'm not sure if we can do that without patching gcc?

Changes to linker scripts are not necessary: Theres a ld feature which automatically adds __start_[section]/__stop_[section] symbols for orphan sections. We'd only have to modify the scripts to support --gc-sections: https://sourceware.org/binutils/docs/ld/Orphan-Sections.html#Orphan-Sections

Things to fix:

• runnable/mangle.d fails: mangle.d emits an extern reference to _D6object12__ModuleInfoZ. It's undefined in the object file, but when linking _D6object12__ModuleInfoZ actually gets defined in the application BSS instead of staying undefined and being resolved by the runtime linker to the symbol in druntime. Do you know what could cause this?
• runnable/untag doesn't compile: unresolvable R_X86_64_TPOFF32 relocation against symbol '_D3std6getopt10optionCharw'. It compiles when using -fPIC but I though compiling the application with -fPIC shouldn't be necessary? Any idea about this?

Update 1

• Fixed up Solaris, FreeBSD, MinGW
• Tested on MinGW and Linux
• --enable-shared-phobos configure flag added
• build libgdruntime.so with -fversion=Shared

Update 2

• Reduced & fixed phobos unittest fail

Update 3
Unfortunately binutils uses PROVIDE to provide the __start_/__stop_ symbols for orphan sections. This works for statically linked as there is only one __start_dminfo symbol in that case. But for shared applications we need PROVIDE_HIDDEN symbols. I propose to send a patch to binutils:
binutils-2.24/ld/scripttempl/elf.sc

   ${WRITABLE_RODATA-${RODATA}}
   .${RODATA_NAME}1      ${RELOCATING-0} :
   {
      *(.${RODATA_NAME}1);
+     PROVIDE_HIDDEN (__start_dminfo = .);
+     KEEP(*(dminfo));
+     PROVIDE_HIDDEN (__stop_dminfo = .);
   }

[dejlek]

I wonder are there any tests in the DMD repository for the DSO support? - It would be nice to run these (if they exist) and check if all is well.

[jpf91]

These tests are here:
https://github.com/D-Programming-Language/druntime/tree/master/test/shared

and work fine when run in the same way DMD runs them. This especially means linking only against libgdruntime.so. If we link a plugin against libgdruntime and libgphobos2.so, a constructor in phobos crashes. I debugged this and the problem is that if a library is loaded via rt_loadLibrary it can't access it's moduleinfo in its constructors. This is an upstream bug though:
https://issues.dlang.org/show_bug.cgi?id=13287

(probably one that's hard to fix. I think the overall plugin design is quite fragile in D. TLS constructors complicate everything and it's obvious that nobody planned how TLS constructors would work with plugins when they were introduced)

[jpf91]

@ibuclaw How exactly do you regenerate the configure file and the Makefile.in files in libphobos?

[jpf91]

OK, found the phobos unittest problem: A unittest in std.stdio calls fork. fork doesn't copy existing threads. So if std.parallelism tests ran before, there are active threads when forking which are not copied. The std.parallelism destructor then tries to pthread_join these threads in the forked process and crashes. So this is not a GDC bug. DMD runs the tests for every module separately. I just added a workaround to always run std.stdio tests first.

So this now passes all tests for static linking and for shared linking only two tests in the testsuite fail. So I think this is ready for merging, as shared library support is optional and must be enabled explicitly anyway.

[ibuclaw]

Haven't given this too much of a look at (yet). But most of what I can see at a glimpse seems reasonable.

Things you might need to be aware of:

• TLS bug found in LDC:
  ldc-developers/druntime@04ce1ac
• Upstream commit that fixed it:
  fix for Issue 11543 - multiple definition of std.regex with shared library dlang/druntime#791

Also, the .minfo section may require to be bracketed? You can emit data to custom sections safely with get_section/switch_to_section

[jpf91]

@ibuclaw minfo needs to be bracketed. As it's a 'orphan section' (not handled in any linker script) ld automatically adds bracketing symbols for us, see https://sourceware.org/binutils/docs/ld/Orphan-Sections.html#Orphan-Sections

But there's a big problem: The bracketing symbols are global, so in an application there can only be one instance. We need hidden bracketing symbols instead, see Update3 in the initial post. The main question is whether we submit these changes to the binutils maintainers or whether we try to find a working workaround like dmd did. Changin the binutils scripts also should make gc-sections work without the hack David had to use in LDC.

[albatr0z]

Hello, would like to ask why is this closed? Are there bigger issues, will shared lib be implemented for Phobos?

[jpf91]

@albatr0z The code in this pull request is outdated and I didn't get round to updating it. We still want shared library support in GDC though.

@ibuclaw Any idea how to solve the bracketing issue? This is basically the only thing blocking shared libary support. If you don't have a specific idea I could ask on the GCC list.

[jpf91]

@ibuclaw ping

[ibuclaw]

I think the likely target this will affect is OpenBSD, maybe replace with a sorry("...") message for now?

[ibuclaw]

I don't really have an answer for linker woes.