Shared runtime

[MartinNowak]

This already bitrots for about a year and it's still very big.
Piecewise merging caused delays for several months and
I also lack an idea how to further split it down.
I thought we might try something different to make this reviewable.

---

This serves as Meta Pull Request, please don't pull it.
I'll create subsequent pull requests for each commit
which should get discussed and merged separately.
I'll regularly rebase this one until hopefully nothing is left.

[nazriel]

Woohooo!
It's rolling!

@dawgfoto thank you very much for this gift :) 👍

[andralex]

yes this is amazing tanks

[ghost]

This is a good news, but what exatly now is supported and what not?

[nazriel]

AFAIK it's for link-time shared libraries.

dlang/dmd#1043 was pulled before to make this happen.

I guess @dawgfoto plans were to first implement link-time shared libraries and then pick up the runtime plugin-like loading

See: http://forum.dlang.org/thread/hmhaldyfziejrplgzazt@forum.dlang.org?page=3#post-50EF5875.1060009:40dawg.eu
for more information

[WalterBright]

I suspect that this would be made easier by have druntime be a shared library/DLL itself. That way, the gc and the thread management is automatically shared.

[MartinNowak]

I suspect that this would be made easier by have druntime be a shared library/DLL itself. That way, the gc and the thread management is automatically shared.

That's really a necessity for any further shared library support and is addressed by this pull request.
The other part is link-time shared libraries.

[MartinNowak]

• add array container add basic Array container #396

• store per-dso data druntime implementation for the new ELF registration #397, collect GC roots of DSO #416, add TLS range support #421

  The necessary compiler support was added. An initializer is written to every ELF object file the calls _d_dso_registry and passes it the _minfo and _deh section. Multiple initializers get merged by the linker so that only a single call per DSO occurs. This currently only weak-links against _d_dso_registry to simplify development.

• refactor initialization deferred

  This can be done automatically in the loader callback as on Windows,
  but it would thread-serialize module initialization. We then would also have
  to make sure that initialization doesn't deadlock when a launched thread also
  loads a library.
  There was a discussion about this here and I now tend to that Rainer Schuetze's argument outweights any deadlock and serialization concerns.

• I will time how long phobos takes for static initialization.

• dsoify GC roots, use GC roots from SectionGroup #432

• dsoify TLS scanning, SectionGroup support for TLS #441

  There is a bug in the current implementation, it calls malloc in the SIGUSR1 thread handler and malloc is not async safe.

• dsoify EH tables, use EH tables from SectionGroup #431

  The EH tables now only contain local addresses 2cc91c2 so we could optimize the unwinding by using a two level searching. First identifying the DSO of the callee and the scan all it's EH tables.

• dsoify ModuleInfos, add SectionGroup #424

  This is a runtime loading issue, but we need to discuss which libraries get thread-local initalization
  when creating a new thread. IMO we should only initialize the libraries of the parent thread and not
  follow the Windows model here which initializes all libraries of the process (in DLL_THREAD_ATTACH).
  This is necessary so that an unrelated thread doesn't prevent unloading and it also might wastes
  computation time and memory.
  And thread "impersonating" is a really ugly hack to perform per-thread initialization for already running threads.

• Makefile flag to build druntime as shared library, add dll build target #429

  Maybe we still want to discuss whether phobos/druntime should be one or two libraries.

• solve unittesting

  The problem here is that linking a unittest module in an executable with the shared druntime causes ODR errors because the module already is in the shared library.
  One approach was to compile the whole runtime with unittests into a shared library and use a unittest runner that only runs a single test at once so that we can keep make's parallelism. Unittests ran approx. twice as fast with that. The same would work for phobos but dmd needs a lot of memory/time (6GB) for compilation and a bugfix in the ELF code.

• add druntime unittest framework

  To automate certain test scenarios we need something like d_do_test in dmd.

• revise installation process, dmd.conf and inform package maintainers

• remove the weak linking of _d_dso_registry from the compiler when all of this is in a published release of druntime

[MartinNowak]

decide upon initialization point

This can be done automatically in the loader callback as on Windows,
but it would thread-serialize module initialization. We then would also have
to make sure that initialization doesn't deadlock when a launched thread also
loads a library.
There was a discussion about this here and I now tend to that Rainer Schuetze's argument outweights any deadlock and serialization concerns.

Ideally this would apply to the runtime initialization itself too. It's a major refactoring though (main, rt_init, rt_term, rt_module...).
I think we should go with the classic route for now and should tackle that before implementing runtime loading.

[jacob-carlborg]

I just looked at the newly committed that added the rt.dso module. For Mac OS X it says "missing integration with rt.memory_osx.onAddImage". Next to the declaration of _static_dsos it says:

Static DSOs loaded by the runtime linker. This includes the executable. These can't be unloaded.

Does that mean that _static_dsos is not for shared libraries loaded by dlopen? I'm asking because rt.memory_osx.onAddImage will be called for every loaded shared library, regardless if it's statically linked or dynamically loaded using dlopen.

[MartinNowak]

It's the same on linux, thre compiler generates an init function that gets called for every shared library and executable. We will be able to distinguish these, the onAddImage comment is a rough hint about what needs to be done.

[jacob-carlborg]

How can you distinguish if a shared library is loaded using dlopen or statically linked?

[MartinNowak]

Because the initializer/finalizers of a linked shared library run before/after main.

[MartinNowak]

The current roadmap is link-time support for linux then FreeBSD (simple) then OSX then the init refactoring then we need to extend the GC so it can finalize memory regions (shared mem unloading, shared library unloading) then we can look into runtime loading. TLS might become difficult on OSX, has anyone tried to implement native support? AFAIK it is only available since 10.6.

[jacob-carlborg]

Because the initializer/finalizers of a linked shared library run before/after main.

Aha, I see. Is that before/after the C main or the D main function? The onAddImage callback is initialized in _d_run_main. So we would need a flag to indicate if the runtime has been initialized, if not already available?

[jacob-carlborg]

TLS might become difficult on OSX, has anyone tried to implement native support?

No, not that I know. I created a ticket for it: 9476.

AFAIK it is only available since 10.6.

Unfortunately it's only available since 10.7. That means we would need to drop the support for 10.6. Although I think it's technically possible to do what the dynamic linker does in 10.7 in the D runtime. Then we could still support 10.6. If we actually want to do that is a different question. It seems Walter don't want to do that.

[WalterBright]

What is the status of this?

1. does it build druntime as a .so ?
2. does it enable user built .so's written in D?
3. can those user built .so's be dynamically loaded and unloaded?

[WalterBright]

My current status:

1. I have successfully build druntime.so for 64 bit linux.
2. I have successfully built a phobos library that does not include druntime.a.
3. I have build an app that calls printf("hello world\n"); uses druntime.so, and runs successfully.

This currently does not work with 32 bit linux since LDIV trashes EBX, which violates PIC protocol. But I'll deal with that later. The main goal for the moment is to get 64 bit linux .so's to work.

[WalterBright]

I'm ok with abandoning support for OSX 10.5 and less in order to support shared libraries with TLS. 10.6 appeared in Aug, 2009. That's nearly 4 years ago, an eternity in internet time.

[WalterBright]

Ah, 10.7 is july 2011. Not so old. Hmmm.

[WalterBright]

Unittests: I suggest simply building a special druntime.so with -unittest and running them all at once.

[MartinNowak]

What is the status of this?

I'll have to rebase/rewrite the last parts of this because of the changes done in the already pulled parts.

1. does it build druntime as a .so ?

Yes, by passing SHARED=1 to make, but this has now been replaced by the dll target (#429).

1. does it enable user built .so's written in D?

Yes.

1. can those user built .so's be dynamically loaded and unloaded?

No.

[WalterBright]

This is great progress.

But we do need to make dll's dynamically loadable and unloadable. A major, major use case for dll's is to provide rapid development turnaround by being able to hot load/unload them while not needing to shut down and restart the whole thing.

[WalterBright]

Let's get this pull request done. Then we can:

1. make sure the dll version of druntime passes the test suite
2. add this to the autotester
3. work on load/unload

[MartinNowak]

Unittests: I suggest simply building a special druntime.so with -unittest and running them all at once.

There is already a better solution. Building a druntime.so with -unittest and use an executable that only run the test passed as command-line argument, here 80950b8. That way we can keep makefile parallelism and reproducing bugs remains simple.

[MartinNowak]

But we do need to make dll's dynamically loadable and unloadable.

Of course, that's the number one reason to do all of this.
I posted an update on outstanding issues in the newsgroup thread.
Extending the GC could already be done in parallel.

archive/forum

[WalterBright]

What remains on this pull request to be merged?

[alexrp]

Everything that's not ticked in the task list above. (This pull request shouldn't be merged itself; it's a meta pull request.)

[WalterBright]

That's not correct, some unticked ones have been merged, like the EH tables thing.

[WalterBright]

A check of the commits shows that these are not merged:

register static GC roots… 877b6bb
support for scanning TLS roots… fca9aae
check for module hijacking… d2fcfe9
add more tests for gc scanning and thread init 39ba09b
add shared library tests 0358002
add SHARED option to posix.mak… 746c453

Should they be merged? Where are we on this? I'd like to get this pull request resolved ASAP.

[MartinNowak]

I haven't rebased this for quite a while because the implementation changed a lot during reviewing/recoding/merging.
The last bit is TLS support (#441) and then there is still the hijacking, the makefile and testing.

[MartinNowak]

What steps do we need to take to allow dynamic loading? There are several scenarios: (a) foreign language app loading exactly one D lib; (b) foreign language app loading more D libs; (c) D app loading one or more D libs.

a - none this already worked, create a -shared library that statically links against phobos, run rt_init/rt_term to initialize the runtime

b - same as (a) but inefficient because it uses multiple GCs and might easily cause ODR violations

c - this pull request is the ground work to make it possible, we still need to implement static initialization