Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Global Initialization Logic for Plugins #1768
Unfortunately the GHC runtime has a deficiency that renders it unable to reinitialize the runtime system after it has been closed down (see https://ghc.haskell.org/trac/ghc/ticket/13693). While it most likely does not matter for normal use cases it is a real show stopper for using it with libelektra, as libelektra dynamically loads and unloads plugins all the time in one process. It looks like the runtime can only be started once per process, even if the associated shared libraries are unloaded inbetween, otherwise it will lead to random behavior like deadloops or segfaults or similar things.
I've already tried all sorts of "easy solutions" in the form of different ways of starting/shutting down the runtime, not doing a real shutdown but manually hack some cleanup actions together picked from the hs_exit routine, and similar, but none of them worked and i slowly run out of ideas.
So basically i see two options here:
i thought about using a global plugin but it seems to suffer from the same issue - loading and unloading everything dynamically which makes it unsuitable to use either. Maybe there is already something like i need available and i'm just not aware about it? If not we should probably discuss how such system would look like.
So summing up the requirements:
Thank you for bringing this up before crashes are reported! We should really write a design decision to document this once and for all.
I am afraid that the effort for this would be too high. Most likely they have much global state.
Thus ref counting does not really solve the issue (#419) and it is impossible to know if you are the most-outer user (if "global" initialization is needed or already done), the current situation is that we never close interpreters which do not allow restarts (and add a flag which allows us to enable shutdown).
A generic solution (that would also fix python/crypto/xerces) is to spawn a process for the problematic plugins. Then no restart is needed, when KDB is closed, the processes are terminated.
To implement this, much code from src/plugins/crypto/gpg.c could be reused. But we would need some communication protocol.
Good to see that this issue has been encountered before and is not limited to the haskell runtime but also to other languages like python or libraries, so we already have some solutions!
Basically that's the reason i brought it up, because the crashes caused by this issue block my typechecker prototype which i had inteded to deliver much earlier. As this plugin gets mounted to fulfill its functionality and thus starts a haskell runtime it runs into segfaults und deadloops due to the reloading.
I totally agree, i have searched for something like that but apparently not good enought because i haven't found the issues you've linked. I'll write a small decision up once i've implemented a solution for the haskell plugins.
Yes, the haskell plugin architecture currently each takes care of the ghc initialization on its own which would become a problem when there is more then 1 haskell plugin in use simultaneously with a simple refcounting approach, as they wouldn't directly know about each other and the runtime's state.
I think this seems the easiest solution for now. Spawning processes certainly results in some performance overhead but I hope we can ignore this for now so i can concentrate on a type system. It would also help in cases like a haskell program using our bindings calling into kdbOpen and thus possibly triggering more haskell plugins although there is already a runtime there. This is already a use case basically, some tests for the haskell bindings are implemented in haskell themself and the realworld tests issue a kdbOpen call. Similar to the crypto libraries i haven't found an obvious public way to check if a runtime system is already up which has been initialized outside libelektra. I'll make another proposal if more sophisticated communication needs are required, maybe some kind of output/string based or binary communication could be sufficient.
The Ruby plugin suffers from the same problem. See "Known Issues" in the README.md. As a result from this, I simply disabled plugin deinitialization for now, however in rare conditions (1 in 100 cases?) Elektra segfaults on shutdown while unloading libruby.
I also do not return anything other than
However, in contrast to your problem, the Ruby-VM initialization is quite robust (
This is an idea I never have thought of. In fact, this could be a working approach for the Ruby plugin too.
A 1:1 forward of the arguments would be the obvious way as communication protocol. (i.e. KeySet for get/set/error; KeySet and Key for open/close).
@mpranj's mmap storage would help in keep the overhead low. But also a text-serialization with dump would most likely be insignificant compared to starting up Haskell/Ruby.
@e1528532 It would be great if you do not hard-code needs for the type checker.
Yes. In these cases I would also not worry about performance too much. Compared to starting up Haskell/Ruby, fork is (most likely) a relatively cheap operation.
As far as I know it should be safe to run all plugins together in a separate process (per KDB). Conflicts only arise in situations where runtimes are unloaded or reloaded. Nevertheless, it might be easier to implement a process per plugin instance, so I would do this as first step. This way, we would also allow incompatible runtime initializations per plugin.
Btw. due to @mpranj work, the plugins will not be used (and thus fork avoided) as long as the configuration does not change.