Global Initialization Logic for Plugins

[e1528532]

Unfortunately the GHC runtime has a deficiency that renders it unable to reinitialize the runtime system after it has been closed down (see https://ghc.haskell.org/trac/ghc/ticket/13693). While it most likely does not matter for normal use cases it is a real show stopper for using it with libelektra, as libelektra dynamically loads and unloads plugins all the time in one process. It looks like the runtime can only be started once per process, even if the associated shared libraries are unloaded inbetween, otherwise it will lead to random behavior like deadloops or segfaults or similar things.

I've already tried all sorts of "easy solutions" in the form of different ways of starting/shutting down the runtime, not doing a real shutdown but manually hack some cleanup actions together picked from the hs_exit routine, and similar, but none of them worked and i slowly run out of ideas.

So basically i see two options here:

• Grab the ghc sources and try to fix this issue there. The downside is that this is quite unpredictable, it could be possibly a quick fix maybe, or its going to be a matter of weeks, or i won't ever find a solution in the worst case. In any way if I managed to fix it it would still take a long time until it reaches upstream ghc versions, especially on more conservative distributions like debian, so i'd avoid this approach.

• Somehow build global plugin initialization logic into libelektra itself which ensures the runtime gets started up before any other work is done (or lazily upon the first hs plugin call) and then gets unloaded after everything is done and the process is about to exit, so that it remains open in a memory location that doesn't get unloaded during the process lifetime.

i thought about using a global plugin but it seems to suffer from the same issue - loading and unloading everything dynamically which makes it unsuitable to use either. Maybe there is already something like i need available and i'm just not aware about it? If not we should probably discuss how such system would look like.

So summing up the requirements:

• load the haskell runtime before any haskell plugin gets called
• keep the haskell runtime and the associated rts libraries open throughout the process lifetime
• exit once there will be no more plugin calls, if it hasn't been started from a haskell context (e.g. standalone haskell application using the bindings)

[markus2330]

Thank you for bringing this up before crashes are reported! We should really write a design decision to document this once and for all.

Grab the ghc sources and try to fix this issue there.

I am afraid that the effort for this would be too high. Most likely they have much global state.

It is discussed in many issues (e.g. #767), and and a proposal was written in #418.

Thus ref counting does not really solve the issue (#419) and it is impossible to know if you are the most-outer user (if "global" initialization is needed or already done), the current situation is that we never close interpreters which do not allow restarts (and add a flag which allows us to enable shutdown).

A generic solution (that would also fix python/crypto/xerces) is to spawn a process for the problematic plugins. Then no restart is needed, when KDB is closed, the processes are terminated.

To implement this, much code from src/plugins/crypto/gpg.c could be reused. But we would need some communication protocol.

[e1528532]

Good to see that this issue has been encountered before and is not limited to the haskell runtime but also to other languages like python or libraries, so we already have some solutions!

Thank you for bringing this up before crashes are reported!

Basically that's the reason i brought it up, because the crashes caused by this issue block my typechecker prototype which i had inteded to deliver much earlier. As this plugin gets mounted to fulfill its functionality and thus starts a haskell runtime it runs into segfaults und deadloops due to the reloading.

We should really write a design decision to document this once and for all.

I totally agree, i have searched for something like that but apparently not good enought because i haven't found the issues you've linked. I'll write a small decision up once i've implemented a solution for the haskell plugins.

Thus ref counting does not really solve the issue

Yes, the haskell plugin architecture currently each takes care of the ghc initialization on its own which would become a problem when there is more then 1 haskell plugin in use simultaneously with a simple refcounting approach, as they wouldn't directly know about each other and the runtime's state.

A generic solution (that would also fix python/crypto/xerces) is to spawn a process for the problematic > plugins. Then no restart is needed, when KDB is closed, the processes are terminated.

To implement this, much code from src/plugins/crypto/gpg.c could be reused.

I think this seems the easiest solution for now. Spawning processes certainly results in some performance overhead but I hope we can ignore this for now so i can concentrate on a type system. It would also help in cases like a haskell program using our bindings calling into kdbOpen and thus possibly triggering more haskell plugins although there is already a runtime there. This is already a use case basically, some tests for the haskell bindings are implemented in haskell themself and the realworld tests issue a kdbOpen call. Similar to the crypto libraries i haven't found an obvious public way to check if a runtime system is already up which has been initialized outside libelektra. I'll make another proposal if more sophisticated communication needs are required, maybe some kind of output/string based or binary communication could be sufficient.

[BernhardDenner]

The Ruby plugin suffers from the same problem. See "Known Issues" in the README.md. As a result from this, I simply disabled plugin deinitialization for now, however in rare conditions (1 in 100 cases?) Elektra segfaults on shutdown while unloading libruby.

I also do not return anything other than 0 for elektra_ruby_open() as this would unload the ruby plugin and unload libruby.

However, in contrast to your problem, the Ruby-VM initialization is quite robust (ruby_setup() simply does nothing once it has been called before).

A generic solution (that would also fix python/crypto/xerces) is to spawn a process for the problematic plugins. Then no restart is needed, when KDB is closed, the processes are terminated.

This is an idea I never have thought of. In fact, this could be a working approach for the Ruby plugin too.
But maybe there are different situations when a fork is necessary and suitable:

• once per plugin (applicable for Ruby plugin)
• for each plugin instance (python plugin, as far as I know)

[markus2330]

I'll make another proposal if more sophisticated communication needs are required, maybe some kind of output/string based or binary communication could be sufficient.

A 1:1 forward of the arguments would be the obvious way as communication protocol. (i.e. KeySet for get/set/error; KeySet and Key for open/close).

@mpranj's mmap storage would help in keep the overhead low. But also a text-serialization with dump would most likely be insignificant compared to starting up Haskell/Ruby.

@e1528532 It would be great if you do not hard-code needs for the type checker.

In fact, this could be a working approach for the Ruby plugin too.

Yes. In these cases I would also not worry about performance too much. Compared to starting up Haskell/Ruby, fork is (most likely) a relatively cheap operation.

for each plugin instance (python plugin, as far as I know)

As far as I know it should be safe to run all plugins together in a separate process (per KDB). Conflicts only arise in situations where runtimes are unloaded or reloaded. Nevertheless, it might be easier to implement a process per plugin instance, so I would do this as first step. This way, we would also allow incompatible runtime initializations per plugin.

Btw. due to @mpranj work, the plugins will not be used (and thus fork avoided) as long as the configuration does not change.

[e1528532]

This works fine now with the pluginprocess library, closing this issue.

[markus2330]

Thank you for closing all the issues!