MoarVM Support of Callbacks in a Different Thread

[Xliff]

…
I am currently working on NaitveCall bindings to a GLib-based multi-media library. However, when I attempted to add a Perl6 "callback" via the GLib signaling mechanism, I get the following error:

MoarVM panic: native callback ran on thread (140407967450880) unknown to MoarVM

As I understand it, this is due to a limitation where MoarVM will not handle being called from a thread created by the C library.

Are there any plans to add such a capability to MoarVM? Can the technical issues involved be explained? Would anyone have any ideas as to a workaround?

[Xliff]

For those interested, here is a backtrace that illustrates the issue.

[jnthn]

It should be possible, just needs somebody with the time to do it. The place where the error is given should be a good point to implement dealing with it in a more useful way.

One would need to react to such a situation by creating the required MoarVM ThreadContext data structure, making sure it's added into the threads list, etc. That's probably not so hard, but will likely need a bit of a refactoring of the thread creation code to make it possible (without a bunch of undesirable copy-paste). One would also need to make sure to mark the thread blocked for GC when returning from the callback (that may fall out of the code that's already there, but it should be checked).

Making sure the thread is marked as application lifetime at a MoarVM level means it won't try and join it at shutdown, which reasonably represents "we don't control its lifetime", but there's another issue: it means that anything creating numerous short-lived threads and doing callbacks with them will leak memory over time. Provided the C code is just re-using the same threads, it'll be fine, though, and this could be treated as a separate follow-up problem (or documented as an implementation restriction). I don't know exactly how we could resolve it; we'd need to get a callback when the thread goes away, or perhaps poll for its existence.

[Xliff]

@jnthn: Thanks for the details! Would something like this be a possible addition for 6.e?

[vrurg]

@Xliff This is not really a matter of language specification because neither of language behaviours are changing. Just a technical implementation which could happen any time when its ready.

[jnthn]

Indeed, NativeCall isn't actually part of the language spec. This can happen as soon as somebody has the time/patience to make it happen.

For @Xliff or anyone who wants to help this happen sooner, but doesn't feel like wading into MoarVM internals: a PR to Rakudo that adds a todo'd NativeCall test that should pass if this is addressed would be great.

[Xliff]

@jnthn: Thanks for the suggestion. Would this be something to add to roast?

[salortiz]

@Xliff: No, Rakudo has it's own tests in https://github.com/rakudo/rakudo/tree/master/t/04-nativecall

[Xliff]

Hello,

Just reporting that I have encountered this issue, again.

Over the past 3 years, I've run into this in the following projects:

GtkPlus
GStreamer

and now:

FluidSynth.

I have bindings for all of these projects. Now Fluidsynth in particular is of a heavily multi-threaded nature, and my bindings cannot unlock their full potential in Rakudo without this issue being looked into.

I would do it myself, but I have utterly no experience writing multi-threaded code in C. If I had some help, I would gladly take a stab at it.

So consider this a call forf help.

Thanks