Calling a Julia function from a non-Julia thread

[barche]

As discussed on julia-users, I would like for it to be possible to call a Julia function from a non-Julia thread:
https://groups.google.com/forum/#!topic/julia-users/TqC79eYTYx8

My use case is calling a Julia OpenGL rendering function from within the QML rendering thread. The current workaround is to force Qt to use the non-threaded rendering engine.

[stevengj]

One way to do this is to have a Julia task waiting to receive requests from your thread. You can send requests from another thread by calling uv_async_send, which is thread-safe.

There was some discussion of this in the julia-users mailing list, and this mechanism is used in ZMQ.jl to some extent.

[stevengj]

See also #5657, which added this discussion of thread-safe callbacks to the manual.

[StefanKarpinski]

Note that as @yuyichao mentioned, this is a very long-term feature request – even when we have fully non-experimental Julia threading, calling Julia code from a non-Julia thread may still not work. Given that reality, the uv_async_send approach is likely to be the most workable in a useful timeframe for you.

[barche]

@stevengj Thanks, I had actually seen that in the docs, but I wonder if it would lead to a deadlock here? The QML application is started from Julia, blocking the main Julia process (i.e. the REPL is unresponsive). Would the request to render then not block until the QML app is closed?

@StefanKarpinski I realise that now, thanks. I'm somewhat reassured by the Qt docs that the option for single-threaded rendering is also used in other circumstances and not likely to go away soon.

[StefanKarpinski]

If the Julia thread is "blocked" in the uv event loop, then it's not really blocked in the sense that C code is blocked – i.e. in the middle a call to the kernel and unable to do anything. So sending work to it via uv_async_send should cause it to do that work. If that doesn't work for some reason, please file issues and we'll make sure that it does work since this use case is quite common and important.

[stevengj]

@barche, if the main thread is blocked in the Qt event loop, not the Julia event loop, then you indeed would have a deadlock problem. Combining two event loops is always problematic. Three options:

• Run the Qt main event loop (i.e. start the QML app) in another thread.
• Run the Julia event loop, not the Qt event loop, but call a Qt event handler every 50ms or so in Julia. (This is what PyCall does to stay responsive while a Qt-based GUI is running.) Since the Qt event loop is mainly responding to human inputs, a 50ms latency is not too noticeable.
• Run the Qt event loop, but call jl_yield every once in a while to handle Julia tasks.

You can also find some other discussion online of merging libuv and Qt event loops, but I haven't seen anything very elegant.

[vchuravy]

For me the uv_async_send approach works, but it has drawbacks that I would like to get around.

1. Until the message is delivered all subsequent messages are ignored (at least that is what I understood from the libuv documentation)
2. The call into Julia doesn't block the C++ side so it needs extra synchronisation between the two runtimes.
3. Sending data along is fragile: https://github.com/JuliaGPU/OpenCL.jl/blob/716add3c4315727ff611cc3ac1b6b086be909a95/src/event.jl#L97-L140

I am aware that the things I am allowed to do a very limited in the cfunction that is called, but would it be feasible to allow the function to acquire locks? That might solve 1. and 2. for me.

[vtjnash]

It can acquire pthread locks, the only restriction is that it can't interact with the julia runtime.

[barche]

OK, I've experimented with uv_async_send, and unfortunately there is a deadlock in this case. I can't find a way to make the main Julia thread -which is executing the Qt event loop- receive the uv_async_send while at the same time have the rendering thread wait for the callback to complete.

@stevengj Running the Qt event loop in another thread (using @threadcall for example) would be interesting, but it would not work on OS X, since there the GUI thread must be the main thread. Also, every call into Julia would have to go through uv_async_send. I already have an option to run the GUI in the background using a uv_timer, but it is indeed not very elegant.

[yuyichao]

If it's waiting in the Qt event loop, use the qt equivalent of uv_async_send

[barche]

@yuyichao Just tried this, the deadlock remains, so it seems that is purely Qt related and I'll not clutter the discussion further with this.