Approach to working with libraries that invoke callbacks from foreign threads

[dom96]

This was raised by @gokr. He started writing a brilliant project in Nim but ran into trouble when he needed to use the pharo MQTT library (library link and nim wrapper).

This library exposes an "asynchronous" API which works by using threads in the background. Using this API is necessary to achieve asynchronous operation for Jester.

---

Side node

As I am writing this, another idea came to me, perhaps @gokr will be willing to try it out:

Use the library's synchronous API in a Nim thread. This will ensure that all threads are handled by nim and thus there should be no trouble.

---

Indeed, the issues seem to arise when attempting to integrate these "foreign threads" with Nim's handling of threads (per-thread GC). You can see @gokr attempting to get this to work using setupForeignGC.

TL;DR

I have no idea how to wrap libraries that call into Nim code via a call-back from a foreign thread.
We need to figure out the best way to accomplish this and document it.

[dom96]

Relevant discussion: https://irclogs.nim-lang.org/24-01-2018.html#10:55:59

[cheatfate]

Its not hard, and parts of such solution is already can be seen in asyncdispatch.

Main idea is here https://github.com/nim-lang/Nim/blob/devel/lib/pure/asyncdispatch.nim#L760-L767.

We create helper function, which will be called from foreign thread. This function must be covered with {.push stackTrace:off.}, and don't use GC memory without system.protect/system.dispose.
This helper function must notify dispatcher somehow. In mentioned example it uses Windows API function PostQueuedCompletionStatus(), on Posix systems you can use pipes (for notification only, or for data transfer. It depends on how much data needs to be transferred). So you can write to pipe to notify dispatcher about new data.

Now when dispatcher got notified you can easily obtain your data in asyncdispatch thread.

[Araq]

With --gc:arc this is a non-issue, closing.