Design issues with multithreaded application using workerfs

[LubosD]

Hello,

I'm porting a non-GUI (but Qt-based) application to the browser using emscripten. It processes large files and does some CPU intensive calculation on them. The original version uses OpenMP to do parallelization.

Initially, I built the application compiled with emscripten as a single threaded, which worked just fine:

• I created a web worker, where I created the module, ran a function and received callbacks about progress.
• I used WORKERFS to provide the large input files.

Now I ported the original OpenMP statement to QtConcurrent and all hell broke loose. I'm struggling to find a working design to basically do any threading at all from the application, because even the simplest things lead to deadlocks.

Attempt no 1:

• I still create a main web worker where I create the module. This is because the code contains unparallelized sections, which of course cannot be run on the main browser thread.
• I spawn threads later on, but these instantly freeze forever, because all syscalls related to file I/O are being proxied to the main thread, which cannot even handle any of these proxied calls, because it's blocked waiting for these threads to complete in the first place.

Attempt no 2:

So I changed the native entry point to spawn an std::thread and continue from there, so that the main web worker is free to receive all of those proxied calls. The result is similar:

• The main web worker receives the request to handle the I/Os, but ends up ignoring them, because emscripten_current_thread_process_queued_calls() calls emscripten_is_main_browser_thread(), which is indeed false, as this is a worker, so the other web workers are blocked forever waiting for a reply.

Question #1: Why is it a hard requirement that these proxied calls are only to be handled by browser's main thread and not by the "runtime main" thread? All of these syscalls work just fine on the worker thread in the single-threaded build of the app.

Now you could probably suggest that I kill the main web worker and instantiate everything from the main browser thread, but the documentation specifically says that WORKERFS can only be used from workers, so I assume all of the proxied file I/O calls will not end well.

Question #2: How is WORKERFS + pthreads supposed to work?

Plus I still need to support environments without SharedArrayBuffer, so it would be annoying to have a worker-based JS code path for single-threaded environments and the same duplicated JS code for multi-threaded environments, where the logic would be run from the main browser thread.

[Alexufo]

SharedArrayBuffer is most effective way to use multythread afaik.
Did you mean you want main worker and other workers for multithreading? It allow do not block main thread.