New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop a fully multithreaded C runtime for pthreads. #3495

Open
juj opened this Issue Jun 3, 2015 · 9 comments

Comments

Projects
None yet
3 participants
@juj
Collaborator

juj commented Jun 3, 2015

Several functions in the C runtime are currently effectively singlethreaded, because they carry global process specific state in JS code that can't be thread local. Therefore calls to these runtime functions from pthreads are proxied to the main thread. Migrating to use more musl can avoid this issue. Also, investigate creating a fully asm.js -based virtual filesystem to be used when targeting pthreads, so that C file I/O can be naturally multithreaded as well.

@juj juj added the multithreading label Jun 3, 2015

@juj

This comment has been minimized.

Show comment
Hide comment
@juj

juj Jun 3, 2015

Collaborator

See https://github.com/kripken/emscripten/blob/incoming/system/lib/pthread/library_pthread.c#L252 for a list of the functions that are currently proxied.

Collaborator

juj commented Jun 3, 2015

See https://github.com/kripken/emscripten/blob/incoming/system/lib/pthread/library_pthread.c#L252 for a list of the functions that are currently proxied.

@juj

This comment has been minimized.

Show comment
Hide comment
@juj

juj Sep 14, 2015

Collaborator

A special note to clarify: this issue is not only performance-related, but this is a problem because this can cause a special kind of deadlocking that native code does not exhibit. If the following conditions are met:

a) the application was not built with --proxy-to-worker mode, and
b) the main thread has entered a blocking loop that waits for a thread before exiting the loop using only atomic lock-free techniques, e.g. an atomic compare-and-swap (CAS), and
c) the loop does not call to any pthread api functions, nor to certain C runtime functions that the pthreads api considers to be "cancellation points" (see list here http://man7.org/linux/man-pages/man7/pthreads.7.html), and
d) at the same time, the worker thread that the main thread waits on, attempts to perform a proxied C runtime operation,

then the main thread and the worker thread will lock up, since they are both waiting for each other to progress: the main thread is spinning (livelocking) to wait until the worker thread flips a bit somewhere, while the worker thread is sleeping to wait for the main thread to perform the proxied operation before it can continue. Conceptually this is a case of a priority inversion failure between two mutexes, even though mutexes are not technically involved here.

The correct solution to this will be to fix up Emscripten C runtime to not require proxying API calls, i.e. developing a fully multithreaded C runtime, which will remove the need for the worker thread to wait for the main thread.

A workaround solution is to audit all instances of blocking atomic loops that an Emscripten main thread can enter, and add a call to a special function emscripten_main_thread_process_queued_calls(); in the body of that loop. This call allows the main thread to assist the worker thread to perform the C runtime call while the main thread is still spinning to wait for the thread, thus avoiding the deadlock.

Collaborator

juj commented Sep 14, 2015

A special note to clarify: this issue is not only performance-related, but this is a problem because this can cause a special kind of deadlocking that native code does not exhibit. If the following conditions are met:

a) the application was not built with --proxy-to-worker mode, and
b) the main thread has entered a blocking loop that waits for a thread before exiting the loop using only atomic lock-free techniques, e.g. an atomic compare-and-swap (CAS), and
c) the loop does not call to any pthread api functions, nor to certain C runtime functions that the pthreads api considers to be "cancellation points" (see list here http://man7.org/linux/man-pages/man7/pthreads.7.html), and
d) at the same time, the worker thread that the main thread waits on, attempts to perform a proxied C runtime operation,

then the main thread and the worker thread will lock up, since they are both waiting for each other to progress: the main thread is spinning (livelocking) to wait until the worker thread flips a bit somewhere, while the worker thread is sleeping to wait for the main thread to perform the proxied operation before it can continue. Conceptually this is a case of a priority inversion failure between two mutexes, even though mutexes are not technically involved here.

The correct solution to this will be to fix up Emscripten C runtime to not require proxying API calls, i.e. developing a fully multithreaded C runtime, which will remove the need for the worker thread to wait for the main thread.

A workaround solution is to audit all instances of blocking atomic loops that an Emscripten main thread can enter, and add a call to a special function emscripten_main_thread_process_queued_calls(); in the body of that loop. This call allows the main thread to assist the worker thread to perform the C runtime call while the main thread is still spinning to wait for the thread, thus avoiding the deadlock.

juj added a commit that referenced this issue Sep 14, 2015

@kripken

This comment has been minimized.

Show comment
Hide comment
@kripken

kripken Sep 15, 2015

Owner

Regarding point (a), what is the main thing stopping a pthreads application from doing that today? Proxy-to-worker seems like the best solution all around here.

Owner

kripken commented Sep 15, 2015

Regarding point (a), what is the main thing stopping a pthreads application from doing that today? Proxy-to-worker seems like the best solution all around here.

@juj

This comment has been minimized.

Show comment
Hide comment
@juj

juj Sep 15, 2015

Collaborator

Yes, --proxy-to-worker will be one way to work around this, but I have not had time to start implementing it. The reason is that it is quite a bit more complicated, since there's a large amount of handwritten JS code that will need to be lifted from the main JS thread to the main browser thread. In general any piece of JS code that stores data that should be visible to all threads, e.g. the filesystem, needs to live in the main browser thread and not the main JS thread in order to decouple the blocking.

Collaborator

juj commented Sep 15, 2015

Yes, --proxy-to-worker will be one way to work around this, but I have not had time to start implementing it. The reason is that it is quite a bit more complicated, since there's a large amount of handwritten JS code that will need to be lifted from the main JS thread to the main browser thread. In general any piece of JS code that stores data that should be visible to all threads, e.g. the filesystem, needs to live in the main browser thread and not the main JS thread in order to decouple the blocking.

@kripken

This comment has been minimized.

Show comment
Hide comment
@kripken

kripken Sep 15, 2015

Owner

Oh ok, I see. Sounds like we need to keep the main thread mostly as it is now, but to not run user code in it, so it just receives proxying requests.

Owner

kripken commented Sep 15, 2015

Oh ok, I see. Sounds like we need to keep the main thread mostly as it is now, but to not run user code in it, so it just receives proxying requests.

@kripken

This comment has been minimized.

Show comment
Hide comment
@kripken

kripken Sep 15, 2015

Owner

So basically, we want to move main() to a pthread, but leave everything else as-is?

Owner

kripken commented Sep 15, 2015

So basically, we want to move main() to a pthread, but leave everything else as-is?

@juj

This comment has been minimized.

Show comment
Hide comment
@juj

juj Sep 15, 2015

Collaborator

Yeah, probably the cleanest way would be to load the whole compiled .js file in the main browser thread as well, but never call main() in it.

Collaborator

juj commented Sep 15, 2015

Yeah, probably the cleanest way would be to load the whole compiled .js file in the main browser thread as well, but never call main() in it.

@gouletr

This comment has been minimized.

Show comment
Hide comment
@gouletr

gouletr Sep 16, 2015

Contributor

What's the actual implications of --proxy-to-worker mode? What would it mean for a large code base of a game engine for instance?

Perhaps it would be a good thing to print in the console each time a call is put in the proxy queue (once per call name, so that it doesn't spam) so that we know where it happens?

Contributor

gouletr commented Sep 16, 2015

What's the actual implications of --proxy-to-worker mode? What would it mean for a large code base of a game engine for instance?

Perhaps it would be a good thing to print in the console each time a call is put in the proxy queue (once per call name, so that it doesn't spam) so that we know where it happens?

@juj

This comment has been minimized.

Show comment
Hide comment
@juj

juj Sep 16, 2015

Collaborator

The intent is that --proxy-to-worker should be as fully transparent to the
application as possible and all code should work nicely with it when
pthreads is enabled as well (since we can do synchronous postMessaging via
the SAB). Currently there will most likely exist a lot of bugs of various
small things that may not be proxied correctly, or may not be proxied
synchronously at all if SAB is not enabled. We are working on this, testing
and comments about your experience are welcome!

2015-09-16 18:04 GMT+03:00 Robert Goulet notifications@github.com:

What's the actual implications of --proxy-to-worker mode? What would it
mean for a large code base of a game engine for instance?

Perhaps it would be a good thing to print in the console each time a call
is put in the proxy queue (once per call name, so that it doesn't spam) so
that we know where it happens?


Reply to this email directly or view it on GitHub
#3495 (comment)
.

Collaborator

juj commented Sep 16, 2015

The intent is that --proxy-to-worker should be as fully transparent to the
application as possible and all code should work nicely with it when
pthreads is enabled as well (since we can do synchronous postMessaging via
the SAB). Currently there will most likely exist a lot of bugs of various
small things that may not be proxied correctly, or may not be proxied
synchronously at all if SAB is not enabled. We are working on this, testing
and comments about your experience are welcome!

2015-09-16 18:04 GMT+03:00 Robert Goulet notifications@github.com:

What's the actual implications of --proxy-to-worker mode? What would it
mean for a large code base of a game engine for instance?

Perhaps it would be a good thing to print in the console each time a call
is put in the proxy queue (once per call name, so that it doesn't spam) so
that we know where it happens?


Reply to this email directly or view it on GitHub
#3495 (comment)
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment