Pthreads

[juj]

This set of commits implements initial pthreads library support for Emscripten code, and allows native code to utilize multithreading with the pthreads API. This backs on experimental research on the JavaScript SharedArrayBuffer and Atomics APIs from https://docs.google.com/document/d/1NDGA_gZJ7M7w1Bh8S0AoDyEqwDdRh4uSoTPSNn77PFk/edit?usp=sharing , and as such is preliminary and unsupported in other browsers except current Firefox Nightly builds. By default pthreads support is disabled, and the feature must be explicitly enabled by specifying the compiler flag -s USE_PTHREADS=1 and the linker flag -lpthread.

This is for experimental use only at the moment, since none of the threading specification has yet been standardized, and it may change a lot for the final proposal, even to the point of making large parts of this pull request obsolete.

This PR contains some unit tests, but more is available in https://github.com/juj/posixtestsuite (especially see emscripten-core/posixtestsuite@ed4bdd6 ), which contains ~300 unit tests for POSIX conformance for pthreads.

There is an accompanying pull request to the LLVM backend at emscripten-core/emscripten-fastcomp#67 .

[kripken]

newline here seems unnecessary

[juj]

Ops. Removed.

[kripken]

Very impressive work!

Bunch of questions above. I am still getting into the code, the answers will help me.

[gunyarakun]

I'm interested in running threads with pthread API in emscripten.

So I tried to build and run a sample pthread program and run posixtestsuite, but I failed to execute.

Could you point what's wrong I did?

I was using the browser FirefoxNightly 39.0a1 (2015-03-23) and emscripten/fastcomp on pthreads branch.

testing with simple pthread program

I fetched a pthread sample source code from http://en.wikipedia.org/wiki/POSIX_Threads

I built html/js with emcc and run it with emrun.

emcc wikipedia.c -lpthread -o wikipedia.html -s TOTAL_MEMORY=268435456 -s PTHREAD_POOL_SIZE=40 -s USE_PTHREADS=1 --emrun 
emrun --browser firefox_nightly wikipedia.html

The error below shows up continuously.

still waiting on run dependencies:
dependency: pthreads
(end of list)

Here is the source code.

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

#define NUM_THREADS     5

void *perform_work(void *argument)
{
   int passed_in_value;

   passed_in_value = *((int *) argument);
   printf("Hello World! It's me, thread with argument %d!\n", passed_in_value);

   /* optionally: insert more useful stuff here */

   return NULL;
}

int main(void)
{
   pthread_t threads[NUM_THREADS];
   int thread_args[NUM_THREADS];
   int result_code, index;

   // create all threads one by one
   for (index = 0; index < NUM_THREADS; ++index) {
      thread_args[index] = index;
      printf("In main: creating thread %d\n", index);
      result_code = pthread_create(&threads[index], NULL, perform_work, (void *) &thread_args[index]);
      assert(0 == result_code);
   }

   // wait for each thread to complete
   for (index = 0; index < NUM_THREADS; ++index) {
      // block until thread 'index' completes
      result_code = pthread_join(threads[index], NULL);
      printf("In main: thread %d has completed\n", index);
      assert(0 == result_code);
   }

   printf("In main: All threads completed successfully\n");
   exit(EXIT_SUCCESS);
}

testing with posixtestsuite

I challenged to run tests in posixtestsuite (https://github.com/juj/posixtestsuite)

• Modify emcc/run_emcc_tests.py emrun option '--browser' to use firefox_nightly

cmd = ['emrun', '--kill_start', '--kill_exit', '--timeout=15', '--silence_timeout=15', '--browser=firefox_nightly', out]

• Install emcc/run_all_tests.py and emcc/run_emcc_tests.py into a directory in $PATH (/usr/local/bin)
• Execute tests in conformance directory.

cd conformance/interfaces
run_all_tests.py

The test started well and launched Firefox, but the results are fail like following.

=== FINISHED ===
3 tests failed:
1-1.c, 2-1.c, 3-1.c

[juj]

@gunyarakun: There is currently an undocumented issue that you will need to work around: in Firefox there is a limit to how many web workers you can spawn. This is bug https://bugzilla.mozilla.org/show_bug.cgi?id=1052398 . To work around this, navigate to about:config and find the pref dom.workers.maxPerDomain and set it to a big enough value, e.g. 50. The value should be larger than what you pass in to -s PTHREAD_POOL_SIZE=x, or otherwise you will see that the run dependencies will never finish. Setting the value to at least 50 should allow running the open posix test suite as well, since some of those tests use quite large number of simultaneous threads.

[juj]

@kripken: Thanks for the reviews, I think I should have addressed all of those in the new batch of commits.

[gunyarakun]

@juj: thank you for your advice. I changed dom.workers.maxPerDomain and toolkit.startup.max_resumed_crashes (for not showing the dialog to be safe mode) and posixtestsuite seemed working well.

But the sample program following doesn't work well. Is my program wrong or is it a known problem?

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>

static void *thread_func(void *vptr_args) {
    for (int i = 0; i < 20; i++) {
        puts("  b");
        sleep(1);
    }

    return NULL;
}

int main(void) {
    pthread_t thread;

    if (pthread_create(&thread, NULL, thread_func, NULL) != 0) {
        return EXIT_FAILURE;
    }

    for (int i = 0; i < 20; i++) {
        puts("a");
        sleep(1);
    }

    if (pthread_join(thread, NULL) != 0) {
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

[juj]

The other pull request should be now in better shape. Rebased this on top of the latest incoming and reran the tests on my Windows box, which look ok.