Spending a lot of time inside __pthread_create calling memset

[svenpilz]

Hi fellow concurrency performance hunters!

I'm trying to profile a performance regression using the profilers of Firefox and Chrome. It showed that some multi-threaded code spends about 65% or 7sec with __pthread_create calling memset. Emscripten is 3.1.32 (eab98adf462c39f3c31d348331c4830bcaa36949).

The code that spawns threads looks like this

std::vector<std::thread> threadPool;
threadPool.reserve(NUM_THREADS); // is navigator.hardwareConcurrency

for (size_t threadIndex = 0; threadIndex < threads; ++threadIndex) {
    threadPool.emplace_back(
         [&](/* Subset Indices per Thread  */) {
            for (/*  Subset Indices per Thread*/) {
                lambda(i); // Only captures by reference.
            }
        },
        /* Subset Indices per Thread */);
    }

for (auto &thread : threadPool) {
    thread.join();
}

Whereas we link with

• -sALLOW_MEMORY_GROWTH=1
• --memory-init-file 0
• -sINITIAL_MEMORY=128mb
• -sSTACK_SIZE=5MB
• -sPTHREAD_POOL_SIZE=navigator.hardwareConcurrency.

Has anybody seen something similar?

Thanks and greetings from rainy Berlin
Sven

[kripken]

pthread_create will memset to 0 the memory the thread needs, which includes the stack and TLS. Your stack size is 5MB, so reducing that could help significantly. Hopefully you don't need that much? The default is 64K these days, which is enough for most applications.

[sbc100]

Actually there was an oversight in the change that reduces the default stack size. The change to actually lower the default pthread stack size just landed in #19043

[sbc100]

I wonder if this is the same issue as #18628?

[svenpilz]

Thank you for your insights. I investigated the code a bit further and guess we start too many new threads. Newer more than the worker pool size, but I guess a thread pool on our C++ side would already help here. Seems there was no change regarding Emscripten. Guess I was mislead here a bit by Linux and Windows not showing that performance issue. Sorry for the noise :)

But I'm still confused why the memset takes so long here. We start less than 100 threads and don't really use TLS on our side, so maybe it would need to memset 500MB. Right? Is this done atomic per thread, or maybe from JS code (we use dynamic heap)?

[sbc100]

The memset is done per thread creation here:

emscripten/system/lib/pthread/pthread_create.c

Line 164 in a90c5ab

memset(block, 0, size);

Its almost certainly the stack size which is causing your problem. Other folks also noticed a slowdown once we started memset'ing the whole pthread stack (see #19043(.

You could confirm this by using a small pthread stack size (e.g. -sDEFAULT_PTHREAD_STACK_SIZE=16kb)

[svenpilz]

Affirmative :). Seems we need at least around 256kb, but that gives a ~9 times performance increase in Firefox.

Naive question from my side: Did I overestimated RAM speeds by thinking also 1GB should be very fast to memset, or is this a limitation of the WASM sandbox preventing reads/writes outside its memory?

Thanks :)

Edit: The 9 times performance increase even with our bad "just start new threads over and over again" pooling.

[sbc100]

Hopefully this will no longer be and issue once we land #19114, and should be completely non-issue once I remove the memset'ing of the stack completely.

In order to see how native memset compares to wasm memset once would need to run some tests but you can expect some amount of slowdown compared to native. Also remember that native plaforms are likely not calling memset for new thread at all since they are likely to be using mmap which provides zero-filled from the OS/hardware.

[svenpilz]

Thanks for the explanation and your help.