Control threading behavior

[Andlon]

I am experiencing some non-determinism problems. This appears to be due to threading behavior. When running without contacts, it looks as if I am able to get deterministic (enough) output by disabling threading on my end. However, when running simulations with contacts with ipc-toolkit, I'm getting very non-deterministic behavior (seen, for example, through very varying number of solver iterations across runs with identical parameters).

I'd like to disable threading in ipc-toolkit to see if I get deterministic results this way. However, I am not so familiar with TBB and I'm not even sure if TBB is the only thing that's being used in ipc-toolkit, or if there are other sources of parallelism. Is there a way to disable parallelism or otherwise control the number of threads in ipc-toolkit?

Note: The problems I'm looking at are extremely sensitive (presumably due to near-singularity of matrices), I don't believe there's anything wrong with ipc-toolkit.

[zfergus]

With multi-threading enabled, it is expected that the results can be non-deterministic because of rounding differences when adding numbers in different orders. We only utilize TBB in the toolkit (excluding experimental CUDA CCD), so to make the results deterministic you can limit TBB's max number of threads to one. The following code shows how to do this:

#include <tbb/global_control.h>
// ...
tbb::global_control thread_limiter(tbb::global_control::max_allowed_parallelism, 1);

As long as this thread_limiter object stays alive, the number of threads will be limited to 1. This example shows the thread limiter stack allocated, so when the object goes out-of-scope, the limit will be released. If you want this limit for the entire length of your program it is best to define it in main or you can define it statically using a std::shared_ptr<tbb::global_control>. For example,

#include <tbb/info.h>
#include <tbb/global_control.h>

static std::shared_ptr<tbb::global_control> thread_limiter;

void set_num_threads(int nthreads)
{
    if (nthreads <= 0) {
        nthreads = tbb::info::default_concurrency();
    } else if (nthreads > tbb::info::default_concurrency()) {
        logger().warn(
            "Attempting to use more threads than available ({:d} > {:d})!",
            nthreads, tbb::info::default_concurrency());
        nthreads = tbb::info::default_concurrency();
    }
    thread_limiter = std::make_shared<tbb::global_control>(
        tbb::global_control::max_allowed_parallelism, nthreads);
}

[Andlon]

@zfergus: thanks, that's very helpful!

[zfergus]

I added this description here.