OpenMP control

[lfarv]

The behaviour of OpenMP, if activated, is fully controlled by the defaults values. Should more control be available ?

• Number of threads
• Scheduling
• …

[matsievskiysv]

I've benchmarked different schedulers with different slice sizes on my desktop machine (1000 particles, 1000 turns).

There's no significant difference. It would be interesting to repeat these calculations on a bigger machine.

---

benchmark.sh:

#!/usr/bin/env bash

rm -f results

for n in $(seq 1 $(nproc)); do
    for sc in auto runtime dynamic "dynamic, 1" "dynamic, 10" "dynamic, 50" \
		   guided "guided, 1" "guided, 10" "guided, 50" \
		   static "static, 1" "static, 10" "static, 50"; do
	echo "Threads: $n; scheduler: $sc" | tee --append results
	OMP_NUM_THREADS="$n" OMP_SCHEDULE="$sc" octave --eval 'bootstrap;trackMultipleParticles' >> results
    done
done

sed -i '/\/at\//d' results
sed -i -Ez 's/\n[0-9a-z, ]+?, +(with|without) +[0-9a-z, ]+?: +([0-9.]+)[0-9a-z, ]+?/; \1: \2/g' results
echo 'threads,without,with,scheduler,slices' > data.csv
sed -E 's/^[Tt]hreads: +([0-9]+); +scheduler: +([0-9a-zA-Z, ]+); +without: +([0-9.]+); +with: +([0-9.]+)/\1,\3,\4,\2/g' results | sed 's/ //g' >> data.csv

plot.py:

#!/usr/bin/env python3

import pandas as pd
import pylab as plt

data = pd.read_csv('data.csv', delimiter=',')

data.slices.unique()

for rad, radname in [('with', 'With longitudinal motion'),
                     ('without', 'Without longitudinal motion')]:
    plt.figure(figsize=(7, 5), dpi=200)
    for scheduler in data.scheduler.unique():
        d = data[data.scheduler == scheduler]
        plt.scatter(d.threads, d[rad], label=scheduler)
    plt.legend()
    plt.grid()
    plt.title(radname)
    plt.xlabel("Threads")
    plt.ylabel("Time, [s]")
    plt.savefig(f"{rad}.png")
    plt.close()

[lfarv]

Thanks for this interesting test, I'll try here!
My feeling is that the default behaviour of OpenMP is clever enough, so that we don't need a very detail control.
If we limit ourselves to the control of the number of threads, here is a proposal:

We introduce a new parameter controlled with the existing setoption/getoption mechanism, as for the max. number of iterations, convergence thresholds or differentiation steps. This "omp_num_threads" parameter is silently initialised with a corresponding environment variable ("OMP_NUM_THREADS") or at a large value if the envvar is not set. It can be modified at any time with setoption('omp_num_threads', value), for instance in startup.m. All this independently of the activation of OpenMP.

Then atpass will call omp_set_num_threads(int num) with num as the minimum of:

• omp_get_max_threads(), or omp_get_thread_limit()? (to be checked),
• the "omp_num_threads" AT parameter,
• the number of particles.

Pros:

• very simple implementation: setoption/getoption is there, no need to interface the OpenMP library routines with Matlab, and a simple #ifdef-protected library call in atpass.c,
• Can be easily extended to other parameters (schedule…)
• No change in the integrators
• as simple in pyat

Cons:

• We may be left when leaving atpass with a small number of threads, possibly affecting other computations. But we can restore the dynamic behaviour at the end of atpass,
• more ?

Additionally, if could be interesting to know in Matlab if OpenMP is active or not, for instance through another read-only getoption parameter. Why not…

[swhite2401]

@lfarv this sounds good to me, small suggestion:
instead of using omp_get_max_threads(), or omp_get_thread_limit() we may define the limit as the number of CPUs

I am still a bit puzzled why OpenMP did not use the full CPU complement of grappa for instance

[lfarv]

How can we get the number of CPUs ?
And what happens if this number is larger than omp_get_thread_limit() ?

I didn't go through the OpenMP doc to see how its limits are determined and how the so-called "dynamic adjustment of the number of threads" works…

[swhite2401]

I think you can use this: omp_get_num_procs
But I haven't gone much through the doc either, I don't really have time to run test these days. We may keep these questions open for later improvements, I think what you propose is already very good.

[lfarv]

Answer: omp_get_num_procs()
And apparently you may ask for less or more threads than processors…

[lfarv]

The control of the number of threads is implemented in #226 and #230