Topology build speed-ups and (including threading)

[garth-wells]

Add supported by threaded mesh graph building and other various performance improvements.

[michalhabera]

Make sure the meaning of num_threads is the same in all interfaces. Here for num_threads=1 you do spawn single thread and keep the caller waiting. I do prefer the other approach, where caller does the work and we spawn num_threads-1 threads, as done in #4270. It avoids any possible overhead and does not rely on the OS scheduling to make sure the caller is not assigned resources when it is waiting.

[garth-wells]

Make sure the meaning of num_threads is the same in all interfaces. Here for num_threads=1 you do spawn single thread and keep the caller waiting. I do prefer the other approach, where caller does the work and we spawn num_threads-1 threads, as done in #4270. It avoids any possible overhead and does not rely on the OS scheduling to make sure the caller is not assigned resources when it is waiting.

The changes in #4270 are inconsistent with the changes that came before it. Everywhere apart from in #4270, num_threads=0 means spawn no (zero) threads, and num_threads=n (n > 0) means spawn n threads. This also allows non-threaded and threaded code paths to be tested for the same use case of a 'single worker'.

[michalhabera]

Make sure the meaning of num_threads is the same in all interfaces. Here for num_threads=1 you do spawn single thread and keep the caller waiting. I do prefer the other approach, where caller does the work and we spawn num_threads-1 threads, as done in #4270. It avoids any possible overhead and does not rely on the OS scheduling to make sure the caller is not assigned resources when it is waiting.

The changes in #4270 are inconsistent with the changes that came before it. Everywhere apart from in #4270, num_threads=0 means spawn no (zero) threads, and num_threads=n (n > 0) means spawn n threads. This also allows non-threaded and threaded code paths to be tested for the same use case of a 'single worker'.

That's not true, see what GJK does,

dolfinx/cpp/dolfinx/geometry/gjk.h

Lines 539 to 542 in 02343da

	if (num_threads <= 1)
	{
	compute_chunk(0, total_size, q);
	}

or also what ordering does,

dolfinx/cpp/dolfinx/graph/ordering.cpp

Lines 194 to 198 in 02343da

	if (nt <= 1)
	{
	for (std::size_t i = 0; i < S.size(); ++i)
	lstmps[i] = create_level_structure(graph, S[i]);
	}

Both have for num_threads=1 serial, non-threaded codepath. This has to be unified, thats a primary concern. Other issues we can benchmark and fix uniformly later.

[garth-wells]

Update for num_thread consistency.