ies_num_threads() vs physical cores

[wkitlasten]

The solves for each iteration takes quite a long time due to a ridiculous number of tiny local solves. I have set ies_num_threads(20) which is the number of physical cores on the host machine, but half the number of logical processors. My CPU usage maxes out at around 30% during the solves and memory usage is always less than 10%. There are other users on the machine, but they are not multithreading.

1. What problem specific conditions influence the optimal number of threads?
2. Given the 30% CPU usage, would increasing the number of threads speed up the solves or is there something else holding it back?
3. What are the risks of 2)?

From the pest++ namual (p133):
the localized upgrade solution process in PESTPP-IES has been multithreaded; this is possible in circumstances such as these where each local solve is independent of every other local solve. The use of multiple threads is invoked through the ies_num_threads() control variable. It should be noted that the optimal number of threads to use is problem-specific. Furthermore, it **should not exceed the number of physical cores of the host machine on which the PESTPP-IES master instance is running.**

[jtwhite79]

So, as you mention @wkitlasten there are lots of little solves with localization - essentially, one solve for each column in the localizer matrix for each lambda value. These solves require various linear algebra components like the simulated value ensemble and obs+noise ensemble for the non-zero localizer entries in the column, the current and initial par ensemble for the localizer column, the prior par cov matrix piece for the localizer column, etc. In a multithreaded setting, access to these containers has to be guarded by semaphores/mutexs to make them thread-safe, meaning one and only one thread can access each of these at a time. Then the solution process, which is independent for each thread. Then results of the solve have to be put into the result parameter ensemble container (which also has to be access controlled with semaphores/mutexs). What this means is that for problems with only a few reals (less than 500 say) and only a few non-zero weighted obs (less than a few hundred), the threads will spend almost all their time fighting/waiting for access to the containers, not actually solving - the SVD needed in the solution process is on a matrix of size num reals X nnz obs, so if those are large, then the SVD solve will take some time and help the threads not just wait on access. The "waiting for access" actually requires clock cycles so it chews up resources...all this is to say that in most cases 3-4 threads would probably be max (at least that is the most Ive used...). Ive been dreaming about offloading the localized solve to the workers for a while but its a massive undertaking...and in most cases, just using more reals is probably faster than waiting a fully localized solve...

[cnicol-gwlogic]

@wkitlasten - I always use ies_autoadaloc(true) and ies_use_approx(false). I'll most often have something like 10k-40k parameters. Number of groups, say around 10-20. I usually start with 300 realz, then bump it up if I need to. Cheers.

[wkitlasten]

I'm a big fan of the autoadaloc, but it sure does increase the parameter upgrade calcs! I've wondered if there is a way to localize by some "post_autoadaloc" grouping based on the correlation coefficients or something to reduce the number of solves.

[cnicol-gwlogic]

Ah I see. So autoadaloc tends to result in a lot of localiser blocks, each of which then has to be churned through the upgrade calcs? I've always been a bit wary of trying manual localisation as I tend to have a lot of data, often responsive across much of the model (eg deep confined systems under stress). Maybe I could get away without localisation in those cases, and up the numrealz a bit; never tried that.

[wkitlasten]

That is my understanding. A good approach seems to be localize by groups using obvious relationships (e.g. pumping rate pars applied after water level obs shouldn't influence those obs, porosity parameters shouldn't influence steady-state head obs, etc) and then, as you say, increase num_reals. I think there is plenty of opportunity for interesting numerical experimentation... likely problem specific though.

[laat0003]

Thanks @wkitlasten and @cnicol-gwlogic for this discussion. It's cleared up a few things for me about localisation and how I can make better use of it. I also always use ies_autoadloc but wasn't clear on exactly how this was impacting the solve.