Rationalise discovery of available memory and CPUs

[benjaminhwilliams]

At present, there are various different approaches within DIALS for discovering the available memory and CPUs on the system. This leads to some patchy behaviour in certain environments, most notably when running on a Slurm-managed cluster node, which uses cgroups to establish the memory allocated to the job. At present, for example, dials.integrate will discover a Slurm node's entire memory and try to use it, rather than the memory limit imposed by the scheduler, and so it will frequently be killed by the OOM killer.

Rather than reinvent the wheel and contribute to the proliferation of standards within DIALS for discovery of resources, I've tried here to strip out all the existing memory and CPU discovery methods and replace them with tools cribbed from Dask and Dask Distributed, on the basis that the devs there have thought longer and harder about cross-platform compatibility than we have. I've also tried to preserve the commit history from those two projects, both to give credit where credit is due and also to provide some history that explains why things are done in this way.

As a side-effect of these changes, I've removed the fall-back option within DIALS integrate to treat swap space as a simple extension of virtual memory and try to allocate to it. Swapping by design doesn't seem to be a good idea and, in trying to test these changes, I've not actually been able to manufacture a situation in which it actually works — on a RHEL8 workstation with 16GB of RAM and 8GB of swap, the OOM killer still intervenes when dials.integrate exceeds the available virtual memory. And, of course, there's no swap anyway in an HPC context, so it's irrelevant there.

I've tested these changes fairly extensively using a Slurm cluster and a PC with RHEL8 and have no concerns but would welcome other tests on other platforms if people have the time and are willing.

[graeme-winter]

I also think we need to add a comment somewhere that the static limits (i.e. CPU_COUNT, MEMORY_LIMIT) should be used in preference to recalculating. If we had a DIALS contributors guide, I would put it there.

I also note that this is logically a generous limit, since it is the full RAM of the machine: how we use this is an interesting question. However you are explicitly not changing the behaviour in this regard so this comment is out of context & what you are suggesting here is probably a substantial improvement on what came before.

Thank you for the contribution, this is a substantial body of effort.

[benjaminhwilliams]

Hoping that merging main into this branch will have sorted test failures that seem unrelated to changes here.

[codecov]

Codecov Report

Attention: Patch coverage is 44.80198% with 223 lines in your changes are missing coverage. Please review.

Project coverage is 78.40%. Comparing base (e36f939) to head (85c455f).
Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2619      +/-   ##
==========================================
- Coverage   78.49%   78.40%   -0.10%     
==========================================
  Files         609      611       +2     
  Lines       75057    75284     +227     
  Branches    10712    10746      +34     
==========================================
+ Hits        58919    59028     +109     
- Misses      13970    14076     +106     
- Partials     2168     2180      +12