Add parameter for controlling GPU memory footprint

[maxpkatz]

PR summary

The CTU hydro allocates a lot of temporary Fab data for its computation. This can result in a memory footprint on GPU builds that is significantly higher than the memory of the StateData itself. A new option, castro.hydro_memory_footprint_ratio, is added which can help limit the amount of temporary Fab data active at any one time.

PR checklist

• test suite needs to be run on this PR
• this PR will change answers in the test suite to more than roundoff level
• all newly-added functions have docstrings as per the coding conventions
• the CHANGES file has been updated, if appropriate
• if appropriate, this change is described in the docs

[maxpkatz]

I tested this option on the Sedov problem on Perlmutter using a 256^3 grid with max_grid_size = 128, running across 4 GPUs. The baseline performance for this case is 81.1 zones / usec / rank, and the maximum Arena size over all ranks is 12.2 GB. Then, for various values of this new option, here are the raw values and ratios with respect to the baseline case:

castro.hydro_memory_footprint_ratio = 1.0:
Throughput: 54.8 (68%)
Arena size: 3.4 GB (28%)

castro.hydro_memory_footprint_ratio = 2.0:
Throughput: 63.8 (79%)
Arena size: 3.6 GB (30%)

castro.hydro_memory_footprint_ratio = 3.0:
Throughput: 71.6 (88%)
Arena size: 4.81 GB (39%)

castro.hydro_memory_footprint_ratio = 6.0:
Throughput: 77.1 (95%)
Arena size: 7.3 GB (60%)

So for this example, castro.hydro_memory_footprint_ratio = 3.0 strikes a fairly reasonable balance between throughput and memory size. I expect the optimal tradeoff point will be problem dependent and distribution mapping dependent, but it's reasonable to expect it will often be in the range of 2.0 - 4.0.

[zingale]

tests: http://groot.astro.sunysb.edu/Castro/test-suite/gpu/2022-04-29-005/index.html