Use 4xA100 to achieve > 1SYPD for target EDMF AMIP

[LenkaNovak]

Since gpu_amip_topo_target_diagedmf is our current target, we want to run it on the faster nodes: either clima's A100 or new-central's V100 / H100.

Results

Running on Clima should be sufficient.

clima A100:

• current driver, 4 GPUs build SYPD ~ 1.51
• current driver w/o coupler diagnostics (still with atmos diags), 4 GPUs build SYPD ~ 1.63

new-central P100

• current driver, 1 GPU build SYPD ~ 0.23
• current driver, 4 GPUs build - memory timeout

Note

• currently this run crashes (saturation adjustment) after ~90-140 days (non deterministic).

Components in PR

• move gpu_amip_topo_target_diagedmf to clima
• increase t_end to 90d
• spilt clima pipeline init from individual runs, so we can run the multiple runs on clima
• remove concurrency block, which was slowing down the clima init, even though the depots are in different locations

[Sbozzolo]

What's the issue with the P100?

[LenkaNovak]

Apparently they're too slow and we're using this run as an SYPD benchmark.

[Sbozzolo]

If you are interested, this build https://buildkite.com/clima/climacoupler-longruns/builds/628 can be fixed by asking more memory per CPU

[LenkaNovak]

That's useful to know, thanks! We'll still run the benchmark on the A100 (as requested by the OKR), but this will be useful for the scaling tables (Cc'ing @juliasloan25 ).

[LenkaNovak]

Update:

The above results are for 200km resolution.

For 100km resolution SYPD on 4xA100: between 0.8 and 1.5 (see builds). This is partly dependent on whether we use coupler/atmos diagnsotics, but removing diagnostics didn't always lead to better SYPD. We also see a large variability between runs of the same config, and even within one simulation. More thorough investigation is being performed as part of CliMA/ClimaAtmos.jl#2914. And we will be presenting a like-for-like comparison and scaling as part of #663.

Notes:

• when server was busy, got these memory errors, some nodes kept running
• 100km edmf run runs to 90d without breaking build

[LenkaNovak]

Now achieving 1.082 https://buildkite.com/clima/climacoupler-longruns/builds/668#018f325b-a6d2-4c2d-b67f-1416484fda11

[Sbozzolo]

Now achieving 1.082 https://buildkite.com/clima/climacoupler-longruns/builds/668#018f325b-a6d2-4c2d-b67f-1416484fda11

While it should be reasonably accurate, I would encourage you not to look at the SYPD printed by the progress log. That is an estimate and does not reflect the actual SYPD in some cases (e.g, first iterations, when callbacks/diagnostics are called).

[LenkaNovak]

Very true, but for avoidance of doubt, this run shows we can achieve at least 380sim days in 1 day of walltime.