debug GPU conservation

[juliasloan25]

Purpose

Debug conservation difference between CPU and GPU #607

To-do

• save sim states to HDF5 daily
• only run CPU/GPU slabplanet sims on buildkite

Content

---

• I have read and checked the items on the review checklist.

[LenkaNovak]

Note, small degradation of conservation total at radiation timestep on CPU (FT64, so small but not negligible):

[LenkaNovak]

CPU (left) vs GPU (right) based on @juliasloan25's investigation:

• significant deviations in states of all models between GPU and CPU after 1 day, e.g. for atmos:
  
• given there was no deviation in states after 1 dt, let's try to keep radiation constant (dt_rad > 1day) as the next step.

[juliasloan25]

Note, small degradation of conservation total at radiation timestep on CPU (FT64, so small but not negligible):
...

Which runs are these plots from? It looks like we have dt_rad: 1hours for most of the runs here (or at least the AMIP runs - slabplanet is using whatever our default is), so I just to make sure we're attributing the conservation change to the right thing :)

[LenkaNovak]

Note, small degradation of conservation total at radiation timestep on CPU (FT64, so small but not negligible):
...

Which runs are these plots from? It looks like we have dt_rad: 1hours for most of the runs here (or at least the AMIP runs - slabplanet is using whatever our default is), so I just to make sure we're attributing the conservation change to the right thing :)

The plots above are was for slabplanet with static albedo. Thanks for setting off the new runs... 🤞

[LenkaNovak]

Hmm, keeping the rad constant doesn't seem to help. Would it be possible to run for 400s and add more debug plots in the coupling loop (with t appended to the plots' titles). Hopefully that'll help us narrow this down.

[juliasloan25]

Hmm, keeping the rad constant doesn't seem to help. Would it be possible to run for 400s and add more debug plots in the coupling loop (with t appended to the plots' titles). Hopefully that'll help us narrow this down.

Sure! Do you think 400s will be enough? I could also run for something like 6 hours and plot every hour

[LenkaNovak]

Hmm, keeping the rad constant doesn't seem to help. Would it be possible to run for 400s and add more debug plots in the coupling loop (with t appended to the plots' titles). Hopefully that'll help us narrow this down.

Sure! Do you think 400s will be enough? I could also run for something like 6 hours and plot every hour

I would try 400s it for now (in case the first time step is overridden by the initial step! and reinit, but hourly plots like you suggest would be the next thing to try ;-)

[LenkaNovak]

I'm seeing differences in conservation but no visible difference in the states. 🤔

It is likely a round off error. The good thing is that we can see the divergence after 400s, but to see where exactly it is coming from, we can

• 1. print the extrema of the plotted fields with more decimal places
• 2. save and exactly compare some key variable arrays (I would start with coupler fields and land fields)

[juliasloan25]

I've added tests to compare the states of each component model simulation at the end of the simulation between CPU and GPU. Looking at slabplanet with static albedo run for 400s, we see that the land and ocean states are approximately equal between CPU and GPU, but the land state is not. The max difference in specific atmos variables is shown below. Given that the only state with differences after 400s is the atmosphere, it will be helpful to set up CPU/GPU conservation tests in atmos.

slabplanet: albedo from static map (CPU vs GPU) - build
atmos_ρe_tot

julia> abs(maximum(cpu_atmos_state[:,1] .- gpu_atmos_state[:,1]))
0.005922795255173696

atmos_ρq_tot

julia> abs(maximum(cpu_atmos_state[:,2] .- gpu_atmos_state[:,2]))
1.5653893756352792e-10

atmos_ρ

julia> abs(maximum(cpu_atmos_state[:,3] .- gpu_atmos_state[:,3]))
2.7223778786833464e-7

atmos_uₕ[1]

julia> abs(maximum(cpu_atmos_state[:,4] .- gpu_atmos_state[:,4]))
0.17414431556971977

atmos_uₕ[2]

julia> abs(maximum(cpu_atmos_state[:,5] .- gpu_atmos_state[:,5]))
0.0863372049825557

[juliasloan25]

don't need this

[juliasloan25]

this has been resolved in #733