Use reference state in amip with diagnostic edmf longrun

[szy21]

The amip target with diagnostic edmf longrun is currently failing in the most recent build. We should fix it soon. Maybe try increasing hyperdiffusion and/or sponge.

Edit: the simulation runs fine with use_reference_state: false, which is what we set now in the main. But it would be good to change back to use_reference_state: true.

[LenkaNovak]

@szy21 Here are some plots at the time of crash after three hours. The NaNs are coming from radiation (maybe related to the rrtmgp issue?) but there are other aspects we should address in this config to ensure the spin up is as smooth as possible. For example, the ocean albedo (atmos is now released for the more realistic version), air density is too low over topo (maybe this will get better with sim time though), initial land T is perhaps too warm, and roughness lengths over land and ocean should be adjusted to more realistic values.

[LenkaNovak]

w fluctuations are pretty high around Antarctica too:

[szy21]

@szy21 Here are some plots at the time of crash after three hours. The NaNs are coming from radiation (maybe related to the rrtmgp issue?) but there are other aspects we should address in this config to ensure the spin up is as smooth as possible. For example, the ocean albedo (atmos is now released for the more realistic version), air density is too low over topo (maybe this will get better with sim time though), initial land T is perhaps too warm, and roughness lengths over land and ocean should be adjusted to more realistic values.

Oh, great, thank you! If it's NaNs in radiation we know how to fix it. It should be already fixed in the latest RRTMGP version.

[szy21]

But this is still with Float64 radiation, I'm a bit surprised it could have issues too. Let's see if the latest RRTMGP version can fix it.

[szy21]

Reopen it to track if we can change back use_reference_state to true

[LenkaNovak]

This now runs to completion with the new update to ClimaAtmos and ClimaLand (see this build).

Note the albedo hooks need still need to be added (see #660 ), but getting close.

[szy21]

Thanks! I actually want to change use_reference_state back to true. I can start a build after the albedo hook is added.

[szy21]

It's still breaking after the update: https://buildkite.com/clima/climacoupler-longruns/builds/465#018e0aba-da1e-4b67-846b-1efe3742af3e. The latest climaatmos release should include all the radiation fixes, we should check if radiation still has NaNs.

[szy21]

Here is the plot for the breaking simulation at hour 3. Radiation seems fine. It might break due to the large vertical velocity near the topography. I will try increasing hyperdiffusion.

[LenkaNovak]

And this is with the ref_state set to true, right? Just a sec, is this using the latest main? And where does it break rn? (do you have the Build?)

[szy21]

Yes, everything I have been talking about is with use_reference_state: true. (use_reference_state: false works fine in the main.) Here is the build that breaks after ~2 hours. This is using the main after #632 is merged, but I don't think any latest change should change the result?

I start to think that there might be something wrong with the combination of reference state, topography, and edmf. Given that use_reference_state: false works fine, maybe we don't need to worry too much about it for now, if there is nothing obviously wrong. I'll think more carefully about the reference state.

[szy21]

The error message shows that there is a negative number in sqrt in RRTMGP. I don't know if it's because the state is wrong or something inside RRTMGP. I'll look into it.

[LenkaNovak]

Sounds good, thanks! And in atmos this combination of reference state, topography, and edmf? Even if we use the slab ocean? 🤔

[szy21]

No, we test edmf and topography separately in atmos as I think it makes more sense to have land with topography. But if this problem persists I can go back to aqualanet for testing. I did test ref state + no topo + edmf in coupler and it seems to work. I may add that to longrun if it’s consistently more stable.

[szy21]

This seems to work now with a stronger sponge: https://buildkite.com/clima/climacoupler-longruns/builds/478#018e1c1b-f16e-47f7-b6b7-dbaa79d48de2. I just realized that we have been using a stronger sponge for topography runs in the atmos, but not for edmf runs. As the coupler jobs use the toml file in atmos I'm using the (weaker) one for edmf. So this might be it. I think it might be easier to just specify the coupler toml file.