XIOS calc timestep warning and crash: incomplete dump

[Denis Sergeev (dennissergeev)]

I'm getting an odd error at a particular CRUN in my simulation. The CRUN in question crashes at the very beginning, at the point of reading a checkpoint file, with little to no error messages after this debug message:

20260709103110.200+0000:P00:DEBUG: Adding checkpoint field: restart_ozone to lfric_checkpoint_read
20260709103110.208+0000:P00:DEBUG: XIOS context definition closing

I also noticed that the previous CRUN (which completes successfully) has this rather cryptic warning:

20260708205750.865+0000:P000:WARN : Danger: start of this run is an operator calc timestep, but start of next run is not. Written dump will be incomplete. Next run must start with an operator calc timestep

How do I fix this? Thanks.

[Steve Mullerworth (stevemullerworth)]

In your configuration, the main timestep is dt. There is also something called the reference_reset_time (in the mixed_solver namelist) which I assume must be a multiple of dt in your configuration. Basically, each NRUN and CRUN has to be a whole number of reference reset times as the dump on a non-reference reset timestep is different to a dump on a reference reset timestep.

The same rule applies for some other science schemes (radiation, radaer and COSP), resulting in similar "Danger" messages when ending the run on a timestep that doesn't divide into their timestep.

[Denis Sergeev (dennissergeev)]

Ah okay, thanks a lot Steve Mullerworth (@stevemullerworth)!

Why didn't it fall over earlier then? This CRUN is day 81 (after 8 successful CRUNs).

[Steve Mullerworth (stevemullerworth)]

That's surprising! Assuming you are doing equal-length CRUNs, I don't know why it would have failed now but not previously!

An intermittent science failure would have been plausible, but this looks like a mismatch in the XIOS setup, which ought to be the same for each CRUN assuming some weird rounding error with the timestep calculations hasn't occurred.

The relevant code is in create_gungho_prognostics_mod is as follows. If the test passes, you get the Danger message. So it relates to both the first and last timesteps of the run. Based on your configuration, would you have expected this warning? And do you know if the warning occurred on preceding CRUNs?

dt = real(clock%get_seconds_per_step(), r_def)

reference_reset_freq = nint(reference_reset_time / dt, i_def)

! snip

 if (mod(clock%get_first_step()-1, reference_reset_freq) == 0 .and.       &

      mod(clock%get_last_step(),    reference_reset_freq) /= 0) then

`

[Denis Sergeev (dennissergeev)]

Thanks for the detailed explanation, much appreciated!

It does seem odd that the run didn't crash earlier. I think what's more likely is that I made a mistake when reloading the suite. I've restarted the whole simulation just in case so hopefully it will run ok.

Thanks again.