Crash detection

[AliceHarang]

Creation of a Crash detection to stop the run if the time-step collapses.

• Detection of the collapsing time-steps
• Output field variables
• Test on CI tests
• Test on real case

[AliceHarang]

Need to improve the crash detection:
Calculate the critical time-step function of the previous ones.

[CyprienBosserelle]

would make more sense to say :
XParam.endtime = XLoop.totaltime
with the same outcome.
i.e. when looking at the model status after the crash it will look like the model reached the expected endtime when in reality it stopped earlier.

[AliceHarang]

Good comments, I will make these changes

[CyprienBosserelle]

Add :
XLoop.nextoutputtime = XLoop.totaltime

this will enforce the request for output without changing the if statement below. same effect but cleaner I think.

[AliceHarang]

I will add this too.

[CyprienBosserelle]

Few things with this solution ATM:

1. This method doesn't see +INF crashes. Because dt will not collapse but instead grow and stabilize
2. I don't think I agree with the weighting. if anything I would simplify and not weighting (dtmin=dt)
3. We need a mechanism for selecting which dt criteria we use:
   the dt<XParam.dtmin or dt<0.1*Xloop.dtmin

On one end the dt<XParam.dtmin is simple and easy for user to control

On the other end dt<0.1*Xloop.dtmin may be more sensitive and detect a wider range of crash. but without a test crash model it is difficult to know .

Possible solutions

Some crash do not affect the model timestep/minimum timestep (e.g. +INF) but we can probably look at the largest time step as well as the smallest to find crazy values.
We have a user defined dtmin and dtmax that detects the crashes using both the minimum timestep and maximum timestep.
Timestep is a good value to look at because locally dt approx to (sqrt(g*h) + hypot(u,v))/dx

Implementing that technique would move the crash detection inside the timestep calculation but that is a more complex operation and more code changes. Should that detection be moved to post v1.0?

Do you agree?

I suggest we do:

• Simplify the min time step detection to be as defined by the user or default as is
• Add task to improve crash detection for version 1.XX

[AliceHarang]

This development is useful to detect the code crashes, locate the source and understand the origin of the crash.
It detects the crash based on the time steps and create a map of the variables at the last steps.
It has been test on a time steps getting too small during the simulation, with different zones output and multi-resolution grid.