New Simulation API: more callbacks, less "progress"

[glwagner]

This PR overhauls the API for Simulation, TimeStepWizard, and printing of progress. It also simplifies the implementation of run!.

After this PR, Simulation no longer accepts the keyword arguments iteration_interval or progress. Instead, progress printing is achieved with callbacks, eg:

progress(sim) = @info "Iteration: $(iteration(sim)), time: $(time(sim))"
simulation.callbacks[:progress] = Callback(progress, IterationInterval(100))

It also refactors the TimeStepWizard so that it can be used as a callback, eg

wizard = TimeStepWizard(cfl=1.0, max_change=1.1, max_Δt=2minutes)
simulation.callbacks[:wizard] = Callback(wizard, IterationInterval(10))

This is a better design for a few reasons:

1. Adaptive time-stepping and progress printing are not longer arbitrarily constrained to occur on the same iteration interval.
2. Both progress printing and adaptive time-stepping can use any schedule (rather than only IterationInterval).
3. The simulation time-step is always simulation.Δt. No more shenanigans like simulation.Δt.Δt.

Eventually, we should also eliminate the "diagnostics" list so that we have only two lists of callback-like objects: simulation.callbacks and simulation.output_writers.

I think this resolves an issue or two but I need to find them. Also, I've so far only updated the examples. There are probably tests and validation cases that need to be updated for the new API as well.

This PR is an important step toward generalizing Oceananigans.Simulation so that it can be used by ClimaAtmos.jl. cc @bischtob @akshaysridhar

[glwagner]

Well, we are almost done with #1138 but not quite. We still need to nuke simulation.diagnostics for that.

[navidcy]

hm... shall I have a look? seems like test are failing?

[navidcy]

No more shenanigans like simulation.Δt.Δt.

:)

[glwagner]

hm... shall I have a look? seems like test are failing?

I think the tests are failing because of some detail about how the simulations are stopped / time-step is aligned at the end of a run --- the overall design can be reviewed!

[glwagner]

@tomchor @navidcy @simone-silvestri can any of you help review this PR?

[tomchor]

This is looking good! Thanks for going through all this effort.

This PR is definitely too big and complex for me fully grasp the details, but I helped with some minor comments wherever I could.

[navidcy]

If tests pass go for it

[tomchor]

Awesome PR!

Quick question: did we skip version 0.64? I seems like this PR bumped the version from 0.63.4 to 0.65.

[glwagner]

That was a mistake, so I'll bump back to 0.64...

[aramirezreyes]

Could someone take a look at this related comment? #1895 (comment) I would argue in favor of restoring the "parameters" argument or an alternative way of passing additional parameters to the callback function.