*Enhance* simulation Callback and model StateModifier utilities

[glwagner]

@glwagner at some point it might good to think about whether exposing similar interfaces to DiffEq ( e.g. https://diffeq.sciml.ai/dev/features/callback_functions/#The-Callback-Types ) makes sense. Probably for the future.

Same could go for run! (e.g. https://diffeq.sciml.ai/dev/basics/integrator/#Initialization-and-Stepping ) too?

Ah wow, that is epic.

I think what's implemented here is more or less analogous to DiscreteCallback.

Reading over the features there makes me realize that we probably want to "align" the time-step (somehow, similar to the root finding feature that's provided for DiffEq) for callbacks. We align time-steps for output, but not for callbacks (yet). That would be nice to add.

I think we could also add an analog of the "continuous callback" --- to the models, not the simulations --- that's executed during update_state!, and therefore can be thought of as a function that "continuously" (eg valid at every moment in time) modifies the state of the model. Perhaps StateModifier or something like that? These objects would not have a schedule since they'd always be executed.

Initialization and finalization might be good wishlist features too for both.

Originally posted by @glwagner in #1894 (comment)

[glwagner]

cc @christophernhill

[glwagner]

cc @christophernhill

[aramirezreyes]

Probably related:

I saw the new Simulation API and the new Callback idea. This brought a change: the old way of building a Simulation included a parameters arguments. This parameters was accesible from the progress function. With the new way, we build callbacks and then register them, but there is no way to pass additional parameters to it.

In one of my usages I was hacking a progress function https://github.com/aramirezreyes/RamirezReyes_ShallowWaterInFPlane/blob/c1971c0cb3fc99fbb2dbf3303a8fd60b17a59d5a/scripts/run_oceananigans_example_cpu.jl#L101 to update some arrays on a parameterization. Unfortunately, this depended on the parameters argument. The more generalized idea of the callbacks is very elegant and works nice, but it loses some power from the lack of ability to add additional parameters to the callback function.

Is there a way to recover this ability? (there are other ways of getting it to work but having it in the design would be nice).

[glwagner]

I opened a new issue for parameterized Callbacks --- check out #2127

[glwagner]

We got this now