Print AMR info with Quiet/Verbose verbosity

[knelli2]

Currently AMR prints either debug info for each element which can result in a lot of output, or it prints nothing. It'd be nice if I could choose a verbosity that prints a summary of what happened with AMR after each phase change. I'm imagining the domain::diagnostic_info function that we print at the beginning of the simulation would be a good thing to print. It gives a good overview of how elements were redistributed (for h-refinement) and how grid points were redistributed (for p-refinement). We'll probably have to do a reduction to get all this info but that shouldn't be too bad.

[nilsdeppe]

This is likely not possible long-term because we will want to switch AMR to not be in its own phase. Additionally, while AMR is currently done effectively at Slab boundaries, we will need to be able to do it locally in space and time, which means the info you are asking for is not well-defined.

[knelli2]

Ah I see. However, I still think it would be nice to have an option to print something about what AMR is doing throughout the simulation. Maybe we need to aggregate this information somehow as AMR does its stuff locally then print every N slabs? I just don't want the only two options for diagnostics to be 100GB in spectre.out or absolutely nothing.

[nilsdeppe]

I agree! We definitely need something! Could we have an Event that collects info about grid structure in a way that's convenient to visualize? I'm not sure what that means (and we probably won't until we try several things), but that seems like what we might want

[knelli2]

What about something similar to ObserveFields except we don't observe any tensors and just dump the grid? Maybe we also dump some other diagnostics to the reductions file like min/max p/h refinement? Number of elements? Etc. And then there we can print this to terminal

[nilsvu]

We can also write every change that AMR makes and/or accumulated data like the number of times h/p refinement/coarsening is triggered to the reductions file. Then visualize this data.

[nilsdeppe]

Yea, I think we should add support for these things. We might not be able to do it in practice for all runs like SpEC does because it may be too much data. However, I think all the requests/suggestions here will be needed for debugging AMR or other things where AMR acts up as a symptom.