MPI: MPI array type for dispatch

[ranocha]

I think it might be good to implement a lightweight MPI array type wrapping a standard array. Then, we can dispatch on the MPI array type to distinguish parallel from serial runs (and we need to pass the coefficients anyway). Moreover, this allows us to redirect broadcasting etc. to the parent array and overload reductions to make use of MPI. In this way, we can also use error-based time step control from OrdinaryDiffEq.jl etc.

We may look at

• https://github.com/CliMA/ClimateMachine.jl/blob/master/src/Arrays/MPIStateArrays.jl
• https://github.com/fverdugo/PartitionedArrays.jl
• https://github.com/jipolanco/PencilArrays.jl
• https://github.com/barche/MPIArrays.jl

[sloede]

I think this is a great idea!

[ranocha]

Since this came up again in our meeting today: I added a few references to related array types to the first post. Right now, I expect we need the following featuers.

• An array type indicating that it's working with MPI. It should wrap some parent array and an MPI communicator, at least.
• It needs to support some common interface functions such as similar etc.
• It needs to support broadcasting to work with fixed step size RK methods
  • Broadcasting should fall back to the parent type. We should ensure that it's compatible with FastBroadcast.jl.
• It needs to support reductions to work with error-based adaptive time stepping, e.g.
  • https://github.com/SciML/DiffEqBase.jl/blob/03e584fe2c18d1c06d174bb5d043545518198e62/src/common_defaults.jl#L25
  • https://github.com/SciML/DiffEqBase.jl/blob/03e584fe2c18d1c06d174bb5d043545518198e62/src/common_defaults.jl#L2
• It needs to support resize! (if the parent array type does) to support AMR
• We need to think about how to handle our current Trixi.wrap_array working with PtrArray etc. with the new MPI array type

Further related notes:

• We might think about disabling unstable_check or setting it to something based on dt, see https://diffeq.sciml.ai/stable/basics/common_solver_opts/#Miscellaneous