Add support for partitioning with PetscSF

[stefanozampini]

This implements a backend for partitioning based on the PetscSF object https://arxiv.org/abs/2102.13018

It comes in two different flavors:

• PETScWrappers::CommunicationPattern is an implementation of Utilities::MPI::NoncontiguousPartitioner
• PETScWrappers::Partitioner is an implementation of Utilities::MPI::Partitioner

There are a few differences wrt the dealII classes:

• API based: PETSc does not need explicit specification of the communication channels, and the user passing MPI_Requests and temporary storage
• the import_from_ghosted does not explicitly zero the ghost data after communication is performed and its signature is thus ArrayView<const ...> (I don't see what is the point of zeroing the ghost data within the internal API when users can do that themselves if needed)

Support for device memory can be easily added by just adding an extra template argument on the memory type of the ArrayView arguments, since PetscSF already support device memory pointers

Tests are copy-pasted from already existing tests.

[bangerth]

Ah, the famous star-forest finally makes an appearance :-)

[bangerth]

Here are some comment.

I don't know my way around the partitioners very well. It would be nice if @kronbichler or @peterrum would take another look at this, but I think most of it is pretty clear at least as far as the interface is concerned. Thanks for adding all of these tests!

[bangerth]

I don't understand this code. The IS object so created just gets a bunch of indices on every process, but no indication which of these are locally owned. How does it determine which ones of these are local?

[stefanozampini]

PetscSFSetGraphLayout(sf, layout, n, nullptr, PETSC_OWN_POINTER, idxs)) does this for you.

[stefanozampini]

Using IS is superfluous here (I could have used a std::vector<PetscInt> instead), it is only to test the new functionality in IndexSet

[bangerth]

Still, I don't know what is happening in the three lines 66-68. You create an index set from a bunch of indices, and then ISGetLocalSize returns how many of these are stored on the current process: https://petsc.org/main/docs/manualpages/IS/ISGetLocalSize/ How does it know? Does make_petsc_is() internally partition indices into locally owned and ghost ones?

[stefanozampini]

the local size corresponds to the number of indices stored in the array, independent of their actual distribution.
IS does not know anything about global indices distribution, it is merely a class around an array of indices.

[stefanozampini]

So the indices returned are passed as "global indices I would like to receive" to the PetscSFSetGraphLayout call

[bangerth]

Let me know once you've got it in a state where you want me to take another look!

[stefanozampini]

@bangerth I think I have addressed your comments. Please take a look.
I have decided to implement get_mpi_communicator with a static variable for the moment, since otherwise I would have to change all the implementations of the partitioners. I will do all the get_mpi_communicator changes in a single PR

[stefanozampini]

/rebuild

[bangerth]

My apologies for not getting to this faster. This looks good now, many thanks!

Out of curiosity, why did you implement this instead of just using the existing classes that do the same thing? Are your classes substantially faster?