Bug in parallel component-wise multigrid renumbering

[jodlbauer]

The multigrid version of DoFRenumbering::component_wise does not work in parallel and issues an error.

One cause may be that in e.g. compute_component_wise(...) (dof_renumbering.cc:809), the non-mg versions of dh.locally_owned_dofs() are used independent of the flag is_level_operation.

(Furthermore, I guess in component_wise(...) (dof_renumbering.cc:674), the size of the new numbers should also be related to the locally owned size, not the global one).

Fixing these, one eventually runs into an assertion in ParallelDistributed<DoFHandlerType>::renumber_mg_dofs(...) (dof_handler_policy.cc:4888), i.e. the feature is not implemented.

I do not quite understand the mechanics behind the renumbering policies etc, but what would it take to extend this to more general renumberings (as stated in the code)? Is it sufficient to change the index_sets, or is there (a lot) more involved?

MWE:
main.txt

[bangerth]

I think @tjhei is the only one with expertise in the parallel MG implementation. He may be able to say more.

That given, since you've already started going down that road, one approach is to fix one issue at a time. That may simply uncover the next issue, and you may run into unimplemented functionality eventually, but we're happy to take small patches that at least move along what's already there. Fixing ParallelDistributed<DoFHandlerType>::renumber_mg_dofs(...) might be a big ask, but you can help the person who eventually implements it if everything before that already works. In other words: Whatever you've already done, break it into small, self-contained patches and make pull requests out of them!

[bangerth]

The small testcase by @jodlbauer looks like this:

#include <iostream>

#include <deal.II/distributed/tria.h>
#include <deal.II/dofs/dof_handler.h>
#include <deal.II/dofs/dof_renumbering.h>
#include <deal.II/fe/fe.h>
#include <deal.II/fe/fe_q.h>
#include <deal.II/fe/fe_system.h>
#include <deal.II/grid/grid_generator.h>

int main(int argc, char *argv[])
{
   dealii::Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv);

   static const int                                  dim = 2;
   dealii::parallel::distributed::Triangulation<dim> tria(MPI_COMM_WORLD,
                                                          decltype(tria)::limit_level_difference_at_vertices,
                                                          decltype(tria)::construct_multigrid_hierarchy);
   dealii::GridGenerator::hyper_cube(tria);
   tria.refine_global(3);

   auto fe_scalar = std::make_unique<dealii::FE_Q<dim>>(1);
   auto fe        = std::make_unique<dealii::FESystem<dim>>(*fe_scalar, dim);

   dealii::DoFHandler<dim> dofhandler(tria);
   dofhandler.distribute_dofs(*fe);
   dofhandler.distribute_mg_dofs();

   dealii::DoFRenumbering::component_wise(dofhandler);
   std::cout << "Finished fine lvl renumbering" << std::endl;

   for (unsigned int lvl = 0; lvl < tria.n_global_levels(); lvl++)
   {
      dealii::DoFRenumbering::component_wise(dofhandler, lvl);
      std::cout << "Finished renumbering on lvl " << lvl << std::endl;
   }

   return 0;
}

[tjhei]

I think @tjhei is the only one with expertise in the parallel MG implementation. He may be able to say more.

I am not too surprised that something might be broken in the renumbering as that is not something I have used extensively. I will take a look.

[jodlbauer]

Finally managed to get the pull request #7176 done (correctly, hopefully). This fixes the first bugs, but still runs into the assertion in dof_handler_policy.

[tjhei]

A testcase is now in master as part of #7176 (tests/multigrid/renumbering_07.cc)

[bangerth]

So does #7176 fix the current issue? @jodlbauer ?

[tjhei]

No, it does not. We are missing the implementation for the renumbering when the owned sets change. It should not be that difficult, but there is definitely code missing.