Unified interface for serializing triangulations.

[pcafrica]

This PR aims at providing a uniform interface for serializing/deserializing all different triangulation classes.

Before this PR, the most abstract interface was available in the parallel::DistributedTriangulationBase class. In order to abstract it to Triangulation, the following steps were/are required:

• Move CellAttachedData from P::DTB to grid/tria.h (internal namespace).
• Move DataTransfer from P::DTB to grid/tria.h (internal namespace) and renamed to CellAttachedDataSerializer. This class currently is hard-coded to use MPI:
  • DataTransfer depended on aliases and members defined in the Triangulation base class, in particular CellStatus, which has been moved to a separate grid/cell_status.h file.
  • The goal is to use a new hierarchy of classes polymorphically, so that T stores a pointer to an abstract DataTransfer interface.
  • Two classes DataTransferSerial and DataTransferParallel will inherit from DataTransfer (still to implement in the new grid/data_transfer.h file). (not possible since DataTransfer is templatized).
  • T and p::DTB will take care of initializing DataTransfer.
  • Implement DataTransfer for the case MPI support is disabled.
• Move save(std::string filename) and load(std::string filename) from p::DTB to T, possibly avoiding clashes with the Boost serialization methods (template <Archive ar> save/load(ar, version)). (Done in Move the save and load function to serial tria #15615)
• Changed interface to DataTransfer: it does not store anymore an MPI communicator as a member, because being it constructed now from T, that would always be MPI_COMM_SELF. Rather, the MPI communicator is now taken as an input by all the DataTransfer methods that needs it.
• Fix documentation references.

@blaisb @peterrum @bangerth

[pcafrica]

@marcfehling @blaisb @peterrum @bangerth anybody willing to review this?

The unchecked listed items will be addressed in subsequent PRs.

[blaisb]

@pcafrica I will review it first thing in the morning :)

[blaisb]

Just a tiny comment. Nice changes really. I did not find anything "wrong" that caught my eyes. The only worry I would have would be that this will break all user code that use the cell status from the triangulations (including ASPECT). I don't know if there could be a provision to keep the cell status defined within the triangulation as an alias?

[pcafrica]

Just a tiny comment. Nice changes really. I did not find anything "wrong" that caught my eyes. The only worry I would have would be that this will break all user code that use the cell status from the triangulations (including ASPECT). I don't know if there could be a provision to keep the cell status defined within the triangulation as an alias?

Thanks for pointing out, my last commit should address this issue.

[peterrum]

Very good 👍

[pcafrica]

Thank you @peterrum! The last 4 commits should do it.

[peterrum]

Very nice!

[bangerth]

Here's the rest of my comments.

[bangerth]

Yes, getting very close!

[bangerth]

Once you make your last changes, you might also want to squash commits into a smaller number.

[tamiko]

Heads up. This pull request should create some noticeable fallout on the regression tester. It breaks at least our performance tests running on bullseye.

[bangerth]

This broke ASPECT and will likely break other codes. See for example geodynamics/aspect#5251, and error messages are of the following form:

FAILED: CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx.o 
/usr/bin/c++ -DDEBUG -DWB_WITH_MPI -I../include -Iinclude -I../contrib/catch -I../contrib/world_builder/include -isystem /opt/deal.II-master/include -isystem /opt/deal.II-master/include/deal.II/bundled -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -isystem /usr/lib/x86_64-linux-gnu/openmpi/include -isystem /opt/trilinos-release-13-2-0/include -isystem /opt/hdf5-1.12.2/include -isystem /opt/p4est-2.3.2/FAST/include -Werror -pedantic -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -Wno-parentheses -Wno-unused-local-typedefs -fopenmp-simd -pthread -O0 -ggdb -Wa,--compress-debug-sections -O3 -g0 -fdiagnostics-color=always -std=gnu++17 -Winvalid-pch -include /__w/aspect/aspect/build/CMakeFiles/aspect.dir/cmake_pch.hxx -MD -MT CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx.o -MF CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx.o.d -o CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx.o -c CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx
In file included from /opt/deal.II-master/include/deal.II/bundled/boost/function/detail/maybe_include.hpp:29,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/function/detail/function_iterate.hpp:14,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/preprocessor/iteration/detail/iter/forward1.hpp:58,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/function.hpp:71,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/signals2/signal.hpp:18,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/signals2.hpp:19,
                 from /opt/deal.II-master/include/deal.II/base/mpi.h:28,
                 from ../include/aspect/global.h:27,
                 from /__w/aspect/aspect/build/CMakeFiles/aspect.dir/cmake_pch.hxx:5,
                 from <command-line>:
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp: In instantiation of 'static R boost::detail::function::function_obj_invoker2<FunctionObj, R, T0, T1>::invoke(boost::detail::function::function_buffer&, T0, T1) [with FunctionObj = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&; T1 = dealii::CellStatus]':
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:931:38:   required from 'void boost::function2<R, T1, T2>::assign_to(Functor) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&; T1 = dealii::CellStatus]'
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:720:7:   required from 'boost::function2<R, T1, T2>::function2(Functor, typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&; T1 = dealii::CellStatus; typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type = int]'
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:1068:16:   required from 'boost::function<R(T0, T1)>::function(Functor, typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&; T1 = dealii::CellStatus; typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type = int]'
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:1121:5:   required from 'typename boost::enable_if_<(! boost::is_integral<Functor>::value), boost::function<R(T0, T1)>&>::type boost::function<R(T0, T1)>::operator=(Functor) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&; T1 = dealii::CellStatus; typename boost::enable_if_<(! boost::is_integral<Functor>::value), boost::function<R(T0, T1)>&>::type = boost::function<unsigned int(const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&, dealii::CellStatus)>&]'
/opt/deal.II-master/include/deal.II/bundled/boost/signals2/detail/slot_template.hpp:160:24:   required from 'void boost::signals2::slot<R(Args ...), SlotFunction>::init_slot_function(const F&) [with F = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; SlotFunction = boost::function<unsigned int(const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&, dealii::CellStatus)>; R = unsigned int; Args = {const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&, dealii::CellStatus}]'
/opt/deal.II-master/include/deal.II/bundled/boost/signals2/detail/slot_template.hpp:85:9:   required from 'boost::signals2::slot<R(Args ...), SlotFunction>::slot(const F&) [with F = aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>; SlotFunction = boost::function<unsigned int(const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&, dealii::CellStatus)>; R = unsigned int; Args = {const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&, dealii::CellStatus}]'
/__w/aspect/aspect/source/particle/world.cc:57:9:   required from 'void aspect::Particle::World<dim>::initialize() [with int dim = 2]'
/__w/aspect/aspect/source/particle/world.cc:[153](https://github.com/geodynamics/aspect/actions/runs/5511284508/jobs/10046617673?pr=3773#step:4:154)1:5:   required from here
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:137:22: error: no match for call to '(aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>) (const dealii::TriaIterator<dealii::CellAccessor<2, 2> >&, dealii::CellStatus)'
  137 |           return (*f)(BOOST_FUNCTION_ARGS);
      |                  ~~~~^~~~~~~~~~~~~~~~~~~~~
In file included from CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx:5:
/__w/aspect/aspect/source/particle/world.cc:58:11: note: candidate: 'aspect::Particle::World<dim>::initialize() [with int dim = 2]::<lambda(const cell_iterator&, dealii::Triangulation<2, 2>::CellStatus)>'
   58 |           [&] (const typename parallel::distributed::Triangulation<dim>::cell_iterator &cell,
      |           ^
/__w/aspect/aspect/source/particle/world.cc:58:11: note:   no known conversion for argument 2 from 'dealii::CellStatus' to 'dealii::Triangulation<2, 2>::CellStatus'
In file included from /opt/deal.II-master/include/deal.II/bundled/boost/function/detail/maybe_include.hpp:29,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/function/detail/function_iterate.hpp:14,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/preprocessor/iteration/detail/iter/forward1.hpp:58,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/function.hpp:71,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/signals2/signal.hpp:18,
                 from /opt/deal.II-master/include/deal.II/bundled/boost/signals2.hpp:19,
                 from /opt/deal.II-master/include/deal.II/base/mpi.h:28,
                 from ../include/aspect/global.h:27,
                 from /__w/aspect/aspect/build/CMakeFiles/aspect.dir/cmake_pch.hxx:5,
                 from <command-line>:
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp: In instantiation of 'static R boost::detail::function::function_obj_invoker2<FunctionObj, R, T0, T1>::invoke(boost::detail::function::function_buffer&, T0, T1) [with FunctionObj = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&; T1 = dealii::CellStatus]':
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:931:38:   required from 'void boost::function2<R, T1, T2>::assign_to(Functor) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&; T1 = dealii::CellStatus]'
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:720:7:   required from 'boost::function2<R, T1, T2>::function2(Functor, typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&; T1 = dealii::CellStatus; typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type = int]'
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:1068:16:   required from 'boost::function<R(T0, T1)>::function(Functor, typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&; T1 = dealii::CellStatus; typename boost::enable_if_<(! boost::is_integral<Functor>::value), int>::type = int]'
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:1121:5:   required from 'typename boost::enable_if_<(! boost::is_integral<Functor>::value), boost::function<R(T0, T1)>&>::type boost::function<R(T0, T1)>::operator=(Functor) [with Functor = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; R = unsigned int; T0 = const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&; T1 = dealii::CellStatus; typename boost::enable_if_<(! boost::is_integral<Functor>::value), boost::function<R(T0, T1)>&>::type = boost::function<unsigned int(const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&, dealii::CellStatus)>&]'
/opt/deal.II-master/include/deal.II/bundled/boost/signals2/detail/slot_template.hpp:[160](https://github.com/geodynamics/aspect/actions/runs/5511284508/jobs/10046617673?pr=3773#step:4:161):24:   required from 'void boost::signals2::slot<R(Args ...), SlotFunction>::init_slot_function(const F&) [with F = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; SlotFunction = boost::function<unsigned int(const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&, dealii::CellStatus)>; R = unsigned int; Args = {const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&, dealii::CellStatus}]'
/opt/deal.II-master/include/deal.II/bundled/boost/signals2/detail/slot_template.hpp:85:9:   required from 'boost::signals2::slot<R(Args ...), SlotFunction>::slot(const F&) [with F = aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>; SlotFunction = boost::function<unsigned int(const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&, dealii::CellStatus)>; R = unsigned int; Args = {const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&, dealii::CellStatus}]'
/__w/aspect/aspect/source/particle/world.cc:57:9:   required from 'void aspect::Particle::World<dim>::initialize() [with int dim = 3]'
/__w/aspect/aspect/source/particle/world.cc:1531:5:   required from here
/opt/deal.II-master/include/deal.II/bundled/boost/function/function_template.hpp:137:22: error: no match for call to '(aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>) (const dealii::TriaIterator<dealii::CellAccessor<3, 3> >&, dealii::CellStatus)'
  137 |           return (*f)(BOOST_FUNCTION_ARGS);
      |                  ~~~~^~~~~~~~~~~~~~~~~~~~~
In file included from CMakeFiles/aspect.dir/Unity/unity_25_cxx.cxx:5:
/__w/aspect/aspect/source/particle/world.cc:58:11: note: candidate: 'aspect::Particle::World<dim>::initialize() [with int dim = 3]::<lambda(const cell_iterator&, dealii::Triangulation<3, 3>::CellStatus)>'
   58 |           [&] (const typename parallel::distributed::Triangulation<dim>::cell_iterator &cell,
      |           ^
/__w/aspect/aspect/source/particle/world.cc:58:11: note:   no known conversion for argument 2 from 'dealii::CellStatus' to 'dealii::Triangulation<3, 3>::CellStatus'

(Taken from https://github.com/geodynamics/aspect/actions/runs/5511284508/jobs/10046617673?pr=3773 for geodynamics/aspect#3773 .)

Does anyone know why the attempted backward compatibility doesn't work here? We should really push a follow-up that restores backward compatibility.

[tamiko]

mhm I fully agree with @bangerth here.

This incompatiblity is not ideal and I think we can fix a large portion of it by simply adding a using declaration at all the old places with a DEAL_II_DEPRECATED annotation. Similarly, the old names can be made available again (and deprecated). See also geodynamics/aspect#5251 (review)

This kind of backwards compatibility should not be too costly.

[pcafrica]

@tamiko

A Triangulation<dim, spacedim>::CellStatus enum was provided with exactly the purpose of backward compatibility, but this doesn't seem to work: https://github.com/dealii/dealii/blob/master/include/deal.II/grid/tria.h#L2238

The reason is still unknown to me, I'm currently trying to produce a MWE that replicates the issue.

[tamiko]

@pcafrica Yes, I realize. I am investigating as well. Try compiling test_dependency/performance.timing_step_68.release according to the regression tester.

[tamiko]

@pcafrica One problem I see is that both objects are not the same. The compiler has no way of converting a ::dealii::CellStatus enum class into the Triangulation<dim>::CellStatus object. So it is likely necessary to simple using ::dealii::CellStatus within the Triangulation class instead of defining a new object there.

[tamiko]

Another subtle point is the difference that an enum class introduces in semantics compared to a plain enum (the former not allowing to assign an enum class object to the underlying type without excliplictly casting). But this should be OK.

[pcafrica]

@pcafrica One problem I see is that both objects are not the same. The compiler has no way of converting a ::dealii::CellStatus enum class into the Triangulation<dim>::CellStatus object. So it is likely necessary to simple using ::dealii::CellStatus within the Triangulation class instead of defining a new object there.

Unfortunately, using enums was not allowed before C++20: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1099r5.html

[masterleinad]

Unfortunately, using enums was not allowed before C++20: open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1099r5.html

That's not quite true. You cannot import them but still alias, see https://godbolt.org/z/4d5EejEaz.

[tamiko]

@masterleinad @pcafrica Let's see what the CI is saying wrt #15707

[pcafrica]

I might have found the non-backward compatible lines, where the new ::dealii::CellStatus is hard-coded:
https://github.com/dealii/dealii/blob/master/include/deal.II/grid/tria.h#L2415
https://github.com/dealii/dealii/blob/master/include/deal.II/grid/tria.h#L3781

By the way, aliasing looks promising and a better solution. Thanks for the fix!

[tamiko]

@pcafrica yw

With the alias this shouldn't be a problem any more. But just to make sure I will run #15707 through the full tester-tng regression tester.