Port preCICE integration to version 3 #666

davidscn · 2024-06-05T14:08:18Z

Since a while already, preCICE version 3 is released. This is a breaking release with a couple of changes.

The new preCICE API interface uses std::span to interface with user-provided vectors, which becomes a bit ugly in the current implementation, as we don't use and pass over full vectors, but we use a static array and only pass the 'active' entries over to preCICE, necessitating to specify the span range manually. I kept it in this way to make the migration a pure technical one.

While running the test case for the preCICE coupling, the fluid solver segfaulted at timestep number two somehwere in ExaDG. Since I don't have Trilinos compiled into my local deal.II, I tried using MultigridCoarseGridPreconditioner::PointJacobi (the AMG requires Trilinos). My best guess is that this causes trouble in the SolverCG (this is at least, where the segfault stems from), maybe it's not even supported. I need to take a closer look.

preCICE version 3 now also supports a proper time interpolation for coupling data. Therefore, I have to update the handling of time-related data later on to make this work with ExaDG and adaptive timestepping. Maybe I will open an issue on this, as I am not sure (a) where the BDF time integrator evaluates the spatial operator in time, i.e., at which time the coupling data has to be retrieved from preCICE and (b) where the time-step size calculation takes place in ExaDG and how to synchronize it with preCICE.

Still, I would first consider to pursue the technical update before adding new features.

davidscn · 2024-06-05T15:18:55Z

Hm compiling and running the whole setup in debug gives me

dealii/source/grid/tria.cc> in function
    const dealii::Manifold<dim, spacedim>& dealii::Triangulation<dim, spacedim>::get_manifold(dealii::types::manifold_id) const [with int dim = 2; int spacedim = 2; dealii::types::manifold_id = unsigned int]
Additional information: 
    No manifold of the manifold id 0 has been attached to the
    triangulation. Please attach the right manifold with
    Triangulation::set_manifold().

but even attaching some manifold in create_grid doesn't resolve the issue. I think there might be some helper triangulation in one of the utility functions called, which is lacking a manifold, but I can't figure out where (and which setting necessitates the manifold).

kronbichler · 2024-06-05T20:34:45Z

I did fix one of those cases in #667 - is that the place you were looking at? If not, I am happy to check in other places as well.

davidscn · 2024-06-06T06:48:27Z

I was looking at the perpendicular flap case. Trying the same as you suggested (using numbers::flat_manifold_id) doesn't resolve the issue here. I can also provide the full stacktrace, although it is a bit messy:

  Viscous step:
  Solver viscous step:                       CG
  Maximum number of iterations:              1000
  Absolute solver tolerance:                 1.0000e-08
  Relative solver tolerance:                 1.0000e-06
  Maximum size of Krylov space:              30
  Preconditioner viscous step:               PointJacobi
  Update preconditioner viscous:             false

--------------------------------------------------------
An error occurred in line <12502> of file <~/dealii/source/grid/tria.cc> in function
    const dealii::Manifold<dim, spacedim>& dealii::Triangulation<dim, spacedim>::get_manifold(dealii::types::manifold_id) const [with int dim = 2; int spacedim = 2; dealii::types::manifold_id = unsigned int]
Additional information: 
    No manifold of the manifold id 0 has been attached to the
    triangulation. Please attach the right manifold with
    Triangulation::set_manifold().

Stacktrace:
-----------
#0  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::Triangulation<2, 2>::get_manifold(unsigned int) const
#1  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::TriaAccessor<1, 2, 2>::get_manifold() const
#2  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: 
#3  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: 
#4  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::TriaAccessor<1, 2, 2>::center(bool, bool) const
#5  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::Triangulation<2, 2>::DistortedCellList dealii::internal::TriangulationImplementation::Implementation::execute_refinement_isotropic<2, 2>(dealii::Triangulation<2, 2>&, bool)
#6  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::Triangulation<2, 2>::DistortedCellList dealii::internal::TriangulationImplementation::Implementation::execute_refinement<2>(dealii::Triangulation<2, 2>&, bool)
#7  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::internal::TriangulationImplementation::PolicyWrapper<2, 2, dealii::internal::TriangulationImplementation::Implementation>::execute_refinement(dealii::Triangulation<2, 2>&, bool)
#8  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::Triangulation<2, 2>::execute_refinement()
#9  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::Triangulation<2, 2>::execute_coarsening_and_refinement()
#10  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::parallel::distributed::Triangulation<2, 2>::copy_local_forest_to_triangulation()
#11  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::parallel::distributed::Triangulation<2, 2>::execute_coarsening_and_refinement()
#12  ~/dealii/build/lib/libdeal_II.g.so.9.6.0-pre: dealii::Triangulation<2, 2>::refine_global(unsigned int)
#13  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}::operator()(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&) const
#14  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: void std::__invoke_impl<void, ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}&, dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&>(std::__invoke_other, ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}&, dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int&&, std::vector<unsigned int, std::allocator<unsigned int> > const&)
#15  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: std::enable_if<is_invocable_r_v<void, ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}&, dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&>, void>::type std::__invoke_r<void, ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}&, dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&>(ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}&, dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int&&, std::vector<unsigned int, std::allocator<unsigned int> > const&)
#16  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: std::_Function_handler<void (dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&), ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)::{lambda(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)#1}>::_M_invoke(std::_Any_data const&, dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int&&, std::vector<unsigned int, std::allocator<unsigned int> > const&)
#17  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: std::function<void (dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)>::operator()(dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > >, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::TriaIterator<dealii::CellAccessor<2, 2> > > > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&) const
#18  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: void ExaDG::GridUtilities::create_triangulation<2>(std::shared_ptr<dealii::Triangulation<2, 2> >&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::Triangulation<2, 2>::cell_iterator>, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::Triangulation<2, 2>::cell_iterator> > >&, ompi_communicator_t* const&, ExaDG::GridData const&, bool, std::function<void (dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::Triangulation<2, 2>::cell_iterator>, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::Triangulation<2, 2>::cell_iterator> > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)> const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)
#19  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: void ExaDG::GridUtilities::create_triangulation_with_multigrid<2>(ExaDG::Grid<2>&, ompi_communicator_t* const&, ExaDG::GridData const&, bool, std::function<void (dealii::Triangulation<2, 2>&, std::vector<dealii::GridTools::PeriodicFacePair<dealii::Triangulation<2, 2>::cell_iterator>, std::allocator<dealii::GridTools::PeriodicFacePair<dealii::Triangulation<2, 2>::cell_iterator> > >&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&)> const&, std::vector<unsigned int, std::allocator<unsigned int> >)
#20  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: ExaDG::FluidFSI::Application<2, double>::create_grid(ExaDG::Grid<2>&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)
#21  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: ExaDG::FluidFSI::ApplicationBase<2, double>::setup(std::shared_ptr<ExaDG::Grid<2> >&, std::shared_ptr<dealii::Mapping<2, 2> >&, std::shared_ptr<ExaDG::MultigridMappings<2, double> >&)
#22  ~/exadg/build_debug/applications/fluid_structure_interaction/perpendicular_flap/solver_precice: ExaDG::FSI::SolverFluid<2, double>::setup(std::shared_ptr<ExaDG::FluidFSI::ApplicationBase<2, double> >, ompi_communicator_t*, bool)
--------------------------------------------------------

From what I understand, this stems from the lambda called here

exadg/applications/fluid_structure_interaction/perpendicular_flap/application.h

Lines 388 to 394 in 27a6077

    
           GridUtilities::create_triangulation_with_multigrid<dim>(grid, 
        
                                                                   this->mpi_comm, 
        
                                                                   this->param.grid, 
        
                                                                   this->param.involves_h_multigrid(), 
        
                                                                   lambda_create_triangulation, 
        
                                                                   {} /* no local refinements */);

and the global refinement (a few lines above the cited code) therein. However, the triangulation within the lambda should contain the manifold after applying the suggestion, so I am not sure what else to change.

kronbichler · 2024-06-06T07:34:34Z

Thank you for the input, I can reproduce this as well. I need to have a deeper look at what happens.

kronbichler · 2024-06-06T08:04:08Z

It seems that this has the origin in dealii/dealii#11566. I did observe some strange entries in the manifold ids.

kronbichler · 2024-06-06T10:42:51Z

Regarding

While running the test case for the preCICE coupling, the fluid solver segfaulted at timestep number two somehwere in ExaDG. Since I don't have Trilinos compiled into my local deal.II, I tried using MultigridCoarseGridPreconditioner::PointJacobi (the AMG requires Trilinos). My best guess is that this causes trouble in the SolverCG (this is at least, where the segfault stems from), maybe it's not even supported. I need to take a closer look.

This should work. I also see a problem in applications/fluid_structure_interaction/cylinder_with_flag, see #664 (comment) - so there is something strange in the multigrid setup. I hope to find it soon.

kronbichler · 2024-06-06T10:43:59Z

With all this minor issues, I did not have the change to thank you for the actual porting work, this is highly appreciated! I will look into it soon.

davidscn · 2024-06-06T14:25:02Z

With all this minor issues, I did not have the change to thank you for the actual porting work, this is highly appreciated! I will look into it soon.

Thank you for your efforts. Do let me know in case I can test something!

davidscn added 12 commits June 5, 2024 15:59

Rename Solverinterface to Participant

0a427a0

Remove mesh_id (not existent anymore)

7ee6208

Update name of the steering methods

cfa5dcb

Technical API changes

5cfeaf3

Get rid of 'unused' variable warnings when preCICE is disabled

fbfb0ce

Prevent compiler warnings about narrowing conversion

a87d6aa

Indentation with clang-format 11

1618b32

Update xml config to v3

3ed1029

Remaining xml config changes

8b7cf92

Rectify sizes of API functions

bf0624c

Indentation using clang-format 11

bb70944

Update debian package in GitHub CI

cd9747d

kronbichler mentioned this pull request Jun 6, 2024

merge_triangulations: Make sure to not copy any manifold ids dealii/dealii#17110

Merged

kronbichler mentioned this pull request Jun 6, 2024

Make sure to initialize multigrid preconditioner for steady linear structure #668

Closed

davidscn mentioned this pull request Jun 11, 2024

Interface data container does not take active faces into account #670

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port preCICE integration to version 3 #666

Port preCICE integration to version 3 #666

davidscn commented Jun 5, 2024 •

edited

Loading

davidscn commented Jun 5, 2024

kronbichler commented Jun 5, 2024

davidscn commented Jun 6, 2024

kronbichler commented Jun 6, 2024

kronbichler commented Jun 6, 2024

kronbichler commented Jun 6, 2024

kronbichler commented Jun 6, 2024

davidscn commented Jun 6, 2024

Port preCICE integration to version 3 #666

Are you sure you want to change the base?

Port preCICE integration to version 3 #666

Conversation

davidscn commented Jun 5, 2024 • edited Loading

davidscn commented Jun 5, 2024

kronbichler commented Jun 5, 2024

davidscn commented Jun 6, 2024

kronbichler commented Jun 6, 2024

kronbichler commented Jun 6, 2024

kronbichler commented Jun 6, 2024

kronbichler commented Jun 6, 2024

davidscn commented Jun 6, 2024

davidscn commented Jun 5, 2024 •

edited

Loading