Distributed triangulation: assertion when distributing DoFs #16272
Comments
|
Hi Mathias. This is an annoying bug and not related to I can replicate the issue by only creating the mesh (with 4 processes) and checking if cells should be ghost or not, based on the vertices: #include <deal.II/distributed/tria.h>
#include <deal.II/grid/grid_generator.h>
using namespace dealii;
int
main(int argc, char *argv[])
{
Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1);
// create mesh
const int dim = 3;
parallel::distributed::Triangulation<dim> triangulation(
MPI_COMM_WORLD,
typename Triangulation<dim>::MeshSmoothing(
Triangulation<dim>::smoothing_on_refinement |
Triangulation<dim>::smoothing_on_coarsening));
const double R = 2.;
const double r = 1.;
const unsigned int n_cells_toroidal = 6;
const double phi = 0.5 * dealii::numbers::PI;
dealii::GridGenerator::torus<dim, dim>(
triangulation, R, r, n_cells_toroidal, phi);
triangulation.refine_global(3);
// determine vertices connected to locally owned cells
std::vector<bool> mask(triangulation.n_vertices(), false);
for (const auto &cell : triangulation.active_cell_iterators())
if (cell->is_locally_owned())
for (const auto v : cell->vertex_indices())
mask[cell->vertex_index(v)] = true;
// determine problematic cells: cells that are not ghost cells but are
// conncted to vertices belonging to locally owned cells
std::vector<typename Triangulation<dim>::cell_iterator> problemtic_cells;
for (const auto &cell : triangulation.active_cell_iterators())
if (cell->is_locally_owned() == false)
{
bool is_ghost = false;
for (const auto v : cell->vertex_indices())
is_ghost |= mask[cell->vertex_index(v)];
if (is_ghost != cell->is_ghost())
problemtic_cells.emplace_back(cell);
}
// print problematic cells
for (const auto &cell : problemtic_cells)
{
// check if locally owned cells are face neighbors
bool face_neighbour = false;
for (const auto f : cell->face_indices())
if (cell->at_boundary() == false)
face_neighbour |= cell->neighbor(f)->is_locally_owned();
std::cout << cell->id() << " " << int(face_neighbour) << std::endl;
}
return 0;
}I don't know what the root cause of this issue is. @tjhei Do you have idea? The options are For now, you could try out I find this bug somewhat worrying, since I used the the torus geometry in the past (with DG; https://github.com/hyperdeal/hyperdeal/blob/3640c397c9a2bd78c80325ee97dc526319197087/examples/vlasov_poisson/cases/torus_hyperball.h#L111). |
|
I did not check closely, but for me the test case by @peterrum runs into an assertion in p4est: This would indicate that we either give an invalid mesh to p4est, or that p4est fails at connecting meshes and building the connectivity. Can you confirm? |
When compiling deal.II with the debug version of p4est, the program just crashes without throwing an exception on my side (also in serial). The stack trace indicates that the issue also occurs during the creation of the coarse grid (also in serial). Which is a good step 👍 |
|
If I see things correctly, p4est has problems with the orientation of the grid. If I orient the 2D base grid of the torus similarly to the 2D circle, I don't see the error. Can you please try this patch: |
|
Note that the numbers come from here: dealii/source/grid/grid_generator.cc Lines 4321 to 4322 in 901ec46 |
|
@kronbichler Great - I can confirm that the problem works fine with your patch - no assertion is thrown when distributing DoFs (and later on creating the sparsity pattern also works). |
|
I am not sure if it is an issue with p4est, or if p4est expects you to give a mesh in the "right" form, which would mean the version after my change. We would need to look up the documentation as to what the expectations from p4est are regarding the mesh and its orientation. Either way, I will commit the change and add the test case you suggested, so we can have a more robust code base for now. |
Hello,
I have a problem with a parallel distributed triangulation that occurs in a 3d scenario when using at least three levels of refinements and at least 4 MPI processes when distributing the degrees of freedom.
The problem is dependent on the mesh and it occurs when using the torus mesh of the GridGenerator class.
I tested this on dealii 9.5.1 and todays master branch. Here is a minimal working example:
distributing_crash.zip
If I run it with at least -np4 I get the following assertion:
I attached a debugger and looked at some variables when the assertion is thrown (it is thrown on proc0):
The minimal working example runs fine in release mode - nevertheless in my production code the program crashes later on when creating the sparsity pattern - and I think this is a consequence of a problem when distributing the DoFs:
Other problems with other meshes (e.g. channel with cylinder) run fine - even with finer meshes on more processes. It is just that I need a torus-like mesh now.
Maybe anybody has an idea what is going wrong here or can help me with this?
Greetings
Mathias
The text was updated successfully, but these errors were encountered: