Trees, but not cells get distributed in P4estMesh?

[JoshuaLampert]

We should double-check that in a P4estMesh really the cells get distributed and not the trees. The issue came up in #1326 (comment).

[ranocha]

I'll assign this to you as a mid-term goal. It's great that you found this issue!

[lchristm]

I think this issue may be caused by the fact that we use p4est with the allow_coarsening=true flag. This causes p4est to keep families of cells on the same rank to allow coarsening the mesh easily. In the example in the linked comment, 96 elements on 24 and 25 ranks are used. So for 24 ranks p4est can always assign 4 cells per rank whereas for 25 ranks p4est might intentionally leave one rank empty.

This could be tested by setting allow_coarsening to false here, running the affected elixir with 24/25 ranks and comparing the partitioning. Note that this might break AMR, so it is probably better to disable it for this test.

[ranocha]

Thanks, @lchristm! Could you please test this, @JoshuaLampert? It would also be great if you could check how many ranks get no cells

[JoshuaLampert]

You're right, @lchristm. Setting allow_coarsening to false leads to an evenly balanced partitioning of the cells, i.e. in the case of 25 ranks in the elixir_advection_amr_unstructured_flag.jl there are 4 ranks that have 3 cells and 21 ranks that have 4 cells, as expected. With allow_coarsening=true and more than 24 ranks there were 24 ranks with 4 cells and all the others got 0, @ranocha. It didn't break even with enabled AMR, btw.

[ranocha]

Thanks, @JoshuaLampert! Did AMR try to coarsen cells that were distributed across multiple cores in your tests?

[JoshuaLampert]

I am not quite sure how to check that. Can you give me a hint?

[ranocha]

You could test it with indicators that always want to coarsen the mesh, similar to https://github.com/trixi-framework/Trixi.jl/blob/main/examples/tree_2d_dgsem/elixir_advection_amr_coarsen_twice.jl
If that works as well, we should consider changing the default.

[JoshuaLampert]

Thanks, @ranocha. I ran this elixir for a P4estMesh instead of the TreeMesh and allow_coarsening=false with many MPI ranks and it worked. But I think it only coarsened the mesh that is owned by the current rank and not across multiple cores. In one of my test cases according to the AMR part of the AnalysisCallback the root rank owned 25 elements on level 4 initially and in the end it owned 3 elements: 2 elements on level 3 and 1 element on level 2. I.e. it didn't coarsen its whole mesh twice since it didn't have enough cells to perform another coarsening (on the other hand it seems like it distributed 1 element to another rank since $2\cdot 4 + 1\cdot 4\cdot 4 = 24 = 25 - 1$). In another test case the root rank initially owned 2 cells on level 3 and in the end it still owned 2 cells on level 3, so at least for that rank it didn't coarsen at all.
Does that make sense?

[ranocha]

Thanks a lot, @JoshuaLampert!

@sloede What's your opinion - how shall we proceed? I expect there can be situations where both options for allow_coarsening can be better.

[sloede]

If I understand allow_coarsening correctly, it will disallow coarsening a set of 4 (2D) or 8 (3D) sibling cells if it is distributed across multiple ranks. This situation (splitting of siblings) can happen even in large simulations with "enough" cells to go around for each rank. Since that would mean that the result of the simulation may change depending on the number of ranks used, even in the case that there are enough cells, I think this is not an option for us. Thus, we should leave the allow_for_coarsening option true, which is the only sane thing to do in my opinion.

[ranocha]

Okay - so we will leave everything as it is. Is this GitHub issue enough to document this or do you think we should discuss it somewhere else - maybe in the FAQ section of the docs at https://trixi-framework.github.io/Trixi.jl/stable/troubleshooting/?

[sloede]

@JoshuaLampert a definitive test would be if you create a P4estMesh with 2x2 trees and ref level 1. This should give you 16 cells, right? If you now set allow_for_coarsening=false, run on 3 MPI ranks, and let it coarsen once, you should end up with not 4 but either 7 or 10 cells (depending on where the space filling curve is cut).

[sloede]

I think it would be good to document this somewhere either in a docstring or the Trixi manual.

[ranocha]

I think it would be good to document this somewhere either in a docstring or the Trixi manual.

A docstring isn't really an option since we don't expose this functionality (unless someone calls partition! manually). Thus, it would be great if you, @JoshuaLampert, could summarize this behavior in a new section in https://github.com/trixi-framework/Trixi.jl/blob/main/docs/src/troubleshooting.md.

[JoshuaLampert]

@JoshuaLampert a definitive test would be if you create a P4estMesh with 2x2 trees and ref level 1.

I did that and I get as expected initially $5 + 5 + 6 = 16$ cells and after one coarsening step we have $3 + 3 + 4 = 10$ cells.

Thus, it would be great if you, @JoshuaLampert, could summarize this behavior in a new section in https://github.com/trixi-framework/Trixi.jl/blob/main/docs/src/troubleshooting.md.

I will do that.

[sloede]

@JoshuaLampert a definitive test would be if you create a P4estMesh with 2x2 trees and ref level 1.

I did that and I get as expected initially

Great! Thanks for confirming my mental model 🙂 Thus, I think this definitely settles it. IMHO once this is documented in Trixi.jl, we can close this issue.