Ghost particles are not generated across periodic boundaries

[blaisb]

Dear all,
The ParticleHandler has the capability to generate ghost particles in cells which are ghost for other subdomains. This is achieved by identifying if a cell has neighbouring cells that belong to another subdomain. When this is the case, the exchange_ghost() routine of the particle_handler sends the necessary information to generate the ghost on the other side.

To identify if a cell is a neighbour to another domain, the vertex_to_cell mapping is used and the subdomains connected to these vertices are identified. The main issue with this is if you have a periodic boundary condition. When there is a periodic boundary condition, the vertex are in essence duplicated at the periodic BC. So this vertex_to_cell container does not allow us to identify that the cell is indeed a ghost for another subdomain, because that subdomain is not connected to it by any cell.

What I suggest is to do the following:

• In the GridToolCache, add the information that is collected by GridTools::collect_coinciding_vertices. This way, the GridToolCache will have a map containing all vertex which are duplicated vertices.
• In the exchange_ghost routine of the particle handler, add a second loop where we loop over all the vertices which are duplicated and we identify the subdomains which are connected to these duplicated vertices. Then we use it to generate ghost particles.

In essence, this would mean that if a cell is a ghost for another subdomain by a connection through a vertex, or a vertex which is a duplication of its own vertex, then ghost particles would be generated.

What would be your take on this? We really need this feature for our projects, so I would like to be able to cook up a working solution as soon as possible.

@gassmoeller I know this will be of interest to you, but maybe @peterrum or @bangerth you also have better suggestions? I am all ears here :).

Suggested label : particles

[kronbichler]

I agree, that sounds like a good plan. I am not entirely sure if there are better choices performance-wise, as std::map is a bad data structure for this kind of things. However, that is unrelated to what you plan here and something that can be revisited later (if necessary).

[gassmoeller]

I agree that this may be a bug, I do not think I ever used or tested the ghost particle algorithm with periodic boundaries. Your approach in general sounds correct, and as @kronbichler pointed out we can look into the most efficient way later if we notice that it is relevant for performance.