Finalize T8codeMesh before MPI is finalized

[jmark]

When finalizing the cmesh object in t8code some MPI calls are made since some shared memory arrays must be freed.
Naturally, this has to happen before MPI itself shuts down. Otherwise you get one of the following error messages:

Attempting to use an MPI routine (MPII_Comm_get_attr) before initializing or after finalizing MPICH

Julia uses a garbage collector, thus the order of finalization of objects is not determined by the program flow.

Is there a possibility in Julia resp. Trixi to deterministically finalize modules/objects before MPI shuts down?

[sloede]

Could this be helpful? https://juliaparallel.org/MPI.jl/stable/reference/environment/#MPI.add_finalize_hook!

[ranocha]

There seem to be two cases where clean-up code could be called:

• by a finalizer as implemented in a previous version of the code
• by a finalizer hook of MPI.jl

It would be nice to run the first in a long-running (interactive) Julia session using multiple meshes. However, only the second one is guaranteed to be called before MPi is finalized - finalizer hooks are called in MPI.finalize just before MPI is finalized, see
https://github.com/JuliaParallel/MPI.jl/blob/2a2bd68a6dc53660108bbc16690e4972de6318d5/src/environment.jl#L250-L256

How would you weight the requirement to avoid piling up garbage without a classical finalizer, @sloede? Would an approach like adding both a classical finalizer and an MPI finalize hook work - where both use the same implementation under the hood and check whether they need to clean up at all?

[jmark]

I just implemented a version with the MPI finalizer hook. It works as intended. But indeed, this does not solve the problem with long-running sessions. For that I did not find a satisfying solution yet on how the two finalizers know about each other.

[sloede]

I would probably continue to use the normal, per-object finalizer function, but guard the actual calls to any MPI functions behind if !MPI.Finalized() ... end. That way you get complete finalization/cleanup during long running sessions. And if something is not fully cleaned up after MPI has been finalized, you usually do not care anymore anyways.

[jmark]

Thanks for the thought, @sloede !

This is my proposed solution: dde2802

I still want to make sure that T8coeMesh is properly cleaned up before shutdown since sc_finalize is called by the
MPI finalizer hook. A feature I added in a previous commit: 6074e87

sc_finalize checks if all references resp. memory allocated by t8code has been properly freed and throws an exception if this is not the case. This assures code quality and might be a sensible addition for P4estMesh, too.

[ranocha]

sc_finalize checks if all references resp. memory allocated by t8code has been properly freed and throws an exception if this is not the case. This assures code quality and might be a sensible addition for P4estMesh, too.

This sounds really good. Would you be willing to make another PR for the P4estMesh?

[jmark]

There still might be one issue which I haven't considered yet. When creating a closure like this

 MPI.add_finalize_hook!(function ()
                                   trixi_t8_unref_forest(mesh.forest)
                               end)

then there is still a reference to mesh dangling around even when the instance of mesh is not needed anymore.
This would mean that finalizer actually never gets called even during long running sessions. Did I miss anything?

[ranocha]

Good point! So we should remove the finalizer hook again and just keep the ordinary finalizer with the check whether MPI has already been finalized in place?

[jmark]

Yes, we should do that. I made the sc_finalize logic optional. It can be activated by setting an environment variable. During production runs sc_finalize is not mandatory, but will be helpfull during development and debugging.

12a20e5