Report error and abort in CollectiveMutex during stack unwinding

[gassmoeller]

When we throw an exception inside a CollectiveMutex::ScopedLock the destructor of ScopedLock tries to unlock the CollectiveMutex. This can lead to hard to debug MPI deadlocks, because other MPI ranks are likely trying to communicate with the ranks that throw the exception, while the ranks that throw the exceptions wait in the MPI barrier to unlock the mutex. We should not try to communicate while an exception is uncaught, because we do not know if all processes threw or just a subset.

The current behavior made it very hard to debug geodynamics/aspect#4984.

[gassmoeller]

Actually, let me mark this WIP for now. Even with this change we run into a deadlock. I confirmed that we do not get into it if I call MPI_Abort in case an exception is uncaught, but I did not include that in this PR (because I want the exception to propagate up).

I first want to investigate where we get stuck, but afterwards: Would it be fair to say that if an exception is raised within a CollectiveMutex, we cannot recover anyway, because other ranks are likely waiting for communication? Should we make sure we do not run into a deadlock in this case by calling MPI_Abort (possibly after printing an error and the exception)? We do this already in the HDF5 output here. Or would this be the responsibility of the exception handler that catches the exception further up the call stack?

[tjhei]

Yes, I would do the same as what we do for HDF5 and for TimerOutput. Communication in a destructor is not going to be a recoverable situation, I think.

[bangerth]

I agree.

[gassmoeller]

Ok, I think I made the requested changes. Now we should abort whenever a CollectiveMutex is destroyed or unlocked (from a ScopedLock) when an exception is thrown. Let me know if you want me to change the error message.

[masterleinad]

Looks OK to me.

[drwells]

/rebuild