Pytest hangs at a barrier if one rank hits an exception

[jwallwork23]

I think we might have come across a bug where Pytest hangs upon reaching an MPI barrier if one of the MPI ranks hits an exception - see code snippet below. Is this a known issue?
cc @stephankramer @ddundo

Minimal failing example:

from mpi4py import MPI 
import pytest

@pytest.mark.parallel(nprocs=2)
def test_mine():

    if MPI.COMM_WORLD.rank == 0:
        raise ValueError("Test")
    MPI.COMM_WORLD.barrier()  # it hangs here if there is a barrier

[connorjward]

I think this is expected behaviour. This will deadlock even without using pytest and just running under mpiexec.

Is there any more information about the actual deadlock that you can provide? mpi-pytest has a parallel_assert function that might be useful for you.

[stephankramer]

Yeah, sorry there's a been a bit of miscommunication here. This is indeed fully expected: if some MPI process raises an exception but others do not, you get a deadlock on the remaining processes. This issue stems from a discussion we had a while ago where we were wondering if there's anything we could do to make the debugging of failing parallel tests a bit more transparent. The thing that's different from running it under pytest is that it doesn't even show the exception or its backtrace at all, it just hangs, because pytest catches the exception to be printed later (depending on options). This means that any parallel test that fails with an exception for any reason, unless it's been raised by all processes, is likely to not give any useful feedback and just hang. Ideally you'd have the test fail immediately if an (uncaught) exception is raised by any of the processes, and then for pytest to print the backtrace. So this is more of a question/feature request to ask whether this could be achieved somehow - e.g. I'd imagine you could hook into pytests catching of exceptions and have it call MPI_Abort().

Just looking into it now again. One option would be to use https://pypi.org/project/pytest-timeout/ so that at least you get it to print the exception on the failing process after some timeout - but then you'd have to carefully choose some safe enough time out for all your parallel tests, and the response would not be immediate.

[connorjward]

In our CI we use pytest-timeout for this exact reason.

Are you using mpi-pytest in "forking" or "non-forking" mode? I.e. with mpiexec on the outside or inside? If it is on the outside then things are generally more transparent. This is part of the change that I submitted to Thetis to work with our new containers. Note that mpiexec on the inside is no longer possible by default with Firedrake because we ship with OpenMPI instead of MPICH, which disallows nested MPI Init calls.

When I debug failing parallel tests I use tmux-mpi (made by Will Saunders). I run something like

$ tmux-mpi 2 python -m pytest my_failing_test.py --pdb

and then I can catch exceptions and step through each rank separately. David has something similar working for VSCode I believe.

[stephankramer]

Thanks, yes we've been using the traditional "forking" so far (this came up in the CI for https://github.com/mesh-adaptation/) - but we should change to non-forking there as well indeed. Good to see that this fixes the not getting any output for such cases, and we'll just have to set a timeout then indeed.

Yeah, I know how to do this locally - I usually do something like

mpiexec - n 2 xterm -e pytest ....

I was just hoping to get something more informative in the CI output - but the new set up with "non-forking" and pytest-split is an improvement already.

Just played around a bit with the idea of putting a MPI Abort in a hook - for example you can put the following in conftest.py:

from mpi4py import MPI
import traceback
def pytest_exception_interact(node, call, report):
    traceback.print_tb(call.excinfo.tb)
    MPI.COMM_WORLD.Abort()

which works to avoid the hang, but the problem is that it causes mpiexec to immediately kill all other processes and thus all of pytest, so you would loose info if the other processes also raise an exception. I guess you could give it a few seconds of sleep before it MPI-Aborts to give the other processes the change to print their stuff that's all a little messy and also it still kills the entire session if you're running a pytest session with multiple tests. So we'll just stick with a timeout

[connorjward]

You can also try setting PYOP2_SPMD_STRICT=1 in your CI which adds in a couple of checks and barriers to help identify where deadlocks are happening.

[connorjward]

Can this issue be closed?

[jwallwork23]

Can this issue be closed?

Yes, will close now. Thanks for the useful info @connorjward!