sporadic python MPI unit test failure in test_sub_bbtree_box: incompatible constructor arguments

[drew-parsons]

After rebuilding dolfinx (0.3.0) against PETSc 3.16 (and scalapack 2.23, scotch 7.0.1), the test_sub_bbtree_box python unit test from python/test/unit/geometry/test_bounding_box_tree.py fails inconsistently under mpi (openmpi)

$ mpirun -n 2 python3 -mpytest test_bounding_box_tree.py  -v
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.9.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.9.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/projects/fenics/build/fenics-dolfinx/python/test/unit/geometry/.hypothesis/examples')
rootdir: /projects/fenics/build/fenics-dolfinx/python, configfile: setup.cfg
plugins: asyncio-0.18.2, xvfb-2.0.0, arraydiff-0.5.0, cov-3.0.0, hypothesis-6.36.0, flaky-3.7.0, astropy-header-0.2.0, remotedata-0.3.3, filter-subpackage-0.1.1, doctestplus-0.12.0, mpi-0.6, openfiles-0.5.0, mock-3.6.1
asyncio: mode=legacy
collecting ... cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/projects/fenics/build/fenics-dolfinx/python/test/unit/geometry/.hypothesis/examples')
rootdir: /projects/fenics/build/fenics-dolfinx/python, configfile: setup.cfg
plugins: asyncio-0.18.2, xvfb-2.0.0, arraydiff-0.5.0, cov-3.0.0, hypothesis-6.36.0, flaky-3.7.0, astropy-header-0.2.0, remotedata-0.3.3, filter-subpackage-0.1.1, doctestplus-0.12.0, mpi-0.6, openfiles-0.5.0, mock-3.6.1
asyncio: mode=legacy
collected 31 items                                                                                                                                                                                                                                                                       
collected 31 items                                                                                                                                                                                                                                                                       

test_bounding_box_tree.py::test_padded_bbox[True] 
test_bounding_box_tree.py::test_padded_bbox[True] SKIPPED (This test should only be run in serial.)                                                                                                                                                                                [  3%]SKIPPED (This test should only be run in serial.)                                                                                                                                                                                [  3%]
test_bounding_box_tree.py::test_padded_bbox[False] 
test_bounding_box_tree.py::test_padded_bbox[False] SKIPPED (This test should only be run in serial.)                                                                                                                                                                               [  6%]SKIPPED (This test should only be run in serial.)                                                                                                                                                                               [  6%]
test_bounding_box_tree.py::test_empty_tree 
test_bounding_box_tree.py::test_empty_tree PASSED                                                                                                                                                                                                                                  [  9%]PASSED                                                                                                                                                                                                                                  [  9%]
...
test_bounding_box_tree.py::test_sub_bbtree_box[7-ct0] 
test_bounding_box_tree.py::test_sub_bbtree_box[7-ct0] PASSED                                                                                                                                                                                                                       [ 87%]PASSED                                                                                                                                                                                                                       [ 87%]
test_bounding_box_tree.py::test_sub_bbtree_box[7-ct1] 
test_bounding_box_tree.py::test_sub_bbtree_box[7-ct1] PASSED                                                                                                                                                                                                                       [ 90%]FAILED                                                                                                                                                                                                                       [ 90%]
test_bounding_box_tree.py::test_sub_bbtree_box[13-ct0] 
test_bounding_box_tree.py::test_sub_bbtree_box[13-ct0] PASSED                                                                                                                                                                                                                      [ 93%]PASSED                                                                                                                                                                                                                      [ 93%]
test_bounding_box_tree.py::test_sub_bbtree_box[13-ct1] 
test_bounding_box_tree.py::test_sub_bbtree_box[13-ct1] PASSED                                                                                                                                                                                                                      [ 96%]PASSED                                                                                                                                                                                                                      [ 96%]
test_bounding_box_tree.py::test_surface_bbtree_collision 
test_bounding_box_tree.py::test_surface_bbtree_collision SKIPPED (This test should only be run in serial.)                                                                                                                                                                         [100%]SKIPPED (This test should only be run in serial.)                                                                                                                                                                         [100%]

==================================================================================================================================== warnings summary ====================================================================================================================================
../../../../../../../../../../usr/lib/python3/dist-packages/pytest_asyncio/plugin.py:191
  /usr/lib/python3/dist-packages/pytest_asyncio/plugin.py:191: DeprecationWarning: The 'asyncio_mode' default value will change to 'strict' in future, please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.
    config.issue_config_time_warning(LEGACY_MODE, stacklevel=2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================================================================================================= 21 passed, 10 skipped, 1 warning in 2.24s ========================================================================================================================


======================================================================================================================================== FAILURES ========================================================================================================================================
_______________________________________________________________________________________________________________________________ test_sub_bbtree_box[7-ct1] _______________________________________________________________________________________________________________________________

ct = <CellType.tetrahedron: 4>, N = 7

    @pytest.mark.parametrize("ct", [cpp.mesh.CellType.hexahedron, cpp.mesh.CellType.tetrahedron])
    @pytest.mark.parametrize("N", [7, 13])
    def test_sub_bbtree_box(ct, N):
        """
        Test that the bounding box of the stem of the bounding box tree is what we expect
        """
        mesh = UnitCubeMesh(MPI.COMM_WORLD, N, N, N, cell_type=ct)
        tdim = mesh.topology.dim
        fdim = tdim - 1
    
        def marker(x):
            return numpy.isclose(x[1], 1.0)
    
        facets = locate_entities_boundary(mesh, fdim, marker)
        f_to_c = mesh.topology.connectivity(fdim, tdim)
        cells = numpy.unique([f_to_c.links(f)[0] for f in facets])
>       bbtree = BoundingBoxTree(mesh, tdim, cells)
E       TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
E           1. dolfinx.cpp.geometry.BoundingBoxTree(mesh: dolfinx.cpp.mesh.Mesh, tdim: int, padding: float = 0.0)
E           2. dolfinx.cpp.geometry.BoundingBoxTree(mesh: dolfinx.cpp.mesh.Mesh, tdim: int, entity_indices: numpy.ndarray[numpy.int32], padding: float = 0.0)
E       
E       Invoked with: <dolfinx.cpp.mesh.Mesh object at 0x7f6d09f25ae0>, 3, array([], dtype=float64)

test_bounding_box_tree.py:464: TypeError

See also debian test logs at https://ci.debian.net/data/autopkgtest/unstable/amd64/f/fenics-dolfinx/20037551/log.gz or https://ci.debian.net/packages/f/fenics-dolfinx/

The failure is sporadic in the sense that if I run only test_sub_bbtree_box on 2 processes,

$ mpirun -n 2 python3 -mpytest test_bounding_box_tree.py -k sub_bbtree  -v

then sometimes it passes, sometimes it still fails. What would make it behave inconsistently like that?

Has anyone else experienced this kind of failure or know what's causing it? Is it likely to have been fixed since 0.3.0?

[jorgensd]

The error looks to be due to the fact that the input array has gotten the wrong dtype (float64) insstead of int32.
I guess this might be an issue whenever one calls: cells = numpy.unique([f_to_c.links(f)[0] for f in facets]) on an empty set

In [1]: import numpy as np                                                                                                                                                                                     

In [2]: a = np.unique([])                                                                                                                                                                                      

In [3]: a.dtype                                                                                                                                                                                                
Out[3]: dtype('float64')

In [4]: a = np.unique([1,2,3])                                                                                                                                                                                 

In [5]: a.dtype                                                                                                                                                                                                
Out[5]: dtype('int64')

On main, the unique has been removed:

dolfinx/python/test/unit/geometry/test_bounding_box_tree.py

Line 381 in 8c61d6f

cells = [f_to_c.links(f)[0] for f in top_facets]

[drew-parsons]

Thanks @jorgensd . I'll try patching it without unique and see if that clears it up.

[drew-parsons]

the unique is removed in 0.4.1 and the tests now pass reliably, so can close this bug now.