[BUG]: nanobind 2.11.0 fails mixed mesh tests

[drew-parsons]

Summarize the issue

Debian builds of dolfinx 0.10.0 using nanobind 2.11.0 (ABI 18) are failing on 32-bit architectures.

=========================== short test summary info ============================
FAILED test/unit/mesh/test_create_mixed_mesh.py::test_create_mixed_mesh - Typ...
ERROR test/unit/io/test_vtkhdf.py::test_read_write_mixed_topology - TypeError...
ERROR test/unit/io/test_vtkhdf.py::test_write_mixed_topology_data - TypeError...
ERROR test/unit/mesh/test_mixed_topology.py::test_mixed_cell_pairs - TypeErro...
ERROR test/unit/refinement/test_uniform.py::test_uniform_refine_mixed_mesh - ...
= 1 failed, 2847 passed, 100 skipped, 32 xfailed, 7 warnings, 4 errors in 1635.41s (0:27:15) =

It's the mixed mesh test test_create_mixed_mesh that fails, with create_mesh() errors also in the 4 other tests, mixed meshes in each case

e.g. armhf https://buildd.debian.org/status/fetch.php?pkg=fenics-dolfinx&arch=armhf&ver=1%3A0.10.0.post5-5%2Bb1&stamp=1770987666&raw=0

i386 https://buildd.debian.org/status/fetch.php?pkg=fenics-dolfinx&arch=i386&ver=1%3A0.10.0.post5-5%2Bb1&stamp=1770802493&raw=0

These 32-bit architectures use mpich rather than openmpi, but I guess it's not the MPI implementation at fault here

How to reproduce the bug

python3 -m pytest -v python/test -k "mixed_mesh or mixed_topology or mixed_cell"

Minimal Example (Python)

Output (Python)

____________________________ test_create_mixed_mesh ____________________________

    def test_create_mixed_mesh():
        nx = 7
        ny = 11
        nz = 8
        n_cells = nx * ny * nz
        cells: list = [[], [], []]
        orig_idx: list = [[], [], []]
        idx = 0
        for i in range(n_cells):
            iz = i // (nx * ny)
            j = i % (nx * ny)
            iy = j // nx
            ix = j % nx
    
            v0 = (iz * (ny + 1) + iy) * (nx + 1) + ix
            v1 = v0 + 1
            v2 = v0 + (nx + 1)
            v3 = v1 + (nx + 1)
            v4 = v0 + (nx + 1) * (ny + 1)
            v5 = v1 + (nx + 1) * (ny + 1)
            v6 = v2 + (nx + 1) * (ny + 1)
            v7 = v3 + (nx + 1) * (ny + 1)
            if iz < nz // 2:
                cells[0] += [v0, v1, v2, v3, v4, v5, v6, v7]
                orig_idx[0] += [idx]
                idx += 1
            elif iz == nz // 2:
                # pyramid
                cells[1] += [v0, v1, v2, v3, v6]
                orig_idx[1] += [idx]
                idx += 1
                # tet
                cells[2] += [v0, v1, v4, v6]
                cells[2] += [v4, v6, v5, v1]
                cells[2] += [v5, v6, v7, v1]
                cells[2] += [v6, v7, v3, v1]
                orig_idx[2] += [idx, idx + 1, idx + 2, idx + 3]
                idx += 4
            else:
                # tet
                cells[2] += [v0, v1, v2, v6]
                cells[2] += [v1, v2, v3, v6]
                cells[2] += [v0, v1, v4, v6]
                cells[2] += [v4, v6, v5, v1]
                cells[2] += [v5, v6, v7, v1]
                cells[2] += [v6, v7, v3, v1]
                orig_idx[2] += [idx, idx + 1, idx + 2, idx + 3, idx + 4, idx + 5]
                idx += 6
    
        n_points = (nx + 1) * (ny + 1) * (nz + 1)
        sqxy = (nx + 1) * (ny + 1)
        geom = []
        for v in range(n_points):
            iz = v // sqxy
            p = v % sqxy
            iy = p // (nx + 1)
            ix = p % (nx + 1)
            geom += [[ix / nx, iy / ny, iz / nz]]
    
        if MPI.COMM_WORLD.rank == 0:
            cells_np = [np.array(c) for c in cells]
            geomx = np.array(geom, dtype=np.float64)
        else:
            cells_np = [np.zeros(0) for c in cells]
            geomx = np.zeros((0, 3), dtype=np.float64)
    
        hexahedron = coordinate_element(CellType.hexahedron, 1)
        pyramid = coordinate_element(CellType.pyramid, 1, variant=1)
        tetrahedron = coordinate_element(CellType.tetrahedron, 1)
    
        part = _cpp.mesh.create_cell_partitioner(GhostMode.none)
        max_cells_per_facet = 2
>       mesh = create_mesh(
            MPI.COMM_WORLD,
            cells_np,
            [hexahedron._cpp_object, pyramid._cpp_object, tetrahedron._cpp_object],
            geomx,
            part,
            max_cells_per_facet,
        )
E       TypeError: create_mesh(): incompatible function arguments. The following argument types are supported:
E           1. create_mesh(arg0: MPICommWrapper, arg1: collections.abc.Sequence[ndarray[dtype=int64, shape=(*), order='C', writable=False]], arg2: collections.abc.Sequence[dolfinx.cpp.fem.CoordinateElement_float32], arg3: ndarray[dtype=float32, order='C', writable=False], arg4: collections.abc.Callable[[MPICommWrapper, int, collections.abc.Sequence[dolfinx.cpp.mesh.CellType], collections.abc.Sequence[numpy.ndarray[dtype=int64, writable=False]]], dolfinx.cpp.graph.AdjacencyList_int32], arg5: int | None) -> dolfinx.cpp.mesh.Mesh_float32
E           2. create_mesh(comm: MPICommWrapper, cells: ndarray[dtype=int64, shape=(*, *), order='C', writable=False], element: dolfinx.cpp.fem.CoordinateElement_float32, x: ndarray[dtype=float32, order='C', writable=False], partitioner: collections.abc.Callable[[MPICommWrapper, int, collections.abc.Sequence[dolfinx.cpp.mesh.CellType], collections.abc.Sequence[numpy.ndarray[dtype=int64, writable=False]]], dolfinx.cpp.graph.AdjacencyList_int32] | None, max_facet_to_cell_links: int | None = 2) -> dolfinx.cpp.mesh.Mesh_float32
E           3. create_mesh(arg0: MPICommWrapper, arg1: collections.abc.Sequence[ndarray[dtype=int64, shape=(*), order='C', writable=False]], arg2: collections.abc.Sequence[dolfinx.cpp.fem.CoordinateElement_float64], arg3: ndarray[dtype=float64, order='C', writable=False], arg4: collections.abc.Callable[[MPICommWrapper, int, collections.abc.Sequence[dolfinx.cpp.mesh.CellType], collections.abc.Sequence[numpy.ndarray[dtype=int64, writable=False]]], dolfinx.cpp.graph.AdjacencyList_int32], arg5: int | None) -> dolfinx.cpp.mesh.Mesh_float64
E           4. create_mesh(comm: MPICommWrapper, cells: ndarray[dtype=int64, shape=(*, *), order='C', writable=False], element: dolfinx.cpp.fem.CoordinateElement_float64, x: ndarray[dtype=float64, order='C', writable=False], partitioner: collections.abc.Callable[[MPICommWrapper, int, collections.abc.Sequence[dolfinx.cpp.mesh.CellType], collections.abc.Sequence[numpy.ndarray[dtype=int64, writable=False]]], dolfinx.cpp.graph.AdjacencyList_int32] | None, max_facet_to_cell_links: int | None = 2) -> dolfinx.cpp.mesh.Mesh_float64
E       
E       Invoked with types: Intracomm, list, list, ndarray, nanobind.nb_func, int

test/unit/mesh/test_create_mixed_mesh.py:86: TypeError

Version

0.10.0

DOLFINx git commit

debian_1:0.10.0.post5-5+b1
(debian package)

Installation

Debian packages (apt install) with dolfinx source from https://salsa.debian.org/science-team/fenics/fenics-dolfinx

Additional information

Reported on the debian bug tracking system https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1127672

[chrisrichardson]

Maybe need to set the dtype for the cells array to np.int64? These type errors are always hard to debug - I guess, if you can get in before the offending call, and check the types of all the arguments, that is the best.

[jorgensd]

Each cells array should definitely be int64, and I don’t think that is the default cast by numpy

[drew-parsons]

While inspecting it, consider also that nanobind 2.11.0 is also poisoning 64-bit architectures, causing extremely slow execution (so CI times out).
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1127800
https://ci.debian.net/data/autopkgtest/testing/amd64/f/fenics-dolfinx/68633180/log.gz

I get the slow-down on my own machine: dolfinx does not import (in good time) if I launch as a serial job python3 -c "import dolfinx", though it does import in good time if I run as MPI, even with 1 process, mpirun -n 1 python3 -c "import dolfinx".

And then test_vtkhdf.py seems to be sitting in its own layer of hell in an MPI test run.

nanobind 2.11.0 is making everything crazy. It feels like multiple different bugs though perhaps it's just one underlying problem.

[drew-parsons]

I get the slow-down on my own machine

The problem on my local computer was probably a cache conflict. serial dolfinx now loads normally after deleting the cache with rm -r ~/.cache/fenics/

[drew-parsons]

With the fenics cache refreshed, I can reproduce errors in the same 5 tests locally (on amd64) that failed on the 32-bit architectures.

The difference is that the tests pass on the 64-bit system (with openmpi) when run in serial. I reproduce failure when running in mpi, e.g. via

mpirun -n 2 python3 -m pytest -v -k "test_create_mixed_mesh or test_read_write_mixed_topology or test_write_mixed_topology_data or test_mixed_cell_pairs or test_uniform_refine_mixed_mesh" python/test/

Testing in mpi (openmpi) with 1 process (mpirun -n 1) succeeds however, same as a serial run.

It looks like nanobind 2.11 has introduced a sensitivity to MPI runtime conditions, which depends on the MPI implementation.

Evidently the problem is triggered with mpich even in a serial run, triggering the incompatible function arguments error.

openmpi on the other hand fails by timing out, with no error emitted, but only in an multiprocess job (n≥2).
The failing 64-bit (openmpi) test only sees one failing test because it gets a timeout rather than an error. So the test simply doesn't reach the other 4 failing tests.

[jhale]

What those tests have in common that they are using mixed topology meshes.

[chrisrichardson]

Works OK for me with nanobind 2.11.0 - so it must be some combination of other factors. Do you have e.g. a docker container with a failing case?

[drew-parsons]

We have the problem with the debian unstable build. I don't have a docker container myself but if it's possible set one up pointing at debian unstable instead of ubuntu, and use no installation instructions (i.e. not rebuilding from source) other than

sudo apt-get update
sudo apt-get dist-upgrade -y
sudo apt-get install -y fenicsx

(or without sudo if the docker contain is root)
then it ought to pull in the affected packages.

Timo Röhling has identified the nanobind commit causing the trouble, in the nanobind bug report he linked above.

[chrisrichardson]

I can confirm that adding the dtype np.int64 to the cells arrays fixes the problem. This is already fixed in main.