Canonical PETSc-sync callback fires on fancy-indexed array copies — serial warning, parallel hang/crash

[lmoresi]

Symptom

Serial: mesh.redistribute_nodes(...) (and anything that touches mesh.Gamma_P1) emits swallowed log warnings:

Callback error in ... canonical_data_callback ...: could not broadcast input array from shape (64,2) into shape (65,2)

Results are unaffected in serial. In parallel the same defect is fatal: at np=2 the MMPDE mover crashes/hangs with General MPI error / MPI_ERR_BUFFER inside coord_dm.getGlobalVec(), with one rank spinning at 100% CPU (mismatched collectives).

Minimal repro

import sympy, underworld3 as uw
mesh = uw.meshing.UnstructuredSimplexBox(minCoords=(0, 0), maxCoords=(1, 1),
                                         cellSize=0.4, refinement=1, qdegree=2)
x, y = mesh.X
d = sympy.Abs((x - 0.35) * 0.866 + (y - 1.0) * 0.5)
h = sympy.sqrt(0.08**2 + (1.6 * d) ** 2)
mesh.redistribute_nodes(1.0 / h**2, slip_surfaces=True)   # 25 swallowed callback errors

Run the same under mpirun -n 2 and the mover dies with MPI_ERR_BUFFER / hangs.

Root cause

NDArray_With_Callback.__array_finalize__ (utilities/nd_array_callback.py:185) copies _callbacks and _owner onto every derived array — including fancy-indexed copies. A masked in-place update such as

n.data[nonzero] /= mag[nonzero, None]     # discretisation_mesh.py:_update_projected_normals

expands to __getitem__ (a copy), an in-place divide on the copy, then __setitem__ back into the parent. The in-place divide on the copy triggers the canonical PETSc-sync callback, which tries to pack the subset (64 rows — the box corner node has a zero-magnitude fallback normal and is masked out) into the full 65-row vec.

• Serial: the pack raises ValueError, is caught and logged (nd_array_callback.py:525); the later __setitem__ on the parent syncs correctly, so values survive. Cosmetic only.
• Parallel: the mask is partition-dependent. On the rank owning the zero-magnitude node, the pack raises before the collective sync (createSubDM + localToGlobal + globalToLocal in pack_raw_data_to_petsc); on other ranks the pack succeeds and enters the collectives. Mismatched collectives poison the communicator: MPI_ERR_BUFFER, then a crash or spin at the mover's next collective (getGlobalVec, mmpde.py:249).

The same pattern exists in the SwarmVariable canonical callback (swarm.py:473).

Fix direction

The canonical callback must only ever sync the canonical storage it was registered on (available in the closure), never a derived view/copy of it: skip the pack when the triggering array is a fancy-indexed copy (the parent __setitem__ performs the real sync on all ranks, symmetrically), and sync the full canonical when the trigger is a partial view. Fix both the MeshVariable and SwarmVariable callbacks.

Found while validating the 3D adaptivity capstone's redistribute→adapt composition in parallel; fix incoming on feature/adaptivity-3d-design.

Underworld development team with AI support from Claude Code