cell_size() diverges on a rank that owns no cells (np=8, tiny mesh) — and that mesh already hangs in construction on development

[lmoresi]

Found while adversarially reviewing #697; not caused by it, filed so it is not lost.

What happens

A 14-cell UnstructuredSimplexBox at np=8 gives cells per rank [2,2,2,2,0,2,2,2]. Calling mesh.cell_size() then diverges — per-rank stacks from the #678 supervisor:

ranks 0,1,2,3,5,6,7   inside _assemble_cell_size   (discretisation_mesh.py:3296)
rank 4                already past it, in the next statement

Rank 4 is the starved one. _assemble_cell_size short-circuits on radii.size == 0 and returns before touching var.data, which its peers do evaluate.

Why this is not #697

• _assemble_cell_size is documented as deliberately rank-local, and its empty-partition early return dates from Nitsche BC: scale penalty by a local per-cell mesh size #275 (a7f0b11f) — long before Make mesh.cell_size() local and partition independent #692 or Take the cell radius from PETSc, and fix the radius accessors with it (#694) #697.
• The same mesh at np=8 hangs on development earlier still, during mesh construction, before cell_size() is reached. So the configuration is pathological independently, and a clean before/after attribution is not available at that size.

At realistic mesh sizes np=8 is green: both partition-independence guards pass at np=2/4/8, and test_1069 passes at np=2 and np=8 on the default local_h=True path.

What needs deciding

Two separate things, and they may have different owners:

1. Whether a rank owning zero cells is a supported state for cell_size(), or whether the early return needs to be paired with whatever its peers do. The rank-local claim in the docstring is the thing to check, not assume.
2. Why a 14-cell mesh at np=8 hangs during construction at all. That is the same family as Fix cell-family detection on empty MPI ranks #691 (isSimplex rank-local on an empty rank) and Passive swarm save/advection hangs or crashes on empty MPI ranks #679/fix: swarm empty-rank save deadlock (MPIO dtype) and estimate_dt reshape crash #680 (empty-rank swarm save), all found in the last week.

Reproduce with the supervisor so a hang reports instead of sitting:

pixi run -e amr-dev python scripts/mpi_supervisor.py --silence 60 -- \
  mpirun -n 8 python -u probe.py

Underworld development team with AI support from Claude Code

[lmoresi]

Fixed and demonstrated in #697 (97cade70).

There were two collectives behind that early return, not one. My first attempt hoisted only var.data and the job still hung, one step later, with the populated ranks in pack_raw_data_to_petsc — assigning into var.data fires the array's write-back callback, which is collective as well.

The early return is gone entirely. A starved rank walks the same path and writes a zero-length slice:

data = var.data                    # createSubDM + createGlobalVector
n = min(data.shape[0], radii.shape[0])
data[:n, 0] = radii[:n]            # pack_raw_data_to_petsc; n == 0 is fine

The reproducer is a region submesh, which is the realistic case and the one the maintainer pointed at — a submesh keeps only the cells carrying its label, so a partition whose share of the parent lies outside that region owns none. A box split at z=0.85 gives an outer slab with cells per rank [12, 11, 0, 19, 0, 0, 0, 0] at np=8: five starved ranks out of eight. It hung before and returns now.

Correcting the original report: I used an over-decomposed 14-cell full mesh. That was a poor choice — degenerate, and it hits an unrelated intermittent hang in mesh construction that masked the real defect. A full mesh having a zero-cell rank is an artefact of over-decomposition; a submesh having one is normal.

Guarded by tests/parallel/test_0779_submesh_cell_size_starved_rank.py, which asserts every rank returns (a regression hangs rather than fails) and separately that the field is still filled where cells exist, so the fix cannot be satisfied by never filling it.

Underworld development team with AI support from Claude Code