Reloading a checkpointed mesh built with refinement>=1 gives a corrupt DM: the integral of 1 is 0.07 on a unit square and the kinetic energy is negative

[lmoresi]

petsc_save_checkpoint on a mesh built with refinement=2 (UnstructuredSimplexBox(cellSize=4/64, refinement=2), 9824 cells, three-level hierarchy) writes a mesh file that uw.discretisation.Mesh(<mesh h5>) reloads with the same cell count and a rebuilt three-level hierarchy, but with the vertices in a different order: the same-index coordinate mismatch between the reloaded mesh and the mesh rebuilt by the original call is 1.0 (the domain size) at both the vertices and the P2 nodes. read_timestep maps the saved field by DOF index, so a P2 velocity read back onto the reloaded mesh is scrambled (the render is a jumble of triangles); read onto a mesh rebuilt by the original call it is correct. Meshes without refinement round-trip correctly (the Re 100 and Re 400 cavity runs from the same driver render fine).

The reload appears to re-refine the saved base rather than restore the refined DM, which renumbers the points. Workaround used in ~/+Simulations/navier_stokes_supg/scripts/render_ns.py (-uw_rebuild cavity:64:2): rebuild the mesh the way the run did and read the fields into that. Related to the split-mesh round-trip defect #640 in shape (a checkpoint that does not preserve the DOF map).

Underworld development team with AI support from Claude Code

[lmoresi]

Correction: the checkpoint reader is not at fault. read_timestep remaps by nearest neighbour onto the variable's own DOF coordinates, and on the reloaded mesh it placed exactly the saved values at every DOF (mismatch 0.0 against the field read onto a mesh rebuilt by the original call, matched by coordinate). The scramble is the reloaded mesh itself:

	reloaded from final.mesh.00000.h5	rebuilt by the original call
cells	9824	9824
hierarchy levels	3	3
integral of 1 over the unit square	0.0725	1.0000
integral of u.u	-0.0711	0.0709
coordinate section offsets	in point order	in point order
renderer's triangulation	cell areas 0 to 0.44, total 640 (a jumble)	correct

Same vertex count, same coordinate values, but the cell closures refer to vertices that do not carry those coordinates: the reload regenerates the refined topology by refining the saved base and then takes the fine-level coordinates from the file by index, which do not correspond to the regenerated point numbering. Every DM-level operation on such a mesh is wrong (integrals, solves), not only the rendering, so this is a real trap for restart from a refined mesh's checkpoint, and it is silent. Meshes without refinement reload correctly.

[lmoresi]

Where to look: the checkpoint reload with a hierarchy (discretisation_mesh.py, the sidecar block around lines 1095-1137) loads the saved fine coordinates as deformed_canon, re-refines the coarse sidecar on COMM_SELF for reference_canon, rebuilds the live hierarchy by distribute-then-refine, and leaves _pending_hierarchy_stamp = (reference_canon, deformed_canon, cdim) for a later step (uses at lines 687 689 1152 ) to stamp the saved coordinates onto the rebuilt fine DM. On this undeformed mesh the two canonical clouds hold the same points in different orders, and the stamped result has the cells pointing at the wrong vertices, so the stamping is by index where it needs a coordinate match (or the sidecar's refinement order differs from the live one). Not chased further here.