Skip to content

Keep snapshot PETSc I/O paths compact - #667

Merged
lmoresi merged 2 commits into
underworldcode:developmentfrom
gthyagi:bugfix/compact-snapshot-io-paths
Sep 2, 2026
Merged

Keep snapshot PETSc I/O paths compact#667
lmoresi merged 2 commits into
underworldcode:developmentfrom
gthyagi:bugfix/compact-snapshot-io-paths

Conversation

@gthyagi

@gthyagi gthyagi commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Make model snapshots robust when meshes and benchmark output directories have long names on parallel PETSc/HDF5 stacks.

This PR changes only the snapshot backend and its focused tests:

  • use deterministic mesh_0000 bulk stems instead of expanding mesh.name into every filename;
  • preserve the complete original mesh name/path in wrapper metadata for exact model matching;
  • keep snapshot artifacts beside the wrapper, but invoke native PETSc/HDF5 reads and writes from the artifact parent directory using short relative names;
  • apply the short-path handling consistently to checkpoint writes, exact model restore, and selective-variable extraction;
  • restore the process working directory even when native I/O raises.

Related to #645. This does not change the general Mesh.write_timestep() API or claim to fix long paths outside the snapshot backend.

Root cause

A Zhong thermal-convection checkpoint loaded a reusable mesh whose mesh.name was its 123-character absolute source path. The snapshot backend sanitised that complete name into every PETSc bulk filename, producing a 346-character absolute filename. On Gadi's PETSc 3.25.4 / HDF5 1.12.2p / Open MPI 4.1.7 stack, the first collective write terminated with PETSc signal 11 and exit status 59.

Compact mesh stems fixed that case, but a deliberately longer restart-validation directory still produced 253-271 character absolute paths and reproduced the crash. Passing only basenames to native I/O removes both sources of path growth while preserving the on-disk layout.

Gadi validation

All Zhong runs used the same cellsize=1/16 tetrahedral mesh, 16 MPI ranks, P2/P1/P1 fields, SUPG predictor-corrector, and quadrature degree 4.

State Job Longest relevant path Result
Before fix 178007756 346 chars Exit 59; 9,552-byte wrapper; empty bulk directory
Compact stems only 178008332 250 chars Exit 0; complete endpoint checkpoint
Long replay path before relative I/O 178008633 271 chars Exit 59 at midpoint checkpoint write
Complete fix 178009196 271 chars as an absolute path Exit 0; exact checkpoint restore and replay

The final job completed in 7:24, used 10.52 GB peak memory, and wrote a 215,179,680 byte checkpoint. Immediate restore and endpoint replay errors were all exactly zero for U, P, T, Tdot, RMS velocity, mean temperature, and upper/lower Nusselt numbers.

Tests

  • Gadi focused snapshot suite: 27 passed in 185.12 s.
  • Gadi 16-rank Zhong checkpoint/restart/replay: passed with every stored field and diagnostic error equal to 0.0.
  • Pre-fix four-rank short-path snapshot control: passed, supporting path length rather than checkpoint state as the failure owner.

Use deterministic mesh_0000-style identifiers for PETSc-HDF5 snapshot bulk files instead of expanding a loaded mesh's complete source pathname. Preserve the original mesh name in wrapper metadata for exact restore matching. Add a regression proving long source names do not enter bulk filenames.
Keep snapshot artifacts beside their wrapper while changing into the artifact parent directory for native PETSc/HDF5 reads and writes. This prevents valid but long user output roots from crossing MPI-I/O pathname limits even after bulk filenames have been compacted.

Apply the same short-path handling to exact checkpoint restore and selective variable extraction, and add a focused regression that verifies the helper restores the process working directory.
@gthyagi
gthyagi requested a review from lmoresi as a code owner September 2, 2026 01:33
@lmoresi
lmoresi merged commit f4551f9 into underworldcode:development Sep 2, 2026
2 checks passed
lmoresi added a commit that referenced this pull request Sep 2, 2026
#667 removed the largest single source of path growth: the snapshot backend
expanded mesh.name -- the full source path for a mesh loaded from file -- into
every generated filename, turning a 123-character mesh name into a 346-character
output filename by itself. It now uses deterministic mesh_0000 stems and short
relative names.

The troubleshooting entry would otherwise have read as though nothing had been
done about it. The advice still stands for every other writer, so the note says
what the fix covers and what it does not rather than softening the section.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv
lmoresi added a commit that referenced this pull request Sep 2, 2026
…indings (#645, #635) (#650)

* Document the two HPC findings that were sitting in issues (#645, #635)

Both were carefully measured reports that had nowhere to live except the
issue tracker, so the next person to hit either would rediscover it.

#645 -- a long output path segfaults parallel HDF5 output. The failure tracks
the length of the full GENERATED filename, including the suffixes UW3 appends
(output.mesh.U.00000.h5), and it fires well below the advertised limits: every
component under NAME_MAX, PATH_MAX=4096, PETSC_MAX_PATH_LEN=4096, and a
286-character name still crashed. Written into the troubleshooting guide with
the reported thresholds marked as observations on one stack rather than a
portable limit, since the first unsafe length was never established.

#635 -- Gadi rank placement for a MatMult-bound Stokes solve. ~97% of solve
time inside MatMult with the solver path held fixed (four SNES, 24 KSP, same
mesh hash), so runtime tracks throughput rather than convergence. The table
goes into the performance guide.

The binding result is deliberately NOT written as a recommendation. Explicit
core binding was slower at 8 ranks, but the runs were concurrent on different
nodes and a repeat of the same configuration varied by 8.6% on its own -- the
node-to-node noise is the size of the effect. The doc says that outright rather
than letting a reader take the table as advice.

Neither issue is closed by this: #635 asks for an investigation we cannot run
from here, and both keep their open questions listed in the text.

Underworld development team with AI support from Claude Code

* Record that the snapshot backend no longer amplifies output path length

#667 removed the largest single source of path growth: the snapshot backend
expanded mesh.name -- the full source path for a mesh loaded from file -- into
every generated filename, turning a 123-character mesh name into a 346-character
output filename by itself. It now uses deterministic mesh_0000 stems and short
relative names.

The troubleshooting entry would otherwise have read as though nothing had been
done about it. The advice still stands for every other writer, so the note says
what the fix covers and what it does not rather than softening the section.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants