Keep snapshot PETSc I/O paths compact - #667
Merged
lmoresi merged 2 commits intoSep 2, 2026
Merged
Conversation
Use deterministic mesh_0000-style identifiers for PETSc-HDF5 snapshot bulk files instead of expanding a loaded mesh's complete source pathname. Preserve the original mesh name in wrapper metadata for exact restore matching. Add a regression proving long source names do not enter bulk filenames.
Keep snapshot artifacts beside their wrapper while changing into the artifact parent directory for native PETSc/HDF5 reads and writes. This prevents valid but long user output roots from crossing MPI-I/O pathname limits even after bulk filenames have been compacted. Apply the same short-path handling to exact checkpoint restore and selective variable extraction, and add a focused regression that verifies the helper restores the process working directory.
lmoresi
added a commit
that referenced
this pull request
Sep 2, 2026
#667 removed the largest single source of path growth: the snapshot backend expanded mesh.name -- the full source path for a mesh loaded from file -- into every generated filename, turning a 123-character mesh name into a 346-character output filename by itself. It now uses deterministic mesh_0000 stems and short relative names. The troubleshooting entry would otherwise have read as though nothing had been done about it. The advice still stands for every other writer, so the note says what the fix covers and what it does not rather than softening the section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv
lmoresi
added a commit
that referenced
this pull request
Sep 2, 2026
…indings (#645, #635) (#650) * Document the two HPC findings that were sitting in issues (#645, #635) Both were carefully measured reports that had nowhere to live except the issue tracker, so the next person to hit either would rediscover it. #645 -- a long output path segfaults parallel HDF5 output. The failure tracks the length of the full GENERATED filename, including the suffixes UW3 appends (output.mesh.U.00000.h5), and it fires well below the advertised limits: every component under NAME_MAX, PATH_MAX=4096, PETSC_MAX_PATH_LEN=4096, and a 286-character name still crashed. Written into the troubleshooting guide with the reported thresholds marked as observations on one stack rather than a portable limit, since the first unsafe length was never established. #635 -- Gadi rank placement for a MatMult-bound Stokes solve. ~97% of solve time inside MatMult with the solver path held fixed (four SNES, 24 KSP, same mesh hash), so runtime tracks throughput rather than convergence. The table goes into the performance guide. The binding result is deliberately NOT written as a recommendation. Explicit core binding was slower at 8 ranks, but the runs were concurrent on different nodes and a repeat of the same configuration varied by 8.6% on its own -- the node-to-node noise is the size of the effect. The doc says that outright rather than letting a reader take the table as advice. Neither issue is closed by this: #635 asks for an investigation we cannot run from here, and both keep their open questions listed in the text. Underworld development team with AI support from Claude Code * Record that the snapshot backend no longer amplifies output path length #667 removed the largest single source of path growth: the snapshot backend expanded mesh.name -- the full source path for a mesh loaded from file -- into every generated filename, turning a 123-character mesh name into a 346-character output filename by itself. It now uses deterministic mesh_0000 stems and short relative names. The troubleshooting entry would otherwise have read as though nothing had been done about it. The advice still stands for every other writer, so the note says what the fix covers and what it does not rather than softening the section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make model snapshots robust when meshes and benchmark output directories have long names on parallel PETSc/HDF5 stacks.
This PR changes only the snapshot backend and its focused tests:
mesh_0000bulk stems instead of expandingmesh.nameinto every filename;Related to #645. This does not change the general
Mesh.write_timestep()API or claim to fix long paths outside the snapshot backend.Root cause
A Zhong thermal-convection checkpoint loaded a reusable mesh whose
mesh.namewas its 123-character absolute source path. The snapshot backend sanitised that complete name into every PETSc bulk filename, producing a 346-character absolute filename. On Gadi's PETSc 3.25.4 / HDF5 1.12.2p / Open MPI 4.1.7 stack, the first collective write terminated with PETSc signal 11 and exit status 59.Compact mesh stems fixed that case, but a deliberately longer restart-validation directory still produced 253-271 character absolute paths and reproduced the crash. Passing only basenames to native I/O removes both sources of path growth while preserving the on-disk layout.
Gadi validation
All Zhong runs used the same
cellsize=1/16tetrahedral mesh, 16 MPI ranks, P2/P1/P1 fields, SUPG predictor-corrector, and quadrature degree 4.178007756178008332178008633178009196The final job completed in
7:24, used10.52 GBpeak memory, and wrote a215,179,680byte checkpoint. Immediate restore and endpoint replay errors were all exactly zero forU,P,T,Tdot, RMS velocity, mean temperature, and upper/lower Nusselt numbers.Tests
27 passedin185.12 s.0.0.