Summary
Parallel UW3 HDF5 output can terminate with a native PETSc segmentation fault when the full generated output filename becomes long, even though the path is valid for the filesystem and shorter than PETSc's advertised path limit.
This was isolated while running a layered spherical-shell benchmark on Gadi. A compact output directory fixed the failure without changing the mesh, labels, fields, Stokes solve, MPI size, or output operations.
Affected workflow
The failure was triggered by the normal UW3 timestep writer:
mesh.write_timestep(
"output",
index=0,
outputPath=output_dir,
meshVars=[velocity, pressure],
)The first generated production mesh filename was 284 characters:
/scratch/m18/tg7098/uw3-mantle-convection-benchmarks/output/zhong2008/stokes_response_layered_viscosity/output_response_layered_cellsize_0.125_ri_0.55_rlid_0.972_rint_0.775_ro_1_fitlid_true_P2P1_l_2_depth_0.5_harm_zhong_etaL_10000_bc_fs_rotated_stol_1e-05_ncpus_8/output.mesh.00000.h5
The corresponding P2 velocity filename was 286 characters:
/scratch/m18/tg7098/uw3-mantle-convection-benchmarks/output/zhong2008/stokes_response_layered_viscosity/output_response_layered_cellsize_0.125_ri_0.55_rlid_0.972_rint_0.775_ro_1_fitlid_true_P2P1_l_2_depth_0.5_harm_zhong_etaL_10000_bc_fs_rotated_stol_1e-05_ncpus_8/output.mesh.U.00000.h5
Controlled results
The same diagnostic, mesh, labels, solved fields, and write sequence were used while changing only output-path padding:
| Full filename |
Result |
| 242-character mesh / 244-character variables |
Pass |
| 252-character mesh / 254-character variables |
PETSc segmentation fault |
| 254-character mesh / 256-character variables |
PETSc segmentation fault |
| 255-character mesh |
PETSc segmentation fault |
| 284-character mesh |
PETSc segmentation fault |
A staged test failed during mesh.write() with a 259-character filename, before P2/P1 variable output, compatibility projection, or XDMF generation. Boundary-label removal and output-format variations did not explain the failure. Memory exhaustion was also excluded.
The exact first unsafe length was not established. Therefore, 252 characters should be reported as the shortest tested failure on this stack, not as a portable threshold.
Environment
- Gadi Linux HPC
- PETSc 3.25.4, custom UW Open MPI build
- HDF5 1.12.2p
- Open MPI 4.1.7
- MPI-enabled h5py 3.16.0
- GPFS filesystem:
PATH_MAX=4096, NAME_MAX=255
- PETSc configuration:
PETSC_MAX_PATH_LEN=4096
Every individual path component was below NAME_MAX. The failure therefore occurs below the advertised filesystem and PETSc full-path limits and appears to be in the active native PETSc/HDF5/MPI-I/O stack.
The issue has only been reproduced systematically on this Gadi stack. A macOS environment using PETSc 3.25.0, HDF5 1.14.6, and Open MPI 5.0.10 did not show the original benchmark failure, but the complete threshold matrix was not repeated there.
Validated workaround
Shorten the user-generated case/output directory and retain fixed configuration in the metrics file rather than encoding every fixed parameter in the directory name.
Reducing the longest production filename from 286 to 189 characters made the unchanged eight-rank benchmark pass end to end. A subsequent 192-rank, 1/64-resolution run also completed all mesh, Stokes, HDF5, XDMF, and postprocessing stages.
Proposed documentation
Add a note to the checkpoint/HDF5 output documentation:
- Full filenames include the user output path plus UW3-generated suffixes such as
output.mesh.U.00000.h5.
- Some native parallel PETSc/HDF5 stacks may fail well below filesystem
PATH_MAX.
- Keep output roots and generated case identifiers compact, particularly on HPC systems.
- Store detailed fixed parameters in HDF5 metadata or a metrics file instead of the directory name.
- A native segmentation fault during
mesh.write() or mesh.write_timestep() should prompt checking the full generated filename length.
Reproducer and investigation log
Summary
Parallel UW3 HDF5 output can terminate with a native PETSc segmentation fault when the full generated output filename becomes long, even though the path is valid for the filesystem and shorter than PETSc's advertised path limit.
This was isolated while running a layered spherical-shell benchmark on Gadi. A compact output directory fixed the failure without changing the mesh, labels, fields, Stokes solve, MPI size, or output operations.
Affected workflow
The failure was triggered by the normal UW3 timestep writer:
The first generated production mesh filename was 284 characters:
The corresponding P2 velocity filename was 286 characters:
Controlled results
The same diagnostic, mesh, labels, solved fields, and write sequence were used while changing only output-path padding:
A staged test failed during
mesh.write()with a 259-character filename, before P2/P1 variable output, compatibility projection, or XDMF generation. Boundary-label removal and output-format variations did not explain the failure. Memory exhaustion was also excluded.The exact first unsafe length was not established. Therefore, 252 characters should be reported as the shortest tested failure on this stack, not as a portable threshold.
Environment
PATH_MAX=4096,NAME_MAX=255PETSC_MAX_PATH_LEN=4096Every individual path component was below
NAME_MAX. The failure therefore occurs below the advertised filesystem and PETSc full-path limits and appears to be in the active native PETSc/HDF5/MPI-I/O stack.The issue has only been reproduced systematically on this Gadi stack. A macOS environment using PETSc 3.25.0, HDF5 1.14.6, and Open MPI 5.0.10 did not show the original benchmark failure, but the complete threshold matrix was not repeated there.
Validated workaround
Shorten the user-generated case/output directory and retain fixed configuration in the metrics file rather than encoding every fixed parameter in the directory name.
Reducing the longest production filename from 286 to 189 characters made the unchanged eight-rank benchmark pass end to end. A subsequent 192-rank, 1/64-resolution run also completed all mesh, Stokes, HDF5, XDMF, and postprocessing stages.
Proposed documentation
Add a note to the checkpoint/HDF5 output documentation:
output.mesh.U.00000.h5.PATH_MAX.mesh.write()ormesh.write_timestep()should prompt checking the full generated filename length.Reproducer and investigation log