-
Notifications
You must be signed in to change notification settings - Fork 521
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to write independently in parallel hdf5 #2330
Comments
Have you installed the parallel build of libhdf5? What does the There are different builds of libhdf5: % conda search hdf5 | grep "1\.14\.2"
hdf5 1.14.2 mpi_mpich_h3618df7_0 conda-forge
hdf5 1.14.2 mpi_openmpi_h01be5f8_0 conda-forge
hdf5 1.14.2 nompi_hedada53_100 conda-forge You must use one of the "mpi" builds. |
@ajelenak Thanks for great comment. builds of my hdf5 and other related follows from
|
You seem to be using the right build of libhdf5. The error comes from the library not h5py. Someone who also runs h5py with MPI could be able to help more. |
Hi @ekdldkqkr, Note that this check was added in HDFGroup/hdf5@6633210 for initial support of UnifyFS. I'd advise against it, but you may be able to disable this check by setting the environment variable However, I believe what's happening is that collective MPI I/O was requested, but HDF5 had to break collective I/O for some reason (one example is type conversions may have been needed with the criteria for that not being supported). I don't know if h5py exposes a routine for it, but HDF5 has the C API H5Pget_mpio_no_collective_cause that is called on a Dataset Transfer Property List (DXPL) and returns a |
Thanks @jhendersonHDF for a helpful suggestion how to resolve the problem. I can confirm the H5Pget_mpio_no_collective_cause function is not available in h5py. Since the h5py code used here is from an example in its documentation, it must have run successfully for an older libhdf5 version. This then implies that perhaps the recent libhdf5b features for MPI-based computing may require revisiting the current h5py support for it. |
Based on some discussion with other folks, I'm wondering if this isn't due to changes made in ROMIO that are picked up in the install of MPICH from conda forge. The issue is that HDF5 is picking up an If the UnifyFS backend is not being used, then setting the environment variable |
Hi, I'm having trouble to run parallel hdf5 from h5py, tutorial in documents.
https://docs.h5py.org/en/latest/mpi.html
I have tried the following:
and error occurs
It does create hdf5 file, but seems to write dataset with only one processing units.
I checked created hdf5 file with h5dump
results different from following which is done in docuents
gives following
The text was updated successfully, but these errors were encountered: