NetCDF's use of HDF5 HL DS APIs and parallel I/O performance.

[brtnfld]

I noticed that NetCDF calls the High-level DS APIs by all the processes doing I/O. For example, it has sync_netcdf5_file --> nc4_rec_write_metadata --> write_var --> (H5DSdetach_scale, H5DSattach_scale, etc...).

Has there been an investigation into the possibility of closing the file and reopening it with only one process and then doing all the metadata updating (.i.e. the DS calls)?

All the HL DS APIs are "serial," so all the processes are re-writing the same data, and this data also seems to be very small. Usually, doing many little writes results in poor parallel performance on a parallel file system, especially when you have tens of thousands of processes writing what amounts to the same data.

The HL APIs are not meant to be called by all the processes since this, for the most part, just duplicates the I/O. Also, HDF5 does not test any of the HL APIs in a parallel setting.

I could be missing something subtle that doesn't allow NetCDF to do this, so any insight/feedback would be much appreciated.

[WardF]

Scott, I'll tag @edhartnett for his feedback since he implemented the HL calls in netCDF-4; I can offer supposition, but that's a poor substitute for first hand info :).

[edwardhartnett]

I don't think you are missing anything subtle. This just never occurred to me. Good idea!

[edwardhartnett]

I believe this issue should be closed.

I note that it is now possible to create files that don't do the dimscales.

[brtnfld]

The issue can be revisited in the future if needed. Closing.