parallel=True seem to make netcdf4 ignore format='NETCDF4' and fall back on 'NETCDF3_classic'

[ThibHlln]

Hi there,

I am running into issues when I try to create a NetCDF file (in the NETCDF4 flavour) with parallel I/O using mpi4py and Open MPI. When I create a Dataset(my_file_name, 'w', format='NetCDF4', parallel=True), and then check the file format, I get NETCDF3_CLASSIC. However, when I create a Dataset(my_file_name, 'w', format='NetCDF4', parallel=False), I get what I want (i.e. a NETCDF4 file).

I know the NetCDF-C library is properly configured for Parallel IO because I don't run into any issue with parallel=True as long as I am not using unlimited dimensions and I don't put them as the first dimension for a multi-dimensional variable. When I try to do this, I get thrown an exception:

File "netCDF4/_netCDF4.pyx", line 2533, in netCDF4._netCDF4.Dataset.createVariable
RuntimeError: NetCDF: NC_UNLIMITED in the wrong index

This error is not surprising because the file is a NETCDF3_CLASSIC, not a NETCDF4 (against my will), and as far as I understand it, there is this requirement for version NETCDF3_CLASSIC that the unlimited dimensions should be the first dimensions of a multi-dimensional variable, which is not the case for NETCDF4.

I am using a High Performance Cluster running on Linux, python 2.7.14, netCDF4 1.4.0, netcdf-c 4.6.1, mpi4py 3.0.0, Open MPI 2.1.1.

Is it the expected behaviour that the use of parallel I/O overwrites user-defined file format to fall back onto NETCDF3_CLASSIC?

Here is my piece of code, if this is of any help.

self.database = Dataset(self.db_file, 'w', parallel=True)
# create structure of NetCDF file
# metadata
self.database.description = "Monte Carlo Simulations."
# dimensions
self.database.createDimension("NbSamples", 50000)
self.database.createDimension("NbParameters", 10)
self.database.createDimension("NbObjFunctions", 14)
# variables
params = self.database.createVariable("Parameters", np.float32, ("NbSamples", "NbParameters"),
                                                           zlib=True, complevel=compression)
params.units = ', '.join(self.model.parameters.names)
objfns = self.database.createVariable("ObjFunctions", np.float32, ("NbSamples", "NbObjFunctions"),
                                                        zlib=True, complevel=compression)
objfns.units = ', '.join(self.obj_fn_names)
if self.save_sim:
    # dimension
    self.database.createDimension("DateTime", None)  # Unlimited dimension
    # coordinate variables
    times = self.database.createVariable("DateTime", np.float64, ("DateTime",), zlib=True)
    times.units = 'seconds since 1970-01-01 00:00:00.0'
    # variable
    simu = self.database.createVariable("Simulations", np.float32, ("NbSamples", "DateTime"),
                                                          zlib=True, complevel=compression)

[jswhit]

Are you sure the netcdf-c library with built with parallel HDF5 support (and not pnetcdf which uses the NETCDF3/CDF-5 format)?

See https://www.unidata.ucar.edu/software/netcdf/docs/getting_and_building_netcdf.html#build_parallel (which discusses how to build with either parallel HDF5 NETCDF4 support or pnetcdf NETCDF3 support)

[ThibHlln]

Very good point. Thank you for your prompt answer and help on this @jswhit.
I must check this, but this is very likely the cause for this behaviour indeed.

[jswhit]

Any update on @ThibHlln ?

[ThibHlln]

Hi @jswhit,

Since I have not installed the C-library on the High Performance Cluster myself, I am still waiting for the answer from the administrator to shade some more light on this thread. I will edit this post as soon as I get the answer. Apologies for the delay.

[ThibHlln]

Hi Jeff,

I finally got more details about the way the C library was built:

cat /opt/software/netcdf-c/4.6.1/lib/libnetcdf.settings

NetCDF C Configuration Summary

==============================

General

---

NetCDF Version: 4.6.1
Configured On: Thu Jul 12 10:47:26 IST 2018
Host System: x86_64-pc-linux-gnu
Build Directory: /opt/builddir/netcdf-c-4.6.1
Install Prefix: /opt/software/netcdf-c/4.6.1

Compiling Options

---

C Compiler: /opt/software/openmpi/2.1.1/bin/mpicc
CFLAGS:
CPPFLAGS: -I/opt/software/hdf5/1.10.2/include -I/opt/software/hdf4/2.1.3/include -I/opt/software/parallel-netcdf/1.9.0/include
LDFLAGS: -L/opt/software/hdf5/1.10.2/lib -L/opt/software/hdf4/2.1.3/lib -L/opt/software/parallel-netcdf/1.9.0/lib
AM_CFLAGS:
AM_CPPFLAGS:
AM_LDFLAGS:
Shared Library: yes
Static Library: yes
Extra libraries: -lpnetcdf -ljpeg -lmfhdf -ldf -lhdf5_hl -lhdf5 -lm -lz -lcurl -ldl

Features

---

NetCDF-2 API: yes
HDF4 Support: yes
NetCDF-4 API: yes
NC-4 Parallel Support: yes
PNetCDF Support: yes
DAP2 Support: yes
DAP4 Support: yes
Diskless Support: yes
MMap Support: no
JNA Support: no
CDF5 Support: yes
ERANGE fill Support: yes

So it seems that it was built with PNetCDF, right?

[jswhit]

Looks like your netcdf library has mpi parallel support (NC-4 Parallel Support). You also need to link the MPI parallel version of the HDF5 library.

I don't know why the C library is switching the file format to NETCDF3_CLASSIC though - that is strange. It could be that you can't have PNetCDF and NC-4 parallel enabled simultaneously. You might try leaving off '--enable-pnetcdf' when building the netcdf-c library.

Are you using mpi4py to create mpi processes? I don't see that in your example script.

Also, have you tried running examples/mpi_example.py?

[ThibHlln]

Yes, I am using mpi4py to create my processes.

When I run mpi_example.py, I get:

Traceback (most recent call last): File "mpi_example.py", line 11, in <module> Traceback (most recent call last): File "mpi_example.py", line 11, in <module> Traceback (most recent call last): File "mpi_example.py", line 11, in <module> v = nc.createVariable('var', np.int, 'dim') v = nc.createVariable('var', np.int, 'dim') File "netCDF4/_netCDF4.pyx", line 2533, in netCDF4._netCDF4.Dataset.createVariable File "netCDF4/_netCDF4.pyx", line 2533, in netCDF4._netCDF4.Dataset.createVariable v = nc.createVariable('var', np.int, 'dim') File "netCDF4/_netCDF4.pyx", line 2533, in netCDF4._netCDF4.Dataset.createVariable File "netCDF4/_netCDF4.pyx", line 3488, in netCDF4._netCDF4.Variable.__init__ File "netCDF4/_netCDF4.pyx", line 3488, in netCDF4._netCDF4.Variable.__init__ File "netCDF4/_netCDF4.pyx", line 3488, in netCDF4._netCDF4.Variable.__init__ File "netCDF4/_netCDF4.pyx", line 1733, in netCDF4._netCDF4._ensure_nc_success File "netCDF4/_netCDF4.pyx", line 1733, in netCDF4._netCDF4._ensure_nc_success Traceback (most recent call last): File "mpi_example.py", line 11, in <module> File "netCDF4/_netCDF4.pyx", line 1733, in netCDF4._netCDF4._ensure_nc_success RuntimeError: Attempting CDF-5 operation on strict CDF or CDF-2 file v = nc.createVariable('var', np.int, 'dim') File "netCDF4/_netCDF4.pyx", line 2533, in netCDF4._netCDF4.Dataset.createVariable RuntimeError: RuntimeError: Attempting CDF-5 operation on strict CDF or CDF-2 file Attempting CDF-5 operation on strict CDF or CDF-2 file File "netCDF4/_netCDF4.pyx", line 3488, in netCDF4._netCDF4.Variable.__init__ File "netCDF4/_netCDF4.pyx", line 1733, in netCDF4._netCDF4._ensure_nc_success RuntimeError: Attempting CDF-5 operation on strict CDF or CDF-2 file

I hope this helps.

[jswhit]

It does look as though the C library is trying to use pnetcdf instead of hdf5. I think this is probably a bug in the C lib. The only workaround I can suggest is to recompile netcdf-c without --enable-pnetcdf.

[ThibHlln]

Thank you for your help on this issue @jswhit !

[jswhit]

I'd like to leave this ticket open - @ThibHlln if you are able to try mpi_example.py again after recompiling the C lib without --enable-pnetcdf please report back here.

[jswhit]

@WardF - would you expect pnetcdf (for NETCDF3 files) and nc4-parallel (for NETCDF4 files) to work with the same build of the C library?

[wkliao]

I have been building and testing NetCDF with both parallel I/O methods enabled. It runs fine.

If file create flag NC_NETCDF4 is used, then NetCDF library picks HDF5 driver underneath to perform parallel I/O and the file will be in the HDF5-based format. If NC_NETCDF4 is not used, then PnetCDF is used and file will be in the classic format.
For file open case, NetCDF first checks the file format and picks the right HDF5 or PnetCDF driver automatically. There is no need to use any format flag to open a file.

[jswhit]

The python library sets NC_NETCDF4 as the create flag by default, yet the error messages above indicate that the format flag is being changed somewhere. @ThibHlln, could you try running with format='NETCDF4' set explicitly?