parallel=True fallback to format='NETCDF4' with NetCDF-C-4.6.2 and NetCDF-C-4.6.3

[gihankarunarathne]

I have tested this on;

hdf5-1.10.4
NETCDF_C_VERSION 4.6.2 / 4.6.3
netcdf4-python-1.4.3

I can pass the format as NETCDF4_CLASSIC with fix for #890 , it isn't giving the error anymore. But the data_model is setting to the NETCDF4 when parallel=True and using NetCDF-C-4.6.2 and NetCDF-C-4.6.3. These are the args which I'm using for the build NetCDF-C.

./configure --prefix=/usr/local --enable-netcdf-4 --enable-shared --enable-parallel4 --enable-parallel-tests --disable-da

So, I don't this, it's related to #820 . When I tried to create a NETCDF4_CLASSIC with python interpreter with parallel=True;

>>> import netCDF4
>>> import numpy as np
>>> ncfile = netCDF4.Dataset('mpi_data_test.nc', mode='w', parallel=True, format='NETCDF4_CLASSIC')
>>> ncfile.data_model
'NETCDF4'
>>> ncfile = netCDF4.Dataset('mpi_data_test.nc', mode='w', parallel=False, format='NETCDF4_CLASSIC')
>>> ncfile.data_model
'NETCDF4_CLASSIC'

When I tried to run the mpi_example.py. Head of mpi_example.py

# to run: mpirun -np 4 python mpi_example.py
from mpi4py import MPI
import numpy as np
from netCDF4 import Dataset
rank = MPI.COMM_WORLD.rank  # The process ID (integer 0-3 for 4-process run)
nc = Dataset('/tmp/parallel_test.nc', 'w', parallel=True, format='NETCDF4_CLASSIC', comm=MPI.COMM_WORLD, info=MPI.Info())
# below should work also - MPI_COMM_WORLD and MPI_INFO_NULL will be used.
#nc = Dataset('/tmp/parallel_test.nc', 'w', parallel=True, format='NETCDF4_CLASSIC')
d = nc.createDimension('dim',4)
v = nc.createVariable('var', np.int, 'dim')

Tried on the output file:

>>> import numpy as np
>>> from netCDF4 import Dataset
>>> nc = Dataset('/tmp/parallel_test.nc', 'r')
>>> nc.data_model
'NETCDF4'

---

But it's working fine with NetCDF-C-4.6.1;

hdf5-1.10.4
NETCDF_C_VERSION 4.6.1
netcdf4-python-1.4.3

>>> import netCDF4
>>> import numpy as np
>>> ncfile = netCDF4.Dataset('mpi_data_test.nc', mode='w', parallel=True, format='NETCDF4_CLASSIC')
>>> ncfile.data_model
'NETCDF4_CLASSIC'
>>> ncfile = netCDF4.Dataset('mpi_data_test.nc', mode='w', parallel=True, format='NETCDF4')
>>> ncfile.data_model
'NETCDF4'

[jswhit]

data_model is just a property that returns a c-lib call nc_inq_format. Don't know why 4.6.2 is returning something different than 4.6.1 did.

[jswhit]

Never mind - I think I see the problem. Stay tuned.

[jswhit]

My bad - it's fixed by pull request #892. Will have to create a 1.4.3.1 release.

[gihankarunarathne]

@jswhit I try to test your code. This's what I did!

git clone https://github.com/Unidata/netcdf4-python.git && \
cd netcdf4-python && git fetch origin issue891 && git checkout issue891 && \
pip uninstall netCDF4 && python setup.py build && python setup.py install

But when I try to load the netCDF4 module, import fails;

>>> import netCDF4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/src/netcdf4-python/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ModuleNotFoundError: No module named 'netCDF4._netCDF4'

This worked before me. pip list show me netCDF4 1.4.3.1 also. Same happens with the master branch.
Previously I downloaded the code via release tag.

[jswhit]

You can't import from the netcdf4-python directory (the directory that setup.py exists in). Move up one level and try again.

[gihankarunarathne]

@jswhit yes, it's working fine!

>>> import netCDF4
>>> ncfile = netCDF4.Dataset('mpi_data_test.nc', mode='w', parallel=True, format='NETCDF4_CLASSIC')
>>> ncfile.data_model
'NETCDF4_CLASSIC'
>>> ncfile = netCDF4.Dataset('mpi_data_test.nc', mode='w', parallel=True, format='NETCDF4')
>>> ncfile.data_model
'NETCDF4'

Tested on;

hdf5-1.10.4
NETCDF_C_VERSION 4.6.2
netcdf4-python-1.4.3.1

[jswhit]

OK - I'll merge tomorrow and make a new release Thursday.

[gihankarunarathne]

Great. Thanks a lot!

[jswhit]

v 1.4.3.1 is released

[gihankarunarathne]

Working fine with `netcdf4-C 4.6.3 also;

hdf5-1.10.4
NETCDF_C_VERSION 4.6.3
netcdf4-python-1.4.3.1