Builds breaking from cdms2 issue related to libnetcdf

[mauzey1]

CMOR CI checks and nightly builds have recently been breaking due to an issue with cdms2. The libnetcdf library has recently been updated but the nightly version of cdms2 relies on the older version. This has caused the Python tests to break since they use cdms2. They generate the following error.

Traceback (most recent call last):
  File "Test/test_python_CMIP6_CV_externalvariables.py", line 21, in <module>
    import cdms2
  File "/home/circleci/project/workspace/test_linux_cmor/miniconda/envs/py3.6/lib/python3.6/site-packages/cdms2/__init__.py", line 8, in <module>
    from . import dataset
  File "/home/circleci/project/workspace/test_linux_cmor/miniconda/envs/py3.6/lib/python3.6/site-packages/cdms2/dataset.py", line 8, in <module>
    from . import Cdunif
ImportError: libnetcdf.so.13: cannot open shared object file: No such file or directory

According to @dnadeau4, there is a dependency issue with the OSX version of cdms and hdf5 concerning glibfortran. This has prevented newer version of cdms2 from being built that uses the newer libnetcdf 4.6.3. The current workaround is to build CMOR with the older libnetcdf 4.6.2. However, we want to go back to using the latest version of libnetdf as soon as possible.

Any suggestions are welcomed.

[durack1]

@mauzey1 thanks!

I think this should be a required issue to be resolved before 3.5.0 is released, otherwise we're going to have users trying to create a new CMOR350 env and hitting the libnetcdf issue

[doutriaux1]

@durack1 I know that @dnadeau4 was working on it before he left for vacation. Probably will be fixed in a couple of weeks.

[taylor13]

Is it time to clean up CMOR3 so it doesn't rely on cdms? As far as I know, users only really need the udunits library that comes with cdms.
I know some of our "test" codes probably rely on cdms, but couldn't they be removed from the build that is distributed to the world? it seems to me that cdms just adds a whole lot of baggage but no needed function for cmor.

[doutriaux1]

@taylor13 I agree. I think it's only used for testing and for PrePARE to open files via CdUnif, we could replace this with the python netcdf4

[mauzey1]

@doutriaux1 cdms2 is used in https://github.com/PCMDI/cmor/blob/master/Lib/pywrapper.py where it is mostly used for the isVariable function. It's also checking whether some variables are an instance of cdms2.axis.TransientAxis.

CMOR also contains source for cdTime, which is used by cmor_axis.c for time axis values. I'm not sure how they relate to the use of cdms2.

[doutriaux1]

@mauzey1 in the file above all the calls to cdms2 are checked against has_cdms2 so it's ok. But in this file: https://github.com/PCMDI/cmor/blob/master/LibCV/PrePARE/PrePARE.py#L37 we require cdms2
I believe this line: https://github.com/PCMDI/cmor/blob/master/LibCV/PrePARE/PrePARE.py#L387 could be replaced with straight netcdf4 (provided a few tweaks thereafter)
it shouldn't be too much effort i think to completely remove the runtime dependency on cdms2

[durack1]

@doutriaux1 just a comment here, you're advocating replacing a dependency on cdms2 with a dependency of netcdf4-python. So other than cutting out cdms2 from the deal, what is the gain here?

This just means that you replace one dependency with another

[doutriaux1]

the gain is when netcdf change netcdf4 python gets changed at the same time we do not need to immediately rebuild everythin (libcdms, libdrs, cdtime, etc...)