File open fails when lon and lat are variables (not dimensions)

[durack1]

The attached file is unreadable in the current version of cdms2 (v2.8). I believe the issue is that the variables (not dimensions) lon and lat included in the file are assumed to be standard dimensions (which they aren't), and these assumptions lead to a read failure.

cdmsReadFail.zip

The file looks like:

ncdump -h 161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none.nc 
netcdf \161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none {
dimensions:
	expt = 18 ;
	level = 61 ;
	layer = 60 ;
	site = 100 ;
variables:
	float lon(site) ;
		lon:long_name = "ERA-Interim longitude" ;
		lon:units = "degree_north" ;
		lon:standard_name = "longitude" ;
	float lat(site) ;
		lat:long_name = "ERA-Interim latitude" ;
		lat:units = "degree_east" ;
		lat:standard_name = "latitude" ;
	float time(site) ;
		time:long_name = "ERA-Interim fractional day of the year 2014" ;
		time:units = "days since 2014-1-1 0:0:0" ;
		time:standard_name = "time" ;
		time:calendar = "gregorian" ;
	float sst(site) ;
		sst:title = "sea surface temperature" ;
		sst:units = "K" ;
		sst:long_name = "ERA-Interim sea surface temperature (= \"missing_value\" over land)" ;
		sst:standard_name = "sea_surface_temperature" ;
		sst:missing_value = -9.99f ;
		sst:FillValue = -9.99f ;
		sst:coordinates = "lon lat time" ;
...

And the read failure looks like:

In [1]: infile = '161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none.nc'
In [2]: import cdms2 as cdm
In [3]: f = cdm.open(infile)
---------------------------------------------------------------------------
CDMSError                                 Traceback (most recent call last)
<ipython-input-3-da3bd29babf2> in <module>()
----> 1 f = cdm.open(infile)
/uvcdat280/lib/python2.7/site-packages/cdms2/dataset.pyc in openDataset(uri, mode, template, dods, dpath, hostObj)
    357 
    358             # The file exists
--> 359             file1 = CdmsFile(path, "r")
    360             if libcf is not None:
    361                 if hasattr(file1, libcf.CF_FILETYPE):
/uvcdat280/lib/python2.7/site-packages/cdms2/dataset.pyc in __init__(self, path, mode, hostObj, mpiBarrier)
   1137                 # Get grid information for the variable. gridkey has the form
   1138                 # (latname,lonname,order,maskname, abstract_class).
-> 1139                 gridkey, lat, lon = var.generateGridkey(self._convention_, self.variables)
   1140 
   1141                 # If the variable is gridded, lookup the grid. If no such grid exists,
/uvcdat280/lib/python2.7/site-packages/cdms2/avariable.pyc in generateGridkey(self, convention, vardict)
    239         lon, nlon = convention.getVarLonId(self, vardict)
    240         if (lat is not None) and (lat is lon):
--> 241             raise CDMSError, "Axis %s is both a latitude and longitude axis! Check standard_name and/or axis attributes."%lat.id
    242 
    243         # Check for 2D grid
CDMSError: Axis lon is both a latitude and longitude axis! Check standard_name and/or axis attributes.

[doutriaux1]

when did you create your conda env? I uploaded an updated cdms2 last night.

[durack1]

@doutriaux1 here we go:

(uvcdat280) bash-3.2$ ls -ald /uvcdat280/lib/python2.7/site-packages/cdms2
drwxr-xr-x  100 duro  THE-LAB\Domain Users  3400 Nov 16 15:33 /uvcdat280/lib/python2.7/site-packages/cdms2
(uvcdat280) bash-3.2$ conda update cdms2
Fetching package metadata .......
# All requested packages already installed.
# packages in environment at /uvcdat280:
#
cdms2                     2.8                 np111py27_2    uvcdat

[doutriaux1]

please do a force

[durack1]

@doutriaux1 same issue, it's a problem with cdms:

(uvcdat280) bash-3.2$ conda update cdms2 --force
Fetching package metadata .......
Solving package specifications: ..........
Package plan for installation in environment /uvcdat280:
The following NEW packages will be INSTALLED:
    cdms2: 2.8-np111py27_2 uvcdat
Proceed ([y]/n)? y
Pruning extracted packages from the cache ...
[      COMPLETE      ]|########################################################################################################################################################| 100%
Extracting packages ...
[      COMPLETE      ]|########################################################################################################################################################| 100%
Linking packages ...

(uvcdat280) bash-3.2$ ipython
Python 2.7.12 | packaged by conda-forge | (default, Sep  8 2016, 14:41:48) 
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]: import cdms2 as cdm
In [2]: infile = '161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none.nc'
In [3]: f = cdm.open(infile)
---------------------------------------------------------------------------
CDMSError                                 Traceback (most recent call last)
<ipython-input-3-da3bd29babf2> in <module>()
----> 1 f = cdm.open(infile)
/uvcdat280/lib/python2.7/site-packages/cdms2/dataset.pyc in openDataset(uri, mode, template, dods, dpath, hostObj)
    357 
    358             # The file exists
--> 359             file1 = CdmsFile(path, "r")
    360             if libcf is not None:
    361                 if hasattr(file1, libcf.CF_FILETYPE):
/uvcdat280/lib/python2.7/site-packages/cdms2/dataset.pyc in __init__(self, path, mode, hostObj, mpiBarrier)
   1137                 # Get grid information for the variable. gridkey has the form
   1138                 # (latname,lonname,order,maskname, abstract_class).
-> 1139                 gridkey, lat, lon = var.generateGridkey(self._convention_, self.variables)
   1140 
   1141                 # If the variable is gridded, lookup the grid. If no such grid exists,
/uvcdat280/lib/python2.7/site-packages/cdms2/avariable.pyc in generateGridkey(self, convention, vardict)
    239         lon, nlon = convention.getVarLonId(self, vardict)
    240         if (lat is not None) and (lat is lon):
--> 241             raise CDMSError, "Axis %s is both a latitude and longitude axis! Check standard_name and/or axis attributes."%lat.id
    242 
    243         # Check for 2D grid
CDMSError: Axis lon is both a latitude and longitude axis! Check standard_name and/or axis attributes.

[durack1]

@doutriaux1 curiously, in a previous (and not identical) version of the file, if the lon:units were changed from "degree_north" to "degree north" (remove the underscore) it seemed to be readable by the checker/cdms

@dnadeau4 pinging you into this thread

[dnadeau4]

degree_north and degree_east are accepted and CDMS2 should work with this file. cf-conventions

I will enhanced CDMS2 to make sure your lat/lon are detected correctly

[durack1]

@dnadeau4 out of curiosity, I tried to update the file to include lon:axis = "X" and similar for lat and I still get the identical error:

[durack1@oceanonly RFMIP]$ ncdump -h test.nc | more                                                                                  
netcdf test {                                                                                                                        
dimensions:                                                                                                                          
        expt = 18 ;                                                                                                                  
        level = 61 ;                                                                                                                 
        layer = 60 ;                                                                                                                 
        site = 100 ;                                                                                                                 
variables:                                                                                                                           
        float lon(site) ;                                                                                                            
                lon:long_name = "ERA-Interim longitude" ;                                                                            
                lon:units = "degree_north" ;                                                                                         
                lon:standard_name = "longitude" ;                                                                                    
                lon:axis = "X" ;                                                                                                     
        float lat(site) ;                                                                                                            
                lat:long_name = "ERA-Interim latitude" ;                                                                             
                lat:units = "degree_east" ;                                                                                          
                lat:standard_name = "latitude" ;                                                                                     
                lat:axis = "Y" ;                                                                                                     
        float time(site) ;                                                                                                           
                time:long_name = "ERA-Interim fractional day of the year 2014" ;                                                     
                time:units = "days since 2014-1-1 0:0:0" ;                                                                           
                time:standard_name = "time" ;                                                                                        
                time:calendar = "gregorian" ;                                                                                        
        float sst(site) ;                                                                                                            
                sst:title = "sea surface temperature" ;                                                                              
                sst:units = "K" ;                                                                                                    
                sst:long_name = "ERA-Interim sea surface temperature (= \"missing_value\" over land)" ;                              
                sst:standard_name = "sea_surface_temperature" ;                                                                      
                sst:missing_value = -9.99f ;                                                                                         
                sst:FillValue = -9.99f ;                                                                                             
                sst:coordinates = "lon lat time" ;

And the error:

(uvcdatNightly) duri@ocean:[RFMIP]:[2858]> ipython
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:42:40)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]: import cdms2 as cdm
In [2]: infile = 'test.nc'
In [3]: f = cdm.open(infile)
---------------------------------------------------------------------------
CDMSError                                 Traceback (most recent call last)
<ipython-input-3-da3bd29babf2> in <module>()
----> 1 f = cdm.open(infile)
/uvcdatNightly/lib/python2.7/site-packages/cdms2/dataset.pyc in openDataset(uri, mode, template, dods, dpath, hostObj)
    307
    308             # The file exists
--> 309             file1 = CdmsFile(path,"r")
    310             if libcf is not None:
    311                 if hasattr(file1, libcf.CF_FILETYPE):
/uvcdatNightly/lib/python2.7/site-packages/cdms2/dataset.pyc in __init__(self, path, mode, hostObj, mpiBarrier)
   1063                 # Get grid information for the variable. gridkey has the form
   1064                 # (latname,lonname,order,maskname, abstract_class).
-> 1065                 gridkey, lat, lon = var.generateGridkey(self._convention_, self.variables)
   1066
   1067                 # If the variable is gridded, lookup the grid. If no such grid exists,
/uvcdatNightly/lib/python2.7/site-packages/cdms2/avariable.pyc in generateGridkey(self, convention, vardict)
    236         lon, nlon = convention.getVarLonId(self, vardict)
    237         if (lat is not None) and (lat is lon):
--> 238             raise CDMSError, "Axis %s is both a latitude and longitude axis! Check standard_name and/or axis attributes."%lat.id
    239
    240         # Check for 2D grid
CDMSError: Axis lon is both a latitude and longitude axis! Check standard_name and/or axis attributes.

[durack1]

@dnadeau4 if you amend the lon:units = "degree north" (remove the underscore) it loads fine:

(uvcdatNightly) duro@ocean:[RFMIP]:[2858]> ipython
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:42:40)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]: import cdms2 as cdm
In [2]: infile = 'test_lonUnitTweak.nc'
In [3]: f = cdm.open(infile)
In [4]:

And for completeness the ncdump of the top of the file:

netcdf test_lonUnitTweak {
dimensions:
        expt = 18 ;
        level = 61 ;
        layer = 60 ;
        site = 100 ;
variables:
        float lon(site) ;
                lon:long_name = "ERA-Interim longitude" ;
                lon:standard_name = "longitude" ;
                lon:axis = "X" ;
                lon:units = "degree north" ;
        float lat(site) ;
                lat:long_name = "ERA-Interim latitude" ;
                lat:units = "degree_east" ;
                lat:standard_name = "latitude" ;
                lat:axis = "Y" ;

[durack1]

@dnadeau4 just wondering where this issue is sitting? We're hoping to get @RobertPincus's data published in the input4MIPs project, however the publisher/cdms can't deal with his data currently..

[dnadeau4]

Nothing yet, I am working on issue #7. @doutriaux1 gave me a priority list check label #priority and talk with @doutriaux1.