Unable to write unsigned int8 variable in netcdf4 file (non-netcdf3 type)

[durack1]

It appears that something is going awry with writing netcdf4 data types. When I try to write a byte (uint8) variable to a file, I get the standard netcdf library error:
'cdunif: NetCDF: Attempting netcdf-4 operation on strict nc3 netcdf-4 file'

I'm wondering if this has any relationship with #406

If the write statement is dropped into a try Write: except Pass block it seems to write at the second attempt, placed after this try/except block.

The written file looks something like below:

      [durack1@oceanonly WOD13]$ ncdump -h WOD13_masks.nc
      netcdf WOD13_masks {
      dimensions:
              depth = 102 ;
              bound = 2 ;
              latitude = 180 ;
              longitude = 360 ;
      variables:
              float depth(depth) ;
                      depth:bounds = "bounds_depth" ;
              double bounds_depth(depth, bound) ;
              float latitude(latitude) ;
                      latitude:bounds = "bounds_latitude" ;
                      latitude:axis = "Y" ;
              double bounds_latitude(latitude, bound) ;
              float longitude(longitude) ;
                      longitude:bounds = "bounds_longitude" ;
                      longitude:axis = "X" ;
                      longitude:modulo = 360. ;
                      longitude:topology = "circular" ;
              double bounds_longitude(longitude, bound) ;
              byte basinmask(depth, latitude, longitude) ;
                      basinmask:missing_value = 125b ;

[durack1]

Peeling this problem back a little, it may be due to tangles with attribute assignment within cdms2

Traceback (most recent call last):
  File "make_masks.py", line 118, in <module>
    basinmask.setMissing(vl)
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/cdms2/tvariable.py", line 557, in setMissing
    self.set_fill_value(value)
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/cdms2/tvariable.py", line 550, in set_fill_value
    AbstractVariable.setMissing(self, value)
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/cdms2/avariable.py", line 433, in setMissing
    self.missing_value = value
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/numpy/ma/core.py", line 3408, in set_fill_value
    _fill_value[()] = target
TypeError: 'numpy.uint8' object does not support item assignment

[durack1]

Ok so it seems the problem is that cdms2 is attempting to write a uint8 variable (^NC_ubyte - unsigned byte) as an int8 (^NC_byte - byte) and consequently is falling over.. And getting out of range results in the missing_value attribute

So this tangle could be causing the issue..

https://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/NetCDF_002d4-Atomic-Types.html

[durack1]

Seems like https://github.com/UV-CDAT/uvcdat/blob/master/Packages/cdms2/Src/Cdunifmodule.c#L190-216 has no unsigned types, however this type is declared further down..

[aashish24]

@durack1 Do you have a patch for it? It would be great if you can submit a patch for it.

[doutriaux1]

@aashish24 that's ok I'll fix this Thursday, @durack1 pretty much did all the detective work, thank you Paul!

[durack1]

My pleasure gents.. Happy to kick the tyres on this stuff..

[durack1]

@doutriaux1 it also seems that NC_STRING is another type which may not be declared here, although is in other places.. May as well add that too, so we're fully netcdf-4 compliant..

[durack1]

Nope looks to me like NC_STRING still isn't a type that cdms2 can write.. From my foggy memory this had something to do with HDF vs NC "classic" types and how going down the full NC4 enabling would require major changes to the CDUNIF module.. And I would caveat that this is really foggy..

[doutriaux1]

we're not going to be full blown NC4 we can do tweaks that work within cdunfi framework. Maybe when we rewrite the reader to use VTK reeaders we can revisit that issue.

[durack1]

Ok sounds good.. I have no idea what the VTK readers is, but it sounds cool (and newer than CDUNIF)..

It would really be nice to have NC_STRING types available, as it's a great way to write out metadata-rich, completely self describing files..

[durack1]

Oh and what is the milestone for VTK reader implementation? UV-CDAT 2.1?

[durack1]

This issue is related to #537

[durack1]

You folks might want to assign this as a 2.1 milestone to get if off the to-do list

[aashish24]

done

[durack1]

It seems that this bug (or possibly #537) is causing problems elsewhere:

[durack1]

@doutriaux1 another 2.3 candidate..

[dnadeau4]

@durack1 it seems that netCDF 3 does not handle unsigned int. This is something that was added in netCDF4. I am not sure what to do if somebody tries to create a netCDF 3 files with a unsigned int type.

[durack1]

@dnadeau4, @doutriaux1 has now converted cdms2 to write netcdf-4 files as the default if I remember correctly, so unless a user specifically tries to write a netcdf-3 file then this situation shouldn't be encountered..

[jbeezley]

It raises error NC_EBADTYPE.

https://github.com/Unidata/netcdf-c/blob/master/libsrc/nc3internal.c#L88

[dnadeau4]

Thanks @jbeezley I will add the new types to CDMS2 and see what happens when I create a netCDF3 file with unsigned int. I'll make it stop in a nicer fashion than "fill_value" assignment error. 😄

[durack1]

@dnadeau4 the behavior that I'd expect, would be throwing error/warning and the file to be written default back to netcdf-4 - so it can contain the unsupported type.