New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to write unsigned int8 variable in netcdf4 file (non-netcdf3 type) #481

Open
durack1 opened this Issue Jun 23, 2014 · 21 comments

Comments

Projects
None yet
5 participants
@durack1
Member

durack1 commented Jun 23, 2014

It appears that something is going awry with writing netcdf4 data types. When I try to write a byte (uint8) variable to a file, I get the standard netcdf library error:
'cdunif: NetCDF: Attempting netcdf-4 operation on strict nc3 netcdf-4 file'

I'm wondering if this has any relationship with #406

If the write statement is dropped into a try Write: except Pass block it seems to write at the second attempt, placed after this try/except block.

The written file looks something like below:

      [durack1@oceanonly WOD13]$ ncdump -h WOD13_masks.nc
      netcdf WOD13_masks {
      dimensions:
              depth = 102 ;
              bound = 2 ;
              latitude = 180 ;
              longitude = 360 ;
      variables:
              float depth(depth) ;
                      depth:bounds = "bounds_depth" ;
              double bounds_depth(depth, bound) ;
              float latitude(latitude) ;
                      latitude:bounds = "bounds_latitude" ;
                      latitude:axis = "Y" ;
              double bounds_latitude(latitude, bound) ;
              float longitude(longitude) ;
                      longitude:bounds = "bounds_longitude" ;
                      longitude:axis = "X" ;
                      longitude:modulo = 360. ;
                      longitude:topology = "circular" ;
              double bounds_longitude(longitude, bound) ;
              byte basinmask(depth, latitude, longitude) ;
                      basinmask:missing_value = 125b ;
@durack1

This comment has been minimized.

Member

durack1 commented Jun 23, 2014

Peeling this problem back a little, it may be due to tangles with attribute assignment within cdms2

Traceback (most recent call last):
  File "make_masks.py", line 118, in <module>
    basinmask.setMissing(vl)
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/cdms2/tvariable.py", line 557, in setMissing
    self.set_fill_value(value)
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/cdms2/tvariable.py", line 550, in set_fill_value
    AbstractVariable.setMissing(self, value)
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/cdms2/avariable.py", line 433, in setMissing
    self.missing_value = value
  File "/usr/local/uvcdat/2014-06-19-numpy1.9.0b1/lib/python2.7/site-packages/numpy/ma/core.py", line 3408, in set_fill_value
    _fill_value[()] = target
TypeError: 'numpy.uint8' object does not support item assignment
@durack1

This comment has been minimized.

Member

durack1 commented Jun 23, 2014

Ok so it seems the problem is that cdms2 is attempting to write a uint8 variable (^NC_ubyte - unsigned byte) as an int8 (^NC_byte - byte) and consequently is falling over.. And getting out of range results in the missing_value attribute

So this tangle could be causing the issue..

https://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/NetCDF_002d4-Atomic-Types.html

@durack1

This comment has been minimized.

Member

durack1 commented Jun 23, 2014

Seems like https://github.com/UV-CDAT/uvcdat/blob/master/Packages/cdms2/Src/Cdunifmodule.c#L190-216 has no unsigned types, however this type is declared further down..

@aashish24

This comment has been minimized.

Contributor

aashish24 commented Jun 24, 2014

@durack1 Do you have a patch for it? It would be great if you can submit a patch for it.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jun 24, 2014

@aashish24 that's ok I'll fix this Thursday, @durack1 pretty much did all the detective work, thank you Paul!

@durack1

This comment has been minimized.

Member

durack1 commented Jun 24, 2014

My pleasure gents.. Happy to kick the tyres on this stuff..

@durack1

This comment has been minimized.

Member

durack1 commented Jun 25, 2014

@doutriaux1 it also seems that NC_STRING is another type which may not be declared here, although is in other places.. May as well add that too, so we're fully netcdf-4 compliant..

@durack1

This comment has been minimized.

Member

durack1 commented Aug 5, 2014

Nope looks to me like NC_STRING still isn't a type that cdms2 can write.. From my foggy memory this had something to do with HDF vs NC "classic" types and how going down the full NC4 enabling would require major changes to the CDUNIF module.. And I would caveat that this is really foggy..

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 5, 2014

we're not going to be full blown NC4 we can do tweaks that work within cdunfi framework. Maybe when we rewrite the reader to use VTK reeaders we can revisit that issue.

@durack1

This comment has been minimized.

Member

durack1 commented Aug 5, 2014

Ok sounds good.. I have no idea what the VTK readers is, but it sounds cool (and newer than CDUNIF)..

It would really be nice to have NC_STRING types available, as it's a great way to write out metadata-rich, completely self describing files..

@durack1

This comment has been minimized.

Member

durack1 commented Aug 5, 2014

Oh and what is the milestone for VTK reader implementation? UV-CDAT 2.1?

@durack1

This comment has been minimized.

Member

durack1 commented Aug 6, 2014

This issue is related to #537

@durack1

This comment has been minimized.

Member

durack1 commented Sep 22, 2014

You folks might want to assign this as a 2.1 milestone to get if off the to-do list

@aashish24 aashish24 added this to the 2.1 milestone Sep 22, 2014

@aashish24

This comment has been minimized.

Contributor

aashish24 commented Sep 22, 2014

done

@durack1

This comment has been minimized.

Member

durack1 commented Oct 1, 2014

It seems that this bug (or possibly #537) is causing problems elsewhere:
140930_cdms2-inttypebug

@durack1

This comment has been minimized.

Member

durack1 commented Feb 19, 2015

@doutriaux1 another 2.3 candidate..

@doutriaux1 doutriaux1 removed this from the 2.4 milestone Sep 16, 2015

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Sep 28, 2015

@durack1 it seems that netCDF 3 does not handle unsigned int. This is something that was added in netCDF4. I am not sure what to do if somebody tries to create a netCDF 3 files with a unsigned int type.

@durack1

This comment has been minimized.

Member

durack1 commented Sep 28, 2015

@dnadeau4, @doutriaux1 has now converted cdms2 to write netcdf-4 files as the default if I remember correctly, so unless a user specifically tries to write a netcdf-3 file then this situation shouldn't be encountered..

@jbeezley

This comment has been minimized.

Contributor

jbeezley commented Sep 28, 2015

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Sep 28, 2015

Thanks @jbeezley I will add the new types to CDMS2 and see what happens when I create a netCDF3 file with unsigned int. I'll make it stop in a nicer fashion than "fill_value" assignment error. 😄

@durack1

This comment has been minimized.

Member

durack1 commented Sep 28, 2015

@dnadeau4 the behavior that I'd expect, would be throwing error/warning and the file to be written default back to netcdf-4 - so it can contain the unsupported type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment