New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems writing netcdf data - _FillValue error #1479

Closed
durack1 opened this Issue Aug 4, 2015 · 7 comments

Comments

Projects
None yet
3 participants
@durack1
Member

durack1 commented Aug 4, 2015

@doutriaux1 this error has appeared, which doesn't seem obviously wrong (user error) - I'm curious if this is related to #1470:

#!/bin/env python

# -*- coding: utf-8 -*-
"""
Created on Tue Aug  4 11:24:35 2015

@author: durack1
"""
import os
import cdms2 as cdm

# Set outfile criteria
on = 0
cdm.setNetcdfShuffleFlag(on)
cdm.setNetcdfDeflateFlag(on)
cdm.setNetcdfDeflateLevelFlag(on)

# Read data
path = '/work/cmip5/historical/atm/mo/ts/'
inFile = 'cmip5.CCSM4.historical.r1i1p1.mo.atm.Amon.ts.ver-v20120604.latestX.xml'
inFile = os.path.join(path,inFile)
fIn = cdm.open(inFile)
time = fIn.getAxis('time')
slab = fIn('ts',time=('1850','1900'))
print 'slab _FillValue:',slab._FillValue,type(slab._FillValue)
print 'slab type:',slab.dtype

# Write data
outFile = './testGemma.nc'
if os.path.exists(outfile):
  os.remove(outFile)
  print 'outFile removed..'
fOut = cdm.open(outFile,'w')
fOut.write(slab.astype('float32'))
fOut.close()
fIn.close()

Yields the following output:

[durack1@crunchy ~/Desktop]$ testGemma.py
slab _FillValue: 1e+20 <type 'float'>
slab type: float32
cdunif: NetCDF: Not a valid data type or _FillValue type mismatch
If you are trying to write a NetCDF4 type (e.g. int64) please do not turn off cdms_shuffle and cdms_deflate
Traceback (most recent call last):
  File ".//testGemma.py", line 32, in <module>
    fOut.write(slab.astype('float32'))
  File "/usr/local/uvcdat/2015-07-09/lib/python2.7/site-packages/cdms2/dataset.py", line 1796, in write
    id=varid, extend=extend, fill_value=fill_value, index=index)
  File "/usr/local/uvcdat/2015-07-09/lib/python2.7/site-packages/cdms2/dataset.py", line 1721, in createVariableCopy
    setattr(newvar, attname, attval)
  File "/usr/local/uvcdat/2015-07-09/lib/python2.7/site-packages/cdms2/fvariable.py", line 145, in __setattr__
    setattr(self._obj_, name, value)
IOError: cdunif: NetCDF: Not a valid data type or _FillValue type mismatch
If you are trying to write a NetCDF4 type (e.g. int64) please do not turn off cdms_shuffle and cdms_deflate
@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 4, 2015

@durack1 it worked for me 2 minutes ago on crunchy with the same version... Odd.
Although I didn't use the setNeCDFDeflate flags... I think you're right it is the same issue as #1470

@doutriaux1 doutriaux1 added this to the 2.3 milestone Aug 4, 2015

@durack1

This comment has been minimized.

Member

durack1 commented Aug 4, 2015

Ok @doutriaux1 @dnadeau4 it seems like the type of the _FillValue is getting confused during read/write:

    float ts(time, lat, lon) ;
        ts:standard_name = "surface_temperature" ;
        ts:long_name = "Surface Temperature" ;
        ts:comment = "TS no change, CMIP5_table_comment: \"\"skin\"\" temperature (i.e., SST for open ocean)" ;
        ts:units = "K" ;
        ts:original_name = "TS" ;
        ts:cell_methods = "time: mean (interval: 30 days)" ;
        ts:cell_measures = "area: areacella" ;
        ts:history = "2011-10-21T17:19:45Z altered by CMOR: replaced missing value flag (-1e+32) with standard missing value (1e+20)." ;
        ts:missing_value = 1.e+20f ;
        ts:_FillValue = 1.e+20f ;
        ts:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_CCSM4_historical_r0i0p0.nc areacella: areacella_fx_CCSM4_historical_r0i0p0.nc" ;

So this kludge gets it synced back to a consistent type with slab (float32):

slab._FillValue = np.array(slab._FillValue).astype('f')
fOut.write(slab)
@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Sep 18, 2015

@dnadeau4 does #1524 fix this?

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Sep 18, 2015

working on it. I have to create a new xml file on my machine. These nc files are big!

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Sep 18, 2015

yes it does! 👍 😄

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Sep 18, 2015

awesome! Want to post a PR with a test for this before I close?

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Sep 18, 2015

I have a test with a 395Mb file and 1 xml file. Not sure how to post this..

@doutriaux1 doutriaux1 closed this Sep 19, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment