cdms2.write can't write a _FillValue attribute?

[jypeter]

The following script shows that you can add a _FillValue attribute to a variable, but it will not be written to the output file. Other attributes starting with a '_' are not written either, so I wonder if all the attributes starting with the underline character are silently ignored

Is that a bug or a feature?

Tested with 2.1

#!/usr/bin/env python

# Can't write a _FillValue attribute to a Variable
# J-Y Peterschmitt - LSCE - 07/2015

import numpy as np
import cdms2

data = np.random.random((10, 10))

data = np.ma.masked_less(data, 0.5)

dummy_grid = cdms2.createUniformGrid(10, 10, 1, 0, 10, 1)

dummy_var = cdms2.createVariable(data,
                                 axes=[dummy_grid.getLatitude(),
                                       dummy_grid.getLongitude()],
                                 id='my_var')
dummy_var.units = '10e3 dummy'
dummy_var._FillValue = 1.e+20
dummy_var.dummy_att = 'Dummy att'
dummy_var._dummy_att = 'Dummy att starting with _'

cdms2.setNetcdfShuffleFlag(0)
cdms2.setNetcdfDeflateFlag(0)
cdms2.setNetcdfDeflateLevelFlag(0)

# Test 1

f = cdms2.open('dummy1.nc', 'w')
f.write(dummy_var)
f.write(dummy_var, fill_value=999.999)
f.close()

# Test 2

f = cdms2.open('dummy2.nc', 'w')
f.write(dummy_var, fill_value=999.999)
f.close()

# Test 3

var_att = dummy_var.attributes
var_att['another_att'] = 'New att'
var_att['_another_att'] = 'New att starting with _'

f = cdms2.open('dummy3.nc', 'w')
f.write(dummy_var, attributes=var_att)
f.close()

# The end

[dnadeau4]

Jean-Yves, thanks for the program which is quite helpful. I will try to run it and take a look today. Attributes starting with "_" should be written in the netcdf files.

[dnadeau4]

I have been trying to reproduce your error without success. I am able to create FillValue attributes and other attributes starting with "". I am using cdms 2.5 which is the cdms found in the master repository.

I also tried the libcdms tarball libcdms-1.0.0.tar.gz found in http://uv-cdat.llnl.gov/cdat/resources/ and did not have any problem creating attributes starting with "_".

Can you tell me what version of UV-CDATyou are using?

Here are some of my results:

ncdump -h dummy1.nc
netcdf dummy1 {
...
    double my_var(latitude, longitude) ;
        my_var:_FillValue = 1.e+20 ;
        my_var:dummy_att = "Dummy att" ;
        my_var:units = "10e3 dummy" ;
        my_var:missing_value = 1.e+20 ;

// global attributes:
        :Conventions = "CF-1.0" ;
}

cdump -h dummy3.nc
netcdf dummy3 {
...
    double my_var(latitude, longitude) ;
        my_var:_FillValue = 1.e+20 ;
        my_var:name = "variable_1" ;
        my_var:another_att = "New att" ;
        my_var:dummy_att = "Dummy att" ;
        my_var:units = "10e3 dummy" ;
        my_var:_another_att = "New att starting with _" ;

// global attributes:
        :Conventions = "CF-1.0" ;
}

Thanks!

[jypeter]

I'm using the cdms2 I got from UV-CDAT 2.1.0 (sorry, I have not had time to updgrade yet)
How do I get the cdms2 version number? Well, maybe version that does not works for me works in your cdms 2.5 as well... :-)

>>> numpy.__version__
'1.9.0'
>>> cdms2.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__version__'

I wanted to check the changelog page to see if some changes about cdms2 were listed there, but it seems to be down like the pages with dynamic content : http://uvcdat-dev.llnl.gov/changelog.html

@chaosphere2112 is this changelog page a static page or does it have some dynamic content?

Now that I think of it, this version number is a bit confusing. I have checked my old cdms2 manual (cdms5.pdf, when cdms2 was introduced to replace cdms), and CDMS was already version 5.0 back in October 2007! But I guess not many people still have this pdf

[durack1]

@dnadeau4 @jypeter for what it's worth I think you're both talking about the same cdms2 "version":

cdms2/setup.py: version='5.0'

Since the evolution to UV-CDAT, the "version" of cdms2 as been tied to the UV-CDAT versioning, so:

In [1]: import cdms2
In [2]: cdms2.
...
In [2]: cdms2.cdat_info.
...
In [2]: cdms2.cdat_info.Version
Out[2]: '2.2.0-204-geef3c60'

[jypeter]

Thanks @durack1 ! So the version of cdms2 is the same as the version of UV-CDAT?

>>> cdms2.cdat_info.Version
'2.1.0'

This only seems to confirm that I'm using UV-CDAT 2.1.0
Does the 2.5 mentioned by @dnadeau4 mean that he is using some UV-CDAT 2.5?

What do you get with your 2.2.0-204-geef3c60 version when you run the example script I gave? Can you get the attributes with the underline prefix?

[chaosphere2112]

@durack1 @jypeter That use of cdat_info is incidental; cdat_info is a module packaged with UV-CDAT, and will always only reflect the current version of UV-CDAT. @dnadeau4 I believe was referring to the fact that the current version number of cdms2 in the setup.py script is 5.0; the way to get at that from userland is a little obscure:

import pkg_resources
pkg_resources.require("cdms2")[0].version

[jypeter]

That is indeed obscure! And I'm also getting '5.0'

>>> cdms2.cdat_info.Version
'2.1.0'
>>> import pkg_resources
>>> pkg_resources.require("cdms2")[0].version
'5.0'

[durack1]

@chaosphere2112 good point - so 5.0 is the "current" cdms2 version.. However I caveat that this version number hasn't been updated/revised in a very, very long time (since CDAT 5.2 (Sept 2009), or since the cdms2 v5.0 manual was written in October 2007) and there have been many bugfixes and tweaks to cdms2 (the largest being ESMF integration in mid-2012) since v5.0 was tagged, so the UV-CDAT tag is a more reliable versioning measure.. @doutriaux1 will have an opinion on this I'm sure..

The oldest version of cdms2 that I could find in the UV-CDAT repo is: '5.0' (Dec 2010)

@dnadeau4, cdms2/cdms3? versioning information is something else to add to the discussion list for planned "upgrades" (along with the existing issues: #481, #537, #576, #885, #1059 and #1288)

[durack1]

@jypeter I do think you're onto something here, so I just ran your script with a uvcdat_version tweak (as noted above the cdms2 version is '5.0') so we know what version I'm testing:

[durack1@oceanonly ~/150730_cdms2Test]$ ls *.nc | xargs -n 1 ncdump -h
netcdf dummy1 {
dimensions:
    latitude = 10 ;
    bound = 2 ;
    longitude = 10 ;
variables:
    int latitude(latitude) ;
        latitude:bounds = "bounds_latitude" ;
        latitude:units = "degrees_north" ;
        latitude:axis = "Y" ;
    double bounds_latitude(latitude, bound) ;
    int longitude(longitude) ;
        longitude:bounds = "bounds_longitude" ;
        longitude:units = "degrees_east" ;
        longitude:axis = "X" ;
        longitude:modulo = 360. ;
        longitude:topology = "circular" ;
    double bounds_longitude(longitude, bound) ;
    double my_var(latitude, longitude) ;
        my_var:_FillValue = 1.e+20 ;
        my_var:dummy_att = "Dummy att" ;
        my_var:units = "10e3 dummy" ;
        my_var:missing_value = 1.e+20 ;

// global attributes:
        :Conventions = "CF-1.0" ;
        :uvcdat_version = "2.2.0-204-geef3c60" ;
}
netcdf dummy2 {
dimensions:
    latitude = 10 ;
    bound = 2 ;
    longitude = 10 ;
variables:
    int latitude(latitude) ;
        latitude:bounds = "bounds_latitude" ;
        latitude:units = "degrees_north" ;
        latitude:axis = "Y" ;
    double bounds_latitude(latitude, bound) ;
    int longitude(longitude) ;
        longitude:bounds = "bounds_longitude" ;
        longitude:units = "degrees_east" ;
        longitude:axis = "X" ;
        longitude:modulo = 360. ;
        longitude:topology = "circular" ;
    double bounds_longitude(longitude, bound) ;
    double my_var(latitude, longitude) ;
        my_var:_FillValue = 1.e+20 ;
        my_var:dummy_att = "Dummy att" ;
        my_var:units = "10e3 dummy" ;
        my_var:missing_value = 999.999 ;

// global attributes:
        :Conventions = "CF-1.0" ;
        :uvcdat_version = "2.2.0-204-geef3c60" ;
}
netcdf dummy3 {
dimensions:
    latitude = 10 ;
    bound = 2 ;
    longitude = 10 ;
variables:
    int latitude(latitude) ;
        latitude:bounds = "bounds_latitude" ;
        latitude:units = "degrees_north" ;
        latitude:axis = "Y" ;
    double bounds_latitude(latitude, bound) ;
    int longitude(longitude) ;
        longitude:bounds = "bounds_longitude" ;
        longitude:units = "degrees_east" ;
        longitude:axis = "X" ;
        longitude:modulo = 360. ;
        longitude:topology = "circular" ;
    double bounds_longitude(longitude, bound) ;
    double my_var(latitude, longitude) ;
        my_var:_FillValue = 1.e+20 ;
        my_var:name = "variable_1" ;
        my_var:another_att = "New att" ;
        my_var:dummy_att = "Dummy att" ;
        my_var:units = "10e3 dummy" ;
        my_var:_another_att = "New att starting with _" ;

// global attributes:
        :Conventions = "CF-1.0" ;
        :uvcdat_version = "2.2.0-204-geef3c60" ;
}

It seems that the fill_value argument to cdms2.write is not overwriting the existing variable _FillValue as you noted above - I have seen some of these unusual behaviors with the _FillValue and missing_value attributes before.

Like @dnadeau4 I don't seem to have any problems with _NewAtt writing either..

[doutriaux1]

@jypeter this is post 2.2.0 anyway. But I noticed that dummy2.nc has the wrong _FillValue (1.e20 where it should say 999) Will take a look unless @dnadeau4 beats me to it.
@jypeter best way to know the version you're using is:

import cdat_info
print cdat_info.version()

example:

[2, 2, '0-215-gc8c7923']

which means 215 commits past 2.2.0 exact commit: c8c7923

Hope this helps.

[doutriaux1]

@durack1 my opinion is to make sure all PAckages are built reflecting the actual git commit used, sort of what is done in pcmdi_metrics
https://github.com/PCMDI/pcmdi_metrics/blob/master/setup.py#L5-L20

[doutriaux1]

#1476

[jypeter]

Thanks @doutriaux1 !

So, we know that there was a problem, in 2.1, and that it is partly solved (except for the fill_value=999.999 problem) in master. But can somebody say if it was already solved in 2.2? I only work with your official releases and I still have not had time to install 2.2.0

And #1476 mean that the UV-CDAT specific packages will have a .__version__ attribute? This will be useful if one day you can install (some) UV-CDAT packages inside another python distribution

[dnadeau4]

@jypeter I found the code in cdms2 where attributes are being set, I think I need a special call to the netcdf library to set _FillValue and it will require me to change/add new methods to cdms2. I need to investigate a little more, but a solution is feasible. As well, it seems like _FillValue for cdms2 can only be float. A lot to think about before making this happen, I suggest that you use NCO "ncatted" command for the time being. I will let you know this week if I can make a quick patch for you.

[jypeter]

@dnadeau4 don't worry about a quick patch, this whole issue is not blocking for me. I only started it because somebody mentioned the problem on the uvcdat-support list

[doutriaux1]

Fixed via #1524