PrePARE issues

[taylor13]

I reviewed the python PrePARE code and noticed the following potential problems:

1. I think the missing_value and _FillValue attribures are checked against the CMIP6 specified value. These are both float32 variables, and there is an if test for equivalence as follows:

 if 1 - (table_value / file_value) < 0.00001:

which is preceded by a test to trap the case of file_value=0.

This test is not rigid enough because, for example, a file_value = table_value/100. will satisfy the test even though we want the table_value and file_value to be the same. The expression on the left side of the if test comparison should be inside an absolute value function.

2. In testing whether time-label portion of the filename is consistent with time coordinates stored in the file, the code expects the bounds to be stored in a variable named "time_bnds" or "climatology_bnds". Currently the output specifications don't actually require these names; they require that the bounds be stored in a variable named in the "bounds" attribute or the "climatology" attribute. I wonder if we can modify PrePARE to extract the name of the bounds variable instead of having it hard wired. Is it too late to require these as the bounds names?

[glevava]

Hi Karl, regarding point 2 of your issue, I already add the following fix on my PrePARE instance (be part of my next PR):

        # Get first and last time bounds
        try:
            if 'bounds' in infile.variables['time'].__dict__.keys():
                bndsvar = infile.variables['time'].__dict__['bounds']
                startimebnds = infile.variables[bndsvar][0][0]
                endtimebnds = infile.variables[bndsvar][-1][1]
            else:
                startimebnds = infile.variables['time_bnds'][0][0]
                endtimebnds = infile.variables['time_bnds'][-1][1]
        except BaseException:
            startimebnds = 0
            endtimebnds = 0

[glevava]

For the point 1, do you mean it should be:
if abs(1 - (table_value / file_value)) < 0.00001:

If yes, I will include this change in my PR.

[taylor13]

Yes, that was my suggestion. The intent is to make sure file_value is the same (or almost the same) as table_value. Note that the code shown above is preceded by a test for a special case. We could also eliminate the test for the case when file_value is 0. by writing it:

if abs(table_value - file_value) < 0.00001*table_value 

This would require file_value to be equal to table_value when table_value=0., and when not, the two would be required to very nearly be the same (i.e., to within 0.001%). The approximate condition will allow for truncation errors. These truncation errors should be avoidable, so perhaps we should raise a warning when exact equivalence is not found. What do you think?

[taylor13]

closed by mistake, but the original fix has been submitted as a pull request by @glevava

[dnadeau4]

This has been merged with @glevava solution.
b11e29e

[taylor13]

Might use my equation that eliminates the if test for 0.

[taylor13]

To be more general (e.g., for cases where the table_value<0), we need another abs function in the test:

if abs(table_value - file_value) < 0.00001*abs(table_value) 

[doutriaux1]

@dnadeau4 is c38e4ad part of any branch? Is that in the stuff you're going to merge from the prepare branch? Should I take over that branch and do the merge myself?

[dnadeau4]

@doutriaux1 this is done, not sure why @taylor13 reopened it. The climatology was merged but guillaume changed a couple of line that need to be merged again and all tests need to be run.

[taylor13]

I haven't rechecked, but last time I looked the correct if-test specified in #332 (comment) had not been implemented. That's why I reopened.

[taylor13]

In #332 (comment) , the < symbol should be replaced with symbol for less than or equal to. Thanks, @doutriaux1 , for catching the error.

with this change the test will be passed even if both the fill value and table value are 0.0