esmf regridder fills in data where non should be

[doutriaux1]

I am not sure if this is a cdms2 or esmf problem.

when starting with a variable covering only part of the gobe and regidding it to a full globe, both regrid2 and libcf return the "expected" behaviour i.e set all ares that were not present originally to "missing".

In contrast esmf seems to do a filling of the data in the newly created space.

example test:

from __future__ import print_function
import cdms2
import unittest
import regrid2
import cdat_info


class ESMFHalfGlobe(unittest.TestCase):
    def setUp(self):
        self.clt = cdms2.open(cdat_info.get_sampledata_path()+"/clt.nc")("clt",latitude=(-90,0),time=slice(0,1),squeeze=1)
        self.T42 = cdms2.createGaussianGrid(64)

    def testRegrid2(self):
        reg = self.clt.regrid(self.T42,regridTool="regrid2")
        nmiss = 128*64-reg.count()
        print("REGRID2 Number of missing:",nmiss)

    def testEsmfLinear(self):
        reg = self.clt.regrid(self.T42,regridTool="esmf",regridMethod="linear")
        nmiss = 128*64-reg.count()
        print("ESMF Linear Number of missing:",nmiss)

    def testEsmfConservative(self):
        reg = self.clt.regrid(self.T42,regridTool="esmf",regridMethod="conservative")
        nmiss = 128*64-reg.count()
        print("ESMF Conservative Number of missing:",nmiss)

    def testlibcf(self):
        reg = self.clt.regrid(self.T42,regridTool="libcf")
        nmiss = 128*64-reg.count()
        print("libcf Number of missing:",nmiss)

produces:

RUNNNIG FROM: /var/folders/nv/3xl0t1xx4yxb6tyd0yqdm238001cpd/T/tmpyFdzCs
('Names:', ['/git/cdms/tests/test_EsmfHalfGlobe.py'])
Executing nosetests -s /git/cdms/tests/test_EsmfHalfGlobe.py in /private/var/folders/nv/3xl0t1xx4yxb6tyd0yqdm238001cpd/T/tmpyFdzCs
WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =    0.00, boundMax =   -2.00
ESMF Conservative Number of missing: 0
.ESMF Linear Number of missing: 0
.REGRID2 Number of missing: 3968
./Users/doutriaux1/anaconda2/envs/nightly2/lib/python2.7/site-packages/cdms2/avariable.py:1152: Warning:
avariable.regrid: We chose regridMethod = linear for you among the following choices:
    'conserve' or 'linear' or 'patch'
  warnings.warn(message, Warning)
libcf Number of missing: 4224
.
----------------------------------------------------------------------
Ran 4 tests in 2.603s
OK
Ran 1 tests, 0 failed (100.00% success)

Note that ESMF regridding have 0 missing values.

[durack1]

@doutriaux1 is this the linux vs linux vs MacOS issue? So it's how masks are dealt with on different systems? So PCMDI/pcmdi_metrics#303, and I'm sure a couple of other open issues around the place?

[doutriaux1]

@durack1 nope this is true on all oses. Linux and Mac OSX.

[durack1]

@doutriaux1 this is worrying (from above #208 (comment)):

ESMF Conservative Number of missing: 0
.ESMF Linear Number of missing: 0
.REGRID2 Number of missing: 3968
...
libcf Number of missing: 4224

[doutriaux1]

@durack1 yes it is VERY worrysome. It might be an ESMF bug though, we should try to reproduce this outside of cdms2

[doutriaux1]

Then again this is kind of a twisted case where you regrid data to an area where you had no data intersecting originally...

[durack1]

@doutriaux1 the differences between REGRID2: 3968 and libcf: 4224 means that this is not just ESMF.. I'd not be surprised if this is the issue that's causing the problems between platforms

[doutriaux1]

I'm not too concerned about the difference there, depending on the interpolation technique you could have libcf being a little more strict than regrid2

[dnadeau4]

@doutriaux1 Can you provide us with some pictures? I think the issue might be the count() method actually.

[doutriaux1]

picture are full, I'll upload them shortly.

[dnadeau4]

full of missing value? full???

[doutriaux1]

[durack1]

@dnadeau4 that equatorial stripe looks very similar to the behavior in #195 (comment)

[dnadeau4]

Last time it was bounds issues in the input file.

[jypeter]

By the way, is it normal that boundMax is smaller than coordMax , if you have retrieved (-90, 0). I would expect it to be >= coordMax
coordMin = -90.00, boundMin = -90.00, coordMax = 0.00, boundMax = -2.00

[dnadeau4]

You do not have a mask in clt so ESMF fills the grid with 0s. Your latitude for clt is only in the South Hemisphere.

self.clt.getLatitude()[:]
array([-90., -86., -82., -78., -74., -70., -66., -62., -58., -54., -50.,
       -46., -42., -38., -34., -30., -26., -22., -18., -14., -10.,  -6.,
        -2.], dtype=float32)

You get the same plot for each case, except that regrid2 and libcf will mask the value.
The linear plot was interesting, ESMF try to smear the data all the way to the pole.

Do you know why vcs plot it reverse?

[dnadeau4]

@jypeter You are right, we go this info reversed 😄 Thanks!

https://github.com/UV-CDAT/cdms/blob/master/Lib/mvCdmsRegrid.py#L390-L395

[jypeter]

@dnadeau4 the reverse plotting is an oooold (when Dean and Bob were kids and I'm not sure @doutriaux1 was even born :p) vcs feature, that makes it possible to make a kind of quick Quality Control:

• if the values of the latitude axis are increasing, the plot will have a normal look (-90 at the bottom of the plot)
• if the values of the latitude axis are decreasing (e.g. from +90 to -90, e.g. in the IPSL model), the plot will have the South pole at the top of the plot

[dnadeau4]

@doutriaux1 I was thinking of adding this code for ESMF, but 0 could be real data. There seems to be no way for ESMF to set non-interpolated value to something else than 0. That is why I did not mask the 0 value from ESMF method.

+            interpolate = ro(self, **keywords)
+            if(regridTool == 'esmf'):
+                interpolate = cdms2.MV2.masked_where((interpolate == 0), interpolate)
+            return interpolate

[dnadeau4]

Also, I set unMappedAction to IGNORE to avoid ESMF to stop. I could set it to ERROR which will tell your program, that your grid is not right.

Here is what get with your program when I set it to ESMF.UnmappedAction.ERROR
Is this better?

Traceback (most recent call last):
  File "tests/test_regrid22.py", line 27, in testEsmfConservative
    reg = self.clt.regrid(self.T42,regridTool="esmf",regridMethod="conservative")
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/cdms2/avariable.py", line 1180, in regrid
    **keywords)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/cdms2/mvCdmsRegrid.py", line 428, in __init__
    self.regridObj.computeWeights(**args)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/regrid2/mvGenericRegrid.py", line 155, in computeWeights
    self.tool.computeWeights(**args)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/regrid2/mvESMFRegrid.py", line 272, in computeWeights
    ignore_degenerate=False)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/util/decorators.py", line 64, in new_func
    return func(*args, **kwargs)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/api/regrid.py", line 128, in __init__
    dstFracField=dst_frac_field)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/util/decorators.py", line 52, in new_func
    return func(*args, **kwargs)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/interface/cbindings.py", line 1759, in ESMP_FieldRegridStore
    '.    '+constants._errmsg)
ValueError: ESMC_FieldRegridStore() failed with rc = 513.    Please check the log files (named "*ESMF_LogFile").

[doutriaux1]

@dnadeau4 I think it should run. Why don't you set it to so me really big number like 1.E20 or even bigger (can you set it to numpy.nan ?) And then mask everything set to this value back to a mask.

[dnadeau4]

I did that and ESMF set it back to 0.

@@ -449,7 +451,7 @@ coordMin = %7.2f, boundMin = %7.2f, coordMax = %7.2f, boundMax = %7.2f
                 re.search('conserv', self.regridMethod) is None:
             dstData *= missingValue
         else:
-            dstData *= 0.0
+            dstData *= 1e20

[dnadeau4]

This is now working in fixesmf branch.