New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

esmf regridder fills in data where non should be #208

Closed
doutriaux1 opened this Issue Jan 19, 2018 · 22 comments

Comments

Projects
None yet
4 participants
@doutriaux1
Member

doutriaux1 commented Jan 19, 2018

I am not sure if this is a cdms2 or esmf problem.

when starting with a variable covering only part of the gobe and regidding it to a full globe, both regrid2 and libcf return the "expected" behaviour i.e set all ares that were not present originally to "missing".

In contrast esmf seems to do a filling of the data in the newly created space.

example test:

from __future__ import print_function
import cdms2
import unittest
import regrid2
import cdat_info


class ESMFHalfGlobe(unittest.TestCase):
    def setUp(self):
        self.clt = cdms2.open(cdat_info.get_sampledata_path()+"/clt.nc")("clt",latitude=(-90,0),time=slice(0,1),squeeze=1)
        self.T42 = cdms2.createGaussianGrid(64)

    def testRegrid2(self):
        reg = self.clt.regrid(self.T42,regridTool="regrid2")
        nmiss = 128*64-reg.count()
        print("REGRID2 Number of missing:",nmiss)

    def testEsmfLinear(self):
        reg = self.clt.regrid(self.T42,regridTool="esmf",regridMethod="linear")
        nmiss = 128*64-reg.count()
        print("ESMF Linear Number of missing:",nmiss)

    def testEsmfConservative(self):
        reg = self.clt.regrid(self.T42,regridTool="esmf",regridMethod="conservative")
        nmiss = 128*64-reg.count()
        print("ESMF Conservative Number of missing:",nmiss)

    def testlibcf(self):
        reg = self.clt.regrid(self.T42,regridTool="libcf")
        nmiss = 128*64-reg.count()
        print("libcf Number of missing:",nmiss)

produces:

RUNNNIG FROM: /var/folders/nv/3xl0t1xx4yxb6tyd0yqdm238001cpd/T/tmpyFdzCs
('Names:', ['/git/cdms/tests/test_EsmfHalfGlobe.py'])
Executing nosetests -s /git/cdms/tests/test_EsmfHalfGlobe.py in /private/var/folders/nv/3xl0t1xx4yxb6tyd0yqdm238001cpd/T/tmpyFdzCs
WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =    0.00, boundMax =   -2.00
ESMF Conservative Number of missing: 0
.ESMF Linear Number of missing: 0
.REGRID2 Number of missing: 3968
./Users/doutriaux1/anaconda2/envs/nightly2/lib/python2.7/site-packages/cdms2/avariable.py:1152: Warning:
avariable.regrid: We chose regridMethod = linear for you among the following choices:
    'conserve' or 'linear' or 'patch'
  warnings.warn(message, Warning)
libcf Number of missing: 4224
.
----------------------------------------------------------------------
Ran 4 tests in 2.603s
OK
Ran 1 tests, 0 failed (100.00% success)

Note that ESMF regridding have 0 missing values.

@doutriaux1 doutriaux1 added this to the 3.0 milestone Jan 19, 2018

@durack1

This comment has been minimized.

Member

durack1 commented Jan 19, 2018

@doutriaux1 is this the linux vs linux vs MacOS issue? So it's how masks are dealt with on different systems? So PCMDI/pcmdi_metrics#303, and I'm sure a couple of other open issues around the place?

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 19, 2018

@durack1 nope this is true on all oses. Linux and Mac OSX.

@durack1

This comment has been minimized.

Member

durack1 commented Jan 19, 2018

@doutriaux1 this is worrying (from above #208 (comment)):

ESMF Conservative Number of missing: 0
.ESMF Linear Number of missing: 0
.REGRID2 Number of missing: 3968
...
libcf Number of missing: 4224
@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 19, 2018

@durack1 yes it is VERY worrysome. It might be an ESMF bug though, we should try to reproduce this outside of cdms2

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 19, 2018

Then again this is kind of a twisted case where you regrid data to an area where you had no data intersecting originally...

@durack1

This comment has been minimized.

Member

durack1 commented Jan 19, 2018

@doutriaux1 the differences between REGRID2: 3968 and libcf: 4224 means that this is not just ESMF.. I'd not be surprised if this is the issue that's causing the problems between platforms

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 19, 2018

I'm not too concerned about the difference there, depending on the interpolation technique you could have libcf being a little more strict than regrid2

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Jan 22, 2018

@doutriaux1 Can you provide us with some pictures? I think the issue might be the count() method actually.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 23, 2018

picture are full, I'll upload them shortly.

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Jan 23, 2018

full of missing value? full???

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 23, 2018

crap

@durack1

This comment has been minimized.

Member

durack1 commented Jan 23, 2018

@dnadeau4 that equatorial stripe looks very similar to the behavior in #195 (comment)

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Jan 23, 2018

Last time it was bounds issues in the input file.

@jypeter

This comment has been minimized.

jypeter commented Jan 24, 2018

By the way, is it normal that boundMax is smaller than coordMax , if you have retrieved (-90, 0). I would expect it to be >= coordMax
coordMin = -90.00, boundMin = -90.00, coordMax = 0.00, boundMax = -2.00

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Feb 6, 2018

You do not have a mask in clt so ESMF fills the grid with 0s. Your latitude for clt is only in the South Hemisphere.

self.clt.getLatitude()[:]
array([-90., -86., -82., -78., -74., -70., -66., -62., -58., -54., -50.,
       -46., -42., -38., -34., -30., -26., -22., -18., -14., -10.,  -6.,
        -2.], dtype=float32)

You get the same plot for each case, except that regrid2 and libcf will mask the value.
The linear plot was interesting, ESMF try to smear the data all the way to the pole.

Do you know why vcs plot it reverse?

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Feb 6, 2018

@jypeter You are right, we go this info reversed 😄 Thanks!

https://github.com/UV-CDAT/cdms/blob/master/Lib/mvCdmsRegrid.py#L390-L395

@jypeter

This comment has been minimized.

jypeter commented Feb 6, 2018

@dnadeau4 the reverse plotting is an oooold (when Dean and Bob were kids and I'm not sure @doutriaux1 was even born :p) vcs feature, that makes it possible to make a kind of quick Quality Control:

  • if the values of the latitude axis are increasing, the plot will have a normal look (-90 at the bottom of the plot)
  • if the values of the latitude axis are decreasing (e.g. from +90 to -90, e.g. in the IPSL model), the plot will have the South pole at the top of the plot
@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Feb 6, 2018

@doutriaux1 I was thinking of adding this code for ESMF, but 0 could be real data. There seems to be no way for ESMF to set non-interpolated value to something else than 0. That is why I did not mask the 0 value from ESMF method.

+            interpolate = ro(self, **keywords)
+            if(regridTool == 'esmf'):
+                interpolate = cdms2.MV2.masked_where((interpolate == 0), interpolate)
+            return interpolate
@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Feb 6, 2018

Also, I set unMappedAction to IGNORE to avoid ESMF to stop. I could set it to ERROR which will tell your program, that your grid is not right.

Here is what get with your program when I set it to ESMF.UnmappedAction.ERROR
Is this better?

Traceback (most recent call last):
  File "tests/test_regrid22.py", line 27, in testEsmfConservative
    reg = self.clt.regrid(self.T42,regridTool="esmf",regridMethod="conservative")
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/cdms2/avariable.py", line 1180, in regrid
    **keywords)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/cdms2/mvCdmsRegrid.py", line 428, in __init__
    self.regridObj.computeWeights(**args)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/regrid2/mvGenericRegrid.py", line 155, in computeWeights
    self.tool.computeWeights(**args)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/regrid2/mvESMFRegrid.py", line 272, in computeWeights
    ignore_degenerate=False)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/util/decorators.py", line 64, in new_func
    return func(*args, **kwargs)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/api/regrid.py", line 128, in __init__
    dstFracField=dst_frac_field)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/util/decorators.py", line 52, in new_func
    return func(*args, **kwargs)
  File "/software/anaconda2/envs/py2/lib/python2.7/site-packages/ESMF/interface/cbindings.py", line 1759, in ESMP_FieldRegridStore
    '.    '+constants._errmsg)
ValueError: ESMC_FieldRegridStore() failed with rc = 513.    Please check the log files (named "*ESMF_LogFile").

@dnadeau4 dnadeau4 added the invalid label Feb 6, 2018

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Feb 6, 2018

@dnadeau4 I think it should run. Why don't you set it to so me really big number like 1.E20 or even bigger (can you set it to numpy.nan ?) And then mask everything set to this value back to a mask.

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Feb 6, 2018

I did that and ESMF set it back to 0.

@@ -449,7 +451,7 @@ coordMin = %7.2f, boundMin = %7.2f, coordMax = %7.2f, boundMax = %7.2f
                 re.search('conserv', self.regridMethod) is None:
             dstData *= missingValue
         else:
-            dstData *= 0.0
+            dstData *= 1e20
@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Feb 13, 2018

This is now working in fixesmf branch.

@dnadeau4 dnadeau4 closed this Feb 13, 2018

@doutriaux1 doutriaux1 removed the invalid label Feb 14, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment