Implement the GetValues class

[travissluka]

This class does the interpolation of State variables to observations locations, and passes it to UFO to run the forward operator.
I need to check and see what the best way to do the interpolation is now (there are generic interpolation routines in OOPS that use SABER/BUMP... I'll try to track these down for you)

TestGetValues should pass afterward.

[loganchen39]

Thanks, I'm also checking the code.

[travissluka]

@loganchen39 I think oops/generic/InterpolatorUnstructured.h is what we want to use for the interpolation. There is a similar InterpolatorAtlas.h but that one does not have apply_ad() to apply the adjoint of the interpolation, so we can't use it with our LinearGetValues class (we might as well use the same interpolation for GetValues and LinearGetValues)

[travissluka]

@mmiesch do you know what the plans are for adding the masking in InterpolatorUnstructured?

@loganchen39 You can go ahead and start working on GetValues. If we don't have masking by the end of that work then we can just use nco/cdo to fill in the missing values of the sst background to create an "aqua planet" for our tests. We'll worry about masking later.

[loganchen39]

Ok,

…

On 11/2/2020 7:32 PM, Travis Sluka wrote: @mmiesch <https://github.com/mmiesch> do you know what the plans are for adding the masking in |InterpolatorUnstructured|? @loganchen39 <https://github.com/loganchen39> You can go ahead and start working on |GetValues|. If we don't have masking by the end of that work then we can just use nco/cdo to fill in the missing values of the sst background to create an "aqua planet" for our tests. We'll worry about masking later. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMUO2PEWVEBEB35A4KCGJH3SN5FQPANCNFSM4THV6QPQ>.

[mmiesch]

@mmiesch do you know what the plans are for adding the masking in InterpolatorUnstructured?

I wasn't aware of any plans but it would be pretty straightforward to do. I could create a ZenHub ticket. I personally would be unlikely to get to it in November but maybe December. Ot, others could do it sooner.

[travissluka]

note to self: https://github.com/JCSDA-internal/oops/issues/952 will be needed to properly finish our GetValues

[loganchen39]

@travissluka How do we define the observation grid/functionSpace which will be needed for interpolation? I have uploaded the obs data we'll use to branch feature/GetValues, which has grid of 0.05x0.05 degree, i.e. 3600x7200. For each day it has 2 files for "day" and "night" respectively. Be aware if you use ncview to view the data, it will show almost all of the data are missing values, while actually it's not. Jim and I used to plot the data, and during early periods of 1980s, about 20%~30% are valid data. In this case the grid/FunctionSpace should be easy to define.

I checked your "obs_sst.nc". Will we use it? How to define the Unstructured Grid that can be used for interpolation? Maybe we can have a discussion sometime.

[travissluka]

@loganchen39 those files should not be placed in this repository. The test data is intended to be small so that all the ctests run in just a couple of seconds. obs_sst.nc has only ~100 observations. Additionally, the files you put up are the raw input files that need to be converted into ioda format, that is handled by the ioda-converters repo, and there are already tests in there to convert GDS2.0 formatted SST files into ioda format.

ioda/ufo handles all the loading of observations, so this isn't something you need to worry about. The locations of observations are provided by ufo::Locations & locs in the GetValues constructor. Hopefully this will all make more sense after the jedi academy next week.

[loganchen39]

@travissluka Ok, I'll delete the raw SST files which are too big. I have following questions.

1. For current test/implementation of class GetValues, I'm supposed to use your obs_sst.nc file, right? If so, in order to use the function of apply_ad() from InterpolatorUnstructured.h , we need to provide "const atlas::FunctionSpace *out_fspace_;" which should contain obs grid info. So we need to create such functionSpace object from the location info. from file obs_sst.nc, right?

2. For the large raw SST files, should I try to test to convert to ioda format, or should I just leave it alone for now?

[travissluka]

1. Correct. Unfortunately now that I look at it more I realize that Locations does not have the C++ methods we need to access the data 🤦 That all exists on the Fortran side. JEDI team is supposed to be rewriting Locations at some point soon (https://github.com/JCSDA-internal/ufo/issues/41). I need to think about whether or not I want to put temporary interfaces on the C++ side of Locations until they get a chance to rewrite it, or if I want to write the Fortran glue code on our side for now.... I'll get back to you about that. (I'll get it figured out while you're doing the academy)
2. ignore for now. we know we have the tools to do that, and they work. Plus, those tools are going to change soon because of ioda changing, so there isn't really any point looking at it now.