point2grid: support the 2D time variable

[hsoh-u]

Requested from RT ticket (Binyu, https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97459)

Describe the New Feature

The sample file of RT-Ticket 97459 has (x,y) dimensions for coordinate variables (lat/lon/time) and MET's NCCF library does not support 2 dimensional time variable. The 2D time variable does not make sense for the gridded data. But it could be used for the point observation data (sparse point data). It would be nice to support 2D time variable for the point2grid.

Background assessment from helpdesk ticket:

From John:
Binyu,

I'm adding my response here and will then reassign this ticket to Howard Soh to see if he has any more ideas.

I downloaded your file to our anonymous ftp site:

ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data_20201116/VOLCAT_HIMAWARI-8_FLDK_s2020296_173000_v300250_VCB_w167_FLDK_b2020295_204000_g001.nc

I'm able to replicate the same error you're getting:

terminate called after throwing an instance of 'netCDF::exceptions::NcInvalidCoords'

After some testing I see that this is caused by the time variable having 2 dimensions: int time(y, x) ;

When reading 2-D gridded data, the code expects the time variable to be 1 dimensional. A file may contain multiple timesteps and a coordinate variable defines what those timesteps are. To sidestep this issue, I just removed the "time(y, x)" variable entirely. But then MET isn't able to read the projection info:

ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out projection from information in netCDF file.

Looking closely at the latitude and longitude variables, I see that this really is not gridded data at all. Instead, it's a dense set of pixels, but it doesn't appear that the lat/lon's are evenly spaced or follow any sort of projection (e.g. Lambert Conformal, Mercator, PolarStereographic, and so on).

Since this is a dense collection of individual points, you'll still need to run the Point2Grid tool to place them onto a real grid. And that's the question to Howard. Is there anyway that Binyu could format this NetCDF file that would enable it to be read easily by the Point2Grid tool?

From Howard:
The point2grid handles a CF complaint NetCDF which has

1. 2D variables with lat/lon dimensions and the time variable with 1 value OR
2. 3D variables with time/lat/lon dimensions and the time variable with 1 dimension.

The point2grid should be enhanced to support the 2-dimensional time variable (if the enhancement is approved).

The sample NetCDF has two time offsets: 1603387817 and 1603387833 ("seconds since 1970-01-01 00:00:00"). One possible way without changing MET applications is generating two files (one file with time offset 1603387817 and the other file with time offset 1603387817).

netcdf1:
int time(1);
time = 1603387817
fill the missing value for the time 1603387833

netcdf2:
int time(1);
time = 1603387833
fill the missing value for the time 1603387817

Acceptance Testing

point2grid should process the file at /d3/personal/hsoh/data/RT-97459//VOLCAT_HIMAWARI-8_FLDK_s2020296_173000_v300250_VCB_w167_FLDK_b2020295_204000_g001.nc

Time Estimate

1 or 2 days

Sub-Issues

Consider breaking the new feature down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

NONE.

Funding Source

2791541

Define the Metadata

Assignee

• Select engineer(s) or no engineer required (Howard)
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[hsoh-u]

The sample file does not have the standard_name attribute. MET fails to find the projection information (lat/lon).
It is a CF complaint NetCDF file (the "standard_name" attribute is optional if the "long_name" attribute exists). The long_name attribute is not standardized. MET does not identify the coordinate variables from the long_name. The "standard_name" attribute is required for MET.

[JohnHalleyGotway]

While the changes merged for PR #1616 did not cause any of the MET unit tests to fail, they did cause 2 failures for METplus which were found by GitHub actions:
https://github.com/dtcenter/METplus/runs/1697787026?check_suite_focus=true

On kiowa, here's a plot_data_plane command that works fine for met-9.1:

/usr/local/met-9.1/bin/plot_data_plane \
/d1/projects/METplus/METplus_Data/model_applications/space_weather/glotec_vs_glotec/GLO_20190422_without_cosmic/GloTEC_TEC_2015_03_17.nc \
plot.ps \
'name="TEC"; level="(20150317_000500,*,*)"; file_type=NETCDF_NCCF;'

But the same command fails with these changes:

/d1/projects/MET/MET_pull_requests/met-10.0.0_beta3/feature_1580/MET-feature_1580_2d_time_into_develop/met/bin/plot_data_plane \
/d1/projects/METplus/METplus_Data/model_applications/space_weather/glotec_vs_glotec/GLO_20190422_without_cosmic/GloTEC_TEC_2015_03_17.nc \
plot.ps \
'name="TEC"; level="(20150317_000500,*,*)"; file_type=NETCDF_NCCF;'

And here's the output:

WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) -> the requested time 20150317_000500 for "TEC" variable does not exist (20150317_000416 and 20150317_235544).
WARNING: 
ERROR  : 
ERROR  : plot_data_plane -> trouble getting field "name="TEC"; level="(20150317_000500,*,*)"; file_type=NETCDF_NCCF;" from file "/d1/projects/METplus/METplus_Data/model_applications/space_weather/glotec_vs_glotec/GLO_20190422_without_cosmic/GloTEC_TEC_2015_03_17.nc"
ERROR  : 

NOTE!!! that the reported range of timestamps (20150317_000416 and 20150317_235544) does not match the ACTUAL range of timestamps within the file:
1426550700 to 1426636500 which is 20150317_000500 to 20150317_235500.

[JohnHalleyGotway]

Howard, please fix this and also add this example as a new MET unit test. Perhaps just add it to unit_plot_data_plane.xml.

Once it is fixed, please do a new pull request.

[hsoh-u]

lat/long variables will be decided by

• By checking standard_name attribute
• By checking units attribute

GLO_20190422_without_cosmic/GloTEC_TEC_2015_03_17.nc does have the standard_name attribute.
If there are two or more variables with "degrees_north" or "degrees_east", it will log proper message and quit.

The sample data file (VOLCAT_HIMAWARI-8_FLDK_s2020296_173000_v300250_VCB_w167_FLDK_b2020295_204000_g001.nc) has multiple variables with "degree_north:

   geospatial_lat_lon_extent:geospatial_lat_units = "degrees_north" ;
   feature_lat_lon_extent:geospatial_lat_units = "degrees_north" ;
   pc_feature_lat_lon_extent:geospatial_lat_units = "degrees_north" ;
   volcano_info:latitude_units = "degrees_north" ;
   feature_latitude:units = "degrees_north" ;
   pc_feature_latitude:units = "degrees_north" ;
   feature_polygon_latitude:units = "degrees_north" ;
   pc_feature_polygon_latitude:units = "degrees_north" ;
   latitude:units = "degrees_north" ;
   pc_latitude:units = "degrees_north" ;