Handle expected no input for ascii2nc and point2grid

[georgemccabe]

Based on discussion in this MET Help ticket: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98329

Describe the Enhancement

Logan (NOAA) would like to process data with ascii2nc that occasionally has no obs data, which is expected. When this is the case, we would like ascii2nc to write out a file with no observations in it. This could be a configurable setting in case other users prefer the tool to skip writing a file if no data is present.

The data is then passed into point2grid. It is unclear how point2grid would react to an empty input file. We should also investigate what is needed to handle this situation in that application as well.

Time Estimate

Estimate the amount of work required here.
1 day.

Sub-Issues

No sub-issues required.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2791541

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: John HG
• Select scientist(s) or no scientist required: Logan

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
  No impacts.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

As of met-9.1, zero observations causes ascii2nc to error out:

ERROR  : Zero observations retained.
ERROR  : Cannot create NetCDF Observation file: sample_ascii_obs_empty.nc

Updated ascii2nc to instead print a warning message and create an empty output file:

WARNING: 
WARNING: Zero observations retained!
WARNING: 
DEBUG 2: Processing observations for 0 headers.
DEBUG 1: Creating NetCDF Observation file: sample_ascii_obs_empty.nc
DEBUG 2: Finished processing 0 observations for 0 headers.

As of met-9.1, zero observations causes point2grid to error out:

WARNING: get_grid_mapping(MET_obs)  no valid point observation data
ERROR  : process_point_file() ->  No valid data for the variable [APCP]

Updated point2grid to instead print a warning message and create an output file with default values:

WARNING: 
WARNING: get_grid_mapping(MET_obs) -> no valid point observation data!
WARNING: 
WARNING: 
WARNING: process_point_file() -> No valid data for the variable [APCP]
WARNING: 
DEBUG 1: Writing output file: sample_point2grid.nc

[JohnHalleyGotway]

This is fixed for the met-10.0.0 beta3 release. @LoganDawson-NOAA once that release is available, please test to confirm that ascii2nc and point2grid handle zero observations as you need and expect. If you find any issues, we can just re-open this issue.

[LoganDawson-NOAA]

@JohnHalleyGotway I tested out met-10.0.0 beta3 on a date with zero observations, and ascii2nc isn't quite producing output that point2grid can process. For comparison, I've included differences from the headers of the netcdf files resulting from ascii2nc when observations exist versus when there are no obs to process. Both files were produced with met-10.0.0 beta3.

When there are observations to be processed by ascii2nc, the dimensions are listed as follows and key variable and global attribute differences are:

dimensions:
        mxstr = 16 ;
        mxstr2 = 40 ;
        mxstr3 = 80 ;
        nobs = 3 ;
        nhdr = 3 ;
        nhdr_typ = 1 ;
        nhdr_sid = 1 ;
        nhdr_vld = 3 ;
        nobs_qty = 2 ;
        obs_var_num = 1 ;
variables:
        int obs_vid(nobs) ;
                obs_vid:long_name = "index of variable names at var_name" ;
                obs_vid:_FillValue = -9999 ;
        char obs_var(obs_var_num, mxstr2) ;
                obs_var:long_name = "variable names" ;

// global attributes:
                :MET_Obs_version = "1.02" ;
                :use_var_id = "true" ;

When no observations exist, the dimensions are listed as follows and key variable and global attribute differences are:

dimensions:
        mxstr = 16 ;
        mxstr2 = 40 ;
        mxstr3 = 80 ;
        nobs = UNLIMITED ; // (0 currently)
        nhdr = UNLIMITED ; // (0 currently)
        nhdr_typ = UNLIMITED ; // (0 currently)
        nhdr_sid = UNLIMITED ; // (0 currently)
        nhdr_vld = UNLIMITED ; // (0 currently)
        nobs_qty = UNLIMITED ; // (0 currently)
variables:
        int obs_gc(nobs) ;
                obs_gc:long_name = "grib code corresponding to the observation type" ;
                obs_gc:_FillValue = -9999 ;

// global attributes:
                :MET_Obs_version = "1.02" ;
                :use_var_id = "false" ;

In the correct output, the value for obs_var is "Fscale" which is the variable expected by point2grid. Without the obs_var variable in the ascii2nc output, point2grid begins creating a netcdf output file with the correct lat/lon data, but it hangs and has to be forcibly terminated.

For additional reference, I've attached files showing the commands and output that result from ascii2nc and point2grid when processing zero observations:
ascii2nc_empty.txt
point2grid_empty.txt

The same information is provided when there are observations available:
ascii2nc_w-obs.txt
point2grid_w-obs.txt

[JohnHalleyGotway]

Logan, You're right. When I run the commands listed below to test, it just hangs as well: MET_BIN=/Volumes/d1/projects/MET/MET_development/MET-develop/met/bin touch empty_obs.txt $MET_BIN/ascii2nc empty_obs.txt empty_obs.nc -format met_point $MET_BIN/point2grid empty_obs.nc G211 empty_grid.nc -field 'name="Fscale"; level="";' I apologize for this. When I ran this test on my development feature branch it worked fine. I'll look into it and figure out what's going on.

[JohnHalleyGotway]

Reopening this issue and assigning it to the met-10.0.0-beta4 release.

My original point2grid test did actually run fine:

point2grid empty_obs.nc G211 empty_grid.nc -field 'name="TMP"; level="Z2";' -v 10

However, running with any name string that is NOT a standard GRIB code abbreviation (like "Fscale" or "bad_string") causes point2grid to hang:

point2grid empty_obs.nc G211 empty_grid.nc -field 'name="Fscale"; level="Z2";' -v 10

Reopening and will fix.

[JohnHalleyGotway]

@hsoh-u I found/fixed the bug that was causing point2grid to hang. It was a cut/paste error.

In PointToGridConfInfo::get_var_id(), we had var_name_map.end() that should be def_var_name_map.end(). Fixing that gets rid of the runtime hang.

But bigger question. When the input to point2grid has 0 observations, and I request "Fscale", it errors out with:

bin/point2grid empty_obs.nc G211 empty_grid.nc -field 'name="Fscale"; level="";'
ERROR  : process_point_file() -> Invalid GRIB code [Fscale]
ERROR  : Try setting the "name" in the "-field" command line option to one of the available names:
ERROR  : 	GRIB codes: 

I can make this go away by adding an entry for "Fscale" to the var_name_map in the config file:

var_name_map = [
...
{ key = "Fscale";   val = "Fscale"; } 
];

And then rerunning with that config file, I now get a warning about 0 obs with an output file.

bin/point2grid empty_obs.nc G211 empty_grid.nc -field 'name="Fscale"; level="";' -config Point2GridConfig 
WARNING: get_grid_mapping(MET_obs) -> no valid point observation data!
WARNING: process_point_file() -> No valid data for the variable [Fscale]
DEBUG 1: Writing output file: empty_grid.nc

Can you confirm that this is the desired behavior? If the user provides a variable name that (1) doesn't appear in the input NetCDF file and (2) isn't listed in the config file, the point2grid tool should error out? Is that the right behavior?

[hsoh-u]

I don't think it's right behavior.
It's a side effect. Without mapping in config file, MET knows "Fscale" is not valid GRIB code. MET exits with an error message (without scanning the GRIB code variable). The existence at the mapping from the configuration makes MET scan the GRIB code variable like the other filtering logic (Note: The mapping is "GRIB code_string" to "variable_name", for example, "11" to "TMP") ==> Should not allow non-numeric key at the mapping configuration. If allowed, MET should detect this early.

Why this happens:
Step 1: If the variable index is used, MET knows if the requested variable exists or not very quickly without scanning a whole variable. This is an advantage of using variable name and index mapping (only if variable index is used).

Step 2: Scanning & filtering all obs data. The mapping allows "variable name" for GRIB code. The obs data could be filtered by requested GRIB code, but MET knows it after scanning all obs. data.

The point2grid runs fast now, so two scanning for GRIB code will be OK.

[JohnHalleyGotway]

Howard, I didn't follow all of those details, but it sounds like you have a good idea for how it should work! Let's proceed as follows:
(1) I'll clarify the point of this issue to make sure you understand the needed behavior.
(2) I'll push my local feature branch (feature_1630_point2grid_hang) up to GitHub and assign you to this issue.
(3) You pull my feature branch code, make whatever updates should be made, and push back up to GitHub.
(4) I'll pull your updates from GitHub, recompile, and retest to make sure I understand them and that it handles Logan's situation correctly.

Logan is running point2grid every day using daily storm reports as input to ascii2nc and point2grid. When storms are present, it works fine and as expected. But when there are no storms, ascii2nc errors out. I've already updated ascii2nc to NOT error out with 0 input obs. And now we want point2grid to do the same. With 0 input observations to process, it should just create an output field of all bad data values and counts and mask values of 0.

Ideally, Logan would NOT need to specify "Fscale" in the config file. Instead, point2grid would just process the commands, realize that no Fscale observations are present, print a warning, and produce an "empty" output file.

[JohnHalleyGotway]

Fixed in develop by PR #1660. @LoganDawson-NOAA, please test this functionality again once met-10.0.0_beta4 is released.