Enhance Ensemble-Stat and Gen-Ens-Prod to read all ensemble members from a single input file

[j-opatz]

Describe the New Feature

In our discussions with CPC, we've been presented with NMME netCDF data files where all of the ensemble members are located in the same data structure in 1 file. CPC confirmed this is the normal structure for ensemble files, and has been encountered during work with other entities (e.g. NRL). While Python embedding is currently a viable solution to this file design type, there's a desire for MET's EnsembleStat tool to do this natively.

Acceptance Testing

Sample NMME file can be found on kiowa, under /d1/projects/MET/MET_test_data/CPC_NMME. Variables of interest are ensmem(ensmem) and fcst(ensmem,target,lat,lon)
Tests should successfully read in all ensemble members in datafile into EnsembleStat with available processing options mirroring what was available for individual member files.

Time Estimate

Estimate the amount of work required here.
Unknown until later date, input from Engineers

Sub-Issues

Consider breaking the new feature down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

None

Funding Source

TBD when on a beta schedule

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

Make sure to test solution with NOAA/CPC data.

[JohnHalleyGotway]

Note that we can also test this support using NOAA GEFS data:
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gens/prod/
For example, the files in gefs.20211001/00/atmos/pgrb2sp25/gep[01-30].t00z.pgrb2s.0p25.f000 contain data for 30 members, and their GRIB2 headers identify each of the 30 members. We can literally just concatenate all the member files to put them in a single GRIB2 file.

cat gep* > gepmem.t00z.pgrb2s.0p25.f000

Here's some 2m TMP records for each member:

wgrib2 gepmem.t00z.pgrb2s.0p25.f000 | grep TMP
10:4090980:d=2021100100:TMP:2 m above ground:anl:ENS=+1
33:15317808:d=2021100100:TMP:2 m above ground:anl:ENS=+2
...
654:319669475:d=2021100100:TMP:2 m above ground:anl:ENS=+29
677:331135932:d=2021100100:TMP:2 m above ground:anl:ENS=+30

Make sure that the changes for this issue also support GRIB records stored in this manner.

[JohnHalleyGotway]

@j-opatz consider this option for the logic...

Currently, the number of input ensemble member files defines the size of the ensemble. Ensemble-Stat attempts to read each field array entry listed in the config file from each of the input ensemble member files.

The issue we're trying solve is handling data files that contain multiple members, rather than just one. Since an ensemble of size 1 should probably be considered a degenerate case, we could consider adjusting the logic like this. If only one ensemble file is specified on the command line, do NOT define the ensemble as size 1. Instead, look in the config file and apply special processing logic. Each field array entry can contain multiple "level" settings. For example:

level = [ "(0,0,*,*)", "(0,1,*,*)", "(0,2,*,*)" ];

Use the number of level settings for each field to define the ensemble size. If we do this, we should require the number level entry for each field array to remain the same over all fields.
So the "name" string would have to remain the same, and only the "level" string would change.

Would that logic suffice?

[georgemccabe]

Notes from discussion with @JohnHalleyGotway on 12/27/2021:

• Enhancements should be made for EnsembleStat and GenEnsProd, so put common logic into libraries as possible
• Potential solution could involve adding a config variable called something like "ens_member_ids" which would be a list of strings to substitute into field info for each ensemble.

ens_member_ids = []; (default)
ens_member_ids = ["0", "1", "2", "3"];

Each value in the list could be substituted whenever a keyword such as MET_ENS_MEMBER_ID is found in the ens.field dictionary.

GRIB Example

name="name1"; level="L0"; GRIB_ens="+MET_ENS_MEMBER_ID";

NetCDF Example 1

name="name1"; level="(0,MET_ENS_MEMBER_OPTION,*,*)";

NetCDF Example 2

name="SFC_TMP_MEMMET_ENS_MEMBER_ID";

• If multiple strings may be need to substituted into the field info, the "ens_member_ids" could also be set to a list of dictionaries similar to message_type_group_map. This could be a later enhancement since it may not be necessary.

ens_member_options = [
  {key="ID1"; val="0,1,2,3";},
  {key="ID2"; val="4,5,6,7";},
];

This would set MET_ENS_MEMBER_ID1 and MET_ENS_MEMBER_ID2 (MET_ENS_MEMBER_{key}) with each value in val. Both lists of values should be the same length.

• We will need additional error checking to ensure that the ens_member_options list is only set if the file list is 1 and the list is not set if the file list is >1.
• We will need to figure out how to handle the control member if set. Currently the control file is set with the -ctrl command line argument. The app errors out if this file is found in the ensemble file list, which would be true in this situation. We would need to specify the control member field info (with a new -ctrl_field argument?) to pull out the correct field.

[georgemccabe]

A few notes:

• The NetCDF test example demonstrating the new functionality results in warnings because the timing information cannot be read properly by MET.

WARNING: parse_cf_time_string() -> Unsupported time step in the CF convention time unit "months since 1960-01-01 00:00:00"

"months" is not currently a supported time interval in MET (see

MET/met/src/libcode/vx_nc_util/nc_utils.cc

Lines 3562 to 3580 in e903f8f

	if(tok.has("second") ||
	tok.has("seconds") ||
	tok.has("s")) sec_per_unit = 1;
	else if(tok.has("minute") ||
	tok.has("minutes") ||
	tok.has("min")) sec_per_unit = 60;
	else if(tok.has("hour") ||
	tok.has("hours") ||
	tok.has("hr") ||
	tok.has("h")) sec_per_unit = 3600;
	else if(tok.has("day") ||
	tok.has("days") ||
	tok.has("d")) sec_per_unit = 86400;
	else {
	mlog << Warning << "\n" << method_name
	<< "Unsupported time step in the CF convention time unit \""
	<< str << "\"\n\n";
	return;
	}

). In theory this time unit could be supported in MET because months and years can be computed if a relative time is available.

• When reading multiple files, the tools will skip files that don't exist and still process the available data as long as the ens.vld_thresh threshold is met. However, when reading fields from a single file, the tools will error if a field is not found. If there is a chance that a single file with multiple ensembles data set may sometimes not include all of the same ensembles at a given time, then enhancements will be needed to support skipping these ensembles properly.