Feature 1695 ensemble single file (gen_ens_prod)

[georgemccabe]

Added support for running gen_ens_prod on a single file that contains all ensemble members

Changes include:

• Modified string_value function to substitute values for MET_ENS_MEMBER_ID and return ConcatString instead of ConcatString*
• Created class EnsVarInfo to store all info to read ensemble files/fields
• Refactored logic in gen_ens_prod to read from EnsVarInfo

Note: EnsVarInfo class will be moved into the library code in the next pull request so that it can also be used by ensemble_stat

Expected Differences

• Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]
  

New configuration file options for gen_ens_prod: ens_member_ids and control_id

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
  

Pull Request Testing

• Describe testing already performed for these changes:
  

Added unit tests that demonstrate new functionality

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

Review code changes, run gen_ens_prod unit tests and ensure that they all pass (new data for testing is available on seneca and kiowa)

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [Yes]
  
  New output will be generated from 4 new unit tests. No other differences should result.

• Please complete this pull request review by 1/11/2022.
  

Pull Request Checklist

See the METplus Workflow for details.

• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Linked issues with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[georgemccabe]

Based on discussion in a NOAA telecon, I updated the error message if the valid thresh is not met so that it is more clear what values are being compared when the threshold is not met.

Old error message:

ERROR : process_ensemble() -> 2 of 3 missing fields for "fcst(1,0,*,*)" exceeds the maximum allowable specified by "ens.ens_thresh" (1) in the configuration file.

New error message:

ERROR : process_ensemble() -> 1 of 3 (0.333333) fields found for "fcst(1,0,*,*)" does not meet the threshold specified by "ens.ens_thresh" (1) in the configuration file.

[JohnHalleyGotway]

I approve of these changes (BUT PLEASE SEE NOTE BELOW)

I have one minor recommendation. In order to avoid lots of versions of config file contents floating around... whenever I add a new config option to one of the config file, I also add it (with its default value) to all instances of that config file in the "test/config" directory. This isn't a "rule", just a recommendation. So I'd recommend adding those 2 new config options to this existing file:

https://github.com/dtcenter/MET/blob/feature_1695_ensemble_single_file/test/config/GenEnsProdConfig

I ran a full regression test vs develop for this feature branch in kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1695_gen_ens_prod.

All existing output is the same, but it introduces 4 new output files:

ERROR: folder MET-develop/test_output missing 4 files
    gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_NO_CTRL.nc 
    gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_WITH_CTRL.nc 
    gen_ens_prod/gen_ens_prod_SINGLE_FILE_NC_NO_CTRL.nc 
    gen_ens_prod/gen_ens_prod_SINGLE_FILE_NC_WITH_CTRL.nc 

Checking these new outputs, I see:

1. The output fields look reasonable via ncview.
2. Diffing them with ncdiff produces the expected result:

/usr/local/nco/bin/ncdiff gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_NO_CTRL.nc gen_ens_prod/gen_ens_prod_SINGLE_FILE_GRIB_WITH_CTRL.nc grib_diff.nc

The standard deviation fields have 0 differences, but the rest do. The mean/mean +/- stadev fields have lot of diffs because the control (when specified) is included in the computation of the mean. The min/max/range fields have fewer diffs... only at grid points where the control has the extreme value.
3. I noted the same result in the ncdiff output for the NetCDF NO/WITH_CTRL output files.

Reviewing the code changes, I note that we:

• Add to gen_ens_prod new config options named ens_memeber_ids and control_id.
• Added user's guide documentation for these including examples.
• Changes to the vx_config library to parse them.
• Change to the gen_ens_prod to no longer interpret the number of input files as the number of ensemble members to be processed.
• Change e->string_value() to return a ConcatString instance instead of a pointer to one. That's need to be able to substitute in the ensemble member id string value.

[georgemccabe]

I updated the other test config file for GenEnsProd to include the new default values as suggested in the PR review.