Feature #2870 removing_MISSING_warning

[JohnHalleyGotway]

This PR includes the following changes:

• Define new parse_file_list_type(const StringArray&) utility function to loop through the input files and determine the file type of the first non-missing file.
• Define new log_missing_file(...) utility function to write a warning message about missing files or just DEBUG(3) if the missing file name begins with the MISSING keyword from METplus wrappers.
• Update Ensemble-Stat and Gen-Ens-Prod to call these utility functions.

Note the following two tweaks to the logic:

• Previously if the first input file was missing, Ensemble-Stat and Gen-Ens-Prod would error out with:

ERROR  : grd_file_type() -> file does not exist "MISSING"

Calling parse_file_list_type(...) instead enables the first file in the list to be missing. So that's an improvement.

• Previously these tools actually called open() on the first input file, and that may have applied a little extra validation logic. But that isn't REALLY needed here because we open the files later when we process them. Not opening them here is every so slightly more efficient but it's possible that if a runtime error was going to occur, it may have occurred sooner rather than later in the logic.

Expected Differences

• Do these changes introduce new tools, command line arguments, or configuration file options? [No]
  
  If yes, please describe:
  

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
  
  If yes, please describe:
  

Pull Request Testing

• Describe testing already performed for these changes:
  

1. Ran Gen-Ens-Prod:

cd scripts; export TEST_OUT_DIR=../out;
../bin/gen_ens_prod \
-ens MISSING \
    ../data/sample_fcst/2009123112/*gep*/d01_2009123112_02400.grib \
    MISSING/path/info /a/real/bad/path -config config/GenEnsProdConfig \
-out ${TEST_OUT_DIR}/gen_ens_prod/gen_ens_prod_20100101_120000V_ens.nc \
-v 3 -log gen_ens_prod.log

Observe the following log messages... 2 DEBUG and 1 WARNING:

DEBUG 3: process_command_line() -> cannot open input ensemble file: MISSING
DEBUG 3: process_command_line() -> cannot open input ensemble file: MISSING/path/info
WARNING: 
WARNING: process_command_line() -> cannot open input ensemble file: /a/real/bad/path
WARNING: 

2. Ran Ensemble-Stat:

cd scripts; export TEST_OUT_DIR=../out;
../bin/ensemble_stat 9 \
/a/bad/path \
../data/sample_fcst/2009123112/*gep*/d01_2009123112_02400.grib \
MISSING \
MISSING/path/to/MISSING config/EnsembleStatConfig \
-grid_obs ../data/sample_obs/ST4/ST4.2010010112.24h \
-point_obs /a/real/bad/path \
-point_obs ${TEST_OUT_DIR}/ascii2nc/precip24_2010010112.nc \
-outdir ${TEST_OUT_DIR}/ensemble_stat \
-v 3 -log ensemble_stat.log

Observe the following log messages...

WARNING:
WARNING: process_command_line() -> cannot open input ensemble file: /a/bad/path
WARNING:
DEBUG 3: process_command_line() -> cannot open input ensemble file: MISSING
DEBUG 3: process_command_line() -> cannot open input ensemble file: MISSING/path/to/MISSING
...
WARNING:
WARNING: process_point_obs() -> cannot open observation netCDF file: /a/real/bad/path
WARNING:

I'll note that providing a -grid_obs bad path as the first one in the list (-grid_obs /a/bath/path) does result in a runtime error. So this PR doesn't solve all the issues with bad data inputs... just the ones described in the issue.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  
  The code for this PR is compiled and available for testing as the met_test user in seneca:/d1/projects/MET/MET_pull_requests/met-12.0.0/beta5/MET-feature_2870_remove_MISSING_warning.

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]
  I couldn't decide whether or not documenting the subtle difference between a warning and debug log message for missing files that start with the MISSING keyword is really warranted. So I did NOT add a note about to the user's guide. Do you think I should?

• Do these changes include sufficient testing updates? [Yes]
  I updated the existing unit_gen_ens_prod.xml file. It includes 3 references to a gep3 file that does NOT actually exist. I increased the verbosity level to 3 for all these runs and updated those 3 references to:

1. MISSING
2. MISSING/&DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib
3. &DATA_DIR_MODEL;/grib1/arw-tom-gep3/arw-tom-gep3_2012040912_F024.grib

The first 2 should produce a DEBUG(3) log message and the 3rd should still produce a warning message.

• Will this PR result in changes to the MET test suite? [No]
  
  If yes, describe the new output and/or changes to the existing output:
  

• Will this PR result in changes to existing METplus Use Cases? [No]
  
  If yes, create a new Update Truth METplus issue to describe them.
  I will change the log output from METplus ensemble use cases, but not the actual output.

• Do these changes introduce new SonarQube findings? [No]
  
  If yes, please describe:
  This PR passed the SonarQube quality gate check.

• Please complete this pull request review by [Fri 5/3/24].
  

Pull Request Checklist

See the METplus Workflow for details.

• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the version that will include these changes
  Select: Coordinated METplus-X.Y Support project for bugfix releases or MET-X.Y.Z Development project for official releases
• After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

Note that all of the differences flagged in this testing workflow run are unrelated to these changes. They are remaining diffs from @hsoh-u recent PR that modified the UGRID output. We need to update develop-ref to move past them, but @hsoh-u is making changes to reduce the overall number of new/modified files.

egrep -i "warning|error|file1:" comp_dir.log  | egrep -B 1 ERROR | grep file1
file1: /data/output/met_test_truth/grid_stat_ugrid/grid_stat_UGRID_MPAS_DIAG_000000L_20120409_120000V.stat
file1: /data/output/met_test_truth/grid_stat_ugrid/grid_stat_UGRID_MPAS_DIAG_000000L_20120409_120000V_pairs.nc
file1: /data/output/met_test_truth/grid_stat_ugrid/grid_stat_UGRID_MPAS_OUT_TO_GRID_000000L_20120409_120000V.stat
file1: /data/output/met_test_truth/grid_stat_ugrid/grid_stat_UGRID_MPAS_OUT_TO_GRID_000000L_20120409_120000V_pairs.nc
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V.stat
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V_cnt.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V_ctc.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V_cts.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V_eclv.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V_fho.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_DIAG_TEMP_000000L_20120409_120000V_mpr.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V.stat
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V_cnt.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V_ctc.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V_cts.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V_eclv.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V_fho.txt
file1: /data/output/met_test_truth/point_stat_ugrid/point_stat_UGRID_MPAS_OUT_TEMP_000000L_20120409_120000V_mpr.txt

[georgemccabe]

These changes look good. These changes will play nicely with the METplus wrappers logic that handles missing files, which either sets MISSING or MISSING/some/path instead of a file path. The tests look to demonstrate these cases. The new output looks good. I don't think it is necessary to update the documentation to describe this change. I approve.