Sort files read from directories to provide consistent behavior across platforms.

[JohnHalleyGotway]

Describe the Enhancement

This issue has arisen via METplus at least once but also shows up in cross-platform testing prior to official releases. I several places, the MET code reads/processes files from a directory. However, the order in which those files are read can vary from platform to platform. The order in which files are read can cause small differences in the results.

For example, when running through METplus GitHub actions, the units for "PWAT" in the MTD output files change:

ERROR: FCST_UNITS differs:
A: kg/m^2
B: kg_m^-2

One possible explanation is the order in which the GRIB2 tables are being read:

grep '"PWAT"' grib2_*
grib2_af_gpp.txt:0 14 0 25 57 0 1 3 "PWAT" "Precipitable water" "kg m^-2"
grib2_all.txt:0 1 0 255 0 0 1 3  "PWAT"  "Precipitable Water"  "kg/m^2"

Rather than remaining subject to the order provided by the filesystem, update the library code to sort those files. Recommend running a regression test versus the develop branch to assess the impact of this change.

Time Estimate

1 day

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None needed.

Relevant Deadlines

None

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @JohnHalleyGotway
• Select scientist(s) or no scientist required: none needed

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
  No impacts.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

Regression test run 1 vs develop after adding the logic to sort files from directories in kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1989/run1/test_regression.log

• 13 files flagged with differences:

grep -i "ERROR:|file1:" test_regression.log | egrep -B 1 ERROR: | grep file1
file1: MET-feature_1989_sort/test_output/climatology/grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pairs.nc
file1: MET-feature_1989_sort/test_output/climatology/grid_stat_WMO_CLIMO_1.5DEG_240000L_20120410_000000V_pairs.nc
file1: MET-feature_1989_sort/test_output/grid_diag/grid_diag_out.nc
file1: MET-feature_1989_sort/test_output/grid_stat/grid_stat_GFS_FOURIER_240000L_20120410_000000V_pairs.nc
file1: MET-feature_1989_sort/test_output/grid_stat/grid_stat_GRIB2_NAM_RTMA_120000L_20120409_120000V_pairs.nc
file1: MET-feature_1989_sort/test_output/met_test_scripts/stat_analysis/job_filter_SL1L2.stat
file1: MET-feature_1989_sort/test_output/pcp_combine/subtract_ROT_LL_CEIL_minus_MTERH.nc
file1: MET-feature_1989_sort/test_output/perc_thresh/grid_stat_PERC_THRESH_120000L_20120409_120000V_pairs.nc
file1: MET-feature_1989_sort/test_output/rmw_analysis/rmw_analysis_out.nc
file1: MET-feature_1989_sort/test_output/tc_gen/tc_gen_2016_genmpr.txt
file1: MET-feature_1989_sort/test_output/tc_gen/tc_gen_2016.stat
file1: MET-feature_1989_sort/test_output/tc_rmw/tc_rmw_hwrf_gonzalo.nc
file1: MET-feature_1989_sort/test_output/tc_rmw/tc_rmw_pressure_lev_out.nc

Regression test run 2 vs develop after renaming grib2_af_gpp.txt to grib2_gpp_af.txt in kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1989/run2/test_regression.log

• 3 files flagged with differences:

egrep -i "ERROR:|file1:" test_regression.log | egrep -B 1 ERROR: | grep file1 | wc -l
3

1. MET-feature_1989_sort/test_output/met_test_scripts/stat_analysis/job_filter_SL1L2.stat

• This is fine and is caused by a difference in the order of files read by stat_analysis. Sorting the contents of these files first and re-diffing flags no differences.

2. MET-feature_1989_sort/test_output/tc_gen/tc_gen_2016_genmpr.txt

• The differences are confined to the GENMPR line's INDEX column. This is affected by the order in which data is read. No of the actual results actually change, only the order in which they were identified and processed.

3. MET-feature_1989_sort/test_output/tc_gen/tc_gen_2016.stat

• The differences are all from the GENMPR lines listed in 2. When excluding GENMPR lines from these files, there are no more differences.