Feature 1575 large_diffs (#1741)

* Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files.

* Per #1575, define config strings to be parsed from the config files.

* Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat.

* Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint.

* Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output.

* Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options.

* Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated.

* Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh().

* Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library.

* Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that.

* Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh.

* Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions.

* Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members.

* Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it.

* Per #1575, need to actually set the mpr_thresh!

* Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same.

* Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations.

* Per #1575, add documentation about mpr_column and mpr_thresh.

* Per #1575, mpr_columns can also include CLIMO_CDF.

* Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options.

met/data/config/GridStatConfig_default

@@ -43,6 +43,8 @@ regrid = {
 //
 censor_thresh       = [];
 censor_val          = [];
+mpr_column          = [];
+mpr_thresh          = [];
 cat_thresh          = [];
 cnt_thresh          = [ NA ];
 cnt_logic           = UNION;

met/data/config/PointStatConfig_default

@@ -38,6 +38,8 @@ regrid = {
 //
 censor_thresh  = [];
 censor_val     = [];
+mpr_column     = [];
+mpr_thresh     = [];
 cat_thresh     = [ NA ];
 cnt_thresh     = [ NA ];
 cnt_logic      = UNION;

met/docs/Users_Guide/config_options.rst

@@ -640,12 +640,12 @@ to be verified. This dictionary may include the following entries:
     metadata of any output files, but the user can set the "desc" entry
     accordingly.
 
-    Examples of user-defined conversion functions include:
+    Examples of user-defined data censoring operations include:
 
     .. code-block:: none
 		    
       censor_thresh = [ >12000 ];
-      censor_val    = [ 12000 ];
+      censor_val    = [  12000 ];
 
   * Several configuration options are provided to override and correct the
     metadata read from the input file. The supported options are listed
@@ -678,6 +678,25 @@ to be verified. This dictionary may include the following entries:
       is_wind_direction    = boolean;
       is_prob              = boolean;
 
+  * The "mpr_column" and "mpr_thresh" entries are arrays of strings and
+    thresholds to specify which matched pairs should be included in the
+    statistics. These options apply to the Point-Stat and Grid-Stat tools.
+    They are parsed seperately for each "obs.field" array entry.
+    The "mpr_column" strings specify MPR column names ("FCST", "OBS",
+    "CLIMO_MEAN", "CLIMO_STDEV", or "CLIMO_CDF"), differences of columns
+    ("FCST-OBS"), or the absolute value of those differences ("ABS(FCST-OBS)").
+    The number of "mpr_thresh" thresholds must match the number of "mpr_column"
+    entries, and the n-th threshold is applied to the n-th column. Any matched
+    pairs which do not meet any of the specified thresholds are excluded from
+    the analysis. For example, the following settings exclude matched pairs
+    where the observation value differs from the forecast or climatological
+    mean values by more than 10:
+
+    .. code-block:: none
+
+      mpr_column = [ "ABS(OBS-FCST)", "ABS(OBS-CLIMO_MEAN)" ];
+      mpr_thresh = [ <=10, <=10 ];
+
   * The "cat_thresh" entry is an array of thresholds to be used when
     computing categorical statistics.
 
@@ -981,6 +1000,8 @@ or
    obs = {
      censor_thresh = [];
      censor_val    = [];
+     mpr_column    = [];
+     mpr_thresh    = [];
      cnt_thresh    = [ NA ];
      cnt_logic     = UNION;
      wind_thresh   = [ NA ];

met/docs/Users_Guide/grid-stat.rst

@@ -221,13 +221,15 @@ __________________________
                      type = [ { method = NEAREST; width = 1; } ]; }
   censor_thresh  = [];
   censor_val     = [];
+  mpr_column     = [];
+  mpr_thresh     = [];
   eclv_points    = 0.05;
   rank_corr_flag = TRUE;
   tmp_dir        = "/tmp";
   output_prefix  = "";
   version        = "VN.N";
 
-The configuration options listed above are common to many MET tools and are described in :numref:`config_options`.
+The configuration options listed above are common to multiple MET tools and are described in :numref:`config_options`.
 
 ___________________________
 

met/docs/Users_Guide/point-stat.rst

@@ -334,6 +334,8 @@ ________________________
                      type = [ { method = NEAREST; width = 1; } ]; }
   censor_thresh  = [];
   censor_val     = [];
+  mpr_column     = [];
+  mpr_thresh     = [];
   eclv_points    = 0.05;
   rank_corr_flag = TRUE;
   sid_inc        = [];
@@ -347,7 +349,7 @@ ________________________
   output_prefix  = "";
   version        = "VN.N";
 
-The configuration options listed above are common to many MET tools and are described in :numref:`config_options`.
+The configuration options listed above are common to multiple MET tools and are described in :numref:`config_options`.
 
 _________________________
 

met/scripts/config/GridStatConfig_APCP_12

@@ -42,6 +42,8 @@ regrid = {
 //
 censor_thresh       = [];
 censor_val          = [];
+mpr_column          = [];
+mpr_thresh          = [];
 cat_thresh          = [];
 cnt_thresh          = [ NA ];
 cnt_logic           = UNION;
@@ -52,7 +54,6 @@ nc_pairs_var_name   = "";
 nc_pairs_var_suffix = "";
 rank_corr_flag      = FALSE;
 
-
 //
 // Forecast and observation fields to be verified
 //

met/scripts/config/GridStatConfig_APCP_24

@@ -42,6 +42,8 @@ regrid = {
 //
 censor_thresh       = [];
 censor_val          = [];
+mpr_column          = [];
+mpr_thresh          = [];
 cat_thresh          = [];
 cnt_thresh          = [ NA ];
 cnt_logic           = UNION;
@@ -51,6 +53,7 @@ eclv_points         = 0.05;
 nc_pairs_var_name   = "";
 nc_pairs_var_suffix = "";
 rank_corr_flag      = FALSE;
+
 //
 // Forecast and observation fields to be verified
 //

met/scripts/config/GridStatConfig_POP_12

@@ -42,6 +42,8 @@ regrid = {
 //
 censor_thresh       = [];
 censor_val          = [];
+mpr_column          = [];
+mpr_thresh          = [];
 cat_thresh          = [];
 cnt_thresh          = [ NA ];
 cnt_logic           = UNION;

met/scripts/config/GridStatConfig_all

@@ -42,6 +42,8 @@ regrid = {
 //
 censor_thresh       = [];
 censor_val          = [];
+mpr_column          = [];
+mpr_thresh          = [];
 cat_thresh          = [];
 cnt_thresh          = [ NA ];
 cnt_logic           = UNION;

met/scripts/config/PointStatConfig

@@ -33,6 +33,8 @@ regrid = {
 
 censor_thresh = [];
 censor_val    = [];
+mpr_column    = [];
+mpr_thresh    = [];
 cnt_thresh    = [ NA ];
 cnt_logic     = UNION;
 wind_thresh   = [ NA ];

met/src/basic/vx_config/config_constants.h

@@ -380,7 +380,7 @@ struct MaskLatLon {
 //
 
 enum DuplicateType {
-   DuplicateType_None,   // Apply no logic for duplicate point obs
+   DuplicateType_None,  // Apply no logic for duplicate point obs
    DuplicateType_Unique // Filter out duplicate observation values
 };
 
@@ -394,7 +394,7 @@ enum ObsSummary {
    ObsSummary_None,    // Keep all observations, no statistics
    ObsSummary_Nearest, // Keep only the observation closest in time
    ObsSummary_Min,     // Keep only smallest value
-   ObsSummary_Max,     // Keep only largest valueXS
+   ObsSummary_Max,     // Keep only largest value
    ObsSummary_UW_Mean, // Calculate un-weighted mean
    ObsSummary_DW_Mean, // Calculate time weighted mean
    ObsSummary_Median,  // Calculate median
@@ -536,6 +536,8 @@ static const char conf_key_obs_qty[]           = "obs_quality";
 static const char conf_key_convert[]           = "convert";
 static const char conf_key_censor_thresh[]     = "censor_thresh";
 static const char conf_key_censor_val[]        = "censor_val";
+static const char conf_key_mpr_column[]        = "mpr_column";
+static const char conf_key_mpr_thresh[]        = "mpr_thresh";
 static const char conf_key_cnt_thresh[]        = "cnt_thresh";
 static const char conf_key_cnt_logic[]         = "cnt_logic";
 static const char conf_key_cat_thresh[]        = "cat_thresh";

met/src/basic/vx_config/config_util.cc

@@ -2285,46 +2285,6 @@ void check_mctc_thresh(const ThreshArray &ta) {
 
 ///////////////////////////////////////////////////////////////////////////////
 
-bool check_fo_thresh(const double f,   const double o,
-                     const double cmn, const double csd,
-                     const SingleThresh &ft, const SingleThresh &ot,
-                     const SetLogic type) {
-   bool status = true;
-   bool fcheck = ft.check(f, cmn, csd);
-   bool ocheck = ot.check(o, cmn, csd);
-   SetLogic t  = type;
-
-   // If either of the thresholds is NA, reset the logic to intersection
-   // because an NA threshold is always true.
-   if(ft.get_type() == thresh_na || ot.get_type() == thresh_na) {
-      t = SetLogic_Intersection;
-   }
-
-   switch(t) {
-      case(SetLogic_Union):
-         if(!fcheck && !ocheck) status = false;
-         break;
-
-      case(SetLogic_Intersection):
-         if(!fcheck || !ocheck) status = false;
-         break;
-
-      case(SetLogic_SymDiff):
-         if(fcheck == ocheck) status = false;
-         break;
-
-      default:
-         mlog << Error << "\ncheck_fo_thresh() -> "
-              << "Unexpected SetLogic value of " << type << ".\n\n";
-         exit(1);
-         break;
-   }
-
-   return(status);
-}
-
-///////////////////////////////////////////////////////////////////////////////
-
 const char * statlinetype_to_string(const STATLineType t) {
    const char *s = (const char *) 0;
 

met/src/basic/vx_config/config_util.h

@@ -80,10 +80,6 @@ extern void         check_climo_n_vx(Dictionary *dict, const int);
 extern InterpMthd   int_to_interpmthd(int);
 extern void         check_mctc_thresh(const ThreshArray &);
 
-extern bool         check_fo_thresh(const double, const double, const double, const double,
-                                    const SingleThresh &, const SingleThresh &,
-                                    const SetLogic);
-
 extern const char * statlinetype_to_string(const STATLineType);
 extern void         statlinetype_to_string(const STATLineType, char *);
 extern STATLineType string_to_statlinetype(const char *);

met/src/libcode/vx_statistics/met_stats.cc

@@ -1207,7 +1207,7 @@ void SL1L2Info::set(const PairDataPoint &pd_all) {
    zero_out();
 
    // Apply continuous filtering thresholds to subset pairs
-   pd = subset_pairs(pd_all, fthresh, othresh, logic);
+   pd = pd_all.subset_pairs_cnt_thresh(fthresh, othresh, logic);
 
    // Check for no matched pairs to process
    if(pd.n_obs == 0) return;

met/src/libcode/vx_statistics/pair_base.h

@@ -71,6 +71,8 @@ class PairBase {
       MaskLatLon    *mask_llpnt_ptr; // Pointer to Lat/Lon thresholds
                                      // which is not allocated
 
+      //////////////////////////////////////////////////////////////////
+
       ConcatString msg_typ;          // Name of the verifying message type
       StringArray  msg_typ_vals;     // Message type values to be included
 

met/src/libcode/vx_statistics/pair_data_ensemble.cc

@@ -758,7 +758,7 @@ void PairDataEnsemble::compute_ssvar() {
 //
 ////////////////////////////////////////////////////////////////////////
 
-PairDataEnsemble PairDataEnsemble::subset_pairs(const SingleThresh &ot) const {
+PairDataEnsemble PairDataEnsemble::subset_pairs_obs_thresh(const SingleThresh &ot) const {
 
    // Check for no work to be done
    if(ot.get_type() == thresh_na) return(*this);

met/src/libcode/vx_statistics/pair_data_ensemble.h

@@ -142,7 +142,7 @@ class PairDataEnsemble : public PairBase {
       void compute_phist();
       void compute_ssvar();
 
-      PairDataEnsemble subset_pairs(const SingleThresh &ot) const;
+      PairDataEnsemble subset_pairs_obs_thresh(const SingleThresh &ot) const;
 };
 
 ////////////////////////////////////////////////////////////////////////