diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index b85706f034..a475d51459 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -4,7 +4,9 @@ - [ ] Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
-- [ ] Do these changes include sufficient documentation and testing updates? **[Yes or No]** +- [ ] Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? **[Yes or No]** + +- [ ] Do these changes include sufficient testing updates? **[Yes or No]** - [ ] Will this PR result in changes to the test suite? **[Yes or No]**
If **yes**, describe the new output and/or changes to the existing output:
diff --git a/met/data/config/TCStatConfig_default b/met/data/config/TCStatConfig_default index 3fc882f0f2..d385b01353 100644 --- a/met/data/config/TCStatConfig_default +++ b/met/data/config/TCStatConfig_default @@ -111,6 +111,12 @@ column_thresh_val = []; column_str_name = []; column_str_val = []; +// +// Stratify by excluding strings in non-numeric data columns. +// +column_str_exc_name = []; +column_str_exc_val = []; + // // Similar to the column_thresh options above // @@ -123,6 +129,12 @@ init_thresh_val = []; init_str_name = []; init_str_val = []; +// +// Similar to the column_str_exc options above +// +init_str_exc_name = []; +init_str_exc_val = []; + // // Stratify by the ADECK and BDECK distances to land. // diff --git a/met/docs/Users_Guide/config_options.rst b/met/docs/Users_Guide/config_options.rst index d8acbab286..e1f65cc7b4 100644 --- a/met/docs/Users_Guide/config_options.rst +++ b/met/docs/Users_Guide/config_options.rst @@ -3748,17 +3748,19 @@ Where "job_name" is set to one of the following: Job command FILTERING options that may be used only when -line_type has been listed once. These options take two arguments: the name of the data column to be used and the min, max, or exact value for that column. - If multiple column eq/min/max/str options are listed, the job will be + If multiple column eq/min/max/str/exc options are listed, the job will be performed on their intersection: .. code-block:: none - "-column_min col_name value" e.g. -column_min BASER 0.02 - "-column_max col_name value" - "-column_eq col_name value" - "-column_thresh col_name threshold" e.g. -column_thresh FCST '>273' - "-column_str col_name string" separate multiple filtering strings - with commas + "-column_min col_name value" e.g. -column_min BASER 0.02 + "-column_max col_name value" + "-column_eq col_name value" + "-column_thresh col_name threshold" e.g. -column_thresh FCST '>273' + "-column_str col_name string" separate multiple filtering strings + with commas + "-column_str_exc col_name string" separate multiple filtering strings + with commas Job command options to DEFINE the analysis job. Unless otherwise noted, diff --git a/met/docs/Users_Guide/config_options_tc.rst b/met/docs/Users_Guide/config_options_tc.rst index 155a394ed0..11f7330b4b 100644 --- a/met/docs/Users_Guide/config_options_tc.rst +++ b/met/docs/Users_Guide/config_options_tc.rst @@ -517,8 +517,8 @@ For example: Stratify by performing string matching on non-numeric data columns. Specify a comma-separated list of columns names and values -to be checked. May add using the "-column_str name string" job command -options. +to be included in the analysis. +May add using the "-column_str name string" job command options. For example: @@ -531,6 +531,23 @@ For example: column_str_name = []; column_str_val = []; +**column_str_exc_name, column_str_exc_val** + +Stratify by performing string matching on non-numeric data columns. +Specify a comma-separated list of columns names and values +to be excluded from the analysis. +May add using the "-column_str_exc name string" job command options. + +For example: + +| column_str_exc_name = [ "LEVEL" ]; +| column_str_exc_val = [ "TD" ]; +| + +.. code-block:: none + + column_str_exc_name = []; + column_str_exc_val = []; **init_thresh_name, init_thresh_val** @@ -567,6 +584,23 @@ For example: init_str_name = []; init_str_val = []; +**init_str_exc_name, init_str_exc_val** + +Just like the column_str_exc options above, but apply the string matching only +when lead = 0. If lead = 0 string does match, discard the entire track. +May add using the "-init_str_exc name thresh" job command options. + +For example: + +| init_str_exc_name = [ "LEVEL" ]; +| init_str_exc_val = [ "HU" ]; +| + +.. code-block:: none + + init_str_exc_name = []; + init_str_exc_val = []; + **water_only** Stratify by the ADECK and BDECK distances to land. Once either the ADECK or @@ -747,8 +781,10 @@ Where "job_name" is set to one of the following: "-track_watch_warn name" "-column_thresh name thresh" "-column_str name string" + "-column_str_exc name string" "-init_thresh name thresh" "-init_str name string" + "-init_str_exc name string" Additional filtering options that may be used only when -line_type has been listed only once. These options take two arguments: the name @@ -758,11 +794,13 @@ Where "job_name" is set to one of the following: .. code-block:: none - "-column_min col_name value" For example: -column_min TK_ERR 100.00 - "-column_max col_name value" - "-column_eq col_name value" - "-column_str col_name string" separate multiple filtering strings - with commas + "-column_min col_name value" For example: -column_min TK_ERR 100.00 + "-column_max col_name value" + "-column_eq col_name value" + "-column_str col_name string" separate multiple filtering strings + with commas + "-column_str_exc col_name string" separate multiple filtering strings + with commas Required Args: -dump_row diff --git a/met/docs/Users_Guide/gsi-tools.rst b/met/docs/Users_Guide/gsi-tools.rst index 0e7a6fb92a..019e5c3f7b 100644 --- a/met/docs/Users_Guide/gsi-tools.rst +++ b/met/docs/Users_Guide/gsi-tools.rst @@ -230,7 +230,7 @@ The GSID2MPR tool writes the same set of MPR output columns for the conventional - PRS_MAX_WGT - Pressure of the maximum weighing function -The gsid2mpr output may be passed to the Stat-Analysis tool to derive additional statistics. In particular, users should consider running the **aggregate_stat** job type to read MPR lines and compute partial sums (SL1L2), continuous statistics (CNT), contingency table counts (CTC), or contingency table statistics (CTS). Stat-Analysis has been enhanced to parse any extra columns found at the end of the input lines. Users can filter the values in those extra columns using the **-column_thresh** and **-column_str** job command options. +The gsid2mpr output may be passed to the Stat-Analysis tool to derive additional statistics. In particular, users should consider running the **aggregate_stat** job type to read MPR lines and compute partial sums (SL1L2), continuous statistics (CNT), contingency table counts (CTC), or contingency table statistics (CTS). Stat-Analysis has been enhanced to parse any extra columns found at the end of the input lines. Users can filter the values in those extra columns using the **-column_thresh**, **-column_str**, and **-column_str_exc** job command options. An example of the Stat-Analysis calling sequence is shown below: @@ -425,7 +425,7 @@ The GSID2MPR tool writes the same set of ORANK output columns for the convention - TZFND - d(Tz)/d(Tr) -The gsidens2orank output may be passed to the Stat-Analysis tool to derive additional statistics. In particular, users should consider running the **aggregate_stat** job type to read ORANK lines and ranked histograms (RHIST), probability integral transform histograms (PHIST), and spread-skill variance output (SSVAR). Stat-Analysis has been enhanced to parse any extra columns found at the end of the input lines. Users can filter the values in those extra columns using the **-column_thresh** and **-column_str** job command options. +The gsidens2orank output may be passed to the Stat-Analysis tool to derive additional statistics. In particular, users should consider running the **aggregate_stat** job type to read ORANK lines and ranked histograms (RHIST), probability integral transform histograms (PHIST), and spread-skill variance output (SSVAR). Stat-Analysis has been enhanced to parse any extra columns found at the end of the input lines. Users can filter the values in those extra columns using the **-column_thresh**, **-column_str**, and **-column_str_exc** job command options. An example of the Stat-Analysis calling sequence is shown below: diff --git a/met/docs/Users_Guide/stat-analysis.rst b/met/docs/Users_Guide/stat-analysis.rst index 50655fc573..ce2d8c7654 100644 --- a/met/docs/Users_Guide/stat-analysis.rst +++ b/met/docs/Users_Guide/stat-analysis.rst @@ -522,13 +522,14 @@ This job command option is extremely useful. It can be used multiple times to sp .. code-block:: none - -column_min col_name value - -column_max col_name value - -column_eq col_name value - -column_thresh col_name thresh - -column_str col_name string - -The column filtering options may be used when the **-line_type** has been set to a single value. These options take two arguments, the name of the data column to be used followed by a value, string, or threshold to be applied. If multiple column_min/max/eq/thresh/str options are listed, the job will be performed on their intersection. Each input line is only retained if its value meets the numeric filtering criteria defined or matches one of the strings defined by the **-column_str** option. Multiple filtering strings may be listed using commas. Defining thresholds in MET is described in :numref:`config_options`. + -column_min col_name value + -column_max col_name value + -column_eq col_name value + -column_thresh col_name thresh + -column_str col_name string + -column_str_exc col_name string + +The column filtering options may be used when the **-line_type** has been set to a single value. These options take two arguments, the name of the data column to be used followed by a value, string, or threshold to be applied. If multiple column_min/max/eq/thresh/str options are listed, the job will be performed on their intersection. Each input line is only retained if its value meets the numeric filtering criteria defined, matches one of the strings defined by the **-column_str** option, or does not match any of the string defined by the **-column_str_exc** option. Multiple filtering strings may be listed using commas. Defining thresholds in MET is described in :numref:`config_options`. .. code-block:: none diff --git a/met/docs/Users_Guide/tc-stat.rst b/met/docs/Users_Guide/tc-stat.rst index f1ddfbeb7d..c66a6f6894 100644 --- a/met/docs/Users_Guide/tc-stat.rst +++ b/met/docs/Users_Guide/tc-stat.rst @@ -251,7 +251,16 @@ _________________________ column_str_name = []; column_str_val = []; -The **column_str_name** and **column_str_val** fields stratify by performing string matching on non-numeric data columns. Specify a comma-separated list of columns names and values to be checked. The length of the **column_str_val** should match that of the **column_str_name**. Using the **-column_str name val** option within the job command lines may further refine these selections. +The **column_str_name** and **column_str_val** fields stratify by performing string matching on non-numeric data columns. Specify a comma-separated list of columns names and values to be **included** in the analysis. The length of the **column_str_val** should match that of the **column_str_name**. Using the **-column_str name val** option within the job command lines may further refine these selections. + +_________________________ + +.. code-block:: none + + column_str_exc_name = []; + column_str_exc_val = []; + +The **column_str_exc_name** and **column_str_exc_val** fields stratify by performing string matching on non-numeric data columns. Specify a comma-separated list of columns names and values to be **excluded** from the analysis. The length of the **column_str_exc_val** should match that of the **column_str_exc_name**. Using the **-column_str_exc name val** option within the job command lines may further refine these selections. _________________________ @@ -260,7 +269,7 @@ _________________________ init_thresh_name = []; init_thresh_val = []; -The **init_thresh_name** and **init_thresh_val** fields stratify by applying thresholds to numeric data columns only when lead = 0. If lead =0, but the value does not meet the threshold, discard the entire track. The length of the **init_thresh_val** should match that of the **init_thresh_name**. Using the **-init_thresh name val** option within the job command lines may further refine these selections. +The **init_thresh_name** and **init_thresh_val** fields stratify by applying thresholds to numeric data columns only when lead = 0. If lead = 0, but the value does not meet the threshold, discard the entire track. The length of the **init_thresh_val** should match that of the **init_thresh_name**. Using the **-init_thresh name val** option within the job command lines may further refine these selections. _________________________ @@ -269,7 +278,16 @@ _________________________ init_str_name = []; init_str_val = []; -The **init_str_name** and **init_str_val** fields stratify by performing string matching on non-numeric data columns only when lead = 0. If lead =0, but the string does not match, discard the entire track. The length of the **init_str_val** should match that of the **init_str_name**. Using the **-init_str name val** option within the job command lines may further refine these selections. +The **init_str_name** and **init_str_val** fields stratify by performing string matching on non-numeric data columns only when lead = 0. If lead = 0, but the string **does not** match, discard the entire track. The length of the **init_str_val** should match that of the **init_str_name**. Using the **-init_str name val** option within the job command lines may further refine these selections. + +_________________________ + +.. code-block:: none + + init_str_exc_name = []; + init_str_exc_val = []; + +The **init_str_exc_name** and **init_str_exc_val** fields stratify by performing string matching on non-numeric data columns only when lead = 0. If lead = 0, and the string **does** match, discard the entire track. The length of the **init_str_exc_val** should match that of the **init_str_exc_name**. Using the **-init_str_exc name val** option within the job command lines may further refine these selections. _________________________ diff --git a/met/src/basic/vx_config/config_constants.h b/met/src/basic/vx_config/config_constants.h index 1ae8d5d90d..e63a6935f0 100644 --- a/met/src/basic/vx_config/config_constants.h +++ b/met/src/basic/vx_config/config_constants.h @@ -1037,10 +1037,14 @@ static const char conf_key_column_thresh_name[] = "column_thresh_name"; static const char conf_key_column_thresh_val[] = "column_thresh_val"; static const char conf_key_column_str_name[] = "column_str_name"; static const char conf_key_column_str_val[] = "column_str_val"; +static const char conf_key_column_str_exc_name[] = "column_str_exc_name"; +static const char conf_key_column_str_exc_val[] = "column_str_exc_val"; static const char conf_key_init_thresh_name[] = "init_thresh_name"; static const char conf_key_init_thresh_val[] = "init_thresh_val"; static const char conf_key_init_str_name[] = "init_str_name"; static const char conf_key_init_str_val[] = "init_str_val"; +static const char conf_key_init_str_exc_name[] = "init_str_exc_name"; +static const char conf_key_init_str_exc_val[] = "init_str_exc_val"; static const char conf_key_water_only[] = "water_only"; static const char conf_key_rirw_track[] = "rirw.track"; static const char conf_key_rirw_time_adeck[] = "rirw.adeck.time"; diff --git a/met/src/libcode/vx_analysis_util/stat_job.cc b/met/src/libcode/vx_analysis_util/stat_job.cc index 346ba9e03f..870d5cac1d 100644 --- a/met/src/libcode/vx_analysis_util/stat_job.cc +++ b/met/src/libcode/vx_analysis_util/stat_job.cc @@ -172,7 +172,8 @@ void STATAnalysisJob::clear() { wmo_fisher_stats.clear(); column_thresh_map.clear(); - column_str_map.clear(); + column_str_inc_map.clear(); + column_str_exc_map.clear(); by_column.clear(); @@ -306,7 +307,8 @@ void STATAnalysisJob::assign(const STATAnalysisJob & aj) { wmo_fisher_stats = aj.wmo_fisher_stats; column_thresh_map = aj.column_thresh_map; - column_str_map = aj.column_str_map; + column_str_inc_map = aj.column_str_inc_map; + column_str_exc_map = aj.column_str_exc_map; by_column = aj.by_column; @@ -507,9 +509,16 @@ void STATAnalysisJob::dump(ostream & out, int depth) const { thr_it->second.dump(out, depth + 1); } - out << prefix << "column_str_map ...\n"; - for(map::const_iterator str_it = column_str_map.begin(); - str_it != column_str_map.end(); str_it++) { + out << prefix << "column_str_inc_map ...\n"; + for(map::const_iterator str_it = column_str_inc_map.begin(); + str_it != column_str_inc_map.end(); str_it++) { + out << prefix << str_it->first << ": \n"; + str_it->second.dump(out, depth + 1); + } + + out << prefix << "column_str_exc_map ...\n"; + for(map::const_iterator str_it = column_str_exc_map.begin(); + str_it != column_str_exc_map.end(); str_it++) { out << prefix << str_it->first << ": \n"; str_it->second.dump(out, depth + 1); } @@ -948,8 +957,8 @@ int STATAnalysisJob::is_keeper(const STATLine & L) const { // // column_str // - for(map::const_iterator str_it = column_str_map.begin(); - str_it != column_str_map.end(); str_it++) { + for(map::const_iterator str_it = column_str_inc_map.begin(); + str_it != column_str_inc_map.end(); str_it++) { // // Check if the current value is in the list for the column @@ -957,6 +966,18 @@ int STATAnalysisJob::is_keeper(const STATLine & L) const { if(!str_it->second.has(L.get_item(str_it->first.c_str(), false))) return(0); } + // + // column_str_exc + // + for(map::const_iterator str_it = column_str_exc_map.begin(); + str_it != column_str_exc_map.end(); str_it++) { + + // + // Check if the current value is not in the list for the column + // + if(str_it->second.has(L.get_item(str_it->first.c_str(), false))) return(0); + } + // // For MPR lines, check mask_grid, mask_poly, and mask_sid // @@ -1125,7 +1146,10 @@ void STATAnalysisJob::parse_job_command(const char *jobstring) { column_thresh_map.clear(); } else if(jc_array[i] == "-column_str" ) { - column_str_map.clear(); + column_str_inc_map.clear(); + } + else if(jc_array[i] == "-column_str_exc" ) { + column_str_exc_map.clear(); } else if(jc_array[i] == "-set_hdr" ) { hdr_name.clear(); @@ -1376,12 +1400,30 @@ void STATAnalysisJob::parse_job_command(const char *jobstring) { col_value.add_css(jc_array[i+2]); // If the column name is already present in the map, add to it - if(column_str_map.count(col_name) > 0) { - column_str_map[col_name].add(col_value); + if(column_str_inc_map.count(col_name) > 0) { + column_str_inc_map[col_name].add(col_value); } // Otherwise, add a new map entry else { - column_str_map.insert(pair(col_name, col_value)); + column_str_inc_map.insert(pair(col_name, col_value)); + } + i+=2; + } + else if(jc_array[i] == "-column_str_exc") { + + // Parse the column name and value + col_name = to_upper((string)jc_array[i+1]); + col_value.clear(); + col_value.set_ignore_case(1); + col_value.add_css(jc_array[i+2]); + + // If the column name is already present in the map, add to it + if(column_str_exc_map.count(col_name) > 0) { + column_str_exc_map[col_name].add(col_value); + } + // Otherwise, add a new map entry + else { + column_str_exc_map.insert(pair(col_name, col_value)); } i+=2; } @@ -2461,14 +2503,23 @@ ConcatString STATAnalysisJob::get_jobstring() const { } // column_str - for(map::const_iterator str_it = column_str_map.begin(); - str_it != column_str_map.end(); str_it++) { + for(map::const_iterator str_it = column_str_inc_map.begin(); + str_it != column_str_inc_map.end(); str_it++) { for(i=0; isecond.n(); i++) { js << "-column_str " << str_it->first << " " << str_it->second[i] << " "; } } + // column_str_exc + for(map::const_iterator str_it = column_str_exc_map.begin(); + str_it != column_str_exc_map.end(); str_it++) { + + for(i=0; isecond.n(); i++) { + js << "-column_str_exc " << str_it->first << " " << str_it->second[i] << " "; + } + } + // by_column if(by_column.n() > 0) { for(i=0; i column_thresh_map; // ASCII column string matching - map column_str_map; + map column_str_inc_map; + map column_str_exc_map; StringArray hdr_name; StringArray hdr_value; diff --git a/met/src/tools/other/ascii2nc/little_r_handler.cc b/met/src/tools/other/ascii2nc/little_r_handler.cc index b3d49787ea..a604a96355 100644 --- a/met/src/tools/other/ascii2nc/little_r_handler.cc +++ b/met/src/tools/other/ascii2nc/little_r_handler.cc @@ -66,6 +66,8 @@ static const string lr_grib_names[] = { // Little-R regular expression used to determine file type static const char *lr_rpt_reg_exp = "FM-[0-9]"; +static const char *lr_dtg_reg_exp = "[0-9]\\{14\\}"; + //////////////////////////////////////////////////////////////////////// @@ -137,46 +139,49 @@ bool LittleRHandler::_readObservations(LineDataFile &ascii_file) int n_data_hdr; StringArray mappedTypes; StringArray unmappedTypes; + bool is_bad_header = false; while (ascii_file.read_fwf_line(data_line, lr_rpt_wdth, n_lr_rpt_wdth)) { + // Check for expected header line if (!check_reg_exp(lr_rpt_reg_exp, data_line[4])) { mlog << Error << "\nLittleRHandler::_readObservations() -> " << "the fifth entry of the little_r report on line " - << data_line.line_number() << " does not match \"" + << data_line.line_number() + << " does not match the regular expression \"" << lr_rpt_reg_exp << "\":\n\"" << data_line[4] << "\"\n\n"; return false; } // Store the message type - ConcatString concat_string = (string)data_line[4]; - concat_string.ws_strip(); + ConcatString cs = (string)data_line[4]; + cs.ws_strip(); ConcatString hdr_typ; - if (_messageTypeMap[concat_string] != "") + if (_messageTypeMap.count(cs) > 0) { - hdr_typ = _messageTypeMap[concat_string]; - if (!mappedTypes.has(concat_string)) { + hdr_typ = _messageTypeMap[cs]; + if (!mappedTypes.has(cs)) { mlog << Debug(5) - << "Switching little_r report type \"" << concat_string + << "Switching little_r report type \"" << cs << "\" to message type \"" << hdr_typ << "\".\n"; - mappedTypes.add(concat_string); + mappedTypes.add(cs); } } else { - hdr_typ = concat_string; + hdr_typ = cs; hdr_typ.replace(" ", "_", false); - - if (!unmappedTypes.has(concat_string)) { - mlog << Warning << "\nLittleRHandler::_processObs() -> " + + if (!unmappedTypes.has(cs)) { + mlog << Warning << "\nLittleRHandler::_readObservations() -> " << "Storing message type as \"" << hdr_typ - << "\" for unexpected report type \"" << concat_string << "\".\n\n"; - unmappedTypes.add(concat_string); + << "\" for unexpected report type \"" << cs << "\".\n\n"; + unmappedTypes.add(cs); } } @@ -188,16 +193,29 @@ bool LittleRHandler::_readObservations(LineDataFile &ascii_file) // Store the valid time in YYYYMMDD_HHMMSS format - ConcatString hdr_vld_str; - - concat_string = data_line[17]; - concat_string.ws_strip(); - hdr_vld_str << cs_erase; - hdr_vld_str.format("%.8s_%.6s", - concat_string.text(), concat_string.text()+8); + time_t hdr_vld = 0; - time_t hdr_vld = _getValidTime(hdr_vld_str.text()); + if (check_reg_exp(lr_dtg_reg_exp, data_line[17])) + { + ConcatString hdr_vld_str; + cs = data_line[17]; + cs.ws_strip(); + hdr_vld_str << cs_erase; + hdr_vld_str.format("%.8s_%.6s", cs.text(), cs.text()+8); + hdr_vld = _getValidTime(hdr_vld_str.text()); + is_bad_header = false; + + } else + { + mlog << Warning << "\nLittleRHandler::_readObservations() -> " + << "the 18 entry of the little_r report on line " + << data_line.line_number() + << " does not match the timestring regular expression \"" + << lr_dtg_reg_exp << "\":\n\"" << data_line[17] << "\"\n\n"; + is_bad_header = true; + } + // Store the station location double hdr_lat = atof(data_line[0]); @@ -211,7 +229,8 @@ bool LittleRHandler::_readObservations(LineDataFile &ascii_file) // Observation of sea level pressure in pascals. - if (!is_eq(atof(data_line[18]), lr_missing_value)) + if (!is_eq(atof(data_line[18]), lr_missing_value) && + !is_bad_header) { ConcatString obs_qty = (is_eq(atof(data_line[19]), lr_missing_value) ? na_string : (string)data_line[19]); @@ -237,12 +256,16 @@ bool LittleRHandler::_readObservations(LineDataFile &ascii_file) int i_data = 0; while (ascii_file.read_fwf_line(data_line, lr_meas_wdth, n_lr_meas_wdth)) { + // Check for the end of report if (is_eq(atof(data_line[0]), lr_end_value) && - is_eq(atof(data_line[2]), lr_end_value)) + is_eq(atof(data_line[2]), lr_end_value)) break; + // Skip data lines if the header line is bad + if (is_bad_header) continue; + // Retrieve pressure and height double obs_prs = (is_eq(atof(data_line[0]), lr_missing_value) ? @@ -305,7 +328,7 @@ bool LittleRHandler::_readObservations(LineDataFile &ascii_file) if (n_data_hdr != i_data) { - mlog << Warning << "\nprocess_little_r_obs() -> " + mlog << Warning << "\nLittleRHandler::_readObservations() -> " << "the number of data lines specified in the header (" << n_data_hdr << ") does not match the number found in the data (" diff --git a/met/src/tools/tc_utils/tc_stat/tc_stat_conf_info.cc b/met/src/tools/tc_utils/tc_stat/tc_stat_conf_info.cc index 3b21161363..1bdc3af262 100644 --- a/met/src/tools/tc_utils/tc_stat/tc_stat_conf_info.cc +++ b/met/src/tools/tc_utils/tc_stat/tc_stat_conf_info.cc @@ -22,6 +22,16 @@ using namespace std; #include "vx_log.h" +//////////////////////////////////////////////////////////////////////// + +// Functions for parsing config entries +static void parse_conf_thresh_map(MetConfig &, + const char *, const char *, + map &); +static void parse_conf_string_map(MetConfig &, + const char *, const char *, + map &); + //////////////////////////////////////////////////////////////////////// // // Code for class TCStatConfInfo @@ -63,7 +73,7 @@ void TCStatConfInfo::clear() { //////////////////////////////////////////////////////////////////////// void TCStatConfInfo::read_config(const char *default_file_name, - const char *user_file_name) { + const char *user_file_name) { // Read the config file constants Conf.read(replace_path(config_const_filename).c_str()); @@ -84,8 +94,7 @@ void TCStatConfInfo::read_config(const char *default_file_name, void TCStatConfInfo::process_config() { int i; - StringArray sa, sa_val, sa_new; - ThreshArray ta_val, ta_new; + StringArray sa; ConcatString poly_file; // Conf: Version @@ -119,12 +128,12 @@ void TCStatConfInfo::process_config() { // Conf: TCStatJob::InitInc sa = Conf.lookup_string_array(conf_key_init_inc); - for(i=0; i " - << "the \"column_thresh_name\" and \"column_thresh_val\" " - << "entries must have the same length.\n\n"; - exit(1); - } + parse_conf_thresh_map(Conf, + conf_key_column_thresh_name, conf_key_column_thresh_val, + Filter.ColumnThreshMap); - // Add entries to the map - for(i=0; i 0) { - Filter.ColumnThreshMap[sa[i]].add(ta_val[i]); - } - else { - ta_new.clear(); - ta_new.add(ta_val[i]); - Filter.ColumnThreshMap.insert(pair(sa[i], ta_new)); - } - } // end for i - - // Conf: TCStatJob::ColumnStrName, TCStatJob::ColumnStrVal - sa = Conf.lookup_string_array(conf_key_column_str_name); - sa_val = Conf.lookup_string_array(conf_key_column_str_val); - - // Check that they are the same length - if(sa.n_elements() != sa_val.n_elements()) { - mlog << Error - << "\nTCStatConfInfo::process_config() -> " - << "the \"column_str_name\" and \"column_str_val\" " - << "entries must have the same length.\n\n"; - exit(1); - } + // Conf: TCStatJob::ColumnStrIncName, TCStatJob::ColumnStrIncVal + parse_conf_string_map(Conf, + conf_key_column_str_name, conf_key_column_str_val, + Filter.ColumnStrIncMap); - // Add entries to the map - for(i=0; i 0) { - Filter.ColumnStrMap[sa[i]].add(sa_val[i]); - } - else { - sa_new.clear(); - sa_new.set_ignore_case(1); - sa_new.add(sa_val[i]); - Filter.ColumnStrMap.insert(pair(sa[i], sa_new)); - } - } // end for i + // Conf: TCStatJob::ColumnStrExcName, TCStatJob::ColumnStrExcVal + parse_conf_string_map(Conf, + conf_key_column_str_exc_name, conf_key_column_str_exc_val, + Filter.ColumnStrExcMap); // Conf: TCStatJob::InitThreshName, TCStatJob::InitThreshVal - sa = Conf.lookup_string_array(conf_key_init_thresh_name); - ta_val = Conf.lookup_thresh_array(conf_key_init_thresh_val); + parse_conf_thresh_map(Conf, + conf_key_init_thresh_name, conf_key_init_thresh_val, + Filter.InitThreshMap); - // Check that they are the same length - if(sa.n_elements() != ta_val.n_elements()) { - mlog << Error - << "\nTCStatConfInfo::process_config() -> " - << "the \"init_thresh_name\" and \"init_thresh_val\" " - << "entries must have the same length.\n\n"; - exit(1); - } - - // Add entries to the map - for(i=0; i 0) { - Filter.InitThreshMap[sa[i]].add(ta_val[i]); - } - else { - ta_new.clear(); - ta_new.add(ta_val[i]); - Filter.InitThreshMap.insert(pair(sa[i], ta_new)); - } - } // end for i - - // Conf: TCStatJob::InitStrName, TCStatJob::InitStrVal - sa = Conf.lookup_string_array(conf_key_init_str_name); - sa_val = Conf.lookup_string_array(conf_key_init_str_val); - - // Check that they are the same length - if(sa.n_elements() != sa_val.n_elements()) { - mlog << Error - << "\nTCStatConfInfo::process_config() -> " - << "the \"init_str_name\" and \"init_str_val\" " - << "entries must have the same length.\n\n"; - exit(1); - } + // Conf: TCStatJob::InitStrIncName, TCStatJob::InitStrIncVal + parse_conf_string_map(Conf, + conf_key_init_str_name, conf_key_init_str_val, + Filter.InitStrIncMap); - // Add entries to the map - for(i=0; i 0) { - Filter.InitStrMap[sa[i]].add(sa_val[i]); - } - else { - sa_new.clear(); - sa_new.set_ignore_case(1); - sa_new.add(sa_val[i]); - Filter.InitStrMap.insert(pair(sa[i], sa_new)); - } - } // end for i + // Conf: TCStatJob::InitStrExcName, TCStatJob::InitStrExcVal + parse_conf_string_map(Conf, + conf_key_init_str_exc_name, conf_key_init_str_exc_val, + Filter.InitStrExcMap); // Conf: TCStatJob::WaterOnly Filter.WaterOnly = Conf.lookup_bool(conf_key_water_only); @@ -311,7 +248,7 @@ void TCStatConfInfo::process_config() { // Conf: Jobs Jobs = Conf.lookup_string_array(conf_key_jobs); - if(Jobs.n_elements() == 0) { + if(Jobs.n() == 0) { mlog << Error << "\nTCStatConfInfo::process_config() -> " << "must specify at least one entry in \"jobs\".\n\n"; @@ -322,3 +259,73 @@ void TCStatConfInfo::process_config() { } //////////////////////////////////////////////////////////////////////// + +void parse_conf_thresh_map(MetConfig &conf, + const char *conf_key_name, const char *conf_key_val, + map &m) { + StringArray sa; + ThreshArray ta_val, ta_new; + + sa = conf.lookup_string_array(conf_key_name); + ta_val = conf.lookup_thresh_array(conf_key_val); + + // Check that they are the same length + if(sa.n() != ta_val.n()) { + mlog << Error + << "\nTCStatConfInfo::parse_conf_thresh_map() -> " + << "the \"" << conf_key_name << "\" and \"" << conf_key_val << "\" " + << "entries must have the same length.\n\n"; + exit(1); + } + + // Add entries to the map + for(int i=0; i 0) { + m[sa[i]].add(ta_val[i]); + } + else { + ta_new.clear(); + ta_new.add(ta_val[i]); + m.insert(pair(sa[i], ta_new)); + } + } // end for i + + return; +} + +//////////////////////////////////////////////////////////////////////// + +void parse_conf_string_map(MetConfig &conf, + const char *conf_key_name, const char *conf_key_val, + map &m) { + StringArray sa, sa_val, sa_new; + + sa = conf.lookup_string_array(conf_key_name); + sa_val = conf.lookup_string_array(conf_key_val); + + // Check that they are the same length + if(sa.n() != sa_val.n()) { + mlog << Error + << "\nTCStatConfInfo::parse_conf_string_map() -> " + << "the \"" << conf_key_name << "\" and \"" << conf_key_val << "\" " + << "entries must have the same length.\n\n"; + exit(1); + } + + // Add entries to the map + for(int i=0; i 0) { + m[sa[i]].add(sa_val[i]); + } + else { + sa_new.clear(); + sa_new.set_ignore_case(1); + sa_new.add(sa_val[i]); + m.insert(pair(sa[i], sa_new)); + } + } // end for i + + return; +} + +//////////////////////////////////////////////////////////////////////// diff --git a/met/src/tools/tc_utils/tc_stat/tc_stat_job.cc b/met/src/tools/tc_utils/tc_stat/tc_stat_job.cc index e472af405b..6f6f812878 100644 --- a/met/src/tools/tc_utils/tc_stat/tc_stat_job.cc +++ b/met/src/tools/tc_utils/tc_stat/tc_stat_job.cc @@ -112,11 +112,11 @@ TCStatJob *TCStatJobFactory::new_tc_stat_job(const char *jobstring) { a = job->parse_job_command(jobstring); // Check for unused arguments - if(a.n_elements() > 0) { + if(a.n() > 0) { // Build list of unknown args - for(i=0; i " @@ -220,9 +220,11 @@ void TCStatJob::clear() { LineType.clear(); TrackWatchWarn.clear(); ColumnThreshMap.clear(); - ColumnStrMap.clear(); + ColumnStrIncMap.clear(); + ColumnStrExcMap.clear(); InitThreshMap.clear(); - InitStrMap.clear(); + InitStrIncMap.clear(); + InitStrExcMap.clear(); EventEqualLead.clear(); EventEqualCases.clear(); @@ -301,9 +303,11 @@ void TCStatJob::assign(const TCStatJob & j) { LineType = j.LineType; TrackWatchWarn = j.TrackWatchWarn; ColumnThreshMap = j.ColumnThreshMap; - ColumnStrMap = j.ColumnStrMap; + ColumnStrIncMap = j.ColumnStrIncMap; + ColumnStrExcMap = j.ColumnStrExcMap; InitThreshMap = j.InitThreshMap; - InitStrMap = j.InitStrMap; + InitStrIncMap = j.InitStrIncMap; + InitStrExcMap = j.InitStrExcMap; DumpFile = j.DumpFile; open_dump_file(); @@ -423,8 +427,14 @@ void TCStatJob::dump(ostream & out, int depth) const { thr_it->second.dump(out, depth + 1); } - out << prefix << "ColumnStrMap ...\n"; - for(str_it=ColumnStrMap.begin(); str_it!= ColumnStrMap.end(); str_it++) { + out << prefix << "ColumnStrIncMap ...\n"; + for(str_it=ColumnStrIncMap.begin(); str_it!= ColumnStrIncMap.end(); str_it++) { + out << prefix << str_it->first << ": \n"; + str_it->second.dump(out, depth + 1); + } + + out << prefix << "ColumnStrExcMap ...\n"; + for(str_it=ColumnStrExcMap.begin(); str_it!= ColumnStrExcMap.end(); str_it++) { out << prefix << str_it->first << ": \n"; str_it->second.dump(out, depth + 1); } @@ -435,8 +445,14 @@ void TCStatJob::dump(ostream & out, int depth) const { thr_it->second.dump(out, depth + 1); } - out << prefix << "InitStrMap ...\n"; - for(str_it=InitStrMap.begin(); str_it!= InitStrMap.end(); str_it++) { + out << prefix << "InitStrIncMap ...\n"; + for(str_it=InitStrIncMap.begin(); str_it!= InitStrIncMap.end(); str_it++) { + out << prefix << str_it->first << ": \n"; + str_it->second.dump(out, depth + 1); + } + + out << prefix << "InitStrExcMap ...\n"; + for(str_it=InitStrExcMap.begin(); str_it!= InitStrExcMap.end(); str_it++) { out << prefix << str_it->first << ": \n"; str_it->second.dump(out, depth + 1); } @@ -501,7 +517,7 @@ bool TCStatJob::is_keeper_track(const TrackPairInfo &pair, map::const_iterator str_it; // Check TrackWatchWarn for each TrackPoint - if(TrackWatchWarn.n_elements() > 0) { + if(TrackWatchWarn.n() > 0) { // Assume track will not be kept keep = false; @@ -539,7 +555,11 @@ bool TCStatJob::is_keeper_track(const TrackPairInfo &pair, keep = false; n.RejInitThresh += pair.n_points(); } - else if(InitStrMap.size() > 0) { + else if(InitStrIncMap.size() > 0) { + keep = false; + n.RejInitStr += pair.n_points(); + } + else if(InitStrExcMap.size() > 0) { keep = false; n.RejInitStr += pair.n_points(); } @@ -567,10 +587,10 @@ bool TCStatJob::is_keeper_track(const TrackPairInfo &pair, } } - // Check InitStr + // Check InitStrInc if(keep == true) { - for(str_it=InitStrMap.begin(); str_it!= InitStrMap.end(); str_it++) { + for(str_it=InitStrIncMap.begin(); str_it!= InitStrIncMap.end(); str_it++) { // Retrieve the column value v_str = pair.line(i_init)->get_item(str_it->first.c_str()); @@ -584,6 +604,23 @@ bool TCStatJob::is_keeper_track(const TrackPairInfo &pair, } } + // Check InitStrExc + if(keep == true) { + + for(str_it=InitStrExcMap.begin(); str_it!= InitStrExcMap.end(); str_it++) { + + // Retrieve the column value + v_str = pair.line(i_init)->get_item(str_it->first.c_str()); + + // Check the string value + if(str_it->second.has(v_str)) { + keep = false; + n.RejInitStr += pair.n_points(); + break; + } + } + } + // Check OutInitMask if(keep == true) { @@ -606,11 +643,11 @@ bool TCStatJob::is_keeper_track(const TrackPairInfo &pair, // MET-667 Check this track for required lead times // If no required lead times were defined, do nothing. - if(keep == true && LeadReq.n_elements() > 0){ + if(keep == true && LeadReq.n() > 0){ // Loop through the points and see if any of the // lead times are in the list of required lead times // defined in the configuration file. - for(int j=0; j::const_iterator str_it; // Check TC-STAT header columns - if(AModel.n_elements() > 0 && + if(AModel.n() > 0 && !AModel.has(line.amodel())) { keep = false; n.RejAModel++; } - else if(BModel.n_elements() > 0 && + else if(BModel.n() > 0 && !BModel.has(line.bmodel())) { keep = false; n.RejBModel++; } - else if(Desc.n_elements() > 0 && + else if(Desc.n() > 0 && !Desc.has(line.desc())) { keep = false; n.RejDesc++; } - else if(StormId.n_elements() > 0 && + else if(StormId.n() > 0 && !has_storm_id(StormId, (string)line.basin(), (string)line.cyclone(), line.init())) { keep = false; n.RejStormId++; } - else if(Basin.n_elements() > 0 && + else if(Basin.n() > 0 && !Basin.has(line.basin())) { keep = false; n.RejBasin++; } - else if(Cyclone.n_elements() > 0 && + else if(Cyclone.n() > 0 && !Cyclone.has(line.cyclone())) { keep = false; n.RejCyclone++; } - else if(StormName.n_elements() > 0 && + else if(StormName.n() > 0 && !StormName.has(line.storm_name())) { keep = false; n.RejStormName++; } else if(InitBeg > 0 && line.init() < InitBeg) { keep = false; n.RejInit++; } else if(InitEnd > 0 && line.init() > InitEnd) { keep = false; n.RejInit++; } - else if(InitInc.n_elements() > 0 && + else if(InitInc.n() > 0 && !InitInc.has(line.init())) { keep = false; n.RejInit++; } - else if(InitExc.n_elements() > 0 && + else if(InitExc.n() > 0 && InitExc.has(line.init())) { keep = false; n.RejInit++; } - else if(InitHour.n_elements() > 0 && + else if(InitHour.n() > 0 && !InitHour.has(line.init_hour())) { keep = false; n.RejInitHour++; } - else if(Lead.n_elements() > 0 && + else if(Lead.n() > 0 && !Lead.has(line.lead())) { keep = false; n.RejLead++; } else if(ValidBeg > 0 && line.valid() < ValidBeg) { keep = false; n.RejValid++; } else if(ValidEnd > 0 && line.valid() > ValidEnd) { keep = false; n.RejValid++; } - else if(ValidInc.n_elements() > 0 && + else if(ValidInc.n() > 0 && !ValidInc.has(line.valid())) { keep = false; n.RejValid++; } - else if(ValidExc.n_elements() > 0 && + else if(ValidExc.n() > 0 && ValidExc.has(line.valid())) { keep = false; n.RejValid++; } - else if(ValidHour.n_elements() > 0 && + else if(ValidHour.n() > 0 && !ValidHour.has(line.valid_hour())) { keep = false; n.RejValidHour++; } - else if(InitMask.n_elements() > 0 && + else if(InitMask.n() > 0 && !InitMask.has(line.init_mask())) { keep = false; n.RejInitMask++; } - else if(ValidMask.n_elements() > 0 && + else if(ValidMask.n() > 0 && !ValidMask.has(line.valid_mask())) { keep = false; n.RejValidMask++; } - else if(LineType.n_elements() > 0 && + else if(LineType.n() > 0 && !LineType.has(line.line_type())) { keep = false; n.RejLineType++; } // Check that PROBRIRW lines include the requested probability type @@ -701,27 +738,45 @@ bool TCStatJob::is_keeper_line(const TCStatLine &line, // Check the column threshold if(!thr_it->second.check_dbl(v_dbl)) { - keep = false; - n.RejColumnThresh++; - break; + keep = false; + n.RejColumnThresh++; + break; } } } - // Check ColumnStrMap + // Check ColumnStrIncMap if(keep == true) { // Loop through the column string matching - for(str_it=ColumnStrMap.begin(); str_it!= ColumnStrMap.end(); str_it++) { + for(str_it=ColumnStrIncMap.begin(); str_it!= ColumnStrIncMap.end(); str_it++) { // Retrieve the column value v_str = line.get_item(str_it->first.c_str()); // Check the string value if(!str_it->second.has(v_str)) { - keep = false; - n.RejColumnStr++; - break; + keep = false; + n.RejColumnStr++; + break; + } + } + } + + // Check ColumnStrExcMap + if(keep == true) { + + // Loop through the column string matching + for(str_it=ColumnStrExcMap.begin(); str_it!= ColumnStrExcMap.end(); str_it++) { + + // Retrieve the column value + v_str = line.get_item(str_it->first.c_str()); + + // Check the string value + if(str_it->second.has(v_str)) { + keep = false; + n.RejColumnStr++; + break; } } } @@ -805,10 +860,10 @@ double TCStatJob::get_column_double(const TCStatLine &line, v = atof(line.get_item(sa[0].c_str())); // If multiple columns, compute the requested difference - if(sa.n_elements() > 1) { + if(sa.n() > 1) { // Loop through the column - for(i=1; i 0) s << "-init_beg " << unix_to_yyyymmdd_hhmmss(InitBeg) << " "; if(InitEnd > 0) s << "-init_end " << unix_to_yyyymmdd_hhmmss(InitEnd) << " "; - for(i=0; i 0) s << "-valid_beg " << unix_to_yyyymmdd_hhmmss(ValidBeg) << " "; if(ValidEnd > 0) s << "-valid_end " << unix_to_yyyymmdd_hhmmss(ValidEnd) << " "; - for(i=0; isecond.n_elements(); i++) { + for(i=0; isecond.n(); i++) { s << "-column_thresh " << thr_it->first << " " << thr_it->second[i].get_str() << " "; } } - for(str_it=ColumnStrMap.begin(); str_it!= ColumnStrMap.end(); str_it++) { - for(i=0; isecond.n_elements(); i++) { + for(str_it=ColumnStrIncMap.begin(); str_it!= ColumnStrIncMap.end(); str_it++) { + for(i=0; isecond.n(); i++) { s << "-column_str " << str_it->first << " " << str_it->second[i] << " "; } } + for(str_it=ColumnStrExcMap.begin(); str_it!= ColumnStrExcMap.end(); str_it++) { + for(i=0; isecond.n(); i++) { + s << "-column_str_exc " << str_it->first << " " + << str_it->second[i] << " "; + } + } for(thr_it=InitThreshMap.begin(); thr_it!= InitThreshMap.end(); thr_it++) { - for(i=0; isecond.n_elements(); i++) { + for(i=0; isecond.n(); i++) { s << "-init_thresh " << thr_it->first << " " << thr_it->second[i].get_str() << " "; } } - for(str_it=InitStrMap.begin(); str_it!= InitStrMap.end(); str_it++) { - for(i=0; isecond.n_elements(); i++) { + for(str_it=InitStrIncMap.begin(); str_it!= InitStrIncMap.end(); str_it++) { + for(i=0; isecond.n(); i++) { s << "-init_str " << str_it->first << " " << str_it->second[i] << " "; } } + for(str_it=InitStrExcMap.begin(); str_it!= InitStrExcMap.end(); str_it++) { + for(i=0; isecond.n(); i++) { + s << "-init_str_exc " << str_it->first << " " + << str_it->second[i] << " "; + } + } if(WaterOnly != default_water_only) s << "-water_only " << bool_to_string(WaterOnly) << " "; if(RIRWTrack != default_rirw_track) { @@ -1223,7 +1294,7 @@ ConcatString TCStatJob::serialize() const { s << "-match_points " << bool_to_string(MatchPoints) << " "; if(EventEqual != default_event_equal) s << "-event_equal " << bool_to_string(EventEqual) << " "; - for(i=0; i " << "the track-based " << TCStatLineType_TCMPR_Str @@ -1611,7 +1682,7 @@ void TCStatJobFilter::filter_tracks(TCLineCounts &n) { if(EventEqual == true) event_equalize_tracks(); // Check for no common cases - if(EventEqualSet == true && EventEqualCases.n_elements() == 0) { + if(EventEqualSet == true && EventEqualCases.n() == 0) { mlog << Debug(1) << "Event equalization of tracks found no common cases.\n"; } @@ -1650,7 +1721,7 @@ void TCStatJobFilter::filter_lines(TCLineCounts &n) { if(EventEqual == true) event_equalize_lines(); // Check for no common cases - if(EventEqualSet == true && EventEqualCases.n_elements() == 0) { + if(EventEqualSet == true && EventEqualCases.n() == 0) { mlog << Debug(1) << "Event equalization of lines found no common cases.\n"; } @@ -1793,7 +1864,7 @@ StringArray TCStatJobSummary::parse_job_command(const char *jobstring) { a = TCStatJob::parse_job_command(jobstring); // Loop over the StringArray elements - for(i=0; i " << "this function may only be called when using the " << "-column option in the job command line:\n" @@ -1904,7 +1975,7 @@ void TCStatJobSummary::do_job(const StringArray &file_list, // // If not specified, assume TCMPR by adding it to the LineType - if(LineType.n_elements() == 0) LineType.add(TCStatLineType_TCMPR_Str); + if(LineType.n() == 0) LineType.add(TCStatLineType_TCMPR_Str); // Add the input file list TCSTFiles.add_files(file_list); @@ -1913,7 +1984,7 @@ void TCStatJobSummary::do_job(const StringArray &file_list, if(LineType.has(TCStatLineType_TCMPR_Str)) { // TCMPR and non-TCMPR LineTypes cannot be mixed - for(i=0; i " << "the track-based " << TCStatLineType_TCMPR_Str @@ -1950,7 +2021,7 @@ void TCStatJobSummary::summarize_tracks(TCLineCounts &n) { if(EventEqual == true) event_equalize_tracks(); // Check for no common cases - if(EventEqualSet == true && EventEqualCases.n_elements() == 0) { + if(EventEqualSet == true && EventEqualCases.n() == 0) { mlog << Debug(1) << "Event equalization of tracks found no common cases.\n"; } @@ -1992,7 +2063,7 @@ void TCStatJobSummary::summarize_lines(TCLineCounts &n) { if(EventEqual == true) event_equalize_lines(); // Check for no common cases - if(EventEqualSet == true && EventEqualCases.n_elements() == 0) { + if(EventEqualSet == true && EventEqualCases.n() == 0) { mlog << Debug(1) << "Event equalization of lines found no common cases.\n"; } @@ -2039,7 +2110,7 @@ void TCStatJobSummary::process_pair(TrackPairInfo &pair) { for(i=0; i&m) { mlog << Debug(5) << "Summary Map Insert (" << it->first << ") " - << it->second.Val.n_elements() << " values: " + << it->second.Val.n() << " values: " << it->second.Val.serialize() << "\n"; // Add the pair to the map @@ -2132,7 +2203,7 @@ void TCStatJobSummary::add_map(map&m) { mlog << Debug(5) << "Summary Map Add (" << it->first << ") " - << it->second.Val.n_elements() << " values: " + << it->second.Val.n() << " values: " << it->second.Val.serialize() << "\n"; // Add the value for the existing key @@ -2165,11 +2236,11 @@ void TCStatJobSummary::do_output(ostream &out) { // Setup the output table out_at.set_size((int) SummaryMap.size() + 1, - ByColumn.n_elements() + 24); + ByColumn.n() + 24); // Left-justify case info and right-justify summary output for(i=0; isecond.Val.n_elements(); i++) { + for(i=0; isecond.Val.n(); i++) { if(!is_bad_data(it->second.Val[i])) { v.add(it->second.Val[i]); init.add(it->second.Init[i]); @@ -2244,7 +2315,7 @@ void TCStatJobSummary::do_output(ostream &out) { // Build index array index.clear(); - for(i=0; isecond.Val.n_elements()); - out_at.set_entry(r, c++, v.n_elements()); + out_at.set_entry(r, c++, it->second.Val.n()); + out_at.set_entry(r, c++, v.n()); out_at.set_entry(r, c++, mean_ci.v); out_at.set_entry(r, c++, mean_ci.v_ncl[0]); out_at.set_entry(r, c++, mean_ci.v_ncu[0]); @@ -2352,11 +2423,11 @@ void TCStatJobSummary::compute_fsp(NumArray &total, NumArray &best, mlog << Debug(4) << "Computing frequency of superior performance for " - << Column.n_elements() << " columns and " - << case_list.n_elements() << " cases.\n"; + << Column.n() << " columns and " + << case_list.n() << " cases.\n"; // Loop over the columns being summarized - for(i=0; isecond.Hdr.n_elements(); k++) { + for(k=0; ksecond.Hdr.n(); k++) { // Check if entry matches the current case if(strncasecmp(Column[i].c_str(), it->first.c_str(), @@ -2498,9 +2569,9 @@ bool is_time_series(const TimeArray &init, const NumArray &lead, dsec = bad_data_int; // The arrays should all be of the same length > 1 - if(init.n_elements() != lead.n_elements() || - init.n_elements() != valid.n_elements() || - init.n_elements() < 2) { + if(init.n() != lead.n() || + init.n() != valid.n() || + init.n() < 2) { mlog << Debug(4) << "Skipping time-series computations since the array " << "lengths differ.\n"; @@ -2513,7 +2584,7 @@ bool is_time_series(const TimeArray &init, const NumArray &lead, dvalid = valid[1] - valid[0]; // Loop over the entries to determine the time spacing - for(i=0; i= mean); @@ -2609,8 +2680,8 @@ int compute_time_to_indep(const NumArray &val, int ds) { exp_runs = 1.0 + 2.0*(n_abv * n_bel)/(n_abv + n_bel); // Calculate effective sample size, time to independence - eff_size = val.n_elements()*(n_run_abv + n_run_bel)/exp_runs; - tind = ds*val.n_elements()/eff_size; + eff_size = val.n()*(n_run_abv + n_run_bel)/exp_runs; + tind = ds*val.n()/eff_size; return(nint(tind)); } @@ -2622,7 +2693,7 @@ StringArray intersection(const StringArray &s1, const StringArray &s2) { int i; // Add elements common to both list - for(i=0; isecond.Hdr.n_elements(); + r += it->second.Hdr.n(); } // Format the output table out_at.set_size(r + 1, - 9 + ByColumn.n_elements() + 15); - setup_table(out_at, 9 + ByColumn.n_elements(), get_precision()); + 9 + ByColumn.n() + 15); + setup_table(out_at, 9 + ByColumn.n(), get_precision()); // Initialize row and column indices r = c = 0; @@ -3339,7 +3410,7 @@ void TCStatJobRIRW::do_mpr_output(ostream &out) { out_at.set_entry(r, c++, "WINDOW_END"); // Write case column names - for(i=0; isecond.Hdr.n_elements(); i++,r++) { + for(i=0; isecond.Hdr.n(); i++,r++) { // Initialize column counter c = 0; @@ -3384,14 +3455,14 @@ void TCStatJobRIRW::do_mpr_output(ostream &out) { // Write case column values sa = it->first.split(":"); - for(j=1; jsecond.Hdr[i]; sa = cs.split(":"); - for(j=0; j 0) { if(!mask_poly.latlon_is_inside_dege(lat, lon)) { @@ -3967,7 +4038,7 @@ bool check_masks(const MaskPoly &mask_poly, const Grid &mask_grid, } // - // Check grid masking. + // Check grid masking // if(mask_grid.nx() > 0 || mask_grid.ny() > 0) { mask_grid.latlon_to_xy(lat, -1.0*lon, grid_x, grid_y); @@ -3977,7 +4048,7 @@ bool check_masks(const MaskPoly &mask_poly, const Grid &mask_grid, } // - // Check area mask. + // Check area mask // if(mask_area.nx() > 0 || mask_area.ny() > 0) { if(!mask_area.s_is_on(nint(grid_x), nint(grid_y))) { diff --git a/met/src/tools/tc_utils/tc_stat/tc_stat_job.h b/met/src/tools/tc_utils/tc_stat/tc_stat_job.h index f97d5ae581..4ad98be987 100644 --- a/met/src/tools/tc_utils/tc_stat/tc_stat_job.h +++ b/met/src/tools/tc_utils/tc_stat/tc_stat_job.h @@ -286,13 +286,15 @@ class TCStatJob { map ColumnThreshMap; // ASCII column string matching - map ColumnStrMap; + map ColumnStrIncMap; + map ColumnStrExcMap; // Numeric column thresholds map InitThreshMap; // ASCII column string matching - map InitStrMap; + map InitStrIncMap; + map InitStrExcMap; // Variables to the store the analysis job specification ConcatString DumpFile; // Dump TrackPairInfo used to a file diff --git a/test/config/TCStatConfig_ALAL2010 b/test/config/TCStatConfig_ALAL2010 index 714029745f..8f59d36f5b 100644 --- a/test/config/TCStatConfig_ALAL2010 +++ b/test/config/TCStatConfig_ALAL2010 @@ -112,6 +112,12 @@ column_thresh_val = []; column_str_name = []; column_str_val = []; +// +// Stratify by excluding strings in non-numeric data columns. +// +column_str_exc_name = []; +column_str_exc_val = []; + // // Similar to the column_thresh options above // @@ -124,6 +130,12 @@ init_thresh_val = []; init_str_name = []; init_str_val = []; +// +// Similar to the column_str_exc options above +// +init_str_exc_name = []; +init_str_exc_val = []; + // // Stratify by the ADECK and BDECK distances to land. // diff --git a/test/config/TCStatConfig_PROBRIRW b/test/config/TCStatConfig_PROBRIRW index 9843408d2a..bf443a9e66 100644 --- a/test/config/TCStatConfig_PROBRIRW +++ b/test/config/TCStatConfig_PROBRIRW @@ -112,6 +112,12 @@ column_thresh_val = []; column_str_name = []; column_str_val = []; +// +// Stratify by excluding strings in non-numeric data columns. +// +column_str_exc_name = []; +column_str_exc_val = []; + // // Similar to the column_thresh options above // @@ -124,6 +130,12 @@ init_thresh_val = []; init_str_name = []; init_str_val = []; +// +// Similar to the column_str_exc options above +// +init_str_exc_name = []; +init_str_exc_val = []; + // // Stratify by the ADECK and BDECK distances to land. // diff --git a/test/xml/unit_ascii2nc.xml b/test/xml/unit_ascii2nc.xml index fc1d1c4b86..2edb2bf9b7 100644 --- a/test/xml/unit_ascii2nc.xml +++ b/test/xml/unit_ascii2nc.xml @@ -76,6 +76,18 @@ + + &MET_BIN;/ascii2nc + \ + &DATA_DIR_OBS;/ascii/OBS:2015080700_bad_record \ + &OUTPUT_DIR;/ascii2nc/OBS:2015080700_bad_record.nc \ + -v 1 + + + &OUTPUT_DIR;/ascii2nc/OBS:2015080700_bad_record.nc + + + &MET_BIN;/ascii2nc \ diff --git a/test/xml/unit_stat_analysis.xml b/test/xml/unit_stat_analysis.xml index 799011f8a6..1558d101fd 100644 --- a/test/xml/unit_stat_analysis.xml +++ b/test/xml/unit_stat_analysis.xml @@ -315,6 +315,21 @@ + + &MET_BIN;/stat_analysis + \ + -lookin &OUTPUT_DIR;/point_stat/point_stat_GRIB1_NAM_GDAS_120000L_20120409_120000V.stat \ + -job filter -line_type MPR -fcst_var TMP -fcst_lev Z2 -vx_mask DTC165 \ + -column_str OBS_SID KDLN,KDHT,KDEN,KDLS,KDMA,KDMN,KDVT,KDEW \ + -column_str_exc OBS_SID KDLN,KDHT \ + -dump_row &OUTPUT_DIR;/stat_analysis/POINT_STAT_FILTER_OBS_SID.stat \ + -v 1 + + + &OUTPUT_DIR;/stat_analysis/POINT_STAT_FILTER_OBS_SID.stat + + + OUTPUT_DIR &OUTPUT_DIR;/stat_analysis diff --git a/test/xml/unit_tc_stat.xml b/test/xml/unit_tc_stat.xml index 8a7891a2fd..7353d72f68 100644 --- a/test/xml/unit_tc_stat.xml +++ b/test/xml/unit_tc_stat.xml @@ -15,7 +15,6 @@ &TEST_DIR; true - &MET_BIN;/tc_stat \ @@ -33,6 +32,20 @@ + + &MET_BIN;/tc_stat + \ + -lookin &OUTPUT_DIR;/tc_pairs/alal2010.tcst \ + -job filter -dump_row &OUTPUT_DIR;/tc_stat/ALAL2010_FILTER_STRINGS.tcst \ + -init_str LEVEL TS,HU -init_str_exc WATCH_WARN HUWARN \ + -column_str LEVEL HU -column_str_exc WATCH_WARN TSWATCH \ + -v 2 + + + &OUTPUT_DIR;/tc_stat/ALAL2010_FILTER_STRINGS.tcst + + + &MET_BIN;/tc_stat \