Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Area Ratio calculation consistent (FCST/OBS) #1643

Closed
17 of 20 tasks
lindsayrblank opened this issue Jan 27, 2021 · 0 comments · Fixed by #1650 or #1652
Closed
17 of 20 tasks

Make Area Ratio calculation consistent (FCST/OBS) #1643

lindsayrblank opened this issue Jan 27, 2021 · 0 comments · Fixed by #1650 or #1652
Assignees
@JohnHalleyGotway
Labels
priority: blocker Blocker requestor: METplus Team METplus Development Team requestor: Navy/NRL Naval Research Laboratory type: enhancement Improve something that it is currently doing
Projects
MET-10.0.0-beta4 (3/2/21)
Milestone
MET 10.0.0

Comments

@lindsayrblank
Copy link

lindsayrblank commented Jan 27, 2021 •
edited by JohnHalleyGotway

Describe the Enhancement

Write out area ratios with FCST/OBS --> currently calculated with whichever value is larger is on the bottom (FCST or OBS), not consistent.

Time Estimate

4 hours

Sub-Issues

No sub issues required

Relevant Deadlines

MET 10.0.0

Funding Source

TBD

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required: John HG
  • Select scientist(s) or no scientist required: No scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
  • Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s), Project(s), Milestone, and Linked issues
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@lindsayrblank lindsayrblank added type: enhancement Improve something that it is currently doing component: application code priority: blocker Blocker alert: NEED ACCOUNT KEY Need to assign an account key to this issue requestor: METplus Team METplus Development Team labels Jan 27, 2021
@lindsayrblank lindsayrblank added this to the MET 10.0.0 milestone Jan 27, 2021
@lindsayrblank lindsayrblank added this to To do in MET-10.0.0-beta4 (3/2/21) via automation Jan 27, 2021
JohnHalleyGotway added a commit that referenced this issue Feb 1, 2021
@JohnHalleyGotway
Per #1643, redefine the contents of the existing AREA_RATIO output co…
0b37b5f 
…lumn from MODE. Define it as FCST/OBS object area instead of min/max. Update the User's Guide to note the change and also clarify that the MTD VOLUME_RATIO output really is FCST/OBS.
@JohnHalleyGotway JohnHalleyGotway moved this from To do to In progress in MET-10.0.0-beta4 (3/2/21) Feb 1, 2021
@JohnHalleyGotway JohnHalleyGotway linked a pull request Feb 1, 2021 that will close this issue
Feature 1643 area_ratio #1650
Merged
10 tasks
JohnHalleyGotway added a commit that referenced this issue Feb 1, 2021
@JohnHalleyGotway
Per #1643, redefine the contents of the existing AREA_RATIO output co…
1186ec3 
…lumn from MODE. Define it as FCST/OBS object area instead of min/max. Update the User's Guide to note the change and also clarify that the MTD VOLUME_RATIO output really is FCST/OBS. (#1650)
@JohnHalleyGotway JohnHalleyGotway moved this from In progress to Done in MET-10.0.0-beta4 (3/2/21) Feb 1, 2021
@JohnHalleyGotway JohnHalleyGotway mentioned this issue Feb 3, 2021
Merged
10 tasks
@JohnHalleyGotway JohnHalleyGotway linked a pull request Feb 3, 2021 that will close this issue
Update develop-ref after #1650 #1652
Merged
10 tasks
JohnHalleyGotway added a commit that referenced this issue Feb 3, 2021
@JohnHalleyGotway
Update develop-ref after #1650 (#1652)
35eadf2 
* Per 1646, one line fix for cut-and-paste error. (#1647)

* Per #1643, redefine the contents of the existing AREA_RATIO output column from MODE. Define it as FCST/OBS object area instead of min/max. Update the User's Guide to note the change and also clarify that the MTD VOLUME_RATIO output really is FCST/OBS. (#1650)

* Feature 1644 ps_log (#1651)

* Per #1644, write rejection reason codes at verbosity 2 when there are 0 matched pairs.

* Per #1644, add a few sentences to Point-Stat, Practical Information chapter about debugging 0 matched pairs.

* The mode_conv.pl logic was slightly broken. MET PR #1650 should have broken the NB but it did not. Turns out the diffing logic is NOT properly distinguishing between single and pair object lines. It does this by looking for an underscore in the OBJECT_ID column. When we added FCST_UNITS and OBS_UNITS, that shifted OBJECT_ID up 2 spots, but the code was still checking the (0-based) 20th column instead of the 22nd. Fixing this now and will rerun NB20210202 to confirm it works again.

* The diffing logic for MODE pair lines still was not correct. We'd added the ASPECT_DIFF and CURVATURE_RATIO columns a while ago, but they were missing from the diff logic. This logic really is not good. We need to make it more robust, reading the version-specific header columns from a table file instead of hard-coding them!

Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
JohnHalleyGotway added a commit that referenced this issue Feb 18, 2021
@JohnHalleyGotway @hsoh-u @jprestop
Update develop-ref after #1662 (#1669)
e9ec2c1 
* Per 1646, one line fix for cut-and-paste error. (#1647)

* Per #1644, no actual code changes here. Just formatting and spacing. For example, replace double ;; with single ;'

* Per #1644, FOUND THE BUG! It's a copy/paste error. We had var_name_map.end() that should be def_var_name_map.end(). Fixing that gets rid of the runtime hang.'

* Per #1643, redefine the contents of the existing AREA_RATIO output column from MODE. Define it as FCST/OBS object area instead of min/max. Update the User's Guide to note the change and also clarify that the MTD VOLUME_RATIO output really is FCST/OBS. (#1650)

* Feature 1644 ps_log (#1651)

* Per #1644, write rejection reason codes at verbosity 2 when there are 0 matched pairs.

* Per #1644, add a few sentences to Point-Stat, Practical Information chapter about debugging 0 matched pairs.

* The mode_conv.pl logic was slightly broken. MET PR #1650 should have broken the NB but it did not. Turns out the diffing logic is NOT properly distinguishing between single and pair object lines. It does this by looking for an underscore in the OBJECT_ID column. When we added FCST_UNITS and OBS_UNITS, that shifted OBJECT_ID up 2 spots, but the code was still checking the (0-based) 20th column instead of the 22nd. Fixing this now and will rerun NB20210202 to confirm it works again.

* The diffing logic for MODE pair lines still was not correct. We'd added the ASPECT_DIFF and CURVATURE_RATIO columns a while ago, but they were missing from the diff logic. This logic really is not good. We need to make it more robust, reading the version-specific header columns from a table file instead of hard-coding them!

* Feature 1653 rscripts (#1654)

* Per #1653, update plot_cnt.R and plot_mpr.R to remove the version-specific header columns.

* Per #1653, nice enhancments to these Rscripts to make them more independent of the MET version number.

* Per #1653, more tweaks

* Per #1653, if no input files are provided, error out with a useful message.

* Per #1653, while the scripts ran fine using R 4.0.2 on my Mac, they fail on eyewall using R 3.4.0. Adding as.character() to get past that error.

* Feature 1655 nc_log (#1656)

* #1630 Display a warning instead of error message with invalid variable if the input data is empty

* Feature 1658 grib_tables (#1659)

* Per #1658, update MXUPHL entries.

* Per #1658, updating long name for MAXREF, MAXUVV, and MAXDVV.

* Modified format of release notes

* Feature 1450 hersbach (#1662)

* Per #1450, add new ECNT columns for Hersback CRPS. Still need to actually compute the stats though.

* Per #1450, update NumArray functions to only sort if the data is not yet sorted. And check for bad data when computing the standard deviation.

* Per #1450, add code to compute the empirical CRPS value.

* Per #1450, large change to the new output for the empirical CRPS. In order to aggregate decomposed empirical CRPS reliability and potential correctly, we'd need to write (n+1)*2 additional columns. While the empirical crps can be aggregated as a weighted mean, the decomposition cannot. It just isn't feasible to do this in the ECNT line type. If this reliability and potential really are required, recommend that we add an entirely new CRPS line type instead of tacking onto ECNT. These changes simply remove reliabilit and potential from the output.

* Per #1450 and #1451, replacing single CRPS_CLIMO column with CRPSCL and CRPSCL_EMP which will be needed for #1451.

* Per #1450, delete temp files I'd accidentally committed.

* Per #1450, update the user's guide with CRPS updates.

* Fix bug replacing crpss_emp with crpss_gaus.

Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
Co-authored-by: hsoh-u <hsoh@ucar.edu>
Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu>
JohnHalleyGotway added a commit that referenced this issue Feb 25, 2021
@JohnHalleyGotway @hsoh-u
@jprestop @j-opatz
Update develop-ref after #1681 and #1676 (#1683)
ba0cc8e 
* Per 1646, one line fix for cut-and-paste error. (#1647)

* Per #1644, no actual code changes here. Just formatting and spacing. For example, replace double ;; with single ;'

* Per #1644, FOUND THE BUG! It's a copy/paste error. We had var_name_map.end() that should be def_var_name_map.end(). Fixing that gets rid of the runtime hang.'

* Per #1643, redefine the contents of the existing AREA_RATIO output column from MODE. Define it as FCST/OBS object area instead of min/max. Update the User's Guide to note the change and also clarify that the MTD VOLUME_RATIO output really is FCST/OBS. (#1650)

* Feature 1644 ps_log (#1651)

* Per #1644, write rejection reason codes at verbosity 2 when there are 0 matched pairs.

* Per #1644, add a few sentences to Point-Stat, Practical Information chapter about debugging 0 matched pairs.

* The mode_conv.pl logic was slightly broken. MET PR #1650 should have broken the NB but it did not. Turns out the diffing logic is NOT properly distinguishing between single and pair object lines. It does this by looking for an underscore in the OBJECT_ID column. When we added FCST_UNITS and OBS_UNITS, that shifted OBJECT_ID up 2 spots, but the code was still checking the (0-based) 20th column instead of the 22nd. Fixing this now and will rerun NB20210202 to confirm it works again.

* The diffing logic for MODE pair lines still was not correct. We'd added the ASPECT_DIFF and CURVATURE_RATIO columns a while ago, but they were missing from the diff logic. This logic really is not good. We need to make it more robust, reading the version-specific header columns from a table file instead of hard-coding them!

* Feature 1653 rscripts (#1654)

* Per #1653, update plot_cnt.R and plot_mpr.R to remove the version-specific header columns.

* Per #1653, nice enhancments to these Rscripts to make them more independent of the MET version number.

* Per #1653, more tweaks

* Per #1653, if no input files are provided, error out with a useful message.

* Per #1653, while the scripts ran fine using R 4.0.2 on my Mac, they fail on eyewall using R 3.4.0. Adding as.character() to get past that error.

* Feature 1655 nc_log (#1656)

* #1630 Display a warning instead of error message with invalid variable if the input data is empty

* Feature 1658 grib_tables (#1659)

* Per #1658, update MXUPHL entries.

* Per #1658, updating long name for MAXREF, MAXUVV, and MAXDVV.

* Modified format of release notes

* Feature 1450 hersbach (#1662)

* Per #1450, add new ECNT columns for Hersback CRPS. Still need to actually compute the stats though.

* Per #1450, update NumArray functions to only sort if the data is not yet sorted. And check for bad data when computing the standard deviation.

* Per #1450, add code to compute the empirical CRPS value.

* Per #1450, large change to the new output for the empirical CRPS. In order to aggregate decomposed empirical CRPS reliability and potential correctly, we'd need to write (n+1)*2 additional columns. While the empirical crps can be aggregated as a weighted mean, the decomposition cannot. It just isn't feasible to do this in the ECNT line type. If this reliability and potential really are required, recommend that we add an entirely new CRPS line type instead of tacking onto ECNT. These changes simply remove reliabilit and potential from the output.

* Per #1450 and #1451, replacing single CRPS_CLIMO column with CRPSCL and CRPSCL_EMP which will be needed for #1451.

* Per #1450, delete temp files I'd accidentally committed.

* Per #1450, update the user's guide with CRPS updates.

* Fix bug replacing crpss_emp with crpss_gaus.

* #1657 Added TIME_EPSILON

* #1657 Corrected 1 second offset by the precision error

* #1657 Added AccumTime

* #1657 Read the time from "bounds" attribute and set the max value from the bounds time variable

* #1657 Corrected 1 second offset by the precision error

* Per #1439, add check_mask_names() utility function which errors out if the list of masking region names is non-unique. Update Point-Stat and Grid-Stat to call it. (#1679)

* Feature 1451 crpss (#1676)

* Per #1450, add new ECNT columns for Hersback CRPS. Still need to actually compute the stats though.

* Per #1450, update NumArray functions to only sort if the data is not yet sorted. And check for bad data when computing the standard deviation.

* Per #1450, add code to compute the empirical CRPS value.

* Per #1450, large change to the new output for the empirical CRPS. In order to aggregate decomposed empirical CRPS reliability and potential correctly, we'd need to write (n+1)*2 additional columns. While the empirical crps can be aggregated as a weighted mean, the decomposition cannot. It just isn't feasible to do this in the ECNT line type. If this reliability and potential really are required, recommend that we add an entirely new CRPS line type instead of tacking onto ECNT. These changes simply remove reliabilit and potential from the output.

* Per #1450 and #1451, replacing single CRPS_CLIMO column with CRPSCL and CRPSCL_EMP which will be needed for #1451.

* Per #1450, delete temp files I'd accidentally committed.

* Per #1450, update the user's guide with CRPS updates.

* Per #1451, instead of computing the climo crps on the fly, compute and store it separately for each point.

* Per #1451, the ECNT line type will no longer be written separately for each CDF bin. Removing the bin-related arguments from the write_ecnt functions.

* Per #1451, the climo_cdf.write_bins option no longer applies. Since Ensemble-Stat will no longer compute stats separately for each climo bin, I'm removing the reference to write_bins from the Ensemble-Stat config files.

* Per #1451, compute and store the climo CRPS for each point. Also, break apart the normal climo computation into separate functions for crps, ign, and pit.

* Per #1451, update Ensemble-Stat logic to no longer subset pairs into climo CDF bins. We had done this to be consistent with the use of climo data in point and grid-stat. But this change to the handling of climo data is consitent with the NOAA/EMC approach.

* Per #1451, split out the setting of climo CDF thresholds into a separate function so that it can also be called by stat-analysis.

* Per #1451, in the Ensemble-Stat ORANK line type, rename CLIMO to CLIMO_MEAN and add a CLIMO_STDEV column.

* Per #1451, also need to update gsidens2orank to write a climo_stdev column.

* Per #1451, switch from constant pointer to ClimoCDFInfo object to a copy to make the logic of doing this in Stat-Analysis a little easier.

* Per #1451, the HiRA method in Point-Stat computes an ECNT output line type. Needed to call set_climo_cdf() there so that we know how many climo values to use when computing the empirical climo CRPS.

* Per #1451, need to store climo_cdf for both grid and point verification.

* Per #1451, update to write the CLIMO_STDEV header column for the ORANK line type.

* Per #1451, in Ensemble-Stat when doing point verification, check for empty OBS_UNIT string and write NA instead.

* Per #1451, update unit tests by enhancing the climatology call to Ensemble-Stat to also include point verification. Tweak the Ensemble-Stat cofiguration for that and also add a call to pb2nc to prepare the point observations for use.

* Per #1450, added a new section to the Ensemble-Stat chapter describing how climo mean/stdev are used in the computation of the skill scores.

* Update ensemble-stat.rst

Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>

* #1677 Update the refence time (from time_bnds variable) (#1680)

Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>

* Feature 1135 stat_analysis (#1681)

* Per #1135, add fcst/obs_init/valid_inc/exc options for STAT-Analysis jobs.

* Per #1135, update all the STATAnalysis config files to include entries for the new fcst/obs_init/valid_inc/exc options.

* Per #1135, add documentation for fcst/obs_init/valid_inc/exc options to the STAT-Analysis chapter. Also, clarify the description for the existing options.

* Per #1135, adding another call to stat_analysis to check the time filtering options.

* Per #1135, just renaming stat_analysis output file.

* Apply suggestions from code review

Co-authored-by: jprestop <jpresto@ucar.edu>

Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
Co-authored-by: jprestop <jpresto@ucar.edu>

Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
Co-authored-by: hsoh-u <hsoh@ucar.edu>
Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu>
Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>
@TaraJensen TaraJensen added requestor: Navy/NRL Naval Research Laboratory and removed alert: NEED ACCOUNT KEY Need to assign an account key to this issue labels Mar 31, 2021
@JohnHalleyGotway JohnHalleyGotway mentioned this issue Mar 10, 2022
Merged
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JohnHalleyGotway JohnHalleyGotway

Labels
priority: blocker Blocker requestor: METplus Team METplus Development Team requestor: Navy/NRL Naval Research Laboratory type: enhancement Improve something that it is currently doing
Projects
No open projects
MET-10.0.0-beta4 (3/2/21)
  
Done
Milestone
MET 10.0.0
Development

Successfully merging a pull request may close this issue.

Feature 1643 area_ratio dtcenter/MET
Update develop-ref after #1650 dtcenter/MET
3 participants
@lindsayrblank @JohnHalleyGotway @TaraJensen