Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Uniform Fractions Skill Score UFSS in appendixC.rst #1431

Merged
merged 1 commit into from Jul 29, 2020

Conversation

ahijevyc
Copy link
Contributor

In Roberts and Lean 2008, UFSS is defined as 0.5 +􏰆 fo/2 where fo is observed frequency (fraction of observed points exceeding the threshold over the domain)--see Section 2d, halfway down the right column of page 83. The critical word is observed.

However, the User's Guide v9.0.3 it says UFSS is equivalent to the forecast event proportion--a subtle but important difference if there is any forecast frequency bias. UFSS will differ depending on if it is based on observed event proportion or forecasted event proportion.

Now, I do believe, behind the scenes, MET based the UFSS on the observed frequency, consistent with Roberts and Lean 2008. Only the User's Guide needs to be revised.

I also revised the description of UFSS and formulation to match Roberts and Lean 2008. UFSS is simply halfway between perfect skill (unity) and a random FSS (equal to observed event proportion).

In Roberts and Lean 2008, UFSS is defined as 0.5 +􏰆 fo/2 where fo is observed frequency  (fraction of observed points exceeding the threshold over the domain)--see Section 2d, halfway down the right column of page 83.  The critical word is observed.  

However, the User's Guide v9.0.3 it says UFSS is equivalent to the forecast event proportion--a subtle but important difference if there is any forecast frequency bias. UFSS will differ depending on if it is based on observed event proportion or forecasted event proportion. 

Now, I do believe, behind the scenes, MET based the UFSS on the observed frequency, consistent with Roberts and Lean 2008. Only the User's Guide needs to be revised.

I also revised the description of UFSS and formulation to match Roberts and Lean 2008. UFSS is simply halfway between perfect skill (unity) a random FSS (equal to observed event proportion).
@ahijevyc
Copy link
Contributor Author

I removed the reference to UFSS needing to be exceeded in order to have skill. It depends on your baseline. Roberts and Lean (2008) offer FSSrandom as a possible baseline, based on a random forecast FSS. It is only equal to the total observed event proportion. For rare events, this is a much lower "skill threshold" than UFSS.

@JohnHalleyGotway JohnHalleyGotway self-requested a review July 29, 2020 22:14
Copy link
Collaborator

@JohnHalleyGotway JohnHalleyGotway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve of this change. I double-checked the code and see that on the following line, UFSS really is only a function of the observation rate, and not the forecast rate: line:https://github.com/NCAR/MET/blob/0cee9fbd0b3c691fd3cf4dd1d665f5d2b0db9a37/met/src/libcode/vx_statistics/met_stats.cc#L3161

@JohnHalleyGotway JohnHalleyGotway merged commit 2770bdc into dtcenter:task_1217_lyx_to_sphinx Jul 29, 2020
@JohnHalleyGotway JohnHalleyGotway added this to In progress in MET-9.1-beta3 (7/29/20) via automation Jul 29, 2020
@JohnHalleyGotway JohnHalleyGotway moved this from In progress to Done in MET-9.1-beta3 (7/29/20) Jul 29, 2020
@JohnHalleyGotway JohnHalleyGotway added this to the MET 9.1 milestone Jul 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

2 participants