Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new Ensemble-Stat line type to store the decomposition of the empirical CRPS into reliability and potential. #1663

Open
4 of 21 tasks
JohnHalleyGotway opened this issue Feb 15, 2021 · 2 comments
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue MET: Ensemble Verification MET: Library Code priority: medium Medium Priority requestor: UK Met Office United Kingdom Met Office required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: new feature Make it do something new

Comments

@JohnHalleyGotway
Copy link
Collaborator

Describe the New Feature

MET #1450 adds the empirical CRPS and corresponding skill score to the ECNT line type, as described in Hersbach, 2000. During that development, the UK MetOffice pointed out that their existing verification system (ver) is able to decompose the CRPS into its components: CRPS = Reliability + Potential. Doing so for a single case is straight-forward enough but aggregating those scores over multiple cases is challenging. Properly doing so requires that an alpha and beta value be stored for each ensemble member. Adding 2 * number of ensemble members to the existing ECNT line type was not practical, and so the decomposition was not computed.

This task is to add a new Ensemble-Stat line type for the CRPS decomposition. It should include the CRPS_EMP, CRPS_REL, CRPS_UNC, and the alpha and beta values for each member. Also for this issue, ensure that Stat-Analysis can aggregate multiple lines together.

Don't forget to define corresponding METdatadb and/or METviewer issues to store this new line type!

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
  • Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

New Feature Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s), Project(s), Milestone, and Linked issues
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added component: library code type: new feature Make it do something new priority: medium Medium Priority requestor: UK Met Office United Kingdom Met Office alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle labels Feb 15, 2021
@JohnHalleyGotway JohnHalleyGotway added this to the MET 10.1.0 milestone Feb 15, 2021
@JohnHalleyGotway
Copy link
Collaborator Author

Here are some development notes I used when testing the python and R packages for doing the empirical CRPS decomposition:
Dev_Notes.txt

@TaraJensen TaraJensen removed the alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle label May 25, 2021
@TaraJensen TaraJensen added this to To do in MET-10.1.0-beta2 (9/01/21) via automation May 25, 2021
@TaraJensen TaraJensen added the required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone label May 25, 2021
@JohnHalleyGotway JohnHalleyGotway added this to To do in MET-10.1.0-beta3 (10/06/21) via automation Aug 26, 2021
@JohnHalleyGotway JohnHalleyGotway moved this from To do to To do: Ensemble-Stat Reorg in MET-10.1.0-beta3 (10/06/21) Aug 26, 2021
@JohnHalleyGotway JohnHalleyGotway removed this from To do: Ensemble-Stat Reorg in MET-10.1.0-beta3 (10/06/21) Oct 6, 2021
@JohnHalleyGotway JohnHalleyGotway added this to To do in MET-10.1.0-beta4 (11/17/21) via automation Oct 6, 2021
@JohnHalleyGotway JohnHalleyGotway moved this from To do to To Do: Ensemble Verification in MET-10.1.0-beta4 (11/17/21) Oct 6, 2021
@JohnHalleyGotway JohnHalleyGotway removed this from To Do: Ensemble Verification in MET-10.1.0-beta4 (11/17/21) Nov 15, 2021
@JohnHalleyGotway JohnHalleyGotway added this to To do in MET-10.1.0-beta5 (1/14/22) via automation Nov 15, 2021
@TaraJensen TaraJensen added this to To do in MET-10.1.0-beta6 (3/2/22) via automation Jan 6, 2022
@JohnHalleyGotway JohnHalleyGotway removed this from the MET 10.1.0 milestone Feb 3, 2022
@JohnHalleyGotway
Copy link
Collaborator Author

@mpm-meto, is this feature required for MET-11.0.0-beta4 or can it be pushed to the next release in favor of other higher priority issues?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue MET: Ensemble Verification MET: Library Code priority: medium Medium Priority requestor: UK Met Office United Kingdom Met Office required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: new feature Make it do something new
Projects
None yet
Development

No branches or pull requests

2 participants