Skip to content

Home

Jump to bottom
Peter Rochford edited this page Aug 23, 2023 · 31 revisions

Summary

This package contains a collection of functions for calculating the skill of model predictions against observations. It includes metrics such as root-mean-square-error (RMSE) difference, centered root-mean-square (RMS) difference, and skill score (SS), as well as a collection of functions for producing target and Taylor diagrams. The more valuable feature of the toolbox are the plotting functions for target and Taylor diagrams and the ability to easily customize the diagrams.

Installation

To install the package simply use the pip3 command:

pip3 install SkillMetrics

If you are upgrading the package then include the upgrade option:

pip3 install SkillMetrics --upgrade

Note that SkillMetrics now only supports Python 3 because Python 2 has been deprecated. Using pip to install may therefore not work.

Diagram Examples

The target and Taylor diagrams produced by the several examples posted on this wiki can be viewed by following the links shown below. This is a useful starting point for users looking to identify the best example from which to begin creating a diagram for their specific need by modifying the accompanying Python script.

It is expected that examples will be added over time as the SkillMetrics package continues to evolve. So, revisit this page whenever the need arises to produce new diagrams.

Tutorial

An easy tutorial is provided here in the form of a series of example scripts to illustrate how to produce target and Taylor diagrams in a variety of formats using the Skill Metrics package. For background information on target diagrams the reader is referred to Jolliff et al. (2009) and for Taylor diagrams to the primer. There are 7 examples for target diagrams and 8 examples for Taylor diagrams that successively progress from very simple to more customized figures. These series of examples provide an easy tutorial on how to use the various options of the target_diagram and taylor_diagram functions. They also provide a quick reference in future for how to produce the diagrams with specific features. The diagrams produced by each script are in Portable Network Graphics (PNG) format and have the same file name as the script with a "png" suffix. Examples of the diagrams produced can be found in the Examples folder with the same file name as the script and ending in '_example.png'. For example the diagram produced by target1.m is named target1_example.png. All target diagram examples require the target_data.pkl file while the Taylor examples require the taylor_data.pkl file. Follow the links below to navigate each of the target and Taylor diagram examples.

Examples will be added over time as the SkillMetrics package continues to evolve.

Target Diagrams

Taylor Diagrams

All Statistics

There is also a simple program in the Examples folder (all_stats.py) that provides examples of how to calculate the various skill metrics used or available in the toolbox. All the calculated skill metrics are written to an Excel file for easy viewing and manipulation. The script requires the target_data.pkl file.

  • bias : Calculate the bias between two variables
  • bias_percent : Calculate the percentage bias between two variables
  • brier_score : Calculate Brier score (BS) between two variables
  • centered_rms_dev : Calculate centered root-mean-square difference (CRMSD, E') between two variables
  • check_taylor_stats : Checks input statistics satisfy Taylor diagram relation to <1%.
  • kling_gupta_eff09 : Calculate the Kling-Gupta efficiency from 2009 paper (KGE09) between two variables
  • kling_gupta_eff12 : Calculate the Kling-Gupta efficiency from 2012 paper (KGE12) between two variables
  • nash_sutcliffe_eff : Calculate Nash-Sutcliffe efficiency between two variables
  • rmsd : Calculate the root-mean-square deviation (RMSD) between two variables
  • skill_score_brier : Calculate Brier skill score (BSS) between two variables
  • skill_score_murphy : Calculate non-dimensional skill score (SS) between two variables

Plotting Options

A list of the currently available plotting options for the diagrams can be found by following the links below:

Reading Data

Functions are provided for reading data for which skill metrics and diagrams are desired as a courtesy to users. However, these functions are not included within the skill_metrics module. Refer to the Reading Data Wiki page for the list of available functions and how to use.

Trouble Shooting

For problems encountered using the package refer to the FAQ page of the Wiki.

How to cite SkillMetrics

Peter A. Rochford (2016) SkillMetrics: A Python package for calculating the skill of model predictions against observations, https://github.com/PeterRochford/SkillMetrics

@misc{rochfordskillmetrics, 
  title={SkillMetrics: A Python package for calculating the skill of model predictions against observations}, 
  author={Peter A. Rochford}, 
  year={2016}, 
  url={https://github.com/PeterRochford/SkillMetrics}
}

References

  • Jolliff, J. K., J. C. Kindle, I. Shulman, B. Penta, M. Friedrichs, R. Helber, and R. Arnone (2009), Skill assessment for coupled biological/physical models of marine systems, J. Mar. Sys., 76(1-2), 64-82, doi:10.1016/j.jmarsys.2008.05.014
  • Taylor, K. E. (2001) Summarizing multiple aspects of model performance in a single diagram, J. Geophys. Res., 106(D7), 7183–7192, doi:10.1029/2000JD900719.
Clone this wiki locally