Reliability (xarray-contrib#164)

* moved _preprocess_dims, _stack_input_if_needed, _preprocess_weights from deterministic.py to utils.py * added reliability to probabilistic.py * added accessor and test to test_accessor_probabilistic.py * added accessor and test to test_accessor_probabilistic.py * make FORECAST_PROBABILITY_DIM a variable * precommit * added tests * rtd * CHANGELOG * rtd att 2 * moved samples to a new coordinate * Update xskillscore/core/probabilistic.py Co-authored-by: Aaron Spring <aaronspring@users.noreply.github.com> * added fillna option * rtd * quick-start.ipynb * rtd * combine discrimination outputs into single object * remove fillna kwarg from reliability * replace xr.ufuncs with np * linting issues * rtd * rtd * added reliability to list of metrics in quick-start Co-authored-by: Aaron Spring <aaronspring@users.noreply.github.com>
raybellwaves · Sep 2, 2020 · de4b538 · de4b538
1 parent ffc6903
commit de4b538
Show file tree

Hide file tree

Showing 49 changed files with 462 additions and 241 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -24,6 +24,7 @@ Features
 - Added contingency table and associated metrics (:pr:`119`, :pr:`153`). `Dougie Squire`_
 - Added ``rank_histogram`` and ``discrimination`` to probabilistic metrics (:pr:`136`).
   `Dougie Squire`_
+- Added ``reliability`` to probabilistic metrics (:pr:`164`). `Dougie Squire`_
 - Added ``dim`` and ``weights`` for probabilistic metrics. (:pr:`121`) `Aaron Spring`_
 - Added ``keep_attrs`` for all metrics. (:pr:`122`) `Andrew Huang`_
 - Added ranked probability score ``rps``. (:pr:`163`) `Aaron Spring`_

diff --git a/docs/source/api.rst b/docs/source/api.rst
@@ -57,6 +57,7 @@ Currently, most of our probabilistic metrics are ported over from
     rps
     rank_histogram
     discrimination
+    reliability
 
 Contingency-based Metrics
 -------------------------

diff --git a/docs/source/api/xskillscore.Contingency.accuracy.rst b/docs/source/api/xskillscore.Contingency.accuracy.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.accuracy
+xskillscore.Contingency.accuracy
 ================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.bias_score.rst b/docs/source/api/xskillscore.Contingency.bias_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.bias\_score
+xskillscore.Contingency.bias\_score
 ===================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.correct_negatives.rst b/docs/source/api/xskillscore.Contingency.correct_negatives.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.correct\_negatives
+xskillscore.Contingency.correct\_negatives
 ==========================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.equit_threat_score.rst b/docs/source/api/xskillscore.Contingency.equit_threat_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.equit\_threat\_score
+xskillscore.Contingency.equit\_threat\_score
 ============================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.false_alarm_rate.rst b/docs/source/api/xskillscore.Contingency.false_alarm_rate.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.false\_alarm\_rate
+xskillscore.Contingency.false\_alarm\_rate
 ==========================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.false_alarm_ratio.rst b/docs/source/api/xskillscore.Contingency.false_alarm_ratio.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.false\_alarm\_ratio
+xskillscore.Contingency.false\_alarm\_ratio
 ===========================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.false_alarms.rst b/docs/source/api/xskillscore.Contingency.false_alarms.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.false\_alarms
+xskillscore.Contingency.false\_alarms
 =====================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.gerrity_score.rst b/docs/source/api/xskillscore.Contingency.gerrity_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.gerrity\_score
+xskillscore.Contingency.gerrity\_score
 ======================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.heidke_score.rst b/docs/source/api/xskillscore.Contingency.heidke_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.heidke\_score
+xskillscore.Contingency.heidke\_score
 =====================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.hit_rate.rst b/docs/source/api/xskillscore.Contingency.hit_rate.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.hit\_rate
+xskillscore.Contingency.hit\_rate
 =================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.hits.rst b/docs/source/api/xskillscore.Contingency.hits.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.hits
+xskillscore.Contingency.hits
 ============================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.misses.rst b/docs/source/api/xskillscore.Contingency.misses.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.misses
+xskillscore.Contingency.misses
 ==============================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.odds_ratio.rst b/docs/source/api/xskillscore.Contingency.odds_ratio.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.odds\_ratio
+xskillscore.Contingency.odds\_ratio
 ===================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.odds_ratio_skill_score.rst b/docs/source/api/xskillscore.Contingency.odds_ratio_skill_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.odds\_ratio\_skill\_score
+xskillscore.Contingency.odds\_ratio\_skill\_score
 =================================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.peirce_score.rst b/docs/source/api/xskillscore.Contingency.peirce_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.peirce\_score
+xskillscore.Contingency.peirce\_score
 =====================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.success_ratio.rst b/docs/source/api/xskillscore.Contingency.success_ratio.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.success\_ratio
+xskillscore.Contingency.success\_ratio
 ======================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.table.rst b/docs/source/api/xskillscore.Contingency.table.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.table
+xskillscore.Contingency.table
 =============================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.Contingency.threat_score.rst b/docs/source/api/xskillscore.Contingency.threat_score.rst
@@ -1,4 +1,4 @@
-xskillscore.Contingency.threat\_score
+xskillscore.Contingency.threat\_score
 =====================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.brier_score.rst b/docs/source/api/xskillscore.brier_score.rst
@@ -1,4 +1,4 @@
-xskillscore.brier\_score
+xskillscore.brier\_score
 ========================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.crps_ensemble.rst b/docs/source/api/xskillscore.crps_ensemble.rst
@@ -1,4 +1,4 @@
-xskillscore.crps\_ensemble
+xskillscore.crps\_ensemble
 ==========================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.crps_gaussian.rst b/docs/source/api/xskillscore.crps_gaussian.rst
@@ -1,4 +1,4 @@
-xskillscore.crps\_gaussian
+xskillscore.crps\_gaussian
 ==========================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.crps_quadrature.rst b/docs/source/api/xskillscore.crps_quadrature.rst
@@ -1,4 +1,4 @@
-xskillscore.crps\_quadrature
+xskillscore.crps\_quadrature
 ============================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.effective_sample_size.rst b/docs/source/api/xskillscore.effective_sample_size.rst
@@ -1,4 +1,4 @@
-xskillscore.effective\_sample\_size
+xskillscore.effective\_sample\_size
 ===================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.mae.rst b/docs/source/api/xskillscore.mae.rst
@@ -1,4 +1,4 @@
-xskillscore.mae
+xskillscore.mae
 ===============
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.mape.rst b/docs/source/api/xskillscore.mape.rst
@@ -1,4 +1,4 @@
-xskillscore.mape
+xskillscore.mape
 ================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.median_absolute_error.rst b/docs/source/api/xskillscore.median_absolute_error.rst
@@ -1,4 +1,4 @@
-xskillscore.median\_absolute\_error
+xskillscore.median\_absolute\_error
 ===================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.mse.rst b/docs/source/api/xskillscore.mse.rst
@@ -1,4 +1,4 @@
-xskillscore.mse
+xskillscore.mse
 ===============
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.pearson_r.rst b/docs/source/api/xskillscore.pearson_r.rst
@@ -1,4 +1,4 @@
-xskillscore.pearson\_r
+xskillscore.pearson\_r
 ======================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.pearson_r_eff_p_value.rst b/docs/source/api/xskillscore.pearson_r_eff_p_value.rst
@@ -1,4 +1,4 @@
-xskillscore.pearson\_r\_eff\_p\_value
+xskillscore.pearson\_r\_eff\_p\_value
 =====================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.pearson_r_p_value.rst b/docs/source/api/xskillscore.pearson_r_p_value.rst
@@ -1,4 +1,4 @@
-xskillscore.pearson\_r\_p\_value
+xskillscore.pearson\_r\_p\_value
 ================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.r2.rst b/docs/source/api/xskillscore.r2.rst
@@ -1,4 +1,4 @@
-xskillscore.r2
+xskillscore.r2
 ==============
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.rank_histogram.rst b/docs/source/api/xskillscore.rank_histogram.rst
@@ -1,4 +1,4 @@
-xskillscore.rank\_histogram
+xskillscore.rank\_histogram
 ===========================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.reliability.rst b/docs/source/api/xskillscore.reliability.rst
@@ -0,0 +1,6 @@
+xskillscore.reliability
+=======================
+
+.. currentmodule:: xskillscore
+
+.. autofunction:: reliability
diff --git a/docs/source/api/xskillscore.rmse.rst b/docs/source/api/xskillscore.rmse.rst
@@ -1,4 +1,4 @@
-xskillscore.rmse
+xskillscore.rmse
 ================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.smape.rst b/docs/source/api/xskillscore.smape.rst
@@ -1,4 +1,4 @@
-xskillscore.smape
+xskillscore.smape
 =================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.spearman_r.rst b/docs/source/api/xskillscore.spearman_r.rst
@@ -1,4 +1,4 @@
-xskillscore.spearman\_r
+xskillscore.spearman\_r
 =======================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.spearman_r_eff_p_value.rst b/docs/source/api/xskillscore.spearman_r_eff_p_value.rst
@@ -1,4 +1,4 @@
-xskillscore.spearman\_r\_eff\_p\_value
+xskillscore.spearman\_r\_eff\_p\_value
 ======================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.spearman_r_p_value.rst b/docs/source/api/xskillscore.spearman_r_p_value.rst
@@ -1,4 +1,4 @@
-xskillscore.spearman\_r\_p\_value
+xskillscore.spearman\_r\_p\_value
 =================================
 
 .. currentmodule:: xskillscore

diff --git a/docs/source/api/xskillscore.threshold_brier_score.rst b/docs/source/api/xskillscore.threshold_brier_score.rst
@@ -1,4 +1,4 @@
-xskillscore.threshold\_brier\_score
+xskillscore.threshold\_brier\_score
 ===================================
 
 .. currentmodule:: xskillscore