docs/source/quick-start.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quick Start\n",
    "\n",
    "See the API for more detailed information, examples, formulas, and references for each function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import xarray as xr\n",
    "import xskillscore as xs\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we generate some sample gridded data. Our data has three time steps, and a 4x5 latitude/longitude grid. `obs` replicates some verification data and `fct` some forecast (e.g. from a statistical or dynamical model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs = xr.DataArray(\n",
    "       np.random.rand(3, 4, 5),\n",
    "       coords=[\n",
    "           xr.cftime_range(\"2000-01-01\", \"2000-01-03\", freq=\"D\"),\n",
    "           np.arange(4),\n",
    "           np.arange(5),\n",
    "       ],\n",
    "       dims=[\"time\", \"lat\", \"lon\"],\n",
    "       name='var'\n",
    "   )\n",
    "fct = obs.copy()\n",
    "fct.values = np.random.rand(3, 4, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deterministic Metrics\n",
    "\n",
    "`xskillscore` offers a suite of correlation-based and distance-based deterministic metrics.\n",
    "\n",
    "### Correlation-Based \n",
    "\n",
    "* Pearson Correlation (`pearson_r`)\n",
    "* Pearson Correlation p value (`pearson_r_p_value`)\n",
    "* Pearson Correlation effective p value (`pearson_r_eff_p_value`)\n",
    "* Spearman Correlation (`spearman_r`)\n",
    "* Spearman Correlation p value (`spearman_r_p_value`)\n",
    "* Spearman Correlation effective p value (`spearman_r_eff_p_value`)\n",
    "* Effective Sample Size (`effective_sample_size`)\n",
    "* Coefficient of Determination (`r2`)\n",
    "\n",
    "### Distance-Based\n",
    "\n",
    "* Mean Error (`me`)\n",
    "* Root Mean Squared Error (`rmse`)\n",
    "* Mean Squared Error (`mse`)\n",
    "* Mean Absolute Error (`mae`)\n",
    "* Median Absolute Error (`median_absolute_error`)\n",
    "* Symmetric Mean Absolute Percentage Error (`smape`)\n",
    "* Mean Absolute Percentage Error (`mape`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling the functions is very straight-forward. All deterministic functions take the form `func(a, b, dim=None, **kwargs)`. **Notice that the original dataset is reduced by the dimension passed.** I.e., since we passed `time` as the dimension here, we are returned an object with dimensions `(lat, lon)`. For correlation metrics `dim` cannot be `[]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (lat: 4, lon: 5)>\n",
      "array([[ 0.99509676, -0.88499394,  0.94083077,  0.96521259, -0.13696899],\n",
      "       [-0.90613709,  0.51585291,  0.72875703,  0.19331043,  0.79754067],\n",
      "       [-0.80112059, -0.95632624, -0.23640403, -0.57684283,  0.43389289],\n",
      "       [ 0.00230351, -0.58970109, -0.87332763, -0.99992557, -0.31404248]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "r = xs.pearson_r(obs, fct, dim='time')\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (lat: 4, lon: 5)>\n",
      "array([[0.06306879, 0.30832471, 0.22009394, 0.1684121 , 0.91252786],\n",
      "       [0.2780348 , 0.6549502 , 0.48019675, 0.87615511, 0.41226788],\n",
      "       [0.40847506, 0.1888421 , 0.84806222, 0.60856901, 0.71427925],\n",
      "       [0.99853354, 0.59849112, 0.32391484, 0.00776728, 0.79663312]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "p = xs.pearson_r_p_value(obs, fct, dim=\"time\")\n",
    "print(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify multiple axes for deterministic metrics. Here, we apply it over the latitude and longitude dimension (a pattern correlation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (time: 3)>\n",
      "array([-0.16920304, -0.06326809,  0.18040449])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "r = xs.pearson_r(obs, fct, dim=[\"lat\", \"lon\"])\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All deterministic metrics except for `pearson_r_eff_p_value`, `spearman_r_eff_p_value`, and `effective_sample_size` can take the kwarg `weights=...`. `weights` should be a DataArray of the size of the reduced dimension (e.g., if time is being reduced it should be of length 3 in our example).\n",
    "\n",
    "Weighting is a common practice when working with observations and model simulations of the Earth system. When working with rectilinear grids, one can weight the data by the cosine of the latitude, which is maximum at the equator and minimum at the poles (as in the below example). More complicated model grids tend to be accompanied by a cell area varaible, which could also be passed into this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs2 = xr.DataArray(\n",
    "       np.random.rand(3, 180, 360),\n",
    "       coords=[\n",
    "           xr.cftime_range(\"2000-01-01\", \"2000-01-03\", freq=\"D\"),\n",
    "           np.linspace(-89.5, 89.5, 180),\n",
    "           np.linspace(-179.5, 179.5, 360),\n",
    "       ],\n",
    "       dims=[\"time\", \"lat\", \"lon\"],\n",
    "    )\n",
    "fct2 = obs2.copy()\n",
    "fct2.values = np.random.rand(3, 180, 360)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make weights as cosine of the latitude and broadcast\n",
    "weights = np.cos(np.deg2rad(obs2.lat))\n",
    "_, weights = xr.broadcast(obs2, weights)\n",
    "\n",
    "# Remove the time dimension from weights\n",
    "weights = weights.isel(time=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray (time: 3)>\n",
      "array([-0.0020303 , -0.00498588, -0.00401522])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "r_weighted = xs.pearson_r(obs2, fct2, dim=[\"lat\", \"lon\"], weights=weights)\n",
    "print(r_weighted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray (time: 3)>\n",
      "array([ 5.72646719e-05, -4.32380560e-03,  4.17909845e-05])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "r_unweighted = xs.pearson_r(obs2, fct2, dim=[\"lat\", \"lon\"], weights=None)\n",
    "print(r_unweighted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also pass the optional boolean kwarg `skipna=...`. If `True`, ignore any NaNs (pairwise) in `a` and `b` when computing the result. If `False`, return NaNs anywhere there are pairwise NaNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (time: 3, lat: 4, lon: 5)>\n",
      "array([[[       nan,        nan,        nan,        nan,        nan],\n",
      "        [       nan,        nan,        nan,        nan,        nan],\n",
      "        [0.02058449, 0.96990985, 0.83244264, 0.21233911, 0.18182497],\n",
      "        [0.18340451, 0.30424224, 0.52475643, 0.43194502, 0.29122914]],\n",
      "\n",
      "       [[       nan,        nan,        nan,        nan,        nan],\n",
      "        [       nan,        nan,        nan,        nan,        nan],\n",
      "        [0.60754485, 0.17052412, 0.06505159, 0.94888554, 0.96563203],\n",
      "        [0.80839735, 0.30461377, 0.09767211, 0.68423303, 0.44015249]],\n",
      "\n",
      "       [[       nan,        nan,        nan,        nan,        nan],\n",
      "        [       nan,        nan,        nan,        nan,        nan],\n",
      "        [0.96958463, 0.77513282, 0.93949894, 0.89482735, 0.59789998],\n",
      "        [0.92187424, 0.0884925 , 0.19598286, 0.04522729, 0.32533033]]])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "obs_with_nans = obs.where(obs.lat > 1)\n",
    "fct_with_nans = fct.where(fct.lat > 1)\n",
    "print(obs_with_nans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (time: 3)>\n",
      "array([0.51901116, 0.41623426, 0.32621064])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "mae_with_skipna = xs.mae(obs_with_nans, fct_with_nans, dim=['lat', 'lon'], skipna=True)\n",
    "print(mae_with_skipna)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (time: 3)>\n",
      "array([nan, nan, nan])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "mae_without_skipna = xs.mae(obs_with_nans, fct_with_nans, dim=['lat', 'lon'], skipna=False)\n",
    "print(mae_without_skipna)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probabilistic Metrics\n",
    "\n",
    "`xskillscore` offers a suite of probabilistic metrics.\n",
    "\n",
    "* Continuous Ranked Probability Score with the ensemble distribution (`crps_ensemble`)\n",
    "* Continuous Ranked Probability Score with a Gaussian distribution (`crps_gaussian`)\n",
    "* Continuous Ranked Probability Score with numerical integration of the normal distribution (`crps_quadrature`)\n",
    "* Brier scores of an ensemble for exceeding given thresholds (`threshold_brier_score`)\n",
    "* Brier Score (`brier_score`)\n",
    "* Ranked Probability Score (`rps`)\n",
    "* Discrimination (`discrimination`)\n",
    "* Rank Histogram (`rank_histogram`)\n",
    "* Reliability (`reliability`)\n",
    "* Receiver operating characteristic (ROC) (`roc`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create some data with an ensemble member dimension. In this case, we envision an ensemble forecast with multiple members to validate against our theoretical observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs3 = xr.DataArray(\n",
    "       np.random.rand(4, 5),\n",
    "       coords=[np.arange(4), np.arange(5)],\n",
    "       dims=[\"lat\", \"lon\"],\n",
    "       name='var'\n",
    "   )\n",
    "fct3 = xr.DataArray(\n",
    "       np.random.rand(3, 4, 5),\n",
    "       coords=[np.arange(3), np.arange(4), np.arange(5)],\n",
    "       dims=[\"member\", \"lat\", \"lon\"],\n",
    "       name='var'\n",
    "   )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuous Ranked Probability Score with the ensemble distribution. Pass `dim=[]` to get the same behaviour as `properscoring.crps_ensemble` without any averaging over `dim`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (lat: 4, lon: 5)>\n",
      "array([[0.19918258, 0.10670612, 0.11858151, 0.15974459, 0.26841063],\n",
      "       [0.08038415, 0.13237479, 0.23778382, 0.18009214, 0.08326884],\n",
      "       [0.08589149, 0.11666573, 0.21579228, 0.09646599, 0.12855359],\n",
      "       [0.19891371, 0.10470738, 0.05289158, 0.107965  , 0.11143681]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "crps_ensemble = xs.crps_ensemble(obs3, fct3, dim=[])\n",
    "print(crps_ensemble)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CRPS with a Gaussian distribution requires two parameters: $\\mu$ and $\\sigma$ from the forecast distribution. Here, we just use the ensemble mean and ensemble spread."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (lat: 4, lon: 5)>\n",
      "array([[0.19821619, 0.11640329, 0.14219455, 0.15912935, 0.28104703],\n",
      "       [0.08953392, 0.11758925, 0.25156378, 0.095484  , 0.10679842],\n",
      "       [0.05069082, 0.07081479, 0.24529056, 0.08700853, 0.09535839],\n",
      "       [0.1931706 , 0.11233935, 0.0783092 , 0.09593862, 0.11037143]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "crps_gaussian = xs.crps_gaussian(obs3, fct3.mean(\"member\"), fct3.std(\"member\"), dim=[])\n",
    "print(crps_gaussian)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CRPS quadrature metric requires a callable distribution function. Here we use `norm` from `scipy.stats`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray (lat: 4, lon: 5)>\n",
      "array([[0.52852898, 0.58042038, 0.46945497, 0.25013942, 0.23370234],\n",
      "       [0.39109762, 0.24071855, 0.25557803, 0.28994381, 0.23764056],\n",
      "       [0.40236669, 0.33477031, 0.24063375, 0.45538915, 0.48236113],\n",
      "       [0.42011508, 0.4174865 , 0.24837346, 0.43954946, 0.44689198]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "from scipy.stats import norm\n",
    "crps_quadrature = xs.crps_quadrature(obs3, norm, dim=[])\n",
    "print(crps_quadrature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use a threshold Brier Score, to score hits over a certain threshold. Ranked Probability Score for two categories yields the same result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' ()>\n",
      "array(0.15555556)\n",
      "Coordinates:\n",
      "    threshold  float64 0.5\n"
     ]
    }
   ],
   "source": [
    "threshold_brier_score = xs.threshold_brier_score(obs3, fct3, 0.5, dim=None)\n",
    "print(threshold_brier_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' ()>\n",
      "array(0.15555556)\n"
     ]
    }
   ],
   "source": [
    "brier_score = xs.brier_score(obs3>.5, (fct3>.5).mean('member'))\n",
    "print(brier_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray ()>\n",
      "array(0.15555556)\n"
     ]
    }
   ],
   "source": [
    "rps = xs.rps(obs3>.5, fct3>.5, category_edges=np.array([0.5]))\n",
    "print(rps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_rank' (rank: 4)>\n",
      "array([3, 8, 6, 3])\n",
      "Coordinates:\n",
      "  * rank     (rank) float64 1.0 2.0 3.0 4.0\n"
     ]
    }
   ],
   "source": [
    "rank_histogram = xs.rank_histogram(obs3, fct3)\n",
    "print(rank_histogram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray (event: 2, forecast_probability: 5)>\n",
      "array([[0.        , 0.08333333, 0.        , 0.66666667, 0.25      ],\n",
      "       [0.125     , 0.5       , 0.        , 0.375     , 0.        ]])\n",
      "Coordinates:\n",
      "  * forecast_probability  (forecast_probability) float64 0.1 0.3 0.5 0.7 0.9\n",
      "  * event                 (event) bool True False\n"
     ]
    }
   ],
   "source": [
    "disc = xs.discrimination(obs3 > 0.5, (fct3 > 0.5).mean(\"member\"))\n",
    "print(disc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' (forecast_probability: 5)>\n",
      "array([0.        , 0.2       ,        nan, 0.72727273, 1.        ])\n",
      "Coordinates:\n",
      "  * forecast_probability  (forecast_probability) float64 0.1 0.3 0.5 0.7 0.9\n",
      "    samples               (forecast_probability) float64 1.0 5.0 0.0 11.0 3.0\n"
     ]
    }
   ],
   "source": [
    "rel = xs.reliability(obs3 > 0.5, (fct3 > 0.5).mean(\"member\"))\n",
    "print(rel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8229166666666666"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARIAAAEGCAYAAACpcBquAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkBUlEQVR4nO3deXgV5fn/8fedjSYsCRCQfVcUBESigloBrYItgnJ5uVTUtlLKFy12+aFSCkX9WqFYWyko5qetWncLRcQFK4r6U1IIhk0Wq6iQIA2IIIQIJLl/f8ygh5hlknPmzFnu13WdK3PmzDnzSSB3nueZmWdEVTHGmHCkBB3AGBP/rJAYY8JmhcQYEzYrJMaYsFkhMcaELS3oAA2Vm5ur3bp1CzqGMUlnzZo1e1S1TU2vxV0h6datG4WFhUHHMCbpiMintb1mXRtjTNiskBhjwmaFxBgTNiskxpiwWSExxoTNt6M2IvJXYBRQqqqn1vC6APcB3wcOAT9S1ff8ymNMPFlcVMKcZVvZua+cDjmZTBnRm0sHdozZffjZInkEGFnH6xcDJ7qPCcADPmYxJm4sLiph6qINlOwrR4GSfeVMXbSBxUUlMbsP3wqJqr4F7K1jkzHAY+ooAHJEpL1feYyJF3OWbaX8aOVx68qPVjJn2daI76Oy7AuOTSUSzj6CHCPpCOwIeV7srvsWEZkgIoUiUrh79+6ohDMmKDv3lTdofWP3oarsXnQXX65aFPY+giwkUsO6GmdZUtV8Vc1T1bw2bWo8Q9eYhNEhJ7NB6xu7DxEhd/QtZJ04OOx9BFlIioHOIc87ATsDymJMzJgyojeZ6anHrctMT2XKiN4R+fxVq1bRdfvLfCcthbTstqS36hj2PoIsJEuA68QxGNivqp8FmMeYmHDpwI7cPbYfHXMyEaBjTiZ3j+0XsaM2zzzzDP9etojpF3WN2D7ErzlbReQpYBiQC/wX+B2QDqCqC9zDv/NwjuwcAn6sqvVejZeXl6d20Z4xDaeqiAhVVVXs3buX3NzcBr1fRNaoal5Nr/l2HomqXl3P6wrc6Nf+jTHfWL58OdOmTWPp0qXk5uY2uIjUx85sNSZJqCpVVVW+fLYVEmMS2LHTJS644AIKCgpo27atL/uxQmJMgnrttdfo1q0bb7zxBgDOsKQ/rJAYk6DOOOMMrr/+ek4//XTf92WFxJgEs2LFCioqKsjOzub+++8nOzvb931aITEmgWzcuJHzzz+fP/3pT1Hdb9xN/myMqd2pp57Kk08+yZgxY6K6X2uRGJMA8vPz+eCDDwC46qqryMyM3HU5XlghSXKLi0o4Z9brdL/tRc6Z9XpE57ww0bF3716mT5/OfffdF1gG69oksWOT2xyb++LY5DZAxGfjMv5p1aoVK1eupHPnzvVv7BNrkSSxaEygY/yhqsyYMYN58+YB0KNHD9LT0wPLYy2SJBaNCXSMP6qqqtiwYQO5ublfX4wXJCskSaxDTiYlNRSNSE6gYyJLVfnqq6/IzMzkmWeeIS0tLfAiAta1SWp+T6BjIu8Xv/gFI0eO5PDhw2RkZJCSEhu/wtYiSWLHBlT9vu2BiZwhQ4aQlZVFRkZG0FGO49vERn6xiY1MsqmsrGTr1q306dMn0Bx1TWwUG+0iY0ytpk2bxllnnUVJSeye42NdG2Ni3OTJk+nRowcdO8Zul9NaJMbEoMOHD/Pwww+jqnTo0IEJEyYEHalOVkiMiUGPPfYY48ePp6CgIOgonljXxpgYNH78ePr06cOQIUOCjuKJtUiMiREHDhzghhtuYNeuXYgI55xzTtCRPLNCYkyM+OCDD1i4cCGrV68OOkqDWdfGmIBVVlaSmprKoEGD+Pjjj2nZsmXQkRrMWiTGBOjzzz9nyJAh/OMf/wCIyyICVkiMCVSTJk3Izs6mefPmQUcJi3VtjAlAaWkpLVu2pFmzZrz66qsxcQVvOKxFYkyUHTp0iHPPPffrk8zivYiAtUiMibqsrCwmT57MoEGDgo4SMVZIjImSjz76iMOHD9OnTx9uuummoONElBUSY6JAVbn66qspLy9n3bp1MTMhUaT4WkhEZCRwH5AKPKSqs6q9ng08DnRxs9yjqn/zM5MxQRARHnvsMY4ePZpwRQR8HGwVkVRgPnAx0Ae4WkSqz8xyI7BJVQcAw4A/ikhsTf1kTBjWr1//9UzvJ598Mv369Qs4kT/8LI1nAh+q6jZVPQI8DVS/j6ACzcUZtm4G7AUqfMxkTFQ98MADzJ49m/379wcdxVd+FpKOwI6Q58XuulDzgFOAncAG4GZVrar+QSIyQUQKRaRw9+7dfuU1JuLmzp1LQUEB2dnZQUfxlZ+FpKaD49UniB0BrAU6AKcB80SkxbfepJqvqnmqmtemTZtI5zQmot555x1GjBjBl19+SXp6ekzPbBYpfhaSYiD0HoKdcFoeoX4MLFLHh8DHwMk+ZjLGd3v27KGkpISDBw8GHSVq/Cwkq4ETRaS7O4B6FbCk2jbbgQsAROQEoDewzcdMxvjmyy+/BGDMmDGsXbuWDh06BJwoenwrJKpaAdwELAM2A8+q6vsiMlFEJrqb3QmcLSIbgOXAraq6x69MxvjljTfeoHv37qxcuRKAtLTkOkXL1+9WVV8CXqq2bkHI8k7gIj8zGBMNffv2ZeTIkZx00klBRwlE4p0ZY0wUFRYWoqq0bduWJ554gtatWwcdKRBWSIxppHXr1jF48GDmzp0bdJTAJVdHzpgI6t+/P/PmzWPcuHFBRwmctUiMaaAnn3yS4uJiRISJEyfSrFmzoCMFzgqJMQ2wZ88eJk2axO9///ugo8QU69oY0wC5ubm8/fbbSXt0pjbWIjHGgz/84Q88/vjjAPTr148mTZoEnCi2WCExph4VFRW88sorLFu2LOgoMcu6NsbUQlWpqKggPT2dpUuXWiukDtYiMaYWt912G2PHjuXo0aNkZWWRmpoadKSYZS0SY2rRvXt3Dh06ZAXEAyskxoSoqqpix44ddO3alYkTJ9b/BgN46NqIY5yIzHCfdxGRM/2PZkz0TZs2jUGDBrFr166go8QVLy2S+4Eq4HzgDuAAsBA4w8dcxgTihhtuIDc3lxNOOCHoKHHFy2DrWap6I/AVgKp+AdhM7yZhHD16lIULFwLQq1cvfv3rXyfEbTSjyUshOereWkIBRKQNTgvFmISQn5/P5ZdfzqpVq4KOEre8dG3mAv8E2orIXcDlwHRfUxkTRRMnTqRnz56ceaYN/TVWvS0SVX0CuAW4G/gMuFRVn/U7mDF+OnToEL/85S/Zt28fqampjBw5MuhIcc3LUZu/q+oWVZ2vqvNUdbOI/D0a4YzxS1FREQsWLODNN98MOkpC8NK16Rv6xB0vGeRPHGP8paqICOeccw7btm2jffv2QUdKCLW2SERkqogcAPqLyJcicsB9Xgo8H7WExkTI/v37Of/883n11VcBrIhEUK2FRFXvVtXmwBxVbaGqzd1Ha1WdGsWMxkREZWUlZWVllJWVBR0l4dTbtVHVqSLSEjgR+E7I+rf8DGZMpOzbt4/mzZvTqlUrCgoKSEmxa1Ujzctg63jgLZwbXd3ufp3pbyxjIqOsrIzvfve73HzzzQBWRHzi5ad6M87p8J+q6nBgILDb11TGREjTpk258sorGTt2bNBREpqXozZfqepXIoKINFHVLSLS2/dkxoRhx44dVFRU0L17d377298GHSfheSkkxSKSAywG/iUiXwA7/QxlTDhUlcsuu4yKigree+89685EgZfB1svcxZki8gaQDbziaypjwiAiPPjgg6iqFZEoqbOQiEgKsF5VTwVQVTsN0MSsLVu2sHr1aq699loGDbJzJqOpznKtqlXAOhHpEqU8xjTa3XffzS233MKBAweCjpJ0vIyRtAfeF5FVwNdn8qjq6PreKCIjgfuAVOAhVZ1VwzbDgD8D6cAeVR3qJbgx1S1YsICdO3fSvHnzoKMkHS+F5PbGfLB7Tc584EKgGFgtIktUdVPINjk4M7CNVNXtItK2MfsyyauwsJA5c+bw6KOPkpmZSc+ePYOOlJS8DLY2dlzkTOBDVd0GICJPA2OATSHb/BBYpKrb3X2VNnJfJklt2bKFwsJC9uzZQ6dOnYKOk7T8HNLuCOwIeV7srgt1EtBSRFaIyBoRua6mDxKRCSJSKCKFu3fbuXAGDh8+DMC4cePYuHGjFZGA+VlIapr0Uqs9T8OZkuAHwAhguoh86+7MqpqvqnmqmtemTZvIJzVx5e2336Znz56sXbsWgMzMzGADGW+FREQyG3E2azHQOeR5J759Ilsx8IqqlqnqHpxregY0cD8myXTt2pUBAwbQrl27oKMYl5eL9i4B1uKehCYip4nIEg+fvRo4UUS6i0gGcBVQ/X3PA98VkTQRyQLOAjY3IL9JIlu2bEFV6dKlCy+++KIVkhjipUUyE2fgdB+Aqq4FutX3JlWtAG7CuVp4M/Csqr4vIhNFZKK7zWacArUeWIVziHhjQ78Jk/iKioro378/+fn5QUcxNfBy+LdCVfc35j4fqvoS8FK1dQuqPZ8DzGnwh5ukMmDAAO644w6uvPLKoKOYGnhpkWwUkR8CqSJyooj8BXjX51zGAPD888+zZ88eUlJSuO2228jJyQk6kqmBl0Lyc5wJoA8DTwL7gV/4mMkYAEpLS7nmmmuYOXNm0FFMPbx0bXqr6jRgmt9hjAnVtm1bXnvtNfr37x90FFMPLy2Se0Vki4jcKSJ969/cmPDMnz+fpUuXAjB48GCysrICTmTq4+VOe8OBYTjTK+aLyAYRsSmnjC+OHj3Ko48+yuOPPx50FNMAolr9ZNM6Nhbph3P7zitVNcO3VHXIy8vTwsLCIHZtfFZVVUVKSgr79u2jadOmpKenBx3JhBCRNaqaV9NrXk5IO0VEZorIRmAezhEbu7DBRNTtt9/OddddR2VlJTk5OVZE4oyXwda/AU8BF6mqzdVqfJGenk5GRiCNXBMBDeraxALr2iQOVaW0tJQTTjjh6+eNOfHRREejujYi8qz7dYOIrA95bBCR9X6FNcljxowZDBo0iNJSZxoaKyLxq66uzc3u11HRCGKSzxVXXEFKSgo2NUT8q+sm4p+5i5NU9dPQBzApOvFMoqmsrOS1114DoF+/ftx+++3WEkkAXk5Iu7CGdRdHOohJDvPnz+fCCy9kzZo1QUcxEVRr10ZE/gen5dGj2phIc+Adv4OZxPSzn/2Mdu3a2X1nEkxdLZIngUtwJiO6JOQxSFXHRSGbSRCHDx/md7/7HWVlZTRp0oQrrrgi6EgmwuoqJKqqnwA3AgdCHohIK/+jmUTx7rvvctddd/Hqq68GHcX4pK6jNk/iHLFZgzNpc+iImAI9fMxlEsjw4cPZsmULvXr1CjqK8UldR21GuV+7q2oP9+uxhxURU6eDBw8yZswYVq5cCWBFJMF5udbmHBFp6i6PE5F77V7Apj4HDhzggw8+4NNPPw06iokCL9faPAAMEJEBOFf+Pgz8HbB79JpvKSsrIysri/bt27Nu3Tq7fiZJeDmPpEKdC3LGAPep6n04h4CNOc7Bgwc577zzmDbNmUzPikjy8NIiOSAiU4Frce5BkwrYNd7mW5o2bcrQoUM599xzg45iosxLIbkS52bfP1HVXe74iN0+wnxt165dALRr145777034DQmCF6mWtwFPAFki8go4CtVfcz3ZCYuVFVVMWrUKMaMGUO8TUlhIqfeFomIXIHTAlmBcy7JX0Rkiqr+w+dsJg6kpKRwzz33kJaWZhffJTEvXZtpwBmqWgogIm2A1wArJEls27ZtbNy4kdGjRzNs2LCg45iAeTlqk3KsiLg+9/g+k8B+85vfMGHCBMrKyoKOYmKAlxbJKyKyDGfeVnAGX1+qY3uTBPLz8ykuLqZp06ZBRzExwMtg6xTgQaA/MADIV9Vb/Q5mYs/GjRuZOHEiFRUVtGjRgj59+gQdycQIr12Ud4E3gdeBlf7FMbHsrbfe4oUXXmDnTruZgDmel2ttxgOrgMuAy4ECEfmJ38FM7KisrARg0qRJvP/++3TpYpdameN5aZFMAQaq6o9U9XpgEOCpayMiI0Vkq4h8KCK31bHdGSJSKSKXe4ttoqWgoIC+ffuydetWAHJycoINZGKSl0JSjDuhkesAsKO+N7mn0s/Hmd+1D3C1iHyrU+1uNxtY5iWwia7s7Gxyc3PtRt6mTl6O2pQA/xaR53EmNBoDrBKRXwGoam3nRJ8JfKiq2wBE5Gn3vZuqbfdzYCFwRsPjG7/s2LGDzp07c8opp/D222/byWamTl5aJB8Bi3GKCMDzwGc4VwDXdRVwR45vuRS7674mIh1xxl4W1BVARCaISKGIFO7evdtDZBOOoqIievfuzWOPOVdCWBEx9am3RaKqtzfys2v631f9Yow/A7eqamVd/1lVNR/IB+eWnY3MYzw69dRTmTx5MhdfbHcdMd546do0VjHQOeR5J6D6ccM84Gm3iOQC3xeRClVd7GMuU4vly5dzxhln0KJFC2bNmhV0HBNH/DzVfTVwooh0F5EM4CqcW1t8zZ3/tZuqdsO5dmeSFZFg7Nq1i1GjRvHb3/426CgmDvnWIlHVChG5CedoTCrwV1V9X0Qmuq/XOS5ioqtdu3YsWbKEwYMHBx3FxCGpbw4JETkJZ97WE1T1VBHpD4xW1f+NRsDq8vLytLCwMIhdJ6RHHnmE7t27M3SoTcFr6iYia1Q1r6bXvHRt/i8wFTgKoKrrcbopJs4dOXKEe+65h7/85S9BRzFxzkvXJktVV1U7qlLhUx4TJapKRkYGr7/+Oi1atAg6jolzXloke0SkJ+6hW/c09s98TWV8dc899zB58mRUlbZt2/Kd73wn6EgmznlpkdyIcw7HySJSAnwM2E3E49iuXbsoLS2lsrKStDQ/zwAwyaLewdavN3Tutpeiqgfq3dhHNtjaOKrKl19+SXZ2NqpKVVUVqampQccycaSuwVYvkz/PqPYcAFW9IyLpTFTceeedPP7446xcuZLWrVtbETER5aVdGzop53eAUcBmf+IYv1x44YXs37+fli1bBh3FJCDPXZuv3yDSBFiiqiP8iVQ369p4V1VVxerVqznrrLOCjmISQLjnkVSXBfQIL5KJhrlz53L22Wezdu3aoKOYBOdljGQD31y1mwq0AWx8JA789Kc/pVmzZgwYMCDoKCbBeRkjGRWyXAH8V1XthLQYVVFRwdy5c7npppto2rQp48ePDzqSSQJ1dm1EJAV4UVU/dR8lVkRi2/Lly/n1r3/Niy++GHQUk0TqbJGoapWIrBORLqq6PVqhTOONGDGCoqIiTjvttKCjmCTiZbC1PfC+iCwXkSXHHn4HM96Vl5czbtw4NmzYAGBFxESdlzGSxk61aKKktLSUt956i4suuoh+/foFHcckIS+F5PvVb9EpIrNx7rxnAnTkyBEyMjLo2rUrmzdvtvvwmsB46dpcWMM6mxU4YAcPHmT48OHMnj0bwIqICVStLRIR+R9gEtBDRNaHvNQceMfvYKZumZmZnHTSSfTq1SvoKMbU2bV5EngZuBsIvd3mAVXd62sqU6vPP/+clJQUWrZsyd/+9reg4xgD1FFIVHU/sB+4OnpxTF2qqqoYOXIkWVlZrFixwm5cZWJGws1qs7iohDnLtrJzXzkdcjKZMqI3lw7sWP8b40BKSgrTp08nKyvLioiJKQlVSBYXlTB10QbKj1YCULKvnKmLnHMr4rmYlJSU8J///Idhw4YxevTooOMY8y1+3iAr6uYs2/p1ETmm/Gglc5ZtDShRZEyePJmrrrqKQ4cOBR3FmBolVItk577yBq2PFwsWLGD79u1kZWUFHcWYGiVUi6RDTmaD1sey//znP0ydOpWqqiratGnDoEGDgo5kTK0SqpBMGdGbzPTj5yLNTE9lyojeASVqvH/+85889NBDFBcXBx3FmHo1eKrFoNU31WK8H7VRVUQEVeW///0v7dq1CzqSMUCYs8jHm0sHdoyrwhGqqKiICRMmsHDhQrp06WJFxMSNhOraxLuqqiqOHDnC0aNHg45iTIMkXIskHu3Zs4fc3FwGDRpEUVERKSlW30188fV/rIiMFJGtIvKhiNxWw+vXiMh69/GuiCTdLMVFRUX07NmT5557DsCKiIlLvv2vFZFUYD7OlAN9gKtFpE+1zT4Ghqpqf+BOnHsMJ5WTTz6ZH/7wh5x77rlBRzGm0fz883cm8KGqblPVI8DTwJjQDVT1XVX9wn1aAHTyMU9MWblyJV999RWZmZk88MADtG/fPuhIxjSan4WkI7Aj5Hmxu642N+BMW/AtIjJBRApFpHD37t0RjBiMnTt3cv755zNt2rSgoxgTEX4OttZ0eWqNJ62IyHCcQlJj+15V83G7PXl5efF14ksNOnTowBNPPMHw4cODjmJMRPjZIikGOoc87wTsrL6RiPQHHgLGqOrnPuYJ3LPPPsuaNWsAGDt2rN3Q2yQMPwvJauBEEekuIhnAVcBxt7EQkS7AIuBaVf3AxyyBO3z4MFOnTuWuu+4KOooxEedb10ZVK0TkJmAZzj2D/6qq74vIRPf1BcAMoDVwvztRT0Vtp+DGuyZNmvDGG2/QunXroKMYE3EJd61NrLn//vspLS1l5syZQUcxJixJda1NLFFV3nvvPUpLS6moqCAtzX7cJjHZ/2yflJeXk5mZyYMPPkhlZaUVEZPQ7HxsH8yaNYuzzz6b/fv3k5qaSkZGRtCRjPGV/Zn0wemnn862bdto1qxZ0FGMiQprkUSIqrJp0yYALrroIvLz80lNTa3nXcYkBiskEXLfffcxcOBANm7cGHQUY6LOujYRcv3111NVVUXfvn2DjmJM1FmLJAyVlZU8/PDDVFZW0rJlS371q1/ZHfBMUrJCEoaXX36Z8ePHs2TJkvo3NiaBWdcmDKNGjeLNN9/kvPPOCzqKMYGyFkkDHTlyhEmTJvHRRx8BWBExBiskDbZ9+3aee+45VqxYEXQUY2KGdW08qqqqIiUlhV69erF161ZatWoVdCRjYoa1SDw4dOgQI0aM4MEHHwSwImJMNVZIPEhLSyMrK4usrKygoxgTk6xrU4f9+/eTlpZG06ZNWbx4sZ0jYkwtrEVSi8rKSkaOHMnll1/+9Y29jTE1sxZJLVJTU7nxxhvJzs62ImJMPayQVFNaWsr27dvJy8tj3LhxQccxJi5Y16aa8ePHM3r0aMrLy4OOYkzcsBZJNfPnz+fTTz8lMzMz6CjGxA1rkQCffPIJs2fPRlXp3Lmz3dDbmAayQgI88sgjzJo1i5KSkqCjGBOXrJAAM2bMoKioiE6dOgUdxZi4lLSFZNOmTXzve99j9+7dpKSk0K1bt6AjGRO3kraQ7N69m08++YS9e/cGHcWYuJd0R20OHjxIs2bNGDp0KJs3byY9PT3oSMbEvaRqkaxbt44ePXrw0ksvAVgRMSZCkqqQdO/enQsuuMBmejcmwpKikKxbt46KigpatGjBU089RdeuXYOOZExCSfhCsmPHDoYMGcL06dODjmJMwvJ1sFVERgL3AanAQ6o6q9rr4r7+feAQ8CNVfS+cfS4uKmHOsq3s3FdOh5xMpozozbx587jkkkvC+VhjTB18a5GISCowH7gY6ANcLSJ9qm12MXCi+5gAPBDOPhcXlTB10QZK9pVT9uEqPvnoA6Yu2kCrgSNo06ZNOB9tjKmDn12bM4EPVXWbqh4BngbGVNtmDPCYOgqAHBFp39gdzlm2lfKjlVQdPczeZfP54s1HKT9ayZxlWxv/XRhj6uVn16YjsCPkeTFwlodtOgKfhW4kIhNwWix06dKl1h3u3Odc+p+S3oQTrrqL1GatjltvjPGHny2SmqYV00Zsg6rmq2qequbV1UXpkPPNpf/prTuR0iTrW+uNMZHnZyEpBjqHPO8E7GzENp5NGdGbzPTU49ZlpqcyZUTvxn6kMcYDPwvJauBEEekuIhnAVUD1u20vAa4Tx2Bgv6p+Vv2DvLp0YEfuHtuPjjmZCNAxJ5O7x/bj0oEdG/1NGGPq59sYiapWiMhNwDKcw79/VdX3RWSi+/oC4CWcQ78f4hz+/XG4+710YEcrHMZEma/nkajqSzjFInTdgpBlBW70M4Mxxn8Jf2arMcZ/VkiMMWGzQmKMCZsVEmNM2MQZ74wfIrIb+NTDprnAHp/jhMsyhi/W80HsZ/Sar6uq1nhGaNwVEq9EpFBV84LOURfLGL5YzwexnzES+axrY4wJmxUSY0zYErmQ5AcdwAPLGL5YzwexnzHsfAk7RmKMiZ5EbpEYY6LECokxJmxxX0hEZKSIbBWRD0XkthpeFxGZ676+XkROj8GM17jZ1ovIuyIyIJbyhWx3hohUisjl0czn7rvejCIyTETWisj7IvJmLOUTkWwReUFE1rn5wr7SvYH5/ioipSKysZbXw/s9UdW4feBMT/AR0APIANYBfapt833gZZzZ2AYD/47BjGcDLd3li6OZ0Uu+kO1ex7ma+/IY/BnmAJuALu7ztjGW7zfAbHe5DbAXyIhixvOA04GNtbwe1u9JvLdIoj7BtB8ZVfVdVf3CfVqAM1NczORz/RxYCJRGMdsxXjL+EFikqtsBVDWaOb3kU6C5ewuWZjiFpCJaAVX1LXeftQnr9yTeC0ltk0c3dBs/NXT/N+D8ZYiWevOJSEfgMmABwfDyMzwJaCkiK0RkjYhcF7V03vLNA07BmUp0A3CzqlZFJ54nYf2e+DqxURREbIJpH3nev4gMxykk5/qaqNpua1hXPd+fgVtVtdL5gxp1XjKmAYOAC4BMYKWIFKjqB36Hw1u+EcBa4HygJ/AvEXlbVb/0OZtXYf2exHshifoE043gaf8i0h94CLhYVT+PUjbwli8PeNotIrnA90WkQlUXRyWh93/nPapaBpSJyFvAACAahcRLvh8Ds9QZkPhQRD4GTgZWRSGfF+H9nkRrsMenAaQ0YBvQnW8GufpW2+YHHD+ItCoGM3bBmbf27Fj8GVbb/hGiP9jq5Wd4CrDc3TYL2AicGkP5HgBmussnACVAbpR/jt2ofbA1rN+TuG6RaEATTPuQcQbQGrjf/atfoVG6WtRjvkB5yaiqm0XkFWA9UIVzr+kaD3UGkQ+4E3hERDbg/LLeqqpRm1pARJ4ChgG5IlIM/A5ID8kX1u+JnSJvjAlbvB+1McbEACskxpiwWSExxoTNCokxJmxWSIwxYbNCEudEZLKIbBaRJ+rYZpiILI1mrtqIyOhjV8eKyKUi0ifktTtE5HtRzDJMRM6O1v4SWVyfR2IAmIRzNuzHQQfxQlWXAEvcp5cCS3Gu2kVVZ0R6fyKSpqq1XRw3DDgIvBvp/SYba5HEMRFZgHPp+hIR+aWInOnOZ1Lkfu1dw3uGunN2rHW3a+6unyIiq925KG6vZX8HReSPIvKeiCwXkTbu+tNEpMB97z9FpKW7frKIbHLXP+2u+5GIzHNbAqOBOW6WniLyiIhcLiIXi8izIfsdJiIvuMsXichKN8NzItKshpwrROT37pwkN4vIJSLyb/f7fU1EThCRbsBE4Jfu/r8rIm1EZKH7c1gtIueE8++TVKJ5iq49fDnt+RPcU62BFkCau/w9YKG7PAxY6i6/AJzjLjfDaZVehDMBsOD8cVkKnFfDvhS4xl2eAcxzl9cDQ93lO4A/u8s7gSbuco779Uch73uEkNPtjz13M20HmrrrHwDG4Vzn81bI+luBGTXkXAHcH/K8Jd+cfDke+KO7PBP4PyHbPQmc6y53ATYH/e8bLw/r2iSWbOBRETkR55c+vYZt3gHudcdUFqlqsYhchFNMitxtmgEn4vzShqoCnnGXHwcWiUg2TpE4NiPZo8Bz7vJ64AkRWQws9vpNqHPK+SvAJSLyD5zrQG4BhgJ9gHfcSwkygJW1fMwzIcudgGfc+TUygNq6gd8D+oRc4dxCRJqr6gGv2ZOVFZLEcifwhqpe5jbdV1TfQFVniciLONdVFLiDmwLcraoPNnB/9V1f8QOcmblGA9NFpG8DPvsZ4EacyXhWq+oBd1Kgf6nq1R7eXxay/BfgXlVdIiLDcFoiNUkBhqhqeQNyGmyMJNFk41xVCk4X4ltEpKeqblDV2UAhzqXsy4CfHBtvEJGOItK2hren4HQ9wJmR7P+p6n7gCxH5rrv+WuBNEUkBOqvqGzitiRyclk6oA0DzWr6XFThTA/6Ub1oXBcA5ItLLzZklIifV8v5QoT+X6+vY/6vATceeiMhpHj7bYIUk0fwBuFtE3sG5CrUmvxCRjSKyDigHXlbVV3HGB1a6V6f+g5p/wcuAviKyBmeCnjvc9dfjDJquB05z16cCj7ufVwT8SVX3Vfu8p4Ep7iBoz9AXVLUSZ6zmYvcrqrobp0A+5e6rAKcQ1mcm8JyIvM3xN8t+Abjs2GArMBnIcweHN+EMxhoP7Opf45mIHFTVbx0lMcZaJMaYsFmLxBgTNmuRGGPCZoXEGBM2KyTGmLBZITHGhM0KiTEmbP8fEYuHxSgxtMgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC for probabilistic forecasts and bin_edges='continuous' default\n",
    "roc = xs.roc(obs3 > 0.5, (fct3 > 0.5).mean(\"member\"), return_results='all_as_metric_dim')\n",
    "\n",
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot([0, 1], [0, 1], 'k:')\n",
    "roc.to_dataset(dim='metric').plot.scatter(y='true positive rate', x='false positive rate')\n",
    "roc.sel(metric='area under curve').values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contingency-Based\n",
    "\n",
    "To work with contingency-based scoring, first instantiate a `Contingency` object by passing in your observations, forecast, and observation/forecast bin edges. See https://www.cawcr.gov.au/projects/verification/#Contingency_table for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "dichotomous_category_edges = np.array([0, 0.5, 1]) # \"dichotomous\" mean two-category\n",
    "dichotomous_contingency = xs.Contingency(obs, fct,\n",
    "                                        dichotomous_category_edges,\n",
    "                                        dichotomous_category_edges,\n",
    "                                        dim=['lat','lon'])\n",
    "dichotomous_contingency_table = dichotomous_contingency.table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
       "<defs>\n",
       "<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
       "<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "</symbol>\n",
       "<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
       "<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "</symbol>\n",
       "</defs>\n",
       "</svg>\n",
       "<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
       " *\n",
       " */\n",
       "\n",
       ":root {\n",
       "  --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
       "  --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
       "  --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
       "  --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
       "  --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
       "  --xr-background-color: var(--jp-layout-color0, white);\n",
       "  --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
       "  --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
       "}\n",
       "\n",
       "html[theme=dark],\n",
       "body.vscode-dark {\n",
       "  --xr-font-color0: rgba(255, 255, 255, 1);\n",
       "  --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
       "  --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
       "  --xr-border-color: #1F1F1F;\n",
       "  --xr-disabled-color: #515151;\n",
       "  --xr-background-color: #111111;\n",
       "  --xr-background-color-row-even: #111111;\n",
       "  --xr-background-color-row-odd: #313131;\n",
       "}\n",
       "\n",
       ".xr-wrap {\n",
       "  display: block;\n",
       "  min-width: 300px;\n",
       "  max-width: 700px;\n",
       "}\n",
       "\n",
       ".xr-text-repr-fallback {\n",
       "  /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-header {\n",
       "  padding-top: 6px;\n",
       "  padding-bottom: 6px;\n",
       "  margin-bottom: 4px;\n",
       "  border-bottom: solid 1px var(--xr-border-color);\n",
       "}\n",
       "\n",
       ".xr-header > div,\n",
       ".xr-header > ul {\n",
       "  display: inline;\n",
       "  margin-top: 0;\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-obj-type,\n",
       ".xr-array-name {\n",
       "  margin-left: 2px;\n",
       "  margin-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-obj-type {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-sections {\n",
       "  padding-left: 0 !important;\n",
       "  display: grid;\n",
       "  grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
       "}\n",
       "\n",
       ".xr-section-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-section-item input {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-item input + label {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label {\n",
       "  cursor: pointer;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label:hover {\n",
       "  color: var(--xr-font-color0);\n",
       "}\n",
       "\n",
       ".xr-section-summary {\n",
       "  grid-column: 1;\n",
       "  color: var(--xr-font-color2);\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".xr-section-summary > span {\n",
       "  display: inline-block;\n",
       "  padding-left: 0.5em;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in + label:before {\n",
       "  display: inline-block;\n",
       "  content: '►';\n",
       "  font-size: 11px;\n",
       "  width: 15px;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label:before {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label:before {\n",
       "  content: '▼';\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label > span {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-summary,\n",
       ".xr-section-inline-details {\n",
       "  padding-top: 4px;\n",
       "  padding-bottom: 4px;\n",
       "}\n",
       "\n",
       ".xr-section-inline-details {\n",
       "  grid-column: 2 / -1;\n",
       "}\n",
       "\n",
       ".xr-section-details {\n",
       "  display: none;\n",
       "  grid-column: 1 / -1;\n",
       "  margin-bottom: 5px;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked ~ .xr-section-details {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-array-wrap {\n",
       "  grid-column: 1 / -1;\n",
       "  display: grid;\n",
       "  grid-template-columns: 20px auto;\n",
       "}\n",
       "\n",
       ".xr-array-wrap > label {\n",
       "  grid-column: 1;\n",
       "  vertical-align: top;\n",
       "}\n",
       "\n",
       ".xr-preview {\n",
       "  color: var(--xr-font-color3);\n",
       "}\n",
       "\n",
       ".xr-array-preview,\n",
       ".xr-array-data {\n",
       "  padding: 0 5px !important;\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-array-data,\n",
       ".xr-array-in:checked ~ .xr-array-preview {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-array-in:checked ~ .xr-array-data,\n",
       ".xr-array-preview {\n",
       "  display: inline-block;\n",
       "}\n",
       "\n",
       ".xr-dim-list {\n",
       "  display: inline-block !important;\n",
       "  list-style: none;\n",
       "  padding: 0 !important;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list li {\n",
       "  display: inline-block;\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list:before {\n",
       "  content: '(';\n",
       "}\n",
       "\n",
       ".xr-dim-list:after {\n",
       "  content: ')';\n",
       "}\n",
       "\n",
       ".xr-dim-list li:not(:last-child):after {\n",
       "  content: ',';\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-has-index {\n",
       "  font-weight: bold;\n",
       "}\n",
       "\n",
       ".xr-var-list,\n",
       ".xr-var-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-var-item > div,\n",
       ".xr-var-item label,\n",
       ".xr-var-item > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-even);\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-var-item > .xr-var-name:hover span {\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-var-list > li:nth-child(odd) > div,\n",
       ".xr-var-list > li:nth-child(odd) > label,\n",
       ".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-odd);\n",
       "}\n",
       "\n",
       ".xr-var-name {\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-var-dims {\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-var-dtype {\n",
       "  grid-column: 3;\n",
       "  text-align: right;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-var-preview {\n",
       "  grid-column: 4;\n",
       "}\n",
       "\n",
       ".xr-var-name,\n",
       ".xr-var-dims,\n",
       ".xr-var-dtype,\n",
       ".xr-preview,\n",
       ".xr-attrs dt {\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-var-name:hover,\n",
       ".xr-var-dims:hover,\n",
       ".xr-var-dtype:hover,\n",
       ".xr-attrs dt:hover {\n",
       "  overflow: visible;\n",
       "  width: auto;\n",
       "  z-index: 1;\n",
       "}\n",
       "\n",
       ".xr-var-attrs,\n",
       ".xr-var-data {\n",
       "  display: none;\n",
       "  background-color: var(--xr-background-color) !important;\n",
       "  padding-bottom: 5px !important;\n",
       "}\n",
       "\n",
       ".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
       ".xr-var-data-in:checked ~ .xr-var-data {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".xr-var-data > table {\n",
       "  float: right;\n",
       "}\n",
       "\n",
       ".xr-var-name span,\n",
       ".xr-var-data,\n",
       ".xr-attrs {\n",
       "  padding-left: 25px !important;\n",
       "}\n",
       "\n",
       ".xr-attrs,\n",
       ".xr-var-attrs,\n",
       ".xr-var-data {\n",
       "  grid-column: 1 / -1;\n",
       "}\n",
       "\n",
       "dl.xr-attrs {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  display: grid;\n",
       "  grid-template-columns: 125px auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt,\n",
       ".xr-attrs dd {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  float: left;\n",
       "  padding-right: 10px;\n",
       "  width: auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt {\n",
       "  font-weight: normal;\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-attrs dt:hover span {\n",
       "  display: inline-block;\n",
       "  background: var(--xr-background-color);\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-attrs dd {\n",
       "  grid-column: 2;\n",
       "  white-space: pre-wrap;\n",
       "  word-break: break-all;\n",
       "}\n",
       "\n",
       ".xr-icon-database,\n",
       ".xr-icon-file-text2 {\n",
       "  display: inline-block;\n",
       "  vertical-align: middle;\n",
       "  width: 1em;\n",
       "  height: 1.5em !important;\n",
       "  stroke-width: 0;\n",
       "  stroke: currentColor;\n",
       "  fill: currentColor;\n",
       "}\n",
       "</style><pre class='xr-text-repr-fallback'>&lt;xarray.DataArray &#x27;histogram_observations_forecasts&#x27; (time: 3, observations_category: 2, forecasts_category: 2)&gt;\n",
       "array([[[5, 6],\n",
       "        [6, 3]],\n",
       "\n",
       "       [[6, 5],\n",
       "        [4, 5]],\n",
       "\n",
       "       [[5, 5],\n",
       "        [4, 6]]])\n",
       "Coordinates:\n",
       "  * time                          (time) object 2000-01-01 00:00:00 ... 2000-...\n",
       "    observations_category_bounds  (observations_category) &lt;U10 &#x27;[0.0, 0.5)&#x27; &#x27;...\n",
       "    forecasts_category_bounds     (forecasts_category) &lt;U10 &#x27;[0.0, 0.5)&#x27; &#x27;[0....\n",
       "  * observations_category         (observations_category) int64 1 2\n",
       "  * forecasts_category            (forecasts_category) int64 1 2</pre><div class='xr-wrap' hidden><div class='xr-header'><div class='xr-obj-type'>xarray.DataArray</div><div class='xr-array-name'>'histogram_observations_forecasts'</div><ul class='xr-dim-list'><li><span class='xr-has-index'>time</span>: 3</li><li><span class='xr-has-index'>observations_category</span>: 2</li><li><span class='xr-has-index'>forecasts_category</span>: 2</li></ul></div><ul class='xr-sections'><li class='xr-section-item'><div class='xr-array-wrap'><input id='section-eeff72e6-7892-496d-9e5e-2deede42be29' class='xr-array-in' type='checkbox' checked><label for='section-eeff72e6-7892-496d-9e5e-2deede42be29' title='Show/hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-array-preview xr-preview'><span>5 6 6 3 6 5 4 5 5 5 4 6</span></div><div class='xr-array-data'><pre>array([[[5, 6],\n",
       "        [6, 3]],\n",
       "\n",
       "       [[6, 5],\n",
       "        [4, 5]],\n",
       "\n",
       "       [[5, 5],\n",
       "        [4, 6]]])</pre></div></div></li><li class='xr-section-item'><input id='section-1f4d5106-41e8-45f9-ab96-423db3d643e0' class='xr-section-summary-in' type='checkbox'  checked><label for='section-1f4d5106-41e8-45f9-ab96-423db3d643e0' class='xr-section-summary' >Coordinates: <span>(5)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>time</span></div><div class='xr-var-dims'>(time)</div><div class='xr-var-dtype'>object</div><div class='xr-var-preview xr-preview'>2000-01-01 00:00:00 ... 2000-01-...</div><input id='attrs-5595c24a-179b-463a-a505-da1957e8c137' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-5595c24a-179b-463a-a505-da1957e8c137' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-e1ea445a-4e69-44af-8058-13232e6cfd0e' class='xr-var-data-in' type='checkbox'><label for='data-e1ea445a-4e69-44af-8058-13232e6cfd0e' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([cftime.DatetimeGregorian(2000, 1, 1, 0, 0, 0, 0),\n",
       "       cftime.DatetimeGregorian(2000, 1, 2, 0, 0, 0, 0),\n",
       "       cftime.DatetimeGregorian(2000, 1, 3, 0, 0, 0, 0)], dtype=object)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>observations_category_bounds</span></div><div class='xr-var-dims'>(observations_category)</div><div class='xr-var-dtype'>&lt;U10</div><div class='xr-var-preview xr-preview'>&#x27;[0.0, 0.5)&#x27; &#x27;[0.5, 1.0]&#x27;</div><input id='attrs-0b4e547b-74df-418b-8015-a942e897c833' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-0b4e547b-74df-418b-8015-a942e897c833' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-e0b6037c-34ec-46e4-91dc-f499c5fc732b' class='xr-var-data-in' type='checkbox'><label for='data-e0b6037c-34ec-46e4-91dc-f499c5fc732b' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([&#x27;[0.0, 0.5)&#x27;, &#x27;[0.5, 1.0]&#x27;], dtype=&#x27;&lt;U10&#x27;)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>forecasts_category_bounds</span></div><div class='xr-var-dims'>(forecasts_category)</div><div class='xr-var-dtype'>&lt;U10</div><div class='xr-var-preview xr-preview'>&#x27;[0.0, 0.5)&#x27; &#x27;[0.5, 1.0]&#x27;</div><input id='attrs-74386f53-0046-4339-8e1b-d45b4b40632f' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-74386f53-0046-4339-8e1b-d45b4b40632f' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-3dced063-cfda-4c67-936c-3bad673a0aa7' class='xr-var-data-in' type='checkbox'><label for='data-3dced063-cfda-4c67-936c-3bad673a0aa7' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([&#x27;[0.0, 0.5)&#x27;, &#x27;[0.5, 1.0]&#x27;], dtype=&#x27;&lt;U10&#x27;)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>observations_category</span></div><div class='xr-var-dims'>(observations_category)</div><div class='xr-var-dtype'>int64</div><div class='xr-var-preview xr-preview'>1 2</div><input id='attrs-11ae962c-1574-465a-95b9-21bf2cafb093' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-11ae962c-1574-465a-95b9-21bf2cafb093' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-c501bffb-61c7-40d4-94e0-0614a685dbc6' class='xr-var-data-in' type='checkbox'><label for='data-c501bffb-61c7-40d4-94e0-0614a685dbc6' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([1, 2])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>forecasts_category</span></div><div class='xr-var-dims'>(forecasts_category)</div><div class='xr-var-dtype'>int64</div><div class='xr-var-preview xr-preview'>1 2</div><input id='attrs-a2dfdb74-af66-4c44-ae10-74e148c57168' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-a2dfdb74-af66-4c44-ae10-74e148c57168' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-b61f05b7-330f-4dee-b060-ce9d4c77e1a2' class='xr-var-data-in' type='checkbox'><label for='data-b61f05b7-330f-4dee-b060-ce9d4c77e1a2' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([1, 2])</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-48beb471-9e31-4ded-8767-18cd4debed6d' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-48beb471-9e31-4ded-8767-18cd4debed6d' class='xr-section-summary'  title='Expand/collapse section'>Attributes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'></dl></div></li></ul></div></div>"
      ],
      "text/plain": [
       "<xarray.DataArray 'histogram_observations_forecasts' (time: 3, observations_category: 2, forecasts_category: 2)>\n",
       "array([[[5, 6],\n",
       "        [6, 3]],\n",
       "\n",
       "       [[6, 5],\n",
       "        [4, 5]],\n",
       "\n",
       "       [[5, 5],\n",
       "        [4, 6]]])\n",
       "Coordinates:\n",
       "  * time                          (time) object 2000-01-01 00:00:00 ... 2000-...\n",
       "    observations_category_bounds  (observations_category) <U10 '[0.0, 0.5)' '...\n",
       "    forecasts_category_bounds     (forecasts_category) <U10 '[0.0, 0.5)' '[0....\n",
       "  * observations_category         (observations_category) int64 1 2\n",
       "  * forecasts_category            (forecasts_category) int64 1 2"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dichotomous_contingency_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">histogram_observations_forecasts</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>observations_category</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>observations_category_bounds</th>\n",
       "      <th>[0.0, 0.5)</th>\n",
       "      <th>[0.5, 1.0]</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>forecasts_category</th>\n",
       "      <th>forecasts_category_bounds</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>[0.0, 0.5)</th>\n",
       "      <td>5.33</td>\n",
       "      <td>4.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>[0.5, 1.0]</th>\n",
       "      <td>5.33</td>\n",
       "      <td>4.67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             histogram_observations_forecasts  \\\n",
       "observations_category                                                       1   \n",
       "observations_category_bounds                                       [0.0, 0.5)   \n",
       "forecasts_category forecasts_category_bounds                                    \n",
       "1                  [0.0, 0.5)                                            5.33   \n",
       "2                  [0.5, 1.0]                                            5.33   \n",
       "\n",
       "                                                         \n",
       "observations_category                                 2  \n",
       "observations_category_bounds                 [0.5, 1.0]  \n",
       "forecasts_category forecasts_category_bounds             \n",
       "1                  [0.0, 0.5)                      4.67  \n",
       "2                  [0.5, 1.0]                      4.67  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dichotomous_contingency_table.to_dataframe().pivot_table(index=['forecasts_category','forecasts_category_bounds'],\n",
    "                                                         columns=['observations_category','observations_category_bounds']).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scores based on the constructed contingency table can be called via class methods. The available methods are:\n",
    "\n",
    "* `bias_score`\n",
    "* `hit_rate`\n",
    "* `false_alarm_ratio`\n",
    "* `false_alarm_rate`\n",
    "* `success_ratio`\n",
    "* `threat_score`\n",
    "* `equit_threat_score`\n",
    "* `odds_ratio`\n",
    "* `odds_ratio_skill_score`\n",
    "* `accuracy`\n",
    "* `heidke_score`\n",
    "* `peirce_score`\n",
    "* `gerrity_score`\n",
    "* `roc`\n",
    "\n",
    "Below, we share a few examples of these in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([1.        , 1.11111111, 1.1       ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(dichotomous_contingency.bias_score())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([0.33333333, 0.55555556, 0.6       ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(dichotomous_contingency.hit_rate())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([0.54545455, 0.45454545, 0.5       ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(dichotomous_contingency.false_alarm_rate())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([-0.41176471,  0.2       ,  0.2       ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(dichotomous_contingency.odds_ratio_skill_score())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can leverage multi-category edges to make use of some scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_category_edges = np.array([0, 0.25, 0.75, 1])\n",
    "multicategory_contingency = xs.Contingency(obs, fct,\n",
    "                                              multi_category_edges,\n",
    "                                              multi_category_edges,\n",
    "                                              dim=['lat','lon'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([0.25, 0.25, 0.5 ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(multicategory_contingency.accuracy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([-0.14503817, -0.25      ,  0.2481203 ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(multicategory_contingency.heidke_score())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([-0.1496063 , -0.24193548,  0.25      ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(multicategory_contingency.peirce_score())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'histogram_observations_forecasts' (time: 3)>\n",
      "array([-0.15212912, -0.11160714,  0.25      ])\n",
      "Coordinates:\n",
      "  * time     (time) object 2000-01-01 00:00:00 ... 2000-01-03 00:00:00\n"
     ]
    }
   ],
   "source": [
    "print(multicategory_contingency.gerrity_score())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5035528250988777"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARIAAAEGCAYAAACpcBquAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlKUlEQVR4nO3deXwV5dnw8d+VjSYsCRiQfRVRFBCJCqKCWgVbBOXlU9Gito+U8qjFtr6olMKD+lig2EUKinm1VetupYi4YEVRq6QQDZssFlEhQRoQwxohy/X+MQMewkkyOSdzJifn+n4++WRmzpy5L0LOlXvuuRdRVYwxJhpJQQdgjIl/lkiMMVGzRGKMiZolEmNM1CyRGGOilhJ0AHWVnZ2tXbt2DToMYxLOhx9+uFtVW4d7Le4SSdeuXcnPzw86DGMSjoh8Ud1rdmtjjImaJRJjTNQskRhjomaJxBgTNUskxpio+ZZIROTPIlIsIuureV1EZK6IbBGRtSJytl+xGGNOtKigiMGz3qLbXa8weNZbLCooivhaftZIHgOG1/D6FUBP92sC8JCPsRhjQiwqKGLKwnUUlZSiQFFJKVMWros4mfiWSFT1XWBPDaeMAp5QRx6QJSLt/IrHGPOtOUs3U1pWQcXBrzk6lUhpWQVzlm6O6HpBtpF0ALaH7Be6x04gIhNEJF9E8nft2hWT4IxpzHaUlKKq7Fp4H/tWLjzueCSCTCQS5ljYWZZUNVdVc1Q1p3XrsD10jTF10D4rHREhe+QdZPQceNzxSASZSAqBTiH7HYEdAcViTMJYuXIlXba9xndSkkjJbENqK+dGID01mcnDekV0zSATyWLgBvfpzUBgr6p+GWA8xiSE5557jn8tXci0y7vQISsdATpkpTNzdB+u6h+2daFW4tecrSLyDDAUyAb+A/wPkAqgqgtERIB5OE92DgE/VtVaR+Pl5OSoDdozpu5UFRGhsrKSPXv2kJ2dXaf3i8iHqpoT7jXfRv+q6rW1vK7ALX6Vb4z51rJly5g6dSpLliwhOzu7zkmkNtaz1ZgEoapUVlb6cm1LJMY0Yke7S1x66aXk5eXRpk0bX8qxRGJMI/Xmm2/StWtX3n77bQCcZkl/WCIxppE655xzuPHGGzn7bP+HsVkiMaaRWb58OeXl5WRmZvLggw+SmZnpe5mWSIxpRNavX88ll1zCH/7wh5iWG3eTPxtjqnfmmWfy9NNPM2rUqJiWazUSYxqB3NxcPvnkEwDGjh1LenpkY2YiZYnEmDi3Z88epk2bxgMPPBBYDHZrY0yca9WqFStWrKBTp061n+wTq5EYE4dUlenTpzNv3jwAunfvTmpqamDxWI3EmDhUWVnJunXryM7OPjYYL0iWSIyJI6rKN998Q3p6Os899xwpKSmBJxGwWxtj4srPf/5zhg8fzuHDh0lLSyMpqWF8hK1GYkwcGTRoEBkZGaSlpQUdynF8m9jILzaxkUk0FRUVbN68md69ewcaR00TGzWMepExplpTp07lvPPOo6go8gWs/Ga3NsY0cJMmTaJ79+506BDZfKqxYDUSYxqgw4cP8+ijj6KqtG/fngkTJtR7GfGyZKcxJkJPPPEE48ePJy8vz5frx82SncaYyI0fP55//vOfDBo0yJfrH12yM1S8LtlpjAmxf/9+brrpJnbu3ImIMHjwYN/Kqm5pznhcstMYE+KTTz7hxRdfZNWqVb6XVd3SnPG4ZKcxBqefCMCAAQP47LPPuPLKK30vc/KwXqSnJh93LF6X7DQm4X311VcMGjSIv/3tbwC0bNkyJuVe1b8DM0f3qbclO60fiTEBatKkCZmZmTRv3jzmZV/Vv0PEiaMqSyTGBKC4uJiWLVvSrFkz3njjjQYxgjcadmtjTIwdOnSICy644Fgns3hPImA1EmNiLiMjg0mTJjFgwICgQ6k3lkiMiZFPP/2Uw4cP07t3b2699dagw6lXlkiMiQFV5dprr6W0tJQ1a9Y0mAmJ6ouviUREhgMPAMnAI6o6q8rrmcCTQGc3lvtV9S9+xmRMEESEJ554grKyskaXRMDHxlYRSQbmA1cAvYFrRaTqzCy3ABtUtR8wFPidiDSsqZ+MicLatWuPzfR+2mmn0adPn4Aj8oefqfFcYIuqblXVI8CzQNV1BBVoLk6zdTNgD1DuY0zGxNRDDz3E7Nmz2bt3b9Ch+MrPRNIB2B6yX+geCzUPOB3YAawDblPVyqoXEpEJIpIvIvm7du3yK15j6t3cuXPJy8sjMzMz6FB85WciCfdwvOoEscOA1UB74Cxgnoi0OOFNqrmqmqOqOa1bt67vOI2pV++//z7Dhg1j3759pKamNuiZzeqLn4mkEAhdQ7AjTs0j1I+BherYAnwGnOZjTMb4bvfu3RQVFXHgwIGgQ4kZPxPJKqCniHRzG1DHAournLMNuBRARE4GegFbfYzJGN/s27cPgFGjRrF69Wrat28fcESx41siUdVy4FZgKbAReF5VPxaRiSIy0T3tXuB8EVkHLAPuVNXdfsVkjF/uzX2ek9p2pN319zN41lssWfefoEOKKV/7kajqq8CrVY4tCNneAVzuZwzG+G1RQRFPbKokrdvZpLTqcGz+U6DeRtc2dI2vZ4wxMZSfn89vX99EWVpzWl85meR051lBNPOfxiNLJMZEaM2aNQwcOJDNy54L+3qk85/GI0skxkSob9++zJs3j54XVu1n6Yh0/tN4ZInEmDp6+umnKSwsRESYOHEid408q17nP41HlkiMqYPdu3dz880385vf/ObYsfqe/zQe2TQCxtRBdnY27733Hqeeeupxx+tz/tN4ZDUSYzz47W9/y5NPPglAnz59aNKkScARNSyWSIypRXl5Oa+//jpLly4NOpQGy25tjKmGqlJeXk5qaipLliyxWkgNrEZiTDXuuusuRo8eTVlZGRkZGSQnJ9f+pgRlNRJjqtGtWzcOHTpkCcQDSyTGhKisrGT79u106dKFiRMn1v4GA3i4tRHHOBGZ7u53FpFz/Q/NmNibOnUqAwYMYOfOnUGHEle81EgeBCqBS4B7gP3Ai8A5PsZlTCBuuukmsrOzOfnkk4MOJa54aWw9T1VvAb4BUNWvAZvp3TQaZWVlvPjiiwCccsop3H777Y1iGc1Y8pJIytylJRRARFrj1FCMaRRyc3MZM2YMK1euDDqUuOXl1mYu8HegjYjcB4wBpvkalTExNHHiRHr06MG551rTX6RqrZGo6lPAHcBM4EvgKlV93u/AjPHToUOH+MUvfkFJSQnJyckMHz486JDimpenNn9V1U2qOl9V56nqRhH5ayyCM8YvBQUFLFiwgHfeeSfoUBoFL7c2Z4TuuO0lA/wJxxh/qSoiwuDBg9m6dSvt2rWr8fxFBUXMWbqZHSWltM9KZ/KwXgk9yrc61dZIRGSKiOwH+orIPhHZ7+4XAy/FLEJj6snevXu55JJLeOONNwA8JZEpC9dRVFKKwrFJnRcVFMUg2vhSbSJR1Zmq2hyYo6otVLW5+3WSqk6JYYzG1IuKigoOHjzIwYMHPZ0/Z+lmSssqjjuWaJM6e1XrrY2qThGRlkBP4Dshx9/1MzBj6ktJSQnNmzenVatW5OXlkZTkbaxqdZM3J9Kkzl55aWwdD7yLs9DV3e73Gf6GZUz9OHjwIBdeeCG33XYbgOckAtVP3pxIkzp75eWnehtOd/gvVPVioD+wy9eojKknTZs25ZprrmH06NF1fu/kYb0SflJnr7w8tflGVb8REUSkiapuEhH7SZoGbfv27ZSXl9OtWzd+/etfR3SNo09n7KlN7bwkkkIRyQIWAf8Qka+BHX4GZUw0VJWrr76a8vJyPvroozrdzlSV6JM6e+WlsfVqd3OGiLwNZAKv+xqVMVEQER5++GFUNaokYryrMZGISBKwVlXPBFBV6wZoGqxNmzaxatUqrr/+egYMsD6TsVRjulbVSmCNiHSOUTzGRGzmzJnccccd7N+/P+hQEo6XNpJ2wMcishI41pNHVUfW9kYRGQ48ACQDj6jqrDDnDAX+CKQCu1V1iJfAjalqwYIF7Nixg+bNmwcdSsLxkkjujuTC7pic+cBlQCGwSkQWq+qGkHOycGZgG66q20SkTSRlmcSVn5/PnDlzePzxx0lPT6dHjx5Bh5SQvDS2Rtouci6wRVW3AojIs8AoYEPIOdcBC1V1m1tWcYRlmQS1adMm8vPz2b17Nx07dgw6nITlZ5N2B2B7yH6heyzUqUBLEVkuIh+KyA3hLiQiE0QkX0Tyd+2yvnAGDh8+DMC4ceNYv369JZGA+ZlIwk16qVX2U3CmJPg+MAyYJiKnnvAm1VxVzVHVnNatW9d/pCauvPfee/To0YPVq1cDkJ5uXdaD5imRiEh6BL1ZC4FOIfsdObEjWyHwuqoeVNXdOGN6+tWxHJNgunTpQr9+/Wjbtm3QoRiXl0F7VwKrcTuhichZIrLYw7VXAT1FpJuIpAFjgarvewm4UERSRCQDOA/YWIf4TQLZtGkTqkrnzp155ZVXLJE0IF5qJDNwGk5LAFR1NdC1tjepajlwK85o4Y3A86r6sYhMFJGJ7jkbcRLUWmAlziPi9XX9R5jGr6CggL59+5Kbmxt0KCYML49/y1V1byTrfKjqq8CrVY4tqLI/B5hT54ubhNKvXz/uuecerrnmmqBDMWF4qZGsF5HrgGQR6SkifwI+8DkuYwB46aWX2L17N0lJSdx1111kZWUFHZIJw0si+RnOBNCHgaeBvcDPfYzJGACKi4v54Q9/yIwZM4IOxdTCy61NL1WdCkz1OxhjQrVp04Y333yTvn37Bh2KqYWXGsnvRWSTiNwrImfUfrox0Zk/fz5LliwBYODAgWRkZAQckamNl5X2LgaG4kyvmCsi60QksimnjKlFWVkZjz/+OE8++WTQoZg6ENWqnU1rOFmkD87yndeoappvUdUgJydH8/Pzgyja+KyyspKkpCRKSkpo2rQpqampQYdkQojIh6qaE+41Lx3STheRGSKyHpiH88TGBjaYenX33Xdzww03UFFRQVZWliWROOOlsfUvwDPA5apqc7UaX6SmppKWFkgl19SDOt3aNAR2a9N4qCrFxcWcfPLJx/Yj6fhoYiOiWxsRed79vk5E1oZ8rRORtX4FaxLH9OnTGTBgAMXFzjQ0lkTiV023Nre530fEIhCTeH7wgx+QlJSETQ0R/6pNJKr6pbt5s6reGfqaiMwG7jzxXcaEt6igiDlLN1O05wBN92zmvluu46r+fejTp4/vZdriVv7z0iHtsjDHrqjvQEzjtaigiCkL11FUUsq+j15hw6N38Iv5C1lUUBSTMhUoKillysJ1vpaZyGpqI/lvEVkH9KrSRvIZzrB/YzyZs3QzpWUVADQ/6wqyR96JZndnztLNMSnzqNKyCl/LTGQ1tZE8DbwGzATuCjm+X1X3+BqVaVSKdu+jZMXztDjv/5CU9h2ann4hADtKSn0rs7pr+1lmIqvp1kZV9XPgFmB/yBci0sr/0Exj0Wzvp+xd8RzffP7RccfbZ/k312p11/azzERWW41kBPAhzqTNoc/mFOjuY1ymEfnfm8dye1pLKpqffOxYemoyk4fVdRpg7yYP68WUheuOu73xu8xEVm2NRFVHuN+7qWp39/vRL0sipkYHDhxg1KhRrFixgqv6d+B344fRISsdATpkpTNzdB9fn6Bc1b8DM0f3iWmZiazWLvIiMhhYraoHRWQccDbwx6OLWhkTzv79+/nkk0/44osvGDRoEFf17xDzD3EQZSYqL2NtHgL6iUg/nJG/jwJ/BWyNXnOCgwcPkpGRQbt27VizZo2Nn0kQXvqRlKszIGcU8ICqPgDYKs3mBAcOHOCiiy5i6lRnMj1LIonDS41kv4hMAa7HWYMmGbAx3uYETZs2ZciQIVxwwQVBh2JizEsiuQZnse//UtWdItIZWz7ChNi5cycAbdu25fe//33A0Zgg1JpI3OTxFHCOiIwAVqrqE/6HZiIVyzEmlZWVjBgxguTkZPLy8mwEb4Ly8tTmBzg1kOU4fUn+JCKTVfVvPsdmInB0jMnR/hNHx5gAviSTpKQk7r//flJSUiyJJDAvtzZTgXNUtRhARFoDbwKWSBqgmsaY1Gci2bp1K+vXr2fkyJEMHTq03q5r4pOXpzZJR5OI6yuP7zMBiNUYk1/96ldMmDCBgwcP1ut1TXzyUiN5XUSW4szbCk7j66s1nG8C1D4rnaIwSaO+x5jk5uZSWFhI06ZN6/W6Jj55WddmMvAw0BfoB+RWnejINByTh/UiPTX5uGP1NcZk/fr1TJw4kfLyclq0aEHv3r2jvqZpHLzeonwAvAO8BazwLxwTLT/HmLz77ru8/PLL7NhhiwmY49U6i7yIjAem4yQRwekaf4+q/tn/8E5ks8jHXkVFBcnJTi2npKSErKysYAMygYhqgSxgMtBfVX+kqjcCA/A4X6uIDBeRzSKyRUTuquG8c0SkQkTGeLmuiZ28vDzOOOMMNm92ZhazJGLC8ZJICnEnNHLtB7bX9ia3K/18nPldewPXisgJN9XuebOBpV4CNrGVmZlJdna2LeRtauTlqU0R8C8ReQlnQqNRwEoR+SWAqlbXJ/pcYIuqbgUQkWfd926oct7PgBeBc+oevvHL9u3b6dSpE6effjrvvfeedTYzNfJSI/kUWISTRABeAr7EGQFc0yjgDhxfcyl0jx0jIh2Aq4EFNQUgIhNEJF9E8nft2uUhZBONgoICevXqxRNPOCMhLImY2ngZa3N3hNcO99tXtWX3j8CdqlpR0y+rquYCueA0tkYYj/HozDPPZNKkSVxxRfhVR2y9GFOVl1ubSBUCnUL2OwJVnxvmAM+6SSQb+J6IlKvqIh/jMtVYtmwZ55xzDi1atGDWrFlhz4n1WB4TH/zs6r4K6Cki3UQkDRgLLA49wZ3/tauqdsUZu3OzJZFg7Ny5kxEjRvDrX/+6xvNsvRgTjm81ElUtF5FbcZ7GJAN/VtWPRWSi+3qN7SImttq2bcvixYsZOHBgjefZejEmHC/TCJyKM2/ryap6poj0BUaq6v/W9l5VfZUq43KqSyCq+iNPEZt69dhjj9GtWzeGDBnCZZeFW531eLEay2Pii5dbm/8HTAHKAFR1Lc5tiolzR44c4f777+dPf/qT5/f4OZbHxC8vtzYZqrqyylOVcp/iMTGiqqSlpfHWW2/RokULz+872qBqT21MKC+JZLeI9MB9dOt2Y//S16iMr+6//36++OIL5s6dS5s2ber8flsvxlTlJZHcgtOH4zQRKQI+A8b5GpXx1c6dOykuLqaiooKUFD97AJhEUevo32MnijTFmS1tf60n+yheRv82tE5bqsq+ffvIzMxEVamsrDw2otcYL2oa/evlqc30KvsAqOo99RJdI9QQO23de++9PPnkk6xYsYKTTjrJkoipV17qtaGTcn4HGAFs9CecxiFWEzDXxWWXXcbevXtp2bJlIOWbxs3LWJvfhe6LyP1U6aFqjtdQOm1VVlayatUqzjvvPAYNGsSgQYNiWr5JHJF0kc8Autd3II1JdZ2zYt1pa+7cuZx//vmsXr06puWaxOOljWQd347aTQZaA9Y+UoPJw3od10YCwXTa+slPfkKzZs3o169fTMs1icdLG8mIkO1y4D+qah3SahBkp63y8nLmzp3LrbfeStOmTRk/frzvZRpTYyIRkSTgFVU9M0bxNBpBddpatmwZt99+O926dePqq6+OefkmMdWYSFS1UkTWiEhnVd0Wq6BM5IYNG0ZBQQFnnXVW0KGYBOKlsbUd8LGILBORxUe//A7MeFdaWsq4ceNYt87pq2JJxMSalzaSSKdaNDFSXFzMu+++y+WXX06fPn2CDsckIC+J5HtVl+gUkdk4K++ZAB05coS0tDS6dOnCxo0bbR1eExgvtzbhZrsJPyuwiZkDBw5w8cUXM3v2bABLIiZQ1dZIROS/gZuB7iKyNuSl5sD7fgdmapaens6pp57KKaecEnQoxtR4a/M08BowEwhdbnO/qu7xNSpTra+++oqkpCRatmzJX/7yl6DDMQaoIZGo6l5gL3Bt7MIxNamsrGT48OFkZGSwfPlyW7jKNBg2q00cSUpKYtq0aWRkZFgSMQ2KJZI4UFRUxL///W+GDh3KyJEjgw7HmBP4uUCWqSeTJk1i7NixHDp0KOhQjAnLaiRxYMGCBWzbto2MjIygQzEmLKuRNFD//ve/mTJlCpWVlbRu3ZoBAwYEHZIx1bJE0kD9/e9/55FHHqGwsDDoUIypledZ5BuKeJlFPlKqioigqvznP/+hbdu2QYdkDFDzLPJWI2lACgoKOPfcc9m2bRsiYknExA1LJA1IZWUlR44coaysLOhQjKkTe2rTAOzevZvs7GwGDBhAQUEBSUmW30188fU3VkSGi8hmEdkiIneFef2HIrLW/fpARBJuluKCggJ69OjBCy+8AGBJxMQl335rRSQZmI8z5UBv4FoR6V3ltM+AIaraF7gXZ43hhHLaaadx3XXXccEFFwQdijER8/PP37nAFlXdqqpHgGeBUaEnqOoHqvq1u5sHdPQxngZlxYoVfPPNN6Snp/PQQw/Rrl27oEMyJmJ+JpIOwPaQ/UL3WHVuwpm24AQiMkFE8kUkf9euXfUYYjB27NjBJZdcwtSpU4MOxZh64Wdja7jhqWE7rYjIxTiJJGz9XlVzcW97cnJy4qvjSxjt27fnqaee4uKLLw46FGPqhZ81kkKgU8h+R2BH1ZNEpC/wCDBKVb/yMZ7APf/883z44YcAjB492hb0No2Gn4lkFdBTRLqJSBowliqLj4tIZ2AhcL2qfuJjLIE7fPgwU6ZM4b777gs6FGPqnW+3NqpaLiK3Aktx1gz+s6p+LCIT3dcXANOBk4AH3Yl6yqvrghvvmjRpwttvv81JJ50UdCjG1Dsba+OzBx98kOLiYmbMmBF0KMZEpaaxNtaz1UeqykcffURxcTHl5eWkpNiP2zRO9pvtk9LSUtLT03n44YepqKiwJGIaNeuP7YNZs2Zx/vnns3fvXpKTk0lLSws6JGN8ZX8mfXD22WezdetWmjVrFnQoxsSE1UjqiaqyYcMGAC6//HJyc3NJTk4OOCpjYsMSST154IEH6N+/P+vXrw86FGNiLuFubRYVFDFn6WZ2lJTSPiudycN6cVX/moYAeXPjjTdSWVnJGWecUQ9RGhNfEqpGsqigiCkL11FUUooCRSWlTFm4jkUFRRFdr6KigkcffZSKigpatmzJL3/5S1sBzySkhEokc5ZuprSs4rhjpWUVzFm6OaLrvfbaa4wfP57FixfXfrIxjVhC3drsKCmt0/HajBgxgnfeeYeLLroomrCMiXsJVSNpn5Vep+PhHDlyhJtvvplPP/0UwJKIMSRYIpk8rBfpqcc/kk1PTWbysF6er7Ft2zZeeOEFli9fXs/RGRO/EurW5ujTmUie2lRWVpKUlMQpp5zC5s2badWqld/hGhM3EiqRgJNM6vq499ChQ4waNYoxY8bw05/+1JKIMVUk1K1NpFJSUsjIyCAjIyPoUIxpkBKuRlIXe/fuJSUlhaZNm7Jo0SLrI2JMNaxGUo2KigqGDx/OmDFjji3sbYwJz2ok1UhOTuaWW24hMzPTkogxtbBEUkVxcTHbtm0jJyeHcePGBR2OMXHBbm2qGD9+PCNHjqS0NLLersYkIquRVDF//ny++OIL0tO993Y1JtFZjQT4/PPPmT17NqpKp06dbEFvY+rIEgnw2GOPMWvWLIqKIptOwJhEZ4kEmD59OgUFBXTs2DHoUIyJSwmbSDZs2MB3v/tddu3aRVJSEl27dg06JGPiVsImkl27dvH555+zZ8+eoEMxJu4l3FObAwcO0KxZM4YMGcLGjRtJTU0NOiRj4l5C1UjWrFlD9+7defXVVwEsiRhTTxIqkXTr1o1LL73UZno3pp4lRCJZs2YN5eXltGjRgmeeeYYuXboEHZIxjUqjTyTbt29n0KBBTJs2LehQjGm0fG1sFZHhwANAMvCIqs6q8rq4r38POAT8SFU/iqbMcAtgzZs3jyuvvDKayxpjauBbjUREkoH5wBVAb+BaEeld5bQrgJ7u1wTgoWjKDF0A6+CWlXz+6SdMWbiOVv2H0bp162gubYypgZ+3NucCW1R1q6oeAZ4FRlU5ZxTwhDrygCwRaRdpgUcXwKosO8yepfP5+p3Ho1oAyxjjjZ+3Nh2A7SH7hcB5Hs7pAHwZepKITMCpsdC5c+dqCzy60FVSahNOHnsfyc1aHXfcGOMPP2sk4aYV0wjOQVVzVTVHVXNqukUJXegq9aSOJDXJOOG4Mab++ZlICoFOIfsdgR0RnONZfSyAZYypOz8TySqgp4h0E5E0YCxQdbXtxcAN4hgI7FXVL6teyKur+ndg5ug+dMhKR4AOWenMHN2nzuvYGGPqxrc2ElUtF5FbgaU4j3//rKofi8hE9/UFwKs4j3634Dz+/XG05UayAJYxJjq+9iNR1VdxkkXosQUh2wrc4mcMxhj/NfqercYY/1kiMcZEzRKJMSZqlkiMMVETp70zfojILuALD6dmA7t9DidaFmP0Gnp80PBj9BpfF1UN2yM07hKJVyKSr6o5QcdRE4sxeg09Pmj4MdZHfHZrY4yJmiUSY0zUGnMiyQ06AA8sxug19Pig4ccYdXyNto3EGBM7jblGYoyJEUskxpioxX0iEZHhIrJZRLaIyF1hXhcRmeu+vlZEzm6AMf7QjW2tiHwgIv0aUnwh550jIhUiMiaW8bll1xqjiAwVkdUi8rGIvNOQ4hORTBF5WUTWuPFFPdK9jvH9WUSKRWR9Na9H9zlR1bj9wpme4FOgO5AGrAF6Vznne8BrOLOxDQT+1QBjPB9o6W5fEcsYvcQXct5bOKO5xzTAn2EWsAHo7O63aWDx/QqY7W63BvYAaTGM8SLgbGB9Na9H9TmJ9xpJzCeY9iNGVf1AVb92d/NwZoprMPG5fga8CBTHMLajvMR4HbBQVbcBqGos4/QSnwLN3SVYmuEkkvJYBaiq77plVieqz0m8J5LqJo+u6zl+qmv5N+H8ZYiVWuMTkQ7A1cACguHlZ3gq0FJElovIhyJyQ8yi8xbfPOB0nKlE1wG3qWplbMLzJKrPia8TG8VAvU0w7SPP5YvIxTiJ5AJfI6pSbJhjVeP7I3CnqlY4f1BjzkuMKcAA4FIgHVghInmq+onfweEtvmHAauASoAfwDxF5T1X3+RybV1F9TuI9kcR8gukIeCpfRPoCjwBXqOpXMYoNvMWXAzzrJpFs4HsiUq6qi2ISoff/592qehA4KCLvAv2AWCQSL/H9GJilToPEFhH5DDgNWBmD+LyI7nMSq8YenxqQUoCtQDe+beQ6o8o53+f4RqSVDTDGzjjz1p7fEH+GVc5/jNg3tnr5GZ4OLHPPzQDWA2c2oPgeAma42ycDRUB2jH+OXam+sTWqz0lc10g0oAmmfYhxOnAS8KD7V79cYzRa1GN8gfISo6puFJHXgbVAJc5a02EfdQYRH3Av8JiIrMP5sN6pqjGbWkBEngGGAtkiUgj8D5AaEl9UnxPrIm+MiVq8P7UxxjQAlkiMMVGzRGKMiZolEmNM1CyRGGOiZokkzonIJBHZKCJP1XDOUBFZEsu4qiMiI4+OjhWRq0Skd8hr94jId2MYy1AROT9W5TVmcd2PxABwM05v2M+CDsQLVV0MLHZ3rwKW4IzaRVWn13d5IpKiqtUNjhsKHAA+qO9yE43VSOKYiCzAGbq+WER+ISLnuvOZFLjfe4V5zxB3zo7V7nnN3eOTRWSVOxfF3dWUd0BEficiH4nIMhFp7R4/S0Ty3Pf+XURauscnicgG9/iz7rEficg8tyYwEpjjxtJDRB4TkTEicoWIPB9S7lARedndvlxEVrgxvCAizcLEuVxEfuPOSXKbiFwpIv9y/71visjJItIVmAj8wi3/QhFpLSIvuj+HVSIyOJr/n4QSyy669uVLt+fPcbtaAy2AFHf7u8CL7vZQYIm7/TIw2N1uhlMrvRxnAmDB+eOyBLgoTFkK/NDdng7Mc7fXAkPc7XuAP7rbO4Am7naW+/1HIe97jJDu9kf33Zi2AU3d4w8B43DG+bwbcvxOYHqYOJcDD4bst+Tbzpfjgd+52zOA/xty3tPABe52Z2Bj0P+/8fJltzaNSybwuIj0xPnQp4Y5533g926bykJVLRSRy3GSSYF7TjOgJ86HNlQl8Jy7/SSwUEQycZLE0RnJHgdecLfXAk+JyCJgkdd/hDpdzl8HrhSRv+GMA7kDGAL0Bt53hxKkASuqucxzIdsdgefc+TXSgOpuA78L9A4Z4dxCRJqr6n6vsScqSySNy73A26p6tVt1X171BFWdJSKv4IyryHMbNwWYqaoP17G82sZXfB9nZq6RwDQROaMO134OuAVnMp5VqrrfnRToH6p6rYf3HwzZ/hPwe1VdLCJDcWoi4SQBg1S1tA5xGqyNpLHJxBlVCs4txAlEpIeqrlPV2UA+zlD2pcB/HW1vEJEOItImzNuTcG49wJmR7J+quhf4WkQudI9fD7wjIklAJ1V9G6c2kYVT0wm1H2hezb9lOc7UgD/h29pFHjBYRE5x48wQkVOreX+o0J/LjTWU/wZw69EdETnLw7UNlkgam98CM0XkfZxRqOH8XETWi8gaoBR4TVXfwGkfWOGOTv0b4T/gB4EzRORDnAl67nGP34jTaLoWOMs9ngw86V6vAPiDqpZUud6zwGS3EbRH6AuqWoHTVnOF+x1V3YWTIJ9xy8rDSYS1mQG8ICLvcfxi2S8DVx9tbAUmATlu4/AGnMZY44GN/jWeicgBVT3hKYkxViMxxkTNaiTGmKhZjcQYEzVLJMaYqFkiMcZEzRKJMSZqlkiMMVH7/0GdgVLmimpwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC for deterministic forecasts and bin_edges\n",
    "roc = xs.roc(obs, fct, np.linspace(0, 1, 11), return_results='all_as_metric_dim')\n",
    "\n",
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot([0,1], [0,1], 'k:')\n",
    "roc.to_dataset(dim='metric').plot.scatter(y='true positive rate', x='false positive rate')\n",
    "roc.sel(metric='area under curve').values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparative\n",
    "\n",
    "Tests to compare whether one forecast is significantly better than another one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sign test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "length = 100\n",
    "obs_1d = xr.DataArray(\n",
    "       np.random.rand(length),\n",
    "       coords=[\n",
    "           np.arange(length),\n",
    "       ],\n",
    "       dims=[\"time\"],\n",
    "       name='var'\n",
    "   )\n",
    "fct_1d = obs_1d.copy()\n",
    "fct_1d.values = np.random.rand(length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# given you want to test whether one forecast is better than another forecast\n",
    "significantly_different, walk, confidence = xs.sign_test(fct_1d, fct_1d+.2, obs_1d, time_dim='time', metric='mae', orientation='negative')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7ff0d31aabe0>]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEWCAYAAACNJFuYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABBx0lEQVR4nO3dd3xV9f3H8dc3e4+bvUMGBMJIEBARVBxVcG+oWlcdtQra1mprbWvtsNU6UNufo1brAAcO3IqLIYhAEkYImWTvRfa49/v74ybXhCQQIMm9ufk8H4/7yL3nnHvP9zDuO99xvl+ltUYIIYToy8HaBRBCCGF7JByEEEIMIOEghBBiAAkHIYQQA0g4CCGEGEDCQQghxAASDmJCUEpdp5TaNNLHCmGvJByEsFFKqbuUUhVKqUal1AtKKdfDHJuilNqhlGrt+ZnSZ991SimjUqq5z+O0MbgEMY5JOAhhg5RSZwP3AmcAsUAc8MAQx7oA7wGvAP7AS8B7Pdt7bdFae/V5fD2KxRd2QMJB2A2l1L1KqTylVJNSKlMpdfFhjtVKqRVKqXylVI1S6mGllMMhxzyilKpXShUopZb02X69Umpfz3nylVK3jMLlXAv8R2u9V2tdDzwIXDfEsacBTsDjWusOrfUqQAGnj0K5xAQh4SDsSR6wCPDF/Fv2K0qpsMMcfzEwB5gNXAjc0GfficB+IBD4B/AfpZTq2VcFnAf4ANcDjymlZg92AqXUQqVUw2EeC4coWzKQ0ed1BhCilAoY4thduv9cOLt6tvdK7QnBbKXU/UoppyHOKwQg4SDsiNb6Ta11mdbapLV+HcgB5h3mLX/XWtdprYuAx4HlffYVaq2f01obMTfThAEhPef5UGudp82+AT7DHEqDlWmT1trvMI+hOr69gMY+r3ufew/j2N7je4/dAEwHgoFLe67z7iHOKwQg4SDsiFLqJ0qp9N7fyjF/IQYe5i3FfZ4XAuF9Xlf0PtFat/Y89eo5zxKl1FalVF3PeZYe4TzHohlzzaRX7/OmYRzbe3wTgNY6X2td0BOau4E/AZeNcHmFnZFwEHZBKRUDPAfcDgRorf2APZjb3ocS1ed5NFA2jPO4AmuBR4CQnvN8NNR5lFKLDhkldOhj0BoHsBeY1ef1LKBSa107xLEz+zR7Aczs2T4YPVR5hegl4SDshSfmL71qMHcaY645HM7dSil/pVQUsBJ4fRjncQFce87T3dNR/aOhDtZabzxklNChj41DvPV/wI1KqWlKKX/gd8CLQxz7NWAEViilXJVSt/ds/xIsNZ2QnudJwP2YRzcJMSQJB2EXtNaZwD+BLUAlMAPYfIS3vQfsANKBD4H/DOM8TcAK4A2gHvgxsO5Yy32Y83yCuSP8K8xNXoXAH3r3K6U+Vkr9tufYTuAi4CdAA+aO9Yt6toN5OOwupVQL5lrO28BfR7rMwr4oWexHTERKKQ0kaq1zrV0WIWyR1ByEEEIMIOEghBBiAGlWEkIIMYDUHIQQQgxgF7fQBwYG6tjYWGsXQwghxpUdO3bUaK2DBttnF+EQGxvL9u3brV0MIYQYV5RShUPtk2YlIYQQA0g4CCGEGEDCQQghxAASDkIIIQaQcBBCCDGAhIMQQogBJByEEEIMIOEghBAjpLiulQ92DVwzqr6lkze3F9NtNPXb3t5l5I3vi2np6B6rIg6bXdwEJ4QQ1tba2c11/91GXnULRpPmwpQIAEwmzYo1aWzMqaG0oY07z5xsec9fPtzHy1sL2ZxXw+NXptB/MT/rkpqDEEKMgAc/yCS/poW4IE/ue2cPRbXmpcef35TPxpwaEoK9WPVFDtsK6gD4bG8FL28tJCHYi/fSy3h7Z6k1iz+AhIMQQhynj3aXs3pbMbeeGs//bpiHUrBiTRo7Cuv5xyf7OSc5lHduW0CUwYM716Sxv6KJX6/dxfQIHz64YyEnTjJw/3t7KKhpsfalWEg4CCHEcSipb+XetbuYFeXHL86aTKS/Bw9dMpP04gaWP7uVYG9XHrp0Bt5uzqxalkpVUwfnP7mJzm4Tq5al4ubsyOPLUnBxcmDF6jQ6u02DnmOw7aNJwkEIIY5Rt9HEnWvSMWl4clkqzo7mr9RzZ4axbG4URq15fFkqfh4uAMyK8uPus6fQaTTxwAXJxAV5ARDm684/Lp3J7tJGHv40q985dpU0sPiRr7nt1Z2M5fo7drHYz5w5c7TMyiqEGGuPfZ7NE1/k8MSyFEsHdC+TSVPV1EGor9uA95U3thHm6z5g+/3v7uHlrYW8dMM8Tp0cRHNHN+eu2kjlwXbau0z86cJkfnJS7IiVXym1Q2s9Z7B9UnMQQohjsK2gjie/zOGS2REDggHAwUENGgzAoMEAcN+5U5kS4s0v38iguqmD37+3h+K6Vv53w4ksnhLEnz/cx77ygyN6HUORcBBCiKPU0NrJnWvSiDZ48KcLp4/Y57o5O7JqeSpN7V1c/n/f8vbOUlackci8SQYevnwWvu7OrFidRlunccTOORQJByGEOApaa+5du5vq5g5WLU/Fy3VkbxebEurN786bxoHaVubG+nP74gQAAr1cefSKWeRUNfPgh5kjes7ByE1wQghxFFZvK+aTvRX8ZkkSMyP9RuUcV58YjcHDhflxBpwcf/gdflFiELecGscz3+RzSmIg50wPG5Xzg9QchBBi2HIqm/jTB3tZlBjITYviRu08SinOnRlGgJfrgH2/PGsKMyN9uWftbsoa2katDBIOQggxDO1dRu5YnYanixP/vGIWDg7WmerCxcmBVctSLcNojabRGXEq4SCEED1MJs2f3s/knbSSAfv+9tE+siqaeOSKWQR7Dz4KaazEBnry4EXT2Xagjqe+zB2Vc0ifgxBC9HhhcwEvbC7A2VGREOTNjEhfANZnVvLSlkJuOHkSi6cEW7mUZpfMjmRLXi3uLqPzO77cBCeEEMDukkYu+fdmFiYEklXRhKuTAx+sWERzezdLnthAuJ87b9+2AFcnR2sX1UJrfVwzuR7uJjipOQghJryWjm5WrEkj0MuVx65MYX9FE8uf28rv391DRc/dyauWp9pUMACjOsW3hIMQYsL7w7q9FNa28NpN8/HzcOHEuABuPz2RVV/kAPCPS2cS3zMP0kQh4SCEmNDeSy/lrR0lrDg9gflxAZbtK05PYF/5QUJ8XLl8TqQVS2gdEg5CiAmrqLaV372zhxNi/FlxRmK/fU6ODjz3k0Gb4ycEGcoqhJiQuowmVqxJAwVPLEvpdyeykHAQQtgxrTX7K5owDXKj2GOfZ5Ne3MDfLplBpL+HFUpn2yQchBB2a833xZz9+Ab+8en+ftu/za3h39/kceWcKM6bGW6l0tk2CQchhF3KqWzigff34uniyDMb8ticWwNAXUsnd76ezqRAT/5wwTQrl9J2STgIIexO33mQPlq5iLhAT+56PZ3a5g5+/VYGDa1drFqWioeLjMkZioSDEMLuPPRxlnkepMtnERPgyZPLZ9PQ2sUFT21m/b4q7lmSxPQIX2sX06ZJOAgh7MoX+yp58dsDXH9yLIuTzPMgTQv34TdLkyhtaGPxlCBuODnWuoUcB6ROJYSwG5UH2/nVmxlMC/PhnnOS+u27bkEsEX7uzI8PGNVpJ+yF1WoOSqkopdRXSql9Sqm9SqmVPdsNSqnPlVI5PT/9rVVGIcT4YTRp7no93TIPkptz/3mQlFL8KDkUHzdnK5VwfLFms1I38Eut9VRgPvBzpdQ04F7gC611IvBFz2shhDisZzbk8W1eLX+8YBoJwRNrHqTRYLVw0FqXa6139jxvAvYBEcCFwEs9h70EXGSVAgohxo20onoe/Sybc2eEccWcKGsXxy7YRIe0UioWSAW+A0K01uVgDhBg0JU1lFI3K6W2K6W2V1dXj1lZhRC25WB7FyvWpBHi48ZfL54xofoT2traaGpqGpXPtnqHtFLKC1gL3Km1Pjjcv1it9bPAs2Be7Gf0SiiEsFVaa+5/dw+l9W28cctJ+HrYV3+CyWSiqamJuro66uvrLT97H+3t7cyYMYNLLrlkxM9t1XBQSjljDoZXtdZv92yuVEqFaa3LlVJhQJX1SiiEsGVv7yzlvfQyfnHWZObEGqxdnGPS3d1NQ0PDgACoq6ujoaEBo9FoOdbBwQE/Pz/8/f2JiIjAYDAQHj46039YLRyUuYrwH2Cf1vrRPrvWAdcCD/X8fM8KxRNC2LiCmhbuf28P8yYZ+PniBGsX57A6OzstX/i9j97XjY2N/Y51cXHB39+f4OBgpkyZgr+/PwaDAX9/f3x9fXFwGJveAGvWHE4GrgF2K6XSe7b9FnMovKGUuhEoAi63TvGEELaqs9vEitVpODs68PiVKTg6WL+foaOjo9+Xf98AOLRfwN3dHYPBQHR0tOXLvzcAPD09baLfxGrhoLXeBAz1J3DGWJZFCDG+PPLZfnaXNvJ/V59AuJ/7mJ23bwDU1tZavvxra2tpaWnpd6yXlxcGg4H4+HjLF39AQAD+/v64ubmNWZmPldU7pIUQ4mhsyK7m2Q35XHViNOdMDx3xz+/q6rJ84R/689AA8Pb2xmAwkJiYSEBAgKUGYDAYcHFxGfGyjSUJByHEqMutaibYx/W4706uae7gF29kMDnEi/vPO/bpto1GY7/f+vsGwMGDB/sd21sDSExMxGAw9AuB8R4AhyPhIIQYVXtKG7nkX98yNcybN29dgIvTsXWomkyaX72ZwcH2Ll756bwB02McSmvNwYMHLV/+fQOgvr4erX8YAe/u7k5AQACxsbGWAOgNAVdX12Mq73gn4SCEGDWtnd2sWJOGq7MDGSWNPPp5NvcuSTryGwfxwuYCvt5fzYMXJpMU6mPZ3t7ebvnyr6mp6RcEXV1dluOcnZ0xGAyEhYWRnJyMwWAgMDAQg8GAh4csE3ooCQchxKh5YF0mBTUtvPrTE3k/o5xnNuSxMCGQhYmBR/U5u4rr+dcn6Zwf60q8LuP993dbwqBvP4BSCj8/PwIDA5k0aZKlBhAQEIC3t7dNjAIaLyQchBCj4v2MMl7fXsztixNYEB9IapQ/3x+o46430vlk5SICvAY217S3t1NTU2OpAdTU1FBVXUNNbS0XumiohM8/Bw8PDwICAkhMTCQwMLBfM5Cj4+Gbm8TwSDgIIUZccV0rv317N6nRfqw8MxEAdxdHnlyeyoVPb+LeNd9xz2nh1NbWUl1dbQmC5uZmy2c4ODhgMBio6nRiX3cIyxZN48SpsQQEBEgz0BiQcBBCjKhuo4mVa9JQmHjgR1Hk7M+y1AZqamq42q0aXdrNq6+aj3dzcyMwMJCEhAQCAgIICgoiMDAQPz8/PtpTycOr07h9cQKX/WiKdS9sgpFwEEIcl66uLmpqaqiurqa6upotmQeIqalhlmMH77y6zXKcr68vgYGBzJtzAh/nNLOj0sgT15/C7LjQQfsCemsfs/vUPsTYkXAQQgxLR0dHvxDofTQ0NFiOUUrRYHTFy9efhbPiLbWAwMDAfvcEzFvUyTmPb+DX7+7n/TuC8HDp/1XUW/sAeGJZKs6ONrG6wIQi4SCE6Kejo4Pq6mqqqqqorq42dwpXVfW7OczR0ZHAwEAiIyNJSUkhKCgIF08/rnllL+4ezrz/84V4ug799WLwdOHxK1O46j/f8cC6TP5+2cx++5/4IoedRQ08uTyVKIP0L1iDhIMQE1RnZ6flt//eIDg0BJycnAgMDCQmJoagoCDLw9/fv9/soFprbn55BzWtXbxz3bzDBkOvBQmB/OzUeP71dR6nTA7i3JlhAGzJq+Wpr3K5/IRIzp81OtNRiyOTcBDCznV3d1NbW0tlZWW/EOjbHOTo6EhQUJAlBIKDgwkKCsLPz29YU0S/8l0Rn2dW8rtzpzI9wnfYZbvrrMl8m1fLvW/vYlaUL54uTtz1ejqTAjz54wXJx3K5YoRIOAhhJ7TW1NfXU1VVRWVlJdXV1VRWVlJbW2uZKsLBwYGAgAAiIiJISUkhODiY4ODgATWBo7G/ook/f5DJqZODuOHkSUf1XmdHB1YtS2Xpqo2sXJOOv4cLtS0dPH/tycOqfYjRI3/6QoxDra2tVFZWWmoDvWHQd7qI3gVjkpKSCAkJITg4mICAgBG9Say9y8gdq3fi7ebMI5fPwuEY1lWIDvDgLxdPZ+WadICjrn2I0SHhIIQNMxqN1NTUWIKg99H3ZjEPDw9CQkKYPXs2wcHBhISEmDuIx2DG0D9/mEl2ZTMv3TCPIO9jn6DuwpQIMssPUt/SedS1DzE6JByEsBHNzc1UVlZSUVFBVVUVFRUV1NTUYDKZgB/6BeLj4y01gZCQELy8vKxS3k/3VvDK1iJuWjSJUycHHffn/WbJ1BEolRgpEg5CjDGTyURtbS0VFRWWMKioqOg3gZy3tzehoaEkJiYSEhJCaGioTc0bVN7Yxj1rdzEjwpe7zz62WVaFbZNwEGIUdXZ29guA3mah7u5uwNxBHBwcTEJCgiUEQkJCjmvuoLzqZjxdnAj1Hd5SlAfbuyiqbR12O7/RpLlzTTqd3SZWLU895vUZhG2TcBBihLS2tlJRUUF5ebklDPqOFHJzcyM0NJQ5c+YQGhpKaGgogYGBI1obKKpt5cKnNuPr7sxHKxfh6374lde6jSau/+/37CyqZ/VN85kfF3DEc/zrq1y+K6jj4ctmMinQc6SKLmyMhIMQx6C5uZny8nLKysosgdDY2GjZ7+PjY1lUpjcIfH19R3U9gS6jiTvWpKGAyoPt/Pbt3Tz149TDnnPVl7nsKKzH38OZu15P56MVi/D3HLoje0dhHY9/kcMFs8K57ITIUbgKYSskHIQ4gqamJsrKyigvL7cEQt/RQgaDgaioKObOnUtYWBihoaFWmVL60c+zyShu4F9XzeZAbQv/+GQ/p2wP5Mq50YMe/11+LU99mcOlsyO5/uRYLv7XZu5Zu4tnrjlh0EBpbOtixep0wv3c+PPF02XhHDsn4SBEH83NzZSVlVnCoG8QKKUIDAwkdtIkwsLCiAgPJzQ01CbWGN6cW8P/fZPH8nlRLJ0Rhsmk2Zxbwx/XZXJCjIGE4P4jmhpaO7nz9XRiAjx54MJkvFyduOecJP784T5e/a6Iq+fH9Dtea8197+ym8mA7b956Ej5uh2+uEuOfhIOYsNra2ixBUFZWRmlpKU1NTZb9gYGBxMfHExYWRnh4OCEhIRhx5JJ/f4t3Fbw6L8omZgutbe7grtfTiQv05P7zpgHg4KB49IoUznl8A3esTuOd2xbg5mzu29Bac8/aXdQ0d/D2z07Gq+dO5BtOnsSGnBoe/CCTubEGpoR6W87x5vYSPthVzt1nTyE12n/sL1KMOQkHMSF0dXVRUVFBaWmpJQjq6uos+w0GAzExMYSHhxMeHk5YWNigN5H99p3d7Cs3T0z3+Ppsqw/j1Fpz91u7aGjt4sXr5/Wb+jrEx41HLp/FjS9t5++fZPGH881zFb22rYhP91by26VJzIj8YYSSg4Pin5fPYskTG7lj9U7W3b4QN2dH8qqb+cO6vSyID+DWU+PH/BqFdUg4CLujtaampobS0lLLo7Ky0nIzmY+PD+Hh4aSkpBAREUFYWBju7u5H/NxP9pTz2ndF3HJqHA0tXfzr6zxOTghkQXzgaF/SkF769gBfZlXxx/OnMS3cZ8D+M6aGcN2CWP67+QCLEgOJ9PfgT+9nsigxkJ8ujBtwfJC3K/+8YhbXvrCNv3y4j9+dN5U7XkvDzdmBx65MwfEYpscQ45OEgxj3WltbKS0tpaSkhJKSEkpLS+no6ADA1dWV8PBwFixYQEREBBEREXh7ex/hEwcqbWjj12/tYmakL788awrdJhPfF9Zx1+vpfLzyFAyHGeEzWjLLDvLXj7I4IymYaxfEDnncvUuS2Jpfy6/e3EWApwvebk7884qh50E6dXIQNy2axHMbC8ipaiKz/CDP/2QOIT7Du29C2AcJBzGumEwmqqqqLEFQUlJCbW0tYO4wDgkJYfr06URGRhIREUFgYOBxj6oxmjR3rUnHaNKsWma+6csFB55cnsrFT3/Lr9/axXM/6T/C59XvCvlmfzWrlqda2vpHUmtnN3es3omfhzP/uGzmYa/RzdmRp36cynlPbqKupZMXr59LsPfhv+jvPjuJrfl1bM2v47oFsZw5LWSkL0HYOAkHYdPa29spKSmhuLiY4uJiSktL6ezsBMwTzkVFRZGSkkJkZCTh4eGjMtncU1/msu1AHY9eMYvYPjd9JYf7cu+SJP70QSYvby3kJyfFApBWVM/v39uL0aR56OOsUVmX4MEPMsmvaeGVG08kwOvIo6USgr159po5VDd1cNqU4CMe7+LkwL+vns3bO0u5+ZSBzU/C/kk4CJuhtaaxsZGioiKKioooLi6mqqoK+KFWMHPmTKKiooiKisLPz2/Ux9p/f6COJ77I5uLUCC6ZPfCmr+tPjmVTbg1//nAfc2MNRPi7s2JNGqE+bixKDOTFbw+wMCFwRH/z/mh3Oau3FfOz0+I5OWH4/R2nHOXkeJH+Hqw4I/FoiyfshOq9tX88mzNnjt6+fbu1iyGOUm8TUW8YFBUVWYaSurq6EhkZaQmCiIiIMb+foLG1i6WrNuLooPhwxUK8hxjbX9vcwTlPbMTP3ZnJId58sreCN26Zz/QIXy56+lsqGtv45M5TRqTNvqS+laVPbGRSkBdv3XqSTQylFeOXUmqH1nrOYPuk5iDGjNFopLy8nMLCQgoLCykqKrJ0HPv4+BATE0N0dDRRUVEEBwcf88pkI0Frzb1v76LyYDtv/WzBkMEAEODlymNXpHDNC9+RU9XML8+azAkxBgCeXJ7K+U9u4q7X03n5xhOHNdqnqb2LZzfkc/X8mH6B0m00ceeadEwaVi1LkWAQo0rCQYya7u5uSktLLWFQXFxsWaksMDCQ5ORkSyCM9rxDR2vN98V8vKeCe85JIiXK74jHL0wM5LdLprKv4iC3LU6wbE8I9uKPF0zjnrW7eWZDHredlnCYT+m9E3kP6zLK+K6gjtU3zbcEypNf5rK9sJ7Hr0whJkAmvBOjy6rhoJR6ATgPqNJaT+/ZZgBeB2KBA8AVWut6a5VRDJ/RaKS0tJSCggJLGPROTR0SEkJqaioxMTHExMTg6Wm7X265VU088P5eFiYEcstRdMbeNMSxV8yJYkN2DY9+ls1JcQGHvcN47c5S1mWUcXJCAJtza3n6q1xWnJHItoI6nvwyh0tmR3BRasRRX5MQR8uqfQ5KqVOAZuB/fcLhH0Cd1vohpdS9gL/W+p7DfY70OViHyWSivLycgoICDhw4QFFRkaVmEBoaSmxsLLGxsURHRw/rJjNb0N5l5KKnN1PV1MEnKxcRPEJj+xvbulj6xEYcHOCjFYsGbabKr27mvCc3MTPSl1d/Op9fvpHOuowynvvJHO5/dw8uTg58sGKRZboLIY6XzfY5aK03KKViD9l8IXBaz/OXgK+Bw4aDGBtaa2pra8nPzyc/P58DBw5Y+gyCg4NJTU21BMJ4CYNDPfRxFlkVTfz3urkjFgwAvu7OrFqewhXPbOV37+7h8StT+jWjdXabWLkmHRenH+5EfvCi6ewsauDGl7bj5KB4+7YFEgxizNjiv7QQrXU5gNa6XCk16KBspdTNwM0A0dGDT0ksjl9LS4slDPLz8zl40DyvkJ+fH9OmTSMuLo7Y2FirrWM8kr7YV8mL3x7g+pNjWZx05HsBjtYJMQZWnpHIo59nc0piEJf2WQ/h4U+z2F3ayDPXnECYrzlYvd2cWbU8laue28ovfjSFmZF+I14mIYZii+EwLFrrZ4FnwdysZOXi2A2j0UhxcTG5ubnk5+dTXl4OmFcxi4uLszz8/e1rZs7Kg+3c/dYupoX5cO+S0ZtM7+eLE9iUW8Pv39vD7Bh/JgV68k12Nc9tLOCa+TGcnRza7/iUKD/Sfv8jWYpTjDlbDIdKpVRYT60hDKiydoHsXWNjIzk5OeTm5lJQUEBnZycODg5ERkayePFiy7TV1hxaOppMJs0v3kinrdPIquWpuDqN/HQXvRwdFI9fmcKSJzayYnUaz1xzAr98I50pId7cd+7UQd8jwSCswRbDYR1wLfBQz8/3rFsc+9NbO8jJySEnJ4fq6moAfH19mTFjBgkJCUyaNMkmFrEZitaa1k4jnoO0wbd0dB/V9mc25LM5t5a/XzpjwKI4oyHcz52/XzqTW1/ZwdJVG2nrNPLqT+ePyhxMQhwraw9lXY258zlQKVUC/AFzKLyhlLoRKAIut14J7UdbW5slDHJzc2lvb8fBwYGYmBhSUlJITEwckUnqxsrdb+3i88xK3r99IdEBPyzJ+fRXuTyxPodXfnoi8yYZLNs/2FXGnWvS+fulM/u19acXN/DPz/Zz7owwrpgTNWblP2d6KFedGM2r3xXx4EXT+y2sI4QtkOkz7Fh9fT1ZWVns37+foqIitNZ4enqSmJjI5MmTiYuLs+nawVDe3lnCL97IQCmYGelnmUZi+4E6rnx2KyatCfNx4+OVp+Dr4UxxnXnKiebObtydHfngjoXEBXnR1N7Fuas2YTRpPlq5CF/3sV36sstoYndpI6lRoz9HlBCDsdmhrGJkaa2prKxk3759ZGVlWSatCw4OZuHChUyePJmIiIhx/UV0oKaF+9/dw7xYA9ecFMMdq9N47PNsbjk1npVr0onwc+ehS2dw7QvbuPftXaxansrKNWkAvHnLSfz0f9tZsSaNt392Mve/u4eS+lbeuOWkMQ8GAGdHB2bLkpvCRkk4jHNaa0pLS8nMzCQrK4v6evPN5NHR0fzoRz9iypQpGAyGI3zK+NDZbWLFmjScHB14bFkKEX7ubM6t4d/f5PFtXi2VB9t589aTSI3251c/msLfPs7iyme2sLOogSeXpzIn1sA/Lp3JzS/vYNmz5u2/OGsyc2Lt489HiJEk4TAOaa0pKSkhMzOTzMxMDh48iIODA3FxcSxcuJApU6bY9PQUAB3dRm57ZSdzJxn6rUustea+d/eggD9fNL1fLeefn+9nV0kj/3f1bCL8zPcC/P78aXx/oI704gZ+fc4Uy9QUNy2KY1NuDRtzarj8hEjOnxUOwI+SQ7l6fjSvbC1i3iQDP198+LmOhJioJBzGCa01FRUV7Nmzh71799LY2IijoyPx8fGcfvrpTJkyBTe38bOM498/3s8XWVV8ub+KGRG+lnUJXvmuiNe+KwJgWrgPV50YA8DGnGqe+SafH58YzTnTwyyf4+HixPPXzuXLrCqu77NUpkPPkNE3d5RwzfyYfuf+3bnTiPT34JLUCFkTWYghSIe0jauvr2f37t3s3r2bmpoaSw0hOTmZpKSkcRUIvb7KquL6F79n+bwothXU0dTezccrF1HT3MkFT21iflwAJq35/kAd79++EH9PF5b0rJew7vaFuLvIkE8hRoJ0SI8z7e3t7N27l4yMDIqLiwFzH8K5557LtGnT8PDwOMIn2K6qg+386s0MkkK9+cP5yeRXt3DR05v51ZsZlDa04e3mzCOXz0KjWfL4Ru5YnUaIjxuNbV3874Z5EgxCjBEJBxuhtaagoIC0tDT27duH0WgkMDCQM844gxkzZuDr62vtIh43k0nzyzczaOnsZs1y801f08J9+M3SJB54PxOAl26YR5C3eXjtI1fM4vr/fk9WRRMPXJDM1DAfaxZfiAlFwsHKDh48SFpaGmlpaTQ2NuLm5sbs2bNJSUkhLCxsXA87PdRzG/PZmFPDXy+eQWLIDzd9XbcglgM1LUT6e3Bqn3WOF08J5rdLkyhraOcnJ8UM9pFCiFEi4WAFWmtyc3PZsWMH2dnZaK2Ji4vjzDPPJCkpCScn+/tryShu4OFP97NkeijL5/W/E1kpxQMXTh/0fTefEj/odiHE6LK/byEb1tbWRlpaGtu3b6e+vh5PT09OPvlkZs+ebXeznPbV3NHNijVpBHu78tAlM+2qNiSEvZJwGAM1NTV89913ZGRk0NXVRXR0NKeffjpTp07F0dH+O1h//+4eiutaWXPzSfh6jP2dyEKIoyfhMEq01hQVFbF582ZycnJwdHRkxowZnHjiiYSGhh75A+zEO2klvJ1WysozEvtNhCeEsG0SDiNMa012djabNm2ipKQEDw8PTj31VObOnWvzdy2PtMLaFn73zh7mxvpzx+lyJ7IQ44mEwwjRWrN//36++eYbKioq8PPzY8mSJaSmpuLsPPGaUjq7TaxYnWZe3GZZKk6OsmCNEOOJhMNx0lqTl5fHF198QUVFBQaDgQsvvJAZM2ZMiP6EoTz6eTYZJY38+6of5kESQowfhw0HZR5WEqm1Lh6j8owrJSUlrF+/nsLCQvz8/LjooouYMWOG3S6nOVybcmr4v2/yWD4viiUzwo78BiGEzTlsOGittVLqXeCEsSnO+HDw4EHWr1/P7t278fT0ZOnSpcyePXtC1xR61TZ3cNcb6SQEe/H785KtXRwhxDEaTrPSVqXUXK3196NeGhtnNBrZvHkzmzZtwmQysWjRIhYuXIiLi4u1i2YTtNbc/dYuGtu6eOl6mQdJiPFsOOGwGLhFKVUItAAKc6Vi5qiWzMYUFxfz/vvvU11dzdSpUznrrLPs+sa1Y/Hitwf4MquKP54/jWnhMg+SEOPZcMJhyaiXwoZ1dXWxfv16tm3bho+PD8uXL2fy5MnWLpbNySw7yN8+yuKMpGCu7bOughBifDpiOGitCwGUUsHA+Fs84DhUVlaydu1aqqurmTdvHqeffjqurq7WLpbNae3s5o7VO/HzcOYfl8n0GELYgyOGg1LqAuCfQDhQBcQA+wC77W3UWrN9+3Y+/fRT3NzcuPrqq4mPlwnghvLgB5nk17Twyo0nEuAl4SmEPRhOs9KDwHxgvdY6VSm1GFg+usWyHpPJxEcffcSOHTtISEjgoosumnB3Nh+Nj3aXs3pbMT87Ld6y1KcQYvwbzoD8Lq11LeCglHLQWn8FpIxusayjo6OD1157jR07drBw4UJ+/OMf23wwdBlN/O2jfewsqh+1c+wta+RP72fS3mXst72kvpV71+5iVpQfvzhL+mGEsCfDqTk0KKW8gI3Aq0qpKqB7dIs19lpbW3nppZeorq7m/PPPZ/bs2dYu0rA89nk2z2zI5+20Uj5ZuWjEm3UOtndxy8s7KKlvo8to4sGLzOsudBtN3LkmHZOGVctScJbpMYSwK8P5H70B8ANWAp8AecD5o1imMdfZ2clrr71GXV0dV1111bgJhm9za/j3N3ksnhJEY1sXv3ozA631iH2+1pr73tlDeWM7Z04N4eWthXy2twKAVV/msr2wnj9fNJ2YANuuXQkhjt5wwkEBnwJfA17A6z3NTHbBZDKxdu1aysrKuPTSS8dNx3NdSyd3vp7OpEBPnr5qNr87dypf7a/mv5sPjNg53tpRwvsZZfzirMk8fVUq0yN8+PXaXbyXXspTX+ZwyewILkqNGLHzCSFsxxHDQWv9gNY6Gfg55hFL3yil1o96ycaA1poPPviA7Oxsli5dSlJSkrWLNCxaa379VgYNrV08uTwVDxcnrpkfw5lTQ3jo4yz2lDYOeE9aUT1ZFQcHbM+vbmZbQd2g2/+wbi8nxQVw66nxuDo5smpZKp3dJlauSSfa4MGfhljaUwgx/h1NQ3EVUAHUAsGjU5yxlZ+fT1paGosWLWLOnDnWLs6w/W9LIev3VXHvkiSSw30B8zrM/7hsJv6ezqxYk0Zr5w/dQjmVTSx/bivLnt1KeWObZXttcwfLnt3Kj5/bSkZxg2V7R7eRO1an4erkwGNXpuDoYL5vIS7Ii79ePINAL1dWLU/Fy1Um9RXCXh0xHJRSP1NKfQ18AQQCN9nL1Bnx8fFcffXVLF682NpFGbZ95Qf5y0f7OD0pmOtPju23z+DpwmNXplBQ08ID6zIBaO8yf9F7uDjR2W3uRDaaNFpr7lm7i4bWLgyeLqxYk0ZzhzlQHv5kP3vLDvLwZbMI9e1/3+NFqRF8f98ZzIz0G4vLFUJYyXB+9YsB7tRap49yWaxivPQxALR1mr/ofd2deXiIO5EXxAdy22nxPP1VHosmB7L9QD1ZFU3897q51LZ08qs3M/j317l4uzmzfl8Vfzh/Gsnhvix7dgu/f28PF8wK5/lNBVx7UgxnTgsZtBxyB7QQ9m8402fcOxYFsZYN2dUsTAjEwWF4X3j1LZ14uTkNGLrZ3NGNAjwPaWrp7DbR2tmNn0f/mVtNJk1tSydB3gOHnlY1tRPsPXCmkj99kEledTMv33D4O5HvPHMym3NrufvNXbR1Gblx4SQWJwWjtWZDdjWPrc/BUSlOTwrmugWxKKVYcUYij6/P4ZM9FSSFevObpVOH9echhLBPE3pw+sacan7ywjZe2FwwrOOL61o59eGvuO6/2zCafhgy2tTexXmrNnL+U5to6fihrd9o0lzzn+847ZGvKWto6/dZ9769iwUPfUF6n7Z+gH9/nce8v3zB+xll/bZ/vLuc1duKuPmUOBYmHv5OZGdHB55cnoqTgyI53IdfnzMFMP/G/+eLpxPu54afR//ax+2LE5gXa8Bo0jy5PBU3Z5luW4iJTI3kuHhrmTNnjt6+fftRv09rzS0v7+Cr/VW8c9vJTI/wHfLYbqOJK57Zwu7SRrqMml+fM4XbTktAa82dr6fzfkYZGrhsdiQPXz4LgCfW5/DY+mycHRWp0f6svmk+jg6K99JLWbkmHWdHRZivOx+uWIi3mzNpRfVc9n9bcFDg5uTIRysXEWXwoLShjSWPb2BSoCdv/WzBsG84K2tow8fdeUDHcUNrJ11GPaDW0tZppKa5gyiDx9H9QQohxiWl1A6t9aCjcWy25qCUOkcptV8plauUGpWmLaUUf790JgGertyxOq3fb/2HeuKLHHYWNfDoFSmcOyOMRz/LJr24gbd3lvJeehl3njmZn5+WwJs99wZ8f6COJ77I5uLUCB66ZCbbCup4+qtciuta+d07e5gd7cf/bjiRkvpW7n93D03tXaxYk0aojxvv3HYyACvXpNHRbeSu3juRl6ce1Z3I4X7ug44o8vNwGbQ5y93FUYJBCAHYaM1BKeUIZANnASXA98ByrXXmYMcfa82h19b8WpY/t7Xfb/19bcmr5cfP/7C/sa2LpU9sBMy/hSdH+LL6pvmYtObKZ7aQU9mMt5sTTo4OfLhiIV6uTtz1ejrv7yonNsCDqqYOPlphrhX01i6mhHiTW93MG7fM54QYA+syylixOo0pId7sr2zisStncXFq5DFfoxBCHGo81hzmAbla63ytdSewBrhwtE42Py6AOxabf+t/L7203776lk7uej2dSQGe/PEC8yzlvu7OrFqeQsXBdpwcHXi8514AZ0cHnliWCkBVUwerlqfi7eaMUooHL5pOhJ87edUt/PXiGZbf0G8/PYF5kwzsr2zizjMSOSHGAMAFs8K5/IRI9lc2cUlqhASDEGJM2WrN4TLgHK31T3teXwOcqLW+vc8xNwM3A0RHR59QWFh4XOfsNpq48tmtZFc0Wdr6tdbc/PIOvh6iT+Kr/VUEeroyI7L/9vTiBhrbujh1clC/7QU1LewubeSCWeH9ttc0d7A+s5LL50RZbjgD8yI676aVcWFK+IBRUEIIcbwOV3Ow1XC4HDj7kHCYp7W+Y7Djj7dZqVdxXStLV20kIdiLN245iTXbirj/vb387typ/HRR3HF/vhBC2JLx2KxUAkT1eR0JlA1x7IiJMnjwt0tmkFbUwN1vZvDnD/dx6uQgbjh50mifWgghbIqthsP3QKJSapJSygVYBqwbixOfNzOcK+dE8W56Gd5uzjxy+axh3yAnhBD2wiYbsrXW3Uqp2zFPFe4IvKC13jtW5//DBdMwas0Vc6IGHfIphBD2zibDAUBr/RHwkTXO7eHixCODDGkVQoiJwlablYQQQljRhA6HgwcP8sorr9DQ0GDtogghhE2Z0OHQ3t5OaWkpr7zyCq2trdYujhBC2IwJHQ7BwcEsW7aMhoYGVq9eTVdXl7WLJIQQNmFChwNATEwMl1xyCSUlJaxduxaTyWTtIgkhhNVN+HAAmDZtGkuWLGH//v288cYbdHZ2WrtIQghhVRIOPebNm8eSJUvIzs7mxRdfpKmpydpFEkIIq5Fw6GPevHksW7aMmpoann/+ecrLy61dJCGEsAoJh0NMnjyZ66+/Hq01zz//PFu2bMEWJycUQojRJOEwiLCwMG699VYSExP57LPPeO2112hubrZ2sYQQYsxIOAzBw8ODK6+8kqVLl1JQUMDTTz9NWlqa1CKEEBOChMNhKKWYO3cut956K8HBwaxbt46XX36Z2tpaaxdNCCFGlU0u9nO0Rmqxn8PRWrNjxw7Wr19PV1cXJ554Iqeccgpubm6jel4hhBgth1vsx2ZnZbU1SinmzJlDUlISX3zxBVu2bGHXrl2cdtpppKam4ujoaO0iCiHEiJGawzEqKyvj008/paioCIPBwOLFi0lOTkYpWRhICDE+jLs1pI+WNcIBzE1N2dnZfPnll1RVVRESEsKpp55KUlKShIQQwuZJOIwyk8nEnj17+Oabb6irqyM4OJhFixYxbdo0HBykz18IYZskHMZIb0hs2LCB2tpa/P39WbBgAbNmzcLZ2dnaxRNCiH4kHMaYyWRi//79bN68mdLSUjw8PJgzZw5z587Fy8vL2sUTQghAwsFqtNYUFhayZcsWsrOzcXR0ZPr06cydO5eIiAhrF08IMcHJUFYrUUoRGxtLbGwstbW1fPfdd2RkZJCRkUF4eDhz584lOTlZmpyEEDZHag5jrKOjg4yMDLZv3051dTWurq7MnDmTE044gZCQEGsXTwgxgUizkg3SWlNUVMTOnTvZu3cvRqOR8PBwUlJSmDFjhtx5LYQYdRIONq6trY2MjAzS09OprKzEycmJKVOmMGvWLOLj42U4rBBiVEg4jBNaayoqKkhLS2PPnj20tbXh6enJ9OnTmTFjBuHh4XJznRBixEg4jENGo5GcnBx27dpFdnY2RqORgIAAkpOTmT59OkFBQdYuohBinJNwGOfa29vJzMxk9+7dHDhwAIDg4GCSk5NJTk4mICDAugUUQoxLEg52pKmpiczMTPbu3UtxcTEAISEhTJ06lWnTpkmNQggxbBIOdurgwYNkZmaSmZlpCYrAwECSkpJISkqSPgohxGFJOEwATU1N7Nu3j3379lFYWIjWGh8fHyZPnkxSUhIxMTE4Ock9j0KIH0g4TDCtra3k5OSQlZVFXl4eXV1duLi4kJCQwOTJk0lISMDT09PaxRRCWJlMnzHBeHh4MGvWLGbNmkVXVxcFBQVkZWWRk5NDZmYmAJGRkSQmJjJ58mRCQkKk+UkI0Y/UHCYQrTXl5eVkZ2eTk5NDWVkZAN7e3sTHx5OYmEhcXJzcnS3EBGFzzUpKqcuBPwJTgXla6+199v0GuBEwAiu01p8e6fMkHI5Nc3Mzubm55OTkkJeXR0dHB0opoqKiiI+PJz4+nrCwMLlDWwg7ZYvhMBUwAc8Av+oNB6XUNGA1MA8IB9YDk7XWxsN9noTD8TOZTJSUlJCbm0tubi7l5eUAuLu7ExcXZ3n4+flZt6BCiBFjc30OWut9wGDt3BcCa7TWHUCBUioXc1BsGdsSTjwODg5ER0cTHR3N6aefTktLC3l5eeTn55Ofn8/evXsBMBgMlqCIjY3F3d3dyiUXQowGW+uQjgC29nld0rNtAKXUzcDNANHR0aNfsgnG09OTmTNnMnPmTLTWVFdXW4Kid8pxgLCwMCZNmsSkSZOIjo7GxcXFyiUXQoyEUQsHpdR6IHSQXfdprd8b6m2DbBu03Utr/SzwLJiblY6pkGJYlFIEBwcTHBzM/PnzMRqNlJaWkp+fT0FBAVu3buXbb7/FwcGBiIgIywJHUVFRspCREOPUqIWD1vrMY3hbCRDV53UkUDYyJRIjxdHR0dIEddppp9HZ2UlxcTEFBQUcOHCATZs2sXHjRhwdHYmIiCAmJobY2FgiIyOlZiHEOGFrzUrrgNeUUo9i7pBOBLZZt0jiSFxcXCyjm8C82l1hYSGFhYX9wsLBwYHw8HCio6OJiYkhOjpahs0KYaOsNVrpYuBJIAhoANK11mf37LsPuAHoBu7UWn98pM+T0Uq2raOjg+LiYg4cOEBhYSFlZWWYTCbAPGlgby0kJiYGb29vK5dWiInD5oayjjQJh/Glq6uLkpISioqKKCoqori4mK6uLgD8/PyIjo4mKiqK6OhogoKC5O5tIUaJzQ1lFRObs7OzZYQTmBc2qqiooLi4mKKiIvLy8ti1axcArq6uREVFWR4RERHSbyHEGJBwEFbX23EdERHB/Pnz0VpTX19vqVUUFxeTm5sLmEdOhYSEEBkZSVRUFJGRkfj7+0vtQogRJs1KYlxoa2ujpKSk36OzsxMwTzQYGRlJRESE5aerq6uVSyyE7ZNmJTHuubu7k5iYSGJiImCe7qO6urpfWGRnZ1uODwoKstRGIiMjCQ4OljmihDgKUnMQdqO9vZ3S0lJKSkosP9va2gBwcnIiPDyc8PBwS2j4+flJc5SY0GS0kpiQevsuSktLLY/y8nKMRvM8ju7u7v0CIzw8XIbSiglFmpXEhKSUwmAwYDAYmDFjBmAeGVVVVUVpaSllZWWUlpayadMmen9J8vb2tgRGWFgY4eHhsmqemJAkHMSE4ujoSFhYGGFhYZZtnZ2dVFRUUFZWZnns37/fst/Hx8cSFhIYYqKQcBATnouLi+Uu7V7t7e2WwCgvL6esrIysrCzLfh8fH8LCwggNDbWEhre3t/RhCLsh4SDEINzc3Cyzy/bqGxgVFRWUl5f3q2F4enoSGhpqCYzQ0FAMBoMEhhiXJByEGKbBAqO3Sao3LMrLy9myZYtl7igXFxdLYPQ+goKCcHKS/3rCtsm/UCGOw2BNUt3d3VRXV1NeXm4JjrS0NMv8UQ4ODgQFBRESEkJoaKjlp4eHh7UuQ4gBJByEGGFOTk4DOr211tTV1VnCorKykoKCAsscUmAeKRUSEtIvNAICAuTmPWEVEg5CjAGlFAEBAQQEBJCcnGzZ3tLSQmVlpSUwKioqyM/PtzRLOTk5WWoZfR9SyxCjTcJBCCvy9PQkLi6OuLg4y7bu7m5qamqorKy0PLKzs0lPT7cc4+XlRUhICMHBwZaf0pchRpL8SxLCxjg5OVk6r/tqbm62hEVVVRWVlZVs27bNcsd3b+2kd73v3uCQWWvFsZBwEGKc8PLywsvLy7IcK5gnIKytrbWERVVVFeXl5WRmZlqOcXZ2JigoiKCgoH7BIfdliMORcBBiHOsd+RQUFNSvL6Ozs5Pq6mqqqqosj7y8PDIyMizHuLq6Wpqj+gaHp6enhIaQcBDCHrm4uFhmn+2rtbXVEhqVlZVUV1ezb98+du7caTnG3d3dEhi9oREUFCShMcFIOAgxgXh4eBATE0NMTIxlm9aalpYWqqqqLMFRXV3N3r17aW9vtxzXGxqBgYH9wkOap+yThIMQE5xSytKf0XfUlNaa5ubmfoFRU1NDZmZmv9BwdXXtFxq9P2W9jPFNwkEIMSilFN7e3nh7ew8IjZaWFmpqaizBUVNTQ05OTr/htk5OTgQEBFgCozc0DAaDDLkdB+RvSAhxVPrWNPrOMwXmtb57axi9P0tKStizZ0+/9/v7+1sCo+/D3d19jK9GDEXCQQgxYtzd3QfMNQXQ1dVFbW2tJTB6H3l5eZb7NMDcJ9I3LAICAggMDMTPz0+mERljEg5CiFHn7Ow86I19JpOJhoaGfoFRW1tLVlYWra2tluMcHR0xGAyWsOj7U2obo0PCQQhhNQ4ODpalXCdPntxvX2trK7W1tf1Co7q6muzsbMvcU/BDbaN37qreh8FgwNHRcawvyW5IOAghbJKHhwceHh5ERUX12240Gqmvr7cER21tLbW1tWRnZ9PS0mI5TimFn5/fgNAICAjAx8dHRlIdgYSDEGJccXR0tPRJTJkypd++9vb2fqFRV1dHTU0NhYWFlvU04IeRVL01jL7PPTw8JDiQcBBC2BE3N7dB7wzXWtPU1GSpZfQ+KisrycrK6tdM5ebmZgmM3iav3ucTqX9DwkEIYfeUUvj4+ODj48OkSZP67evtFO8bGnV1dRQVFbF79+5+x7q7u/cLjb7h4ebmNpaXNOokHIQQE1rfTvHExMR++7q7uy39G3V1ddTW1lJfX8+BAwf6reIH/YPD39+/X3CMxxqHhIMQQgyhdyW+oKCgAfu6urqor6+nrq6u32Ow4Ohtqjo0OAwGg81OaCjhIIQQx8DZ2dkyzfmh+tY4+gZIaWkpe/fuRWvd73P6hkbf8PDx8bHazX9WCQel1MPA+UAnkAdcr7Vu6Nn3G+BGwAis0Fp/ao0yCiHEsTpcjcNoNNLQ0EBdXV2/AOmdn6rvHeMODg74+flZQqNvgPj7++Ps7Dx61zBqn3x4nwO/0Vp3K6X+DvwGuEcpNQ1YBiQD4cB6pdRkrbXxMJ8lhBDjhqOjo2Xo7KFMJhNNTU2W4Oj7s7i4mI6Ojn7He3l5MX36dM4+++wRL6dVwkFr/Vmfl1uBy3qeXwis0Vp3AAVKqVxgHrBljIsohBBjzsHBAV9fX3x9fQeMqtJa09bW1i806uvr8fX1HZWy2EKfww3A6z3PIzCHRa+Snm1CCDGhKaUsd40feh/HaBi1cFBKrQdCB9l1n9b6vZ5j7gO6gVd73zbI8XqQbSilbgZuBgbMACmEEOL4jFo4aK3PPNx+pdS1wHnAGfqHrvsSoO9EKpFA2RCf/yzwLMCcOXMGDRAhhBDHxipjpJRS5wD3ABdorVv77FoHLFNKuSqlJgGJwDZrlFEIISYya/U5PAW4Ap/33PyxVWt9q9Z6r1LqDSATc3PTz2WkkhBCjD1rjVZKOMy+vwB/GcPiCCGEOISsuyeEEGIACQchhBADSDgIIYQYQPWdAGq8UkpVA4XH8RGBQM0IFWe8mIjXDBPzuuWaJ46jve4YrfXACaCwk3A4Xkqp7VrrOdYux1iaiNcME/O65ZonjpG8bmlWEkIIMYCEgxBCiAEkHMyetXYBrGAiXjNMzOuWa544Ruy6pc9BCCHEAFJzEEIIMYCEgxBCiAEmdDgopc5RSu1XSuUqpe61dnlGg1IqSin1lVJqn1Jqr1JqZc92g1Lqc6VUTs9Pf2uXdTQopRyVUmlKqQ96Xtv1dSul/JRSbymlsnr+zk+y92sGUErd1fPve49SarVSys0er1sp9YJSqkoptafPtiGvUyn1m57vt/1KqaNaS3TChoNSyhF4GlgCTAOW96xhbW+6gV9qracC84Gf91znvcAXWutE4Iue1/ZoJbCvz2t7v+4ngE+01knALMzXbtfXrJSKAFYAc7TW0wFHzGvR2+N1vwicc8i2Qa+z5//5MiC55z3/6vneG5YJGw6Y16bO1Vrna607gTWY17C2K1rrcq31zp7nTZi/LCIwX+tLPYe9BFxklQKOIqVUJHAu8HyfzXZ73UopH+AU4D8AWutOrXUDdnzNfTgB7kopJ8AD8yJhdnfdWusNQN0hm4e6zguBNVrrDq11AZCL+XtvWCZyOEQAxX1e2/161UqpWCAV+A4I0VqXgzlAgGArFm20PA78GjD12WbP1x0HVAP/7WlKe14p5Yl9XzNa61LgEaAIKAcatdafYefX3cdQ13lc33ETORyGvV61PVBKeQFrgTu11getXZ7RppQ6D6jSWu+wdlnGkBMwG/i31joVaME+mlIOq6eN/UJgEhAOeCqlrrZuqWzCcX3HTeRwGPZ61eOdUsoZczC8qrV+u2dzpVIqrGd/GFBlrfKNkpOBC5RSBzA3GZ6ulHoF+77uEqBEa/1dz+u3MIeFPV8zwJlAgda6WmvdBbwNLMD+r7vXUNd5XN9xEzkcvgcSlVKTlFIumDtu1lm5TCNOmddh/Q+wT2v9aJ9d64Bre55fC7w31mUbTVrr32itI7XWsZj/br/UWl+NHV+31roCKFZKTenZdAbmJXft9pp7FAHzlVIePf/ez8Dct2bv191rqOtcByxTSrkqpSYBicC2YX+q1nrCPoClQDaQB9xn7fKM0jUuxFyV3AWk9zyWAgGYRzbk9Pw0WLuso/hncBrwQc9zu75uIAXY3vP3/S7gb+/X3HPdDwBZwB7gZcxr1NvddQOrMferdGGuGdx4uOsE7uv5ftsPLDmac8n0GUIIIQaYyM1KQgghhiDhIIQQYgAJByGEEANIOAghhBhAwkEIIcQAEg5CHIOe2U9v63kerpR6y9plEmIkyVBWIY5BzzxVH2jzLKBC2B0naxdAiHHqISBeKZWO+eajqVrr6Uqp6zDPiukITAf+CbgA1wAdwFKtdZ1SKh7zlPFBQCtwk9Y6a6wvQoihSLOSEMfmXiBPa50C3H3IvunAjzFPj/wXoFWbJ8LbAvyk55hngTu01icAvwL+NRaFFmK4pOYgxMj7SpvXzmhSSjUC7/ds3w3M7JkhdwHwpnkqIMA83YMQNkPCQYiR19HnuanPaxPm/3MOQENPrUMImyTNSkIcmybA+1jeqM3raRQopS4H88y5SqlZI1k4IY6XhIMQx0BrXQts7lno/eFj+IirgBuVUhnAXuxwiVoxvslQViGEEANIzUEIIcQAEg5CCCEGkHAQQggxgISDEEKIASQchBBCDCDhIIQQYgAJByGEEAP8PzIgaIBXQT9BAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "walk.plot()\n",
    "confidence.plot(c='gray')\n",
    "(-1*confidence).plot(c='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MAE test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a worse forecast with high but different to perfect correlation\n",
    "fct_1d_worse=fct_1d.copy()\n",
    "step=3\n",
    "fct_1d_worse[::step]=fct_1d[::step].values+.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray 'var' ()>\n",
      "array(0.00966918)\n",
      "<xarray.DataArray 'var' ()>\n",
      "array(0.01083478)\n",
      "MAEs significantly different at level 0.05 : False\n"
     ]
    }
   ],
   "source": [
    "# half-with of the confidence interval at level alpha is larger than the MAE differences,\n",
    "# therefore not significant\n",
    "alpha = 0.05\n",
    "significantly_different, diff, hwci = xs.mae_test(fct_1d, fct_1d_worse, obs_1d, time_dim='time', dim=[], alpha=alpha)\n",
    "print(diff)\n",
    "print(hwci)\n",
    "print(f\"MAEs significantly different at level {alpha} : {bool(significantly_different)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessors\n",
    "\n",
    "You can also use `xskillscore` as a method of your `xarray` Dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xr.Dataset()\n",
    "ds[\"obs_var\"] = obs\n",
    "ds[\"fct_var\"] = fct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case that your Dataset contains both your observation and forecast variable, just pass them as strings into the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray (lat: 4, lon: 5)>\n",
      "array([[ 0.99509676, -0.88499394,  0.94083077,  0.96521259, -0.13696899],\n",
      "       [-0.90613709,  0.51585291,  0.72875703,  0.19331043,  0.79754067],\n",
      "       [-0.80112059, -0.95632624, -0.23640403, -0.57684283,  0.43389289],\n",
      "       [ 0.00230351, -0.58970109, -0.87332763, -0.99992557, -0.31404248]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "print(ds.xs.pearson_r(\"obs_var\", \"fct_var\", dim=\"time\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also pass in a separate Dataset that contains your observations or forecast variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.DataArray (lat: 4, lon: 5)>\n",
      "array([[ 0.99509676, -0.88499394,  0.94083077,  0.96521259, -0.13696899],\n",
      "       [-0.90613709,  0.51585291,  0.72875703,  0.19331043,  0.79754067],\n",
      "       [-0.80112059, -0.95632624, -0.23640403, -0.57684283,  0.43389289],\n",
      "       [ 0.00230351, -0.58970109, -0.87332763, -0.99992557, -0.31404248]])\n",
      "Coordinates:\n",
      "  * lat      (lat) int64 0 1 2 3\n",
      "  * lon      (lon) int64 0 1 2 3 4\n"
     ]
    }
   ],
   "source": [
    "ds = ds.drop_vars(\"fct_var\")\n",
    "print(ds.xs.pearson_r(\"obs_var\", fct, dim=\"time\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resampling\n",
    "-   randomly resample the `time` dimension and then take mean over `time` to get resample threshold\n",
    "-   resample over `member` dimension to get uncertainty due to member sampling in hindcasts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create large one-dimensional array\n",
    "s = 1000\n",
    "f = xr.DataArray(np.random.normal(size=s), dims='member', coords={'member':np.arange(s)}, name='var')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65.1 ms ± 1.78 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
      "1.44 ms ± 41.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "# resample with replacement in that one dimension\n",
    "iterations = 100\n",
    "%timeit f_r = xs.resampling.resample_iterations(f, iterations, 'member', replace=True)\n",
    "# resample_iterations_idx is much (50x) faster because it involves no loops\n",
    "%timeit f_r = xs.resampling.resample_iterations_idx(f, iterations, 'member', replace=True)\n",
    "# but both do the same resampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- use `resample_iterations` for very large data, because very robust, chunksize stays contants and only more tasks are added\n",
    "- use `resample_iterations_idx` for small data always and very large data only, when chunked to small chunks in the other dimensions, because the function increases the input chunksize by factor `iterations`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ff0d3119580>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlVUlEQVR4nO3deXxU5dn/8c8VtoBsCqgI1KBlEUgMyL5GRcGC4AIKdYFqRVwrtiiiP8EFpYobLuXBpxZ8FLdWK0qrgBoRAZElIgIKSBQEAdEAQbAErt8fc0gnIftCksP3/Xrlxcw59zn3dZLhO2fuOXOPuTsiIhIuMWVdgIiIlDyFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXY4oM/u3mQ07Av3EmZmbWeWS7tfMepjZl1H3U82sd0nsO9jfF2aWVFL7k6OT6Tr3cDOzIcAooA2wB9gATAf+4iH+45tZHJFjreLuGYXYzoFm7r6uENukAr9397lFqHMasMnd7yrstiJ50Zl7iJnZH4EngIeBE4ETgJFAN6BqGZZWYR16JSBS7rm7fkL4A9QhcqZ+cT7t+gHLgV3ARmB81LokImeV0e1Tgd7B7Y7AkmDbrcCjwfJY4AVgB5AGfAqcEKxLJnKWC3Aq8H7Q7gfgRaButr7+BKwAdgKvALG5HEclYFKwn6+BGwAHKufQ76+BD4N9/gC8EiyfF2yzB0gHLj30OwBuB74H/i/77yWo8w5gFfAT8LdDdQLDgfnZavWghhHAfuA/QX9v5fA7rgY8DmwOfh4HqkX/fYA/AtuALcDv8vhbJwP3AwsO9QfUC37vu4K/U1xU+5bAHOBH4EvgkgI+buKCYxwGfBv8ju8s6/8TR9uPztzDqwuRYHgzn3Z7gCuBukT+w15nZhcUsI8ngCfcvTaRoH41WD6MyJNLEyLhMRLYm8P2BjwInAScFrQfn63NJUBfoCmQQCQsc3IN0B9oC7QHBuVR933AbOBYoDHwJIC79wzWn+7uNd39leD+icBxwMlEAjknlwF9iPwemgP5DrO4+1QiwfpQ0N/5OTS7E+gMJAKnE3lCjd73iUR+142Aq4GnzezYPLodAlwRtD8VWEjkyeg4YDUwDsDMjiES7DOA44GhwDNm1jrYT0EeN92BFsDZwN1mdloedUkJU7iHV33gB48abzazBWaWZmZ7zawngLsnu/vn7n7Q3VcALwG9CtjHfuDXZlbf3dPdfVHU8nrAr939gLsvdfdd2Td293XuPsfdf3H37cCjOfQ92d03u/uPRM40E3Op5RLgcXffGLR9MJ+6TwZOcvd97j4/n+M8CIwL6szpSQrgqai+JxAJw5JwGXCvu28Lfkf3EAnnQ/YH6/e7+7+InJG3yGN/f3P39e6+E/g3sN7d5waPk9eIPDlC5Iky1d3/5u4Z7r4M+AfBk2YBHzf3uPted/8M+IzIk5McIQr38NoB1I8eI3b3ru5eN1gXA2BmnczsAzPbbmY7iZxl1y9gH1cTOUtdY2afmln/YPn/Ae8CL5vZZjN7yMyqZN/YzI43s5fN7Dsz20VkKCd7399H3f4ZqJlLLScRGR445Js86r6NyKuGxcGVKVfl0RZgu7vvy6dN9r5Pyqd9QZ1E1mPJvu8dnvUN47x+RxAZPjtkbw73D217MtApOBlIM7M0Ik80J0KBHzcF/dtJKVC4h9dC4BdgYD7tZgAzgSbuXgeYQiT4IPLSu8ahhmZWCWhw6L67r3X3oURetv8Z+LuZHROcRd7j7q2ArkTOAq/Moe8HiYzNJgRDO5dH9V1YW4gM6xzyq9wauvv37n6Nu58EXEtkuOHXeey7IFcVZe97c3A7++/wxELuezORoM1p36VpI/Chu9eN+qnp7tcF6/N63Eg5oHAPKXdPI/IS/hkzG2RmNc0sxswSgWOimtYCfnT3fWbWEfht1LqvgFgz6xeced9FZBwfADO73MwauPtBIm+cAhwwszPNLD54MthFZOjgQA5l1iIyjJBmZo2A0cU45FeBm82scTDmPCa3hmY22MwaB3d/IhKwh+rbCpxShP5vCPo+DhhL5M1fiAxHtDazRDOL5fD3FPLr7yXgLjNrYGb1gbuJvMIpbW8Dzc3sCjOrEvx0iBo3z+txI+WAwj3E3P0h4FYiwxDbiATJ/xC58mNB0Ox64F4z200kOF6N2n5nsP5/ge+InIVuiuqiL/CFmaUTeXN1SDB8cSLwdyLBvprIlSk5BdI9QDsiV63MAl4vxuE+S2Qo6DNgWT776gB8EtQ9E/iDu28I1o0HpgdDEZcUov8ZRN6k/Tr4uR/A3b8C7gXmAmuB7OP7fwVaBf39M4f93k/kiqQVwOfBsd1fiLqKxN13A+cSeQN2M5Ehlj/z3yf3XB83Uj7oQ0wiIiGkM3cRkRBSuIuIhJDCXUQkhBTuIiIhVC4mQapfv77HxcWVdRlylPvyy8gsvi1a5PUBT5HyY+nSpT+4e4Oc1pWLcI+Li2PJkiVlXYYc5ZKSkgBITk4u0zpECsrMcv0ktoZlRERCSOEuIhJCCncRkRAqF2PuIqVh//79bNq0iX378pvQMWLcuHEArF69ujTLEim02NhYGjduTJUqh02umiuFu4TWpk2bqFWrFnFxcZjlP2FhTEzkhayulpHyxN3ZsWMHmzZtomnTpgXeTsMyElr79u2jXr16BQp2kfLKzKhXr16BX4EeonCXUFOwSxgU5XGscBcRCSGFu8gRMn78eCZNmgTA3Xffzdy5c3Nt+89//pNVq1blun7KlCk8//zzQOTDV4X5EGBaWhrPPPNM5v3NmzczaFBe3ycuFZHCXQRg83L4ZXfkZ/PyUu/u3nvvpXfv3rmuzyvcMzIyGDlyJFdemdM3F+Yve7ifdNJJ/P3vfy/SvqT8UriLlKIJEybQokULevfunTl3DcDw4cMzA3XMmDG0atWKhIQE/vSnP7FgwQJmzpzJ6NGjSUxMZP369SQlJTF27Fh69erFE088keVVAMALL7xA165dadOmDYsXLwY4rE2bNm1ITU1lzJgxrF+/nsTEREaPHk1qaipt2rQBIm9C/+53vyM+Pp62bdvywQcfADBt2jQuuugi+vbtS7NmzbjttttyPN64uDjGjh1Lly5daN++PcuWLaNPnz6ceuqpTJkyJbPdww8/TIcOHUhISMi8BBXgggsu4IwzzqB169ZMnTo1c3nNmjW58847Of300+ncuTNbt25F8pbvpZBm1gR4nshXpx0Eprr7E2Y2HrgG2B40Hevu/wq2uQO4msj3Ut7s7u+WQu0iBXbLLbeQkpKSe4P/pPPz/si3ktWoYlC1Zr77TExM5PHHH891/dKlS3n55ZdZvnw5GRkZtGvXjjPOOCNLmx9//JE33niDNWvWYGakpaVRt25dBgwYQP/+/bMMl6SlpfHhhx8CkeCOtmfPHhYsWMC8efO46qqrWLlyZa51TZw4kZUrV2b+PlJTUzPXPf300wB8/vnnrFmzhnPPPZevvvoKgJSUFJYvX061atVo0aIFN910E02aNMm+e5o0acLChQsZNWoUw4cP5+OPP2bfvn20bt2akSNHMnv2bNauXcvixYtxdwYMGMC8efPo2bMnzz33HMcddxx79+6lQ4cOXHzxxdSrV489e/bQuXNnJkyYwG233cazzz7LXXfdlesxSsGuc88A/ujuy8ysFrDUzOYE6x5z90nRjc2sFZHvXWwNnATMNbPm7p7TFySLhNZHH33EhRdeSI0aNQAYMGDAYW1q165NbGwsv//97+nXrx/9+/fPdX+XXnppruuGDh0KQM+ePdm1axdpaWlFqnn+/PncdNNNALRs2ZKTTz45M9zPPvts6tSpA0CrVq345ptvcgz3Q8cZHx9Peno6tWrVolatWsTGxpKWlsbs2bOZPXs2bdu2BSA9PZ21a9fSs2dPJk+ezBtvvAHAxo0bWbt2LfXq1aNq1aqZv5szzjiDOXPmHNavZJVvuLv7FmBLcHu3ma0GGuWxyUDgZXf/BdhgZuuAjsDCEqhXpEjyOsMGYPNyvvwhcv7Ron4lOKltifSb3yVslStXZvHixbz33nu8/PLLPPXUU7z//vs5tj3mmGMK3I+ZUblyZQ4ePJi5rCDXSef1ncrVqlXLvF2pUiUyMjLybBcTE5Nlm5iYGDIyMnB37rjjDq699tos2yUnJzN37lwWLlxIjRo1SEpKyqy5SpUqmceYV9/yX4UaczezOKAt8Emw6EYzW2Fmz5nZscGyRsDGqM02kcOTgZmNMLMlZrZk+/bt2VeLVHg9e/bkjTfeYO/evezevZu33nrrsDbp6ens3LmT3/zmNzz++OOZQyW1atVi9+7dBe7rlVdeASJn3nXq1KFOnTrExcWxbNkyAJYtW8aGDRvy3XfPnj158cUXAfjqq6/49ttvS/wTu3369OG5554jPT0dgO+++45t27axc+dOjj32WGrUqMGaNWtYtGhRifZ7tCnw9ANmVhP4B3CLu+8ys78A9wEe/PsIcBWQ06nKYacD7j4VmArQvn373E8XRCqodu3acemll5KYmMjJJ59Mjx49Dmuze/duBg4cyL59+3B3HnvsMQCGDBnCNddcw+TJkwt0Jcuxxx5L165d2bVrF8899xwAF198Mc8//zyJiYl06NCB5s2bA1CvXj26detGmzZtOO+887jhhhsy93P99dczcuRI4uPjqVy5MtOmTcty9l0Szj33XFavXk2XLl2AyJulL7zwAn379mXKlCkkJCTQokULOnfuXKL9Hm0sr5dhmY3MqgBvA++6+6M5rI8D3nb3NsGbqbj7g8G6d4Hx7p7rsEz79u1dX9YhJW316tWcdtppBWtcSsMyIiUlp8ezmS119/Y5tc93WMYiA11/BVZHB7uZNYxqdiFw6O35mcAQM6tmZk2BZsDiQh2FiIgUS0GGZboBVwCfm1lKsGwsMNTMEokMuaQC1wK4+xdm9iqwisiVNjfoShkRkSOrIFfLzCfncfR/5bHNBGBCMeoSEZFi0CdURURCSOEuIhJC+iYmOXoVcIKwFZvSSreOXCQ0rlsm/Uo4KNzlqBE3ZlYhWm/Ot8XMG7sVqv+/PDqRGjWOIT19N2d06krnHkk5tnv/nVmcfMqpJDTO+TrvKVOmUKNGDa688kqSkpKYNGkS7dvneDXcYdLS0pgxYwbXX389EJnu9+abb9askCGkYRmRI+yGP43NNdgBPnh3Fl+v/TLHdZruVwpK4S5Sip6dPIkBvTowYugFpK5fC8D/G3U9c2a9CcDjD47nwrM6M+icbjxy3/8jZcknJM/5N49OuLvCTfcr5YuGZURKyaoVKbwz83VeeedDDmRkMOS8JFrFJ2au3/nTT7z/zizeTF6MmbFr505q16lD0jnn0bN3H/547bDMthVlul8pP3TmLlJKli1eyFl9+1O9eg1q1qpNr3POy7L+mFq1qFatGuNH38zcf79F9erVc93XkZru94orrgByn+43NjY2c7pfKd8U7iKlKK8pfytXrsyLb71H79+czwfvzuK6K3L/HtOKMt2vlB8Kd5FSckanrrz/ztvs27uXPem7mTf3nSzrf96Tzu7du+hx1rncNu5BvvzicwBq1KzJnmA63IKoSNP9ypGjMXc5aqRO7Jd1Qbbr3HObFbKo17mfFn86fc6/kEv69qRhoya07dgly/o96en84erL+M8vkel+R497AIC+Ay7i3ttv4Y0X/jd00/3KkVOgKX9Lm6b8lfwU7hr1iGcHNOSEX52S6/qEmA1Z7keH+4qDTQvdX0nTh5gkWolP+SsiIhWPwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJI17nLUSPhf0/Oc330x3USCrC/Fb+vmB/BL+w0wcnJyUyaNIm33367lCsrumnTprFkyRKeeuqpLFMi5yQ5OZmqVavStWvXHNfPnDmTVatWMWbMGIYPH07//v0ZNCj3Tw9n98ADDzB27NjM+127dmXBggWFO6ASoHAXOQLcHXcnJkYvlkvbyJEj81yfnJxMzZo1cwz3jIwMBgwYwIABA4rcf/ZwL4tgBw3LiJSa7zZ+ywVndmLC2D9y6Xm9+H7zJqZNmcxv+53FoHO68cwjDwLw8897uHHYJQw+tzsXnd2Fd2a+DsC9995Lhw4daNOmDSNGjMic+yUpKYlRo0bRs2dPTjvtND799FMuuugimjVrxl133QVEZnts2bIlw4YNIyEhgUGDBvHzzz8fVuPs2bPp0qUL7dq1Y/DgwaQH0x688847tGzZku7du/P666/neHzTpk3jggsu4Pzzz6dp06Y89dRTPProo7Rt25bOnTvz448/ArB+/Xr69u3LGWecQY8ePVizZg0Ab731Fp06daJt27b07t2brVu3ApFZL6+66iqSkpI45ZRTmDx5co79/+1vf6N58+b06tWLjz/+OHN59HTHkydPplWrViQkJDBkyBBSU1OZMmUKjz32GImJiXz00UcMHz6cW2+9lTPPPJPbb7+dadOmceONN2bub+7cufTo0YPmzZtnvnrJ3qZ///4kJyczZswY9u7dS2JiIpdddhkANWvWBCJP8KNHj6ZNmzbEx8dnThuRnJxMUlISgwYNomXLllx22WV5zvNTUAp3kVKUun4t5w8awqvvzCN1/Tq+3fA1L779Hq+++xGrPk9h6aKPWZD8Hg1OaMhrs+fz+nsL6ZZ0NgA33ngjn376KStXrmTv3r1ZhkWqVq3KvHnzGDlyJAMHDuTpp59m5cqVTJs2jR07dgDw5ZdfMmLECFasWEHt2rWzfEkHwA8//MD999/P3LlzWbZsGe3bt+fRRx9l3759XHPNNbz11lt89NFHfP/997ke38qVK5kxYwaLFy/mzjvvpEaNGixfvpwuXbrw/PPPAzBixAiefPJJli5dyqRJkzK/Bap79+4sWrSI5cuXM2TIEB566KHM/a5Zs4Z3332XxYsXc88997B///4s/W7ZsoVx48bx8ccfM2fOHFatWpVjfRMnTmT58uWsWLGCKVOmEBcXx8iRIxk1ahQpKSn06NEDiMylM3fuXB555JHD/4apqXz44YfMmjWLkSNH5jkJ28SJE6levTopKSmZ8/Qc8vrrr5OSksJnn33G3LlzGT16NFu2bAFg+fLlPP7446xatYqvv/46y5NVUSncRUpRw8ZNSGjXAYCF8z5g4bz3ubRvT4ac14vUdWv5JvVrft2yFYvmJ/PYA+NY9skCatWuA8AHH3xAp06diI+P5/333+eLL77I3O+hYYP4+Hhat25Nw4YNqVatGqeccgobN24EoEmTJnTrFvkqwMsvv5z58+dnqW3RokWsWrWKbt26kZiYyPTp0/nmm29Ys2YNTZs2pVmzZpgZl19+ea7Hd+aZZ1KrVi0aNGhAnTp1OP/88zPrSk1NJT09nQULFjB48GASExO59tprMwNt06ZN9OnTh/j4eB5++OEsx9evXz+qVatG/fr1Of744zPP6g/55JNPSEpKokGDBlStWjXXKZETEhK47LLLeOGFF6hcOfdR6MGDB1OpUqUc111yySXExMTQrFkzTjnllMxXHoU1f/58hg4dSqVKlTjhhBPo1asXn376KQAdO3akcePGxMTEkJiYmGWe/aLSmLtIKapevUbmbXfnqhtGMfjy3x3W7uVZyXz0wWye+PO9dOl5Fr8beTPXX389S5YsoUmTJowfPz7LGeOhCb1iYmKyTO4VExOTOR1vTlMBR3N3zjnnHF566aUsy1NSUvKcqjha9r6j68rIyODgwYPUrVs388tBot10003ceuutDBgwgOTk5CxfQlKQKYYLUuOsWbOYN28eM2fO5L777svyBBItjFMq68xd5Ajp2uss/vnKi/y8JzKuvXXLZnb8sJ1t328htnp1+l90KcNG3MSazz/jl19+AaB+/fqkp6cX6XtOv/32WxYuXAjASy+9RPfu3bOs79y5Mx9//DHr1q0D4Oeff+arr76iZcuWbNiwgfXr12duW1S1a9emadOmvPbaa0Ak4D777DMAdu7cSaNGjQCYPn16ofbbqVMnkpOT2bFjB/v378/cf7SDBw+yceNGzjzzTB566CHS0tJIT0/Pc9rjnLz22mscPHiQ9evX8/XXX9OiRQvi4uJISUnJ7OPQ1xsCVKlS5bBhJIhMqfzKK69w4MABtm/fzrx58+jYsWOhjrswdOYuR43sly7mNStklu1KaIbIrr3OYsO6r7hi4LkA1DimJg888T98m/o1j024m5iYGCpXrsKdDzxC7Tp1uOaaa4iPjycuLo4OHToUur/TTjuN6dOnc+2119KsWTOuu+66LOsbNGjAtGnTGDp0aOaTyf3330/z5s2ZOnUq/fr1o379+nTv3j3Pr+7Lz4svvsh1113H/fffz/79+xkyZAinn34648ePZ/DgwTRq1IjOnTtnzjlfEA0bNmT8+PF06dKFhg0b0q5dOw4cOJClzYEDB7j88svZuXMn7s6oUaOoW7cu559/PoMGDeLNN9/kySefzLevFi1a0KtXL7Zu3cqUKVOIjY2lW7duNG3alPj4eNq0aUO7du0y248YMYKEhATatWuXZdz9wgsvZOHChZx++umYGQ899BAnnnhikYd58qMpf6VCONJT/kYrq+l/izPlb2pqKv379y9WKEv5oil/RURE4S4SRnFxcTprP8op3CW0HC+RD4OIlLWiPI4V7hJa36TtJ+PnXQp4qdDcnR07dhAbG1uo7fK9WsbMmgDPAycCB4Gp7v6EmR0HvALEAanAJe7+U7DNHcDVwAHgZnd/t1BViZSAJz/5iZuAk+v+gHH4NdGrbXuW+9+nR54EDu7J2nZrGT05rN5dvUz6lfInNjaWxo0bF2qbglwKmQH80d2XmVktYKmZzQGGA++5+0QzGwOMAW43s1bAEKA1cBIw18yau/uBXPYvUip2/XKQCfN25Lo+Nfa3We5fN20PAMnDs36g5bx9M0q+uAJIndivTPqVcMh3WMbdt7j7suD2bmA10AgYCBz65MF04ILg9kDgZXf/xd03AOuA0rtSX0REDlOoMXcziwPaAp8AJ7j7Fog8AQDHB80aARujNtsULBMRkSOkwOFuZjWBfwC3uPuuvJrmsOywQUszG2FmS8xsyfbt23PYREREiqpA4W5mVYgE+4vufmhy561m1jBY3xDYFizfBDSJ2rwxsDn7Pt19qru3d/f2DRo0KGr9IiKSg3zD3SJTov0VWO3uj0atmgkMC24PA96MWj7EzKqZWVOgGbAYERE5YgpytUw34ArgczNLCZaNBSYCr5rZ1cC3wGAAd//CzF4FVhG50uYGXSkjInJk5Rvu7j6fnMfRAc7OZZsJwIRi1CUiIsWgT6iKiISQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIF+RCTSKaifFG1iBx5OnMXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUriLiISQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhlG+4m9lzZrbNzFZGLRtvZt+ZWUrw85uodXeY2Toz+9LM+pRW4SIikruCnLlPA/rmsPwxd08Mfv4FYGatgCFA62CbZ8ysUkkVKyIiBZNvuLv7PODHAu5vIPCyu//i7huAdUDHYtQnIiJFUJwx9xvNbEUwbHNssKwRsDGqzaZg2WHMbISZLTGzJdu3by9GGSIikl1Rw/0vwKlAIrAFeCRYbjm09Zx24O5T3b29u7dv0KBBEcsQEZGcFCnc3X2rux9w94PAs/x36GUT0CSqaWNgc/FKFBGRwipSuJtZw6i7FwKHrqSZCQwxs2pm1hRoBiwuXokiIlJYlfNrYGYvAUlAfTPbBIwDkswskciQSypwLYC7f2FmrwKrgAzgBnc/UCqVi4hIrvINd3cfmsPiv+bRfgIwoThFiYhI8egTqiIiIZTvmbuIlI24MbPKrO/Uif3KrG8pGTpzFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUriLiIRQ5bIuQORISo39bVmXIHJE6MxdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJC+Ya7mT1nZtvMbGXUsuPMbI6ZrQ3+PTZq3R1mts7MvjSzPqVVuIiI5K4gZ+7TgL7Zlo0B3nP3ZsB7wX3MrBUwBGgdbPOMmVUqsWpFRKRA8g13d58H/Jht8UBgenB7OnBB1PKX3f0Xd98ArAM6lkypIiJSUEUdcz/B3bcABP8eHyxvBGyMarcpWHYYMxthZkvMbMn27duLWIaIiOSkpN9QtRyWeU4N3X2qu7d39/YNGjQo4TJERI5uRQ33rWbWECD4d1uwfBPQJKpdY2Bz0csTEZGiKGq4zwSGBbeHAW9GLR9iZtXMrCnQDFhcvBJFRKSw8p0V0sxeApKA+ma2CRgHTAReNbOrgW+BwQDu/oWZvQqsAjKAG9z9QCnVLiIiucg33N19aC6rzs6l/QRgQnGKEhGR4tEnVEVEQkjhLiISQgp3EZEQUriLiISQwl1EJIQU7iIiIaRwFxEJoXyvcxc52qXG/jbL/bh9M8qoEpGC05m7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSNMPVEBxY2aVdQkiUs7pzF1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICBVr4jAzSwV2AweADHdvb2bHAa8AcUAqcIm7/1S8MkVEpDBK4sz9THdPdPf2wf0xwHvu3gx4L7gvIiJHUGkMywwEpge3pwMXlEIfIiKSh+KGuwOzzWypmY0Ilp3g7lsAgn+PL2YfIiJSSMX9so5u7r7ZzI4H5pjZmoJuGDwZjAD41a9+VcwyRKQkldUXwqRO7Fcm/YZRscLd3TcH/24zszeAjsBWM2vo7lvMrCGwLZdtpwJTAdq3b+/FqUMkN6mxvy3rEkTKRJGHZczsGDOrdeg2cC6wEpgJDAuaDQPeLG6RIiJSOMU5cz8BeMPMDu1nhru/Y2afAq+a2dXAt8Dg4pcpIiKFUeRwd/evgdNzWL4DOLs4RYmISPHoE6oiIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQmh4k75e1Qrq2lRRUTyozN3EZEQUriLiISQhmVEpNzQN0CVHJ25i4iEkM7cRQop+qv74vbNKMNKRHKnM3cRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQmhUFwto2kARESy0pm7iEgIKdxFREJI4S4iEkIKdxGREArFG6oi0aKnBxA5WincReSoV5ZX3JXWjJQKd5Fi0CRiUl5pzF1EJIQU7iIiIaRwFxEJoVILdzPra2Zfmtk6MxtTWv2IiMjhSuUNVTOrBDwNnANsAj41s5nuvqo0+hPR5Y8iWZXWmXtHYJ27f+3u/wFeBgaWUl8iIpJNaV0K2QjYGHV/E9ApuoGZjQBGBHfTzezLPPZXH/ihRCssO2E6Fignx2Mls5v6wA92z64ibt6/ZKooGeXi71KCwnQ8WY7F/lysfZ2c24rSCvec/q95ljvuU4GpBdqZ2RJ3b18ShZW1MB0LhOt4dCzlV5iO50gdS2kNy2wCmkTdbwxsLqW+REQkm9IK90+BZmbW1MyqAkOAmaXUl4iIZFMqwzLunmFmNwLvApWA59z9i2LsskDDNxVEmI4FwnU8OpbyK0zHc0SOxdw9/1YiIlKh6BOqIiIhpHAXEQmhChPuZnafma0wsxQzm21mJ5V1TUVlZg+b2ZrgeN4ws7plXVNRmdlgM/vCzA6aWYW8VC1MU2WY2XNmts3MVpZ1LcVlZk3M7AMzWx08xv5Q1jUVlZnFmtliM/ssOJZ7Sr3PijLmbma13X1XcPtmoJW7jyzjsorEzM4F3g/eeP4zgLvfXsZlFYmZnQYcBP4H+JO7LynjkgolmCrjK6KmygCGVtSpMsysJ5AOPO/ubcq6nuIws4ZAQ3dfZma1gKXABRXxb2NmBhzj7ulmVgWYD/zB3ReVVp8V5sz9ULAHjiHbh6IqEnef7e4Zwd1FRD4HUCG5+2p3z+vTxeVdqKbKcPd5wI9lXUdJcPct7r4suL0bWE3k0+8VjkekB3erBD+lmmEVJtwBzGyCmW0ELgPuLut6SshVwL/LuoijWE5TZVTIAAkzM4sD2gKflHEpRWZmlcwsBdgGzHH3Uj2WchXuZjbXzFbm8DMQwN3vdPcmwIvAjWVbbd7yO5agzZ1ABpHjKbcKciwVWL5TZUjZMrOawD+AW7K9gq9Q3P2AuycSeaXe0cxKddisXH2Hqrv3LmDTGcAsYFwpllMs+R2LmQ0jMtPU2V7O3/goxN+lItJUGeVYMD79D+BFd3+9rOspCe6eZmbJQF+g1N74Lldn7nkxs2ZRdwcAa8qqluIys77A7cAAd/+5rOs5ymmqjHIqeBPyr8Bqd3+0rOspDjNrcOiqODOrDvSmlDOsIl0t8w+gBZErM74BRrr7d2VbVdGY2TqgGrAjWLSoAl/5cyHwJNAASANS3L1PmRZVSGb2G+Bx/jtVxoSyrajozOwlIInItLJbgXHu/tcyLaqIzKw78BHwOZH/9wBj3f1fZVdV0ZhZAjCdyGMsBnjV3e8t1T4rSriLiEjBVZhhGRERKTiFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhP4/E70d3kBcI0cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f_r = xs.resampling.resample_iterations_idx(f, iterations, 'member', replace=True)\n",
    "f.plot.hist(label='distribution')\n",
    "f_r.mean('iteration').plot.hist(label='resampled mean distribution')\n",
    "plt.axvline(x=f.mean('member'), c='k', label='distribution mean')\n",
    "plt.title('Gaussian distribution mean')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ff0d30e0b50>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkgUlEQVR4nO3deXhU5d3/8feXRUAFRUEBQQMWBCQQfSJiWYygIoIiFCq4sViBVrTQ9vcUbVWKG7Yo1V+xfdB6BZ4iiCsuLVjRFFApBhtZBBUlatiJGyggy/f545yESZgkkx0On9d1zZU5+31OZj5zn3vO3MfcHRERiZYa1V0AERGpeAp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIX7EcrMJprZ36q7HEcDMzvdzHaaWc0q2Fb+/7Wit2tmfzGzO8LnaWaWUxHrDdfX3cw+qKj1VRYzyzCzn1R3OaqCwj0BZpZtZrvCN9pmM0s3s+NjpqebmZvZlYWW+2M4fng4fIyZPWhmOeG61pvZ1CK2k/f4UyXsjz4YSsHdP3P34919/+G4XTMbbmZLEljfGHe/uyLKFr6ufxCz7sXuflZFrLs6mdn48D3+tZk9YWZ1qrtMZaVwT9wV7n48kAKcA9xWaPqHwLC8ATOrBQwGPo6Z5zYgFegM1AcuAv4Tbzsxj7EVuhdyVKqKs44jnZn1BiYAvYAkoBXwu+osU3ko3EvJ3TcDCwhCPtZLQFczaxgOXwasADbHzHMe8Ly7b/RAtrvPLEdx6prZU2a2w8zeNbNOeRPMrJmZPWtm28IzhFvD8ZcBtwNXh2cG75nZRWa2MmbZ18xsWczwEjO7qrj1htNqmNkEM/vYzHLNbK6ZnRROSwpre8PM7DMz225mvylqx8zsZDN7ycy+MbN3zOye2NqpmT1sZp+H05ebWfeYaelmdk/McIEmCDP7tZltCI/bB2bWKxzf2cwyw3VuMbOHCpW9Vjg8wszWhMt/YmajC2/LzH5pZlvNbJOZjShmP1ua2b/Cdf0TaBQzrfB2h4fb2xEe+2vNrB3wF+CC8P/5Vcwx+LOZ/d3MvgUuKnxcwvluD/8X2WZ2bcz4As0XsWcHZrYoHP1euM2r4xzjduE6vjKz1RZzVhuWY5qZvRLuy7/N7MxijtHTdrA2vcjMzk50XWZ2iZmtDZf9E2BFbYegcvZXd1/t7l8CdwPDi5n/sKZwLyUzaw70AdYVmrQbeBEYEg7fABQO7qXAL8zsZ2aWbGbFvdAS0R94GjgJeBJ4wcxqm1kNgg+b94DTCGoi48yst7vPB+4DngrPDDoBbwM/MLNGYZB0AJqbWX0zqwf8F7C4uPWG5bkVuAq4EGgGfAlMK1TmbsBZ4bJ3huEUzzTgW6AJwZtuWKHp7xB8wObt+9NmVrekA2ZmZwFjgfPcvT7QG8gOJz8MPOzuDYAzgblFrGYr0A9oAIwApprZuTHTmwAnEByjG4FpdvBDv7AngeUEoX53nP3MK/dxwCNAn7DcPwSy3H0NMAZ4O/x/nhiz2DXAvQRnifGabZqE2z0t3O708PgUy917hE87hdt8qlBZaxO8Tl4FTgFuAWYVWvdQglpxQ4L30r3FbPIfQOtwXe8CswpNj7suM2sEPAv8NtzPj4GuxWznbILXdp73gFPN7ORiljlsKdwT94KZ7QA+J3hz3xVnnpnADWZ2AkHAvVBo+v3AA8C1QCawwcwKv5lfCGs7eY+biinTcnd/xt33Ag8BdYEuBGcIjd19krt/7+6fAI9x8IOnAHffHZanB0Gz0QqCMOgaru8jd89NYL2jgd+4e4677wEmAoPyap6h37n7Lnd/j+DN04lCLGhC+BFwl7t/5+7vAzMKlflv7p7r7vvc/UGgDsGHRkn2h/O2N7Pa4dlTXtPZXsIPOXff6e5Lizher7j7x+HZ178IQqx7zCx7gUnuvtfd/w7sjFc2Mzud4Jje4e573H0RQSgW5QDQwczqufsmd19dwr7Oc/c33f1A+D+OJ2/b/wJeAX5cwjoT0QU4Hpgcvk5eB14mCOE8z7n7MnffRxDWKUWtzN2fcPcdMa+pTuF7rKR1XQ68H/Me+SMFz6QLOx74OmY473n94nb2cKVwT9xVYY0pDWhLzOlzHndfAjQmqCm87O67Ck3f7+7T3L0rcCJBDeOJQrXXq9z9xJjHY8WU6fOYdR8AcghqzGcAzWI/JAiaYk4tZl3/CvetR/g8g+AD6sJwmATWewbwfMy0NQRhGrvd2DfXdwRvqMIaA7Vi96/Qc8JmjzXh6fZXBDXlQ/4nhbn7OmAcQUhsNbM5ZtYsnHwj0AZYa0FTUL946zCzPma21My+CLd9eaFt54ZBU9J+NgO+dPdvY8Z9WkS5vwWuJqilbwqbIdoWv7cFj1kc8bbdrKiZS6EZ8Hn4moxd92kxw4m8DjCzmmY22YKmvm84eJYVe7yLWlczCr5HnOKPyU6Cs7E8ec93FLPMYUvhXkphDScdmFLELH8DfsmhTTKF17PL3acRNF20L2NxWuQ9CZtMmgMbCV7A6wt9SNR398vzNh9nXYXD/V8cGu4lrfdzgmaD2Ol13X1DKfdrG7Av3J94+9od+DVBLbNh2BTxNQfbU78Fjo1Ztknsyt39SXfvRvBh5ARnU7j7R+4+lOD0/wHgmbA5JJ8FV088S/D/PzXc9t8pvi23KJuAhoW2cXpRM7v7Ane/BGgKrCU4a4L4/8/ixueJt+2N4fNij2EJNgItwtdk7LpL+zqAoGmpP3AxwQd4Ujg+keO9iYKvG4sdjmM1Bc8kOwFbwrPWI47CvWz+CFxiZilxpj0CXAIsKjzBzMaFXzzVM7NaYZNMfQ69YiZR/2VmA8Nmj3HAHoJ2/WXANxZ8cVgvrP10MLPzwuW2AEmF3nxvETQddAaWhaf8ZwDnx+xLSev9C3CvmZ0R7m9jM+tf2p3y4NK/54CJZnZsWEO9IWaW+gThvw2oZWZ3UrDGlQVcbmYnmVmT8NgQluksM+sZhvRuYBfB2QVmdp2ZNQ5rnF+FixS+DPEYgmadbcA+M+sDXFrafQz381OC5rDfWXCZbDfginjzmtmpZnZlGMZ7CGqZeWXbQvAdyTFlKEbetrsTfI/wdDg+CxgYHv8fEJzVxNpCcDVJPP8m+HD47/A7oLRwv+aUoXz1CfY3l+DD5r5SLPsKcHbMe+RWiv+QmgncaGbtw+9IfktQkTsiKdzLwN23EbwQ7ogz7Qt3XxieAha2C3iQ4DRyO3Az8KOw7TrPS1bwOvfniynKPIJT9S+B64GBYTvvfoI3UwqwPtzW4wQ1Hzj4Bs41s3fDcn9L8GXVanf/Ppz+NvCpu28N5ylpvQ8TfKn8avj9xFKCD4eyGBuudzPwv8Bsgjc5BFcr/YPg8tNPCUI69nT7fwna87MJ2sNjv/CrA0wOy76ZoJZ+ezjtMmC1me0M92VI4bZqd99BEBJzCY77NeE+l9U1BMfoC4LvcYo646tBcEa4MZz3QuBn4bTXCWqdm81seym2vZlgHzYStFWPcfe14bSpwPcEIT6DQ7/EnAjMCJvgCrTTh6+fKwkuPNgOPArcELPu0phJ8D/eALxP8JpKiLtvJ7gceTLBh0Nr4M1i5p8P/B54I9zmp8T/bu2IYPEzSOTwYmYPAE3cPe7VJCJSkGruclgys7Zm1tECnQmaBYo7ixGRGLVKnkWkWtQnaIppRnDp6YMEzVAikgA1y4iIRJCaZUREIuiwaJZp1KiRJyUlVXcxRCrEBx8EPd+eddYR30miHOaWL1++3d0bx5t2WIR7UlISmZmZ1V0MkQqRlpYGQEZGRrWWQ6LPzOL+ohnULCMiEkkKdxGRCCox3M2shZm9EXbStNrMfh6On2hBn9hZ4ePymGVuM7N1FvSV3bvotYuISGVIpM19H/BLd3/XzOoDyy24qQDAVHcv0IGWmbUn6AL2bIJrlF8zszZexbcoE9m7dy85OTns3l1Ub7eV4667gl+sr1mzpkq3K9FVt25dmjdvTu3atRNepsRwd/dNBL2r4e47zGwNBbvuLKw/MCfse3m9ma0j6Izq7YRLJVIBcnJyqF+/PklJSVi574uSuBo1ghNiXS0jFcHdyc3NJScnh5YtWya8XKna3M0sieD+of8OR401sxUW3Eg2704zp1GwE6cc4nwYmNkoC25plrlt27bSFEMkIbt37+bkk0+u0mAXqWhmxsknn1zqM9CEw93Mjifox3qcu38D/JngVmQpBDX7B/NmjbP4IT+Ddffp7p7q7qmNG8e9TFOk3BTsEgVleR0nFO4W3BPxWWCWuz8H4O5bwjsLHSC4aUDncPYcCnaIn3cDCRERqSKJXC1jwF+BNe7+UMz4pjGzDQBWhc9fBIaYWR0za0nQh/KyiiuyyJGjZs2apKSk0KFDB6644gq++uorALKzszEz7rjj4C0Btm/fTu3atRk7diwQ/NI1LS2NlJQU2rVrx6hRo4Dgx1EnnHACKSkp+Y/XXnutyvdNDm+JXC3TleBGECvNLCscdzswNLwTkRPcFGE0gLuvNrO5BB3r7wNu1pUyciRbkfNVqebfu2df/nJ16tZj5ssZAPx2/E+5474p3HTrr9iw6Ruan57E08/PY/DoXwIwd+YMzmzTlu0797Ai5yv+3623Mn78ePr3D25mtXLlyvxtdO/enZdffrn8OyeRlcjVMkuI347+92KWuZfg5s8iEup47nl8tGZ1/nCdunVp9YM2rH7vP5zd6RwWvPQ8l/a7iq1bgvs9b9q0iebND95GNjk5ucrLLEeuw6JvGZHKNm7cOLKyssq07LdhTbyws85O5r8n3p/QOvbv38+yNxcx4OrrCoy/7MqBzH/xORqdcgo1atak8alN88N9/Pjx9OzZkx/+8IdceumljBgxghNPPBGAxYsXk5KSkr+eZ599ljPPPLP0OyeRpe4HRCrRnt27+HHv7lzYsRVff/UlXXpcVGB617SLWbr4Df4x71l6XzGgwLQRI0awZs0aBg8eTEZGBl26dGHPnuA2st27dycrKyv/oWCXwlRzl6PCH//4xzIvW9o291h16tZj7oLF7Pjma24ZPoQ5Mx7n2pGj86fXPuYY2iWnMHP6NJ5b+Db/+uf8Ass3a9aMkSNHMnLkSDp06MCqVasKb0IkLtXcRapA/QYn8OtJDzDzf/4/e/fuLTDthlE3M+62iZzY8KQC4+fPn58/7+bNm8nNzeW004r7cbjIQQp3kSrSrkNH2rTvwPwXny0w/gdntePKwUMPmf/VV1+lQ4cOdOrUid69e/OHP/yBJk2aAAfb3PMezzzzTJXsgxw5Dot7qKamprpu1iEVbc2aNbRr167c6yn1pZBfbACg9knlq2V3bH5iuZaXaIn3ejaz5e6eGm9+1dxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkX6jKUSNpwisVur4Xx3at0PVVlbS0NKZMmUJqatwr6A6RkZHBlClTDumFMiMjg/79+9OqVSt27dpFv379mDIluKVyeno6I0aM4LXXXqNXr14APP/88wwcOJCnn36aQYMG8fLLL3PHHXdw4MAB9u7dy89//nNGjx7NxIkTeeyxx4i9iU9GRkZ+vzrxDB8+nH79+jFo0CB+8pOf8Itf/IL27dvHnTc9PZ1LL72UZs2axZ1+55130qNHDy6++GKSkpLIzMykUaNGCR2r7Oxs3nrrLa655hoAMjMzmTlzJo888khCy1ckhbtIFXB33D3//qpRkdf18K5duzjnnHMYMGAAXbsGH3rJycnMnj07P9znzJlDp06dgODm5aNGjWLZsmU0b96cPXv2kJ2dnb/e8ePH86tf/apMZXr88ceLnZ6enk6HDh3ihvv+/fuZNGlSmbYLQbg/+eST+eGempqa8IdoRYvWK03kMLLh88+46qLzuff2X3J1nwvZvDGH9L88wjV9ezLokq48+mDQo+R3333L2GE/ZvCl3RjY6wLmv/gcAJMmTeK8886jQ4cOjBo1irwfHKalpTF+/Hh69OhBu3bteOeddxg4cCCtW7fmt7/9LRCETNu2bRk2bBgdO3Zk0KBBfPfdd4eU8dVXX+WCCy7g3HPPZfDgwezcuRMIuj5o27Yt3bp147nnnitxX+vVq0dKSgobNmzIH9e9e3eWLVvG3r172blzJ+vWrcvvyXLHjh3s27ePk08+GYA6deqU6obi7s7YsWNp3749ffv2ZevWrfnT0tLSyMzMZP/+/QwfPpwOHTqQnJzM1KlTeeaZZ8jMzOTaa68lJSWFXbt2kZSUxKRJk+jWrRtPP/00w4cPL/CL3z/84Q907tyZzp07s27dOoBD5jn++OMBmDBhQv6vh6dOnUpGRgb9+vUD4IsvvuCqq66iY8eOdOnShRUrVgAwceJERo4cSVpaGq1ataqwWr7CXaQSZX/8EVcMGsLc+YvI/ngdn63/hFkvL2TugsW8vzKL5Uvf5K2MhTQ+tSlPv7qE5xa+Tde0oKY7duxY3nnnHVatWsWuXbsKNIscc8wxLFq0iDFjxtC/f3+mTZvGqlWrSE9PJzc3Fwju5DRq1ChWrFhBgwYNePTRRwuUbfv27dxzzz289tprvPvuu6SmpvLQQw+xe/dubrrpJl566SUWL17M5s2bS9zPL7/8ko8++ogePXrkjzMzLr74YhYsWMC8efO48sor86eddNJJXHnllZxxxhkMHTqUWbNmceDAgfzpU6dOze9a4aKLCvakCUETzwcffMDKlSt57LHHeOuttw6ZJysriw0bNrBq1SpWrlzJiBEjGDRoEKmpqcyaNYusrCzq1asHQN26dVmyZAlDhgw5ZD0NGjRg2bJljB07lnHjxhV7HCZPnpzfY+f48eMLTLvrrrs455xzWLFiBffddx833HBD/rS1a9eyYMECli1bxu9+97tD+h8qC4W7SCVq2rwFHc89D4C3F73B24te5+rLejCkz4Vkr/uIT7M/4Qdt27N0SQZT77uLd//9FvUbnADAG2+8wfnnn09ycjKvv/46q1cfvNFHXlAmJydz9tln07RpU+rUqUOrVq34/PPPAWjRokV+E8l1113HkiVLCpRt6dKlvP/++3Tt2pWUlBRmzJjBp59+ytq1a2nZsiWtW7fGzLjuuoJ90MdavHgxHTt2pEmTJvTr1y+/75s8Q4YMYc6cOcyZM4ehQwv2n/P444+zcOFCOnfuzJQpUxg5cmT+tPHjx+d3Z/zGG28cst1FixYxdOhQatasSbNmzejZs+ch87Rq1YpPPvmEW265hfnz59OgQYMi9+Pqq68uclpeuYcOHcrbb79d5HwlWbJkCddffz0APXv2JDc3l6+//hqAvn37UqdOHRo1asQpp5zCli1byrydPGpzF6lE9eodm//c3Rl583gGXzfikPnmvJLB4jde5eEHJnFBj56MGHMrP/vZz8jMzKRFixZMnDiR3bt3589fp04dAGrUqJH/PG94377g5iLB7Y8PKjzs7lxyySXMnj27wPisrKxD5i1KXpv7hx9+SLdu3RgwYECBm4h07tyZVatWUa9ePdq0aXPI8snJySQnJ3P99dfTsmVL0tPTE9puvP0prGHDhrz33nssWLCAadOmMXfuXJ544om48x533HEJbSfvea1atfLPNNyd77//vsTyxuvHK299sf/DmjVr5v8Py0M1d5Eq8sMLe/LCU7P47tugXXvLpo3kbt/G1s2bqFuvHv0GXs2wUbewduV7+TflaNSoETt37ixTr4+fffZZfk1z9uzZdOvWrcD0Ll268Oabb+a3I3/33Xd8+OGHtG3blvXr1/Pxxx/nL1uSNm3acNttt/HAAw8cMu3+++/nvvvuKzBu586dZGRk5A9nZWVxxhlnJLxvPXr0YM6cOezfv59NmzbFrd1v376dAwcO8KMf/Yi7776bd999F4D69euzY8eOhLf11FNP5f+94IILAEhKSmL58uUAzJs3L78Zpbh19+jRg1mzZgHB1T+NGjUq9myivFRzl6NG9uS+ZVquPDfriPXDC3uyft2HXN//UgCOPe547nv4f/gs+xOm3nsnNWrUoFat2vzmvgdpcMIJ3HTTTSQnJ5OUlMR5551X6u21a9eOGTNmMHr0aFq3bs1Pf/rTAtMbN25Meno6Q4cOzf8wueeee2jTpg3Tp0+nb9++NGrUiG7duiV0k5AxY8YwZcoU1q9fX2B8nz59DpnX3fn973/P6NGjqVevHscdd1yBWvvUqVP529/+lj/8wgsvkJSUlD88YMAAXn/9dZKTk2nTpg0XXnjhIdvYsGEDI0aMyK9h339/8AX28OHDGTNmDPXq1UuomWXPnj2cf/75HDhwIP+D7qabbqJ///507tyZXr165df8O3bsSK1atejUqRPDhw/nnHPOyV/PxIkTGTFiBB07duTYY49lxowZJW67PNTlr0TW0dzlb3Z2Nv369dOdmyJEXf6KiIjCXSSKkpKSVGs/yincJdIOh2ZHkfIqy+tY4S6RVbduXXJzcxXwckRzd3Jzc6lbt26pltPVMhJZzZs3Jycnh23btpVrPVu+3FWq+Q98+yUANXbuKdd21+yoV67lJTrq1q1L8+bNS7WMwl0iq3bt2rRs2bLc6+lTyt4kNz85AYAm10wu13bLeummCKhZRkQkkhTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQSWGu5m1MLM3zGyNma02s5+H408ys3+a2Ufh34Yxy9xmZuvM7AMz612ZOyAiIodKpOa+D/ilu7cDugA3m1l7YAKw0N1bAwvDYcJpQ4CzgcuAR82sZmUUXkRE4isx3N19k7u/Gz7fAawBTgP6A3kdEs8Argqf9wfmuPsed18PrAM6V3C5RUSkGKX6haqZJQHnAP8GTnX3TRB8AJjZKeFspwFLYxbLCccVXtcoYBTA6aefXuqCi0RdUil/GVuR9OvYI1/CX6ia2fHAs8A4d/+muFnjjDuk5yZ3n+7uqe6e2rhx40SLISIiCUgo3M2sNkGwz3L358LRW8ysaTi9KbA1HJ8DtIhZvDmwsWKKKyIiiUjkahkD/gqscfeHYia9CAwLnw8D5sWMH2JmdcysJdAaWFZxRRYRkZIk0ubeFbgeWGlmWeG424HJwFwzuxH4DBgM4O6rzWwu8D7BlTY3u/v+ii64iIgUrcRwd/clxG9HB+hVxDL3AveWo1wiIlIO+oWqiEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCStW3jEh1qc5+VkSORKq5i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCKpV3QWQI0vShFequwgikgDV3EVEIqjEcDezJ8xsq5mtihk30cw2mFlW+Lg8ZtptZrbOzD4ws96VVXARESlaIjX3dOCyOOOnuntK+Pg7gJm1B4YAZ4fLPGpmNSuqsCIikpgSw93dFwFfJLi+/sAcd9/j7uuBdUDncpRPRETKoDxt7mPNbEXYbNMwHHca8HnMPDnhOBERqUJlDfc/A2cCKcAm4MFwvMWZ1+OtwMxGmVmmmWVu27atjMUQEZF4yhTu7r7F3fe7+wHgMQ42veQALWJmbQ5sLGId09091d1TGzduXJZiiIhIEcoU7mbWNGZwAJB3Jc2LwBAzq2NmLYHWwLLyFVFEREqrxB8xmdlsIA1oZGY5wF1AmpmlEDS5ZAOjAdx9tZnNBd4H9gE3u/v+Sim5iIgUqcRwd/ehcUb/tZj57wXuLU+hRESkfPQLVRGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQgqMdzN7Akz22pmq2LGnWRm/zSzj8K/DWOm3WZm68zsAzPrXVkFFxGRoiVSc08HLis0bgKw0N1bAwvDYcysPTAEODtc5lEzq1lhpRURkYSUGO7uvgj4otDo/sCM8PkM4KqY8XPcfY+7rwfWAZ0rpqgiIpKosra5n+rumwDCv6eE408DPo+ZLyccJyIiVaiiv1C1OOM87oxmo8ws08wyt23bVsHFEBE5upU13LeYWVOA8O/WcHwO0CJmvubAxngrcPfp7p7q7qmNGzcuYzFERCSesob7i8Cw8PkwYF7M+CFmVsfMWgKtgWXlK6KIiJRWrZJmMLPZQBrQyMxygLuAycBcM7sR+AwYDODuq81sLvA+sA+42d33V1LZRUSkCCWGu7sPLWJSryLmvxe4tzyFEhGR8ikx3EXk6JM04ZVq2W725L7Vst0oUvcDIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEVSrPAubWTawA9gP7HP3VDM7CXgKSAKygR+7+5flK6bESprwSnUXQUQOcxVRc7/I3VPcPTUcngAsdPfWwMJwWEREqlBlNMv0B2aEz2cAV1XCNkREpBjlDXcHXjWz5WY2Khx3qrtvAgj/nhJvQTMbZWaZZpa5bdu2chZDRERilavNHejq7hvN7BTgn2a2NtEF3X06MB0gNTXVy1kOERGJUa6au7tvDP9uBZ4HOgNbzKwpQPh3a3kLKSIipVPmcDez48ysft5z4FJgFfAiMCycbRgwr7yFFBGR0ilPs8ypwPNmlreeJ919vpm9A8w1sxuBz4DB5S+miIiURpnD3d0/ATrFGZ8L9CpPoUREpHz0C1URkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkHlvUG2iEiFSZrwSrVsN3ty32rZbmVSzV1EJIIU7iIiEaRwFxGJILW5l0N1tQ+KiJRENXcRkQhSzV1EjnrVeRZeWVfqqOYuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQZH4har6eBERKajSau5mdpmZfWBm68xsQmVtR0REDlUp4W5mNYFpQB+gPTDUzNpXxrZERORQlVVz7wysc/dP3P17YA7Qv5K2JSIihVRWm/tpwOcxwznA+bEzmNkoYFQ4uNPMPqikshwuGgHbq7sQh4mj4lh8+kC/RGc9Ko5Hgo66Y2EPFDu5pONxRlETKivcLc44LzDgPh2YXknbP+yYWaa7p1Z3OQ4HOhYF6XgcpGNRUHmOR2U1y+QALWKGmwMbK2lbIiJSSGWF+ztAazNraWbHAEOAFytpWyIiUkilNMu4+z4zGwssAGoCT7j76srY1hHkqGmCSoCORUE6HgfpWBRU5uNh7l7yXCIickRR9wMiIhGkcBcRiSCFewUqqcsFMzvBzF4ys/fMbLWZjaiOclaVBI5HQzN73sxWmNkyM+tQHeWsCmb2hJltNbNVRUw3M3skPFYrzOzcqi5jVUngWLQ1s7fNbI+Z/aqqy1fVEjge14aviRVm9paZdUpkvQr3CpJglws3A++7eycgDXgwvJoochI8HrcDWe7eEbgBeLhqS1ml0oHLipneB2gdPkYBf66CMlWXdIo/Fl8AtwJTqqQ01S+d4o/HeuDC8H1yNwl+yapwrziJdLngQH0zM+B4ghfxvqotZpVJ5Hi0BxYCuPtaIMnMTq3aYlYNd19E8P8uSn9gpgeWAieaWdOqKV3VKulYuPtWd38H2Ft1pao+CRyPt9z9y3BwKcHvhkqkcK848bpcOK3QPH8C2hH8oGsl8HN3P1A1xatyiRyP94CBAGbWmeCn1Am9cCMokeMlciPwj0RmVLhXnBK7XAB6A1lAMyAF+JOZNajcYlWbRI7HZKChmWUBtwD/IbpnMiVJ5HjJUczMLiII918nMn8kbtZxmEiky4URwGQPflywzszWA22BZVVTxCpV4vFw928IjglhU9X68HE0UpcdUiQz6wg8DvRx99xEllHNveIk0uXCZ0AvgLBt+SzgkyotZdUp8XiY2YkxXyj/BFgUBv7R6EXghvCqmS7A1+6+qboLJdXPzE4HngOud/cPE11ONfcKUlSXC2Y2Jpz+F4JvutPNbCXBafiv3T2S3ZsmeDzaATPNbD/wPsEpZySZ2WyCK6QamVkOcBdQG/KPxd+By4F1wHeEZzRRVNKxMLMmQCbQADhgZuOA9lH94E/gtXEncDLwaHCCy75EeopU9wMiIhGkZhkRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIuj/ADYTFxgMwf4FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we can calculate the distribution of the RMSE of 0 and f resampled over member\n",
    "xs.rmse(f_r, xr.zeros_like(f_r), dim='iteration').plot.hist(label='resampled RMSE distribution')\n",
    "# the gaussian distribution should have an RMSE with 0 of one\n",
    "plt.axvline(x=xs.rmse(f, xr.zeros_like(f)), c='k', label='RMSE')\n",
    "plt.title('RMSE between gaussian distribution and 0')\n",
    "plt.legend()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}