notebooks/03.01-Measuring-Model-Performance-with-Scoring-Functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!--BOOK_INFORMATION-->\n",
    "<a href=\"https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv\" target=\"_blank\"><img align=\"left\" src=\"data/cover.jpg\" style=\"width: 76px; height: 100px; background: white; padding: 1px; border: 1px solid black; margin-right:10px;\"></a>\n",
    "*This notebook contains an excerpt from the book [Machine Learning for OpenCV](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv) by Michael Beyeler.\n",
    "The code is released under the [MIT license](https://opensource.org/licenses/MIT),\n",
    "and is available on [GitHub](https://github.com/mbeyeler/opencv-machine-learning).*\n",
    "\n",
    "*Note that this excerpt contains only the raw code - the book is rich with additional explanations and illustrations.\n",
    "If you find this content useful, please consider supporting the work by\n",
    "[buying the book](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv)!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!--NAVIGATION-->\n",
    "< [First Steps in Supervised Learning](03.00-First-Steps-in-Supervised-Learning.ipynb) | [Contents](../README.md) | [Understanding the k-NN Classifier](03.02-Understanding-the-k-NN-Algorithm.ipynb) >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Measuring-Model-Performance-with-Scoring-Functions\n",
    "\n",
    "One of the most important parts of building a machine learning system is to find a way to\n",
    "measure the quality of the model predictions. In real-life scenarios, a model will rarely get\n",
    "everything right. From earlier chapters, we know that we are supposed to use data from the\n",
    "test set to evaluate our model. But how exactly does that work?\n",
    "\n",
    "The short, but not very helpful, answer is that it depends on the model. People have come\n",
    "up with all sorts of **scoring functions** that can be used to evaluate our model in all possible\n",
    "scenarios. The good news is that a lot of them are actually part of scikit-learn's `metrics`\n",
    "module.\n",
    "\n",
    "Let's have a quick look at some of the most important scoring functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Scoring classifiers using accuracy, precision, and recall\n",
    "\n",
    "In a binary classification task, where there are only two different class labels, there are a number of different ways to measure classification performance. Some common metrics are:\n",
    "\n",
    "- `accuracy_score`: *Accuracy* counts the number of data points in the test set that have been predicted correctly, and returns that number as a fraction of the test set size. Sticking to the example of classifying pictures as cats or dogs, accuracy indicates the fraction of pictures that have been correctly classified as containing either a cat or a dog. This is the most basic scoring function for classifiers.\n",
    "- `precision_score`: *Precision* describes the ability of a classifier not to label as \"cat\" a picture that contains a dog. In other words, out of all the pictures in the test set that the classifier thinks contain a cat, precision is the fraction of pictures that actually contain a cat.\n",
    "- `recall_score`: *Recall* (or *sensitivity*) describes the ability of a classifier to retrieve all the pictures that contain a cat. In other words, out of all the pictures of cats in the test set, recall is the fraction of pictures that have been correctly identified as pictures of cats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's say, we have some ground truth class labels that are either zeros or ones. We can\n",
    "generate them at random using NumPy's **random number generator**. Obviously, this\n",
    "means that whenever we rerun our code, new data points will be generated at random.\n",
    "\n",
    "However, for the purpose of this book, this is not very helpful, as I want you to be able to\n",
    "run the code and always get the same result as me. A nice trick to get that is to fix the seed\n",
    "of the random number generator. This will make sure the generator is initialized the same\n",
    "way every time you run the script.\n",
    "\n",
    "We can fix the seed of the random number generator using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then we can generate five random labels that are either zeros or ones by picking random\n",
    "integers in the range (0, 2):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, 0, 0])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_true = np.random.randint(0, 2, size=5)\n",
    "y_true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's assume we have a classifier that tries to predict the class labels mentioned earlier. For\n",
    "the sake of argument, let's say the classifier is not very smart, and always predicts label 1.\n",
    "We can mock this behavior by hard-coding the prediction labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = np.ones(5, dtype=np.int32)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "What is the accuracy of our prediction?\n",
    "\n",
    "As mentioned earlier, accuracy counts the number of data points in the test set that have\n",
    "been predicted correctly, and returns that number as a fraction of the test set size. We\n",
    "correctly predicted only the second data point (where the true label is 1). In all other cases,\n",
    "the true label was a 0, yet we predicted 1. Hence, our accuracy should be 1/5 or 0.2.\n",
    "\n",
    "A naive implementation of an accuracy metric might sum up all occurrences where the\n",
    "predicted class label matched the true class label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20000000000000001"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(y_true == y_pred) / len(y_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Close enough, Python.\n",
    "\n",
    "A smarter, and more convenient, implementation is provided by scikit-learn's `metrics`\n",
    "module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20000000000000001"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.accuracy_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "That wasn't too hard, was it? However, in order to understand precision and recall, we need\n",
    "a general understanding of **type I** and **type II** errors. Refer to the book for a detailed explanation.\n",
    "\n",
    "In short:\n",
    "If a data point was truly a positive, and we predicted a positive, we\n",
    "got it all right! In this case, the outcome is called a **true positive**. If we thought the data\n",
    "point was a positive, but it was really a negative, we falsely predicted a positive (hence the\n",
    "term, **false positive**). Analogously, if we thought the data point was a negative, but it was\n",
    "really a positive, we falsely predicted a negative (**false negative**). Finally, if we predicted a\n",
    "negative and the data point was truly a negative, we found a **true negative**.\n",
    "\n",
    "Let's quickly calculate these four metrics on our mock-up data. We have a true positive,\n",
    "where the true label is a 1 and we also predicted a 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "truly_a_positive = (y_true == 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "predicted_a_positive = (y_pred == 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You thought it was a 1, and it actually was a 1\n",
    "true_positive = np.sum(predicted_a_positive * truly_a_positive)\n",
    "true_positive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Similarly, a false positive is where we predicted a 1 but the **ground truth** was really a 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You thought it was a 1, but it was actually a 0\n",
    "false_positive = np.sum((y_pred == 1) * (y_true == 0))\n",
    "false_positive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "I'm sure by now you've got the hang of it. But do we even have to do math in order to know\n",
    "about predicted negatives? Our not-so-smart classifier never predicted 0, so (`y_pred == 0`) should never be true:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You thought it was a 0, but it actually was a 1\n",
    "false_negative = np.sum((y_pred == 0) * (y_true == 1))\n",
    "false_negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You thought it was a 0, and it actually was a 0\n",
    "true_negative = np.sum((y_pred == 0) * (y_true == 0))\n",
    "true_negative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To make sure we did everything right, let's calculate accuracy one more time. Accuracy\n",
    "should be the number of true positives plus the number of true negatives (that is,\n",
    "everything we got right) divided by the total number of data points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20000000000000001"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = np.sum(true_positive + true_negative) / len(y_true)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Success! Precision is then given as the number of true positives divided by the number of all\n",
    "true predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20000000000000001"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision = np.sum(true_positive) / np.sum(true_positive + false_positive)\n",
    "precision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Turns out that precision isn't better than accuracy in our case. Let's check our math with\n",
    "scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20000000000000001"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.precision_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, recall is given as the fraction of all positives that we correctly classified as\n",
    "positives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall = true_positive / (true_positive + false_negative)\n",
    "recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.recall_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Perfect recall! But, going back to our mock-up data, it should be clear that this excellent\n",
    "recall score was mere luck. Since there was only a single 1 in our mock-up dataset, and we\n",
    "happened to correctly classify it, we got a perfect recall score.\n",
    "\n",
    "Does that mean our classifier\n",
    "is perfect? Not really! But we have found three useful metrics that seem to measure\n",
    "complementary aspects of our classification performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Scoring regressors using mean squared error, explained variance, and R squared\n",
    "\n",
    "When it comes to regression models, our metrics above don't work anymore. After all, we are now predicting continuous output values, not distinct classification labels. Fortunately, Scikit-Learn provides some other useful scoring functions:\n",
    "\n",
    "- `mean_squared_error`: The most commonly used error metric for regression problems is simply to measure the *squared error* between predicted and true target value for every data point in the training set, averaged across all data points.\n",
    "- `explained_variance_score`: A more sophisticated metric is to measure to what degree a model can explain the variation or dispersion of the test data. Often, the amount of *explained variance* is measured using the *correlation coefficient*.\n",
    "- `r2_score`: The R2 score (pronounced *R squared*) is closely related to the explained variance score, but uses an unbiased variance estimation. It is also known as the *coefficient of determination*.\n",
    "\n",
    "Let's create another mock-up dataset. Let's say we are observing data that looks like a sin as\n",
    "a function of x values. We start by generating 100 equally spaced x values between 0 and\n",
    "10:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "However, real data is always noisy. To honor this fact, we want the target values `y_true` to\n",
    "be noisy, too. We do this by adding noise to the `sin` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "y_true = np.sin(x) + np.random.rand(x.size) - 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here, we use NumPy's `rand` function to add noise in the range [0, 1], but then center the\n",
    "noise around 0 by subtracting 0.5. Hence, we effectively jitter every data point either up or\n",
    "down by a maximum of 0.5.\n",
    "\n",
    "Let's assume our model was smart enough to figure out the `sin(x)` relationship. Hence, the\n",
    "predicted `y` values are given as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "y_pred = np.sin(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "What does this data look like? We can use Matplotlib to visualize them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x2e1e176b898>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnAAAAF6CAYAAACZYGVYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8FXW6P/DPd1JJCEUTRFAwAoLYIhZsNCnSi+BYENeC\nWFjdcvfub+vdcsu65e7dptjWhgiMSJcmCAFFQEUUV0GRiFKTUEIgJJDM8/vjpJw5JTllzpk553ze\nr9e+ljxnzpnHTDJ55luViICIiIiIEofmdAJEREREFB4WcEREREQJhgUcERERUYJhAUdERESUYFjA\nERERESUYFnBERERECYYFHBEREVGCYQFHRERElGBYwBERERElGBZwRERERAkm3ekE4oB7hREREVEi\nUS0dkAoFHPbv3x/Tz8/Pz0d5eXlMz0Hh4TVxJ14X9+E1cSdeF/eJ1zXp1KlTSMexC5WIiIgowbim\nBU7X9RcAjAZQahjGpQFeHwhgEYCS+tB8wzB+G78MiYiIiNzBNQUcgJcA/APAK80cs8EwjNHxSYeI\niIjInVzThWoYxnoAR5zOg4iIiMjt3NQCF4obdF3/BMA+AD8yDONfTidEREREFG+JVMBtBdDFMIwT\nuq6PBLAQQI9AB+q6Pg3ANAAwDAP5+fkxTSw9PT3m56Dw8Jq4E6+L+/CauBOvi/u47ZooEfcsk6br\n+gUAlgaaxBDg2K8BXG0YRktzeoXLiKQeXhN34nVxH14Td+J1cZ84LyPS4jpwrhkD1xJd1zvquq7q\n/30tPLkfdjYrIiIiovhzTReqruuzAQwEkK/r+l4AvwKQAQCGYTwNYBKAR3RdrwVwCsAdhmG4p/mQ\niIiIKE5cU8AZhnFnC6//A55lRshBxSUVmLmtDOVVtcjPSceUogIMKGzrdFpEREQpxTUFHLlfcUkF\nntx8EDV1nobPsqpaPLn5IACwiCMiIoojFnAUspnbyhqLtwY1dYKZ28pYwBERBcBeC4oVFnAUsvKq\n2rDiRESpjL0WFEsJMwuVnJefE7jeDxYnIkplzfVaEEWLf3kpZFOKCixPkwCQlaYwpajAwayIiOIv\nlK5R9lpQLLGAo5A13JzcPp6juKQCsxaXoLSyxrU5ElHiCrVrND8nHWUBijX2WpAd+FNEYRlQ2NbV\nxRDHnBBRrIU6oYu9FhRLHANHSYVjTogo1kLtGh1Q2BbT+3ZEQU46FICCnHRM79uRD5NkC7bAUVLh\nmBMiirVwukaD9VpweRGKFlvgKKlwpiwRxdqUogJkpVn3Gg+na7RhqEdZVS0ETUM9iksqYpAtJSsW\ncJRUor2xEhG1JNquUQ71IDuwWYKSSsMNdNb2I5yFSkQxE82ELg71IDuwgKOkM6CwLSZe0w3l5eVO\np0JE5IfLi5Ad2IVKREQURxzqQXZguU8AOCOKiCheEmVRdHI3FnDExW+JiOLM7Yuik/uxC5U4I4qI\niCjBsIAjzogiIiJKMCzgiIvfEhERJRgWcMQZUURERAmGTSzEGVFEREQJhgUcAeCMKCIiokTCLlQi\nIiKiBMMCjoiIiCjBsIAjIiIiSjAcA0dERFSP2wpSomABR0REBG4rSImFXahERETgtoKUWFjAERER\ngdsKUmJhFyoRERE82weWBSjWQt1WkOPnKJ7YAkdERITothVsGD9XVlULQdP4ueKSihhlS6mOBRwR\nERE8ExWm9+2Igpx0KAAFOemY3rdjSK1oHD9H8cYuVCIionqRbivI8XMUbyzgKGY4HoSIUkW04+eI\nwsUuVIoJjgcholQSzfg5okjw0YBs4dvaVl1nBh0PwlY4Iko2Dfc1p3od2OOReljAUdQCrV4eDMeD\nEFGyinT8XLS4g0RqYgFHUQs0+yqYZBoPwideInKD5mbA8p6UvJLnryk5JtRWtWQaD8InXiJyC86A\nTU0s4ChqwWZf5WUoZGekJWULFZ94iRJfsrSicwZsauLVpahNKSqwtEYBnta2B68JbQHMRMQnXqLE\nFk4rutsLvWD34Ks752Lqgl2uzZuiwwKOoub07Csn8ImXKLGF2oqeCMMlAt2Dr+6ci7d3H3d13hQd\n1/y10XX9BQCjAZQahnFpgNcVgL8CGAmgCsC9hmFsjW+WySEWT5N2z75K1CfeZBnjR5TsQm1FT5Th\nEr734KkLdiVE3hQ5Ny3k+xKA4c28PgJAj/r/TQMwIw45JZ1EWGA3EXKMZs9EInJesNZy33iiDpdI\n1LwpdK5pgTMMY72u6xc0c8g4AK8YhiEANum63k7X9XMNwzgQnwyTg5NPk3LyBFDyBeTb3UD1KUBM\nwDQBEcAUILc11PkXYuZXbRPiydGpNZ+IKHqhtqJHO1xCTBPYtwfybQlwqgqorgJqTgGnTgFnaoA2\n7YGOnaHO6Qx0PA8qt3V0/2E25U3ul0hXsjOAb72+3lsfYwEXhng+lcmRcsjHm4HdOyG7vwBK97f8\nHgDlA34PKOX3Gp8cicguoY7dDXe4hIgAB76F7NwO2fEJ8MWnwInKFvNp/PS8tkDhRTh180hIt95Q\n2a0i+u+L5zAPtw95SVaJVMCFTNf1afB0s8IwDOTn58f0fOnp6TE/h1065GXhUGVNwLgd/w0igtPb\nP8SpZW+g5v13ALMu7M/IrzmGsuz2fvGCDBNn5+VBZWW1+BmJdE1SCa+L+6TyNZmYn4+J13Rr8Zi8\nvDw8vXEPSitr0CEvCw/f0BXDenWwHCc11Ti15k1ULZ2LugN7I0+qsgL45H0c/+R9ICsb2df2Q3b/\nYcgs6guVHvqf7FDzjtaqHaV4cssh1NSaAOqHvGw5hLy8PNvP5TS3/a4kUgG3D8D5Xl+fVx/zYxjG\nswCerf9SysvLY5pYfn4+Yn0Ou0y+7KyAT2WTLzsrqv8GqToJ2bgGUrwcOBjwsoSe4+7lmNFzEmrS\nMptyrDuNuz6bh9L7/xPqukFQw8ZDnR385pBI1ySV8Lq4D69Jy/rka3h2bKEl1vA9k+PHIGuXQda9\nGVJLW1hqqlG94S1Ub3gLyGsLNfp2qP7DQy7kmsvbLk9t2N1YvDWoqTXx1Ibd6JPvpmH20YvX70qn\nTp1COi6RCrjFAL6r6/ocAH0BVHD8W/jsXvJDTBPy3tuQeS8BJ443f7DSgM5doAovAs7K93ytpQFa\n/S95+UHInq/Qf99nwM55mHXhCJRntUN+zTFM3r0c/Uu3ec759lLIhlVQIyZB3TIBKrPlFjkiIjtJ\n5XHI4tcg764Gzpxu/uBWuUCP3lBnFQDZrer/lwOkpwOHD0EO7fc8+JbuB2qDDBWprIDMfhayZim0\nid8BrrwOKsBQk3jjZAnnuKaA03V9NoCBAPJ1Xd8L4FcAMgDAMIynASyDZwmRXfAsI3KfM5kmPrsG\n38ver2HOmgHs+jz4QZ26QF03EOrCXkDXbiGN55C6Ogw8uA8Ddu+AbFwE7PrM/6AzpxtvntodU4Er\n+rriZkZEyU1EIB+8C5n9jKe7M5C0dODiK6B6XQbV8zKgy4VQWlqLn71u91G8uvUQymsE+aePY/Ku\nNxsfXBuV7oc543dA94uh3XY/1IU9bfivihwnSzhHiYS2CXkCk/37Wx48H41U64KQ6lOQJXMgqxd5\nZpH6SkuD6nMD1MCRnqfOKAsrOfAtZMMqyHtvB++iuORKaHc+BHWOp+k51a5JouB1cR9ek9DJscMw\nZz0DbNsU+IBWuVADh0PdPAaq3VlhfbbvgsEAkKUJHqn9F/pvNoCa6oDvU4NGQd12H1RGZsDXYy1g\n3mkqKZdVinMXaot/OFnA2SCVboDy9ZcwZzwBHCnzfzEj09OlOXAkVFv/SQhRn/vMGcimtZAFMwM/\n+WZlQ7v3cairb0qpa5JIeF3ch9ekZSICeXc1xHgBOHXS/4CzCqCGjoW6aShUdk5E55i6YFfAlqyC\nnHQ8N+gsyJLZkA1veZZf8tXlQmgP/RiqQ2hjp+yWKrNQWcDFHws4m5gb10BmPgXUnvF/8bKrod05\nDaqgY8zzkKoTkMWzIWvfDNgCqIaNR8G0H+Lw0WMxz4XCkyq/K4mE16R5croG8vLfIVvW+7+Yng41\n6nao4bdCpWdEdZ7xs3Yg0F9jBWDh5F6eXPZ9A/ONl4DtH/gfmN0Kasp0aNf2jyoPCs5tBRw7qalF\nUlsLmfciZM0S/xfPyod2xzSgKH5j0FROa6g7HoTcNBTmnOeAndut+a5aiKP7v4Hc9z2oNva3BBJR\napCKozCf/G+g5Av/Fwsv8rT4d+piy7lCGUumOndB2uP/Afn8Y5iv/AMoP9R0YPUpyHN/grlzO9Tt\nUzm5KwWwBc4GyfwEK5UVMJ/+vWcxSh9q8BioCVOgsrIdyMxDRCDFyyFzngfqfG5+7c6G9vD/g+rW\ny5nkyE8y/64kKl6TwGRvCcy//ydwxOd7k5kJNX4K1ODRIU1MCFW4Y8mk6gTMl/8BbN3o/2GFF0H7\n3q+gcvMCnicVujtjwW0tcGm//vWvY56Mw35dWWnz2jw+cnJyUFVVFdNzOEH2fwPzjz8D9n5tfSEj\nE+r+70MbfmtYC0vGglIK6oIeUL2LIJ9u9WzR1aD6FGTTOqjzL4Tq2NmxHKlJsv6uJDJeE3/y8RaY\nf/ut/9JI554P7d9/B+3ya6CUvWucXdA+Gx1yM7DrcDVOnTFxTl4WHugTvLhSGZlQV9/o2bnh84+t\nw0mOHYZ88gHUlX0tY/IaisTjpz3HVp0xsXX/SXTIzcAF7Z17EE8U8fpdycvLA4DftHQcW+BskIxP\nsLJvD8z//YX/ZIGzCqA9+jOors2vXu4EOX4M5rN/9OtSRVo6tGk/gupzgzOJUaNk/F1JdLwmVuba\nNyGzn/Xs0eyt95WeiQI5uXHJI5zrIt98BfOZPwClPkuj5p8D7Qe/hepwLoDmJ0o8P6F71DknO7e1\nwCXXMslkC/m2BOaffu5fvPW8DNov/uxY8VZcUoGpC3Zh/KwdmLpgF4pLrPmpNu08N6tbJljfWFcL\n85k/wNxcHMdsiSjRmGuXQV57xq94U4NGQXv8P+JWvIVLdekG7ed/BnpeZn2h/BDMP/wEsrfE8yUX\n3U0qLODIQr75ytPy5tN1oPrfAu37v4HKc2asREPTf1lVLQT1++1tPuhfxKWlQZt0H/Ie/rH1A0wT\n8s8/w3x3TfySJqKEYa5fCXntaWtQ06DuehjaXQ9Bpdk33i0WVE4utMf/A7jiWusLFUdh/vFnkK92\nBF1cl4vuJiYWcNRI9uyC+b+/BE5axwyqm0dD3f2oo+PdZm4rswzuBYCaOsHMbQHWowOQc8t4qHu/\n59muq4EI5KW/wly3vNlztdTSR0TJxXx3DeTVp6zB9Axoj/0S2qCRziQVAZWZBe3hn0BdN9D6QtVJ\nmH/5Fe4+30RWmrVnLitNYUpRQfySJNuwgCMAXsVb1QlLXA0ZC3XHg45vUxVJ079242CoqT9s2mu1\nnsyaAXNj4Ja4UFv6iCg5mJvWQl7+m7XbND0d2vSfQV16lXOJRUilp0Pd932om0dbX6g+hX6z/wuP\nXpyNgpx0KHjGviXjjgmpgu2mBCk/5Jlx5bPCuBo2AWrSvY4Xb0Dk++1p1/aHpGd4Jjd4LTMir/wD\n0u5sqN5FluOba+njTY4ouZjvvwN54a/W4i0tHdrDP03I4q2B0jTgjgeBnNaQpXOaXjh+DP3m/jcG\n/OT3XCMzCbAFLsVJ1QlP8XbcumuBGj7RNcUbAEwpKoi46V/1uR7a9J8B3iul19XBnPG7xsG9DTjI\nlyg1yJefQf75Z+vWVJoGbdq/Q11xjXOJhSCUYR5KKWjj7oIaNt76QtlBmH/9LaSaS8ckOhZwKUxq\nz3j2NT3wrSWuho2HuvUe1xRvADCgsC2m9+0YcdO/uuxqqPt/YA1Wn/LcyLwW6uQgX6LkJ2UHYT71\nP9bFv5UG7cEfQfW53rnEQhDuMA818V6ovgOswW++gvnU7yCBtkWkhMG/SkkknBW2RcSzr+mOTyxx\nddWNnl94FxVvDQYUto2qG1O75iaYR8sgr7/YFDx2GObffgPtx09A5eRiSlFBwNXQOciXKDlI1UnP\nDgu+M+3v+x7U1Tc5lFXowh3moTQNuPdxyInjwL8+anrh848hL/wFmPpvnmMo4fCqJYlwn8rkTQPi\nO5C/Wy+o+7+f1L/Mauh4/8G9+/bAfPoJSG1t1C19ROReUlcH87k/+vc6jLkD2vWDHMoqPJEM81Dp\nGdAe/glwQQ9LXN7fAFnxhq35Ufwk71/qFBPOMhvm5mLIolnWYEFHaNN/nvQbICuloG5/ACi6zvrC\n5x9DjH8C8LT0PT+hOxZO7oXnJ3Rn8UaUJOT1F4BPt1pi6pp+UGPudCij8EU6zENlt/KsE9ehkyUu\nC2dBvFvmKGGwgEsSoT6Vyb49kFf+bj0op7VnlXGHFumNN6WlQZv6b0DhRZa4rH0T5pb1DmVFRLFk\nrlsOWbPEGiy8COrex105ZCSYqCZ05bWF9v1fA629NrkXE+Zzf4KUH7I1T66nGXss4JJEKE9lUn0K\n5tO/B06fbjqgYb2jjufFOkVXUVlZ0B77JZB/jiUur/wD4tO9Ei3eyIicJV/tgMx51hpsn+/Z1znB\neh2intBV0BHagz+yLnJ+shLmjCcgp2tsyZHracYHC7gk0dJTWeOkhYN7LceoyY9AXXRp3PJ0E5XX\n1jMuxHt5kZpqz42s+pQt5+CNjMhZcvIEzOf+BNTVNQWzsqF99xdQ7c5yLrEoRDvMQ/W+EmrC3dbg\nN19BZj0N8dkHNhLh7pxDkWEBlyRaeiqTDSshW6ybuasbBkO7aagD2bqH6toN6s4HrcED30JmPsUb\nGVGCExGYL/8NOFxqiWv3fx+qy4UOZeUOavhE4ErrWGDZuAZSvCLqz+Z6mvHBZUSSSLBlNmTPV5DZ\nz1mDnbtC3fVwnDJzN9XvFmDX55D31jbGZEsx0P1iqCj3QeSNjMg5sm4Z8NEmS0wNHgPV5waHMnIP\npRS0+74P88C/AQf3NcZlznOQrt2hCnuEtTSVt0h3zqHwsAUuyUnVSZjP/B7wXrAxqxW0h/4fVFZi\njf2IFaUU1ORHgM5dLXGZ+zyk5MuoPpsLAxM5Q775qnFmeaOu3aEm3utIPm6kWuVAe/RnQFarpmBd\nLcx//hnrviyPePhHNBMtKHQs4JKYiMB85e9A2UFLXE15FOrc1Jq00BKVlQ3t4f8HZPvcyJ7/X0hN\ndcSfyxsZUfxJdRXMZ/4A1Hq1AmW38myTlZER/I0pSJ17PrT7HrcGD+3Dq5u/jXj4B9fTjA82AyQx\n2bwO+HCjJaYGjoDmu60KAQBUx/Ogfecxz42/Qel+yLyXoCZH1t3ccMOKpBuCiMLnmbA1Ayg9YImr\nex6D6nCuQ1m5m7rqRqj+t0DWr2yMlSM74LGhDv+IduccahkLuCQlR8ohr/lMm+9yIZT+gDMJJQh1\n9U1QOz6xDOSVdcsgV1wLdWmfiD6TNzKi+JEt6/0nbPUfDu0a92+T5SSlPwDZsR0o3Q8AyK85hrLs\n9n7HcfiHe7ALNQl5Zl79HTh1simYkQlt6o+gMjKdSyxBqNvuBwo6WmLmy3+DnKx0KCMiCoVUHIXM\n9nlw7dzVs/sKNUtlZUOb+kOgfivFybuXI6vutOUYDv9wFxZwSUiKVwCfWbdGUbdO4bi3EKmsbGj3\n/8C60OWxI5DXnnEuKSJqlojAfPUpwPtBKz3DM+4twRbrdYoqvAhqzB0AgP6l2/DIznkoqD4KBeE4\nNhdiW2iSkdIDkHkvWoM9L4O6eYwzCSUo1f1iqOG3QpbPa4zJlvUwi/pCu6afg5kRUSCyeR2wbbMl\npsZPhurUxZmEEpQacRvk063AVzvQv3Qb+pduAzIzof3yr1AdWby5CVvgkoiYdTBf/AvgPWsyqxW0\nex+H0nipw6XG3gmcV2iJyaszIMcOO5QREQUixw77r3V5YU+ooeOcSSiBqbQ0Tw+E99Iip0/DfOXv\nENN0LjHyw7/qSUTeWgzs+twSU7c/AOWz3yeFRqVneMaEpHs1VFedgPnKk7bs0kBE0RMRmDOfAqpO\nNAUzMqHd9z0oLc25xBKY6nCu/w41X34G2bDKmYQoIBZwSUIO7oMsfNUavOxqqBTfKitaqnNXqPFT\nrMHtH0A+eNeZhIjIQt5bC3zyviWmxk+G6sgxv9FQNwwGLr3KEpM3XoIcZQ+EW7CASwKNg3e9d1vI\nzYN2z3ehlAr+RgqJGjoW6H6xJSZznoWcPBHkHUQUD3LsMGSOT9dpt15QQ8Y6k1ASUUpBu/sRIMtr\nPbhTVTBfs2fDe4oeC7gkIO+tBXZut8TUHQ9CtTvLoYySi9LSoN3zXSDNqyv1+DHI/JedS4qIIHOe\n918u6V52ndpFnd0BaoJPD8S2zcDWjYHfQHHFAi7BSeVxyOs++/31LoLibgu2UueeDzVikiUm61dC\nvvzMoYyIUpts/xDyoXUog5owBapjZ4cySk5q0Ejgwp6WmPnaM+yBcAEWcAlO5r0InPBa9ygjE9rk\nR9h1GgNq5CTA54+DOfNJyJkzQd5BRLEgp2tgvva0Ndi1O9Tg0c4klMQ8PRCP+fdA+C5XRXHHAi6B\nyc7tkI1rLDE1Sud+f0EUl1Rg6oJdGD9rB6Yu2IXikoqw3q8yMqHdPd0aPPAtZMUbNmZJRC2RN18H\nyg81BZQGbcqj7DqNEdW5i38PxDtvQT7/2KGMCGABl7DkzBnP1HlvnbpA3TLBmYRcrrikAk9uPoiy\nqloIgLKqWjy5+WD4RVzPS/1m9soyA3Jwr43ZElEwcuBbyMr5lpgaNBKqa3eHMoqtaB887aJG3gac\ne74lZr72DKSWPRBOYQGXoGT568ChfZaYNuVRqPQMhzJyt5nbylBTZ505VVMnmLmtLOzPUpPuBfK8\nViSvrYX56gzOzCKKMRGBOetpoK62Kdj2LKhxk51LKobsevC0g8rI8Ezm8h6ec3AvZM3SuOdCHizg\nEpAc2m/Z4gkAVP9boLr3digj9yuvqg0r3hyVmwd1+1RrcOd2yAfvRJIaEYVINq3zn3F/+wNQObnO\nJBRjdj542kF1v9i/B2LJnJB3p3FLa2KyYAGXgMy5zwO1XoVHXluoW7/jXEIJID8n8La/weK+fG88\n6wuuAC650nKMvP4ixHsbMyKyjZyshLz+gjXY+0qoq29yJqE4sPPB0y5qwhTAu2CuOQV5/aUW3+em\n1sRkwQIuwcj2D4HtH1hiSn8AKre1QxklhilFBchKs87MzUpTmFJU0OJ7A914ntpyCBsG3GudmXW0\n3K9llIjsIQtnAZVef+zTM6BNfiipZ9xH++AZCyqvLdT4uy0x2VIM+eLTZt/nttbEZMACLoFI7RlP\n65u37hcn/Jpv8WhWH1DYFtP7dkRBTjoUgIKcdEzv2xEDCtu2+N5gN55Xv66DGjLGEpeVCyBlB+1M\nnSjlyd6vIcUrLDE16jaoDp0cyig+onnwjCU1YDhwfqElZr72DKSuLuh73NiamOicK+MpbLJmqXXi\nglLQ7pyW0E+gDa1bDQVSQ7M6gJCKq3AMKGwb0Wc2d+NRE2/3jMupOOoJ1p6BafwTadN/HkWmRNRA\nRGAa/wTEbAoWdIS6ZaJzScVJw/1q5rYylFfVIj8nHVOKCmy/NxaXVIR1DqWlQbvrIZi//0lTcN8e\nyLplUIPHBHxPfk46ygLcS51sTUx0rvnO6bo+HMBfAaQBeN4wjCd8Xh8IYBGAkvrQfMMwfhvXJB0k\nFUchS+dYYqrfMKgu3RzKKDK+N4rqOjNos7rdN6lINXfjUdk5UBPvhbzwf00vbNsM+XQr1KV94pgl\nUZL6eDPgs96Ypt8PlZEaM+4jffAMVaQP0ap7b6jrBkE2rW2MyaJZkGtugmrT3u/4KUUFlvMA7mhN\nTGSu6ELVdT0NwJMARgDoDeBOXdcDTancYBhGUf3/UqZ4AwBZ8ApQfaop0CrXbxyC2wUaS1ZZYwY8\n1k3N6i11Y6i+A4BuvSyvm3Of4/pIRFGSM2dgGj4TFy6+AriirzMJJaFoxqapSfcC2a2aAqeqIPNn\nBjw2mmEsFJhbWuCuBbDLMIzdAKDr+hwA4wBwo0kAUvIF5F2fHRfG3QWVl1g/+IFuFMGEMzs01t0L\nLXVjKE2Dduc0mP/9b0DDWnAH90HeXgo1LPDCyvHImyjRyZrFgPeYUqVBu31qQg8bcZtoxqaptu2h\nxt4FMZr245aNayA3j4bqcqHf8bFuTUw1bingOgP41uvrvQACPWLdoOv6JwD2AfiRYRj/ikdyThLT\nhDn7WWvw3POhBoxwJqEohNqqFu7s0EBN/xPz8yNPNICWbjyqa3eom4ZCNqxqjMmSOZDrBkG1aRdy\n3ry5EXl4ho0YlpgaMByqc9eIPo8PTYGFOzbN9/t49+X90O/cVcCB+j/hIjBffwHaD/+ThXaMuaWA\nC8VWAF0Mwzih6/pIAAsB9Ah0oK7r0wBMAwDDMJBv8x9zX+np6TE7x6nilThe8oUl1u6hHyGrY8eY\nnC+WOuRl4VBljV+8TVYaWmWmo7SyBh3ysvDwDV0xrFeHFj9v1uKSgE3/s7Yfwe3X94z5dfdlPvA9\nlG99D3Ky0hOoPoWstxaizUM/shzXXN4Tr0msMY3hiuXvCkXGrdekYs6zqK5pGjaiWuch/77HoLUJ\nv+hataMUT245hJpaz5CNsqpaPLnlEPLy8kK61zghXtfl0X4mnlizq/F7AwBZ6Roe7Xeh3/kDfR+f\n+qAcrcY8hquf/XHTgTs+QZuvv0DWNTfGPP94ctvvilsKuH0AvDdZO68+1sgwjONe/16m6/pTuq7n\nG4ZR7vthhmE8C6Ch2UrKy/0OsVV+fj5icQ45cxrmKz77nV55HSo7F6Iyxv9NsTD5srMCDmKdelUH\nvyfhUL6fpQGKwYZ4bW1tTK5JS4oHT8Ws0myUZ7VDfs0xTP5oJQZu3wZ17nmW/AIprawJOedEbU2I\n1e8KRc6N10T27IL59pvW4Og7cOT0GSCCXJ/asNtSoABATa2JpzbsRp98VwwF9xOv69InX8P0a8/x\nu5/0ydf8zh/s+/jc3nRc3bsI+GxbY/zYC3+Bdn43qHS3lBnRi9c16dQptOVx3PKdfR9AD13XC+Ep\n3O4AcJcG6B62AAAgAElEQVT3AbqudwRwyDAM0XX9WngmYIS2f0eCkjVLgCNeA0nT0qHddr9zCUXJ\n7inxbpuWXlxSgRknO6Emu75rNLs9Zlx0K7B4DQY91LRTRrR5swuWkpln2ZAXmsaTAsC552P9BTfi\n1QW7Irp3cA2y5oU6Nq2576N2230wf/t96zjgDSuhBo2yM1Xy4opHD8MwagF8F8BKAJ97Qsa/dF1/\nWNf1h+sPmwTgU13XPwbwNwB3GIaRtLuHS+VxyDKf/U4HjYIqSLyuU28DCtvi+QndsXByLzw/oXtU\nBYfbFrkMOJsrLROz0i+CeO3fGG3eXNGcktonHwA+q/pvGDINT71fFvE2TG7c0SARNfd9VOcV+u+T\nuvg1SNWJeKSWklzz02sYxjIAy3xiT3v9+x8A/hHvvJwib84FTp1sCuTkQo3WnUvIheK1yGWogj6d\nZrWD+fpz0H72JyhNizpvtiZQspK6OphvvGQNXtoHrx7ORU2d9ee7pk7w3PsHQ/o94hpk9mjp+6jG\nTYZsWQ807Al9ohKy7HWoSfc5kW7Sc00BR02kdD9knaWWhRqpQ+XmOZSRe7lpWnrQrtGaY8CeXZAt\n66GuGwggurzd1nVMZBfZuKZpNiPg2W1m4ndQXhx43GjlGUHlGc/vQnNDCdz2sJeoWlxSqW17qOET\nIYtmNb5H1iyBDBiR8L1HbsQ7vguZ818BvPeUO7sD1M0cR+B2AZ9O605j8u7lAABZMBPS53qozCz7\nz8PWBEpwUlMNWfSaJaauvxnqvELk5+wK+NDiq7ldXNz0sJfIWlxSaeh4z761x+qHqNfWQua/AvXQ\nj4O+hyLjijFw1ER2fQ58uNESUxOmQGVkxj2XeGwyn0z8VhrPVnjkywXoX1o/M+tImWc/W7vPwxXN\nKQnIW4uAiiNNgYxMqHGeuWyBxo0Gw6EEzlJZWVC33mOJyQfvQL7+0qGMkhdb4FxERGDOe9Ea7Nod\n6pp+cc8lVWY62r0ch+/TqVl7HuTgh41fy/LXITcNiXoXDbYmUDKR48cgK+Y3fr2+QxFmXTIR5csP\nIz+nAlOKCjC9b0e/fZQDbcXHoQTOU30HQFYvBr75qjFmzn8FaT/8TwezSj5sgXOTj94DvtphCWm3\n3Q+lxf8ypcJMx0B7s4Yzsy0UarQO5OQ2BU5V+c0uJkp1snQOUL9o7/oORZjRcxLKJMvyewnAMoP9\nwavOcdUsdGqiNA3aRGsrHD7/GPLZR84klKRYwLmE1NXBXOCzCfAV10L1vNSRfFJhpmM8ilSVmwc1\nyjp7WNYtgxxOnkKYKBpycB9k/crGr2ddOAI1adYhI4F+LzmUwN1U7yuBi6+wxMw3XoGY/q2mFBkW\ncC4h770NHPTafEJp0CZ+J/gbYiwV1k2KV5GqBo0C2nttv1J7BrJktq3nIEpU5sKZlklb5VntAh4X\n6PfSznUlyX6az1g4fPMV5MN3nUkmCbGAcwE5cxqyZDbWdyjCQ9f9FBMH/B4P9f8V1le3cSwnty2S\nGwvxKlJVRibU2DstMdn4NsR7uQQiF4r1RCbZs8tv0lZ+RuAWmmR6eEwV6oIeUFdZ90OVha9CapOn\nJ8dJLOBcQNYtx/r08zzjPrLbQ5RCmWpl+3iscKRC90Q8i1R1/c1Ax6b9UCEmzIWv2n4eIrvEY4yo\n37CRrt1x9zWdk/7hMZWo8XcD3uO4Sw9A3lnlXEJJhAWcw6S6CrLs9ZDHfcRTsndPxLNIVWlp0Mbf\nbQ1ufQ9S8oXt5yKyQ6zHiMrOT4F/WQe1axOmYOCF7ZL+4TGVqI6d/bfYWjoX0rBbA0WMbdIOk1WL\ngBPHmx33YfdSF9Qkrstx9Lke6Nod2LOrMWS+8TK0f/svKBXaGldE8RLLMaIiAnPBK9bgRZcCvYsA\ncJmcZKPG3AHZtBY4fdoTqDgKWb3Yb4IXhYctcA6SygrIWwsB1G+3FEDrDBXzbgyKD1W/LZDFzu3A\n59ucSYioGTEdI7r9A/8lkybczQeZJKXanQ01eIwlJivnQ05WOpRRcmAB5yBZNg+o9qx9NHn3cmTV\nnba8npWmAE0l/XpsqURdfIX/1Pr5MyEiQd5B5IxYjREV04S5wGf852VXQ3XvHdXnkrup4ROBnNZN\ngVNVkJULnEsoCbCAc4gcKbNsWN+/dBseaVfmN+7jRICVxoHkWo8t1WgTfKbW79kFbN0Y+GAih8Rq\njKh8+C6wt8QS0yZMieozyf1UTmtPEedF1iyBHA/c+0Qt4xi4KDSNTdsR9tg0WToXqD3TFGifj4Ej\n+mGQz56nM7eVBdzEOdRuDI6fcx9V2APoc4OlaDMXvQbtyuugtDQHMyOysnssmtTWQhbOssTUNf2g\nzi+07RzkXurmUZDVi4CGou10DWT5PKjbpzqbWIJiC1yEopliL6UHIO+utsTUmDsCblgfTTdGPJYB\noMho4ycDyuvX78C3kPffcS4hojiQ994GSvc3BTQNauxdziVEcaWysqFG3maJybrlkCPlDmWU2FjA\nRSiaKfaydA7gvZ1Ih05QNwwOeGw03RipsJ9polLnng/Vd4AlJotnQ7xWpCdKJnLmjOfe50XdOASq\nY2eHMiInqP63+O9Ms8xwLqEExi7UCEU6xV4O7IVsKrbE1Jg7oNKCd51F2o2RCvuZJjI15nbIluKm\nYr50P2TTWqgbhzibGFEMyIaVgHdLS3oG1OjbnUuIHKEyMqFG65CZTzXG5J23ILfcClXQ0cHMEg9b\n4CIU6RR7WTIbEK/Wt3PPh7q2n52ptZgLt6RxBxWg5VWWzIF4j40kSgJyusYz696LGjAc6izurpCK\n1A1DAO9ira7OMy6cwsICLkKRjE2TvV9D3t9giWlj74zZwPVU2M800anRtwNpXgX14VLIO6uDv4Eo\nAUnxCqDiSFMgMxNqxCTnEiJHqfR0qDE++0O/txZycK9DGSUmFnARimRsmrn4NWvgvELPbEQX5Ujx\npc7uANVvmCUmbxqQM6eDvIMosUhNNWS5T+vbwFFQbdvH7JzFJRWYumAXxs/agakLdnHilgupvv39\n9oeWxbOdSygBsS8tCg1j0/Lz81Fe3vwsGtnzFfDRJktMG3cXlBZZDR3q8iDcksb91KjbPLOSG4q2\nY4ch61f6rVxOlIhk3TKg0quAysqGGn5rzM7XMPu+YQJXw+x7ALwXuojS0qCNuwvmM39ojMn7GyAj\nb4M67wLnEksgbIGzWbAnP3ORde0jXNADuOLaiM/B5UGSh2p3NtSAEZaYLHsdUlPjUEZE9pDqKsiK\nNywxdfNoqLzYFVKcfZ9A+twA+KwBaC6ZE+Rg8sUCzkbBCqt1m3Z49v7zoo27K+J9/3iDSj5qxEQg\nM6spcPwYZN2bziVEZANZsxQ44bXfZXYrqFsmxPScnH2fOJSmQfNdB3DrRsi3JYHfQBYs4GwUrLB6\n9YuT1gO79QIu6RPxeXiDSj6qTTuowaMtMVm5AFJT7VBGRNGRqpOQVQstMTVkHFRuXkzPy9n3CeaK\na4Gu3S0hcwnHwoWCBZyNghZWWo7la23c5Ihb3wDeoJKVGjYByG7VFKissOyXS5RIZPVioOpEUyAn\nF2ro2Jifl7PvE4tSCprPjFR8tAnyzW5nEkogLOBsFLSwqvHarPeiS6EuviKq8/AGlZxU6zZY3+87\neOi6n2LigN/joet+iuItOyHVp5xOjSgscvKEZ89LL2roeKic1jE/N2ffJ6DLr/aMC/fCVriWscnG\nRlOKCiyznwAgq+40Ju9e3vi1NvbOQG8NS8ONiJvUJ5fikgrMMLuhJrt+9lx2e8zoOgpY8Q4GjR/q\ncHZEoZPVi4FTVU2B3DyoIfGbVc3Z94lFKQVt7J0w//bbpuC2zZA9u6B8ulepCQs4G/kVVnUnMHnn\nYgDAQ9f9FOVZ7ZD/WQamZFZEfXPhDSr5BBxDmZaJWUfyMLD6FJR39yqRS8nJE5A1iy0xdcsEqOyc\nIO8gAnDpVUDhRUDJF40hc/FspD32y2bfFuqSWsmIXag2G1DYFs9P6I4F15p4ZoPnaWJGz0koy24P\nUYpLflBQQcdQZraFrOWMVEoMfq1vrfOgBo10LiFKCA2tcBafvA8p+TLoe1J9SS0WcDHS0H8/68IR\nqEnLtLzGJT8okObGUMqqBZDqqoCvE7lFwNa3YWx9oxBd0ge4sKcl1NxYuFRfUosFXAzIl58Bn38M\nACjPahfwGC75Qb4CTk5pGEN5ohLyNlvhyN3Y+kbR8LTC+awLt/0DiFe3qrdUX1KLBVwMeD8xWGag\neuGSH+TLb/acqsEjO+ehf+k2AICsWshWOHIttr6RLXoXedZK9RJsd4ZUX1KLBZzN5It/Nba+AcDk\n3cuRpVmbeLnkBwXTMIZy4eReeG58N/Sv9Br/cbLSs7I9kQux9Y3sEHAs3PYPAo6FS/UltVjA2cxc\nan1S6H+WienXdeKaRBQ2ldPab+kFeWsRW+HIUYH2e2brG9nq4kCtcP5j4VJ9zb/UaGeME++xbw20\nMXdyyQ+KmBoyFrJ6CXCqfju2k5WQtcugRkxyNjFKSQ2z/hoGjjfM+jPTd6M/W9/IJg27M5h/+VVT\ncPsHkK+/hPJZ8DeV/76yBc5Gfk8IvS6HuugSZ5KhpKByWkMN9mmFW7WAuzOQI4LN+pt1vL0lxtY3\nilrvIv8ZqUvnOpSMO7GAs4ns+jxA69sdDmVDyUQNGQu08vpjeMLTCkcUb82tVdiIrW9kg4B7pH68\nBbLnK2cSciEWcDbxmyXT8zKoiy51JhlKKiqXrXDkDqHs96yGjmfrG9njkis9uzN44R6pTVjA2eD0\nzk+Bzz6yxPyeHIiioIaMBby30jpxHFK8PPgbiGKg2bUKAc+epzePciAzSkZBW+G+YSscwALOFifn\nvmANXHQpVE+2vpF9VG4e1M0+rXArF0Bqqh3KiFKR36y/mmOWtQrV0HFsfSN7XdonQCscx8IBLOCi\nJrt34vRHmywxjn2jWFBDxwJZXq1wlRWQdWyFo/hqWKtwftuP8cx7/9NYvCGnNdTNo51NjpKOpxXO\n52/qtk2Qb3Y7k5CLuGYZEV3XhwP4K4A0AM8bhvGEz+uq/vWRAKoA3GsYxta4J+rDb+xbj95Az8uc\nSYaSmmrdBurmUZDl8xpjsnI+ZOBIqKwsBzOjVCOnqiBvLbLE1NBxUK3Y+kYxcOlVQNfuwJ5djSFz\n6RykPfozB5Nynita4HRdTwPwJIARAHoDuFPX9d4+h40A0KP+f9MAzIhrkgFIyRfApx9aYtqYO6GU\nCvIOouiooeOBrOymQGUFx8JR3MnbS4GqE02BnFy2vlHMBBwL99EmyN4SZxJyCVcUcACuBbDLMIzd\nhmGcBjAHwDifY8YBeMUwDDEMYxOAdrqunxvvRL35tb51vxjodbkzyVBKUHltoAZZB4nLyvmQmhqH\nMqJUI9UBWt+GjIPKyXUoI0oJl1/taYXzkupj4UIu4HRd/z9d14tilEdnAN96fb23PhbuMXEjX38J\nbP/AEtPG3MHWN4o5NcynFe74MciGFQG3OCKym6xdBpysbAq0yoUa7N/6xp9HspNSCtro263BrRsh\ne792JB83CGcMXBqAlbqulwGYCWCWYRh7Y5NWdHRdnwZPNysMw0B+fr7t5zixZhFOen2d0fNStO83\nhAWcS6Snp8fkurtCfj4qR05E1YJZjaH1W3biqcMXo6bWBFC/xdGWQ8jLy8OwXh2cytRPUl+XBBXO\nNTFPVaHcp/Utd+ztaN3lAkts1Y5SPLnlkOt/Ht0sVX5XVu0oxdMb96C0sgYd8rLw8A1dg/6MyOCR\nOLL8ddTu/qIxlvHWQrT79/+KS65uuyYhF3CGYTyu6/oP4BmLNhnAL3Rd3wzgFQDzDcM40ewHNG8f\ngPO9vj6vPhbuMQ25Pgvg2fovpby8PIrUghg8DtoFPT2b13+6FXXDJ+Hw4cP2n4cikp+fj5hcd5eQ\nfsOBN+cBpz1dp6+e06/xj2WDmloTT23YjT75bhkpkfzXJRGFc03MFW9AKr1a0lrl4NT1Q1Dt8/6n\nNuxOiJ9HN0uF3xXfvXUPVdbgidVforKyMuj+pjJ8EvDU/zR+XfPeWpR9vBWqc5eY5xuva9KpU6eQ\njgtrFqphGHUAlgJYquv6JQBeA/ASgKd0XZ8D4FeGYQQsqlrwPoAeuq4XwlOU3QHgLp9jFgP4bv15\n+gKoMAzjQATnso3q1gtp3/s12lYexbHW7ZxMhVKMymsLNXAkZNUCAEB5VuCfv2BbHxGFS6pPQVYu\nsMTU4DFQua39jg32c1dWVYupC3ahvKoW+TnpmFJUkLIbkVPwvXVnbisL/nNR1Bc4rxBomMAgAnlz\nLtS0f49xtu4T1qOQruttdF1/QNf1tQDWA9gMoB+AiwGcABDRdDjDMGoBfBfASgCfe0LGv3Rdf1jX\n9YfrD1sGYDeAXQCeA/BoJOeKhYzCHuw6pbhTt4wHMjMBWLcy8hZs6yOicEnxcuDE8aZAdivPDiEB\nNPdzV1ZVC6n//yc3H+TYuBQWdG/dZh48PTNSrWPh5IN3IPu/sSWnRBq7GfLdXdf1eQBugadwexrA\nQsMwarxe/yGAiP9LDcNYBk+R5h172uvfAmB6pJ9PlGxUm/ZQA0ZA3lqEybuXY0bPSahJy2x8PStN\nYUpRgYMZUrKQmmr/1rebx0Dl5gU8fkpRgaVrLJgWW1soqeXnpKMsQLHW4oNn0XVA567Avj2er0Ug\nbxpQD/4oqnx8u3QbHjIAuPJnNJwWuE0AehiGMcowjLnexRsAGIZhAjjH1uyIqFnqlluBjEz0L92G\nR3bOQ0H1USgICnLSMb1vR1fedCjxSPFywHvsW1Yrz84gQfhtudXMH2R286eugHvrhvDgqTTNb3cG\neX8D5EB08yqb69J1o3AmMfwphGOqokuHiMKh2raHGjAcsnox+pdu82xr1D4f2n8/A5WR4XR6lASk\npgayYr4lpm4eBdW6TbPvG1DY1vIAMXXBrshaWyhpNfx8zNxWFv64yCuv92+FW2ZAPfDDiPOJpEvX\nSfzNcUhxSUVkP7REPtQtt0KKVwBnTnsCR8sh766GGjjC2cQoKQRufRsf9ucE6lZlNz/5FvqhUpoG\nNep2yLN/aIzJ5vWQUTpUx/MiyiXiLl2HcD63Axr62TmYl+yg2p0F1f8WS0yWvw45c8ahjChZSE0N\nZKVv69tIqLzmW98CCdStOr1vRwBImEHjFB+hTiRQV90AnOu1upiYkDeNiM8baZeuU1jAOSDR+tnJ\n/dTwW4F0ry7TI+WQjWucS4iSgmxYARz3muGclQ01dELEnzegsC2en9AdCyf3wvMTPNsi8WGWvIXT\nwKE0Dcpnj1TZvB5yMLKxcMEeMtzaO+bOdsEkl2j97OR+qt3ZnrFwa5Y0xmSZAblxMFQ6x8IlG7uH\nYAT6vP6ds/3Hvg2MrPUtmIjWAaOkFu7PhLrqBsi55wMH6nfarG+Fi3QsXKRduk5gC5wDgvWnu7Wf\nnRJDwFa4d1Y7lxDFhN1DMFbtKA34eetWbgQqjjYdmJkFdUvg1rdI187iwyz5CvdnIngrXCR7CiQW\nFnAOSLR+dkoMDa1w3jgWLvnYPQTj6Y17An7erPIcS0wNGgmV598yEU1ByYdZ8hXJz4TdY+ESBQs4\nByRaPzsljoCtcO+yFS6ZtLRNVbitYKWVNQHj5RleXaWZWVDDAre+RVNQ8mGWfEXyMxG4Fa446Vvh\n+JjjkETqZ6fEEXAs3PLXITcO4bpwSSLYUgcAGuMNrWCfl1Xhg30nmx0r1yEvC4cCFHHe27OpQSOh\n2oS3324o3aBRrQNGSSnSn4ngY+F+EOuUHcMCjijJqOETIetXNq0Ld4TrwiWTcLapWv5lUytcsG2B\nHr6hK55Y/aV1fba605i8u35r66xsz44fQUS7dhYfZslXJD8Tnla4OyDP/rExJpuL69eF62x3iq7A\nLlSiJMN14ZJbONtU+QrUtTmsVwfr59UcwyM753l29QCgBo0KOPatAbtByS0Cj4Wb61xCMcYWOKIk\nxFa45BbqNlWBBOrabPg8c+UCyLwXm17Iyg469s37vQC7Qcl5SksL0Aq3HjJShzo3st0Z3IwtcERJ\nKGAr3DK2wiWrQK1gwQTr2pSa6gC7LowKad033wV6WbyRU9RVNwKdujQFxIQsneNcQjHEAo4oSanh\nE4GMzKbA0XLIhpXOJUQxE6hbdUSPtmF1bcraN/33PG2h9Y3IbZSmQRvrMyP1/Q2Qhk3vkwi7UImS\nlGp3lmdG6urFjTFZNg9y01CozCwHM6NYCDTw++KCnJC6NqX6FGTlAktMDR4N1dq+XReI4ubK64Hz\nLgD2fu35WgTmktlIe/gnTmZlOxZwRElMjagfC3e6fpmIiiOQdcuhho13NjGKi1Bn88naN4ETx5sC\n2a34M0IJy9MKdxfMp/6nKfjhRsi3JVDnFzqXmM3YhUqUxFSb9lCDRllisuINSPUphzIitzFPnvDf\n83TwGKjcPIcyIrJBUV+gSzdLyFw826FkYoMFHFGSU7fcCmS1agpUVkDWLnMuIXKVqqVzgaoTTYFW\nOVBDxzmXEJENlFLQxt1lDW7bBNmzy5mEYoAFHFGSU3ltoAaPscRk5XzIqSqHMiK3kBPHUbXYOkNP\nDR3P1jdKDpddDRReZAklUyscCziiFKCGjQda5TYFTlZC1iwO/gZKWsUlFY17pj64uATFrXs0vZib\nBzVkrHPJEdlIKQVtrE8r3CfvQ0q+cCYhm7GAI0oBKre1X7eYrFoEOXkiyDsoGRWXVODJzQdRVlUL\nAVAmWZjRcxLWdygCAKjht0K1ynE2SSI7XXIl0K2XJWQunOVQMvZiAUeUItTgMUBO66bAqZOQtxY6\nlxDF3cxtZX57qNakZWLWhSOANu38JrwQJTrPWLjJ1uBnH0F2fupMQjZiAUeUxCzdZSsPYP3Aey2v\ny+olkMrjgd9MSSfQNloAUJ7VDmrkbVBZ2XHOiCgOel0O9LzMEjIXzoSIBHlDYmABR5Sk/LrLqmox\no6Yr1ne5vumgmlOQ5a87liPFV7BttPLPHPfbeo0oWSiloI2/2xrc9Tnw6YfOJGQTFnBESSpgd1md\nYFb30ZaYrF0GOVIWz9TIIVOKCpDlc9fPqjuNyR1PQ3lvu0aUZNandcJD/X6FiQN+j4eu+ynWdyiC\nufBViGk6nVrEWMARJamg3WVmBtA+vylQewaydG6csiInDShsi0dOf4yC6qNQIiioPorpB9dg4NDr\nW34zUYJq7I1Iy4UohbLs9p7JO9VtgK0bnU4vYtxKiyhJ5eekoyxAEZefkw41+nbIzCcbY/Luasiw\nCVAdO8czRYoz2f8N+r87G/2lqdWhzWO/wMl0/img5NXc5J3+i16BduX1UGlpDmUXObbAESWpKUUF\nyEpTllhWmsKUogKoG4cAHTo1vWCakMWvxTlDijdzwauAV/GGjuche8Aw5xIiioPmJu/g4D7IprVx\nzsgeLOCIktSAwraY3rcjCnLSoQAU5KRjet+OGFDYFiotDWq8dWq9vL8B8s1XziRLMSdf7QC2bbLE\ntAlToNLY+kbJLejknZpjAABZMgdy5kw8U7IFf3OJXK64pAIzt5WhvKoW+TnpmFJUgAGFbUN674DC\ntkGPVVfdCDl/HvBtSWPMXPAq0r73q5jlQ84QEZjzX7EGCy8CrrzOmYSI4mhKUQGe3HzQ0o2aVXca\nk3cv93xxuBSyYSXUzaODfII7sQWOyMUCLQXy5OaDKC6piPqzlaZBmzDFGvz0Q8gX/7IlH+816KYu\n2GVLzhShf20FvrAuXKrdeg+UUkHeQJQ8AvVGPJK+G/1LtzUeI0vnQqoTa39oFnBELhZsKZCZ22xa\n9uPSq4DuvS0hc0HwBS5DzSdYobdqR6k9eVPIxDRhvuHT+nbJlVC9LncmISIHDChsi+cndMfCyb3w\n/ITuGDh6AJCZ1XRAZQVkVWLtTMMCjsjFgg6+DRIPl1IK2q33WIO7PgM+eT+qfIIVek9v3BN5shQR\neX8DsLfEEtMm3BPkaKLUoNq0D7A/9EJIxVGHMgofCzgiFws6+DZIPBKqR2/gsqstMfONlyF1dRHn\nE6zQK62siTBLioTUnoEssm7cra7pB9W1m0MZEbmHuuVWoHWbpkBNNWTpHOcSChMLOCIXa24pEDtp\nt04BvMdDHfgW8u5bEecTrNDrkJcVME6xIRveAsoONgUCzD4mSlWqVQ7U6DssMVm/EnJwr0MZhYcF\nHJGLNbcUiJ3UeYVQN9xsicmi1yDVpyLKJ1ih9/ANXW3Nm4KTU1WQJbMtMXXTUCjv9f+IEkSsJkWp\nAbcABR2bAqYJc8FMWz471riMCJHLNbcUiJ3U2Mme8VKnT3sCx49BVi2EGntn2Pk0vO673MiwXh1Q\nXl4ek/zJSla8AVR6/ZHLzIQafbtzCRFFqGFSVMO42oZJUQCivjeq9AyoCfdAnv1DU3Dre5CvdkB1\n6xXVZ8caW+CICACgzsqHGuI7qHdBxIN6fWd9ca24+JEjZZC3FlliatgEqHZnO5QRUeRiPRtfXX0j\ncEEPS8yc91LQ2fhuwQKOiBoFHNS7eHbwN5AryYJXgTOnmwJt2kHdMsG5hIiiEJfZ+JPutQZ3fQZ8\nvNmWz48VFnBE1Ejl5EKN8RnU+84qyIFvHcqIwiV7dvnt7ajGTYbKznEoI6LoxGU2fs/LAszGfwVS\na0+RGAss4IjIQvUf7rfRvfnGy84llECc3n1CRGC+/qI12Lkr1E1D4poHkZ3iNxv/Huts/IN7IcUr\nbD2HnRyfxKDr+lkA5gK4AMDXAHTDMPwG3ei6/jWASgB1AGoNw7ja9xgiip5KT4d26z0wn36iKfjx\nFsjO7Z6nVAoolgOtQ/bxFmDndktIm3QflJYWn/MTxUCwSVH2z8a/AOqGwZB3VzfGZPFrkOsGQOXm\n2XouO7ihBe4nANYYhtEDwJr6r4MZZBhGEYs3ohjrcz3gMwPLnPNcwMV9ySPm2561QGprYc57yRq8\n5Auw5NgAAB4vSURBVEqoS/vE5fxEsRSvSVFq/N1AVqumQNUJ144DdkMBNw5AQ//MywDGO5gLEaF+\nUO9t91uDe7+GbFjlTEIJINYDrVsi61cAh/Y1BZQGbdJ9cTk3UbJQ7c6CGjnJEpN1yyD7v3Eoo+Dc\nUMCdYxjGgfp/HwRwTpDjBMBqXdc/1HV9WnxSI0pdqlsvqOsGWmKy6FXIyRPOJORy8RhoHYycrAyw\naO8QqPMuiPm5iZKNGjoOyPcqRUwT5usvOJdQEHEZA6fr+moAHQO89HPvLwzDEF3Xgy28cpNhGPt0\nXe8A4C1d13cYhrE+yPmmAZhW/5nIz8+PIvuWpaenx/wcFB5eE3vUTf0Byj/aBNRUewInKpH11gK0\nmfoDy3GrdpTi6Y17UFpZgw55WXj4hq4Y1quD3+cl83V5tJ+JJ9bsQk2t2RjLStfwaL8LY/7ffPyN\nl3DqRGXj1yq7Fc6+7zGktW953bdkviaJjNfFWdX3P46KP3iVKJ9uRe22zcgv6utcUj6U0wvV6bq+\nE8BAwzAO6Lp+LoB1hmH0bOE9vwZwwjCMP4VwCtm/f78NmQaXn5/P1eVdhtfEPuabBmThq00BTYP2\nH3+D6twFgP/gfcAzQyzQFlvJfl2KSypiPtDal3yzG+Z//RCQpsJRjb8b2ig9pPcn+zVJVLwuzhIR\nmH/6GfDFvxpjaZ27QH7xF6j02LZ9derUCQBUS8e5oQt1MYDv1P/7OwAW+R6g63qurut5Df8GMAzA\np3HLkCiFqWHjgbO9WtNME6bxfOMq5U4P3neTeO8+ISIwZz9rKd7Q4VyoYVy0lygaSilot0+1LCtS\nt+8byLplDmZl5YYC7gkAQ3Vd/xLAkPqvoet6J13XG75T5wB4R9f1jwFsAfCmYRjuXZyFKImojExo\nus+Ehs+2Na5S7vTg/VQmm4s9K8Z70W6fCpWR4VBGRM6zaz1G1aUb1E1DLTFZMhtSedyONKPm+Dpw\nhmEcBjA4QHw/gJH1/94N4Io4p0ZEDa68Huh5mWWNMdN4AdolVyE/Jx1lAYq1eAzeT2VSXQXxXTbk\n8mugLr/GkXyI3MDu9RjV+Mko3n0Ms84fjPKsdsivOYbJi1dj0ORbbc07Em5ogSMil1NKQbvjQUB5\n3TLKDkJWzo/bKulkJUvnAhVHmgLp6dBuf8C5hIhcwO4hHesPa5jR41aUZbeHKIWy7PaYoXrFfZeV\nQFjAEVFI1HkXQA0YbonJmwb6557E9L4dUZCTDgWgICc94AQGso8c2AtZvcQSU8MmQHlvgUaUguwe\n0jFzWxlqfEqlGtFcMcaXfRxEFDI1fjLkg3eAE/VjQGrPwJz1NPp//zcs2OJERGDOeQ6o8/qD1D4f\nauRtziVF5BJ2D+lw8xhftsARUchUbh6U7w4Nn22DbAm4JCPFgGxZD3z2kSWmbrsfKivboYyI3MPu\nIR1OLtDdEhZwRBQWdf0gz4QGL2L8E1LFHRpiTU4ch8x93hrseRnU1Tc6kxCRywwobGvrkA43j/F1\nvoQkooSilIJ29yMwf/M4UFvfjXD8GGTBTKjJjzibnAvZubivvP4iUOk1eDo9A9rdj0CpFtf8JEoZ\nAwrb2jako+Fz4r1AdyhYwBFR2FTH86CGT4IsndMYk+IVkOtvhrqw2Y1UUoqdSxrI5x9DNq6xxNQo\nHarjefYkS0QBNRSEbtsdg12oRBQRNXIS4D3rUQTmzKcgdXXOJWUDuxYBBexZ0sCTz5eY8GEmHrru\np1jfocjzQqcuUMOdX4uKiJzBAo6IIipaVEYmtMkPW4N7SyCrFsYoy9hraDErq6qFoKnFLNIiLtoZ\nbE351DWtQdVzEtZ3uBLaPd+FSueOC0Spil2oRCkumm4+1bsIqu8Az5ZO9WTxLMjl1zRudp9Immsx\ni2TMS7hLGviOl6uuM/3zScvErN4TMKhbr7DzIaImdo5PdQJb4IhSXLTdfEp/AMjNawrU1sJ88S+Q\nWnvXSbKzazMYu9d8CmcGW6DWv8oa0+84ACgHlwwhiobdre1OYAFHlOKiLVpUm3ZQdz1kDe7ZBVkx\nL9rUGsXrZmv3mk/hLGkQqJAON08iCo3dW245gXcBohRnx8rl6pp+kK0bgQ83NsZk6VzI5ddCdbkw\n6hzt7toMZkpRgaU7GYh+zadQlzQIWjCLAF7LhLhlDSqiRObmHRZCxRY4ohRnx0KVSilokx8B8rwK\nlbq6+q7UM1HnGK+brd2LgIYjWMGcd+YkCqqPQomgoJXGfWaJbODmHRZClTiZElFM2LVQpcprC+3u\nR2HO+F1TcO/XkCVzoSbcHVWOdu9v2Bw7FwENR8DWv7rTeGDXYvQv3QZt+s+hivrGPS+iZBSL1vZ4\nYwFHRLYVLarP9f6zUlfMgxRdC1V4UcSfmww325Y0FtLv70P5aYX8mmOYvHs5+pdug+o3jMUbkY3c\nvMNCqFjAEVFAkU6xV3c+BNm5HTh2xBMwTZjP/QnaL/4PQH5EuSTDzTYU/QsUbtryB6DiSFOwoCOU\nfr9zSRElKada2+3CAo6I/ES1Nlxua2j3PAbzb79pCpYdhLzyD8jP/xBxTol+s22JiEBenWEt3pQG\n7f4fQGXnOJcYEbkSJzEQkZ+o14a77CqogSMtMfnwXZxaMd+2HJONFK+AfPiuJaZGTILqfrFDGRGR\nm7GAIyI/dsz6VPr9gM8SIpUv/A2y56uocktGUvIFZO5z1mCXblBjbncmISJyPRZwRCkmlB0N7Jhi\nrzIyoT30YyC7VVOw9gzMZ34PqToZdt7JSiqPw3z6CcB754qsbGhTf8i9TokoKBZwRCkk1B0N7Fgb\nDgBUh05Q9zxmDTaMh5PQdh1IZmLWwXzuj8CRcktcfedxqHPPdygrIkoELOCIUkioY9vsXNBWu+am\ngOPh5O2lYX9WspHFs4HPP7bE1JBx0K65yaGMiChRcBYqUQoJZ2ybnbM+lX4/ZPcO4JvdjTEx/gk5\npxPUpVfZco5EIx+/D3nTsAa794aa+B1nEiKihMIWOKIU4tT2MQ3j4VQrr+UwTBPmM3+A7NsT03O7\nkRzcC/OFP1uDbdt7vkfpfK4mopaxgCNKIXaNbYuE6tAJbX/wG0B53XaqT8H8+39Cjh+N+fndQiqO\nwvzLrwHviRyaBm3aj6HaneVYXkSUWFjAEaUQJzdrB4Csa26E0u+zBg+XwvzHf0NO18QlBydJdZVn\ngePDpZa4mngv1EWXOJQVESUittUTpRindzRQg8cCh/ZD1i1vCpZ8AXnhL8C0f4fSkvO5UmrPwJzx\ne8s4QABQNw2FGjrOoayIKFEl552SiFxLKQV1xzSg95WWuHz4LmT+y0m5vIiIQF7+B/DZR9YXLrsa\n6u5HoZQK/EYioiBYwBFR3Km0NM8iv526WOKycgFk4atJV8TJgpmQTWutwQt6eCYtpKU5kxQRJTQW\ncETkCJWTC+2xXwJ51u5cWfZ60hRxIgJz6VzI8nnWFwo6Qnvsl1BZ2c4kRkQJjwUcETlG5Z8D7Xu/\nAnJyLXFPETcroYs4EYG88TJk0SzrC3ltoX3/11Bt2jmTGBElBRZwROQo1bU7tB/8NkARZ0AWJWYR\nJ6YJee1pyMr51hcyszwtbx06OZMYESUNFnBE5Dh1QQ9PEdfKp4h70/C0YpmmM4lFQOrqIC/+1TrL\nFgBa5UD7/m+gCi9yJjEiSios4IjIFYIWcSvnw5zxO0h1lTOJhUHOnMb/b+/+o6Mq7zyOv+9kICGS\nQmECGEDIKixSl4bUErRYtAYaXU6peHxal812tfhjT7rVmB6K9Ry3f9RVOaWKltbyq3Uph/q0EuAc\nW2hQG4sIFZEW/J2K8qvB5IAICQlM5u4fE5CBhAyTydw7mc/rnDnM3Hnm3u/Mkzt853nu8zyRxfPP\nHbDQP49A1Y9wxoz3JjAR6XWUwImIbziFHSdx7NhK5JHv4zbUexNYHNyGeiKPfB+2vxL7xIBBBL73\nMM6oy7wJTER6JSVwIuIrp5O4s0ansv9DIv9bhfvOzoT2W7v7CHOq6/j6yreZU11H7e4jSYg2yt2x\nlciPKmHP32OfGDyEwNyHcYZf0vELRUQSpARORHzHKRxD4IGfwMjC2CeOHSXy2INENq7FjbTFvb/a\n3UdYtLWehuYwLtDQHGbR1vpuJ3FuWxuRZ58msuih2LVNAYYNJzD3EZwhF3frGCIiHdFSWiLiqdrd\nR1ixo4HG5jCh3CDlRflMLRyAMzifwPcfjQ4IeO3lT1/Q1ob7zDLcVzcRKK/AGTG6y2Os2NFAa1vs\naNbWNpcVOxoSXlbMbagn8qsn4N1d5z5ZNJnAbd/Fye2f0L5FRLqiBE5EPHOqZexUcnWqZQyia7Y6\n2Tlw11x4zp47n9r77xD5USXO9JtwZnwDp292p8dpbA5f0PbzcY83R+ep27gWwme9PhCILkw/baaW\nxxKRHqUETkQ8E0/LmOM4ODO+gVtwCZFfPg4txz8t3NaG+4ff4W7bRMB8GyZ8ESdw7pUhodwgDR0k\na6Hc+L8C3Ugb7svP4675NXzy8bkFBg4icOfcuEeadtbyKCISDyVwIuKZC2kZc4qvIjB6DJFVi2HH\nltgnG+qj16ENKcC5fgbO1V/Byck9/XR5UX5MSx9AdpZDeVF+lzG6LcdxX9uM+/w62Lu740KXf57A\nnKq4V1foquVRRKQrnidwxphbgB8ClwOTrLXbOilXBiwEsoCl1tpHUhakiPSIC20ZcwaFyKr4Ae72\nV4is+gV8fCi2wEcHcFctxl3za5wvTcO56loYMfp0UhRvi5cbicC7u3A3v4C7fTO0tnT8BgYOxrn5\nP3AmTe2w5a8zPXFNnohkFs8TOGAXMAv4RWcFjDFZwCJgGrAPeNUYs85a+2ZqQhSRnpBoy5hTfBWB\ncRNwq1fg1v4Bzl5u63gz7sa10evUcvrBpeO45rLxfHn8ePhsCPr0hT4ObmsLBLKgsR72f4i7fw/u\ngQ9h93twuLHzAPr0xfnqLJyyWQktSJ/Ma/JEJDN5nsBZa98CMMacr9gkoM5a+3572d8AMwElcCJp\n7EJbxs7k5F6EM/tu3Gum4dasxX11E7R1kAC1HIc3Xsd943WSsaqq88VrogMVBnfd/dqZZFyTJyKZ\nLV2+LYYDe894vA8o8SgWEUmiqYUDutVt6FxyKc6378O9+T9xa/+AW7sejiZvkl6Al0ZPYWXhV2l0\ns6NJ5id9mTo48f1155o8ERFIUQJnjNkIDOvgqQestWt74Hh3AncCWGsJhULJPkSMYDDY48eQC6M6\n8acerZdQCC4bi1t+Ny2bNtK6pZYTb+/ETTSZ69uXnMnXsmn8dH5eB63hCNA+4OAvB8nLy2P6uCEJ\n7frmUIi8vDye2vwhHx1tZUheNndfPSrh/XWHzhV/Ur34j9/qJCUJnLW2tJu72A+MPOPxiPZtnR1v\nMbC4/aHb2Hiea1mSIBQK0dPHkAujOvGnZNdLp1NxTCiBCSU4rotTvx+37k14703cPX+PDkg4eRLC\nJ+Hkiei/eQNg+CicglEw/BKc4aOgYBQns7NZUl1H61nzvbWGI/zsz+9THEp8MZviUIDFX4tdacKL\nv1mdK/6kevGfVNVJQUFBXOXSpQv1VWCMMaaQaOL2TeDfvA1JRLwUz1QcjuPAxSNwLh4B10xP6Did\nDSxoaA4zp7pO87iJiCc8XwvVGHOTMWYfcBXwnDFmQ/v2AmPM7wGstWHgO8AG4K3oJvuGVzGLiPfO\nNxVHMp1vYEGy11YVEYmX4549/L73cQ8cONCjB1BTt/+oTvwpmfXy9ZVvdziq1AHWzB6XlGPAuS19\n55OfG2TpTZcl7dipoHPFn1Qv/pPiLtQu1+LzvAVORCQRnbWMJXsqjqmFA6goGUZ+bhCHaJLWGc3j\nJiKpki7XwImIxEjlVBxnT3Uyp7pO87iJiKfUAiciaamjlrGKkmEpGUhQXpRPdlZsD4fmcRORVNLP\nRRFJmT++/RE/+/P7SRu52d1JgLtzXEhsBQkRkWRQAiciKVG7+wiL/nIwdkLcs6b9SCdeJY8iIqAu\nVBFJkRU7Gk4nb6f0xLQfIiKZQAmciKREZyM0NXJTROTCKYETkZRI1bQfIiKZQAmciKREeVE+2cHY\nrxyN3BQRSYx++opISkwtHEBeXl5SR6GKiGQqJXAikjLTxw2hOKSGfxGR7tI3qYiIiEiaUQInIiIi\nkmbUhSoivlO7+4hWORAROQ8lcCLiK7W7j8QsUp/uKzaIiPQEdaGKiK+s2NFwOnk7RSs2iIjEUgIn\nIr6iFRtERLqmBE5EfEUrNoiIdE0JnIj4SnlRPtlZTsw2rdggIhJLP2lFxFdODVTQKFQRkc5lZALn\nui4tLS1EIhEcx+n6BV04ePAgra2tSYgsdVzXJRAIkJOTk5TPQCSZphYOUMImInIeGZnAtbS00KdP\nH4LB5Lz9YDBIVlZWUvaVSuFwmJaWFvr16+d1KCJJpXnkRKS3y8gELhKJJC15S2fBYDDtWg5FuqJ5\n5EQkE2TkIAZ1GX5Kn4X0NppHTkQyQUYmcL1JSUkJhw4d6nYZkd5C88iJSCZQAicivYrmkRORTJDR\n32htd3wtOfvpZHvWknUdbt+7dy+zZ8+muLiYbdu2UVRUhDGGBQsW0NjYyE9/+lNGjx5NVVUVe/bs\nIScnh/nz5zN+/HgOHTpERUUF9fX1fOELX8B1P+0qevbZZ1m+fDknTpxg4sSJPPzww2k5uEKkO8qL\n8mOugQPNIycivY9a4DzywQcfcNddd/HSSy9RV1fHmjVrWLNmDQ8++CBPPvkkCxYs4IorrmDjxo3M\nmzePe+65B4DHHnuMSZMm8eKLL1JWVsb+/fsBeO+991i3bh1r1qyhpqaGrKwsVq9e7eVbFPHE1MIB\nVJQMIz83iAPk5wapKBmmAQwi0qtkdAucl0aOHMnll18OwNixY5kyZQqO4zBu3Dj27t3Lvn37WLJk\nCQBTpkzh8OHDHD16lC1btrB06VIASktLGThwIACbNm1i586d3HjjjUB0qpRQKOTBOxPxnuaRE5He\nTgmcR7Kzs0/fDwQC9O3b9/T9tra2C57mxHVdbrnlFu6///6kxikiIiL+k9EJXGfXqF2oYDBIOJzc\nEW4lJSWsXr2ayspKNm/ezKBBg8jLy2Py5MlUV1dz77338sILL/Dxxx8D0Va62267jTvuuINQKMTh\nw4dpampixIgRSY1LREREvJfRCZyf3XfffVRVVVFaWkpOTg6PP/44AJWVlVRUVHDddddx5ZVXMnz4\ncCDaDTt37lxuvfVWXNclGAzy0EMPKYETERHphZwzRzH2Uu6BAwdiNjQ3N5Obm5u0A/REC1yqJPuz\n8ItQKERjY6PXYchZVC/+ozrxJ9WL/6SqTgoKCgC6nGVfo1BFRERE0owSOBEREZE0owROREREJM0o\ngRMRERFJM0rgRERERNKMEjgRERGRNKMEzgcWLFjAU0891enz69ev5913301hRCIiIuJnmsg3DrW7\nj7BiRwONzWFCuUHKi/JTus7i+vXrKS0tZezYsSk7pkgm8PrcFhFJlBK4LtTuPsKirfW0tkUnPG5o\nDrNoaz1At77oFy5cyG9/+1tCoRAFBQVMmDCBlStXsnLlSk6cOEFhYSFPPPEEu3btoqamhi1btrBw\n4UKWLFnCyy+/fE65fv36JeX9imSKnjq3RURSwfMEzhhzC/BD4HJgkrV2WyflPgCOAm1A2Fp7ZSri\nW7Gj4fQX/CmtbS4rdjQk/CX/t7/9jXXr1lFTU0M4HKasrIwJEyZwww03MHv2bAAeffRRVq1axe23\n3860adMoLS1lxowZAHzmM5/psJyIxK8nzm0RkVTxPIEDdgGzgF/EUfY6a21K1xZpbO54iazOtsdj\n69atlJWVnW41mzZtGgDvvPMO8+fP55NPPqGpqYmpU6d2+Pp4y4lI53ri3BYRSRXPEzhr7VsAxhiv\nQ+lQKDdIQwdf6KHc5H90lZWVLFu2jM997nM888wzvPLKK90qJyKdS+W5LSKSbOk0CtUFNhpjXjPG\n3Jmqg5YX5ZOdFbumbHaWQ3lRfsL7nDx5Mhs2bOD48eMcO3aMmpoaAI4dO8bQoUM5efIk1dXVp8v3\n79+fpqam0487Kyci8euJc1tEJFVS8lPTGLMRGNbBUw9Ya9fGuZsp1tr9xpghQI0x5m1r7UudHO9O\n4E4Aay2hUCjm+YMHDxIMxvfWrx8zmEBWFk9vP0hj00lCF/XhW8VDue6fBsaUi3d/ABMnTmTmzJlM\nnz6dUCjExIkTCQQCzJs3jxkzZjB48GCKi4tpamoiGAwya9YsqqqqWL58OcuWLeu0XCKys7PP+Xx6\ng2Aw2CvfV7rzU73cHAqRl5fHU5s/5KOjrQzJy+buq0cxfdwQr0NLKT/ViXxK9eI/fqsTx3Xdrkul\ngDHmT8D3OhvEcFbZHwLHrLU/jmPX7oEDB2I2NDc3k5ubm0iYHQoGg4TD6XndTLI/C78IhUI0Nqb0\nckmJg+rFf1Qn/qR68Z9U1UlBQQGA01W5tOhCNcZcZIzJO3UfmE508IOIiIhIxvE8gTPG3GSM2Qdc\nBTxnjNnQvr3AGPP79mJDgU3GmL8CfwGes9au9yZiEREREW/5pgu1B6kL9TzUhSqppHrxH9WJP6le\n/EddqD6QAUlr3PRZiIiIpJ+MTOACgUDatpglUzgcJhDIyD8BERGRtJaRM1bm5OTQ0tJCa2srjtNl\nK2WXsrOzaW1tTUJkqeO6LoFAgJycHK9DERERkQuUkQmc4zhJXfxd1yqIiIhIKqn/TERERCTNKIET\nERERSTNK4ERERETSTEbMA+d1ACIiIiIXQPPAEf0QevRmjHktFcfRTXWS7jfVi/9uqhN/3lQv/rul\nuE66lAkJnIiIiEivogROREREJM0ogUuOxV4HIOdQnfiT6sV/VCf+pHrxH1/VSSYMYhARERHpVdQC\nJyIiIpJmMnIprWQxxpQBC4EsYKm19hGPQ8p4xpiRwP8BQ4lOIbPYWrvQ26gEwBiTBWwD9ltrZ3gd\nj4AxZiCwFLiC6Plyu7X2FW+jymzGmEpgDtH62AncZq1t8TaqzGOMWQ7MAD6y1l7Rvm0Q8AwwGvgA\nMNbaw17FqBa4BLX/Z7QIuAEYD9xqjBnvbVQChIEqa+14YDJQoXrxjXuAt7wOQmIsBNZba8cBn0f1\n4yljzHDgu8CV7UlDFvBNb6PKWL8Cys7aNg943lo7Bni+/bFnlMAlbhJQZ61931p7AvgNMNPjmDKe\ntfYf1trt7fePEv0Pabi3UYkxZgTwr0Rbe8QHjDEDgC8DywCstSestR97G5UQ7RnrZ4wJArnAAY/j\nyUjW2peAQ2dtngk83X7/aeDrKQ3qLErgEjcc2HvG430oUfAVY8xoYCKw1eNQBB4H5gIRrwOR0wqB\nBuCXxpjXjTFLjTEXeR1UJrPW7gd+DOwB/gEcsdb+0duo5AxDrbX/aL9fT/RSHc8ogZNeyRjTH3gW\nuNda+4nX8WQyY8yp60he8zoWiREEioGfW2snAk143CWU6YwxnyXaylMIFAAXGWP+3duopCPWWheP\nl+pUApe4/cDIMx6PaN8mHjPG9CGavK201q72Oh7hS8DXjDEfEL3U4CvGmF97G5IQ7TXYZ6091UL9\nO6IJnXinFNhtrW2w1p4EVgNXexyTfOqgMeZigPZ/P/IyGCVwiXsVGGOMKTTG9CV6oek6j2PKeMYY\nh+g1PW9Za3/idTwC1tr7rbUjrLWjiZ4nL1hr1argMWttPbDXGPPP7ZuuB970MCSJdp1ONsbktn+X\nXY8GlvjJOuBb7fe/Baz1MBZNI5Ioa23YGPMdYAPRkULLrbVveByWRFt7yoGdxpgd7dt+YK39vYcx\nifjVfwMr23+Evg/c5nE8Gc1au9UY8ztgO9ER9a/js9n/M4UxZhVwLRAyxuwD/gd4BLDGmG8DHwLG\nuwi1EoOIiIhI2lEXqoiIiEiaUQInIiIikmaUwImIiIikGSVwIiIiImlGCZyIiIhImlECJyIiIpJm\nlMCJiIiIpBklcCIiIiJpRisxiIjEwRhzKdEl9EqttduNMQXAX4FbrLV/8jQ4Eck4WolBRCROxpg7\ngErgSqAa2Gmt/Z63UYlIJlIXqohInKy1S4A6YCtwMfCAtxGJSKZSAicicmGWAFcAT1prW70ORkQy\nk7pQRUTiZIzpT/S6txeBG4B/sdYe8jYqEclEaoETEYnfQmCbtXYO8BzwlMfxiEiGUgInIhIHY8xM\noAz4r/ZN9wHFxpjZ3kUlIplKXagiIiIiaUYtcCIiIiJpRgmciIiISJpRAiciIiKSZpTAiYiIiKQZ\nJXAiIiIiaUYJnIiIiEiaUQInIiIikmaUwImIiIikGSVwIiIiImnm/wErsDjgmbjkSgAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2e1e15eecf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(x, y_pred, linewidth=4, label='model')\n",
    "plt.plot(x, y_true, 'o', label='data')\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "plt.legend(loc='lower left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The most straightforward metric to determine how good our model predictions are is the\n",
    "mean squared error. For each data point, we look at the difference between the predicted\n",
    "and the actual `y` value, and then square it. We then compute the average of this squared\n",
    "error over all the data points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.085318394808423778"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mse = np.mean((y_true - y_pred) ** 2)\n",
    "mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For our convenience, scikit-learn provides its own implementation of the mean squared\n",
    "error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.085318394808423778"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.mean_squared_error(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Another common metric is to measure the scatter or variation in the data: if every data\n",
    "point was equal to the mean of all the data points, we would have no scatter or variation in\n",
    "the data, and we could predict all future data points with a single data value. This would be\n",
    "the world's most boring machine learning problem. Instead, we find that the data points\n",
    "often follow some unknown, hidden relationship, that we would like to uncover. In the\n",
    "previous example, this would be the $y=sin(x)$ relationship, which causes the data to be\n",
    "scattered.\n",
    "\n",
    "We can measure how much of that scatter in the data (or **variance**) we can explain. We do\n",
    "this by calculating the variance that still exists between the predicted and the actual labels;\n",
    "this is all the variance our predictions could not explain. If we normalize this value by the\n",
    "total variance in the data, we get what is known as the **fraction of variance unexplained**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.16397032626629501"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fvu = np.var(y_true - y_pred) / np.var(y_true)\n",
    "fvu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Because this metric is a fraction, its values must lie between 0 and 1. We can subtract this\n",
    "fraction from 1 to get the fraction of variance explained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.83602967373370496"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fve = 1.0 - fvu\n",
    "fve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's verify our math with scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.83602967373370496"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.explained_variance_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Spot on! Finally, we can calculate what is known as the coefficient of determination, or $R^2$\n",
    "(pronounced R squared). $R^2$ is closely related to the fraction of variance explained, and\n",
    "compares the mean squared error calculated earlier to the actual variance in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8358169419264746"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2 = 1.0 - mse / np.var(y_true)\n",
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The same value can be obtained with scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8358169419264746"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.r2_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The better our predictions fit the data, in comparison to taking the simple average, the\n",
    "closer the value of the $R^2$ score will be to 1. The $R^2$ score can take on negative values, as\n",
    "model predictions can be arbitrarily worse than 1. A constant model that always predicts\n",
    "the expected value of `y`, independent of the input `x`, would get a $R^2$ score of 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.r2_score(y_true, np.mean(y_true) * np.ones_like(y_true))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!--NAVIGATION-->\n",
    "< [First Steps in Supervised Learning](03.00-First-Steps-in-Supervised-Learning.ipynb) | [Contents](../README.md) | [Understanding the k-NN Classifier](03.02-Understanding-the-k-NN-Algorithm.ipynb) >"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}