t81_558_class_04_2_multi_class.ipynb

{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# T81-558: Applications of Deep Neural Networks\n", "**Module 4: Training for Tabular Data**\n", "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n", "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Module 4 Material\n", "\n", "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_1_feature_encode.ipynb)\n", "* **Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC** [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_2_multi_class.ipynb)\n", "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_3_regression.ipynb)\n", "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_4_backprop.ipynb)\n", "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_5_rmse_logloss.ipynb)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Google CoLab Instructions\n", "\n", "The following code ensures that Google CoLab is running the correct version of TensorFlow."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Note: not using Google CoLab\n"]}], "source": ["try:\n", "    %tensorflow_version 2.x\n", "    COLAB = True\n", "    print(\"Note: using Google CoLab\")\n", "except:\n", "    print(\"Note: not using Google CoLab\")\n", "    COLAB = False"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC\n", "\n", "* **Binary Classification** - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).\n", "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n", "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n", "\n", "In this class session we will look at some visualizations for all three.\n", "\n", "\n", "It is important to evaluate the level of error in the results produced by a neural network.  In this part we will look at how to evaluate error for both classification and regression neural networks."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Binary Classification and ROC Charts\n", "\n", "Binary classification occurs when a neural network must choose between two options, which might be true/false, yes/no, correct/incorrect, or buy/sell.  To see how to use binary classification, we will consider a classification system for a credit card company.  This classification system must decide how to respond to a new potential customer.  This system will either \"issue a credit card\" or \"decline a credit card.\"  \n", "\n", "When you have only two classes that you can consider, the objective function's score is the number of false positive predictions versus the number of false negatives. False negatives and false positives are both types of errors, and it is important to understand the difference. For the previous example, issuing a credit card would be the positive.  A false positive occurs when a credit card is issued to someone who will become a bad credit risk.  A false negative happens when a credit card is declined to someone who would have been a good risk.  \n", "\n", "Because only two options exist, we can choose the mistake that is the more serious type of error, a false positive or a false negative.  For most banks issuing credit cards, a false positive is worse than a false negative.  Declining a potentially good credit card holder is better than accepting a credit card holder who would cause the bank to undertake expensive collection activities.\n", "\n", "Consider the following program that uses the [wcbreast_wdbc dataset](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/datasets_wcbc.ipynb) to classify if a breast tumor is cancerous (malignant) or not (benign).  "]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>id</th>\n", "      <th>diagnosis</th>\n", "      <th>mean_radius</th>\n", "      <th>...</th>\n", "      <th>worst_concave_points</th>\n", "      <th>worst_symmetry</th>\n", "      <th>worst_fractal_dimension</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>0</th>\n", "      <td>842302</td>\n", "      <td>M</td>\n", "      <td>17.99</td>\n", "      <td>...</td>\n", "      <td>0.2654</td>\n", "      <td>0.4601</td>\n", "      <td>0.11890</td>\n", "    </tr>\n", "    <tr>\n", "      <th>1</th>\n", "      <td>842517</td>\n", "      <td>M</td>\n", "      <td>20.57</td>\n", "      <td>...</td>\n", "      <td>0.1860</td>\n", "      <td>0.2750</td>\n", "      <td>0.08902</td>\n", "    </tr>\n", "    <tr>\n", "      <th>...</th>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "    </tr>\n", "    <tr>\n", "      <th>567</th>\n", "      <td>927241</td>\n", "      <td>M</td>\n", "      <td>20.60</td>\n", "      <td>...</td>\n", "      <td>0.2650</td>\n", "      <td>0.4087</td>\n", "      <td>0.12400</td>\n", "    </tr>\n", "    <tr>\n", "      <th>568</th>\n", "      <td>92751</td>\n", "      <td>B</td>\n", "      <td>7.76</td>\n", "      <td>...</td>\n", "      <td>0.0000</td>\n", "      <td>0.2871</td>\n", "      <td>0.07039</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "<p>569 rows \u00d7 32 columns</p>\n", "</div>"], "text/plain": ["         id diagnosis  mean_radius  ...  worst_concave_points  worst_symmetry  \\\n", "0    842302         M        17.99  ...                0.2654          0.4601   \n", "1    842517         M        20.57  ...                0.1860          0.2750   \n", "..      ...       ...          ...  ...                   ...             ...   \n", "567  927241         M        20.60  ...                0.2650          0.4087   \n", "568   92751         B         7.76  ...                0.0000          0.2871   \n", "\n", "     worst_fractal_dimension  \n", "0                    0.11890  \n", "1                    0.08902  \n", "..                       ...  \n", "567                  0.12400  \n", "568                  0.07039  \n", "\n", "[569 rows x 32 columns]"]}, "metadata": {}, "output_type": "display_data"}], "source": ["import pandas as pd\n", "\n", "df = pd.read_csv(\n", "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n", "    na_values=['NA','?'])\n", "\n", "pd.set_option('display.max_columns', 6)\n", "pd.set_option('display.max_rows', 5)\n", "\n", "display(df)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).\n", "\n", "Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:\n", "\n", "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.\n", "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n", "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n", "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n", "\n", "Types of errors can be seen in Figure 4.ETYP. \n", "\n", "**Figure 4.ETYP: Type of Error**\n", "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n", "\n", "Neural networks classify in terms of probability of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensitive or specific:\n", "\n", "More info on Sensitivity vs Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"]}, {"cell_type": "code", "execution_count": 3, "metadata": {"scrolled": true}, "outputs": [{"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3wU1frH8c9Jp4ReJRSRmgSIoSlFkN5EekKRojT1qnjBXn56vXavIiJFQBQJJpGm0kFEpROQFkB6CYj0DiFlfn+cJSAmkITdPVue9+u1ryyb3ZlvJsuTs2dmnlGWZSGEEML5fEwHEEIIbyUFWAghDJECLIQQhkgBFkIIQ6QACyGEIVKAhRDCECnATqCUekUplaiU2qyU2qiUqm/HZa+0fa2glOp1w+N1lFKj7LUee1FKpdm2wSal1AalVINsvOaCM7Jlst43lFIjTKz7Vhz8fpqnlCpku/+0Umq7UipGKdVRKfWivdYjND/TATydUup+oAMQaVlWslKqGBBgr+VblnWtgFUAegHTbI8nAAn2Wo8dXbYsKwJAKdUaeBdoYjaS+3DC+6ndDf98AmhrWdY+279/sNd6hCYjYMcrDZywLCsZwLKsE5ZlHVFK1VZK/aKUWq+UWqiUKg2glFqmlHpfKbVWKbVTKdXY9niY7bGNtpFPZdvj10aH7wGNbd9/VinVVCk1Rynlo5Taf21UY3vNbqVUSaVUcaXUDKXUOtutoVO3DBQATtsy5VdK/WQbFW9RSj1885Ozeo5t9L9dKTXBNjJcpJTKY/teJaXUkhtG3PfYHn/O9jNvVkq9ecM6XlFK/aGUWgJUdcZGyKGs3k/7b3jfrFVKVQLI6nds25aTbdtxs1Kqq+3x/UqpYkqpcUBF4Afb+6m/Umq07TkllVKzbNt0U3Y+xYgsWJYlNwfegPzARmAnMAY92vMHVgLFbc+JAr603V8G/M92vx2wxHb/M6C37X4AkMd2/4Lta1Ngzg3rzfg38CkwwHa//g3LnAY0st0vB2x3wvZIs22PHcBZoLbtcT+ggO1+MWA3oG76GTN9Dnr0nwpE2L4XD/Sx3V8DdLbdDwLyAq2AL2yv9QHmAA8AtYEttucUsC1/hOn30O3eT7bH9wOv2O73veF3n+nvGHgfGHnDcgvfsJximdzvD4y23Y8Dhtnu+wIFTW8Xd73JFISDWZZ1QSlVG2gMPIh+8/4XCAcWK6VAv4n/vOFlM21f16OLC8Aq4BWlVAgw07KsXTmIEQe8DkwGom3/BmgBhNoyABRQSgVblnU+B8vOqRunIO4HpiilwtHF8B2l1ANAOlAGKAkcveG1WT0HYJ9lWRtt99cDFZRSwUAZy7JmAViWdcW23lboIvy77fn5gcpAMDDLsqxLtue53EfuzN5PN8zNfnvD109s9zP9Hdsej75huadzEKMZushjWVYa+g+pyAUpwE5ge5MuA5YppbYATwKJlmXdn8VLkm1f07D9jizLmqaUWgO0BxYqpQZalrU0mxFWAZWUUsWBTug/AKBHf/dblnU5pz+TPViWtco2h1kcPdovjh4Rpyil9qNHrDfqfYvnJN/wvDQgD7pgZ0YB71qWNf5vDyo1DHD55iiZvJ/6XfvWjU+zfc30d6x0RXb5n9XTyRywgymlql6br7WJALYDxW0jQJRS/kqpsNsspyKw17KsUeidITVvesp59AjuHyz9WXEW8DH6I+hJ27cWAf+6YR0R2f7B7EApVQ09+j8JFASO2Qrrg0D5TF6SnedksCzrHJCklOpkW1+gUiovsBB4VCmV3/Z4GaVUCeBXoLNSKo9tlPiQfX5S+8ni/XTAdj/qhq+rbPez+h3f/HjhHMT4CXjc9jpfpVSBHLxW3EAKsOPlB75WSm1TSm0GQtHTAd2A95VSm9BzerfbkREFbFVKbQSqAVNu+v5mINW2U+TZTF4fB/Th+vQDwNNAHdtOmG3A0Bz+bLmRx7ajcKMtSz/biC7GliUBPdLdkclrs/Ocmz0CPG3b9iuBUpZlLULPja6yjSCnA8GWZW2wZdoIzAB+u5Mf1EEyez+9YfteoO1T0jPAtfdAVr/j/wKFlVJbbe/BB3OQ4RngQdu2Ww/ccvAgsnZtJ4cQwo3ZpmPqWJZ1wnQWkX0yAhZCCENkBCyEEIbICFgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwpAcnYpcrFgxq0KFCg6KInLijz/+AKBqVVds2OVcsi2uk23hmtavX3/CsqziNz+eowJcoUIFEhJcscWs92natCkAy5YtM5rDFci2uE62hWtSSh3I7HGZghBCCEOkAAshhCFSgIUQwhDpByyEyJCSkkJSUhJXrlwxHcUtBQUFERISgr+/f7aeLwVYCJEhKSmJ4OBgKlSowA1X0RDZYFkWJ0+eJCkpibvvvjtbr5EpCCFEhitXrlC0aFEpvrmglKJo0aI5+vQgBVgI8TdSfHMvp9tOCrAQwqUopRg+fHjGvz/66CPeeOMNu6/nnXfe+du/GzS43UVp7E8KsBDCpQQGBjJz5kxOnHDsxT1uLsArV6506PoyIwVYCOFS/Pz8GDx4MJ988sk/vnf8+HG6du1K3bp1qVu3LitWrMh4vGXLlkRGRjJkyBDKly+fUcA7depE7dq1CQsL44svvgDgxRdf5PLly0RERNC7d28A8ufPD0BUVBTz5s3LWGf//v2ZMWMGaWlpPPfcc9StW5eaNWsyfvzfLqqdK1KAhRAu58knnyQmJoazZ8/+7fFnnnmGZ599lnXr1jFjxgwGDhwIwJtvvkmzZs3YsGEDnTt35uDBgxmv+fLLL1m/fj0JCQmMGjWKkydP8t5775EnTx42btxITEzM39YRHR1NXJy+du3Vq1f56aefaNeuHZMmTaJgwYKsW7eOdevWMWHCBPbt23dHP6cchiaEyNSwBcPYeHSjXZcZUSqCkW1G3vZ5BQoUoG/fvowaNYo8efJkPL5kyRK2bduW8e9z585x/vx5li9fzqxZswBo06YNhQsXznjOqFGjMr536NAhdu3aRdGiRbNcd9u2bXn66adJTk5mwYIFPPDAA+TJk4dFixaxefNmpk+fDsDZs2fZtWtXtg85y4wUYCGESxo2bBiRkZEMGDAg47H09HRWrVr1t6IM+hjczCxbtowlS5awatUq8ubNS9OmTW97mFhQUBBNmzZl4cKFxMXF0bNnz4x1fPbZZ7Ru3foOf7LrpAALITKVnZGqIxUpUoQePXowadIkHn30UQBatWrF6NGjee655wDYuHEjERERNGrUiPj4eF544QUWLVrE6dOnAT1KLVy4MHnz5mXHjh2sXr06Y/n+/v6kpKRketZadHQ0EydOJCEhga+++gqA1q1bM3bsWJo1a4a/vz87d+6kTJky5MuXL9c/o8wBCyFc1vDhw/92NMSoUaNISEigZs2ahIaGMm7cOAD+7//+j0WLFhEZGcn8+fMpXbo0wcHBtGnThtTUVGrWrMlrr73Gfffdl7GswYMHU7NmzYydcDdq1aoVv/76Ky1atCAgIACAgQMHEhoaSmRkJOHh4QwZMoTU1NQ7+vlydFn6OnXqWNIP2DVI39frZFtcd6fbYvv27VSvXt1+gZwkOTkZX19f/Pz8WLVqFY8//jgbN9p3/jq7MtuGSqn1lmXVufm5MgUhhHB7Bw8epEePHqSnpxMQEMCECRNMR8oWKcBCCLdXuXJlfv/9d9MxckzmgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAjhUnx9fYmIiCA8PJzu3btz6dKlHC9j4MCBGacsu0LbyaxIARZCuJRrTXK2bt1KQEBAxskWOTFx4kRCQ0MB12g7mRUpwEIIl9W4cWN2794NwMcff0x4eDjh4eGMHKlPk7548SLt27enVq1ahIeHZ3Qxa9q0KQkJCS7TdjIrchywEMIlpaamMn/+fNq0acP69euZPHkya9aswbIs6tevT5MmTdi7dy933XUXc+fOBfhH+8r33nuP0aNHZ3pW3LW2k+3atctoOzl27Ni/tZ1MTk6mYcOGtGrV6o66nmVFCrAQInPDhoG9T+eNiICRt27yc23ECnoE/NhjjzF27Fg6d+6c0fimS5cu/Pbbb7Rp04YRI0bwwgsv0KFDBxo3bpztKM5sO5kVKcBCCJdybQ74Rln1rKlSpQrr169n3rx5vPTSS7Rq1YrXX389W+txZtvJrEgBFkJk7jYjVWd64IEH6N+/Py+++CKWZTFr1iy++eYbjhw5QpEiRejTpw/58+fPaB15I1doO5kVKcBCCJcXGRlJ//79qVevHqAPM7v33ntZuHAhzz33HD4+Pvj7+zN27Nh/vPZa28nIyMh/XH6oVatW9O3bl44dO/6t7eT+/fuJjIzEsiyKFy/O7NmzHfJzSTtKNyUtGK+TbXGdt7ajdCU5aUcph6EJIYQhUoCFEMIQKcBCCGGIFGAhxN/kZL+Q+LucbjspwEKIDEFBQZw8eVKKcC5YlsXJkycJCgrK9mvkMDQhRIaQkBCSkpI4fvy46ShuKSgoiJCQkGw/XwqwECKDv7+/Q065FZmTKQghhDBECrAQQhgiBVgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAghDJECLIQQhkgBFkIIQ6QAu5nTl0+zYPcC9p/Zz55Te4jdGsuBMwekf6u3O3EC5syBfftg716YMQOOHDGdStyGtKN0E0nnknhy3pP88McP+oGzoFD0nNETgOrFqjO+w3gal29sMKVwut27YcgQWLr0+mNKQbdu+n7t2vDFFxAZaSafuCUZAbs4y7KYtGESYWPCWLxnMa80foWf+v5Eo7KNaFyuMQmDEvis7WdcSb1Ck6+a8PT8p7l49aLp2MLR0tLgk0+gZk1Yvx7eegt++QUaN4ZGjWD1avjf//QouF49ePVVSE42nVrcREbALiwtPY3eM3sTlxhHk/JNmNRxEvcUuQeA//j8B4Dad9Wm9l216R/Rn5d/epnP1n7Gwj0LWdZvGaWDS5uMLxzlyhV4+GFYtAg6dIBx46BMGf09H9uYqn59fRswAJ59Ft5+G+bNg59+gsKFzWUXfyMjYBdlWRZD5gwhLjGOt5u9zdJ+SzOKb2byB+RnVNtRLO27lMPnDtN6amtOXz7txMTCKVJToVcvXXzHjoUffrhefDNTuDB89RXMng2JibpgX7rktLji1qQAu6gXl7zIpN8n8WrjV3m58cv4qOz9qh68+0G+j/6eP07+QYdvO8h0hCexLBg6FGbNgk8/1feVyt5rH34Ypk3TUxPdusHVq47NKrJFCrAL+mTVJ3yw8gMer/M4/3nwPzl+ffOKzfm267esTlpN9++6k26lOyClcLpXX4VJk+C11+Dpp3P++q5d9XTF/Pnw6KO6oAujpAC7mN///J3nlzxP52qd+aztZ6jsjnBu0qV6Fz5v9znzd8/n09Wf2jmlcLqff4Z33oGBA+HNN3O/nEGD4D//gZgYmDrVfvlErkgBdiHJqcn0nd2X4nmLM7HjRHx9fO9oeUNqD6Fj1Y689NNL7Dixw04phdOdO6d3plWurKcecvlHOcPLL0PDhvDUU5CUZJ+MIlekALuQN395k63HtjLhoQkUyVPkjpenlGJ8h/HkC8hHv9n9SE1PtUNK4XQjRsChQ3pnWt68d748X1+9rJQUPaKWqQhjpAC7iDVJa3h/xfs8GvEo7au0t9tyS+UvxZh2Y1h7eC0frPjAbssVTrJgAUyYoItwgwb2W26lSvDBB7BwIUycaL/lihyRAuwC0tLTGPTjIMoEl+Hj1h/bfflR4VH0COvBG8veYM+pPXZfvnCQy5dh8GAIDb2zed+sPP44NG8Ow4fDX3/Zf/nitqQAu4Apm6aw5dgWPm79MQWDCjpkHSNbj8Tf159Xlr7ikOULB/jsMz318PnnEBRk/+X7+OhlX7qkz6QTTicF2LBLKZd47efXqF+mPl2rd3XYekoHl2b4/cOJS4xj3eF1DluPsJOTJ/VRD+3bQ9OmjltP1ap6lD1+POza5bj1iExJATZs1JpRHD5/mA9afpDrQ86y67kGz1E8b3GeX/K8dE9zde+8A+fPw3vvOX5d//d/EBioj44QTiUF2KATl07w7vJ3eajKQzxQ/gGHry84MJjXm7zOsv3LmL97vsPXJ3Jp/34YPRr69YPwcMevr2RJvZNv+nRYs8bx6xMZpAAb9Pavb3Ph6gXea+GEUY7N4NqDqVSkEi8seYG09DSnrVfkwGuv6fnZ/+T8LMhcGz5cF+Lnn5fD0pxICrAhf134i7EJY+lbqy+hxUOdtt4A3wD+++B/2XpsK7N2zHLaekU27dqlz1J76ikICXHeeoODdeH/9Vd9E04hBdiQz9Z+xtW0q7zY8EWnr7tbaDcqFanEBys+kLlgV/O//0FAAPz7385f96OPQokS8P77zl+3l5ICbMD55PN8vu5zOlXrRNViVZ2+fl8fX0bcP4J1R9axbP8yp69fZOHoUX2GWr9+UKqU89efJ49u8jN/Pmze7Pz1eyEpwAZM2DCBM1fO8ELDF4xl6BfRj5L5SvL+ChntuIxRo3SbyBEjzGV44gnIl0+fJSccTgqwk11Nu8onqz+hSfkm1A+pbyxHkF8Qz9R/hoV7FrLx6EZjOYTNuXMwZoxuGVm5srkchQvr44JjY+HAAXM5vIQUYCf7dsu3JJ1LMjr6vebxuo8THBAsPSJcwRdfwNmz8IL59wXPPqs7rn1s/9Pixd9JAXYiy7L4aNVH1ChRgzaV2piOQ6GgQgyprS97dOCMjHaMSUmBkSOhWTOoU8d0GihbFnr31k16Tp0yncajSQF2ol8P/MrWY1sZdt8wh5/1ll1P1X8KgPHrxxtO4sW+/x4OH4Zhw0wnue7ZZ3WPiK+/Np3Eo0kBdqKxCWMpFFSI6PBo01EylCtYjg5VOjDp90kkp8ply40YOxbKl4d27Uwnua5WLbj/fp0tXS5p5ShSgJ3k6IWjzNg+gwERA8jrb4em2nb0RJ0nOHbxGDO3zzQdxfvs2AFLl8KQIbpRuit54gl9YsjSpaaTeCwpwE4yccNEUtNTGVpnqOko/9DynpbcU/gexiSMMR3F+4wbB/7+8NhjppP8U7duUKyYPjpDOIQUYCdITU/li/Vf0KJiC6oUrWI6zj/4KB+G1hnK8oPL2fLXFtNxvMfFi/rEi27d9BloriYoSJ8d98MPcu04B5EC7ARzd87l0LlDPFHnCdNRsjQgYgCBvoGMTRhrOor3iI3Vh549/rjpJFkbMkTPAU+YYDqJR5IC7ARjEsZQJrgMD1V9yHSULBXNW5So8Ci+2fwN55LPmY7jHcaM0e0mGzUynSRrFStC27b6OOWUFNNpPI4UYAc7cOYAi/csZmDkQPx8/EzHuaWhtYdy4eoF4hPjTUfxfL//Dhs26BGmixySmKWhQ3WfinnzTCfxOFKAHezrTV9jYdE/or/pKLd1X8h9VC1alckbJ5uO4vkmT9Zdz3r1Mp3k9tq21b2CJ8v7wt6kADtQupXOVxu/otndzahQqILpOLellGJAxABWHlrJHyf+MB3HcyUn656/nTpBkSKm09yenx888gjMnQvHjplO41GkADvQrwd+Zd+ZfQyIGGA6SrY9UusRfJQPX2+SM6Ac5scf9Sm+A9znfcGAAZCaClOnmk7iUaQAO9DkjZMpEFiALtW7mI6SbXcF30WbSm2YsmmKXLLIUSZPhjJloGVL00myLzQU6tXT2aWJv91IAXaQ88nnmb5tOlFhUS535tvtDIgYwOHzh1m8d7HpKJ7nyBFYsAD69nW9M99uZ8AA2LoV1q83ncRjSAF2kPjEeC6lXHKr6YdrHqryEEXyFJGdcY7wzTf6uNr+/U0nybnoaH1yhuyMsxspwA4yeeNkqhatyn0h95mOkmOBfoH0qdGH2Ttmc+qytCO0G8vSZ741agRVXO+MyNsqVAi6dIFp0+DKFdNpPIIUYAfYe3ovKw6toF+tfi7TdjKn+kf052raVb5L/M50FM+RkKCb7/TrZzpJ7vXvD2fO6CMixB2TAuwAMZtjAOhds7fhJLkXUSqC0OKhTN0ie73tZupUCAzUvR/cVbNm+oKhcjSEXUgBtjPLsojZEsMD5R+gXMFypuPkmlKK3jV6s/zgcvaf2W86jvtLTdW9Hzp00B/l3ZWvL/Tsqc+Kk6tl3DEpwHa2/s/1/HHyD/rU6GM6yh3rVUOfpTVtyzTDSTzAkiX6JIY+7v++oE8fffXm6dNNJ3F7UoDtLGZzDAG+AXQLdeOPmTYVClWgUblGTN08FUuO/bwzMTF65Nu2rekkd+7ee6FaNf0ziTsiBdiOUtNTiU2MpX3l9hTOU9h0HLvoXaM3209sl0vX34mLF2HWLOjeXc8Buzul9EU7f/0VDh40ncatSQG2o6X7lnL0wlH61PSAj5k23UO74+/jT8wWGe3k2vff6yLsCdMP11xrIjRNpqfuhBRgO4rZEkPBwIK0q+xCF1e8Q0XzFqVt5bZM2zJNTk3OrZgYfal3V+77m1MVK0KDBvpoCJmeyjUpwHZyKeUSM7fPpFtoN4L8gkzHsas+Nfrw54U/WbZ/meko7uf4cVi4UI8YfTzsv1ufPpCYCJs3m07itjzsHWHOvF3zuHD1QsaRA56kQ5UO5A/IT+zWWNNR3M+MGZCW5h59f3OqWzd9WFpcnOkkbksKsJ3Ebo2lZL6SNCnfxHQUu8vjn4dO1ToxY/sMrqZdNR3HvcTG6k5iNWqYTmJ/xYtDixb6Z5RpiFyRAmwH55LPMWfnHHqE9cDXx806XGVTdFg0p6+cZvEe6ZCWbYcP6yMFoqNd/7JDuRUdDfv2wdq1ppO4JSnAdvD9ju9JTksmOjzadBSHaXlPSwoHFSY2UaYhsi0+Xo8Mo6JMJ3GcTp30pZVi5X2RG1KA7SA2MZbyBctzf8j9pqM4TIBvAF2rd2X2jtlcTrlsOo57iI2FyEj37HyWXYUKQbt2eh44TY6SySkpwHfo5KWTLNqziKiwKLftfJZd0eHRXLh6gXm75Oq4t7V3r/5YHu25n4oyREfDn3/C8uWmk7gdKcB3aOb2maSmp3r09MM1TSs0pWS+kjINkR3Xjgzo0cNsDmfo0AHy5pVpiFyQAnyHYhNjqVK0ChGlIkxHcThfH1+6h3Znzs45nE8+bzqOa4uN1ScqlC9vOonj5csHHTvq5jwpKabTuBUpwHfg6IWjLNu/zCumH67pWaMnV1Kv8P0f35uO4rq2b9cnJ3jD9MM10dFw4gT89JPpJG5FCvAdmL5tOulWOlFhHryX+yb3hdxHSIEQ4hPjTUdxXXFx+rAzd268nlOtW0OBAvrID5FtUoDvQHxiPGHFwwgrEWY6itP4KB96hPZgwe4FnLlyxnQc12NZugg1aQKlS5tO4zxBQfDww7rr21U5WSe7pADn0uFzh1l+cLlXjX6viQqPIiU9he93yDTEP2zdqqcgPPnY36xERenrxS2Wk3WySwpwLk3fNh0Lix5hXrCX+yZ176pLhUIViEuUHgD/EB+vm+506WI6ifO1bKmPC5ZpiGyTApxLcYlx1CpZi6rFqpqO4nRKKXqE9mDx3sWcvHTSdBzXYVl6/rdZMyhRwnQa5wsIgM6dYfZsuWx9NkkBzoWDZw+yKmmVV45+r+kR1oPU9FRm75htOorr2LQJdu3yjmN/s9KjB5w7B4sWmU7iFqQA58J3id8BeHUBjiwdyT2F75FpiBvFxen2jN44/XBN8+ZQpIi0qMwmKcC5EL8tnsjSkVQqUsl0FGOUUvQI68HSfUs5fvG46TjmXTv6oUULKFrUdBpz/P31H6AffoDL0jPkdqQA59C+0/tYe3gtPUK9d/R7TVRYFGlWGjO3zzQdxbz163X/B2+efrgmKgouXID5800ncXlSgHPo2gkI3jz9cE3NkjWpUrSKTEOA/sjt7693Qnm7pk11s3aZhrgtKcA5FL8tnnpl6nF34btNRzFOKUVUWBS/HPiFoxeOmo5jzrXph1atoHBh02nM8/ODrl1hzhx9NWiRJSnAObD71G42/LnBK0++yEpUWBTpVjozts0wHcWcNWvg4EHvPPkiK1FRcOkSzJ1rOolLkwKcA9emH7qHdjecxHWElQgjtHgo8du8+OD7+Hh9DGzHjqaTuI7GjaFUKTkp4zakAOdAXGIcDco2oGzBsqajuJSosCh+O/AbR84fMR3F+dLTdZFp2xYKFjSdxnX4+upmRHPnwnlpXZoVKcDZtOPEDjb/tVmOfshEj7AeWFgZx0d7lZUr9cU35eiHf4qK0mfE/fij6SQuSwpwNsUnxqNQdAv1ohaD2VStWDVqlqzpndMQ8fG6E9hDD5lO4noaNIAyZWQa4hakAGdTXGIcjco1okyBMqajuKQeoT1YeWglh84eMh3FedLS4Lvv9EUpg4NNp3E9Pj7Qvbs+HvjsWdNpXJIU4GzYemwr245vk6MfbiEqXG8br2rU/ttvcPSoHP1wK1FRuj/w99K6NDNSgLMhbmscPspHph9uoVKRStQuXdu7TsqIi9PXQ2vf3nQS11W/vr4unpyUkSkpwLdhWRaxibE0u7sZJfOXNB3HpUWHR7PuyDr2nNpjOorjpaToi1B27KiLsMicUnoUvGgRnJTWpTeTAnwbG/7cwO5Tu4kO86ILLObStdOzvWIUvHSpvgilN114M7eioyE1FWZKz5CbSQG+jditsfj7+NO5upzjfzvlCpajYdmGxG6NNR3F8WJj9XG/rVubTuL6IiKgShW9zcTfSAG+hXQrnbjEOFpXak2RPEVMx3EL0eHRbDm2hcRjiaajOE5ysh7NdekCgYGm07g+pfQo+Oef4c8/TadxKVKAb2HVoVUcOndIph9yoFtoN3yUj2dPQyxYoK/6INMP2RcVpZsWTZ9uOolLkQJ8C7FbYwnyC6JjVTnHP7tK5S/FgxUeJC4xDsuyTMdxjNhYKFZMX/tNZE9oKNSsKdMQN5ECnIXU9FTit8XTvnJ7ggPlIPuciAqLYufJnWw8utF0FPu7eFFf7aFbN912UWRfdLQ+dfvAAdNJXIYU4Cws27+MYxePER0uHzNzqkv1Lvj5+DFtyzTTUezvxx91m0WZfsd5G+4AABjNSURBVMi5ayesyCg4gxTgLMRsiSE4IJj2leUg+5wqmrcobSq1ITYxlnQr3XQc+4qJgZAQ3W5R5EzFivrEjGke+Ic5l6QAZ+JyymVmbJtB19Cu5PHPYzqOW+pdozdJ55L49cCvpqPYz4kTegdcz566z4HIud69YfNm2LrVdBKXIO+iTMzdNZfzV8/Tu0Zv01HcVseqHcnnn4+YzTGmo9jPd9/pEwp6y/si13r00L2CZRQMSAHOVMyWmIy9+SJ38vrnpXP1zkzfPp3k1GTTcexj2rTre/NF7pQsCS1a6G2Z7mHTU7kgBfgmpy+fZt6ueUSHRePr42s6jlvrXaM3Z66cYf5uD7g8+YEDsHy5Hv0qZTqNe+vdW2/PVatMJzFOCvBNZmyfwdW0q/SuKR8z71SLii0oka8EMVs8YBri2kfmnj3N5vAEnTpBnjx6h6aXkwJ8k5gtMVQpWoXapWubjuL2/Hz8iAqL4sc/fuTsFTduyG1Zulg0aAB33206jfsLDoaHH9ZXykhJMZ3GKCnAN0g6l8Qv+3+hV3gvlHzMtIteNXqRnJbMzO1u3AlryxZITIRevUwn8Ry9eun2lAsXmk5ilBTgG8RsjsHCkukHO6pfpj6VilTim83fmI6Se1Om6LPe5MKb9tO6tT6d+xs3fl/YgRRgG8uy+HrT1zQs25BKRSqZjuMxlFL0rdmXn/f/zIEzbngKamoqTJ2qr3pRvLjpNJ4jIEDPp3//PZw+bTqNMVKAbRKOJLD9xHb61uprOorHeaTWIwDuOQpetAj++gv69TOdxPP07atbe3rxVZOlANtM2TSFQN/AjKs6CPupUKgCTco3YcqmKe7XIW3KFChSRF/5WNhX7dr6uOopU0wnMUYKMHA17Srfbv2WTtU6USiokOk4HqlfrX7sOrWL1UmrTUfJvjNnYPZs/VFZGq/bn1L6k8XKlbBrl+k0RkgBBubunMvJyyfpV0s+ZjpKt9Bu5PHLw9ebvjYdJfvi4/VHZJl+cJw+fXRfDS8dBUsBBqZsnkKp/KVoeU9L01E8VnBgMF2qdyEuMY4rqVdMx8meKVOgenWoU8d0Es9111361ORvvvHKU5O9vgCfuHSCuTvn0rtGb/x8pMG2I/Wr1Y8zV87w4x8/mo5ye7t3w4oVevQrx4Q7Vr9++tTkXz2oc142eX0Bnrp5KinpKTL94ATN7m5GSIEQvtz4pekotzd5sv5o3KeP6SSer1MnKFAAvnSD94WdeXUBtiyLiRsmUq9MPWqUrGE6jsfz9fFlQMQAFu5eyMGzB03HyVpqqi7AbdtCmTKm03i+vHn1mXHffad3fHoRry7Aq5NWk3g8kUGRg0xH8RqP3vsoAJN/n2w4yS3Mn68vnz5I3hdOM2gQXLnidQ16vLoAT9wwkXz++YgKizIdxWtUKFSBlve05MuNX5KWnmY6TuYmToRSpeTYX2eKjIR774UJE3TzIy/htQX4XPI5YhNj6RneU6567GSDIgdx8OxBFu9dbDrKPx05AnPnQv/+4O9vOo13GTQINm2CDRtMJ3Eary3AsVtjuZRyiYGRA01H8Todq3akWN5iTNww0XSUf/rqK0hLg8ceM53E+/TsqfsET5hgOonTeG0BnrhhIuElwqlXpp7pKF4nwDeAfrX68f0f33Ps4jHTca5LT4dJk+DBB6GSNGRyukKFoHt33fz+4kXTaZzCKwvwpqObWHdkHYMiB0nfX0MGRg4kNT2Vrze60JlxP/8Me/fCQPlUZMygQXD+vNc06PHKAjxm3RiC/ILoU1OO8TSlWrFqPFD+AcatH0e65SJnQI0ZA0WLQufOppN4r4YNdYOeMWNMJ3EKryvAZ66cYeqWqfQK70WRPEVMx/FqT9Z9kr2n97Jg9wLTUeDQId2b9rHH9DykMEMpeOIJSEiAtWtNp3E4ryvAX238ikspl3iy3pOmo3i9ztU6Uzp/aUavHW06Cowfr+eAH3/cdBLRt6++btxoF3hfOJhXFeB0K50x68Zwf8j9RJaONB3H6/n7+jOk9hDm757P7lO7zQVJTtZ73h96CCpUMJdDaMHBugjHxcHx46bTOJRXFeDFexaz69Qu/lXvX6ajCJvBtQfj5+PH2HVjzYWYPh2OHYMn5VORy3jySbh6VZ8U48G8qgB/vu5zSuQrQdfqXU1HETalg0vTtXpXvtz4JRevGjr06PPPoUoV3RZRuIbq1aFZMxg3Tvfm8FBeU4D3nd7HnJ1zGBw5mEA/ubqBK/lXvX9x5soZpm2Z5vyVb9gAq1bpEZeP1/x3cA//+hccPAhz5phO4jBe844btWYUvj6+DKkzxHQUcZOGZRsSUSqCkWtGOv+QtE8+gfz55aoXruihh6BcOf078lBeUYBPXz7NhA0TiA6PJqRAiOk44iZKKYbfP5xtx7c595C0Q4cgNlYf/F+woPPWK7LHzw+GDdON2j30kDSvKMDj14/nYspFRtw/wnQUkYWosChCCoTw4coPnbfSTz/VnbeeecZ56xQ5M3Cg/uP40UemkziExxfg5NRkPl3zKS0rtqRWqVqm44gs+Pv6M6z+MJbtX0bCkQTHr/DsWfjiC4iKgvLlHb8+kTvBwTB0KMyYoU8T9zAeX4CnbZnG0QtHea7Bc6ajiNsYVHsQBQIL8NFKJ4x2xo/XPQdGyKcil/f00+DrCx9/bDqJ3Xl0AU630vlo1UfUKlmLFhXlECNXVyCwAENrD+W7bd+x7/Q+x63o6lU9/dC8uW4CLlzbXXfpa/N9+SWcOGE6jV15dAGet2se245vY0SDEdL1zE08Xf9pfJUvH69y4GgnJkY3XpfRr/sYPhwuX9bHbHsQjy3AlmXxn1/+Q4VCFeSSQ26kTIEy9K3VlwkbJnDk/BH7ryA1Fd5+GyIioHVr+y9fOEZYGHTsCCNH6vl7D+GxBXj+7vmsO7KOVxu/ir+vXFrGnbzc+GXSrDTeX/6+/RceEwN79sAbb+jOW8J9/N//6asmjxplOondeGQBtiyLN5a9QYVCFehbq6/pOCKHKhauSL9a/Ri/frx9R8GpqfDWW3r027Gj/ZYrnCMyUv/ePv7YY0bBHlmAZfTr/hwyCpbRr/vzsFGwxxVgGf16BruPgmX06xk8bBTscQV47q65Mvr1ENdGwe/+9u6dL+ybb2T06ymujYI//dR0kjvmUQU4NT2VF5a8QKUilWT06wEqFq7IoxGPMm79OHad3JX7BV26BK+/DnXqyOjXE0RG6uv2ffgh/PWX6TR3xKMK8OTfJ7Pt+Dbeb/G+jH49xBtN3yDQN5CXfnop9wsZORKSknQ/ARn9eob33oMrV/QnGjfmMQX4wtULvPbzazQs25DO1eSqtp6idHBpnm/4PDO2z2DFwRU5X8Bff8G778LDD0OTJvYPKMyoUkVfv2/CBNi2zXSaXPOYAvzhig/56+Jf/K/V/+SsNw8z/P7h3BV8F8MXDceyrJy9+M039Ujpgw8cE06Y8/rrkC8fvPCC6SS55hEF+PC5w3y48kOiwqKoH1LfdBxhZ/kC8vHfB//LmsNr+G7bd9l/4fbtuuPZ0KF6xCQ8S7Fi8Mor+ooZS5eaTpMrHlGAn1/yPGlWGu80f8d0FOEgfWv1pVbJWoxYNIILVy/c/gXX+vzmy6dHSsIzPf20bif6zDOQkmI6TY65fQFetGcR07ZM46VGL1GxcEXTcYSD+Pr48nm7zzl07hBvLHvj9i/49ltYvFj3fShe3OH5hCFBQfqkjK1b3bJdpVsX4Mspl3li7hNULlKZFxu9aDqOcLCG5RoyKHIQI1ePZNPRTVk/8fRpePZZqFtX76gRnq1jR+jUSc/373NgG1MHcOsC/PZvb7Pn9B7GdRhHkF+Q6TjCCd5r8R5F8hRhyJwhpKWnZf6kl17SfWPHj9eNvIXnGzVK/66feEJPP7kJty3A245v44MVH9C3Vl+a3d3MdBzhJEXyFOGT1p+w5vAaxq8f/88nrFypC++wYdJs3ZuULQv//S8sWADf5WBHrWFuWYBT0lLoN7sfwYHBfNTSMy/WJ7LWq0YvWlRswQtLXmDPqT3Xv3HxIgwYoP8zvvmmuYDCjH/9C2rXhiefhKNHTafJFrcswG8se4OEIwl80eELiueTHSzeRinFpI6T8FW+9JnVBwvbR85//xt27YKvv4b8+c2GFM7n6wtTpsCFC/oPsRtMRbhdAf7twG+8u/xdBkQMoGtoV9NxhCHlCpZjXIdxrE5azYEzB/Sc7xdf6MsMPfig6XjClNBQ3SNiwQIYM8Z0mttyqwJ89spZHpn1CBULV+TTNu7fCUncmejwaPrU7MOfJw9g/bFDt5p86y3TsYRpTz4JbdvqP8Yufpqy2xRgy7IY+ONAks4lMbXLVIIDg01HEi5gdMuRhJ5UWGlpnJ70OQQGmo4kTFNKX0E5f36IitJTEi7KbQrwu8vfZfq26bzX4j3uC7nPdBzhIgq+9l8KXrbYXRS6bnmVlDT3OxtKOECpUjBtmh4B9+/vsvPBblGA5+ycw6tLX6V3jd4Mv3+46TjCVUyZoltNlilDgfJV+Xn/z4xYJJeaFzYtW+omTDNm6DMiXZCf6QC3s+PEDnrN6MW9pe9lwkMTpNOZ0Natg8GD9Q63tDRKKUWb+m34dM2nRJSKYMC9A0wnFK7g3/+GjRvhtdegZk2Xa8jv0iPgpHNJtJnahiC/IGZFzSKPfx7TkYQr2LkT2reH0qUhPj6jyfpHrT6i+d3NGTJnCPN3zTccUrgEpfTRMbVrQ8+esGqV6UR/47IF+PjF47T8piWnLp9ifu/5lCtYznQk4QoOHoQWLfT9hQt1S0IbPx8/ZvSYQXiJcLrGd+W3A78ZCilcSp48umXlXXdBu3aw6RZ9RJzMJQvw2StnaRPThv1n9jOn1xxq31XbdCThCv76S8/rnTsHixZl2uO3YFBBFvRZQLmC5ejwbQc2/LnBQFDhckqVgiVL9JERrVrpE3ZcgMsV4JOXTtJ6ams2/7WZGT1m8ED5B0xHEq4gKQmaNdNf587Vx/xmoUS+Eix+ZDGFggrR6ptWrD281olBhcsqX163KLUsve/ABY4RdqkCfOjsIRpPbszGoxuZ3n067Sq3Mx1JuIIdO6BhQzh0SBffhg1v+5KyBcuytO9SCgYVpNnXzVi0Z5ETggqXV60a/PQTpKVB48bG54RdpgBvO76NBl824Mj5Iyx6ZBEPV3vYdCThCtauhUaN9HXdfvkFmjbN9kvvKXIPKx5dQeWilWk/rT3TtkxzXE7hPmrU0F3zihSB5s1h3jxjUVyiAE/fNp37Jt5Hanoqv/T/RaYdhDZpEjzwABQoACtW5Kq9ZKn8pVjWbxkNyzak98zePL/4eVLTUx0QVriVu+/W76nq1eGhh+CddyA93ekxjBbglLQU/r3w33T/rjuhxUNZO3AttUrVMhlJuILLl+Gxx2DgQP0xcc0aqFQp14srGFSQhX0W8nidx/lw5Yc0n9Kcoxfco12hcKASJfSnqqgofXHPhx/WV1NxImMFOPFYIo0mN+KT1Z/wVL2n+HXAr5QtWNZUHOEq1q2DevX0ufyvvqq7Wtnhmm6BfoGMaT+GKZ2msO7wOiLGRTBz+0w7BBZuLX9+iImB0aP1YY0REfoIGydxegG+mnaVN5e9yb3j72XPqT3EdYtjVNtRBPgGODuKcCWXLunuVffdB6dO6Xm5t96y+yWFHqn1CGsGrqF0cGm6xnelW3w3GQ17O6V0B7XfftPHDLdurfsJnzrl8FU7rQBblsWs7bOoObYmb/zyBt3DurP9ye30COvhrAjCFaWnw9Spuo/r//4Hgwbpw4PatnXYKmuUrMHagWt5t/m7zNk5h2qjq/H+8ve5nHLZYesUbqB+fX3a8ssvwzff6CMmRo+Gq1cdtkqHF2DLsli6bykNvmxAl/guKKWY03MOMV1i5GoW3iw9XZ+dFBkJjzyi90j//DOMGwcFCzp89f6+/rzY6EU2Dd1E4/KNefGnF6n8WWUmrJ9Acmqyw9cvXFRQkG7ck5AAYWHw1FN6R11MDKTaf+etwwpwcmoyUzZNIfKLSJpPac6hs4eY+NBEtjy+hfZV2jtqtcLVXbqkL5oZFqb3Pp8/D99+q9/wOTjEzF6qFqvKjz1/5Jf+v1C2YFkGzxlM+ZHleeuXtzh+8bjT8wgXEREBS5fC/PkQHAx9+kDFivpqG2fO2G01di3AlmWx9vBanpr3FCGfhNBvdj+upl1lwkMT2PXULh6LfAw/H5dvwCbsLT1dz68NHgxlysDQoZAvn5562L4doqPBx+wRkQ+Uf4CVj65kUZ9FRJaO5PVlrxPySQjd4rsxe8dsrqY57mOocFFKQZs2sGEDfP893HMPPP+8fg/37q13EN/hqPiOq+GV1Cv8sv8X5u2ax5xdc9h7ei+BvoF0rNqRgZEDaVmxpbSQ9EYXL+oRxLx5+uy1Q4d00e3S5frhZS72vlBK0fKelrS8pyXbj29nXMI4YhNjmbF9BoWDCtOmUhvaV25P60qtKZa32O0XKDyDj49uY9mxI/z+u/4EFx+vG74XL64b/LRrp3tMFCqUo0UrKwed4uvUqWPN/GkmG49uZNWhVaw4tIK1h9eSnJZMkF8Qze5uRpdqXegW2o2CQY6fx/NmTW0f15ctW2Y0B6DPrd+/X785V67UB7ivXw8pKbrotmwJ3bpBp07633bmyG2Rmp7Kkr1LiN0ay/zd8zl28RgKRViJMBqVbUTDcg25t9S9VC1W1SU+3bnU+8KTJSfr6Yn4eD0SPn1aF+patfSp8g0b6hOHKlUCX1+UUusty6pz82JyVID9QvystEFp+r6PH7VL16Zh2YY0r9icBys8KP16ncjp/9HS0uDoUT2SPXgQ9uzRHaV27oQtW3SHMoCAAKhbV78BW7bUI10HX6fNWdsi3Upn/ZH1LNyzkOUHl7MqaRXnkvXPHegbSFiJMKoWrUrlIpWpXLQy5QuWJ6RACGUKlHHaYZZSgA1ITdUnCy1aBMuXw+rVel8HQN68EB6OWrs20wKcoz/ZhfMU5s12b1KrZC3uLX0vef3z2ucHELmXmqrnWNPS/nlLTdW3lBR9KE1Kiu6pkJysv16+rG8XL+oLF54/rwvp6dP6duoUHD8Ox47pr2lpf193qVJQubLeQVGr1vVbUJCZbeFgPsqHumXqUrdMXQDS0tPYdnwbG49uZNNfm9j812ZWJa0idmssFn8f2BTNU5QS+UpQIl8JiuYtSuGgwhQOKkyBwAIEBwYTHBBMvoB85PHLQ17/vAT5BRHoF0igbyABvgEE+Abg7+uPn49fxs1X+eLr44uP8sFX6a8WFgrXmtrxeH5+10e9oP/Pbdmi+w5v2gSbN2f50hxPQSQkJNxpXPd27JiejL8mO9svq+fc+Pi1+zd+vfn+Dbemtpcty0H0bPH3h8KFr99KlLh+CwmBsmX17Z579FlELsDVRn3JqcnsO7OPg2cPcujsIQ6dO8Sxi8cybqcun+L0ldOcvnyay6l2PvZ4MuQLyMeFna57JWBvZJcpCKXUceCAPYPlQjHghOEMrkK2xXWyLa6TbXGdq2yL8pZl/ePEhxwVYFeglErI7C+JN5JtcZ1si+tkW1zn6tvCJdpRCiGEN5ICLIQQhrhjAf7CdAAXItviOtkW18m2uM6lt4XbzQELIYSncMcRsBBCeAS3LsBKqRFKKUsp5bUn5iulPlRK7VBKbVZKzVJK5exkdA+glGqjlPpDKbVbKfWi6TymKKXKKqV+VkptV0olKqWeMZ3JJKWUr1Lqd6XUHNNZsuK2BVgpVRZoCRw0ncWwxUC4ZVk1gZ3AS4bzOJVSyhf4HGgLhAI9lVKhZlMZkwoMtyyrOnAf8KQXbwuAZ4DtpkPcitsWYOAT4HnAqyexLctaZFnWtZ54q4EQk3kMqAfstixrr2VZV4FY4GHDmYywLOtPy7I22O6fRxefMmZTmaGUCgHaAxNNZ7kVtyzASqmOwGHLsjaZzuJiHgXmmw7hZGWAQzf8OwkvLTo3UkpVAO4F1phNYsxI9ADN+deazwHz/fOyoJRaApTK5FuvAC8DrZybyJxbbQvLsr63PecV9EfQGGdmcwGZdZ7x6k9FSqn8wAxgmGVZ50zncTalVAfgmGVZ65VSTU3nuRWXLcCWZbXI7HGlVA3gbmCTrdF7CLBBKVXPsiyPvLxtVtviGqVUP6AD0NzyvuMKk4CyN/w7BDhiKItxSil/dPGNsSxrpuk8hjQEOiql2gFBQAGl1FTLsvoYzvUPbn8csFJqP1DHsixXaLjhdEqpNsDHQBPLsrzuImZKKT/0zsfmwGFgHdDLsqxEo8EMUHpE8jVwyrKsYabzuALbCHiEZVkdTGfJjFvOAYu/GQ0EA4uVUhuVUuNMB3Im2w7IfwEL0Tud4r2x+No0BB4BmtneCxtto0Dhotx+BCyEEO5KRsBCCGGIFGAhhDBECrAQQhgiBVgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwpD/B/qlfbnDiDFSAAAAAElFTkSuQmCC\n", "text/plain": ["<Figure size 432x288 with 1 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import scipy.stats as stats\n", "import math\n", "\n", "mu1 = -2\n", "mu2 = 2\n", "variance = 1\n", "sigma = math.sqrt(variance)\n", "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n", "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n", "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\")\n", "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n", "plt.axvline(x=-2,color=\"black\")\n", "plt.axvline(x=0,color=\"black\")\n", "plt.axvline(x=+2,color=\"black\")\n", "plt.text(-2.7,0.55,\"Sensitive\")\n", "plt.text(-0.7,0.55,\"Balanced\")\n", "plt.text(1.7,0.55,\"Specific\")\n", "plt.ylim([0,0.53])\n", "plt.xlim([-5,5])\n", "plt.legend(['Negative','Positive'])\n", "plt.yticks([])\n", "#plt.set_yticklabels([])\n", "plt.show()"]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": ["from scipy.stats import zscore\n", "\n", "# Prepare data - apply z-score to ALL x columns\n", "# Only do this if you have no categoricals (and are sure you \n", "# want to use z-score across the board)\n", "x_columns = df.columns.drop('diagnosis').drop('id')\n", "for col in x_columns:\n", "    df[col] = zscore(df[col])\n", "\n", "# Convert to numpy - Regression\n", "x = df[x_columns].values\n", "y = df['diagnosis'].map({'M':1,\"B\":0}).values # Binary classification, \n", "                                              # M is 1 and B is 0"]}, {"cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": ["%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "from sklearn.metrics import roc_curve, auc\n", "\n", "# Plot a confusion matrix.\n", "# cm is the confusion matrix, names are the names of the classes.\n", "def plot_confusion_matrix(cm, names, title='Confusion matrix', \n", "                            cmap=plt.cm.Blues):\n", "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", "    plt.title(title)\n", "    plt.colorbar()\n", "    tick_marks = np.arange(len(names))\n", "    plt.xticks(tick_marks, names, rotation=45)\n", "    plt.yticks(tick_marks, names)\n", "    plt.tight_layout()\n", "    plt.ylabel('True label')\n", "    plt.xlabel('Predicted label')\n", "    \n", "\n", "# Plot an ROC. pred - the predictions, y - the expected output.\n", "def plot_roc(pred,y):\n", "    fpr, tpr, _ = roc_curve(y, pred)\n", "    roc_auc = auc(fpr, tpr)\n", "\n", "    plt.figure()\n", "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n", "    plt.plot([0, 1], [0, 1], 'k--')\n", "    plt.xlim([0.0, 1.0])\n", "    plt.ylim([0.0, 1.05])\n", "    plt.xlabel('False Positive Rate')\n", "    plt.ylabel('True Positive Rate')\n", "    plt.title('Receiver Operating Characteristic (ROC)')\n", "    plt.legend(loc=\"lower right\")\n", "    plt.show()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### ROC Chart Example"]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Train on 426 samples, validate on 143 samples\n", "Epoch 1/1000\n", "426/426 - 1s - loss: 0.6864 - accuracy: 0.8427 - val_loss: 0.6696 - val_accuracy: 0.9510\n", "Epoch 2/1000\n", "426/426 - 0s - loss: 0.6350 - accuracy: 0.9319 - val_loss: 0.5496 - val_accuracy: 0.9580\n", "Epoch 3/1000\n", "426/426 - 0s - loss: 0.4468 - accuracy: 0.9460 - val_loss: 0.2802 - val_accuracy: 0.9580\n", "Epoch 4/1000\n", "426/426 - 0s - loss: 0.2020 - accuracy: 0.9531 - val_loss: 0.1025 - val_accuracy: 0.9720\n", "Epoch 5/1000\n", "426/426 - 0s - loss: 0.1047 - accuracy: 0.9671 - val_loss: 0.0616 - val_accuracy: 0.9790\n", "Epoch 6/1000\n", "426/426 - 0s - loss: 0.0795 - accuracy: 0.9789 - val_loss: 0.0532 - val_accuracy: 0.9720\n", "Epoch 7/1000\n", "426/426 - 0s - loss: 0.0689 - accuracy: 0.9789 - val_loss: 0.0476 - val_accuracy: 0.9930\n", "Epoch 8/1000\n", "426/426 - 0s - loss: 0.0621 - accuracy: 0.9789 - val_loss: 0.0478 - val_accuracy: 0.9930\n", "Epoch 9/1000\n", "426/426 - 0s - loss: 0.0568 - accuracy: 0.9836 - val_loss: 0.0455 - val_accuracy: 0.9860\n", "Epoch 10/1000\n", "426/426 - 0s - loss: 0.0531 - accuracy: 0.9812 - val_loss: 0.0519 - val_accuracy: 0.9860\n", "Epoch 11/1000\n", "426/426 - 0s - loss: 0.0483 - accuracy: 0.9883 - val_loss: 0.0480 - val_accuracy: 0.9860\n", "Epoch 12/1000\n", "426/426 - 0s - loss: 0.0443 - accuracy: 0.9906 - val_loss: 0.0504 - val_accuracy: 0.9860\n", "Epoch 13/1000\n", "426/426 - 0s - loss: 0.0410 - accuracy: 0.9930 - val_loss: 0.0502 - val_accuracy: 0.9860\n", "Epoch 14/1000\n", "Restoring model weights from the end of the best epoch.\n", "426/426 - 0s - loss: 0.0383 - accuracy: 0.9930 - val_loss: 0.0489 - val_accuracy: 0.9860\n", "Epoch 00014: early stopping\n"]}, {"data": {"text/plain": ["<tensorflow.python.keras.callbacks.History at 0x1a4db23090>"]}, "execution_count": 6, "metadata": {}, "output_type": "execute_result"}], "source": ["# Classification neural network\n", "import numpy as np\n", "import tensorflow.keras\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense, Activation\n", "from tensorflow.keras.callbacks import EarlyStopping\n", "from sklearn.model_selection import train_test_split\n", "\n", "# Split into train/test\n", "x_train, x_test, y_train, y_test = train_test_split(    \n", "    x, y, test_size=0.25, random_state=42)\n", "\n", "model = Sequential()\n", "model.add(Dense(100, input_dim=x.shape[1], activation='relu',\n", "                kernel_initializer='random_normal'))\n", "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n", "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n", "model.add(Dense(1,activation='sigmoid',kernel_initializer='random_normal'))\n", "model.compile(loss='binary_crossentropy', \n", "              optimizer=tensorflow.keras.optimizers.Adam(),\n", "              metrics =['accuracy'])\n", "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n", "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n", "\n", "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n", "          callbacks=[monitor],verbose=2,epochs=1000)\n"]}, {"cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [{"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxN9f/A8dfbvgvTItTYMmNfJtmSkiUJSakoiYRskSgtSkoS8sVXfpJKRSpLaUFZvhVpZN+lYlCWGLvGeP/+OGema8zcuSZ3zp2Z9/PxuA/3LPec9/04c973fD7nfD6iqhhjjDEpyeZ1AMYYY0KbJQpjjDF+WaIwxhjjlyUKY4wxflmiMMYY45clCmOMMX5ZosjkRKSDiCzwOo5QIiLHRaSMB/sNFxEVkRzpve9gEJGNItIoDZ9L8zEpIk1FZE5aPptWIpJbRLaIyBXpud9QYokiHYnIbyJyyj1R/SEi00SkQDD3qarvq2rTYO7Dl4jUE5FvReSYiMSKyGciUjG99p9MPEtEpKvvPFUtoKo7g7S/60RklogcdL//OhHpLyLZg7G/tHITVrl/sw1VraSqS1LZzwXJ8V8eky8DI3y2ryJywv2b2iMio5OWtYi0FJGV7nqHROR9ESmZZJ3iIvKWiOxzj90tIvKCiORX1TPAVGBQGmPO8CxRpL87VLUAUB2oATzlcTxpktyvYhGpCywA5gJXA6WBtcD3wfgFH2q/zEWkLPAjsBuooqqFgbuBKKDgJd6XZ9/dq32LyPVAYVVdkWRRNfdv6iagPfCwz2faAR8AbwBhQCXgDPCdiBRx1ykKLAfyAnVVtSDQBLgMKOtu6gOgk4jkDtLXC22qaq90egG/Abf6TI8E5vtM5wZGAbuAP4FJQF6f5a2BNcBR4BeguTu/MPAWsA/YA7wEZHeXPQR8576fBIxKEtNcoL/7/mrgE+AA8CvQx2e9ocDHwHR3/12T+X7/AyYmM/9L4F33fSMgBngaOOiWSYdAysDns4OAP4D3gCLA527Mh933Jd31hwPxwGngODDena9AOff9NGACMB84hnOiL+sTT1NgKxALTASWJvfd3XWn+/5/JrM83N13J/f7HQSG+CyvjXPCOuL+X44HcvksV+AxYDvwqzvvDZzEdBRYBdzos352t5x/cb/bKqAUsMzd1gm3XNq767fEOb6OAD8AVZMcu4OAdTgn2hz4HM9u7NFuHH8Co935u9x9HXdfdfE5Jt11KgELgb/czz6dQvk9B0xJMi/x/9Kd/giY4L4X4HfgySSfyQZsAF50p18C1gPZUvn73Q7c5PV5xIuX5wFkpVeSP6yS7sH5hs/yscA8oCjOL9DPgFfcZbXdk1UT90AvAUS4y+YAbwL5gSuAlcCj7rLEP0qgoXtSEXe6CHAKJ0Fkc08kzwG5gDLATqCZu+5QIA5o466bN8l3y4dzUr45me/dGdjnvm8EnAVG4ySFm3BOWBUCKIOEz77qfjYvUAy4y91/QWAWMMdn30tIcmLnwkTxl1u+OYD3gRnusjCcE19bd1lftwxSShR/AJ39/P+Hu/v+Pzf2ajgn3Uh3eS2gjruvcGAz0C9J3AvdsklInh3dMsgBDHBjyOMuG4hzjFXAOWlWA4olLQN3uiawH7gBJ8F0wjlec/scu2twEk1en3kJx/Ny4AH3fQGgTpLvnMNnXw/xzzFZECcpDgDyuNM3pFB+s4CBfv4vI9xtPe4zrUDpZLb1ArDcfb8CeCGAv995+Px4ykovzwPISi/3D+s4zq87Bb4BLnOXCc4J0/fXbF3++eX4JjAmmW1e6Z5sfK887gMWu+99/ygF5xdeQ3f6EeBb9/0NwK4k234KeNt9PxRY5ue7lXS/U0Qyy5oDce77Rjgn+/w+yz8Cng2gDBoBf+OeCFOIozpw2Gd6Cakniik+y1oAW9z3DyacTHzKb3fS7fksj8O9ykthebi775I+81YC96awfj9gdpK4b0nlGDuMUxUDzpVQ6xTWS5oo/gsMS7LOVtxf0O6x+3Ayx3NColiGc/INS+E7p5Qo7gNWB/j3sxDonsz3OOoeNwp8yD/JrYE774LjBegObHffb0+63RT2/z7wXCCxZraXtVGkvzbq1IE2wvnFE+bOvxznV/EqETkiIkeAr9z54PyS+yWZ7V0L5AT2+XzuTZwri/Ooc7TPwPnjBLgf5+BP2M7VCdtwt/M0TiJKsNvP9zoMnAOKJ7OsOE41S+K6qnrCZ/p3nKua1MoA4ICqnk6YEJF8IvKmiPwuIkdxTliXXWTj8R8+70/i/CLGjSnxO7vlF+NnO4dI/vsHtD+3Ifxz90aHozgNt2FJPnve/4GIDBCRzW7D+RGcasiEz6R0zCTnWmBAkv//UjhlkOy+k+gCXAdsEZGfRKRlgPu9mBgPk3xbT02cMmyP84Mnvzs/4ZhL7ZgM9P+tIE61XJZjicIjqroU59fsKHfWQZxqoEqqepn7KqxOIx04f6RlL9wSu3GuKMJ8PldIVSulsOsPgXYici3OH9UnPtv51Wcbl6lqQVVt4Ru2n+9zAqf64e5kFt+Dc/WUoIiI5PeZvgbYG0AZJBfDAJyqlRtUtRBO9Ro4v/79xhyAfThXSs4GRcR3OhmLcKrB0uq/wBagvPtdnuaf75Eg8fuIyI047Qb3AEVU9TKc6smEz6R0zCRnNzA8yf9/PlX9MLl9J6Wq21X1PpwfKK8CH7v/x6mV/8XEuA4nGSW3f1XVj3COwefc2VtxEvt5x6SIZMP5f0o4JhcBd7rz/YnEuTkjy7FE4a2xQBMRqa6q53Dqrsck3K8tIiVEpJm77ltAZxFpLCLZ3GURqroP506j10WkkLusrIjclNwOVXU1TsPvFOBrVU34hbQSOCoig0Qkr4hkF5HK7p0mgRqMc2dIHxEpKCJFROQlnOqjF5Ks+4KI5HJPdi2BWQGUQXIK4iSXI+7dK88nWf4nTntLWswHqohIG/dOn8eAq/ys/zxQT0ReE5Gr3PjLich0EbksgP0VxKlGOS4iEUCPANY/i/P/mUNEngMK+SyfAgwTkfLiqCoixdxlScvl/4DuInKDu25+EbldRAK6W0tEOorI5e7/YcIxFe/Gdo6U/w8+B64SkX7iPK9QUERuSGHdL3DatPwZAXQTkavcK8AngGdE5H73uL4Kp1wKAWPcz4x2p99xf0AlHHejRaRqwjRO21DSO66yBEsUHlLVA8C7OPXz4Pw63AGscKseFuH8WkZVV+I0Co/B+dW4FKe6AJy69FzAJpzL84/xfyn9IXArzi1/CbHEA3fg1PH/ivPrfgpOVUag3+c7oBlO4+8+nCqlGkADVd3us+ofbpx7caq+uqvqltTKIAVjcRqGD+L8EX+VZPkbOFdQh0VkXKDfxf0+B3F+jY7EqZ6oiHNnz5kU1v8FJymGAxtFJBbnii0ap10qNU/gVAcewzlxz0xl/a9x7ijbhlPWpzm/emg0TvvPApwE9BZOWYHT5vSOW810j6pG47RZjcf5v9mB05YQqOY43/k4Tpnfq6qnVfUkzt1n37v7quP7IVU9hnODxh04x8V24ObkdqCqPwOxfhIJqroe529joDs9E3gAeBznGNnklkF9VT3krvMXUA+njelHETmGc7UR65YDOP8v76jzTEWWk3D3izHpQpwneaerqr8qnJDkVk3E4NzOu9jreLIiEWkK9FTVNum4z9w4VU4NVXV/eu03lITUA0vGhBq32utHnOqtgTj1/1my+iEUqOoCnCuk9NznGZwbT7Isq3oyxr+6OHflHMSpHmmjqqe8DcmY9GVVT8YYY/yyKwpjjDF+Zbg2irCwMA0PD/c6DGOMyVBWrVp1UFUvT33NC2W4RBEeHk50dLTXYRhjTIYiIr+n9bNW9WSMMcYvSxTGGGP8skRhjDHGL0sUxhhj/LJEYYwxxi9LFMYYY/wKWqIQkakisl9ENqSwXERknIjsEJF1IlIzWLEYY4xJu2BeUUzD6Xo4JbcB5d1XN5xBW4wxxoSYoD1wp6rLRCTczyqtgXfdwUVWiMhlIlLcHYgnpHzw4y7mrtnjdRjGGHNRVJU9a5ayZ83Sf7UdL5/MLsH5g6zEuPMuSBQi0g3nqoNrrrkmXYLzNXfNHjbtO0rF4oVSX9kYY0LAiYP7+Hnm6+xb/wOFS5T7V9vyMlEkHQsYUhhfV1UnA5MBoqKiPOnutmLxQsx8tK4XuzbGmIuiqkRFRXF051Zef/11+vTpQ86cOdO8PS8TRQxQyme6JM7QmMYYY9Lghx9+oEqVKhQsWJApU6YQFhZGqVKlUv9gKrxMFPOAXiIyA7gBiA2kfWLngRO0f3N50IPzZdVOxphQdujQIQYPHsyUKVN4/vnnGTp0KDVq1Lhk2w9aohCRD4FGQJiIxADPAzkBVHUS8AXQAmfw8pNA50C2eyouPhjh+lWxeCFaVy+R7vs1xhh/VJV3332XJ554gsOHDzNw4EAGDhx4yfeT4Ua4K3ptpP71+2avwzDGGM89+eSTvPbaa9SrV49JkyZRpUqVFNcVkVWqGpWW/WS48SiMMSYrO3XqFCdOnCAsLIwuXbpQvnx5unTpQrZswXsszrrwMMaYDOKrr76icuXKPProowBUqFCBRx55JKhJAixRGGNMyNu7dy/33HMPt912Gzlz5qRXr17pun+rejLGmBD2zTffcOedd/L3338zbNgwBg4cSO7cudM1BksUxhgTguLi4siZMyfVqlWjRYsWvPTSS5Qr9++esE4rq3oyxpgQcvToUfr27cuNN95IfHw8YWFhzJgxw7MkAZYojDEmJKgqs2bNIiIigv/85z9ERUVx5swZr8MCrOrJGGM8d+DAATp16sSXX35JjRo1mDt3Ltdff73XYSWyKwpjjPFYoUKFOHjwIGPHjmXlypUhlSTAEoUxxnhi2bJlNGvWjOPHj5M7d25WrFhB3759yZEj9Cp6LFEYY0w6OnjwIJ07d+amm25i27Zt/PbbbwBBf2ju3wjdyIwxJhNRVaZOnUqFChWYPn06Tz31FBs3bqRy5cpeh5aq0LvGMcaYTGr69OlUrFiRSZMmUalSJa/DCZhdURhjTJCcPHmSZ555hpiYGESETz75hKVLl2aoJAGWKIwxJii++OILKlWqxPDhw/nss88AKFKkSEi3RaQk40VsjDEhLCYmhnbt2nH77beTN29eli5dSo8ePbwO61+xRGGMMZfQ8OHDmT9/Pi+//DJr1qyhYcOGXof0r9kId8YY8y+tXLmSvHnzUqVKFQ4dOkRsbCxlypTxOqzz/JsR7uyKwhhj0ig2NpbHHnuMOnXqMGTIEACKFSsWckni37JEYYwxF0lVmTFjBhEREUyaNInevXszffp0r8MKGnuOwhhjLtL06dN58MEHiYqK4vPPP6dWrVpehxRUliiMMSYAZ86cYefOnURGRnLPPfdw9uxZHnzwQbJnz+51aEFnVU/GGJOKxYsXU61aNZo1a8aZM2fInTs3nTt3zhJJAixRGGNMivbv38+DDz7ILbfcQlxcHJMnT0738apDgVU9GWNMMnbs2EHt2rU5fvw4Q4YMYciQIeTNm9frsDxhicIYY3wcPXqUQoUKUbZsWbp06cLDDz9MZGSk12F5yqqejDEGOHHiBIMGDSI8PDyxE7/XXnstyycJsCsKY4zhs88+o1evXuzatYsuXbqQL18+r0MKKZYojDFZ1tmzZ7nnnnuYPXs2lSpV4n//+x8NGjTwOqyQY1VPxpgsJ6GPuxw5clC8eHFGjBjBzz//bEkiBZYojDFZyooVK4iKiuLnn38GYMKECQwaNIhcuXJ5HFnoskRhjMkSDh8+TI8ePahXrx5//vknhw8f9jqkDCOoiUJEmovIVhHZISKDk1l+jYgsFpHVIrJORFoEMx5jTNY0c+ZMIiIimDx5Mv369WPz5s00btzY67AyjKA1ZotIdmAC0ASIAX4SkXmquslntWeAj1T1vyJSEfgCCA9WTMaYrGnLli2Eh4fz1VdfUaNGDa/DyXCCeUVRG9ihqjtV9W9gBtA6yToKFHLfFwb2BjEeY0wWcfr0aV544YXEsaqffvppfvjhB0sSaRTMRFEC2O0zHePO8zUU6CgiMThXE72T25CIdBORaBGJjouLC0asxphMYtGiRVStWpWhQ4eydOlSAHLmzJllOvALhmAmCklmXtJxV+8DpqlqSaAF8J6IXBCTqk5W1ShVjcqZM2cQQjXGZHR//vknHTp0oEmTJqgqCxYsYNSoUV6HlSkEM1HEAKV8pktyYdVSF+AjAFVdDuQBwoIYkzEmk1q4cCEff/wxzz33HOvXr6dJkyZeh5RpBPPJ7J+A8iJSGtgD3Avcn2SdXUBjYJqIROIkigNBjMkYk4msXbuW7du3065dOzp06ED9+vUpXbq012FlOkG7olDVs0Av4GtgM87dTRtF5EURaeWuNgB4RETWAh8CD2nCI5PGGJOC48ePM2DAAGrVqsXgwYM5e/YsImJJIkgko52Xi14bqX/9vtnrMIwxHpkzZw69e/cmJiaGbt268corr1C0aFGvwwp5IrJKVaPS8lnrFNAYk2GsX7+eO++8kypVqjBz5kzq1avndUhZgnXhYYwJaXFxcXz77bcAVKlShfnz57Nq1SpLEunIEoUxJmT98MMP1KpViyZNmrBjxw4AWrRogd0mn74sURhjQs5ff/1Ft27dqF+/PkeOHOHTTz+lXLlyXoeVZVkbhTEmpJw+fZrq1auzd+9eBgwYwNChQylQoIDXYWVpliiMMSEhJiaGkiVLkidPHoYNG0b16tWpVq2a12EZrOrJGOOxU6dO8dxzz1G2bNnETvw6depkSSKEBHRFISK5gGtUdUeQ4zHGZCELFiygZ8+e/PLLL3Ts2JHatWt7HZJJRqpXFCJyO7AeWOhOVxeR2cEOzBiTufXu3ZtmzZqRLVs2Fi1axHvvvceVV17pdVgmGYFcUbwI3AAsBlDVNSJitx8YYy5afHw8ANmzZ6dOnTqEhYUxaNAg8uTJ43Fkxp9A2ijiVPVIknkZq98PY4znfv75Z+rWrcvEiRMB6NChA88//7wliQwgkESxWUTuAbKJSGkRGQusCHJcxphM4tixYzz++ONcf/317Nq1i+LFi3sdkrlIgSSKXkAt4BzwKXAa6BvMoIwxmcOCBQuIjIzkjTfe4NFHH2XLli20a9fO67DMRQqkjaKZqg4CBiXMEJG2OEnDGGNSlCtXLq644go++eQTbrjhBq/DMWmUajfjIvKzqtZMMm+VqtYKamQpsG7GjQldcXFxjB49mqNHjzJ8+HAAzp07R7Zs9siW14LSzbiINAOaAyVEZLTPokI41VDGGJPou+++o3v37mzcuJG77747MUFYksj4/P0P7gc24LRJbPR5LQBuC35oxpiM4NChQ3Tt2pUbb7yRY8eO8dlnn/HRRx9ZgshEUryiUNXVwGoReV9VT6djTMaYDOTQoUPMmDGDJ598kueee478+fN7HZK5xAJpzC4hIsOBikDiDc+qel3QojLGhLTNmzfz0Ucf8fzzz3Pdddexa9cuG440Ewvk2nAa8DYgOFVOHwEzghiTMSZEnTx5kiFDhlCtWjXeeOMNYmJiACxJZHKBJIp8qvo1gKr+oqrPADcHNyxjTKj56quvqFy5Mi+//DL3338/W7dupWTJkl6HZdJBIFVPZ0REgF9EpDuwB7giuGEZY0LJ8ePHeeCBByhWrBiLFy+mUaNGXodk0lEgVxSPAwWAPkB94BHg4WAGZYzxXnx8PNOnTyc+Pp4CBQqwaNEi1q5da0kiC0r1ikJVf3TfHgMeABARu940JhNbtWoVjz76KKtWrSJv3rzcddddNpBQFub3ikJErheRNiIS5k5XEpF3sU4BjcmUYmNj6dOnD7Vr12bPnj3MmDGDtm3beh2W8ViKiUJEXgHeBzoAX4nIEJwxKdYCdmusMZnQXXfdxfjx4+nZsydbtmyhffv2OE2UJivzV/XUGqimqqdEpCiw153emj6hGWPSw86dO7n88sspWLAgw4cPJ1u2bFx//fVeh2VCiL+qp9OqegpAVf8CtliSMCbz+Pvvv3n55ZepVKkSL730EgA33HCDJQlzAX9XFGVEJKErcQHCfaZRVau4NCaDWrZsGd27d2fz5s20a9eOPn36eB2SCWH+EsVdSabHBzMQY0z6GDNmDP379yc8PJz58+fTokULr0MyIc5fp4DfpGcgxpjgOXfuHCdOnKBgwYLcfvvtHDhwgGeeeYZ8+fJ5HZrJAFIduCjU2MBFxlycjRs30r1798SR5kzW9G8GLgpqh/Ei0lxEtorIDhEZnMI694jIJhHZKCIfBDMeY7KSkydP8tRTT1G9enU2b95My5YtyWg/DE1oCKSvJwBEJLeqnrmI9bMDE4AmQAzwk4jMU9VNPuuUB54C6qvqYRGxPqSMuQRWr15N27Zt+e233+jcuTMjR44kLCzM67BMBpXqFYWI1BaR9cB2d7qaiPwngG3XBnao6k5V/Runa/LWSdZ5BJigqocBVHX/RUVvjDlPwhXDNddcwzXXXMPSpUuZOnWqJQnzrwRS9TQOaAkcAlDVtQTWzXgJYLfPdIw7z9d1wHUi8r2IrBCR5gFs1xiTxNmzZxk7diyNGzcmPj6eYsWKsXTpUho2bOh1aCYTCCRRZFPV35PMiw/gc8k995+0gjQHUB5oBNwHTBGRyy7YkEg3EYkWkei4uLgAdm1M1rFy5Upq167N448/Tp48eTh69KjXIZlMJpBEsVtEagMqItlFpB+wLYDPxQClfKZL4nQDknSduaoap6q/AltxEsd5VHWyqkapalTOnDkD2LUxmd/x48d57LHHqFOnDn/++SezZs1i/vz5FClSxOvQTCYTSKLoAfQHrgH+BOq481LzE1BeREqLSC7gXmBeknXm4FZjuT3UXgfsDCx0Y7K2nDlzsmTJEnr37p34hLV14GeCIZC7ns6q6r0Xu2FVPSsivYCvgezAVFXdKCIvAtGqOs9d1lRENuFUZw1U1UMXuy9jsoodO3bw4osvMmHCBAoWLMiqVavIkyeP12GZTC7VB+5E5BecKqGZwKeqeiw9AkuJPXBnsqIzZ84wcuRIhg8fTq5cuZg/fz433nij12GZDCSoD9ypalngJaAWsF5E5ojIRV9hGGPSZvHixVSrVo3nnnuONm3asGXLFksSJl0F9GS2qv6gqn2AmsBRnAGNjDFBpqoMHz6cuLg4vvrqK2bMmMHVV1/tdVgmi0m1jUJECuA8KHcvEAnMBeoFOS5jsqxz587x1ltv0bx5c0qVKsV7773HZZddRt68eb0OzWRRgVxRbMC502mkqpZT1QGq+mOQ4zImS1q3bh0NGjSgW7duTJkyBYDixYtbkjCeCuSupzKqei7okRiThR0/fpwXXniBMWPGUKRIEaZNm8aDDz7odVjGAH4ShYi8rqoDgE9E5IJbo2yEO2MunaFDh/L666/TtWtXRowYQbFixbwOyZhEKd4eKyK1VXWliDRObrlXAxvZ7bEms9i9ezcnTpwgIiKCgwcPsmXLFho0aOB1WCaTCsrtsaq60n0bqarf+L5wGrWNMWlw9uxZRo8eTWRkJI8++igAYWFhliRMyAqkMfvhZOZ1udSBGJMVrFixgqioKAYMGECjRo145513vA7JmFT5a6Noj3NLbGkR+dRnUUHgSLADMyazmT9/PnfccQdXX301n376KW3atLG+mUyG4O+up5U4Y1CUxBmpLsExYHUwgzIms1BV9u7dS4kSJbj11lt58cUX6du3LwULFvQ6NGMClmpfT6HGGrNNRrFt2zZ69uzJtm3b2LRpEwUKFPA6JJOFBaUxW0SWuv8eFpG/fF6HReSvtAZrTGZ3+vRphg4dSpUqVYiOjuapp56yB+ZMhuav6ilhuFMbbNeYAP3xxx80bNiQ7du3c9999zF69Giuuuoqr8My5l/xd3tswtPYpYDsqhoP1AUeBfKnQ2zGZBgJQ/ReeeWVNGzYkAULFvDBBx9YkjCZQiC3x87BGQa1LPAuzjMUHwQ1KmMyiHPnzjFp0iTKli1LTEwMIsKUKVNo0qSJ16EZc8kEkijOqWoc0BYYq6q9gRLBDcuY0Ld27Vrq1atHjx49KF++fOJVhTGZTSCJ4qyI3A08AHzuzssZvJCMCW2qyhNPPEGtWrXYuXMn7733HosWLaJ06dJeh2ZMUAT6ZPbNON2M7xSR0sCHwQ3LmNAlIhw+fJguXbqwdetWOnbsaA/OmUwtoOcoRCQHUM6d3KGqZ4MalR/2HIXxwu+//07fvn157rnnqFmzJufOnSNbtoAGiDQmJAR1zGwRuRHYAbwFTAW2iUj9tOzMmIwmLi6OkSNHUrFiRRYuXMjWrVsBLEmYLCWQgYvGAC1UdROAiEQC7wFpykzGZBQ//PADjz76KBs2bKB169aMGzeOa665xuuwjEl3gSSKXAlJAkBVN4tIriDGZExIWLRoEbGxscyZM4fWrVt7HY4xnkm1jUJEpgFncK4iADoA+VS1U3BDS561UZhgUVXee+89Lr/8cm677TbOnDlDXFyc9dFkMoWgtlEA3YFfgCeBQcBOnKezjck0tmzZwi233EKnTp14++23AcidO7clCWNIpepJRKoAZYHZqjoyfUIyJv2cOnWKl19+mVdffZX8+fPz5ptv0rVrV6/DMiak+Os99mmc7js6AAtFJLmR7ozJ0D777DNeeukl2rdvz5YtW+jWrZvd0WRMEv6uKDoAVVX1hIhcDnyBc3usMRnaH3/8wZo1a2jevDl333034eHh1K5d2+uwjAlZ/n46nVHVEwCqeiCVdY0JefHx8UycOJEKFSrwwAMPcOrUKUTEkoQxqfB3RVHGZ6xsAcr6jp2tqm2DGpkxl9DPP/9M9+7d+emnn7j11luZOHGiDSZkTID8JYq7kkyPD2YgxgTLr7/+Su3atQkLC+ODDz7g3nvvtb6ZjLkIKSYKVf0mPQMx5lJSVdavX0/VqlUpXbo0b7/9NnfccQeXXXaZ16EZk+FYu4PJdH799VdatmxJjRo1WLduHQAPPPCAJQlj0iioiUJEmovIVhHZISKD/azXTkRURKz/KJNmf//9NyNGjKBSpUosXbqUUaNGUbFiRa/DMj45dk0AABxHSURBVCbDC6SvJwBEJLeqnrmI9bMDE4AmQAzwk4jM8+03yl2vINAH+DHQbRuTVHx8PPXq1WPVqlW0bduWsWPHUqpUKa/DMiZTCKSb8doish7Y7k5XE5H/BLDt2jhjV+xU1b+BGUByPasNA0YCpwMP2xjH0aNHAciePTsPP/wwn332GZ988oklCWMuoUCqnsYBLYFDAKq6FmfEu9SUAHb7TMeQZKxtEakBlFLVz/FDRLqJSLSIRNu4xAacxupp06ZRpkwZ5s6dC0DPnj1p2bKlx5EZk/kEkiiyqervSebFB/C55O4/TOyqVkSy4Yx1MSC1DanqZFWNUtWonDltuO6sbtOmTTRq1IjOnTsTERFB2bJlvQ7JmEwtkESxW0RqAyoi2UWkH7AtgM/FAL7X/yWBvT7TBYHKwBIR+Q2oA8yzBm3jz8iRI6lWrRobNmxgypQpLFu2jMqVK3sdljGZWiCJogfQH7gG+BPnhN4jgM/9BJQXkdLuQEf3AvMSFqpqrKqGqWq4qoYDK4BWqhp9kd/BZAEJ46ZcddVVdOjQgS1bttClSxfrwM+YdJDqwEX/auMiLYCxQHZgqqoOF5EXgWhVnZdk3SXAE6klChu4KGvZu3cvffv25cYbb6RPnz5eh2NMhvVvBi5K9fZYEfk/fNoWEqhqt9Q+q6pf4PQ66zvvuRTWbZTa9kzWkdCB35AhQ4iLi6NevXpeh2RMlhXIcxSLfN7nAe7k/LuZjLmk1qxZQ9euXVm1ahVNmzZl4sSJ1mBtjIdSTRSqOtN3WkTeAxYGLSKT5cXGxrJ3715mzpzJ3XffbR34GeOxgJ/M9lEauPZSB2KyLlVl1qxZbN++nSFDhnDTTTexc+dO8uTJ43VoxhgCezL7sIj85b6O4FxNPB380ExW8Msvv9CiRQvat2/P3LlzSXig0pKEMaHDb6IQ55q/GnC5+yqiqmVU9aP0CM5kXmfOnGH48OFUrlyZ77//njfeeIMffvgBe6DSmNDjt+pJVVVEZqtqrfQKyGQNu3fvZtiwYdxxxx2MHTuWEiVKpP4hY4wnAnlaaaWI1Ax6JCbTO3DgAOPHOwMllitXjk2bNjFr1ixLEsaEuBQThYgkXG00wEkWW0XkZxFZLSI/p094JjM4d+4cb731FhEREfTv35+tW7cCUKZMGY8jM8YEwl/V00qgJtAmnWIxmdCGDRvo0aMH3333HTfeeCOTJk2iQoUKXodljLkI/hKFAKjqL+kUi8lk/v77b5o2bcrff//N1KlTeeihh+yZCGMyIH+J4nIR6Z/SQlUdHYR4TCbw7bffctNNN5ErVy4++ugjIiIiCAsL8zosY0wa+WvMzg4UwOkOPLmXMeeJiYnhrrvuonHjxrz77rsANGjQwJKEMRmcvyuKfar6YrpFYjKss2fPMn78eJ599lni4+N55ZVX6NChg9dhGWMukVTbKIxJzQMPPMCMGTO47bbbmDBhAqVLl/Y6JGPMJZTieBQiUlRV/0rneFJl41GEhiNHjpAjRw4KFCjAd999xx9//MFdd91ljdXGhKh/Mx5Fim0UoZgkjPdUlRkzZhAZGcmzzz4LOO0Q7dq1syRhTCZl40iagO3YsYNmzZpx3333UbJkSTp27Oh1SMaYdGCJwgTkgw8+oHLlyvz444+MHz+eFStWUKuWdQFmTFaQlvEoTBYSFxdHzpw5iYqKol27dowcOZKrr77a67CMMekoxcbsUGWN2elj//79DBgwgBMnTvDpp596HY4x5l8KSmO2yZrOnTvH5MmTqVChAjNnzqRSpUrEx8d7HZYxxkNW9WQS7dy5k44dO7J8+XIaNWrEf//7XyIiIrwOyxjjMUsUJlHhwoU5cuQI77zzDg888IDd7mqMAazqKcubN28ebdu2JT4+nmLFirFhwwYefPBBSxLGmESWKLKoXbt20aZNG1q3bs22bdvYt28fANmy2SFhjDmfnRWymLNnzzJq1CgiIyNZsGABr776KqtXr6ZkyZJeh2aMCVHWRpHFxMfHM2XKFG655Rb+85//EB4e7nVIxpgQZ1cUWcDhw4cZNGgQx44dI3fu3Hz//ffMmzfPkoQxJiCWKDIxVeX9998nIiKC119/ncWLFwNQrFgxa6w2xgTMEkUmtW3bNpo0aULHjh0JDw8nOjqaVq1aeR2WMSYDsjaKTKpfv35ER0czceJEunXrRvbs2b0OyRiTQVmiyEQWLlxIREQEpUqV4r///S+5c+fmqquu8josY0wGF9SqJxFpLiJbRWSHiAxOZnl/EdkkIutE5BsRuTaY8WRWf/zxB/fffz9Nmzbl1VdfBeDaa6+1JGGMuSSClihEJDswAbgNqAjcJyIVk6y2GohS1arAx8DIYMWTGZ07d45JkyYRERHBJ598wvPPP8+oUaO8DssYk8kE84qiNrBDVXeq6t/ADKC17wqqulhVT7qTKwB76usivPLKK/To0YNatWqxbt06hg4dSp48ebwOyxiTyQSzjaIEsNtnOga4wc/6XYAvk1sgIt2AbgAFipe9VPFlSMeOHePgwYOULl2a7t27U7p0ae677z673dUYEzTBvKJI7syV7ChJItIRiAJeS265qk5W1ShVjcqZM+clDDHjUFVmz55NxYoVad++PapKsWLFuP/++y1JGGOCKpiJIgYo5TNdEtibdCURuRUYArRS1TNBjCfD+v3332nVqhVt27alaNGijBs3zpKDMSbdBLPq6SegvIiUBvYA9wL3+64gIjWAN4Hmqro/iLFkWMuXL+fWW28FYNSoUfTt25ccOeyuZmNM+gnaFYWqngV6AV8Dm4GPVHWjiLwoIgmPCL8GFABmicgaEZkXrHgymqNHjwJQs2ZNHn74YTZv3syAAQMsSRhj0p2oJttsELKKXhupf/2+2eswgubQoUMMHjyYBQsWsHHjRgoUKOB1SMaYTEBEVqlqVFo+a309hQhV5d133yUiIoK3336b9u3bWzuEMSYkWD1GCIiNjaVNmzYsWbKEunXrMmnSJKpWrep1WMYYA1ii8JSqIiIUKlSIsLAwJk+eTJcuXWw4UmNMSLEzkke+/vpratasSUxMDCLCrFmzeOSRRyxJGGNCjp2V0tm+ffu49957ad68OSdPnmT/frsr2BgT2ixRpKMJEyYQERHBnDlzeOGFF1i3bh01a9b0OixjjPHL2ijS0apVq7jhhhuYMGEC5cuX9zocY4wJiF1RBNHRo0fp168fq1atAmDixIl8/fXXliSMMRmKJYogUFU+/vhjIiMjGTduHEuXLgUgT5489myEMSbDsURxif3666+0bNmSu+++myuuuILly5fTv39/r8Myxpg0s0Rxib3//vssW7aMMWPG8NNPP3HDDf6G4DDGmNBnfT1dAv/73/84c+YMt956K2fOnOHAgQOULGmD9RljQof19eSRgwcP8vDDD9OwYUNefPFFAHLnzm1JwhiTqdjtsWmgqkybNo2BAwcSGxvLoEGDePbZZ70Oy4SYuLg4YmJiOH36tNehmCwkT548lCxZkks5GqglijT44osvePjhh6lfvz6TJk2icuXKXodkQlBMTAwFCxYkPDzc7nYz6UJVOXToEDExMZQuXfqSbdeqngJ08uRJvv/+ewBatGjB3LlzWbZsmSUJk6LTp09TrFgxSxIm3YgIxYoVu+RXsZYoAvDll19SuXJlbrvtNo4cOYKI0KpVK+vAz6TKkoRJb8E45uxM58eePXu4++67adGiBblz5+azzz7jsssu8zosY4xJV5YoUrB//34qVqzI559/zksvvcTatWu56aabvA7LmIuSPXt2qlevTuXKlbnjjjs4cuRI4rKNGzdyyy23cN1111G+fHmGDRuG7+3yX375JVFRUURGRhIREcETTzzhxVfwa/Xq1XTt2tXrMFJ06NAhbr75ZgoUKECvXr1SXO+vv/6iSZMmlC9fniZNmnD48GHAaXPo06cP5cqVo2rVqvz8888AHDhwgObNm6fLdwBLFBfYs2cPAFdccQXDhg1jw4YNDBkyhFy5cnkcmTEXL2/evKxZs4YNGzZQtGhRJkyYAMCpU6do1aoVgwcPZtu2baxdu5YffviBiRMnArBhwwZ69erF9OnT2bx5Mxs2bKBMmTKXNLazZ8/+6228/PLL9O7dO133eTHy5MnDsGHDGDVqlN/1RowYQePGjdm+fTuNGzdmxIgRgJOst2/fzvbt25k8eTI9evQA4PLLL6d48eKJ7abBZnc9uWJjY3nmmWd48803WbFiBTVr1qRPnz5eh2UyiRc+28imvUcv6TYrXl2I5++oFPD6devWZd26dQB88MEH1K9fn6ZNmwKQL18+xo8fT6NGjXjssccYOXIkQ4YMISIiAoAcOXLQs2fPC7Z5/PhxevfuTXR0NCLC888/z1133UWBAgU4fvw4AB9//DGff/4506ZN46GHHqJo0aKsXr2a6tWrM3v2bNasWZNYpVuuXDm+//57smXLRvfu3dm1axcAY8eOpX79+uft+9ixY6xbt45q1aoBsHLlSvr168epU6fImzcvb7/9NhUqVGDatGnMnz+f06dPc+LECb799ltee+01PvroI86cOcOdd97JCy+8AECbNm3YvXs3p0+fpm/fvnTr1i3g8k1O/vz5adCgATt27PC73ty5c1myZAkAnTp1olGjRrz66qvMnTuXBx98EBGhTp06HDlyhH379lG8eHHatGnD+++/f0G5BEOWTxSqyqxZs+jXrx9//PEHvXr1omzZsl6HZcwlFR8fzzfffEOXLl0Ap9qpVq1a561TtmxZjh8/ztGjR9mwYQMDBgxIdbvDhg2jcOHCrF+/HiCxysSfbdu2sWjRIrJnz865c+eYPXs2nTt35scffyQ8PJwrr7yS+++/n8cff5wGDRqwa9cumjVrxubN5/fIEB0dfd5dhxERESxbtowcOXKwaNEinn76aT755BMAli9fzrp16yhatCgLFixg+/btrFy5ElWlVatWLFu2jIYNGzJ16lSKFi3KqVOnuP7667nrrrsoVqzYeft9/PHHWbx48QXf695772Xw4MGpfv/k/PnnnxQvXhyA4sWLJw5otmfPHkqVKpW4XsmSJdmzZw/FixcnKiqKZ555Jk37u1hZOlGoKm3btmXOnDnUrFmTefPmERWVpifcjfHrYn75X0qnTp2ievXq/Pbbb9SqVYsmTZoA/4zXnpyLuWtm0aJFzJgxI3G6SJEiqX7m7rvvJnv27AC0b9+eF198kc6dOzNjxgzat2+fuN1NmzYlfubo0aMcO3aMggULJs7bt28fl19+eeJ0bGwsnTp1Yvv27YgIcXFxicuaNGlC0aJFAViwYAELFiygRo0agHNVtH37dho2bMi4ceOYPXs2ALt372b79u0XJIoxY8YEVjiXQHJdLCX8/1xxxRXs3bs3XeLIkokiLi6OnDlzIiI0aNCAW265hZ49eyYevMZkFgltFLGxsbRs2ZIJEybQp08fKlWqxLJly85bd+fOnRQoUICCBQtSqVIlVq1alVitk5KUEo7vvKT39OfPnz/xfd26ddmxYwcHDhxgzpw5ib+Qz507x/Lly8mbN6/f7+a77WeffZabb76Z2bNn89tvv9GoUaNk96mqPPXUUzz66KPnbW/JkiUsWrSI5cuXky9fPho1apTs8wjBuKK48sorE6uU9u3bxxVXXAE4VxC7d+9OXC8mJoarr74acMrVX/lcSlmuMXvJkiVUrVqVuXPnAjBgwAB69+5tScJkaoULF2bcuHGMGjWKuLg4OnTowHfffceiRYsA58qjT58+PPnkkwAMHDiQl19+mW3btgHOiXv06NEXbLdp06aMHz8+cTqh6unKK69k8+bNiVVLKRER7rzzTvr3709kZGTir/ek212zZs0Fn42MjDyv7j82NpYSJUoAMG3atBT32axZM6ZOnZrYhrJnzx72799PbGwsRYoUIV++fGzZsoUVK1Yk+/kxY8awZs2aC15pTRIArVq14p133gHgnXfeoXXr1onz3333XVSVFStWULhw4cQqqm3btqXbA79ZJlEcOHCATp06cfPNN3PmzJnzLmGNyQpq1KhBtWrVmDFjBnnz5mXu3Lm89NJLVKhQgSpVqnD99dcn3sJZtWpVxo4dy3333UdkZCSVK1dm3759F2zzmWee4fDhw1SuXJlq1aol/tIeMWIELVu25JZbbkk8saWkffv2TJ8+PbHaCWDcuHFER0dTtWpVKlasyKRJky74XEREBLGxsRw7dgyAJ598kqeeeor69esTHx+f4v6aNm3K/fffT926dalSpQrt2rXj2LFjNG/enLNnz1K1alWeffZZ6tSpk3qhBiA8PJz+/fszbdo0SpYsmVil1rVrV6KjowEYPHgwCxcupHz58ixcuDAx6bRo0YIyZcpQrlw5HnnkkcS70gAWL17M7bfffkliTE2W6Gb8ww8/5LHHHuP48eMMHDiQIUOGkC9fviBFaIxj8+bNREZGeh1GpjZmzBgKFiwY0s9SBEvDhg2ZO3dusu1CyR171s14Ks6ePUvlypVZs2YNw4cPtyRhTCbRo0cPcufO7XUY6e7AgQP0798/oJsHLoVMeUVx4sQJhg0bxjXXXEPPnj0T7xywfndMerIrCuMVu6JIxeeff06lSpV49dVXExviRMSShPFERvshZjK+YBxzmSZRxMTE0LZtW+644w7y58/PsmXLGDt2rNdhmSwsT548HDp0yJKFSTcJ41HkyZPnkm430zxHsXPnTr7++mteeeUV+vfvb30zGc+VLFmSmJgYDhw44HUoJgtJGOHuUsrQbRQrV65k+fLl9O3bF3B6akz6FKUxxpgQbqMQkeYislVEdojIBU+jiEhuEZnpLv9RRMID2e6RI0fo2bMnderUYfTo0Zw4cQLAkoQxxgRB0BKFiGQHJgC3ARWB+0SkYpLVugCHVbUcMAZ4NbXt/n0yloiICN5880369OnD+vXrz3s83xhjzKUVzCuK2sAOVd2pqn8DM4DWSdZpDbzjvv8YaCyp3J504uAflCpVip9++omxY8dSqFChSx64McaYfwSzMbsEsNtnOga4IaV1VPWsiMQCxYCDviuJSDcgoWP4M9HR0RuSdpGcRYWRpKyyMCuLf1hZ/MPK4h8V0vrBYCaK5K4MkracB7IOqjoZmAwgItFpbZDJbKws/mFl8Q8ri39YWfxDRKLT+tlgVj3FAKV8pksCSTtPT1xHRHIAhYG/ghiTMcaYixTMRPETUF5ESotILuBeYF6SdeYBndz37YBvNaPdr2uMMZlc0Kqe3DaHXsDXQHZgqqpuFJEXgWhVnQe8BbwnIjtwriTuDWDTk4MVcwZkZfEPK4t/WFn8w8riH2kuiwz3wJ0xxpj0lWn6ejLGGBMcliiMMcb4FbKJIljdf2REAZRFfxHZJCLrROQbEbnWizjTQ2pl4bNeOxFREcm0t0YGUhYico97bGwUkQ/SO8b0EsDfyDUislhEVrt/Jy28iDPYRGSqiOwXkQ0pLBcRGeeW0zoRqRnQhlU15F44jd+/AGWAXMBaoGKSdXoCk9z39wIzvY7bw7K4Gcjnvu+RlcvCXa8gsAxYAUR5HbeHx0V5YDVQxJ2+wuu4PSyLyUAP931F4Dev4w5SWTQEagIbUljeAvgS5xm2OsCPgWw3VK8ogtL9RwaValmo6mJVPelOrsB5ZiUzCuS4ABgGjAROp2dw6SyQsngEmKCqhwFUdX86x5heAikLBRL6+ynMhc90ZQqqugz/z6K1Bt5VxwrgMhEpntp2QzVRJNf9R4mU1lHVs0BC9x+ZTSBl4asLzi+GzCjVshCRGkApVf08PQPzQCDHxXXAdSLyvYisEJHm6RZd+gqkLIYCHUUkBvgC6J0+oYWciz2fAKE7cNEl6/4jEwj4e4pIRyAKuCmoEXnHb1mISDacXogfSq+APBTIcZEDp/qpEc5V5v9EpLKqHglybOktkLK4D5imqq+LSF2c57cqq+q54IcXUtJ03gzVKwrr/uMfgZQFInIrMARopapn0im29JZaWRQEKgNLROQ3nDrYeZm0QTvQv5G5qhqnqr8CW3ESR2YTSFl0AT4CUNXlQB6cDgOzmoDOJ0mFaqKw7j/+kWpZuNUtb+IkicxaDw2plIWqxqpqmKqGq2o4TntNK1VNc2doISyQv5E5ODc6ICJhOFVRO9M1yvQRSFnsAhoDiEgkTqLIimPUzgMedO9+qgPEquq+1D4UklVPGrzuPzKcAMviNaAAMMttz9+lqq08CzpIAiyLLCHAsvgaaCoim4B4YKCqHvIu6uAIsCwGAP8nIo/jVLU8lBl/WIrIhzhVjWFue8zzQE4AVZ2E0z7TAtgBnAQ6B7TdTFhWxhhjLqFQrXoyxhgTIixRGGOM8csShTHGGL8sURhjjPHLEoUxxhi/LFGYkCMi8SKyxucV7mfd8JR6yrzIfS5xex9d63Z5USEN2+guIg+67x8Skat9lk0RkYqXOM6fRKR6AJ/pJyL5/u2+TdZlicKEolOqWt3n9Vs67beDqlbD6WzytYv9sKpOUtV33cmHgKt9lnVV1U2XJMp/4pxIYHH2AyxRmDSzRGEyBPfK4X8i8rP7qpfMOpVEZKV7FbJORMq78zv6zH9TRLKnsrtlQDn3s43dMQzWu33953bnj5B/xgAZ5c4bKiJPiEg7nD633nf3mde9EogSkR4iMtIn5odE5D9pjHM5Ph26ich/RSRanLEnXnDn9cFJWItFZLE7r6mILHfLcZaIFEhlPyaLs0RhQlFen2qn2e68/UATVa0JtAfGJfO57sAbqlod50Qd43bX0B6o786PBzqksv87gPUikgeYBrRX1So4PRn0EJGiwJ1AJVWtCrzk+2FV/RiIxvnlX11VT/ks/hho6zPdHpiZxjib43TTkWCIqkYBVYGbRKSqqo7D6cvnZlW92e3K4xngVrcso4H+qezHZHEh2YWHyfJOuSdLXzmB8W6dfDxOv0VJLQeGiEhJ4FNV3S4ijYFawE9u9yZ5cZJOct4XkVPAbzjdUFcAflXVbe7yd4DHgPE4Y11MEZH5QMBdmqvqARHZ6fazs93dx/fudi8mzvw43VX4jlB2j4h0w/m7Lo4zQM+6JJ+t487/3t1PLpxyMyZFlihMRvE48CdQDedK+IJBiVT1AxH5Ebgd+FpEuuJ0q/yOqj4VwD46+HYgKCLJjm/i9i1UG6eTuXuBXsAtF/FdZgL3AFuA2aqq4py1A44TZxS3EcAEoK2IlAaeAK5X1cMiMg2n47ukBFioqvddRLwmi7OqJ5NRFAb2ueMHPIDza/o8IlIG2OlWt8zDqYL5BmgnIle46xSVwMcU3wKEi0g5d/oBYKlbp19YVb/AaShO7s6jYzjdnifnU6ANzhgJM915FxWnqsbhVCHVcautCgEngFgRuRK4LYVYVgD1E76TiOQTkeSuzoxJZInCZBQTgU4isgKn2ulEMuu0BzaIyBogAmfIx004J9QFIrIOWIhTLZMqVT2N07vmLBFZD5wDJuGcdD93t7cU52onqWnApITG7CTbPQxsAq5V1ZXuvIuO0237eB14QlXX4oyPvRGYilOdlWAy8KWILFbVAzh3ZH3o7mcFTlkZkyLrPdYYY4xfdkVhjDHGL0sUxhhj/LJEYYwxxi9LFMYYY/yyRGGMMcYvSxTGGGP8skRhjDHGr/8H6toaSTFDHD8AAAAASUVORK5CYII=\n", "text/plain": ["<Figure size 432x288 with 1 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["pred = model.predict(x_test)\n", "plot_roc(pred,y_test)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Multiclass Classification Error Metrics\n", "\n", "If you want to predict more than one outcome, you will need more than one output neuron.  Because a single neuron can predict two outcomes, a neural network with two output neurons is somewhat rare.  If there are three or more outcomes, there will be three or more output neurons. The following sections will examine several metrics for evaluating classification error. The following classification neural network will be used to evaluate."]}, {"cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": ["import pandas as pd\n", "from scipy.stats import zscore\n", "\n", "# Read the data set\n", "df = pd.read_csv(\n", "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n", "    na_values=['NA','?'])\n", "\n", "# Generate dummies for job\n", "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n", "df.drop('job', axis=1, inplace=True)\n", "\n", "# Generate dummies for area\n", "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n", "df.drop('area', axis=1, inplace=True)\n", "\n", "# Missing values for income\n", "med = df['income'].median()\n", "df['income'] = df['income'].fillna(med)\n", "\n", "# Standardize ranges\n", "df['income'] = zscore(df['income'])\n", "df['aspect'] = zscore(df['aspect'])\n", "df['save_rate'] = zscore(df['save_rate'])\n", "df['age'] = zscore(df['age'])\n", "df['subscriptions'] = zscore(df['subscriptions'])\n", "\n", "# Convert to numpy - Classification\n", "x_columns = df.columns.drop('product').drop('id')\n", "x = df[x_columns].values\n", "dummies = pd.get_dummies(df['product']) # Classification\n", "products = dummies.columns\n", "y = dummies.values"]}, {"cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Train on 1500 samples, validate on 500 samples\n", "Epoch 1/1000\n", "1500/1500 - 1s - loss: 1.5458 - accuracy: 0.4753 - val_loss: 1.1331 - val_accuracy: 0.4980\n", "Epoch 2/1000\n", "1500/1500 - 0s - loss: 1.1397 - accuracy: 0.4660 - val_loss: 1.0794 - val_accuracy: 0.4980\n", "Epoch 3/1000\n", "1500/1500 - 0s - loss: 1.0356 - accuracy: 0.5453 - val_loss: 0.9845 - val_accuracy: 0.5480\n", "Epoch 4/1000\n", "1500/1500 - 0s - loss: 0.8614 - accuracy: 0.6613 - val_loss: 0.8428 - val_accuracy: 0.6820\n", "Epoch 5/1000\n", "1500/1500 - 0s - loss: 0.8293 - accuracy: 0.6633 - val_loss: 0.8141 - val_accuracy: 0.6920\n", "Epoch 6/1000\n", "1500/1500 - 0s - loss: 0.7700 - accuracy: 0.6887 - val_loss: 0.7984 - val_accuracy: 0.6740\n", "Epoch 7/1000\n", "1500/1500 - 0s - loss: 0.7455 - accuracy: 0.6973 - val_loss: 0.7968 - val_accuracy: 0.6720\n", "Epoch 8/1000\n", "1500/1500 - 0s - loss: 0.7294 - accuracy: 0.6967 - val_loss: 0.7857 - val_accuracy: 0.6820\n", "Epoch 9/1000\n", "1500/1500 - 0s - loss: 0.7247 - accuracy: 0.6953 - val_loss: 0.7723 - val_accuracy: 0.6780\n", "Epoch 10/1000\n", "1500/1500 - 0s - loss: 0.7058 - accuracy: 0.7007 - val_loss: 0.7629 - val_accuracy: 0.6820\n", "Epoch 11/1000\n", "1500/1500 - 0s - loss: 0.6941 - accuracy: 0.7053 - val_loss: 0.7684 - val_accuracy: 0.6680\n", "Epoch 12/1000\n", "1500/1500 - 0s - loss: 0.6793 - accuracy: 0.7100 - val_loss: 0.7606 - val_accuracy: 0.6840\n", "Epoch 13/1000\n", "1500/1500 - 0s - loss: 0.6808 - accuracy: 0.7053 - val_loss: 0.7546 - val_accuracy: 0.6760\n", "Epoch 14/1000\n", "1500/1500 - 0s - loss: 0.6771 - accuracy: 0.7000 - val_loss: 0.7458 - val_accuracy: 0.6920\n", "Epoch 15/1000\n", "1500/1500 - 0s - loss: 0.6644 - accuracy: 0.7147 - val_loss: 0.7473 - val_accuracy: 0.7020\n", "Epoch 16/1000\n", "1500/1500 - 0s - loss: 0.6581 - accuracy: 0.7147 - val_loss: 0.7465 - val_accuracy: 0.6980\n", "Epoch 17/1000\n", "1500/1500 - 0s - loss: 0.6513 - accuracy: 0.7200 - val_loss: 0.7407 - val_accuracy: 0.7000\n", "Epoch 18/1000\n", "1500/1500 - 0s - loss: 0.6496 - accuracy: 0.7207 - val_loss: 0.7542 - val_accuracy: 0.6760\n", "Epoch 19/1000\n", "1500/1500 - 0s - loss: 0.6435 - accuracy: 0.7220 - val_loss: 0.7453 - val_accuracy: 0.6960\n", "Epoch 20/1000\n", "1500/1500 - 0s - loss: 0.6403 - accuracy: 0.7233 - val_loss: 0.7646 - val_accuracy: 0.6920\n", "Epoch 21/1000\n", "1500/1500 - 0s - loss: 0.6323 - accuracy: 0.7307 - val_loss: 0.7517 - val_accuracy: 0.6840\n", "Epoch 22/1000\n", "Restoring model weights from the end of the best epoch.\n", "1500/1500 - 0s - loss: 0.6401 - accuracy: 0.7253 - val_loss: 0.7495 - val_accuracy: 0.6880\n", "Epoch 00022: early stopping\n"]}, {"data": {"text/plain": ["<tensorflow.python.keras.callbacks.History at 0x1a50f2e6d0>"]}, "execution_count": 9, "metadata": {}, "output_type": "execute_result"}], "source": ["# Classification neural network\n", "import numpy as np\n", "import tensorflow.keras\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense, Activation\n", "from tensorflow.keras.callbacks import EarlyStopping\n", "from sklearn.model_selection import train_test_split\n", "\n", "# Split into train/test\n", "x_train, x_test, y_train, y_test = train_test_split(    \n", "    x, y, test_size=0.25, random_state=42)\n", "\n", "model = Sequential()\n", "model.add(Dense(100, input_dim=x.shape[1], activation='relu',\n", "                kernel_initializer='random_normal'))\n", "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n", "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n", "model.add(Dense(y.shape[1],activation='softmax',\n", "                kernel_initializer='random_normal'))\n", "model.compile(loss='categorical_crossentropy', \n", "              optimizer=tensorflow.keras.optimizers.Adam(),\n", "              metrics =['accuracy'])\n", "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n", "                        verbose=1, mode='auto', restore_best_weights=True)\n", "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n", "          callbacks=[monitor],verbose=2,epochs=1000)\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Calculate Classification Accuracy\n", " \n", "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n", "\n", "$ accuracy = \\frac{c}{N} $\n", "\n", "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n", "\n", "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."]}, {"cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": ["pred = model.predict(x_test)\n", "pred = np.argmax(pred,axis=1) \n", "# raw probabilities to chosen class (highest probability)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."]}, {"cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Accuracy score: 0.7\n"]}], "source": ["from sklearn import metrics\n", "\n", "y_compare = np.argmax(y_test,axis=1) \n", "score = metrics.accuracy_score(y_compare, pred)\n", "print(\"Accuracy score: {}\".format(score))"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Calculate Classification Log Loss\n", "\n", "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n", "\n", "The following code shows the output of predict_proba:\n"]}, {"cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Numpy array of predictions\n"]}, {"data": {"text/plain": ["array([[0.    , 0.078 , 0.7665, 0.155 , 0.0005, 0.    , 0.    ],\n", "       [0.    , 0.6634, 0.3353, 0.    , 0.0012, 0.    , 0.    ],\n", "       [0.    , 0.6434, 0.3534, 0.0003, 0.0028, 0.0001, 0.    ],\n", "       [0.    , 0.2482, 0.7391, 0.0116, 0.0011, 0.    , 0.    ],\n", "       [0.    , 0.0369, 0.8035, 0.1595, 0.0001, 0.    , 0.    ]],\n", "      dtype=float32)"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["As percent probability\n", "[ 0.0001  7.8021 76.648  15.5011  0.0477  0.001   0.    ]\n", "Log loss score: 0.7407421676069498\n"]}], "source": ["from IPython.display import display\n", "\n", "# Don't display numpy in scientific notation\n", "np.set_printoptions(precision=4)\n", "np.set_printoptions(suppress=True)\n", "\n", "# Generate predictions\n", "pred = model.predict(x_test)\n", "\n", "print(\"Numpy array of predictions\")\n", "display(pred[0:5])\n", "\n", "print(\"As percent probability\")\n", "print(pred[0]*100)\n", "\n", "score = metrics.log_loss(y_test, pred)\n", "print(\"Log loss score: {}\".format(score))\n", "\n", "# raw probabilities to chosen class (highest probability)\n", "pred = np.argmax(pred,axis=1) "]}, {"cell_type": "markdown", "metadata": {}, "source": ["[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n", "\n", "$ \\mbox{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $\n", "\n", "\n", "You should use this equation only as an objective function for classifications that have two outcomes. The variable y-hat is the neural network\u2019s prediction, and the variable y is the known correct answer.  In this case, y will always be 0 or 1.  The training data have no probabilities. The neural network classifies it either into one class (1) or the other (0).  \n", "\n", "The variable N represents the number of elements in the training set the number of questions in the test.  We divide by N because this process is customary for an average.  We also begin the equation with a negative because the log function is always negative over the domain 0 to 1.  This negation allows a positive score for the training to minimize.\n", "\n", "You will notice two terms are separated by the addition (+).  Each contains a log function.  Because y will be either 0 or 1, then one of these two terms will cancel out to 0.  If y is 0, then the first term will reduce to 0.  If y is 1, then the second term will be 0.  \n", "\n", "If your prediction for the first class of a two-class prediction is y-hat, then your prediction for the second class is 1 minus y-hat.  Essentially, if your prediction for class A is 70% (0.7), then your prediction for class B is 30% (0.3).  Your score will increase by the log of your prediction for the correct class.  If the neural network had predicted 1.0 for class A, and the correct answer was A, your score would increase by log (1), which is 0. For log loss, we seek a low score, so a correct answer results in 0.  Some of these log values for a neural network's probability estimate for the correct class:\n", "\n", "* -log(1.0) = 0\n", "* -log(0.95) = 0.02\n", "* -log(0.9) = 0.05\n", "* -log(0.8) = 0.1\n", "* -log(0.5) = 0.3\n", "* -log(0.1) = 1\n", "* -log(0.01) = 2\n", "* -log(1.0e-12) = 12\n", "* -log(0.0) = negative infinity\n", "\n", "As you can see, giving a low confidence to the correct answer affects the score the most.  Because log (0) is negative infinity, we typically impose a minimum value.  Of course, the above log values are for a single training set element.  We will average the log values for the entire training set.\n", "\n", "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"]}, {"cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/Users/jheaton/miniconda3/envs/tensorflow/lib/python3.7/site-packages/ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n", "  if sys.path[0] == '':\n"]}, {"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc9Xnv8e+jfV8syZJ3G2NjjA3EyIQlCTIQAm0CzXppGkq4SWhIk5S2SZuWNGmSJm3olvbV3jbcJr1pm8ShIQs3IQRIYkgLBmw22xjjDRtbsiXL1r6Pnv4xR2awZXRGo9EZaT7v10svzZwzyzOPj0Zf//Sb3zF3FwAAAICplRN1AQAAAMBsRNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAkOHM7CUzu3oKHucaM/tByNs+YWbnpfqcAJDNCNoAkD2+JOkvQt72ryR9Po21AMCsR9AGgCxgZuslVbr75pB3uVfSBjObl8ayAGBWI2gDwAxhZoVm9hUzaw6+vmJmhQn7/8DMWoJ9HzQzN7Ozg93XSXo44baXmdkxM1sUXL/AzDrMbJUkufuApK2Srpm+VwgAswtBGwBmjjskXSLpQkkXSLpY0qclycyulfR7kq6WdLakK06571pJu8auuPujkr4q6RtmVizp3yV92t1fSLjPzuB5AACTQNAGgJnjNyR93t1b3b1N0uck3RTse4+kf3X3He7eF+xLVCWp+5RtfyqpUtITkpol/eMp+7uD+wEAJoGgDQAzx3xJBxKuHwi2je17OWFf4mVJOiGpPHGDuw9L+n+S1kj6a3f3U+5TLqkjtZIBIHsRtAFg5miWtCTh+uJgmyS1SFqYsG/RKfd9TtLKxA1mtkDSZyX9q6S/TpzvHThX0rMp1gwAWYugDQAzx7clfdrM6sysVtJnJP1HsO9uSbeY2blmVhLsS3SfEuZtm5kpPpr9NUkfUDyofyFhf6GkiyQ9mJ6XAgCzH0EbAGaOP5O0RfHR6W2Sngq2yd1/IunvJf1C0h5JjwX3GQz2PyWp08xeH2z/uKR6SX8STBm5RfGg/sZg//WSNrn72Ig5ACBJdvqUPADATGdm50raLqnQ3UeCbddI+oi7/1qI+z8u6QPuvj29lQLA7EXQBoBZwszeLunHkkolfUPSaJhQDQBID6aOAMDs8VuS2iTtlRSTdFu05QBAdmNEGwAAAEgDRrQBAACANCBoAwAAAGmQF3UByaitrfWlS5dG9vy9vb0qLS2N7PlnOvo3efQuNfQvNfRv8uhdauhfauhfarZu3XrM3etSeYwZFbSXLl2qLVu2RPb8mzZtUlNTU2TPP9PRv8mjd6mhf6mhf5NH71JD/1JD/1JjZgdSfQymjgAAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpEGnQNrNrzWyXme0xs09FWQsAAAAwlSIL2maWK+kfJV0nabWkXzez1VHVAwAAAEylKEe0L5a0x933ufuQpI2SboiwHgDAKYZGRjU86lGXAQAzUl6Ez71A0ssJ1w9Jen1EtQBAVhgddXX0D6u9Z1DHeobU3juo9p6h+PXe+Pf2niG1B5e7Bkb0vnML9OaoCweAGcjcoxmpMLN3S3qLu38wuH6TpIvd/WOn3O5WSbdKUn19/UUbN26c9lrH9PT0qKysLLLnn+no3+TRu9TM5v65uwZiUveQq2vQ1T0c/941FP/qDr7Ht8VvN967vkkqK5AqCkwVBaby4KuiwLS8dEhr5s3O/qXbbD72pgP9Sw39S82GDRu2untjKo8R5Yj2IUmLEq4vlNR86o3c/S5Jd0lSY2OjNzU1TUtx49m0aZOifP6Zjv5NHr1LzUzrn7urq39EbT0Dau0eVNvYV88ro8/xEechHesZ1ODI6LiPU1aYp5qyQtWUFmhBfaFqywpUU1qomrIC1ZQVqrY0/r2mrEDVJQXKzbFxH2em9S+T0LvU0L/U0L/oRRm0n5S0wsyWSTos6UZJ742wHgBIq4Hh2MnA3NY9+OoQHWw/Flweip0envNzTbVBMK4pLdTZc8vi1xMC88nLpQUqys+N4FUCAMZEFrTdfcTMPirpp5JyJX3d3XdEVQ8ATMboqOt435BauwbV2j3wqiCdGKDbugfVPTBy2v3NpJrSAtWWFaquvFDL60pVV16ouuB6XXmh5pYXqq6sSBXFeTIbf9QZAJB5ohzRlrvfJ+m+KGsAgPG4u7oHR9TaNaAjnYM62jWgo90DOto5oKNdgycvt3YPamScVTnKCvNOBuZzGyr0phWFpwXouvL4yHNeLucOA4DZKNKgDQBRGJvCcaRrQEe7BnQkCMynXu4bip123/KiPDVUFKm+okiXLK85eXlueaHmVsRHnmvLC1RSwNsrAGQ7fhMAmFX6h2Jq7uxXS8eAmjv7daRzQFt3Dupf9z0RD9JdA+roGz7tfgV5OUFoLtR58yt05aq5qq8oVH0QpBsqijS3opAADQAIjd8YAGaMwZGYjnQOqLljQC2d/WrpDL53DKg5uDxeiK4okJbUDWlhdYkal1arvrxI9ZWvBOj6ikJVFucz/xkAMKUI2gAygrurrWdQh07069CJfjV39Kulo1/NnfHpHC2d/TrWM3Ta/apK8jWvsljzK4u0bnGV5lcVa15lUXxbVTxMb/7vX6qp6Q0RvCoAQDYjaAOYFqOjrtbuQR060afDHf1BoO7ToRP9OnyiX4c6+jV0ynrQ5YV5mlcVD83nza/QvMpizasq0vzg+7zKIqZyAAAyFr+hAEyJ2KjrSNdAPDS/KkDHL7d0DJy2NnRNaYEWVhdr1bxyXb26XguqirWwulgLq0s0r6pIFUX5Eb0aAABSR9AGEFrv4IgOHu+Lf7X36cDxXh083q+D7b06dKL/tGXu6soLtbC6WGsXVOq6NfO0oDoepBdVF2t+VTGj0QCAWY3fcgBOGpsnfbA9HqYPBN/HLh/rGXzV7SuK8rSkplTnLajUdWvnaVF1STAiHQ/SnJkQAJDNCNpAlnF3tXUPat+xXu1P+Ho5CNSJa0ebSfMri7VoTrGuWjVXi2tKtHhOiZbUlGjJnFJVljC1AwCAMyFoA7NU18Cw9rfFQ/QrobpH+9t61ZsQpgvycrQkCM+XLa/VkiBML66Jj04X5jEqDQDAZBC0gRlsODaqA+292tOaEKSDUJ24FF6OSQurS7SstlSNS+borLpSLauNf82rLFZuDutHAwAw1QjawAwwFHM939yl3a3d2tvao93B10vHel/1AcS68kItqy3V1efWnwzSZ9WVatGcEkamAQCYZgRtIIP0DI6cDNJ7Wnu0p7Vbu1t7dLC9T/7gLyXFR6eX1pTq7LllumZ1vVbUl2l5XZmW1ZaqnOXwAADIGARtIALDsVHta+vVC0e69MKRbr3Q0qUXj/bocEf/ydvk55rOqi3TmgWVurBqWG9+/RqtmFuupbWMTgMAMBMQtIE0co+fDXFnSzxQ7zrSrZ0tXdrb1qPhWHzKR36uaXldmdYvrdZ76xfr7LllWjG3TIvnlCgvN0eStGnTJjWdPz/KlwIAAJJE0AamyNDIqHa3dmvH4S4939J1crS6o2/45G3mVRbpnIZyNZ0zV+fOK9c5DeU6q7ZMBXk5EVYOAADSgaANTMLAcEy7jnRre3Onth/u1PbDXdp1pPvkKcZLCnK1sr5c161p0KqGCp3TUK5VDeWqKimIuHIAADBdCNrABHoHR7SzpSseqJvj33e39igWrPZRWZyvNQsqdMvlS3XegkqtmV+hpTWlymHJPAAAshpBG0gQG3Xtbu3WMwc79MzL8a8Xj3ZrbAW9mtICrVlQqavOnau1Cyp13vxKLawulhmhGgAAvBpBG1ntaNeAnj4Zqk9o26HOk2dNrCzO1wWLqnTN6nqtXViltQsqVV9RSKgGAAChELSRNUZio3q+pUtPvnRCWw8c19MHO9TSOSApvvLH6nkVeudFC3XhoipduKhKy2pLCdUAAGDSCNqYtXoHR/TMyx168qXj2vLSCT118IT6gtHqhdXFWr90TjxUL67S6nkVKspnbWoAADB1CNqYNU70Dunx/e168qUT2vLScW1v7lJs1GUmndtQoXdftFDrl81R45I5aqgsirpcAAAwyxG0MWN1Dwzrif3H9ejedj26t107W7okSYV5OXrd4ip9pGm5GpfO0esWV6mCU5MDAIBpRtDGjNE/FNPWAyf06N5jenRvu7Yd7lRs1FWYl6PGpdX6xDUrdenyGq1dUMUJYAAAQOQI2shY7q5dR7v18K42Pfxim7a8dEJDsVHl5ZguWBQfsb50eY3WLa5mfjUAAMg4BG1klM7+Yf33nmMnw/WRrviqIKsaynXzZUt02dm1Wr90jsoKOXQBAEBmI60gUu6uF4/26KGdR7VpV6ueOtih2KirvDBPb1hRq6Zz6vSmlXWaV1kcdakAAABJIWhj2sVGXVsPnNADO47owZ1HdaC9T5J03vwKffiKs3TFyrl63eIq5ecyzxoAAMxcBG1Mi/6hmJ5uHdGP//NZ/eyFVh3vHVJBbo4uXV6jD73xLL15db3qK1hyDwAAzB4EbaTNwHBMm3a16UfPNetnO1vVPxxTedERXblqrt68ul5XrKxTOcvuAQCAWYqgjSk1NDKq/9rTph8926IHnj+qnsERzSkt0NvXLdCCWKs+9GsbWHoPAABkBYI2UubuemL/cX3/6cP6yfYj6uwfVkVRnn5lbYPeev58Xba8Rnm5Odq0aRMhGwAAZA2CNibt5eN9+t5Th3XPU4d08HifSgpy9ZbzGvTW8+fpjSvqCNUAACCrEbSRlL6hEf1k2xF9d+shPbavXZJ02fIa3X71Cl27pkElBRxSAAAAEkEbIe1p7dZ/bD6oe7YeUvfgiBbPKdHvvXml3v66BVo0pyTq8gAAADIOQRtnNDQyqgeeP6L/2HxAm/cdV0Fujq5b26D3XrxYFy+bIzOLukQAAICMRdDGadp7BvVvjx3Qt544qLbuQS2sLtYfXrtK725cqNqywqjLAwAAmBEI2jhpb1uPvvZf+3XP1kMaHBnVlavm6qZLluhNK+uUm8PoNQAAQDII2tCTLx3XXY/s00M7jyo/N0fvXLdQH3jDMp09tyzq0gAAAGYsgnYWe2xvu77y0It6fP9xVZfk62NXrtBvXrqE6SEAAABTgKCdZdxdj+1r11ce2q0n9h/X3PJCffZtq3Xj+sUqLsiNujwAAIBZg6CdRbYeOK4v37/rZMD+07et1o0XL1ZRPgEbAABgqkUStM3sLyW9TdKQpL2SbnH3jihqyQb7j/Xqzvtf0E+2H1FdeaE+d/15+l/rFxGwAQAA0iiqEe0HJf2Ru4+Y2Zcl/ZGkP4yollmrvWdQf/+z3frm4wdVkJej3716pT70pmWcvREAAGAaRJK43P2BhKubJb0rijpmq9io65uPH9Bf/nSX+oZiunH9It1+9UrVlfMhRwAAgOmSCUOb/1vSd6IuYrZ4+uAJ/ckPt2v74S5dfnaNPnf9eTp7bnnUZQEAAGQdc/f0PLDZQ5Iaxtl1h7v/MLjNHZIaJb3Dz1CImd0q6VZJqq+vv2jjxo1pqTeMnp4elZVl5trSfcOuu18c0sMvj6iy0PTrqwp0cUNuRp0mPZP7l+noXWroX2ro3+TRu9TQv9TQv9Rs2LBhq7s3pvIYaQvaEz6x2c2SPizpKnfvC3OfxsZG37JlS3oLew2bNm1SU1NTZM9/Jg+/2KZP3fOcjnYN6JbLl+n2q1eovCg/6rJOk6n9mwnoXWroX2ro3+TRu9TQv9TQv9SYWcpBO6pVR65V/MOPV4QN2Thd98Cwvvjjndr45Ms6e26ZvveRy3XhoqqoywIAAICim6P9D5IKJT0YTG3Y7O4fjqiWGWnrgRP6+LefVktnvz58xXLdfvUKlusDAADIIFGtOnJ2FM87G4yOur76yD791QO7NL+qSN+97TKtW1wddVkAAAA4RSasOoKQ2nsG9bt3P6tHXmzTr54/T3/+jrWqyMC52AAAACBozxg7mjv1oW9s0bHeIX3x7Wv03osXZ9SKIgAAAHg1gvYMcN+2Fv3+3c+qqiRf93z4Mq1dWBl1SQAAAJgAQTuDubv+7me79ZWHdmvd4ir9800XaW55UdRlAQAAIASCdoaKjbo+/YPt+vYTB/XOdQv1pXesUWEeq4oAAADMFATtDDQwHNPtG5/R/TuO6Lc3LNcnrjmH+dgAAAAzDEE7w/QPxfSBbzypR/e260/eulofeMOyqEsCAADAJBC0M8jAcDxkb97Xrr95zwV6x7qFUZcEAACASSJoZ4iB4Zg+9G9b9Ni+dv31uwnZAAAAM11O1AVAGomN6qPfekq/3H1MX37n+YRsAACAWYCgHTF312fu3aGHdrbqCzecp/c0Loq6JAAAAEwBgnbE/unhvfrW4wd1W9Ny3XTp0qjLAQAAwBQhaEfovm0tuvP+Xbrhwvn65DXnRF0OAAAAphBBOyJ7Wrv1if98VusWV+nOd52vnBzWyQYAAJhNCNoR6B4Y1q3/vlUlBXn6p/ddxBkfAQAAZiGW95tm7q5PfW+bDrT36ZsffL3qK4qiLgkAAABpwIj2NPv+04f14+da9PvXrNQlZ9VEXQ4AAADShKA9jQ539OuzP9yh9Uur9VtvWh51OQAAAEgjgvY0cXd94u5nNequv3nPhcrlw48AAACzGkF7mtzz1GE9tq9dd/zqai2aUxJ1OQAAAEgzgvY06Ogb0pfu26l1i6t043rO/AgAAJANCNrT4Mv371Jn/7C++Pa1rJcNAACQJQjaabazpUsbnzyomy9dqnPnVURdDgAAAKYJQTvN7rz/BZUX5ul3rloRdSkAAACYRgTtNNq8r12/2NWmj2w4W5Ul+VGXAwAAgGlE0E4Td9ed97+ghooivf+ypVGXAwAAgGk2YdA2s4+aWfV0FDObbN53XE8d7NBvb1iuovzcqMsBAADANAszot0g6Ukzu9vMrjUzls0I4Z8e3qvasgK9u5Hl/AAAALLRhEHb3T8taYWkr0l6v6TdZvYlM+Mc4mewo7lTj7zYplsuX8ZoNgAAQJYKNUfb3V3SkeBrRFK1pO+a2Z1prG3G+pdf7ldZYZ7ed8mSqEsBAABARPImuoGZfVzSzZKOSfoXSZ9092Ezy5G0W9IfpLfEmeVE75B+vK1FN65fpMpiVhoBAADIVhMGbUm1kt7h7gcSN7r7qJm9NT1lzVz3PHVIQyOjeu/rF0ddCgAAACI0YdB298+8xr6dU1vOzObu+tYTB7VucZVWNXAWSAAAgGzGOtpTaMuBE9rX1qv3vp652QAAANmOoD2F7n2mWUX5ObpuTUPUpQAAACBiBO0pMhIb1X3bWnTVufUqLQwz9R0AAACzGUF7ijy2r13tvUN62/nzoy4FAAAAGYCgPUV+9GyLygrz1HROXdSlAAAAIAMQtKfA6KjrZy+0qumcOs4ECQAAAEkE7Smxo7lLx3oGdeWquVGXAgAAgAxB0J4Cv9jVKjPpipVMGwEAAEAcQXsK/GJXqy5YWKWassKoSwEAAECGIGinqLNvWM+83MFoNgAAAF4l0qBtZp8wMzez2ijrSMWWA8flLl26vCbqUgAAAJBBIgvaZrZI0pslHYyqhqnwxP7jKsjN0YWLqqIuBQAAABkkyhHtv5X0B5I8whpS9sRLx3X+wkqW9QMAAMCrRBK0zex6SYfd/dkonn+q9A2NaNuhTl28bE7UpQAAACDDmHt6BpTN7CFJDePsukPSH0u6xt07zewlSY3ufuwMj3OrpFslqb6+/qKNGzempd4wenp6VFZWdvL6zvaYvvzkgH73okJdUJcXWV0zxan9Q3j0LjX0LzX0b/LoXWroX2roX2o2bNiw1d0bU3mMtKVDd796vO1mtlbSMknPmpkkLZT0lJld7O5HxnmcuyTdJUmNjY3e1NSUrpIntGnTJiU+/+5H9knaqfdd90aW9gvh1P4hPHqXGvqXGvo3efQuNfQvNfQvetM+DOvu2ySdPIXiRCPamWzb4U7NrywiZAMAAOA0rKOdgu3NnVqzoDLqMgAAAJCBIg/a7r50Jo5m9wyOaP+xXoI2AAAAxhV50J6pnm/ukru0lqANAACAcRC0J2nX0W5J0qp55RFXAgAAgExE0J6kva09Ki3IVUNFUdSlAAAAIAMRtCdpb1uPls8tU7BEIQAAAPAqBO1J2tvao+V1LAIPAACA8RG0J6F3cETNnQNaXlcadSkAAADIUATtSdh/rFeSGNEGAADAGRG0J2EsaC+tZUQbAAAA4yNoT8KhE/2SpIXVxRFXAgAAgExF0J6Ewx19qizOV3lRftSlAAAAIEMRtCfh8Il+LahiNBsAAABnRtCehMMd/VrAtBEAAAC8BoJ2ktydEW0AAABMiKCdpM7+YfUOxfggJAAAAF4TQTtJzR0DkqT5jGgDAADgNRC0k3SsZ1CSVFdeGHElAAAAyGQE7SS1dceDdm0ZQRsAAABnRtBO0tiIdm1ZQcSVAAAAIJMRtJN0rGdQRfk5KivMi7oUAAAAZDCCdpKO9QyptqxQZhZ1KQAAAMhgBO0kHesZZH42AAAAJkTQTlJbN0EbAAAAEyNoJyk+dYQPQgIAAOC1EbST4O7q6h9WZUl+1KUAAAAgwxG0kzA8Kg3FRlVZTNAGAADAayNoJ6F32CWJoA0AAIAJEbST0DcS/15RRNAGAADAayNoJ6GPEW0AAACERNBOwtjUkQqCNgAAACZA0E7C2NQRRrQBAAAwEYJ2EsamjlQU5UVcCQAAADIdQTsJTB0BAABAWATtJPSPuIrzc5WfS9sAAADw2kiMSRiMSSUFuVGXAQAAgBmAoJ2EwZhUTNAGAABACATtJAzGnBFtAAAAhELQTkJ8RJsVRwAAADAxgnYShmKuknxGtAEAADAxgnYSmKMNAACAsAjaSRiMOUEbAAAAoRC0kzAUE1NHAAAAEApBOwmsOgIAAICwCNpJGBxh1REAAACEE1nQNrOPmdkuM9thZndGVUdYI7FRjbhUzNQRAAAAhBDJ8KyZbZB0g6Tz3X3QzOZGUUcy+odjkqTiAv4IAAAAgIlFlRpvk/QX7j4oSe7eGlEdoQ2NjEqSCvMY0QYAAMDEograKyW90cweN7OHzWx9RHWENhxzSVJ+LiPaAAAAmJi5e3oe2OwhSQ3j7LpD0hcl/VzS70haL+k7ks7ycYoxs1sl3SpJ9fX1F23cuDEt9U6krW9Un3ykXx9cW6A3LMiPpIaZrqenR2VlZVGXMSPRu9TQv9TQv8mjd6mhf6mhf6nZsGHDVndvTOUx0jZH292vPtM+M7tN0veCYP2EmY1KqpXUNs7j3CXpLklqbGz0pqam9BQ8gT2tPdIjD2vteavVdOGCSGqY6TZt2qSo/v1mOnqXGvqXGvo3efQuNfQvNfQvelHNg/iBpCslycxWSiqQdCyiWkIZjsXnaBcwdQQAAAAhRLUo9Nclfd3MtksaknTzeNNGMsnJoJ1H0AYAAMDEIgna7j4k6X1RPPdkja06wochAQAAEAapMaShGEEbAAAA4ZEaQxpb3o+pIwAAAAiD1BjS8AgfhgQAAEB4pMaQTk4dybOIKwEAAMBMQNAOieX9AAAAkAxSY0isOgIAAIBkkBpDGmIdbQAAACSB1BgSH4YEAABAMkiNIY0t75eXy4chAQAAMDGCdkgjo0HQzqFlAAAAmBipMaRRjwdtcjYAAADCIDaGFAtGtHONqSMAAACYGEE7pJNBO4egDQAAgIkRtEMadZdJMka0AQAAEAJBO6TYqIvBbAAAAIRF0A4p5i4GswEAABAWQTukUUa0AQAAkASCdkixUZoFAACA8MiOIY06I9oAAAAIj6AdEh+GBAAAQDII2iHFGNEGAABAEgjaIcU/DEnSBgAAQDgE7ZCYOgIAAIBkELRDigVnhgQAAADCIGiHxDraAAAASAZBO6SYi6ANAACA0AjaITGiDQAAgGQQtEPiw5AAAABIBkE7pPg62iRtAAAAhEPQDompIwAAAEgGQTukmDvNAgAAQGhkx5Bioy5mjgAAACAsgnZIzvJ+AAAASAJBO6RRzgwJAACAJBC0Q3IXU0cAAAAQGkE7JJdHXQIAAABmEIJ2EhjQBgAAQFgE7ZCcAW0AAAAkgaAdEjkbAAAAySBoJ4EPQwIAACAsgnZYDGkDAAAgCQTtkFysow0AAIDwCNoh8WFIAAAAJCOSoG1mF5rZZjN7xsy2mNnFUdSRLOZoAwAAIKyoRrTvlPQ5d79Q0meC6xmNAW0AAAAkI6qg7ZIqgsuVkpojqiM0d+ZoAwAAILy8iJ73dkk/NbO/UjzsXxZRHQAAAEBamKfpU35m9pCkhnF23SHpKkkPu/s9ZvYeSbe6+9VneJxbJd0aXD1H0q501BtSraRjET7/TEf/Jo/epYb+pYb+TR69Sw39Sw39S8057l6eygOkLWi/5pOadUqqcnc3M5PU6e4VE90vama2xd0bo65jpqJ/k0fvUkP/UkP/Jo/epYb+pYb+pWYq+hfVHO1mSVcEl6+UtDuiOgAAAIC0iGqO9ock/Z2Z5Uka0CtTQwAAAIBZIZKg7e7/JemiKJ47RXdFXcAMR/8mj96lhv6lhv5NHr1LDf1LDf1LTcr9i2SONgAAADDbcQp2AAAAIA0I2pLM7Foz22Vme8zsU+PsLzSz7wT7HzezpQn7/ijYvsvM3jKddWeKEP37PTN73syeM7OfmdmShH0xM3sm+Lp3eivPDCH6934za0vo0wcT9t1sZruDr5unt/Lohejd3yb07UUz60jYx7Fn9nUzazWz7WfYb2b290F/nzOzdQn7sv3Ym6h3vxH07Dkze9TMLsAyYTcAAAZQSURBVEjY95KZbQuOvS3TV3XmCNG/JjPrTPgZ/UzCvtf8uc8GIfr3yYTebQ/e7+YE+7L6+DOzRWb2CzPbaWY7zOx3xrnN1L33uXtWf0nKlbRX0lmSCiQ9K2n1Kbf5iKR/Di7fKOk7weXVwe0LJS0LHic36teUgf3bIKkkuHzbWP+C6z1Rv4YZ0L/3S/qHce47R9K+4Ht1cLk66teUSb075fYfk/T1hOtZfewFPXiTpHWStp9h/69I+okkk3SJpMeD7Vl97IXs3WVjPZF03VjvgusvSaqN+jVkeP+aJP1onO1J/dzP1q+J+nfKbd8m6ecJ17P6+JM0T9K64HK5pBfH+b07Ze99jGhLF0va4+773H1I0kZJN5xymxskfSO4/F1JV5mZBds3uvugu++XtCd4vGwyYf/c/Rfu3hdc3Sxp4TTXmMnCHH9n8hZJD7r7cXc/IelBSdemqc5MlGzvfl3St6elshnC3R+RdPw1bnKDpH/zuM2Sqsxsnjj2Juyduz8a9Ebife80IY69M0nlPXPWSLJ/vPclcPcWd38quNwtaaekBafcbMre+wja8ea+nHD9kE5v+MnbuPuIpE5JNSHvO9sl24MPKP6/xDFFZrbFzDab2a+lo8AMF7Z/7wz+fPVdM1uU5H1nq9CvP5iutEzSzxM2Z/uxF8aZepztx16yTn3fc0kPmNlWi5/9GOO71MyeNbOfmNl5wTaOvSSYWYniQfCehM0cfwGLTwV+naTHT9k1Ze99Ua2jnUlsnG2nLsVyptuEue9sF7oHZvY+SY165WRFkrTY3ZvN7CxJPzezbe6+Nw11Zqow/fv/kr7t7oNm9mHF/7pyZcj7zmbJvP4bJX3X3WMJ27L92AuD974UmdkGxYP2GxI2Xx4ce3MlPWhmLwQjlHjFU5KWuHuPmf2KpB9IWiGOvWS9TdJ/u3vi6DfHnyQzK1P8PyC3u3vXqbvHucuk3vsY0Y7/b2RRwvWFip+5ctzbWPwkO5WK/8kmzH1nu1A9MLOrJd0h6Xp3Hxzb7u7Nwfd9kjYp/j/LbDJh/9y9PaFn/1evrEGf7cdfMq//Rp3yp1OOvVDO1ONsP/ZCMbPzJf2LpBvcvX1se8Kx1yrp+8q+KYcTcvcud+8JLt8nKd/MasWxl6zXeu/L2uPPzPIVD9nfdPfvjXOTKXvvI2hLT0paYWbLzKxA8YPy1BUI7pU09snSdyn+oQIPtt9o8VVJlin+v+0npqnuTDFh/8zsdZK+qnjIbk3YXm1mhcHlWkmXS3p+2irPDGH6Ny/h6vWKzyeTpJ9KuiboY7Wka4Jt2SLMz67M7BzFP7TyWMI2jr1w7pX0m8En8C+R1OnuLeLYm5CZLZb0PUk3ufuLCdtLzax87LLivRt35YhsZmYNwWehZGYXK55X2hXy5x6SmVUq/hfkHyZsy/rjLziuviZpp7v/zRluNmXvfVk/dcTdR8zso4o3KlfxVQl2mNnnJW1x93sV/wf5dzPbo/hI9o3BfXeY2d2K/4IekfTbp/xpetYL2b+/lFQm6T+D982D7n69pHMlfdXMRhV/E/0Ld8+qsBOyfx83s+sVP8aOK74Kidz9uJl9QfFfPJL0+VP+PDirheydFP8g0MbgP8djsv7YkyQz+7biqzvUmtkhSZ+VlC9J7v7Pku5T/NP3eyT1Sbol2JfVx54UqnefUfyzPP8neN8bcfdGSfWSvh9sy5P0LXe/f9pfQMRC9O9dkm4zsxFJ/ZJuDH6Gx/25j+AlRCpE/yTp7ZIecPfehLty/MUHVm6StM3Mngm2/bGkxdLUv/dxZkgAAAAgDZg6AgAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGgFnIzNab2XNmVhScDW6Hma2Jui4AyCacsAYAZikz+zNJRZKKJR1y9z+PuCQAyCoEbQCYpcysQPFTBQ9IuszdYxGXBABZhakjADB7zZFUJqlc8ZFtAMA0YkQbAGYpM7tX0kZJyyTNc/ePRlwSAGSVvKgLAABMPTP7TUkj7v4tM8uV9KiZXenuP4+6NgDIFoxoAwAAAGnAHG0AAAAgDQjaAAAAQBoQtAEAAIA0IGgDAAAAaUDQBgAAANKAoA0AAACkAUEbAAAASAOCNgAAAJAG/wNvAXy94P6E/wAAAABJRU5ErkJggg==\n", "text/plain": ["<Figure size 864x720 with 1 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["%matplotlib inline\n", "from matplotlib.pyplot import figure, show\n", "from numpy import arange, sin, pi\n", "\n", "#t = arange(1e-5, 5.0, 0.00001)\n", "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n", "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n", "\n", "fig = figure(1,figsize=(12, 10))\n", "\n", "ax1 = fig.add_subplot(211)\n", "ax1.plot(t, np.log(t))\n", "ax1.grid(True)\n", "ax1.set_ylim((-8, 1.5))\n", "ax1.set_xlim((-0.1, 2))\n", "ax1.set_xlabel('x')\n", "ax1.set_ylabel('y')\n", "ax1.set_title('log(x)')\n", "\n", "show()"]}, {"cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Confusion matrix, without normalization\n", "[[ 21   1   0   0   0   0   0]\n", " [  5 183  61   0   0   0   0]\n", " [  0  44 142   1   0   0   0]\n", " [  0   0  10   4   0   0   0]\n", " [  0   8   0   0   0   0   0]\n", " [ 10   7   0   0   0   0   0]\n", " [  3   0   0   0   0   0   0]]\n", "Normalized confusion matrix\n", "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n", " [0.02 0.73 0.24 0.   0.   0.   0.  ]\n", " [0.   0.24 0.76 0.01 0.   0.   0.  ]\n", " [0.   0.   0.71 0.29 0.   0.   0.  ]\n", " [0.   1.   0.   0.   0.   0.   0.  ]\n", " [0.59 0.41 0.   0.   0.   0.   0.  ]\n", " [1.   0.   0.   0.   0.   0.   0.  ]]\n"]}, {"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEmCAYAAAB/OxvoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZn/8c+3E7YQBEIA2UKACSgyEkjEEURAkQFFFpUlIqKiERTc/cnmiAujP1F03NAwIIgOBsUoAiMiv0GEHyhJCDEZ1mAyBCJkQdYYkvDMH/cUFpWb7tvddbvurXzfvO6rq27dPufpkDx96qlzzlVEYGZm9dXT6QDMzGxwnMjNzGrOidzMrOacyM3Mas6J3Mys5pzIzcxqzoncBkTSRpJ+JekJST8dRDsnSPpNO2PrFEn7S7q303HYukeeR97dJL0D+DjwMuApYBZwXkTcMsh2TwROB/aNiFWDDrTiJAUwLiIe6HQsZq08Iu9ikj4OfAP4V2BrYAzwXeDINjS/I3DfupDEi5A0vNMx2DosInx04QFsCjwNHNPLNRuQJfpH0vENYIP02oHAQuATwGPAIuA96bXPAc8BK1MfJwPnAj9qanssEMDw9PzdwINk7wr+DJzQdP6Wpu/bF7gDeCJ93bfptZuALwC3pnZ+A4xey8/WiP//NMV/FPAm4D5gGXBW0/X7ALcBf03XfhtYP712c/pZnkk/73FN7X8a+AtweeNc+p5dUh97p+fbAkuAAzv9d8NH9x0ekXev1wAbAtN6ueZs4J+A8cCeZMnsnKbXX0r2C2E7smT9HUmbR8RnyUb5UyNiZERc3FsgkjYGvgkcFhGbkCXrWTnXjQKuTdduAVwAXCtpi6bL3gG8B9gKWB/4ZC9dv5Tsz2A74F+Ai4B3AhOA/YF/kbRzunY18DFgNNmf3RuADwJExOvSNXumn3dqU/ujyN6dTG7uOCLmkSX5H0saAfwAuDQibuolXrMBcSLvXlsAS6L30scJwOcj4rGIWEw20j6x6fWV6fWVEXEd2Wh0twHG8zywh6SNImJRRMzNuebNwP0RcXlErIqIK4B7gLc0XfODiLgvIpYDV5L9ElqblWSfB6wEfkKWpP8tIp5K/c8FXgkQETMi4vbU73zg+8ABBX6mz0bEihTPi0TERcD9wB+Abch+cZq1nRN591oKjO6jdrstsKDp+YJ07oU2Wn4RPAuM7G8gEfEMWTniFGCRpGslvaxAPI2Ytmt6/pd+xLM0Ilanx41E+2jT68sb3y9pV0nXSPqLpCfJ3nGM7qVtgMUR8bc+rrkI2AP4VkSs6ONaswFxIu9etwF/I6sLr80jZGWBhjHp3EA8A4xoev7S5hcj4vqIeCPZyPQesgTXVzyNmB4eYEz9cSFZXOMi4iXAWYD6+J5ep3xJGkn2ucPFwLmpdGTWdk7kXSoiniCrC39H0lGSRkhaT9Jhkr6SLrsCOEfSlpJGp+t/NMAuZwGvkzRG0qbAmY0XJG0t6YhUK19BVqJZndPGdcCukt4habik44DdgWsGGFN/bAI8CTyd3i2c2vL6o8DOa3xX7/4NmBER7yOr/X9v0FGa5XAi72IRcQHZHPJzgMXAQ8BpwC/SJV8EpgOzgT8BM9O5gfR1AzA1tTWDFyffHrLZL4+QzeQ4gPRBYksbS4HD07VLyWacHB4RSwYSUz99kuyD1KfI3i1MbXn9XOAySX+VdGxfjUk6EjiUrJwE2f+HvSWd0LaIzRIvCDIzqzmPyM3Mas6J3Mys5pzIzcxqrtRELukXkmZImitpct/fYWZm/VXqh52SRkXEMkkbke2bcUCamdB8zWTS8uYRIzaesMu4gS4cLNfwYX1NKe6MakZlNnQWLJjPkiVL2vpPYdhLdoxYtcZi3VyxfPH1EXFoO/vvr7J3bPuwpKPT4x2AcWTTyl4QEVOAKQCvHD8hfnXjrSWHNDBbvWSDToeQS3Iqt3Xbfq+e2PY2Y9VyNtitz1mmAPxt1nf6WgFcutISuaQDgYOB10TEs5JuItvAyMys4gSqz0eIZY7INwUeT0n8ZWS77JmZVZ+AGr3bLTOR/xo4RdJs4F7g9hL7MjNrL4/IIe30dlhZ7ZuZlUfQM6zTQRTm21OZmeWpUWmlPu8dzMyGishKK0WOvpqSLpH0mKQ5TeemSpqVjvmSZqXzYyUtb3qt0I6ZHpGbma1B7RyRX0p2D9gfNk5ExHEv9CR9jewetQ3zIqK3O1+twYnczCxPmz7sjIibJY3N7SJbCHIs8PrB9OHSiplZHqnYMTj7A49GxP1N53aSdKek30nav0gjHpGbma2hXwuCRkua3vR8SlqxXsQksjt1NSwCxkTEUkkTgF9IekVEPNlbI07kZmat+rcgaElE9HufgHRj9LcCExrn0rTtFenxDEnzgF3J7uS1Vk7kZmZ5yl8QdDBwT0QsfKFLaUtgWUSslrQz2f5UD/bVkGvkZmZrEAwbVuzoqyXpCuA2YDdJCyWdnF46nheXVQBeB8yWdBfwM+CUiFjWVx8ekZuZtWrMI2+DiJi0lvPvzjl3FXBVf/twIjczy1OjlZ1O5GZma/A2tmZm9ecR+cAMHyZGbbx+p8PINWqf0zsdQq7Lf3BWp0NYq8P32LbTIZgNnEfkZmY11p5Vm0PGidzMLI9H5GZmNecRuZlZnXnWiplZvQnf6s3MrN48Ijczqz/XyM3Mas4jcjOzmvOI3MysxuQauZlZ/XlEbmZWb3IiNzOrr+yWnU7kZmb1JaEeJ3Izs1rziNzMrObqlMhLm18jaaykOWW1b2ZWJkmFjirwiNzMrJXSURNlz3gfLukySbMl/UzSiJL7MzMbNFFsNF6VEXnZiXw3YEpEvBJ4Evhg6wWSJkuaLmn6ksWLSw7HzKyYdiVySZdIeqy51CzpXEkPS5qVjjc1vXampAck3Svpn4vEWnYifygibk2PfwS8tvWCiJgSERMjYuLoLbcsORwzs2LaOCK/FDg05/zXI2J8Oq5Lfe4OHA+8In3PdyX1uTF62Yk8+nhuZlZJ7UrkEXEzsKxgt0cCP4mIFRHxZ+ABYJ++vqnsRD5G0mvS40nALSX3Z2Y2eOrHAaMb5eF0TC7Yy2np88NLJG2ezm0HPNR0zcJ0rldlJ/K7gZMkzQZGAReW3J+Z2aAJ0dPTU+gAljTKw+mYUqCLC4FdgPHAIuBrL3S9pj4rGaVNP4yI+cDuZbVvZlamMmekRMSjTf1cBFyTni4Edmi6dHvgkb7aq8+Gu2ZmQ6l4aaX/TUvbND09GmjMaLkaOF7SBpJ2AsYBf+yrPS8IMjNrpfaNyCVdARxIVktfCHwWOFDSeLKyyXzgAwARMVfSlcB/A6uAD0XE6r76cCI3M8vRrkQeEZNyTl/cy/XnAef1pw8ncjOzHFVZtVmEE7mZWYvGEv26cCI3M8tTnzzuRG5mtoY2ftg5FJzIzcxyOJGbmdWc79lpZlZzHpGbmdVYlW4aUYQTuZlZDidyM7OacyIfIAHrDa/mPl43/eyLnQ4h19suuKnTIazVm79xZKdDWKs6/SO1DqnRX5FKJXIzs6qo0y97J3Izs1ZeEGRmVm8CapTHncjNzNYkerwgyMys3lxaMTOrM7m0YmZWawKXVszM6s4jcjOzmnON3MyszlwjNzOrt2weeX0yuRO5mdka6rWNbTV3qDIz6zCp2NF3O7pE0mOS5jSdO1/SPZJmS5omabN0fqyk5ZJmpeN7RWJ1Ijcza6Vs+mGRo4BLgUNbzt0A7BERrwTuA85sem1eRIxPxylFOnAiNzNr0aiRFzn6EhE3A8tazv0mIlalp7cD2w8mXidyM7Mc/SitjJY0vemY3M+u3gv8Z9PznSTdKel3kvYv0oA/7DQzy9GPDzuXRMTEAfZxNrAK+HE6tQgYExFLJU0AfiHpFRHxZG/tlD4il/SuVNC/S9LlZfdnZtYO7fqwc+3t6yTgcOCEiAiAiFgREUvT4xnAPGDXvtoqdUQu6RXA2cB+EbFE0qgy+zMza4uSbywh6VDg08ABEfFs0/ktgWURsVrSzsA44MG+2iu7tPJ64GcRsQQgIpa1XpDqSZMBdhgzpuRwzMz61s4bS0i6AjiQrJa+EPgs2SyVDYAb0i+M29MMldcBn5e0ClgNnJKXN1uVncgFRG8XRMQUYArAhAkTe73WzGxotG9BUERMyjl98VquvQq4qr99lF0jvxE4VtIWAC6tmFldlF0jb6dSR+QRMVfSecDvJK0G7gTeXWafZmaDJu9H/iIRcRlwWdn9mJm1izfNMjPrAk7kZmY1V6M87kRuZpbHI3Izszqr0IyUIpzIzcxaqGY3lnAiNzPLUaM87kRuZpanp0aZ3InczCxHjfK4E7mZWSsJhnllp5lZvXXFh52SXtLbN/Z1xwozszqrUR7vdUQ+l2wL2uYfp/E8AG8ebmZdSWRTEOtirYk8InYYykDMzKqkRiXyYvuRSzpe0lnp8fbppqBmZt1J2YKgIkcV9Plhp6RvA+uR3YLoX4Fnge8Bryo3tGrZc8fNOh1Crru+cninQ1ir1c9X94ZPw4dV4x+gVVdFcnQhRWat7BsRe0u6E7L7bkpav+S4zMw6RnTfgqCVknpI995Mt217vtSozMw6rEZ5vFCN/DtkNwPdUtLngFuA/1tqVGZmHdZVNfKI+KGkGcDB6dQxETGn3LDMzDqnbis7C81aAYYBK4Hn+vE9Zma1pYJHn+1Il0h6TNKcpnOjJN0g6f70dfN0XpK+KekBSbMl7V0k1j6TsqSzgSuAbYHtgf+QdGaRxs3M6qqNpZVLgUNbzp0B3BgR44Ab03OAw4Bx6ZgMXFikgyIfdr4TmBARzwJIOg+YAXypSAdmZnWTzVppT1sRcbOksS2njwQOTI8vA24CPp3O/zAiArhd0maStomIRb31UaRMsoAXJ/zhwIMFvs/MrJ7KXxC0dSM5p69bpfPbAQ81XbcwnetVb5tmfZ1syuGzwFxJ16fnh5DNXDEz61r9yNGjJU1vej4lIqYMtNucc32urOuttNIozM8Frm06f3s/gjIzq6V+jLaXRMTEfjb/aKNkImkb4LF0fiHQvM/V9sAjfTXW26ZZF/czMDOzrtDOGvlaXA2cBHw5ff1l0/nTJP0EeDXwRF/1cSi218ouwHnA7sCGjfMRsWu/Qzczq4l2LfaRdAXZB5ujJS0EPkuWwK+UdDLwP8Ax6fLrgDcBD5CVtd9TpI8is1YuBb4IfJVsasx78BJ9M+tiEgxrUyKPiElreekNOdcG8KH+9lFk1sqIiLg+dTIvIs4BDupvR2ZmdSIVO6qgyIh8hbL3GPMknQI8zN+nypiZdaWq7KNSRJFE/jFgJPBhslr5psB7+9uRpHOBpyPiq/39XjOzoVajPF5o06w/pIdPASeWG46ZWecJdcd+5JKm0ctE9Ih4a1+Np31a3kW2Umkx2dJ+M7Nqq1D9u4jeRuTfHkzD6b6exwN7pX5mkpPIJU0m2xyGHcaMGUyXZmZt0xU18oi4cZBt7w9Ma9ps6+q19DMFmAIwYcLE6t7k0czWKXXar7vIh52D4cRsZrUj6jUiL/OXzs3A0ZI2krQJ8JYS+zIza6seFTuqoPCIXNIGEbGi6PURMVPSVGAW2Va4vx9AfGZmQ65ut3orstfKPsDFZPPHx0jaE3hfRJze1/dGxHlkc8/NzGqlRnm8UGnlm8DhwFKAiLgLL9E3sy7XbUv0eyJiQUvhf3VJ8ZiZdVy2jW1FsnQBRRL5Q6m8EpKGAacD95UblplZZ3Xb9MNTycorY4BHgd+mc2ZmXatGA/JCe608RrZC08xsnSB1yV4rDZIuImdhT0RMLiUiM7MKqFEeL1Ra+W3T4w2Bo8k2wTIz61p1mn5YpLQytfm5pMuBG0qLyMysw0SXLQjKsROwY7sDMTOrjAotvy+iSI38cf5eI+8BlgFnlBmUmVmnifpk8l4TebpX555k9+kEeD7d5dnMrGtlC4I6HUVxvc55T0l7WkSsToeTuJmtE7pt98M/Sto7ImaWHo2ZWUW0az9ySbsBzZNGdgb+BdgMeD/ZbTABzoqI6wbSR2/37BweEauA1wLvlzQPeIbsXUdExN4D6bCulj9Xze1lNt6g7HuDmK172llaiYh7gfEAaZuTh4FpwHuAr0fEVwfbR29Z4I/A3sBRg+3EzKxWytvZ8A3AvJyNCAelt0QugIiY17bezMxqoh9L9EdLmt70fEq6F3Ge44Ermp6fJuldwHTgExHxeP8j7T2Rbynp42t7MSIuGEiHZmZV18/SypKImNhnm9L6wBHAmenUhcAXyKZ3fwH4GvDe/sYKvSfyYcBIqNFkSjOzthDD2l9bOQyYGRGPAjS+wgt7Wl0z0IZ7S+SLIuLzA23YzKyuRCk18kk0lVUkbRMRi9LTo4E5A224zxq5mdk6p81zxCWNAN4IfKDp9FckjScrrcxvea1fekvkbxhoo2ZmddfO/cgj4llgi5ZzJ7ar/bUm8ohY1q5OzMzqpKTSSmm8msTMLEdX3SHIzGxdVKM87kRuZtZK9LGjYMU4kZuZtVL7Ns0aCk7kZmY56pPGncjNzNYgKGNlZ2mcyM3MctQojzuRm5mtSa6Rm5nVmWetmJl1gTqNyEv9pSPpnZL+KGmWpO+n2xyZmVWeCh5VUFoil/Ry4Dhgv4gYD6wGTsi5brKk6ZKmL16yuPVlM7Ohl+aRFzmqoMzSyhuACcAd6YfdCHis9aJ0S6QpABMmTIwS4zEzK8Q18r8TcFlEnNnnlWZmFVOV0XYRZf7SuRF4u6StACSNkrRjif2ZmbVNj4odVVDaiDwi/lvSOcBvJPUAK4EPAQvK6tPMrB2y0kpFsnQBpU4/jIipwNQy+zAzK0ONKiueR25mtiYhj8jNzOrNI3IzsxpzjdzMrO7kEbmZWe05kZuZ1Vw7P+yUNB94imyrklURMVHSKLJZfWOB+cCxEfH4QNqv0ypUM7MhIUpZEHRQRIyPiInp+RnAjRExjmwB5RkDjdeJ3MwsR49U6BiEI4HL0uPLgKMGHOtgojAz61Yq+B8wurGDazom5zQXZKvcZzS9vnVELAJIX7caaKyukZuZtWiUVgpa0lQuWZv9IuKRtPfUDZLuGUx8rTwiNzNbQ9HxeLFsHxGPpK+PAdOAfYBHJW0DkL6usc13UU7kZmat0jzyIkefTUkbS9qk8Rg4BJgDXA2clC47CfjlQMN1acXMLEcbp5FvDUxL+5sPB/4jIn4t6Q7gSkknA/8DHDPQDiqVyFdH8MyKVZ0OI9ewqmw8bGaly2rk7fk3HxEPAnvmnF9Kdie1QatUIjczq4o6Dd2cyM3M8tQokzuRm5nlaFdpZSg4kZuZ5ahPGnciNzPLV6NM7kRuZtZCtHf3w7I5kZuZtfKNJczM6q9GedyJ3MwsV40yuRO5mdkaim+IVQVO5GZmOVwjNzOrMVGryooTuZlZHtVoSO5EbmaWo0Z53InczCxPjfL40NwhSNKHJd0t6cdD0Z+Z2aCoH0cFDNWI/IPAYRHx5yHqz8xsUOo0/bD0Ebmk7wE7A1dL+ljZ/ZmZDZZo3z07h0LpI/KIOEXSocBBEbGk7P7MzNqhIjm6kCGpkfdG0mRJ0yVNX7rEed7MKqJGNfKOJ/KImBIREyNi4hajR3c6HDMzoLFIv+//qsDTD83McvRUI0cX0vERuZlZJbWptCJpB0n/laZgz5X0kXT+XEkPS5qVjjcNNNQhGZFHxNih6MfMrB3afIegVcAnImKmpE2AGZJuSK99PSK+OtgOXFoxM2vVxqmFEbEIWJQePyXpbmC79rSecWnFzCxHGZNWJI0F9gL+kE6dJmm2pEskbT7QWJ3IzczyFM/koxtTqNMxObc5aSRwFfDRiHgSuBDYBRhPNmL/2kBDdWnFzGwN/ZpauCQiJvbamrQeWRL/cUT8HCAiHm16/SLgmgEG6xG5mVmedi3RV7ax+cXA3RFxQdP5bZouOxqYM9BYPSI3M2vR5kWb+wEnAn+SNCudOwuYJGk8EMB84AMD7cCJ3MwsT/tmrdyyltaua08PTuRmZrl6qrK1YQFO5GZmOeqTxp3IzczWVKG9xotwIjczy1WfTO5EbmbWonGHoLpwIjczy1GjPO5EbmaWxyPyAeqR2GB4NRebDh9WzbjMrBxVuftPEZVK5GZmlVGfPO5EbmaWp0Z53InczKyV5JWdZmb1V5887kRuZpanRnncidzMLE+NKitO5GZma+rXHYI6zonczKxF3Zboe5WLmVnNeURuZpajTiNyJ3IzsxyukZuZ1Vi2IKjTURTnRG5mlseJ3Mys3lxaMTOruTp92Onph2ZmOVTwKNSWdKikeyU9IOmMdsfqRG5mlqdNmVzSMOA7wGHA7sAkSbu3M9RSE7mkz0i6R9INkq6Q9Mky+zMzaxcV/K+AfYAHIuLBiHgO+AlwZDtjLa1GLmki8DZgr9TPTGBGznWTgcnp6dObbDjs3jaFMBpY0qa22q2qsVU1LqhubFWNC9ad2HZsUzsvuHPmjOtHrK/RBS/fUNL0pudTImJK0/PtgIeani8EXj3YGJuV+WHna4FfRsRyAEm/yrso/cBT8l4bDEnTI2Jiu9tth6rGVtW4oLqxVTUucGyDERGHtrG5vGF7tLH9UksrNfrM18ysNAuBHZqebw880s4OykzktwBvkbShpJHAm0vsy8ysqu4AxknaSdL6wPHA1e3soLTSSkTcIelq4C5gATAdeKKs/nK0vVzTRlWNrapxQXVjq2pc4NgqISJWSToNuB4YBlwSEXPb2Yci2lqqeXHj0siIeFrSCOBmYHJEzCytQzOzdVDZKzunpPmSGwKXOYmbmbVfqSNyMxtakkZExLOdjsOGlld2WmVJ2lqq044XnSXpCOBb6QM1W4c4kQ8BST3pq5NSQZK2A84hW85cyT+3KsUlaQvgw8DXgO0ljepwSDaEui6RS9qo0zG0iojn08MdJQ2vUgJokPTyTsfQ4hGylcB7AW+t4p8ZsC2ApCrsIvocsAr4DHAB8Hzvl3eGpJdW9P9lrXVVIk9TfL4i6UuSNq1APPtKOj49/hBwFfAD4IzGKL0KJJ0KnC9p607HAtlIN7IPb54HXgZ8GjiySgkg/V37nqQvAx+UtEEn44mIp4AbgbcAsyPir1X68wKQ9I/AF4C3VS22uqvCSKItJH0QOAZ4B9m+LttJ+kJE3N/BsDYHvpRGu2NTfGOBA4EvSzqjabTeEamuegpwREQ82slYGiIiJJ0AnA6cDLwXOAgYLumq6PAn9JKOAo4FjgCmAXMjYkUnY0quJPu7/21Jj0fE1zsdUIOkw4HTgJGkvVGq8P+yW1RmVDgYkl4C7E22YuptwJ3ppW9KGtepuCLiWrINwd4GDI+IB8lWvF4ObE22pWWnbQtMjYgFktbrdDBNdgOujIjZwKeAB8gS+zEVGM1tCnwDOApYCXwcQNKunQwqIhZExI1kg5lT07uGjkvv9M4APhIRrwV+TjaYqdS7rDrrikQeEU8CHwK2Ao5OG96cBLwKOLGTn+JHxA3A2cCbJB0XEc+ldwkjgSrUpRcA+0vaLSJWAkg6MY06O2kmsJ+kV0TEyoj4FtmquD3J/uw6aT5wPnByRBwSEc9J+jDwvir8MoyIO4G3A+dK+kCn4yGr3/cAW6bnFwPrAR8B/rlTQXWTrimtRMQKSc+Svf3+R7JNan4N/HvaA7iTsf1S0olk7xB2B/4I7ESWrDrtVmA/4CRJ/x/YhGz2w6SORgU3kf0iniTp/wEbAcuAb6V6cCfNAH4JPC/pQGAM2cDhpMYvw06LiNkptuUViOVxST8DDpL014iYI+mnZO9Wj5P0XxUpTdVWVy0ISh84fRQ4mKx0cWxE3NPZqP4ujXKvAn4KnBER8zsbUUbSNmQb3R9Bth/Ol1JJo6MkbQu8NR2rgE9ExJ86G1Um/ZkdkY6lwPlVia2KJG1P9lnMBLIBzFHAu4BzgXMi4q7ORVd/XZXIAdJb25cCz0fEw52Op5WkA4D5EbGg07G0apSgOv0OppWkjcn+rj7d6VhaNUopVRmJV1n6LGtfsvLYdcDGZJtnvbEqH7TXVdclcjOrPkkHAV8CPuDR+OA5kZvZkEulqfWr+M60jpzIzcxqriumH5qZrcucyM3Mas6J3Mys5pzIbQ2SVkuaJWmOpJ+mW/UNtK0DJV2THh8h6Yxert0s7ZnT3z7OlfTJoudbrrlU0tv70ddYSXP6G6NZmZzILc/yiBgfEXuQLa8+pflFZfr9dyciro6IL/dyyWZAvxO52brOidz68nvgH9JI9G5J3yVbmbeDpEMk3SZpZhq5jwSQdKikeyTdQrYqk3T+3ZK+nR5vLWmapLvSsS/wZWCX9G7g/HTdpyTdIWm2pM81tXW2pHsl/ZZsg61eSXp/aucuSVe1vMs4WNLvJd2XdulD0jBJ5zf1XYU9S8xyOZHbWim7YcJhQGPp+W7ADyNiL+AZsjv4HBwRewPTgY9L2hC4iGxf7P3JVtnm+Sbwu4jYk2znyrlkO+TNS+8GPiXpEGAcsA8wHpgg6XWSJpDtdLkX2S+KVxX4cX4eEa9K/d1Ntj1uw1jgAODNZHuMb5hefyIiXpXaf7+knQr0YzbkumbTLGurjSTNSo9/T7Zb3bbAgoi4PZ3/J7JteG9NO5GuD9xGdiOIPzf2gZf0I7LNkVq9nmyvDSJiNfCEpM1brjkkHY1tiUeSJfZNgGmNmwxLurrAz7SHpC+SlW9GAtc3vXZl2hf+fkkPpp/hEOCVTfXzTVPf9xXoy2xIOZFbnuURMb75RErWzzSfAm6IiEkt140H2rXKTGQbeH2/pY+PDqCPS4GjIuIuSe8m2w+7obWtSH2fHhHNCR9JY/vZr1npXFqxgbqdbL/wfwCQNCLdWOEeYCdJu6Tr1rYd7o3Aqel7h6UNlZ4iG203XA+8t6n2vp2krYCbgaMlbSRpE7IyTl82ARalTa5OaHntGEk9KeadgXtT36c2NsWStGvavMuscjwitwGJiMVpZHuF/n6/ynMi4j5Jk4FrJS0huyPSHjlNfASYIulkYDVwakTcJunWNL3vP1Od/OXAbekdwdPAOyNipqSpwCyyG2P8vkDInwH+kK7/Ey/+hXEv8DuyrY9PiYi/Sfp3spz9m9MAAAA/SURBVNr5TGWdLybbetWscrzXiplZzbm0YmZWc07kZmY150RuZlZzTuRmZjXnRG5mVnNO5GZmNedEbmZWc/8LRIbQI8QZn58AAAAASUVORK5CYII=\n", "text/plain": ["<Figure size 432x288 with 2 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAEmCAYAAACtaxGwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debwcVZ338c/3JoQEE4IQBEkCQVlGYGQLKIsKgjwwsrgiKCAPSwQFVASND8ggyoDiMi4oRkG2EcIimtFoVBxkESRhlYBhQiQSUEgggOwQfs8fdS50On276yZdt6puvm9e9aKrq/qcX/ft/Pr06XNOKSIwM7N66Sk7ADMz6z8nbzOzGnLyNjOrISdvM7MacvI2M6shJ28zsxpy8l7JSDpV0sXp9vqSnpI0pMt13C9p926WmaPOoyU9nJ7PWitQzlOS3tDN2MoiabakXcqOw4rh5N1lKXE9LOk1DfcdIemaEsNqKSL+FhEjI2JJ2bGsCEmrAN8A9kjP59HlLSs9fl73ous+SedL+nKn8yJi84i4ZgBCshI4eRdjKPDJFS1EGf+NOlsHGA7MLjuQKpA0tOwYrHhODMU4CzhB0hqtDkraUdJMSU+k/+/YcOwaSadLugF4BnhDuu/Lkv6Yvtb/t6S1JP2XpCdTGRMayviWpAfSsVskva2POCZICklDJe2Qyu7dnpN0fzqvR9JkSfdJelTSZZLWbCjnYEnz07GT2r0wkkZI+no6/wlJ10sakY7tm77qP56e85saHne/pBMk3ZkeN1XScEmbAHPSaY9L+n3j82p6XY9ItzeS9IdUziJJUxvOC0kbpdujJV0oaWGK9+TeD1NJh6bYvyZpsaS/StqrzfO+X9KJKf6nJZ0raR1Jv5L0T0m/k/TahvMvl/SPFOO1kjZP908CPgJ8tve90FD+5yTdCTyd/qavdF9Jmi7p6w3lT5V0Xru/lVVcRHjr4gbcD+wO/BT4crrvCOCadHtNYDFwMFkL/cC0v1Y6fg3wN2DzdHyVdN9c4I3AaOBu4N5Uz1DgQuDHDTEcBKyVjn0G+AcwPB07Fbg43Z4ABDC06Tn01nlG2v8UcBMwDlgV+AFwSTq2GfAU8PZ07BvAS8Dufbw+Z6eyxwJDgB3T4zYBngbeler/bHrOwxpe15uB9dJreA9wVKvn0ep5pTqPSLcvAU4ia7wMB3ZuOC+AjdLtC4GfA6NSmfcCh6djhwIvAkem53E08BCgNu+Lm8i+JYwFHgFuBbZOz//3wL83nH9YqndV4D+B2xuOnU96bzWVfzswHhjR+F5Mt9dNdb6TLPnPA0aV/e/F2wrkmrIDGGwbrybvLYAngLVZOnkfDNzc9JgbgUPT7WuA05qOXwOc1LD/deBXDfv7NP7jbhHTYmDLdPtUOifv7wO/BHrS/j3Abg3HX58S11DgFODShmOvAV6gRfJOyfLZ3liajn0BuKzp3AeBXRpe14Majn8VOKfV82j1vFg6eV8ITAHGtYgjgI3IEvLzwGYNxz7W8Hc8FJjbcGy19Nh127wvPtKwfyXw/Yb9Y4Gf9fHYNVLZo9P++bRO3oe1ei827L8PeABYRMMHlrd6bu42KUhE3AX8ApjcdGg9YH7TffPJWmO9HmhR5MMNt59tsT+yd0fSZyTdk75yP07WWh+TJ25JHwN2AT4cES+nuzcArkrdGY+TJfMlZK3I9RrjjYingb5+MBxD1tK9r8WxpV6XVPcDLP26/KPh9jM0POd++iwg4ObUTXNYH7EOY+m/VfPf6ZV4IuKZdLNdTLn+hpKGSDozdVM9SZaEe2Nqp9X7ptEvyD6U5kTE9R3OtYpz8i7Wv5N9rW78B/8QWTJstD5ZK7PXci/1mPq3PwfsD7w2ItYg+wagnI/9ErBfRDzRcOgBYK+IWKNhGx4RDwJ/J/uq3lvGamRdNq0sAp4j6/5pttTrIkmp3AdbnNvJ0+n/qzXct27vjYj4R0QcGRHrkbWmv9fbz90U64ss/bdq/jsV5cPAfmTf4EaTfZOAV/+Gfb0/Or1vTif74H29pANXMEYrmZN3gSJiLjAVOK7h7unAJpI+nH5U+hBZv/EvulTtKLI+54XAUEmnAKt3epCk8SnWQyLi3qbD5wCnS9ognbu2pP3SsSuAvSXtLGkYcBp9vK9Sa/o84BuS1kstzB0krQpcBrxb0m7Khv59hqzb4o/9evZZPQvJkuxBqY7DaPjAkPRBSePS7mKypLekqYwlKabTJY1Kz/144OL+xrMcRpE990fJPoD+o+n4w0C/xqJLejvwf4FD0vYdSWPbP8qqzMm7eKeR9QMDENkY5L3JktOjZF/h946IRV2qbwbwK7If1+aTtXQ7fZ0G2I2sdXqFXh1x0jv07lvANOA3kv5J9sPbW9LzmQ18AvgJWSt8MbCgTT0nAH8GZgKPAV8h61ufQ/ZD63fIWr37APtExAs5n3ezI4ETyV7jzVn6Q2A74E+SnkrP65MR8dcWZRxL1oqfB1yfnuNAjNC4kOxv9yDZj9M3NR0/F9gsdWP9rFNhklZPZR4TEQ+mLpNzgR+nbzhWQ4rwxRjMzOrGLW8zsxpy8jYzK5Ck8yQ9IumuPo5L0rclzU2TuLbJU66Tt5lZsc4H9mxzfC9g47RNIptn0VGhyVvSz5RNz56dpvWama1UIuJash/n+7IfcGFkbgLWkPT6TuUWvYDNYRHxmLK1K2ZKujKaVnxLST1L7D2rbKsRy72aZ6G23KSao6p6PFbAVnLz59/PokWLuvovYcjqG0S89Gyuc+PZhbPJRnX1mhIRU/pR3ViWHhG2IN3393YPKjp5Hyfpven2eLKvBUsl7/QkpwD0jHx9rLrV4QWHtHx+/5svlh1CSyOGdXUpbrPa2ektE7teZrz0LKtuun+uc5+7/eznImJFgmj1wdNxGGBhyVvZIvC7AztExDPK1rMeXlR9ZmbdIxi41ZgX0DBLmWwBuIc6PajI6EYDi1Pi/hfgrQXWZWbWPQKkfNuKmwYckkadvBV4IiLadplAsd0mvwaOSusLz2HZWWJmZtXVpZa3pEvIFnsbI2kB2ZpHqwBExDlkS2b8G9kSyM+QLWPQUWHJOyKeJxsCY2ZWM4Ke7vyeFBFtFwGLbJr7J/pbri+XZGbWSsWXfXHyNjNrJgbyB8vl4uRtZraMrv0YWRgnbzOzVtzyNjOrIbe8zczqZkAn6SwXJ28zs2a9k3QqzMnbzKwVt7zNzOpGMKTai745eZuZNfM4bzOzmnKft5lZ3Xi0iZlZPbnlnd9Wm47l2qtPKzuMlrY+eUbZIbR00cd2KDuEPm01YY2yQzBbfm55m5nVTPcutFAYJ28zs1bc8jYzqyG3vM3M6sajTczM6kd07TJoRXHyNjNbhlveZmb15D5vM7MacsvbzKyG3PI2M6sZuc/bzKye3PI2M6sfOXmbmdVLdglLJ28zs3qRUI+Tt5lZ7bjlbWZWQ1VP3oWNhZE0QdJdRZVvZlYkSbm2srjlbWbWTGmrsKJHoQ+VdIGkOyVdIWm1guszM1thIl+ru8yWd9HJe1NgSkS8GXgS+HjzCZImSZoladaihQsLDsfMLJ+VPXk/EBE3pNsXAzs3nxARUyJiYkRMHLP22gWHY2aWT9WTd9F93tFh38ysklba0SbJ+pJ2SLcPBK4vuD4zsxWnfmwlKTp53wN8VNKdwJrA9wuuz8xshQnR09OTaytLYd0mEXE/sFlR5ZuZFamb3SaS9gS+BQwBfhQRZzYdXx+4AFgjnTM5Iqa3K7PaC9aamZWlS90mkoYAZwN7kTVoD5TU3LA9GbgsIrYGDgC+16lcJ28zs2bq6miT7YG5ETEvIl4ALgX2azongNXT7dHAQ50K9QxLM7MW+tFtMkbSrIb9KRExpWF/LPBAw/4C4C1NZZwK/EbSscBrgN07VerkbWbWQj+S96KImNiuqBb3NQ+bPhA4PyK+nkboXSRpi4h4ua9CnbzNzJr0To/vkgXA+Ib9cSzbLXI4sCdARNwoaTgwBnikr0Ld521m1kr3xnnPBDaWtKGkYWQ/SE5rOudvwG4Akt4EDAfarhfilreZWTN1b6hgRLwk6RhgBtkwwPMiYrak04BZETEN+AzwQ0mfJutSOTQi2s5Id/I2M2uhm+O805jt6U33ndJw+25gp/6U6eRtZtaCr2FpZlZDVV+YysnbzKxJ2cu95uHkbWbWgpO3mVkNOXn3g4ChQ6o59Pzio3bofFIJ9v/2dWWH0Ke/fG3vskPoU9X/YVoFVPwtUqnkbWZWFVX/gHfyNjNr1sVJOkVx8jYzayKg4rnbydvMbFmix5N0zMzqx90mZmZ1I3ebmJnVjsDdJmZmdeSWt5lZDbnP28ysbtznbWZWP9k472pnbydvM7NleElYM7NaqnjudvI2M1uGPFTQzKx23OdtZlZTFc/dTt5mZq1UveVd+GVrJB0i6U5Jd0i6qOj6zMy6Qcq3laXQlrekzYGTgJ0iYpGkNYusz8ysK2pwMYaiW97vBK6IiEUAEfFY8wmSJkmaJWnWwkULCw7HzKyz3osxVLnlXXTyFhDtToiIKRExMSImrj1m7YLDMTPLI5ukk2crS9HJ+2pgf0lrAbjbxMzqouot70L7vCNitqTTgT9IWgLcBhxaZJ1mZivMk3QgIi4ALii6HjOzbvEkHTOzmnLyNjOroYrnbidvM7NW3PI2M6sbX0nHzKx+5IsxmJnVU8Vzt5O3mVkrPRXP3oWvKmhmVkfdnGEpaU9JcyTNlTS5j3P2l3S3pNmSftKpTLe8zcyaSDCkSzMsJQ0BzgbeBSwAZkqaFhF3N5yzMfB5shVYF0t6Xady3fI2M2uhiwtTbQ/MjYh5EfECcCmwX9M5RwJnR8RigIh4pFOhfba8Ja3e7oER8WTHkM3MaqofXd5jJM1q2J8SEVMa9scCDzTsLwDe0lTGJlmdugEYApwaEb9uV2m7bpPZZMu5Nj6F3v0A1m9XsJlZXYlsuGBOiyJiYofimjUvlT0U2BjYBRgHXCdpi4h4vK9C+0zeETG+TTBmZoNaFxcVXAA05tNxwEMtzrkpIl4E/ippDlkyn9lnfHlqlnSApP+Xbo+TtG1/Ijczq5Wc/d05+7xnAhtL2lDSMOAAYFrTOT8Dds2q1hiybpR57QrtONpE0neBVYC3A/8BPAOcA2yXJ+rBYssN1ig7hJZGjly17BD6dP3cRWWH0Ke3beyrNll73RrmHREvSToGmEHWn31eutbBacCsiJiWju0h6W5gCXBiRDzartw8QwV3jIhtJN2WAnksfXqYmQ1KoruTdCJiOjC96b5TGm4HcHzacsmTvF+U1EPqYE+XNHs5bwVmZnVU8QmWufq8zwauBNaW9EXgeuArhUZlZlayql+AuGPLOyIulHQLsHu664MRcVexYZmZlaebMyyLknd6/BDgRbKuE8/KNLNBr9qpO0cilnQScAmwHtn4xJ9I+nzRgZmZlan23SbAQcC2EfEMgKTTgVuAM4oMzMysLNlok7KjaC9P8p7fdN5QOgweNzOrtZJb1Xm0W5jqm2R93M8AsyXNSPt7kI04MTMbtCqeu9u2vHtHlMwGftlw/03FhWNmVg21bXlHxLkDGYiZWVUMij5vSW8ETgc2A4b33h8RmxQYl5lZqare8s4zZvt84MdkH0Z7AZeRXQnCzGxQkmCIlGsrS57kvVpEzACIiPsi4mTS0oVmZoNVNy9AXIQ8QwWfV/b94T5JRwEPAh0vjmlmVmdV7zbJk7w/DYwEjiPr+x4NHNbfiiSdCjwVEV/r72PNzAZaxXN3roWp/pRu/hM4uNhwzMzKJ9TV9byL0G6SzlUse5HMV0TE+zoVntZFOYTsyskLyabVm5lVW8n92Xm0a3l/d0UKTte5PADYOtVzKy2St6RJwCSA8ev7gvRmVg217fOOiKtXsOy3AVc1LGjVfMHN3nqmAFMAtt12Yp8tfTOzgVT1ta/zrue9vJyMzax2RPVb3kV+uFwLvFfSCEmjgH0KrMvMrKt6lG8rS+6Wt6RVI+L5vOdHxK2SpgK3ky0re91yxGdmNuAGxWXQJG0PnEs2vnt9SVsCR0TEsZ0eGxGnk40NNzOrlYrn7lzdJt8G9gYeBYiIO/D0eDMb5AbD9PieiJjf1Hm/pKB4zMxKly0JW+2md57k/UDqOglJQ4BjgXuLDcvMrFyDYajg0WRdJ+sDDwO/S/eZmQ1aFW9451rb5BGymZJmZisFqcZrm/SS9ENaTLaJiEmFRGRmVgEVz925uk1+13B7OPBesoWmzMwGraoPFczTbTK1cV/SRcBvC4vIzKxkYhBM0mlhQ2CDbgdiZlYZJU99zyNPn/diXu3z7gEeAyYXGZSZWdlEtbN32+Sdrl25Jdl1KwFejgivFGhmg1o2SafsKNprOw49JeqrImJJ2py4zWylUPVVBfNMIrpZ0jaFR2JmViGScm1laXcNy6ER8RKwM3CkpPuAp8m+UURErFQJ/bXbHVN2CC0tnrlCV6szsxbq0G3Srs/7ZmAb4D0DFIuZWTV0ecVASXsC3wKGAD+KiDP7OO8DwOXAdhExq12Z7ZK3ACLivuUL18ysvro1PT4t6Hc28C5gATBT0rSIuLvpvFHAccCf8pTbLnmvLen4vg5GxDfyVGBmVjdd7jbZHpgbEfMAJF0K7Afc3XTel4CvAifkKbTdD5ZDgJHAqD42M7NBSgxRvg0YI2lWw9a87tNYll5SZEG679XapK2B8RHxi7wRtmt5/z0iTstbkJnZYJFdPT736YsiYmKH4pq9MuxaUg/wTeDQ3DWSo8/bzGyl090x3AuA8Q3744CHGvZHAVsA16Shh+sC0yTt2+5Hy3bJe7flj9XMrN66uJ73TGBjSRuSzVY/APhw78GIeAIY07sv6RrghE6jTfrs846Ix1YwYDOzWurtNunGBYjTfJljgBnAPcBlETFb0mmS9l3eGJdnVUEzs0Gvm1fSiYjpwPSm+07p49xd8pTp5G1m1sJguJKOmdlKRQyOq8ebma1cRKmLTuXh5G1m1kK1U7eTt5nZMgS9sycry8nbzKyFiuduJ28zs2WVe6GFPJy8zcyaeLSJmVlNVb3lXeiHi6SDJN0s6XZJP0iLkpuZVZ5ybmUpLHlLehPwIWCniNgKWAJ8pMV5k3rXwV24aGFR4ZiZ5acaX4C4C3YDtiW75A/ACOCR5pMiYgowBWDbbSdG83Ezs4G2svd5C7ggIj5fYB1mZoVYmfu8rwY+IOl1AJLWlLRBgfWZmXVNj/JtZSms5R0Rd0s6GfhNuszPi8AngPlF1Wlm1g1Zt0m1W96FDhWMiKnA1CLrMDMrQsV7TTzO28xsWUIrc8vbzKyu3PI2M6uZlb7P28yslnJeXLhMTt5mZi04eZuZ1ZB/sDQzqxlR7gScPJy8zcxa6Kl4v4mTt5lZC+42MTOrGXebmJnVkmdYmpnVj8d5m5nVU8Vzd7WS932PPs2Hfjyz7DBaOuNbx5cdgpkNkKzPu9rpu1LJ28ysKqqdup28zcxaq3j2dvI2M2vB3SZmZjVU7dTt5G1m1lrFs7eTt5lZE+Hp8WZm9eNJOmZm9VTx3E1P2QGYmVWScm55ipL2lDRH0lxJk1scP17S3ZLulHS1pA06lenkbWa2DOX+r2NJ0hDgbGAvYDPgQEmbNZ12GzAxIt4MXAF8tVO5Tt5mZi1I+bYctgfmRsS8iHgBuBTYr/GEiPifiHgm7d4EjOtUqJO3mVmTvD0mKXePkTSrYZvUVNxY4IGG/QXpvr4cDvyqU4z+wdLMrAXlH26yKCImtiuqxX3RR50HAROBd3Sq1MnbzKyFLg4VXACMb9gfBzy0bH3aHTgJeEdEPN+pUHebmJm10MXBJjOBjSVtKGkYcAAwbam6pK2BHwD7RsQjeQodkOQt6ThJ90j6r4Goz8xshfSz07udiHgJOAaYAdwDXBYRsyWdJmnfdNpZwEjgckm3S5rWR3GvGKhuk48De0XEXweoPjOzFdLN6fERMR2Y3nTfKQ23d+9vmYW3vCWdA7wBmCbp00XXZ2a2okRXhwoWovCWd0QcJWlPYNeIWFR0fWZm3VD16fGljzZJYyInAYxYc92SozEzSyqevUsfbRIRUyJiYkRMHDZqjbLDMTMD8k+QL0vpLW8zsyrqqXjL28nbzKwVJ2+IiAkDUY+ZWTf4SjpmZnXkK+mYmdVTxXO3k7eZWUsVz95O3mZmyyh3GGAeTt5mZi24z9vMrGb6sdxraZy8zcxaqXj2dvI2M2uhp+L9Jk7eZmYtVDt1O3mbmS3Lk3TMzOqq2tnbydvMrEnvlXSqzMnbzKyFiuduJ28zs1bc8u6HJx95lN9874Kyw2hp6szvlh2CmQ0gT483M6ujauduJ28zs1YqnrudvM3MmkmeYWlmVk/Vzt1O3mZmrVQ8dzt5m5m1UvFeEydvM7Nl+Uo6Zma1U4fp8T1lB2BmZv3nlreZWQtVb3k7eZuZteA+bzOzmskm6ZQdRXtO3mZmrTh5m5nVj7tNzMxqqOo/WHqooJlZC8q55SpL2lPSHElzJU1ucXxVSVPT8T9JmtCpTCdvM7NWupS9JQ0Bzgb2AjYDDpS0WdNphwOLI2Ij4JvAVzqVW2jylvQFSX+R9FtJl0g6ocj6zMy6RTn/y2F7YG5EzIuIF4BLgf2aztkP6L2M2BXAblL7jpvC+rwlTQTeD2yd6rkVuKXFeZOASWn3qeduP3tOl0IYAyzqUlmMWOXsbhUFXY6ti6oaF1Q3tqrGBStPbBt0qZxX3HbrLTNWG6YxOU8fLmlWw/6UiJjSsD8WeKBhfwHwlqYyXjknIl6S9ASwFm1eoyJ/sNwZ+HlEPAsg6b9bnZSe5JRWx1aEpFkRMbHb5XZDVWOralxQ3diqGhc4thUREXt2sbhWLehYjnOWUmS3ScV/qzUzGxALgPEN++OAh/o6R9JQYDTwWLtCi0ze1wP7SBouaSTw7gLrMjOrqpnAxpI2lDQMOACY1nTONOCj6fYHgN9HRNuWd2HdJhExU9I04A5gPjALeKKo+lroeldMF1U1tqrGBdWNrapxgWOrhNSHfQwwAxgCnBcRsyWdBsyKiGnAucBFkuaStbgP6FSuOiT3FSJpZEQ8JWk14FpgUkTcWliFZmYriaJnWE5J4xmHAxc4cZuZdUehLW8zG1iSVouIZ8qOw4rnGZZWWZLW6TRRwV4laV/gO+lHMRvknLwHgKSe9H8nopwkjQVOJptKXMnXrUpxSVoLOA74OjBO0polh2QFG3TJW9KIsmNoFhEvp5sbSBpapX/0vSS9qewYmjxENiN3a+B9VXzNgPXglXG5ZXsBeAn4AvAN4OX2p5dD0roV/VvWzqBK3mk4zlclnSFpdAXi2VHSAen2J4ArgR8Dk3tb41Ug6WjgLEnrlB0LZC3aNMb1ZeBfgM8B+1XpH316r50j6Uzg45JWLTOeiPgncDWwD3BnRDxepdcLQNK/Al8C3l+12OqoCi2GrpD0ceCDwIfJ1lEZK+lLEfG/JYb1WuCM1KqdkOKbAOwCnClpckOrvBSpn/QoYN+IeLjMWHpFREj6CHAs2WprhwG7AkMlXdlp8kLRJL0H2B/YF7gKmB0Rz5cZU3IZ2Xv/u5IWR8Q3yw6ol6S9gWOAkaS1SKrwt6yzyrT+VoSk1YFtyAa2vx+4LR36tqSNy4orIn5JtujW+4GhETGPbObpRcA6ZMtDlm09YGpEzJe0StnBNNgUuCwi7gROBOaSJfMPVqDVNhr4T+A9wIvA8QCSNikzqIiYHxFXkzVgjk7fDkqXvtFNBj4ZETsDPyVrwFTq21TdDIrkHRFPAp8AXge8Ny0q81FgO+DgMn99j4jfAicB/ybpQxHxQvo2MBKoQj/zfOBtkjaNiBcBJB2cWpdluhXYSdLmEfFiRHyHbHbalmSvXZnuB84CDo+IPSLiBUnHAUdU4QMwIm4jm2J9qqSPlR0PWX98D7B22j8XWAX4JPB/ygqq7gZNt0lEPC/pGbKv1v9KtsjLr4EfpTV0y4zt55IOJvsmsBlwM7AhWYIq2w3ATsBHJf0RGEU2auHAUqOCa8g+fA+U9HtgBNm04e+k/t0y3QL8HHhZ0i7A+mSNhY/2fgCWLSLuTLE9W4FYFku6AthV0uMRcZeky8m+lX5I0v9UpNupVgbVJJ30o9GngN3JuiX2j4i/lBvVq1Jr9krgcmByRNxfbkQZSa8nWwx+X7L1Z85I3RWlkrQe8L60vQR8JiL+XG5UmfSa7Zu2R4GzqhJbFUkaR/bbyrZkjZb3AIcApwInR8Qd5UVXT4MqeQOkr63rAi9HxINlx9NM0juA+yNiftmxNOvtXir7m0ozSa8he68+VXYszXq7SarS4q6y9NvUjmRdX9OB15AtUPWuqvxYXieDLnmbWfVJ2hU4A/iYW93Lx8nbzAZc6nYaVsVvoHXh5G1mVkODYqigmdnKxsnbzKyGnLzNzGrIyduWIWmJpNsl3SXp8nQZu+UtaxdJv0i395U0uc25a6Q1avpbx6mSTsh7f9M550v6QD/qmiDprv7GaNZtTt7WyrMRsVVEbEE2tfmoxoPK9Pu9ExHTIuLMNqesAfQ7eZutjJy8rZPrgI1Si/MeSd8jmyE3XtIekm6UdGtqoY8EkLSnpL9Iup5sdiTp/kMlfTfdXkfSVZLuSNuOwJnAG1Or/6x03omSZkq6U9IXG8o6SdIcSb8jW8SqLUlHpnLukHRl07eJ3SVdJ+netPodkoZIOquh7iqsEWL2Cidv65OyiwzsBfRO+94UuDAitgaeJrvSze4RsQ0wCzhe0nDgh2TrSr+NbLZrK98G/hARW5KtCDmbbOW5+1Kr/0RJewAbA9sDWwHbSnq7pG3JVpDcmuzDYbscT+enEbFdqu8esqVme00A3gG8m2yN7uHp+BMRsV0q/0hJG+aox2xADJqFqayrRki6Pd2+jmwVuPWA+RFxU7r/rWRL2t6QVvUcBtxIdvGEv/auoy7pYrIFiJq9k2xtCyJiCfCEpNc2nbNH2nqX+B1JlsxHAVf1XmhX0rQcz2kLSV8m65oZCcxoOHZZWlf9fyXNS89hD+DNDf3ho1Pd9+aoy6xwTt7WyrMRsVXjHSlBP914F/DbiDiw6bytgG7N/BLZIlk/aKrjU8tRxwNzAF8AAAEkSURBVPnAeyLiDkmHkq0n3au5rEh1HxsRjUkeSRP6Wa9ZIdxtYsvrJrL1tjcCkLRauhjBX4ANJb0xndfX0rJXA0enxw5Jixb9k6xV3WsGcFhDX/pYSa8DrgXeK2mEpFFkXTSdjAL+nhaS+kjTsQ9K6kkxvwGYk+o+unfhKUmbpAWyzCrBLW9bLhGxMLVgL9Gr1288OSLulTQJ+KWkRWRXDtqiRRGfBKZIOhxYAhwdETdKuiENxftV6vd+E3Bjavk/BRwUEbdKmgrcTnYxietyhPwF4E/p/D+z9IfEHOAPZMsIHxURz0n6EVlf+K3KKl9ItoypWSV4bRMzsxpyt4mZWQ05eZuZ1ZCTt5lZDTl5m5nVkJO3mVkNOXmbmdWQk7eZWQ39f8KoNg65kxJKAAAAAElFTkSuQmCC\n", "text/plain": ["<Figure size 432x288 with 2 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["import numpy as np\n", "from sklearn import svm, datasets\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import confusion_matrix\n", "\n", "# Compute confusion matrix\n", "cm = confusion_matrix(y_compare, pred)\n", "np.set_printoptions(precision=2)\n", "print('Confusion matrix, without normalization')\n", "print(cm)\n", "plt.figure()\n", "plot_confusion_matrix(cm, products)\n", "\n", "# Normalize the confusion matrix by row (i.e by the number of samples\n", "# in each class)\n", "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", "print('Normalized confusion matrix')\n", "print(cm_normalized)\n", "plt.figure()\n", "plot_confusion_matrix(cm_normalized, products, title='Normalized confusion matrix')\n", "\n", "plt.show()"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": []}], "metadata": {"anaconda-cloud": {}, "kernelspec": {"display_name": "Python 3.6 (tensorflow)", "language": "python", "name": "tensorflow"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5"}}, "nbformat": 4, "nbformat_minor": 4}