notebooks/data-sources/TestGrid/metrics/probability_to_fail.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f3c2cb4b",
   "metadata": {},
   "source": [
    "# Probability To Fail\n",
    "\n",
    "One of the key performance indicators that we would like to create greater visibility into and for a given moment in time, is to calculate what is the probability that a given test will fail. Using this information, we plan to build a model that uses the probability to fail to predict the time to failure. \n",
    "\n",
    "In order to provide maxium flexibility for the end-user of this work, instead of creating a number of dataframes to answer each of these specifc questions, we will define a long and narrow data structure (a list of tuples saved as a csv for now) that contains only 7 columns (\"timestamp\", \"tab\",\"grid\",\"test\",\"test_duration\", \"pass_or_fail\", \"prob\"). This allows visualization tools like Superset (or pandas) to perform the last filter and/or aggregate interested columns to an end user. Which is to say, there may be a lot of repetition within the final data set, but each row should be unique, and it should provide the simplest usability for an end-user. \n",
    "\n",
    "Linked issue(s) : [Issue](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/282)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9ad0ae80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "import gzip\n",
    "import os\n",
    "import pandas as pd\n",
    "import datetime\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.calibration import CalibratedClassifierCV\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.calibration import calibration_curve\n",
    "from sklearn.metrics import brier_score_loss, roc_auc_score\n",
    "from matplotlib import pyplot\n",
    "import numpy as np\n",
    "\n",
    "from ipynb.fs.defs.metric_template import testgrid_labelwise_encoding\n",
    "from ipynb.fs.defs.metric_template import CephCommunication\n",
    "from ipynb.fs.defs.metric_template import save_to_disk, read_from_disk\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "load_dotenv(find_dotenv())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2dce36ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Specify variables\n",
    "METRIC_NAME = \"probability_to_fail\"\n",
    "# Specify the path for input grid data,\n",
    "INPUT_DATA_PATH = \"../../../../data/raw/testgrid_183.json.gz\"\n",
    "\n",
    "# Specify the path for output metric data\n",
    "OUTPUT_DATA_PATH = f\"../../../../data/processed/metrics/{METRIC_NAME}\"\n",
    "\n",
    "# Specify whether or not we are running this as a notebook or part of an automation pipeline.\n",
    "AUTOMATION = os.getenv(\"IN_AUTOMATION\")\n",
    "\n",
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs,\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_path = os.getenv(\"S3_PROJECT_KEY\", \"ai4ci/testgrid/metrics\")\n",
    "s3_input_data_path = \"raw_data\"\n",
    "metric_path = f\"metrics/{METRIC_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a5ec8f57",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import data\n",
    "timestamp = datetime.datetime.today()\n",
    "\n",
    "if AUTOMATION:\n",
    "    filename = f\"testgrid_{timestamp.day}{timestamp.month}.json\"\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    s3_object = cc.s3_resource.Object(s3_bucket, f\"{s3_input_data_path}/{filename}\")\n",
    "    file_content = s3_object.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "    testgrid_data = json.loads(file_content)\n",
    "\n",
    "else:\n",
    "    with gzip.open(INPUT_DATA_PATH, \"rb\") as read_file:\n",
    "        testgrid_data = json.load(read_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c3d718b",
   "metadata": {},
   "source": [
    "## Calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c51ef525",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will now fetch all the tests which are failing i.e. have a status code of 12.\n",
    "failures_list = testgrid_labelwise_encoding(testgrid_data, 12, overall_only=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9ab44c0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>test_duration</th>\n",
       "      <th>failure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-25 19:11:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>91.800000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-08-25 18:31:23</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>17.350000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-08-25 14:51:42</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>91.933333</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-08-23 00:01:04</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>95.300000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-22 08:53:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>101.800000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-08-25 19:11:17  \"redhat-assisted-installer\"   \n",
       "1 2021-08-25 18:31:23  \"redhat-assisted-installer\"   \n",
       "2 2021-08-25 14:51:42  \"redhat-assisted-installer\"   \n",
       "3 2021-08-23 00:01:04  \"redhat-assisted-installer\"   \n",
       "4 2021-08-22 08:53:17  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  test_duration  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall      91.800000   \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall      17.350000   \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall      91.933333   \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall      95.300000   \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall     101.800000   \n",
       "\n",
       "   failure  \n",
       "0    False  \n",
       "1     True  \n",
       "2    False  \n",
       "3    False  \n",
       "4    False  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert to dataframe\n",
    "failures_df = pd.DataFrame(\n",
    "    failures_list,\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"failure\"],\n",
    ")\n",
    "failures_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4a3981f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will fetch all the tests which are passing i.e. have status code of 1.\n",
    "passing_list = testgrid_labelwise_encoding(testgrid_data, 1, overall_only=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "809d6653",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>test_duration</th>\n",
       "      <th>passing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-25 19:11:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>91.800000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-08-25 18:31:23</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>17.350000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-08-25 14:51:42</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>91.933333</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-08-23 00:01:04</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>95.300000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-22 08:53:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>101.800000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-08-25 19:11:17  \"redhat-assisted-installer\"   \n",
       "1 2021-08-25 18:31:23  \"redhat-assisted-installer\"   \n",
       "2 2021-08-25 14:51:42  \"redhat-assisted-installer\"   \n",
       "3 2021-08-23 00:01:04  \"redhat-assisted-installer\"   \n",
       "4 2021-08-22 08:53:17  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  test_duration  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall      91.800000   \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall      17.350000   \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall      91.933333   \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall      95.300000   \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall     101.800000   \n",
       "\n",
       "   passing  \n",
       "0     True  \n",
       "1    False  \n",
       "2     True  \n",
       "3     True  \n",
       "4     True  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert to dataframe\n",
    "passing_df = pd.DataFrame(\n",
    "    passing_list,\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"passing\"],\n",
    ")\n",
    "passing_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b68325ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>test_duration</th>\n",
       "      <th>failure</th>\n",
       "      <th>passing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-25 19:11:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>91.800000</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-08-25 18:31:23</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>17.350000</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-08-25 14:51:42</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>91.933333</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-08-23 00:01:04</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>95.300000</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-22 08:53:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>101.800000</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-08-25 19:11:17  \"redhat-assisted-installer\"   \n",
       "1 2021-08-25 18:31:23  \"redhat-assisted-installer\"   \n",
       "2 2021-08-25 14:51:42  \"redhat-assisted-installer\"   \n",
       "3 2021-08-23 00:01:04  \"redhat-assisted-installer\"   \n",
       "4 2021-08-22 08:53:17  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  test_duration  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall      91.800000   \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall      17.350000   \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall      91.933333   \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall      95.300000   \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall     101.800000   \n",
       "\n",
       "   failure  passing  \n",
       "0    False     True  \n",
       "1     True    False  \n",
       "2    False     True  \n",
       "3    False     True  \n",
       "4    False     True  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.merge(\n",
    "    failures_df,\n",
    "    passing_df,\n",
    "    on=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\"],\n",
    ")\n",
    "df = df.dropna()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "063f2bff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19938777, 7)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be514a81",
   "metadata": {},
   "source": [
    "### Calculate Probability to Fail using SVM and Calibrated Classification Model\n",
    "\n",
    "In order to calculate the probability to fail, some of the approaches we could consider are SVM (support vector machines) and a Calibrated Classification Model. In this section of the notebook, we will compare the SVM and Calibrated Classification model, and choose the model giving better results.\n",
    "\n",
    "The objective of the support vector machine algorithm is to find a hyperplane in an N-dimensional space(N — the number of features) that distinctly classifies the data points. SVM is a good candidate model to calibrate because it does not natively predict probabilities, meaning the probabilities are often uncalibrated. The probabilities are not normalized, but can be normalized when calling the `calibration_curve()` function by setting the `normalize` argument to `True`.\n",
    "\n",
    "An SVM model can be created using the [scikit learn module](https://scikit-learn.org/stable/modules/svm.html).\n",
    "\n",
    "A classification predictive modeling problem requires predicting or forecasting a label for a given observation. An alternative to predicting the label directly, a model may predict the probability of an observation belonging to each possible class label. Although a model may be able to predict probabilities, the distribution and behavior of the probabilities may not match the expected distribution of observed probabilities in the training data. The distribution of the probabilities can be adjusted to better match the expected distribution observed in the data. This adjustment is referred to as calibration, as in the calibration of the model or the calibration of the distribution of class probabilities.\n",
    "\n",
    "A classifier can be calibrated using the [CalibratedClassifierCV](http://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html) sci-kit library. We will generate the labels/classes required for classification and train the model on a subset of the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e37ae2ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "## This helper function will add pass/fail labels to enable classification model\n",
    "## i.e 1-\"Pass\", 0-\"Failure\".\n",
    "def p_f_label(row):\n",
    "    if row[\"passing\"]:\n",
    "        return 1\n",
    "    elif row[\"failure\"]:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b594a72c",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create a sample df with a batch of 10000 to start with.\n",
    "sample_df = df[0:10000]\n",
    "## Add the pass_or_fail label using the helper function `p_f_label`\n",
    "sample_df[\"pass_or_fail\"] = sample_df.apply(lambda row: p_f_label(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "eecbe5bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    9837\n",
       "0.0     135\n",
       "Name: pass_or_fail, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## For all the missing test_duration values, drop the row.\n",
    "sample_df = sample_df.dropna()\n",
    "## Labels for multiclass classification\n",
    "sample_df.pass_or_fail.unique()\n",
    "sample_df.pass_or_fail.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbb03c02",
   "metadata": {},
   "source": [
    "#### Resampling the data\n",
    "From above, we see that we have an imbalanced dataset i.e the classes are not represented equally, and this can lead to misleading classification accuracy results. To handle this issue of imbalanced datasets, we will resample the data. Resampling is a widely-adopted technique for dealing with imbalanced datasets by changing the dataset into a more balanced one by:\n",
    "* Undersampling - This simply deletes instances from the over-represented class (majority class) in different ways. The most obvious way is to do delete instances randomly.\n",
    "* Oversampling - This method adds copies of instances from the under-represented class (minority class) to obtain a balanced dataset. There are multiple ways you can oversample a dataset, like random oversampling."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "169f7d99",
   "metadata": {},
   "source": [
    "Let's implement the basic method of undersampling and oversampling, which uses the `DataFrame.sample` method to get random samples of each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ac703b42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class count\n",
    "count_class_1, count_class_0 = sample_df.pass_or_fail.value_counts()\n",
    "df_class_1 = sample_df[sample_df[\"pass_or_fail\"] == 1]\n",
    "df_class_0 = sample_df[sample_df[\"pass_or_fail\"] == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82d02f9c",
   "metadata": {},
   "source": [
    "##### Undersampling\n",
    "Here, we delete instances randomly from the over-represented class i.e. for our dataset the 'Pass (1)' class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "63fddc1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random under-sampling:\n",
      "0.0    135\n",
      "1.0    135\n",
      "Name: pass_or_fail, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:title={'center':'Count (Pass(1)/Fail(0))'}>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEOCAYAAABy7Vf3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUeUlEQVR4nO3df5BlZX3n8fcnjEAIyIDTmeAMOkRmk4AJSk0RLFPGEmsFRKEqFgsaHV2qJm5wjcGNorGCZqPBJEa0Vl1nA2HYsCBLMKCiCSIETQLYoKCAyARBZgJMIz/VLD/ku3+c03Kn6Z7+cbunmWfer6quPud5nnPOd7p6Pvfc55x7OlWFJKktP7PYBUiS5p/hLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuGuHlWQkyXeS/OwiHf9Pk7xjhmM/kuS/LEANNyV5eb/8/iR/M6H/n5K8eAb72a3/WY4MtL0myWfmu2ZtH4a7tinJ65OMJvlhkruTfDHJb2yH41aSA6cZdipwdlX9e7/NlUn+X1/rfUkuSrLfAtU3ArwJ+HS/vmuSC5Pc0df+8gmb/AXw3iS7TtjPS5L8c79cSX7U1//DJA9OV0dVHVxVV05R42uAR6rqGwNtv5/kniQPJzkryW79fh4FzqL7mY7v+3PAwUl+bbo69MxjuGtKSU4BzgA+BCwHngd8Ejh2EcsCujNNYC3wNxO63lZVewL/AVgKfHSBSngzcOn4C0vva8BvA/dMHFxVdwPfAV47oevVwKUD64dU1Z7919Iha3wr8L/HV5K8ii68jwCeD/wi8IGB8f8HWDse+L3zgHVD1qFFYLhrUkn2Bv4YOLmqLqqqH1XV41X1uar6g37MbknOSPJv/dcZ48GQ5M1JvjZhnz89G09ydpJPJPlCkkeSXJPkBX3fVf0mN/RnsP9pkhJ/HXiwqjZNVn9V3Q/8LfDCfp//tz9jfSjJVUkOHqjr6CQ393VsTvLf+vZlST6f5MEk9yf5apLx/zNHAf84cLzHquqMqvoa8JMpfqxX0oX5oKPZOty3kuQFSb6S5Af9u5Fzkywd6L8jySsn2W5X4BWDNdK9GJ5ZVTdV1QPAf6d7kRr/N2wCHgAOn6Zm7QAMd03lJcDuwGe3MeYP6YLgRcAhwGHA+2ZxjBPozhz3ATYCHwSoqpf1/eNnsZPN+/4qcOtUO06yDPgtYHxK4ovAauDngeuBcweGnwn8TlXtRfdi8JW+/Z3AJmCE7p3Le4HxhzFt8/hTuIXu5zRe4379fr8x5RYQ4E+B5wK/AuwPvH8Gx1oNPDnhxe9g4IaB9RuA5UmeM1WN/fqqJM+ewTH1DGK4ayrPAe6rqie2MeYNwB9X1ZaqGqML6jfO4hifrapr+2OcS/ciMVNLgUcmaf94P1d9A3A3cApAVZ1VVY/0c8vvBw7p350APA4clOTZVfVAVV0/0L4f8Pz+XctX66kn7U11/G15pN9u3NHAlwb2CXB9/07hwSQfr6qNVXVZVT3a/4z/EvjNGRxrsvr2BB4aWB9f3msbNY7vY7BNOwDDXVP5AbAsyZJtjHkucOfA+p1920wNzk3/mC58ZuoBtg6lcW+vqqVVtaKq3lBVY0l2SXJ6kn9N8jBwRz92Wf/9t+iC9s4k/5jkJX37n9O9o/iHJLcnOXXgOFMdf1v2Ah4cWJ9sSubQvv6lVfX2JMuTnN9PFz1Md41hGdObrL4fAoNn4OPLgy8CE2sc38dgm3YAhrum8i/Ao8Bx2xjzb3QX5sY9r28D+BGwx3hHkl+Y5/pupLtoOhOvp7sI/Epgb2DVeFkAVfX1qjqWbsrm74AL+vZHquqdVfWLdBdCT0lyxByOP+5X6KdFkjyL7gz8smm2+RDdVNCvVtWz6S7YZgbH2tgdJisG2m5i6ymXQ4B7q+oHk9U4sH5HVT08g2PqGcRw16Sq6iHgj4BPJDkuyR5JnpXkqCR/1g87D3hfuvvNl/Xjx+9euYHuNroXJdmdmc0TD7qX7m6OqVwLLJ0QXlPZi+6F6gd0LzgfGu/ob2F8Q5K9q+px4GHgyb7vmCQHJgndFMZPxvvozri3mh7pLzDv3q/ummT3fttxv0k39w/wG8CNMwjNvejOuB/q/61/MIN/L1X1GPDlCTWeA5yU5KD+ouz7gLMH6l8B7AtcPUXN2oEY7ppSVX2Ebs76fcAYcBfwNrqzW4A/AUbpzmK/RXeh8k/6bb9Ld7fNl4Hb6G4TnI33Axv6uefjJ6ntMbpg+u0Z7OscuimjzcDNbB1e0F0nuKOf9ngr3bUE6C5KfpkuXP8F+GRVXTGwz6Oz9QeobgX+HVgB/H2//Hz46cXTg3jqZzfxFsipfAA4lO7F5QvARTPYZtynGbgGUlVfAv4MuAL4Pt3P5LSB8a8HNvTXJcad2O9HO5j4l5i0o0r3QaKvAi+ecL/59jr+h4AtVXXGDMZ+BPjXqvpkv34z8LqqunmBa/wnunv/t3VHzvjnBm4AXlZVW/q21wBvrKqnvbjqmc9wl7az/h70U6rq9MWuRe0y3CWpQc65S1KDtnUPMwBJzgKOoZtbfOGEvnfSPRBppKru6+8M+Bjd/bs/Bt488IGQKS1btqxWrVo1h/Ilaed13XXX3VdVI5P1TRvudHck/A+6uwN+Ksn+wH+ku+o+7ii6OwxW0z3741P9921atWoVo6OjMyhFkjQuyZ1T9U07LVNVVwH3T9L1UeBdPPWsDeg+KHJOda6muw95QR65Kkma2pzm3JMcC2yuqhsmdK2guxd63Ka+bbJ9rEv3nPDRsbGxuZQhSZrCrMM9yR50T8f7o2EOXFXrq2pNVa0ZGZl0ykiSNEczmXOf6AXAAXTP2gZYSfcku8PoPgG4/8DYlX2bJGk7mvWZe1V9q6p+vqpWVdUquqmXQ6vqHuAS4E3pHA481P8FGknSdjRtuCc5j+65Gr+UZFOSk7Yx/FLgdron0v0v4HfnpUpJ0qxMOy1TVSdO079qYLmAk4cvS5I0DD+hKkkNMtwlqUFzuVtmp7Xq1C8sdglNueP0Vy92Cc3wd3N+tfC76Zm7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1KBpwz3JWUm2JPn2QNufJ/lOkhuTfDbJ0oG+9yTZmOTWJK9aoLolSdswkzP3s4EjJ7RdBrywqn4N+C7wHoAkBwEnAAf323wyyS7zVq0kaUamDfequgq4f0LbP1TVE/3q1cDKfvlY4PyqerSqvgdsBA6bx3olSTMwH3Pu/xn4Yr+8ArhroG9T3/Y0SdYlGU0yOjY2Ng9lSJLGDRXuSf4QeAI4d7bbVtX6qlpTVWtGRkaGKUOSNMGSuW6Y5M3AMcARVVV982Zg/4FhK/s2SdJ2NKcz9yRHAu8CXltVPx7ougQ4IcluSQ4AVgPXDl+mJGk2pj1zT3Ie8HJgWZJNwGl0d8fsBlyWBODqqnprVd2U5ALgZrrpmpOr6icLVbwkaXLThntVnThJ85nbGP9B4IPDFCVJGo6fUJWkBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lq0LThnuSsJFuSfHugbd8klyW5rf++T9+eJB9PsjHJjUkOXcjiJUmTm8mZ+9nAkRPaTgUur6rVwOX9OsBRwOr+ax3wqfkpU5I0G9OGe1VdBdw/oflYYEO/vAE4bqD9nOpcDSxNst881SpJmqG5zrkvr6q7++V7gOX98grgroFxm/o2SdJ2NPQF1aoqoGa7XZJ1SUaTjI6NjQ1bhiRpwFzD/d7x6Zb++5a+fTOw/8C4lX3b01TV+qpaU1VrRkZG5liGJGkycw33S4C1/fJa4OKB9jf1d80cDjw0MH0jSdpOlkw3IMl5wMuBZUk2AacBpwMXJDkJuBM4vh9+KXA0sBH4MfCWBahZkjSNacO9qk6couuIScYWcPKwRUmShuMnVCWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoOGCvckv5/kpiTfTnJekt2THJDkmiQbk3wmya7zVawkaWbmHO5JVgBvB9ZU1QuBXYATgA8DH62qA4EHgJPmo1BJ0swNOy2zBPjZJEuAPYC7gVcAF/b9G4DjhjyGJGmW5hzuVbUZ+Avg+3Sh/hBwHfBgVT3RD9sErJhs+yTrkowmGR0bG5trGZKkSQwzLbMPcCxwAPBc4OeAI2e6fVWtr6o1VbVmZGRkrmVIkiYxzLTMK4HvVdVYVT0OXAS8FFjaT9MArAQ2D1mjJGmWhgn37wOHJ9kjSYAjgJuBK4DX9WPWAhcPV6IkabaGmXO/hu7C6fXAt/p9rQfeDZySZCPwHODMeahTkjQLS6YfMrWqOg04bULz7cBhw+xXkjQcP6EqSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaNFS4J1ma5MIk30lyS5KXJNk3yWVJbuu/7zNfxUqSZmbYM/ePAV+qql8GDgFuAU4FLq+q1cDl/bokaTuac7gn2Rt4GXAmQFU9VlUPAscCG/phG4DjhitRkjRbw5y5HwCMAX+d5BtJ/irJzwHLq+rufsw9wPLJNk6yLsloktGxsbEhypAkTTRMuC8BDgU+VVUvBn7EhCmYqiqgJtu4qtZX1ZqqWjMyMjJEGZKkiYYJ903Apqq6pl+/kC7s702yH0D/fctwJUqSZmvO4V5V9wB3JfmlvukI4GbgEmBt37YWuHioCiVJs7ZkyO3/K3Bukl2B24G30L1gXJDkJOBO4PghjyFJmqWhwr2qvgmsmaTriGH2K0kajp9QlaQGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWrQ0OGeZJck30jy+X79gCTXJNmY5DNJdh2+TEnSbMzHmfvvAbcMrH8Y+GhVHQg8AJw0D8eQJM3CUOGeZCXwauCv+vUArwAu7IdsAI4b5hiSpNkb9sz9DOBdwJP9+nOAB6vqiX59E7BiyGNIkmZpzuGe5BhgS1VdN8ft1yUZTTI6NjY21zIkSZMY5sz9pcBrk9wBnE83HfMxYGmSJf2YlcDmyTauqvVVtaaq1oyMjAxRhiRpojmHe1W9p6pWVtUq4ATgK1X1BuAK4HX9sLXAxUNXKUmalYW4z/3dwClJNtLNwZ+5AMeQJG3DkumHTK+qrgSu7JdvBw6bj/1KkubGT6hKUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGzTnck+yf5IokNye5Kcnv9e37JrksyW39933mr1xJ0kwMc+b+BPDOqjoIOBw4OclBwKnA5VW1Gri8X5ckbUdzDvequruqru+XHwFuAVYAxwIb+mEbgOOGrFGSNEvzMueeZBXwYuAaYHlV3d133QMsn2KbdUlGk4yOjY3NRxmSpN7Q4Z5kT+BvgXdU1cODfVVVQE22XVWtr6o1VbVmZGRk2DIkSQOGCvckz6IL9nOr6qK++d4k+/X9+wFbhitRkjRbw9wtE+BM4Jaq+suBrkuAtf3yWuDiuZcnSZqLJUNs+1LgjcC3knyzb3svcDpwQZKTgDuB44eqUJI0a3MO96r6GpApuo+Y634lScPzE6qS1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBCxbuSY5McmuSjUlOXajjSJKebkHCPckuwCeAo4CDgBOTHLQQx5IkPd1CnbkfBmysqtur6jHgfODYBTqWJGmCJQu03xXAXQPrm4BfHxyQZB2wrl/9YZJbF6iWndEy4L7FLmI6+fBiV6BF4O/m/Hr+VB0LFe7Tqqr1wPrFOn7LkoxW1ZrFrkOayN/N7WehpmU2A/sPrK/s2yRJ28FChfvXgdVJDkiyK3ACcMkCHUuSNMGCTMtU1RNJ3gb8PbALcFZV3bQQx9KknO7SM5W/m9tJqmqxa5AkzTM/oSpJDTLcJalBhrskNchwb0iSfZPsu9h1SFp8hvsOLsnzkpyfZAy4Brg2yZa+bdUilydpkRjuO77PAJ8FfqGqVlfVgcB+wN/RPdNHWlRJlic5tP9avtj17Cy8FXIHl+S2qlo92z5poSV5EfA/gb156hPqK4EHgd+tqusXp7Kdg+G+g0tyPnA/sIGnHta2P7AWWFZVxy9Wbdq5Jfkm8DtVdc2E9sOBT1fVIYtS2E7CcN/B9Y93OInukcor+uZNwOeAM6vq0cWqTTu3ad5VbuynELVADHdJCyLJx4EXAOew9bvKNwHfq6q3LVZtOwPDvWFJjqmqzy92Hdp5JTmKrd9VbgYuqapLF6+qnYPh3rAkH6iq0xa7Dknbn+HegCS/zORnR7csXlXS1JKs6/9gjxaI97nv4JK8m+5+9gDX9l8Bzkty6mLWJm1DFruA1nnmvoNL8l3g4Kp6fEL7rsBN3ueuZ6Ikb6mqv17sOlrmmfuO70nguZO079f3Sc9EH1jsAlq3aH8gW/PmHcDlSW7jqdvNngccCHirmRZNkhun6gJ8DMECc1qmAUl+BjiMrS+ofr2qfrJ4VWlnl+Re4FXAAxO7gH+uqsnecWqeeObegKp6Erh6seuQJvg8sGdVfXNiR5Irt3s1OxnP3CWpQV5QlaQGGe6S1CDDXZIaZLhLUoP+P+ZdkkK4jB4fAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Divide by class\n",
    "df_class_1_under = df_class_1.sample(count_class_0)\n",
    "df_us = pd.concat([df_class_1_under, df_class_0], axis=0)\n",
    "\n",
    "print(\"Random under-sampling:\")\n",
    "print(df_us.pass_or_fail.value_counts())\n",
    "\n",
    "df_us.pass_or_fail.value_counts().plot(kind=\"bar\", title=\"Count (Pass(1)/Fail(0))\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480baa8f",
   "metadata": {},
   "source": [
    "##### Oversampling\n",
    "Here, we add instances randomly to the under-represented class i.e. for our dataset the 'Fail (0)' class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0cc5eea1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random over-sampling:\n",
      "0.0    9837\n",
      "1.0    9837\n",
      "Name: pass_or_fail, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:title={'center':'Count (target)'}>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEOCAYAAABrSnsUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAASd0lEQVR4nO3de5BedX3H8fdHIihiSSI7CAkYKqkW7ag0Qhwv40iHi5eGP5SiVqJDm06FVmu94GUMonSw05bLqLQZiQShXIoXoqJMRB11lEsQRRExOyAmkctKLqBUMPLtH89v4Um6m2X32eyTZN+vmWf2nN/vd875blj2s+d3znmeVBWSpOntSf0uQJLUf4aBJMkwkCQZBpIkDANJEoaBJAnDQJo0SQaS/CzJU/tdy0iS7NXqG+h3Ldr5GAbapSR5U5LVSX6T5O4kX03ysik4biU5dIxhpwEXVtX/tm2+leRvdnRto9n2+FX1MLCcTp3SVgwD7TKSvAs4B/gXYH/gYOBTwKI+lgV0/uoGFgMXT+I+Z0zWvrr8N7C41Ss9xjDQLiHJvsAZwClV9fmq+m1V/b6qvlRV72lj9kpyTpJftdc5w7/0krw1yXe32edjf+0nuTDJJ5N8JcmDSa5P8uzW9+22yY/aGclfjVDikcCmqlrXtjkTeDnwibbNJ1r7uUnWJnkgyU1JXt5Vz+lJrkxycZIHgLcmOSTJt1tNX281Xty1zcIk30uyKcmPkrxye8dv9W0EFk78v4Z2R4aBdhUvAZ4CfGE7Yz5I55fcC4EXAEcAHxrHMU4EPgLMAgaBMwGq6hWt/wVVtU9VXT7Ctn8G3D68UlUfBL4DnNq2ObV13djqm03nr/T/SfKUrv0sAq4EZgKXtDE3AM8ATgfeMjwwyRzgK8DH2v7eDXwuycB2jg9wG51/H+kxhoF2Fc8Afl1VW7Yz5s3AGVV1X1UN0fnF/pbtjN/WF6rqhnaMS+j80n6iZgIPjjWoqi6uqvuraktV/TuwF/CcriHfr6ovVtWjwADwYuDDVfVIVX0XWNk19q+Bq6vq6qp6tKpWAauBV49RxoOtXukxhoF2FfcD+40xj34gcFfX+l2t7Ym6p2v5IWCfcWy7EXj6WIOSvDvJbUk2J9kE7Avs1zVkbdfygcCGqnpolP5nAW9oU0Sb2v5eBhwwRhlPBzaNVaumF8NAu4rvAw8Dx29nzK/o/IIcdnBrA/gtsPdwR5JnTnJ9twB/sk3bVm8J3K4PvBc4AZhVVTOBzUBG2eZuYHaSvbvaDupaXgt8tqpmdr2eVlVnjXT8Ln8K/OgJfE+aRgwD7RKqajPwYeCTSY5PsneSJyc5Lsm/tmGXAh9q9/vv18YPX2z9EfC8JC9sc/Snj7OEe4E/3k7/DcDMNo8/2jZPB7YAQ8CMJB8G/mi0HVbVXXSmfU5PsmeSlwCv6xpyMfC6JMck2SPJU5K8Msnc0Wpu9c0GrtvO96JpyDDQLqPNsb+LzkXhITp/GZ8KfLEN+RidX563AD8GftDaqKqf07kb6evAGmCrO4uegNOBFW065oQRansEuJDOPP6wc4HXJ9mY5DzgGuBrwM/pTGH9jq2nfUbyZjoXz+9v38vldM6QqKq1dC44f4DH/z3ew+P/X297fIA3ASvaMwfSY+KH20iToz3Z+x3gRcMPnu2AY1wO/Kyqlk5g273onCG9oqrum/TitEszDKSdWJIXAxuAO4Gj6ZwFvaSqbu5nXdr97IgnHCVNnmcCn6dza+064O8NAu0InhlIkryALEnahaeJ9ttvv5o3b16/y5CkXcZNN93066oa8S3Md9kwmDdvHqtXr+53GZK0y0hy12h9ThNJkgwDSdITCIMky5Pcl+QnXW2zk6xKsqZ9ndXak+S8JINJbklyeNc2i9v4NUkWd7X/eZIft23OSxIkSVPqiZwZXAgcu03bacC1VTUfuJbHP0bvOGB+ey0BzodOeABL6XwAyBHA0uEAaWP+tmu7bY8lSdrBxgyDqvo2nScguy0CVrTlFTz+TpKLgIuq4zo6b9x1AHAMsKqqNlTVRmAVcGzr+6Oquq46DzxcxPbflVKStANM9JrB/lV1d1u+h87n0QLMYes33lrX2rbXvm6E9hElWZLOh6GvHhoammDpkqRt9XwBuf1FPyWPMVfVsqpaUFULBgZGvFVWkjQBEw2De9sUD+3r8DsgrmfrD9+Y29q21z53hHZJ0hSaaBisBIbvCFoMXNXVflK7q2ghsLlNJ10DHJ1kVrtwfDRwTet7IMnCdhfRSV37kiRNkTGfQE5yKfBKOp8/u47OXUFnAVckOZnOh3QMf9jH1XQ+jHuQzmfIvg2gqjYk+ShwYxt3RlUNX5R+O507lp4KfLW9dgvzTvtKv0vYrfzirNf0u4Tdij+fk2tX//kcMwyq6o2jdB01wtgCThllP8uB5SO0rwaeP1YdkqQdxyeQJUmGgSTJMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJJEj2GQ5J+S3JrkJ0kuTfKUJIckuT7JYJLLk+zZxu7V1gdb/7yu/by/td+e5JgevydJ0jhNOAySzAH+EVhQVc8H9gBOBD4OnF1VhwIbgZPbJicDG1v72W0cSQ5r2z0POBb4VJI9JlqXJGn8ep0mmgE8NckMYG/gbuBVwJWtfwVwfFte1NZp/UclSWu/rKoerqo7gUHgiB7rkiSNw4TDoKrWA/8G/JJOCGwGbgI2VdWWNmwdMKctzwHWtm23tPHP6G4fYRtJ0hToZZpoFp2/6g8BDgSeRmeaZ4dJsiTJ6iSrh4aGduShJGla6WWa6C+AO6tqqKp+D3weeCkws00bAcwF1rfl9cBBAK1/X+D+7vYRttlKVS2rqgVVtWBgYKCH0iVJ3XoJg18CC5Ps3eb+jwJ+CnwTeH0bsxi4qi2vbOu0/m9UVbX2E9vdRocA84EbeqhLkjROM8YeMrKquj7JlcAPgC3AzcAy4CvAZUk+1touaJtcAHw2ySCwgc4dRFTVrUmuoBMkW4BTquoPE61LkjR+Ew4DgKpaCizdpvkORrgbqKp+B7xhlP2cCZzZSy2SpInzCWRJkmEgSTIMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJNFjGCSZmeTKJD9LcluSlySZnWRVkjXt66w2NknOSzKY5JYkh3ftZ3EbvybJ4l6/KUnS+PR6ZnAu8LWqei7wAuA24DTg2qqaD1zb1gGOA+a31xLgfIAks4GlwJHAEcDS4QCRJE2NCYdBkn2BVwAXAFTVI1W1CVgErGjDVgDHt+VFwEXVcR0wM8kBwDHAqqraUFUbgVXAsROtS5I0fr2cGRwCDAGfSXJzkk8neRqwf1Xd3cbcA+zflucAa7u2X9faRmuXJE2RXsJgBnA4cH5VvQj4LY9PCQFQVQVUD8fYSpIlSVYnWT00NDRZu5Wkaa+XMFgHrKuq69v6lXTC4d42/UP7el/rXw8c1LX93NY2Wvv/U1XLqmpBVS0YGBjooXRJUrcJh0FV3QOsTfKc1nQU8FNgJTB8R9Bi4Kq2vBI4qd1VtBDY3KaTrgGOTjKrXTg+urVJkqbIjB63/wfgkiR7AncAb6MTMFckORm4Czihjb0aeDUwCDzUxlJVG5J8FLixjTujqjb0WJckaRx6CoOq+iGwYISuo0YYW8Apo+xnObC8l1okSRPnE8iSJMNAkmQYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgSWISwiDJHkluTvLltn5IkuuTDCa5PMmerX2vtj7Y+ud17eP9rf32JMf0WpMkaXwm48zgHcBtXesfB86uqkOBjcDJrf1kYGNrP7uNI8lhwInA84BjgU8l2WMS6pIkPUE9hUGSucBrgE+39QCvAq5sQ1YAx7flRW2d1n9UG78IuKyqHq6qO4FB4Ihe6pIkjU+vZwbnAO8FHm3rzwA2VdWWtr4OmNOW5wBrAVr/5jb+sfYRtpEkTYEJh0GS1wL3VdVNk1jPWMdckmR1ktVDQ0NTdVhJ2u31cmbwUuAvk/wCuIzO9NC5wMwkM9qYucD6trweOAig9e8L3N/dPsI2W6mqZVW1oKoWDAwM9FC6JKnbhMOgqt5fVXOrah6dC8DfqKo3A98EXt+GLQauassr2zqt/xtVVa39xHa30SHAfOCGidYlSRq/GWMPGbf3AZcl+RhwM3BBa78A+GySQWADnQChqm5NcgXwU2ALcEpV/WEH1CVJGsWkhEFVfQv4Vlu+gxHuBqqq3wFvGGX7M4EzJ6MWSdL4+QSyJMkwkCQZBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJ9BAGSQ5K8s0kP01ya5J3tPbZSVYlWdO+zmrtSXJeksEktyQ5vGtfi9v4NUkW9/5tSZLGo5czgy3AP1fVYcBC4JQkhwGnAddW1Xzg2rYOcBwwv72WAOdDJzyApcCRwBHA0uEAkSRNjQmHQVXdXVU/aMsPArcBc4BFwIo2bAVwfFteBFxUHdcBM5McABwDrKqqDVW1EVgFHDvRuiRJ4zcp1wySzANeBFwP7F9Vd7eue4D92/IcYG3XZuta22jtIx1nSZLVSVYPDQ1NRumSJCYhDJLsA3wOeGdVPdDdV1UFVK/H6NrfsqpaUFULBgYGJmu3kjTt9RQGSZ5MJwguqarPt+Z72/QP7et9rX09cFDX5nNb22jtkqQp0svdRAEuAG6rqv/o6loJDN8RtBi4qqv9pHZX0UJgc5tOugY4OsmsduH46NYmSZoiM3rY9qXAW4AfJ/lha/sAcBZwRZKTgbuAE1rf1cCrgUHgIeBtAFW1IclHgRvbuDOqakMPdUmSxmnCYVBV3wUySvdRI4wv4JRR9rUcWD7RWiRJvfEJZEmSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAksROFQZJjk9yeZDDJaf2uR5Kmk50iDJLsAXwSOA44DHhjksP6W5UkTR87RRgARwCDVXVHVT0CXAYs6nNNkjRtzOh3Ac0cYG3X+jrgyG0HJVkCLGmrv0ly+xTUNh3sB/y630WMJR/vdwXqE38+J8+zRuvYWcLgCamqZcCyftexu0myuqoW9LsOaST+fE6NnWWaaD1wUNf63NYmSZoCO0sY3AjMT3JIkj2BE4GVfa5JkqaNnWKaqKq2JDkVuAbYA1heVbf2uazpxKk37cz8+ZwCqap+1yBJ6rOdZZpIktRHhoEkyTCQJBkG01qS2Ulm97sOSf1nGEwzSQ5OclmSIeB64IYk97W2eX0uT1KfGAbTz+XAF4BnVtX8qjoUOAD4Ip33hJL6Lsn+SQ5vr/37Xc904K2l00ySNVU1f7x90lRI8kLgP4F9efxdCOYCm4C3V9UP+lPZ7s8wmGaSXAZsAFbw+JsDHgQsBvarqhP6VZuU5IfA31XV9du0LwT+q6pe0JfCpgHDYJppb/dxMp23CJ/TmtcBXwIuqKqH+1WbNMaZ62Cb1tQOYBhI2mkkOQ94NnARW5+5ngTcWVWn9qu23Z1hoMckeW1VfbnfdWh6S3IcW5+5rgdWVtXV/atq92cY6DFJPlJVS/tdh6SpZxhMQ0mey8h/ed3Wv6qk7UuypH3AlXYAnzOYZpK8j87zBAFuaK8AlyY5rZ+1SWNIvwvYnXlmMM0k+TnwvKr6/TbtewK3+pyBdlZJ3lZVn+l3Hbsrzwymn0eBA0doP6D1STurj/S7gN3ZTvFJZ5pS7wSuTbKGx2/dOxg4FPC2PfVVkltG6wJ8W4odyGmiaSjJk4Aj2PoC8o1V9Yf+VSVBknuBY4CN23YB36uqkc5qNQk8M5iGqupR4Lp+1yGN4MvAPlX1w207knxryquZRjwzkCR5AVmSZBhIkjAMJEkYBpIk4P8AF0JKazCzec0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Divide by class\n",
    "df_class_0_over = df_class_0.sample(count_class_1, replace=True)\n",
    "df_os = pd.concat([df_class_1, df_class_0_over], axis=0)\n",
    "\n",
    "print(\"Random over-sampling:\")\n",
    "print(df_os.pass_or_fail.value_counts())\n",
    "\n",
    "df_os.pass_or_fail.value_counts().plot(kind=\"bar\", title=\"Count (target)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80b09e52",
   "metadata": {},
   "source": [
    "#### SVM\n",
    "Now, we will fit the SVM model to calculate the probabilities. Let's run it for both the under and over sampled data and compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "af94fe80",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create X = test_duration and y = pass_or_fail and calculate probabilty for under sampled data\n",
    "X, y = np.array([df_us[\"test_duration\"].values]), np.array(\n",
    "    [df_us[\"pass_or_fail\"].values]\n",
    ")\n",
    "X = X.reshape(-1, 1)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "# split into train/test sets.\n",
    "trainx, testx, trainy, testy = train_test_split(\n",
    "    X, y, train_size=0.8, test_size=0.2, random_state=42\n",
    ")\n",
    "\n",
    "# fit a SVM Model.\n",
    "model = SVC(probability=True).fit(trainx, trainy)\n",
    "\n",
    "# Evaluate the model\n",
    "y_pred = model.predict(testx)\n",
    "\n",
    "# predict probabilities.\n",
    "probs_train = model.predict_proba(trainx)[:, 0]\n",
    "probs_test = model.predict_proba(testx)[:, 0]\n",
    "probs = np.append(probs_train, probs_test, axis=None)\n",
    "df_us[\"prob\"] = probs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0164484e",
   "metadata": {},
   "source": [
    "*NOTE* : The probabilities are not normalized, but can be normalized when calling the `calibration_curve()` function by setting the `normalize` argument to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "30f16507",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVM Model accuracy : 0.8888888888888888\n",
      "SVM Brier Score 0.1111111111111111\n",
      "SVM Roc Score 0.8859890109890111\n"
     ]
    }
   ],
   "source": [
    "# Mean accuracy on the given test data and labels.\n",
    "svm_accuracy = model.score(testx, testy)\n",
    "print(\"SVM Model accuracy :\", svm_accuracy)\n",
    "svm_brier = brier_score_loss(testy, y_pred, pos_label=1)\n",
    "print(\"SVM Brier Score\", svm_brier)\n",
    "svm_roc = roc_auc_score(testy, y_pred)\n",
    "print(\"SVM Roc Score\", svm_roc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce82ef9b",
   "metadata": {},
   "source": [
    "Brier Score is a distance in the probability domain. Which means: the lower the value of this score, the better the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c86e13",
   "metadata": {},
   "source": [
    "The above calculated score is a harsh metric since we require for each sample that each label set be correctly predicted. This does not give us a clear indication of a better model so we will use the reliability diagram to find out the better suited model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dec9b510",
   "metadata": {},
   "source": [
    "##### Reliability Diagram\n",
    "\n",
    "A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency  (x-axis). These plots are commonly referred to as ‘reliability‘ diagrams in forecast literature, although may also be called ‘calibration‘ plots or curves as they summarize how well the forecast probabilities are calibrated.\n",
    "\n",
    "The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0b05c577",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABAPUlEQVR4nO3dd3hUZfbA8e9JIHRCSUDpvXcjSAtYAVdBsWFndWVdF3VFRV13V3fd9WdFVGxYERVEsaCo2Am9SUdQpAaQTigh/fz+uDcyxJQJzMzNzJzP88yTuWXuPTeBe+a9773nFVXFGGNM9IrxOgBjjDHeskRgjDFRzhKBMcZEOUsExhgT5SwRGGNMlLNEYIwxUc4SgYl4IrJJRM5x3/9dRF7x83NviMh/i1l+WESaFVxXRPqKyLpAxO4vEVktIv1DuU8TOSwRmFIRkT4iMldE0kRkn4jMEZHTReQMETkiIlUL+cxSERkpIk1EREVkaYHlCSKSJSKbitmvuts/LCLbRGSMiMSWNn5VfVhV/1TazxWxraqquqGQ+bNUtXX+tG8iKi2f39lh97VTRD4VkXML7LO9qn5/IvswxhKB8ZuIVAc+BZ4FagH1gX8Dmao6H0gFLi3wmQ5AO2CSz+zK7vx8VwEb/Qihs6pWBfoBVwA3nOChhKMa7rF3Br4CPhSR4cHeqYiUC/Y+jPcsEZjSaAWgqpNUNVdVj6rql6q6wl0+AbiuwGeuAz5T1b0+8yYC1xdY501/g1DV9cAcoEv+PBG5QESWicgBt8XSqbDPisiDIvKWz/R7IvKr28JJEZH2BT6SICJficghEZkpIo19Pqsi0qKQffQXkVT3/USgEfCJ+41+tIhMF5FbC3xmhYhc7Mex/6qqTwMPAo+KSIz7ed/LX91FZJ77u9ghIuNEJM5nX+eJyDr3mJ93j+tP7rLhbivvKRHZCzwoIs1F5FsR2Ssie0TkbRGp4bO9TSJyt3sMR0TkVRGpKyKfu7+3r0WkZknHZrxjicCUxk9ArohMEJFBhfznnggki0hDAPckdRVOgvD1FjBMRGJFpB1QFVjgbxAi0gboC6x3p7sCrwF/BmoDLwHTRKSCH5v7HGgJ1AF+AN4usPxq4CEgAVhWyPJiqeq1wBbgQvdS0mM4v49rfI6nM07ranopNv2BG3PrQpblAne4MfcEzgZucfeVALwP3Ifzu1oH9Crw+R7ABqAu8D9AgP8D6gFtgYY4icjXJcC5OF8WLsT5vf4dSMQ5z9xWimMzIWaJwPhNVQ8CfQAFXgZ2i8g0EanrLt8KfA9c637kbKACvz/BpeKcgM7BaQ1M9DOEH0TkCPCju5/n3fkjgJdUdYHbUpkAZAJn+HFMr6nqIVXNxDm5dRaReJ9Vpqtqirv8fqBnfqI7CdOAViLS0p2+FnhXVbNKsY3t7s9aBReo6hJVna+qOaq6CScx9nMXnw+sVtUPVDUHeAb4teC2VfVZ9/NHVXW9qn6lqpmquhsY47O9fM+q6k5V3QbMAhao6lJVzQA+BLqW4thMiFkiMKWiqj+q6nBVbQB0wPmWONZnlQkcSwTXApNVNbuQTb0JDAeuxP9E0A2n9XAFzrfWKu78xsCd7qWQAyJyAOdba73iNua2SB4RkV9E5CCwyV2U4LPa1vw3qnoY2FfSdkvinhzfBa5xW02l+R3kq+/+3FdwgYi0cjuUf3WP62GOHVM9jj8mxUnMvrb6TriXeSa7nfQHcVp0CQU+s9Pn/dFCpn93E4EpOywRmBOmqmuBN3ASQr4PgAYiciYwlN9fFso3FfgDsEFVt5Rin6qqU4B5wL/c2VuB/6lqDZ9XZVWdVPSWAOey1RCclkk80MSdLz7r/PbtX5w7ompx7Nu432EXMm8CzmWns4F0VZ1Xym1eDOzCaVkV9AKwFmipqtVxLtHkH9MOoEH+iiIivtNFxPuwO6+ju71rOP53ZMKcJQLjNxFpIyJ3ikgDd7ohzrfZ+fnrqOoRnGvQrwObVXVxYdty1zsLONFbOR8BbhKRU3AuU90sIj3EUUVE/iAi1UrYRjWcS0h7gco4J7yCzhfnltk4nL6C+e4lsNLYCTTzneGe+POAJylFa8D9dj4SeAC4T1XzClmtGnAQOOz2p/zFZ9l0oKOIXOTeEfRX4JQSdlsNOAykiUh94G5/4zXhwRKBKY1DOJdkFrjX6ucDq4A7C6w3AedyTbF3AqnqYlX95UQCUdWVQApwt5tsbgLGAftxOpGH+7GZN4HNwDZgDT4Jzcc7OCfdfcBp+HTylsL/Af9wL1vdVWD/HXEutZTkgPs7X4lznf8yVX2tiHXvwmntHMJJku/mL1DVPcBlwGM4CbAdsBgnIRbl3ziX5dJwEskHfsRrwojYwDTGeENErgNGqGofD2OIwekjuFpVv/MqDuMtaxEY4wERqYxzS+d4D/Y9QERquLfX5vcfFNYaMlHCEoExISYiA4DdOH0H73gQQk/gF2APzj3/F6nqUQ/iMGWEXRoyxpgoZy0CY4yJcmFXUCohIUGbNGnidRjGGBNWlixZskdVEwtbFnaJoEmTJixeXOit6cYYY4ogIpuLWmaXhowxJspZIjDGmChnicAYY6KcJQJjjIlylgiMMSbKBS0RiMhrIrJLRFYVsVxE5BkRWe8OcdctWLEYY4wpWjBbBG8AA4tZPghniMCWOCNMvRDEWMzWhTDrSeenMcb4CNpzBKqaIiJNilllCPCmO0LSfLcI1qmquiNYMUWtrQthwoWQkwnlKsL106Bhd6+jMsb4KT0rh72Hs2hYq3JQtu9lH0F9jh8SL5Vjw+8dR0RGiMhiEVm8e/fukAQXUTbNcpIACrmZzrQxJizMXb+HgWNncfNbS8jLC05tuLDoLFbV8aqapKpJiYmFPiFtitOoF7+NPqgKjT0rf2+M8VPa0WzunbqCq15ZQIzAPy9oR0xMcEYI9bLExDZ8xoPFGTd1m0exRLbsdOdn7Zaw92fIPOhtPMaYYuXmKZe8MJcNuw/z537NuOOcVlQsHxu0/XmZCKYBI0VkMs7wh2nWPxAkK96FivEw4jt4vifMfAxanANi448bU5bsP5JFjcrliY0R7jqvNfVqVKRTgxpB328wbx+dBMwDWotIqojcKCI3i8jN7iqfARtwxpd9GWe0JhNomYfhx0+g/cVQoRr0vh1SF8LGmV5HZoxxqSofLk3lzCe/Z/Iip+t0YIdTQpIEILh3DV1ZwnIF/hqs/RvX2k+dS0OdhjnTXa+FlCdg5uPQrL+noRljYPuBo9z/4Uq+W7ebro1qkNS4ZshjCLsy1KaUlk+GGo2h0RnOdPmKTqtgxn2weS407uVtfMZEsY+XbeP+D1eRm6f864J2XN+rCbFB6hAuTljcNWRO0MEdziWgTlcc3x9w2nCokuj0FRhjPBNfqTxdGtbgyzuSuaFPU0+SAFgiiGwr3wPNg87Djp8fVxl63QobvoOti7yJzZgolJObx0szf2Hctz8D0L91HSbe2D1oD4r5yxJBJFsxBeonQe3mv1+WdCNUqgUp1iowJhTWbD/Ixc/P5f8+X8uPvx7C6SYFKQN371kiiFQ7V8POlb9vDeSrUBV63gI/fwnbl4Y2NmOiSGZOLk9+uY7B42azI+0oz1/djXFXdi0TCSCfJYJItXwyxJSD9kOLXqf7COf5gpQnQheXMVFm0550Xpz5C4O71OOrO/pxfsdTy1QSAEsEkSkv1+kfaHEuVKld9HoV46HHzc4tpr8WWi3cGHMCjmTm8NFSp1BC61Oq8c2o/oy5vAs1q8R5HFnhLBFEoo0pcGgHdL6i5HV73AxxVWGWtQqMCYRZP+9mwNgU7piyjPW7DgHQqLa3ncElsUQQiVa8CxXiodWgktetXAu63wSrP4Ld64IemjGRKi09m9HvL+faVxcSFxvDuyN60qJONa/D8oslgkiTdQTWTIP2Q5yHx/zRcySUr2R9BcacoNw85ZIX5zL1h23c0r85n93el+5Na3kdlt/syeJIs3Y6ZB85VlLCH1USIOkGmP889L+38NtNjTG/s+9IFjUqOUXi7h7Qmvo1KtGhfrzXYZWatQgizfLJEN8IGvUs3ed63QaxcTBrTHDiMiaCqCpTl6Ry5hPHisQNaH9KWCYBsEQQWQ7tdJ4W7nQ5xJTyT1utLnS7HlZMhv2bgxOfMREgdX8617++iDvfW06LOlXD6hJQUSwRRJJV7zslJTr5cbdQYXrfDhIDs58KbFzGRIgPl6Yy4KkUFm/ax78Ht+e9P/ekRZ2qXod10iwRRJLlk6FeV0hsdWKfj68PXa6GZW9Dmg0WZ0xBtapU4LQmtfjyjmSu79UkaENHhpolgq0LYdaTzs9wtutH+HVF6TqJC9PnDqdVMefpwMRlTBjLzs3j+e/X88w3TpG4fq0SmfDH02lQs2w/F1Ba0X3X0NaF8MYFkJsJ5SrC9Z9Aw+5eR3Vilk8GiYUOl5zcdmo2duoTLXkD+o6CaqcEJDxjws2qbWncM3UFq7cf5MLO9VBVRKTMlYcIhOhuEWyaBXnZzvucDOeJ3HCUl+eWlDgHqiae/Pb6jHJ+L3OfPfltGRNmMrJzeeyLtQx5bg47D2by4jXdeLaMFYkLtOhOBE36QmwFfvs17PnZ03BO2KZZcHCbfyUl/FG7OXS8DBa/Bkf2BGabxoSJzXvTeXnWBoZ2rc83o/oxsMOpXocUdNGdCBp2h+unwdn/gNbnO7dOrpnmdVSlt+JdiKvmHEOg9L0Lso/CvHGB26YxZdSRzBw++CEVcIrEfXtnfx6/rDPxlct7HFloRHcfATjJoGF3yMmE1wfBR3+BxDYnfudNqGWlw5qPod1FTpmIQElsBe0vgoUvOw+bVQ7/e6WNKczMn3bz9w9Wsj3tKJ0axNOiTjXPRwwLtehuEfgqVwEun+h0Gr97NWQc9Doi/6z7DLIOB+6ykK/ku51tL3gx8Ns2xmP7j2Qxasoyrn9tIRXLx7jPBIRHkbhAs0TgK74+XPYG7P3FaRm4Q8mVaSveheoNoHGfwG+7bntocwHMfxEy0gK/fWM8kl8k7uNl2xl5Zgum39aXpCbR2+q1RFBQ075w3kPOYC1l/Qnbw7tg/TfQ6bLSl5TwV/LdkJkGC8YHZ/vGhNDew5nk5SmxMcK9A9swbWRv7hrQmorlY70OzVOWCApzxi3Q4VL49iHnRFtWrZoKmnvyD5EVp14XaDkA5j8HmYeCtx9jgkhVmbJ4K2c+8T2TFm0B4Lz2p9C+XngWiQs0SwSFEYHBz0BiW5h6Y9ktwrZ8MpzaGeq0Ce5++o2Go/th0avB3Y8xQbB1XzrXvbaQ0e+voM0p1enZrJjhW6OUJYKixFWBYW855Rbevca5lbIs2b0OdiwLbmsgX4MkaHamcytpVnrw92dMgHzwQyoDxqbww+b9PHRRByaPOINmieFfJC7QLBEUp1YzGPoK/LoSPvlb2eo8zi8p0fHS0Oyv32g4stspPWFMmEioWoHuTWvx5ah+XHtG44gpEhdolghK0uo86H+f87DZole8jsaRX1Ki+ZlQtU5o9tm4l/Mk9pynITsjNPs0ppSyc/MY9+3PPP21UyUguVUib/yxO/VrBPAZmwhkicAfyXdDq4Hwxb2wZb7X0cDmOZC2NTSXhXwl3w2Hf4WlE0O7X2P8sGpbGoPHzeGJL39iw57DaFlqwZdxlgj8ERMDF78ENRrBlOvg0K/exrNiMsRVhTZ/CO1+myZDwx4weyzkZIV238YUISM7l0c+d4rE7TmcyUvXnsbTwyK7SFygBTURiMhAEVknIutF5N5CljcSke9EZKmIrBCRABbLCbBKNeCKt51bKKdc792JMPuoUw+p7WCIC/Fj8CKQPBoOpsLyd0K7b2OKsGVfOq/O3sCl3Rrw9R39GNDeSqeXVtASgYjEAs8Bg4B2wJUi0q7Aav8ApqhqV2AY8Hyw4gmIuu1gyHOwdT58eb83Maz7HDIPBqekhD9anO2MgjZrDORmexODiXqHMrJ5b7EzaHyrutX47q7+PHppp6gpEhdowWwRdAfWq+oGVc0CJgNDCqyjQHX3fTywPYjxBEaHodDrVlg4HpZNCv3+V7wL1eo5HbdeyG8VHNjsdFgbE2Lfrd3FgKdSuGfqCtbvch5yjLQRw0ItmImgPrDVZzrVnefrQeAaEUkFPgNuLWxDIjJCRBaLyOLdu3cHI9bSOftB50T86d9gx/LQ7ffIHlj/tVtSwsNH4lsPgrodnSE+83K9i8NElX1Hsrjj3WX88Y1FVKlQjvf/0itqi8QFmtedxVcCb6hqA+B8YKKI/C4mVR2vqkmqmpSYGIARuE5WbDmnOF3lBOdhs/R9odnvqqmQlxP6u4UKEoF+d8Pe9bD6Q29jMVEhN0+59IW5fLJ8O7ed3ZJPb+tDt0Y1vQ4rYgQzEWwDGvpMN3Dn+boRmAKgqvOAikBCEGMKnCoJcMWbzh1EU28MzTfj5ZOdb+J1C3a1eKDNhU4JjpTHnecajAmC3YeOFYn7+/lt+eTWPow6txUVykV3kbhAC2YiWAS0FJGmIhKH0xlccPivLcDZACLSFicRlIFrP36qfxr84Un45Vv49r/B3deen2H7D951EhcUEwPJd8HutfBjGI7qZso0VeXdRVs468nveWehUyTunHZ1aXtq9RI+aU5E0BKBquYAI4EZwI84dwetFpH/iMhgd7U7gZtEZDkwCRiu4fYUSLfr4LThMHtMcIe5XD4ZJMYZS7isaH8x1G4BKU+UrfIbJqxt2ZvO1a8s4J6pK2l3anX6tAiPiwThLKhDVarqZzidwL7z/uXzfg3QO5gxhMSgx5x6RMEa5jIvD1ZMgWb9oVoZukc6Jhb63ukc97rPoU3ZfQzEhIf3l6Tyz49WERsj/O/iDlx5eiOrDxQCXncWR4ZgD3O5dT6kbfG+k7gwHS+DGo0h5TFrFZiTVrd6BXo1r81Xo5K5uocViQsVSwSBEl8fLp8QnGEul0+G8lWg7QWB22agxJZ3WgXbl5btQXxMmZSVk8fTX//MU1/9BEDflom8Ovx0To23InGhZIkgkJr0gfP+G9hhLrMzYPVH0PZCZ4yEsqjzlRDfEGY+aq0C47flWw9w4bOzeerrn9i6L92KxHnIEkGgnfGXwA5z+dMXzpjBnS4/+W0FS7k46H07pC6EjTO9jsaUcUezcvnf9DVc/Pwc0o5m88p1SYy5oosVifOQJYJAC/QwlyvehaqnOB3FZVnXa504Zz7udSSmjNu6P50JczczrHsjvhyVzDnt6nodUtSzRBAMgRrm8she+PlLZxQyL0tK+KN8RadVsHk2bJ7rdTSmjDmYkc0UnyJx39/dn4cv7kj1ilYkriywRBAsvsNcfnrHiV07X/2BU1Kicxm8W6gwpw2HKokw8zGvIzFlyLdrd3LemBTunbqC9bsOA1DPRgwrU0pMBCLypIi0D0UwESd/mMvlk05smMvlk6FOezilY+BjC4a4ytBzJGz4DlIXex2N8djew5ncPnkpN7yxmPhK5fnglt60qGMDx5dF/rQIfgTGi8gCEblZROKDHVREOdFhLvesh22Ly05JCX+d/ieoVMtaBVEuN0+57MV5fLZyB3ec04pPbu1Dl4Y1vA7LFKHERKCqr6hqb+A6oAmwQkTeEZEzgx1cRPhtmMvGpRvmcuUUQMpWSQl/VKgKPW+Bn2c4zxaYqLLrUMZvReLu/0NbPr21L7ef05K4cnYVuizz66/jjjbWxn3tAZYDo0RkchBjixyVasAVb0HmYf+GuVR17hZq1g+q1wtJiAHVfQRUjHdqEJmokJenvL1gM2c9MZO33SJxZ7etS+tTbLyAcOBPH8FTwFqc8QIeVtXTVPVRVb0Q6BrsACNG3XYwZJx/w1xuXQD7N0GnMLsslK9iPPS42Xmw7tdVXkdjgmzTniNc9cp87v9wFZ0axNOvZRkYM8SUij8tghVAF1X9s6ouLLCsexBiilz+DnO5fDKUq+Q8TRyuetwMcVVhlrUKItmUxVsZMDaF1dsO8sjQjrz9px40qm3DRoYbfxLBAXyqlIpIDRG5CEBV04ITVgQraZjLnExn1K+2F0CFMG5WV64F3W9yymPsXud1NCZI6teoRHKrRL4a1Y9h3RvZ08Fhyp9E8IDvCV9VDwAPBC2iSFfSMJc/zYCMA2Wz0mhp9RwJ5Ss5YxubiJCZk8tTX/3EmC+d5N67RQIvX5fEKfEVPY7MnAx/EkFh6wR1HIOIV9wwlyvehSp1yn5JCX9USYCkG2Dle05VVhPWlm7Zz4XPzubpb35m24EMKxIXQfxJBItFZIyINHdfY4AlwQ4s4vkOc/nd/5x56fucFkHHy5yWQyTodRvExsGsMV5HYk5QelYOD326hqEvzOVQRg6vDU/iycs722WgCOLP2eZW4J/Au+70V8BfgxZRNOl2HWxb4lw6qdcVDu+EvOzwe4isONXqQrfrYfGr0G801GzsdUSmlLbtP8rE+Zu5ukcj7hnYhmpWHyjiSLg175KSknTx4ggqX5CTCa+fDztXO8Xq4qrA7cudKqaRIm0bPNMFulwNF471Ohrjh7Sj2Xy+cgfDujcCYEfaURssJsyJyBJVTSpsmT/PEbQSkfEi8qWIfJv/CnyYUapcBegzCnKOQvoeOLgNUhd5HVVgxdd3ksCyt52kYMq0L1f/yrljZnL/R6t+KxJnSSCy+dNH8B6wFPgHcLfPywTKnrUg7p9CFTbN8jaeYOhzh1OWe87TXkdiirDncCYj3/mBEROXUKtKHB/e0suKxEUJf/oIclT1haBHEs2a9IXYCpCb5XSsNunrdUSBV7OxU077hwnOGMfVbDCSsiQ3T7n0hblsP5DBXee14s/9mlM+1uoDRYsS+whE5EFgF/AhkJk/X1X3FfWZYIq4PoJ8Wxc6LYEmfaFhhD6wvfcXGJcEZ9wCA/7ndTQG2Hkwg8SqFYiJEb5bu4sGNSvRsm4YP8hoilRcH4E/iWBjIbNVVZsFIrjSithEEC0+GAE/fgJ/W+k8Z2A8kZenvL1wC49+vpZ7Brbm2p5NvA7JBNlJdRaratNCXp4kARMB+t7lDN05b5zXkUStDbsPM+zl+fzzo1V0aViD/q3reB2S8Zg/dw1VFpF/iMh4d7qliFwQ/NBMREpsBe0vgoUv/768hgm6dxdtYdDTs1i74yCPXdqJiTd2p2EtKxIX7fzpDXodyAJ6udPbgP8GLSIT+ZLvhqzDsOBFryOJOg1qVqZ/60S+HtWPy5Ma2tPBBvAvETRX1ceAbABVTQfsX485cXXbQ5sLYP6LkGEFbIMpMyeXJ2as44kZx4rEvXRtEnWqW5E4c4w/iSBLRCoBCiAizfG5e8iYE5J8N2SmOWMzmKBYsnkf5z89i3HfrWfXISsSZ4rmz3MEDwBfAA1F5G2gNzA8mEGZKFCvC7QcAPOecwaxCeexF8qYI5k5PD5jHRPmbaJefCUm3NCdfq1s1DBTNH/uGvoKGIpz8p8EJKnq9/5sXEQGisg6EVkvIvcWsc7lIrJGRFaLyDv+h27CXr/RcHQ/LHrV60giyvYDR3ln4RauO6MxM+5ItiRgSuTPcwTJhc1X1ZQSPhcL/AScC6QCi4ArVXWNzzotgSnAWaq6X0TqqOqu4rZrzxFEmDcvgp2r4PYVEGd3r5yotPRspq/cwVU9nCJxOw9mUNf6AYyP4p4j8OfSkG9doYo44xQvAc4q4XPdgfWqusENYjIwBFjjs85NwHOquh+gpCRgIlC/0fD6IFjyBvS8xetowtIXq37lnx+vYt+RLHo0q0XzxKqWBEyp+HNp6EKf17lAB2C/H9uuD2z1mU515/lqBbQSkTkiMl9EBha2IREZISKLRWTx7t27/di1CRuNe0HjPk4xuuwMr6MJK7sOZXDL20u4+a0lJFatwMd/7U3zRCsSZ0rvRKpKpQJtA7T/ckBLoD9wJfCyiNQouJKqjlfVJFVNSky0650Rp99oOPwrLJ3odSRhIzdPufzFeXz94y7uHtCaj0f2pkP9eK/DMmGqxEtDIvIs7q2jOImjC/CDH9veBjT0mW7gzvOVCixQ1Wxgo4j8hJMYIqwgvylW02Ro2ANmj3VGMysX53VEZdaOtKPUrVaR2BjhgcHtaVizspWKNifNrzGLcfoElgDzgHtU9Ro/PrcIaCkiTUUkDhgGTCuwzkc4rQFEJAHnUtEGvyI3kUMEkkfDwVRYbjeOFSYvT3ljzkbOfnImby3YDMCZretYEjABUWKLQFUnnMiGVTVHREYCM4BY4DVVXS0i/wEWq+o0d9l5IrIGyAXuVtW9J7I/E+ZanO2M2zxrjDOaWayNi5tv/a7D3Dt1BYs37ye5VSJntbEicSaw/Ll9dCXHLg0dtwinHHWnYARWFLt9NIKt/QwmXwkXvQBdrvI6mjJh8sIt/GvaaiqVj+VfF7RjaLf6Vh/InJCTvX30c/dnfk/e1e5PG7XMBFbrQVC3I8x6EjpdATGxXkfkuUa1K3NO2zr8e3AHEqtV8DocE6H8aREsVdWuBeb9oKrdghpZEaxFEOHWfAxTroNLXoWOl3odTchlZOfyzDc/AzB6YBuPozGR5KQGpnE+L719Jnr5+TljSq/NhZDYFlKegLw8r6MJqcWb9nH+M7N4/vtf2Hcky4rEmZDx59LQjcBrIpJ/k/IB4IagRWSiW0wMJN8FU2+EtZ9AuyFeRxR0hzNzePyLtbw5fzP1a1TizRu6k2z1gUwI+XPX0BKgc34iUFUrIG+Cq/3F8P3/wczHoe1g5/bSCPZr2lEmL9rK9T2bcPeA1lSp4M/3M2MCx5+hKuuKyKvAZFVNE5F2InJjCGIz0SomFvreCTtXwrrPS14/DO0/ksXE+c7zAC3qVGPW6DN5cHB7SwLGE/5c638D537/eu70T8DfghSPMY6Ol0GNxpDyGETQtXJV5bOVOzj3qZn8e9pqftl9GMBGDDOe8icRJKjqFCAPnAfFcB7+MiZ4YstD31GwfSms/8braAJi18EMbn5rCbe8/QOnxldi2sg+ViTOlAn+JIIjIlKbY0NVngFYP4EJvs5XQfUGEdEqyM1TLntpHt+v2819g9rw4S29aFevutdhGQP4d9fQKJwaQc1FZA6QCETfDd4m9MrFQZ+/wWd3wcYUaNbP64hKbfuBo5xS3SkS958hHWhYsxLNrBVgyphiWwTuKGP93Fcv4M9Ae1VdEYLYjIGu10LVU2DmY15HUiq5ecrrBYrE9WuVaEnAlEnFJgJVzcUZXjJHVVer6iq3ZLQxoVG+IvS+HTbPhs1zvY7GL+t3HeKyF+fy70/W0KNZLc5uW9frkIwplj99BHNEZJyI9BWRbvmvoEdmTL7ThkOVxLBoFbyzYAvnPz2bjXuO8NQVnXl9+OnUr1HJ67CMKZY/fQRd3J//8ZmnlDxmsTGBEVcZeo6Erx+A1MXQoNByKWVCk4TKnNe+Lg8Obk9CVSsSZ8JDkUXnRGSkqo5z37dX1dUhjawIVnQuSmUehrEdocHpcPUUr6P5TUZ2Lk99/ROCcO8gKxJnyq4TLTrnW0/IBpM13qpQFXreAj/PgO3LvI4GgAUb9jLo6Vm8NHMDhzKyrUicCVv+VhGN7GIvJjx0HwEV4yHlcU/DOJSRzT8+WskV4+eTm6e886ce/O/ijjZgjAlbxfUR1BCRi3GSRXURGeq7UFU/CGpkxhRUMR563AwzH4VfV8EpHTwJY+fBTN5fksqf+jRl1HmtqBxn9YFMeCuuj+D1Yj6nqupJKWrrI4hy6fucvoKW58Jlb4Rst/uOZDF9xXau7dkEgN2HMm3EMBNWTmioSlX9Y/BCMuYEVa4F3W+C2WOh/zpIbB3U3akqn67YwYPTVnMwI5veLRJolljVkoCJKDbSmAk/PUdC+UrO2MZBtPNgBje9uYRbJy2lfs1KfHJrH3sy2EQkSwQm/FRJgKQbYOV7sPeXoOwiN0+5/KV5zPp5N/ef35YP/tKLNqdYkTgTmSwRmPDU6zaIjYPZYwK62dT96eTmKbExwkNDOjDjb8nclNyMcrH2X8VELn9GKKssIv8UkZfd6ZYickHwQzOmGNXqQrfrYflk2L/5pDeXm6e8MmsD54yZyVvuyGHJrRJpklDlpLdtTFnnz9ec14FMoKc7vQ34b9AiMsZfvW8HiYHZT53UZtb9eoihL8zlv9N/pHfzBM5rb0XiTHTxJxE0V9XHgGwAVU3HHjAzZUF8fehyNSx7G9K2ndAm3pq/mQuencXWfek8PawLr1yfxKnxViTORBd/EkGWiFTi2AhlzXFaCMZ4r88doHkw5+lSfSz/+ZkWdapyfsdT+eqOZIZ0qW9PB5uo5M8jkQ8CXwANReRtoDcwPIgxGeO/mo2h8zD4YQL0vdPpOyjG0axcxny1jpgY4b5BbTmjWW3OaFY7RMEaUzaV2CJQ1S+BoTgn/0lAkqp+H9ywjCmFPqMgNwvmPlPsavN+2cvAp1N4edZG0jNzrUicMa4SWwQi8gnwDjBNVY8EPyRjSql2c+h4GSx+zblUVCXhuMUHM7L5v8/WMmnhFhrXrsw7N/WgV/OEIjZmTPTxp4/gCaAvsEZE3heRS0WkYpDjMqZ0+t4F2Udh3rjfLdp1MJOPlm5jRHIzvrg92ZKAMQX4c2lopqreAjQDXgIuB3b5s3ERGSgi60RkvYjcW8x6l4iIikjZHXrKlG2JraD9RbDwZUjfx97DmbwxZyPgdAjPvudM/n5+WyrFxXobpzFlkF/1c927hi4ErgC6ARP8+Ews8BxwLpAKLBKRaaq6psB61YDbgQWlC92YApLvhtUfsvbjx7jy57M4nJlDcqtEmiVWpbYNG2lMkfx5sngK8CPOGMXjcJ4ruNWPbXcH1qvqBlXNAiYDQwpZ7yHgUSDD76iNKcT2Cs1YUrkP9dZOoG1NmH5bXysSZ4wf/OkjeBXn5H+zqn6nqnl+brs+sNVnOtWd9xsR6QY0VNXpxW1IREaIyGIRWbx7924/d2+iSU5uHsPGz+fhQ3+guqTzVqfltKpbzeuwjAkLRV4aEpGzVPVboAowpOCDNic7QpmIxABj8OOZBFUdD4wHZ2Cak9mviSxb96VTr0YlysXG8PDFHWlUqwd88T0x85+HM/7ijHVsjClWcS2Cfu7PCwt5+VN0bhvQ0Ge6gTsvXzWgA/C9iGwCzgCmWYex8UdObh7jU37hnDEzmThvEwB9WibQqHZl6Dcaju6Dxa96G6QxYaK4EcoecN/+R1U3+i4TkaZ+bHsR0NJddxswDLjKZ/tpwG/38YnI98BdqmrjUJpi/bjjIPdMXcGK1DTObVeXQR1PPX6FBknQ7EyY+yycfhPEVfYmUGPChD99BFMLmfd+SR9S1RxgJDADp7N5iqquFpH/iMjg0oVpjGPivE1c+Oxstu0/yrirujL+2tOoW72Qx1r6jYYju2HJGyGP0ZhwU1wfQRugPRAvIkN9FlUH/HqgTFU/Az4rMO9fRazb359tmuikqogIrepW48LO9fjnBe2oVSWu6A807gWN+zjF6JJugPL2DKQxRSnuOYLWOH0BNXD6BfIdAm4KYkzG/CY9K4cnZvxEuVjh7+e3pUez2vTwt0hcv9Hw5mBYOtEZ8N4YU6ji+gg+Bj4WkZ6qOi+EMRkDwJz1e7j3gxVs3XeU4b2a/NYq8FvTZGjYA2aPdUYzK1dMC8KYKObPk8VLReSvOJeJfmtfq+oNQYvKRLW0o9k8PP1H3l28laYJVZjy5550b1qr9BsSgeTR8PYlsHwSnHZ94IM1JgL401k8ETgFGADMxLkN9FAwgzLRbc/hTD5ZsZ2b+zXn89v7nlgSyNfibKjXFWY9CbnZgQvSmAjiTyJooar/BI6o6gTgD0CP4IZlos3uQ5m8Ntu5S7l5YlVm33MW9w5qQ8XyJ1kkLr9VcGAzrHwvAJEaE3n8SQT5X6MOiEgHIB6oE7yQTDRRVT5cmsq5T83kkc/XsnGPM+RFsXcElVbrQVC3o9MqyMsN3HaNiRD+JILxIlIT+CcwDVgDPBbUqExU2HbgKH98YxF3vLucZglV+Oz2PjRNqBL4HYlAv7th73pY/WHgt29MmCuxs1hVX3HfzsQZk8CYk+YUiZvH3sNZPHhhO67t2YTYmCAOHN/mQkhsCylPQPuhEOPPdyBjokNxD5SNKu6Dqjom8OGYSLdlbzr1azpF4h4Z2olGtSrTsFYISkDExEDyXTD1Rlj7CbQrrCK6MdGpuK9F1Up4GeO3nNw8Xvj+F855aiZvukXierdICE0SyNf+YqjdAmY+DjZwvTG/Ke6Bsn+HMhATuVZvT+OeqStYte0gA9rX5Q8Fi8SFSkws9L0TPvoLrPsc2pzvTRzGlDH+jFDWSkS+EZFV7nQnEflH8EMzkWDC3E0MGTeHX9MyeeHqbrx0bRJ1CisSFyodL4MajSHlMWsVGOPyp8fsZeA+3NtIVXUFTklpY4qk7km2zSnVGNKlPl+PSv59uWgvxJaHvqNg+1JY/43X0RhTJviTCCqr6sIC83KCEYwJf0cyc3hw2moe/uxHAHo0q82Tl3emRuUyVOen81VQvYG1Coxx+ZMI9ohIc0ABRORSYEdQozJhKeWn3Zz3VAoT5m0iO1d/axWUOeXioM/fYOsC2JjidTTGeM6fonN/xRkvuI2IbAM2AlcHNSoTVtLSs3lo+hreX5JKs0SnSNzpTU6iPlAodL3WeaYg5XFo1q/k9Y2JYCW2CFR1g6qeAyQCbXDGMu4T7MBM+NhzJJPPV+7glv7N+ey2vmU/CYAzUE3v22HTLNg81+tojPFUkYlARKqLyH0iMk5EzgXSgeuB9cDloQrQlE27DmXwyqwNwLEicaMHBqBIXCidNhyqJMJMq5hioltxLYKJOKOUrcQZkew74DLgYlW1xzKjlKry/pJUzh2TwmMz1v1WJK5mIIvEhUpcZeg5EjZ8B6mLvY7GGM8U10fQTFU7AojIKzgdxI1UNSMkkZkyZ+u+dP7+4Upm/byHpMY1eeSSTsEpEhdKp//JGdd45mNw9RSvozHGE8Ulgt9G8VDVXBFJtSQQvXJy87jy5fnsP5LFQ0Pac3WPxsQEs0hcqFSoCj1vgW//C9uXQb0uXkdkTMgVlwg6i8hB970AldxpAVRVqwc9OuO5TXuO0LBWZcrFxvDYpU6RuAY1Q1gfKBS6j4C5zzp3EA172+tojAm5IvsIVDVWVau7r2qqWs7nvSWBCJedm8dz363nvKdSfisS16t5QuQlAYCK8dDjZlj7Kexc7XU0xoScFWU3v7NqWxpDxs3h8RnrOLddXS7oVM/rkIKvx80QV9VpFRgTZfx5oMxEkdfnbOS/03+kVpU4XrzmNAZ2OMXrkEKjci3ofhPMHgv910Fia68jMiZkrEVggGNF4trXi2do1/p8fUe/6EkC+XqOhPKVnLGNjYkilgii3OHMHP718Sr+N90pEte9aS0ev6wz8ZXLexyZB6okQNINsPI92PuL19EYEzKWCKLY9+t2MeCpFCbO34xC2S0SF0q9boPYOJhtI7Ga6GGJIArtP5LFqCnLGP76IirFxfL+zb345wXtEImA5wJOVrW60O16WD4Z9m/2OhpjQsISQRTan57Fl6t3cttZLZh+Wx9Oa1zT65DKlt63g8TAnLFeR2JMSAQ1EYjIQBFZJyLrReTeQpaPEpE1IrLCHQ6zcTDjiWa7DmYwPuUXVJVmiVWZc89ZjDqvNRXKhVGRuFCJrw9droalb0HaNq+jMSbogpYIRCQWeA4YBLQDrhSRdgVWWwokqWon4H3AykAGmKoyZdFWzh4zkye//IlNe9MBorMzuDT63AGa59QhMibCBbNF0B1Y745nkAVMBo6rWqqq36lqujs5H2gQxHiiztZ96Vz76kJGT11B21Or8/ntfcO/SFyo1GwMnYfBDxPg0E6vozEmqIKZCOoDW32mU915RbkR+LywBSIyQkQWi8ji3bt3BzDEyJVfJG7Z1gP896IOTL7pDJolVvU6rPDSZxTkZsHcZ7yOxJigKhNPFovINUASzuhnv6Oq43GGyyQpKcnucSzGxj1HaOQWiXv80s40rl2ZejUqeR1WeKrdHDpeBotfcy4VVUnwOiJjgiKYLYJtQEOf6QbuvOOIyDnA/cBgVc0MYjwRLTs3j2e/+ZkBT6UwYe4mAHo2r21J4GT1vQuyj8K857yOxJigCWYiWAS0FJGmIhIHDAOm+a4gIl2Bl3CSwK4gxhLRVqQe4MJnZ/PkVz8xoMMpDO4SBUXiQiWxFbS/CBa+DOn7vI7GmKAIWiJQ1RxgJDAD+BGYoqqrReQ/IjLYXe1xoCrwnogsE5FpRWzOFOG12Ru56Lk57E/P4uXrknj2yq4kVK3gdViRJfluyDoEC170OhJjgiKofQSq+hnwWYF5//J5f04w9x/JVBURoVODeK44vSH3DmpLfCW7JTQo6raHNhfA/Beh51+d8QuMiSD2ZHGYOZSRzf0fruShT50icUlNavF/QztZEgi25LshMw0Wjvc6EmMCzhJBGPlu7S7OeyqFSQu3UC5WrEhcKNXrAi0HwLznIfOw19EYE1CWCMLAviNZ/G3yUv74xiKqVSzH1L/04u/nt7UicaHWbzQc3QeLX/U6EmMCyhJBGEg7ms03P+7i9rNb8umtfenayIrEeaJBEjQ70xnoPiu95PWNCROWCMqoX9MyeHGmUySuaUIVZt97Fnec24q4cvYn81S/0XBkNyx5w+tIjAkYO6uUMarKpIVbOHfMTMZ+/ROb84vEWWdw2dC4FzTu4xSjy87wOhpjAsISQRmyee8Rrnp5Afd9sJL29avzxe3JNLEicWVPv9Fw+FdYOtHrSIwJiDJRa8g4ReKuenkBaUezefjijgw7vSExMdYZXCY1TYaGPWD2WGc0s3JxXkdkzEmxFoHHftl9mJzcPMrFxvDk5Z35alQyV/VoZEmgLBOB5NFwMBWWT/I6GmNOmiUCj2Tl5DH2658YODaFN+c5Y+Oe0aw2p8Zbkbiw0OJsqNfVGeQ+N8fraIw5KZYIPLBsq1MkbuzXP3N+x1O5qGtxwzSYMim/VbB/E6x8z+tojDkp1kcQYq/O3sj/pq+hTrWKvHp9Eme3ret1SOZEtR4EdTvCrCeg0+UQY+M/m/BkLYIQyS8H0aVhPMO6N+LLUcmWBMKdCCTfBXvXw+oPvY7GmBNmLYIgO5iRzf99tpaK5WN44ML2nNa4Fqc1ruV1WCZQ2g6GxDaQ8gS0Hwox9t3KhB/7VxtEX6/ZybljZvLuoi3ElYuxInGRKCbGqUy6+0dY+4nX0RhzQiwRBMHew5ncNmkpf3pzMTUrx/HhLb25b5AViYtY7S+G2i0g5XGwZG/CkCWCIDiUkcN363ZxxzmtmDayD50b1vA6JBNMMbHQ9074dSX89IXX0RhTapYIAmT7gaM89916VJUmCVWYc+9Z3H5OSysSFy06XgY1GsPMx6xVYMKOnaVOUl6e8tb8zZz3VArjvl3/W5G46hWtSFxUiS0PfUfB9h9g/TdeR2NMqVgiOAkb9xzhypfn84+PVtG5YTwz/mZF4qJa56ugegNIsVaBCS+WCE5QTm4e17yygDU7DvLYJZ1468YeNKpd2euwjJfKxUGfv8HWBbAxxetojPGbPUdQSut3HaJJ7SqUi43hqSu60Lh2ZepWr+h1WKas6Hqt80xByuPQrJ/X0RjjF2sR+CkzJ5cxX/3EwLGzmOAWievetJYlAXO88hWh9+2waRZsnud1NMb4xRKBH37Ysp8LnpnNM9/8zODO9RhqReJMcU4bDlUSnb4CY0qydSHMetL56RG7NFSCl1M28PDnP3Jq9Yq8/sfTObN1Ha9DMmVdXGXoORK+fgBSFzuD3pvooQpZRyDjABw9cOzn0f2/n7d/E2xf4nymXCW4fho07B7ykC0RFCEvT4mJEbo1rsHVPRpxz8A2VLNbQo2/Tr8R5ox1niu4eorX0ZjSUoXs9ONP2sWd0I9blgZ52UVvW2KgYg2oVMMZ9zr/DrPcLOeSoiUC76UdzeZ/09dQqXws/x7SwYrEmRNToRr0/Ct8+1/YvgzqdfE6ouijCtlHSzhpF7OsuJM5AhXjnZN5pZrOiT2+wbETfEV3/m/vfX7GVTtWnHDrQpgw2EkCsXHQpG+Afwn+sUTgY8bqX/nnR6vYeySLPyc3Q1WtPpA5cd1HwNxnnTuIhr3tdTThKzvDj2/hRSzLzSxmwwIVqx87QVeqCdXrFX7yzj/Z58+rUD0wlWYbdncuB22a5SQBD1oDYIkAgD2HM3ng49VMX7mDdqdW57Xhp9OhfrzXYZlwVzEeetwMMx+FnauhbnuvI/JOTqbPCXp/4SftopblZBS/7QrxUCn+2Ik6sXXR38Z9v6lXqF42BhNq2N2zBJDPEgFwOCOHWT/v5u4BrRmR3IzysXYzlQmQHjfDvOecZwsue93raE5OTlbJJ+2iluUcLX7bFfK/mbsn9ISWRX8b9z2hV4wvGyfzMBe1iWDbgaN8+EMqfz2zBU0SqjD3vrOpWiFqfx0mWCrXgu43weyx0P8+SGzlbTy52aW8vOJzQs9OL37bcVWP/8Zdu3kxl1d8vrFXjIdY+7/npaD+9kVkIPA0EAu8oqqPFFheAXgTOA3YC1yhqpuCGVNenvL2gs088vla8hQu6FSPJglVLAmY4Ok5Eha85IxtPHT8yW8vN8f/yyoFT/bZR4rfdvkqx3/jrtWs+Msrv82LdwrvmbAUtLOfiMQCzwHnAqnAIhGZpqprfFa7Edivqi1EZBjwKHBFsGL6Zfdh7pu6koWb9tG3ZQIPX9yRhrWsPpAJsioJkHSDc4moSiK0GwL1ujm3Gf52st5fzDf0tOOXZR0qfn/lKx9/0q7RCE7tXPIJvWK8Uy/JRJ1gfg3uDqxX1Q0AIjIZGAL4JoIhwIPu+/eBcSIiGoQxHXNy87ju1YUcysjm8Us7celpDeyOIBM6TfrCvHHHXiUpV+n4k3Z8AzilQzHXy2scu2ZerkLQDsNEpmAmgvrAVp/pVKBHUeuoao6IpAG1gT2+K4nICGAEQKNGjU4omHKxMYwd1oXGtSpTx+oDmVDbtRoQQJ2fTZOh9flF39lS3v6NmtAJiwvjqjoeGA+QlJR0wq2F05vYg2HGI036QrmKxx4cOusfnt8yaEy+YCaCbUBDn+kG7rzC1kkVkXJAPE6nsTGRpYw8OGRMYYKZCBYBLUWkKc4JfxhwVYF1pgHXA/OAS4Fvg9E/YEyZUAYeHDKmMEFLBO41/5HADJzbR19T1dUi8h9gsapOA14FJorIemAfTrIwxhgTQkHtI1DVz4DPCsz7l8/7DOCyYMZgjDGmeFZLwRhjopwlAmOMiXKWCIwxJspZIjDGmCgn4Xa3pojsBjaf4McTKPDUchSwY44OdszR4WSOubGqJha2IOwSwckQkcWqGlUjidsxRwc75ugQrGO2S0PGGBPlLBEYY0yUi7ZEEIBRQcKOHXN0sGOODkE55qjqIzDGGPN70dYiMMYYU4AlAmOMiXIRmQhEZKCIrBOR9SJybyHLK4jIu+7yBSLSxIMwA8qPYx4lImtEZIWIfCMijb2IM5BKOmaf9S4RERWRsL/V0J9jFpHL3b/1ahF5J9QxBpof/7Ybich3IrLU/fd9vhdxBoqIvCYiu0RkVRHLRUSecX8fK0Sk20nvVFUj6oVT8voXoBkQBywH2hVY5xbgRff9MOBdr+MOwTGfCVR23/8lGo7ZXa8akALMB5K8jjsEf+eWwFKgpjtdx+u4Q3DM44G/uO/bAZu8jvskjzkZ6AasKmL5+cDnOGOfngEsONl9RmKLoDuwXlU3qGoWMBkYUmCdIcAE9/37wNkS3iPZl3jMqvqdqqa7k/NxRowLZ/78nQEeAh4FMkIZXJD4c8w3Ac+p6n4AVd0V4hgDzZ9jVqC6+z4e2B7C+AJOVVNwxmcpyhDgTXXMB2qIyKkns89ITAT1ga0+06nuvELXUdUcIA2oHZLogsOfY/Z1I843inBW4jG7TeaGqjo9lIEFkT9/51ZAKxGZIyLzRWRgyKILDn+O+UHgGhFJxRn/5NbQhOaZ0v5/L1FYDF5vAkdErgGSgH5exxJMIhIDjAGGexxKqJXDuTzUH6fVlyIiHVX1gJdBBdmVwBuq+qSI9MQZ9bCDquZ5HVi4iMQWwTagoc90A3deoeuISDmc5uTekEQXHP4cMyJyDnA/MFhVM0MUW7CUdMzVgA7A9yKyCeda6rQw7zD25++cCkxT1WxV3Qj8hJMYwpU/x3wjMAVAVecBFXGKs0Uqv/6/l0YkJoJFQEsRaSoicTidwdMKrDMNuN59fynwrbq9MGGqxGMWka7ASzhJINyvG0MJx6yqaaqaoKpNVLUJTr/IYFVd7E24AeHPv+2PcFoDiEgCzqWiDSGMMdD8OeYtwNkAItIWJxHsDmmUoTUNuM69e+gMIE1Vd5zMBiPu0pCq5ojISGAGzh0Hr6nqahH5D7BYVacBr+I0H9fjdMoM8y7ik+fnMT8OVAXec/vFt6jqYM+CPkl+HnNE8fOYZwDnicgaIBe4W1XDtrXr5zHfCbwsInfgdBwPD+cvdiIyCSeZJ7j9Hg8A5QFU9UWcfpDzgfVAOvDHk95nGP++jDHGBEAkXhoyxhhTCpYIjDEmylkiMMaYKGeJwBhjopwlAmOMiXKWCIwn3Gqgb/lMlxOR3SLyqZdxlZaIbHLv10dE5paw7nARqVfK7TcprAqliGwQkdYF5o0VkXv8idUYX5YIjFeOAB1EpJI7fS4n+XRkoLhPm5eaqvYqYZXhQKkSQTEm4/P8i1tS41J3vjGlYonAeOkz4A/u+yuBSfkLRKSKW5d9oVtnfog7v4mIzBKRH9xXL3d+fxH5XkTeF5G1IvJ2YRVl3XWeFpFlIrJKRLq78x8UkYkiMgfnYcNEEZkqIovcV293vdoi8qU4tf5fwSkFnL/twz7v7xGRlSKyXEQeEZFLcWo8ve3uu5KInCYiM0VkiYjMyK8g6c5fLiLLgb8W8bubBFzhM50MbFbVzSLykbvN1SIyopDfwXGtDBG5S0QedN83F5Ev3M/PEpE2RezfRBKva2/bKzpfwGGgE04Z8IrAMpynKT91lz8MXOO+r4FTM6cKUBmo6M5vifN0Ke5n03DqrsQA84A+hez3e+Bl930ybs13nAqWS4BK7vQ7+Z8HGgE/uu+fAf7lvv8DzpOsCfnH5P4cBMzl2PgPtXz2neS+L++uk+hOX4Hz1CzACiDZff84RdelXwV0dt+/CIwssL9K7jq13elNODV4mvhuE7gLeNB9/w3Q0n3fA6f8iuf/XuwV3FfElZgw4UNVV4gzOtyVOK0DX+cBg0XkLne6Is4JeTswTkS64JRQaOXzmYWqmgogIstwTnizC9n1JHf/KSJSXURquPOnqepR9/05QDufRkV1EamKkzyGup+fLiL7C9n+OcDr6o7/oKqF1ZZvjVMU7yt3H7HADjeWGurUpAeYiJNYCjMJGCYiq4GLcEoRANwmIhe77xviJMwSy0y4x9eLY2VIACqU9DkT/iwRGK9NA57A+UbvOyaEAJeo6jrfld1LGDuBzjjf/H0HnPGtqJpL0f++C9ZVyZ8+4jMvBjhDVY8b0KaQq00nSoDVqtqzwPZrlGIbk4EvgZnAClXdKSL9cRJRT1VNF5HvcZKorxyOvyycvzwGOKCqXUoRg4kA1kdgvPYa8G9VXVlg/gzg1vzr/OJUTwWnZPgOdWrNX4vzTbq0rnC32QencmNaIet8ic8AJ24LBJxhL69y5w0Cahby2a+AP4pIZXe9Wu78QzjlsQHWAYni1M9HRMqLSHt1xg044MYGcHVRB6GqvwB7gEc41r8SD+x3k0AbnPLbBe0E6rj9HRWAC9ztHQQ2ishlbkwiIp2L2r+JHJYIjKdUNVVVnylk0UM419FXuJc+HnLnPw9c73aktuH4b/H+yhCRpTjX1W8sYp3bgCRxBgdfA9zszv83kOzGNBSnBHLBY/oCp6Wz2L1ElX956w3gRXdeLM5dPo+6x7IM57IMONUkn3PXK6kJMgnn9/CBO/0FUE5EfsRJEPMLiS8b+A+wECdprfVZfDVwoxvTagof/tNEGKs+aqKKe6nkLg3vcQmMCShrERhjTJSzFoExxkQ5axEYY0yUs0RgjDFRzhKBMcZEOUsExhgT5SwRGGNMlPt/3uMdyEgIhrsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reliability diagram\n",
    "fop, mpv = calibration_curve(testy, probs_test, n_bins=10, normalize=True)\n",
    "# plot perfectly calibrated\n",
    "pyplot.plot([0, 1], [0, 1], linestyle=\"--\")\n",
    "# plot model reliability\n",
    "pyplot.plot(mpv, fop, marker=\".\")\n",
    "pyplot.xlabel(\"Mean predicted Value\")\n",
    "pyplot.ylabel(\"Relative Frequency\")\n",
    "pyplot.title(\"SVM Reliability Diagram\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54d6bec5",
   "metadata": {},
   "source": [
    "In the above graph, the solid line represents the calibration of the SVMs predicted probabilities and the dotted line represents a comparison to a perfectly calibrated model. The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot. Here, we see that calibration probabilities are much further away from the ideal prediction.\n",
    "\n",
    "The position of the points or the curve relative to the diagonal can help to interpret the probabilities; for example:\n",
    "\n",
    "* *Below the diagonal*: The model has over-forecast; the probabilities are too large.\n",
    "* *Above the diagonal*: The model has under-forecast; the probabilities are too small."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f911f375",
   "metadata": {},
   "source": [
    "#### SVM for over sampled data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c61d72bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create X = test_duration and y = pass_or_fail and calculate probabilty for over sampled data\n",
    "X, y = np.array([df_os[\"test_duration\"].values]), np.array(\n",
    "    [df_os[\"pass_or_fail\"].values]\n",
    ")\n",
    "X = X.reshape(-1, 1)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "# split into train/test sets.\n",
    "trainx, testx, trainy, testy = train_test_split(\n",
    "    X, y, train_size=0.8, test_size=0.2, random_state=42\n",
    ")\n",
    "\n",
    "# fit a SVM Model.\n",
    "model = SVC(probability=True).fit(trainx, trainy)\n",
    "\n",
    "# Evaluate the model\n",
    "y_pred = model.predict(testx)\n",
    "\n",
    "# predict probabilities.\n",
    "probs_train = model.predict_proba(trainx)[:, 0]\n",
    "probs_test = model.predict_proba(testx)[:, 0]\n",
    "probs = np.append(probs_train, probs_test, axis=None)\n",
    "df_os[\"prob\"] = probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "97239480",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVM Model accuracy : 0.9110546378653113\n",
      "SVM Brier Score 0.08894536213468869\n",
      "SVM Roc Score 0.9108594984441994\n"
     ]
    }
   ],
   "source": [
    "# Mean accuracy on the given test data and labels.\n",
    "svm_accuracy = model.score(testx, testy)\n",
    "print(\"SVM Model accuracy :\", svm_accuracy)\n",
    "svm_brier = brier_score_loss(testy, y_pred, pos_label=1)\n",
    "print(\"SVM Brier Score\", svm_brier)\n",
    "svm_roc = roc_auc_score(testy, y_pred)\n",
    "print(\"SVM Roc Score\", svm_roc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4170c0bf",
   "metadata": {},
   "source": [
    "Brier Score is a distance in the probability domain. Which means: the lower the value of this score, the better the prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f910ca22",
   "metadata": {},
   "source": [
    "The above calculated score is a harsh metric since we require for each sample that each label set be correctly predicted. This does not give us a clear indication of a better model so we will use the reliability diagram to find out the better suited model.\n",
    "\n",
    "We also see that we achieved a higher accuracy of ~81% with over sampling of the data compared to the ~78% accuracy with the under sampled data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "719aab48",
   "metadata": {},
   "source": [
    "##### Reliability Diagram\n",
    "\n",
    "A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency  (x-axis). These plots are commonly referred to as ‘reliability‘ diagrams in forecast literature, although may also be called ‘calibration‘ plots or curves as they summarize how well the forecast probabilities are calibrated.\n",
    "\n",
    "The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a94959e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABHLklEQVR4nO3dd3hUZfbA8e9JQu+Q0HsnIEUjKkqRoogKiA3Buu7607Xs2ta27lrWsu7adtW1d6UoFhQVEGlSRBAMvXdUeugh5fz+eG80xpQJmZk75XyeZ57MvXPn3nNDmDP3ve97XlFVjDHGxK8EvwMwxhjjL0sExhgT5ywRGGNMnLNEYIwxcc4SgTHGxDlLBMYYE+csEZiYJyIbRKS/9/xuEXk5wPe9LiL/KOb1AyLSsuC2ItJTRFYGI/ZAichSEekTzmOa2GGJwJSKiJwmIrNFJENEdovILBE5UUROFpGDIlK1kPcsFJEbRKS5iKiILCzwerKIHBWRDcUcV739HxCRrSLyhIgkljZ+VX1YVX9f2vcVsa+qqrqukPUzVbVd3nL+RFRa+X5nB7zHTyLyqYgMKHDMjqo67ViOYYwlAhMwEakOfAr8F6gNNALuBzJVdS6wBbigwHs6AanAqHyrK3vr84wA1gcQQhdVrQr0Bi4GfneMpxKNanrn3gWYDHwoIleG+qAikhTqYxj/WSIwpdEWQFVHqWqOqh5W1Umqmu69/gZweYH3XA58pqq78q17C7iiwDZvBhqEqq4BZgFd89aJyDkiskhE9npXLJ0Le6+I3Ccib+dbfk9EfvSucGaISMcCb0kWkckisl9EpotIs3zvVRFpXcgx+ojIFu/5W0BT4BPvG/1fRGSCiNxY4D3pInJeAOf+o6o+DdwH/FNEErz352/+6i4ic7zfxQ8i8oyIlM93rDNEZKV3zs955/V777Urvau8J0VkF3CfiLQSka9EZJeI7BSRd0SkZr79bRCR271zOCgir4hIPRH53Pu9fSkitUo6N+MfSwSmNFYBOSLyhoicVch/7reAXiLSBMD7kBqBSxD5vQ0MF5FEEUkFqgLfBBqEiLQHegJrvOVuwKvA/wF1gBeA8SJSIYDdfQ60AeoC3wHvFHh9JPAgkAwsKuT1YqnqZcAm4FyvKekx3O/j0nzn0wV3dTWhFLv+wIu5XSGv5QA3ezGfAvQD/ugdKxl4H7gL97taCfQo8P6TgHVAPeAhQIBHgIZAB6AJLhHldz4wAPdl4Vzc7/VuIAX3OXNTKc7NhJklAhMwVd0HnAYo8BKwQ0TGi0g97/XNwDTgMu8t/YAK/PYDbgvuA6g/7mrgrQBD+E5EDgLLveM8562/BnhBVb/xrlTeADKBkwM4p1dVdb+qZuI+3LqISI18m0xQ1Rne6/cAp+QlujIYD7QVkTbe8mXAGFU9Wop9bPN+1i74gqouUNW5qpqtqhtwibG39/IgYKmqfqCq2cB/gB8L7ltV/+u9/7CqrlHVyaqaqao7gCfy7S/Pf1X1J1XdCswEvlHVhap6BPgQ6FaKczNhZonAlIqqLlfVK1W1MdAJ9y3xqXybvMEvieAyYLSqZhWyqzeBK4FLCDwRHI+7ergY9621ire+GXCr1xSyV0T24r61NixuZ94VyaMislZE9gEbvJeS8222Oe+Jqh4Adpe035J4H45jgEu9q6bS/A7yNPJ+7i74goi09W4o/+id18P8ck4N+fU5KS4x57c5/4LXzDPau0m/D3dFl1zgPT/le364kOXfdCIwkcMSgTlmqroCeB2XEPJ8ADQWkdOBYfy2WSjPOOBsYJ2qbirFMVVVxwJzgL95qzcDD6lqzXyPyqo6qug9Aa7ZagjuyqQG0NxbL/m2+fnbv7geUbX55dt4wGEXsu4NXLNTP+CQqs4p5T7PA7bjrqwK+h+wAmijqtVxTTR55/QD0DhvQxGR/MtFxPuwt+44b3+X8uvfkYlylghMwESkvYjcKiKNveUmuG+zc/O2UdWDuDbo14CNqjq/sH152/UFjrUr56PAH0SkPq6Z6loROUmcKiJytohUK2Ef1XBNSLuAyrgPvIIGiesyWx53r2Cu1wRWGj8BLfOv8D74c4HHKcXVgPft/Abg78BdqppbyGbVgH3AAe9+ynX5XpsAHCciQ70eQdcD9Us4bDXgAJAhIo2A2wON10QHSwSmNPbjmmS+8drq5wJLgFsLbPcGrrmm2J5AqjpfVdceSyCquhiYAdzuJZs/AM8Ae3A3ka8MYDdvAhuBrcAy8iW0fN7FfejuBk4g303eUngE+KvXbHVbgeMfh2tqKcle73e+GNfOf6GqvlrEtrfhrnb245LkmLwXVHUncCHwGC4BpgLzcQmxKPfjmuUycInkgwDiNVFEbGIaY/whIpcD16jqaT7GkIC7RzBSVaf6FYfxl10RGOMDEamM69L5og/HPlNEanrda/PuHxR2NWTihCUCY8JMRM4EduDuHbzrQwinAGuBnbg+/0NV9bAPcZgIYU1DxhgT5+yKwBhj4lzUFZRKTk7W5s2b+x2GMcZElQULFuxU1ZTCXou6RNC8eXPmzy+0a7oxxpgiiMjGol6zpiFjjIlzlgiMMSbOWSIwxpg4Z4nAGGPinCUCY4yJcyFLBCLyqohsF5ElRbwuIvIfEVnjTXF3fKhiMcYYU7RQXhG8Dgws5vWzcFMEtsHNMPW/EMZijDGmCCFLBKo6g0JmT8pnCPCmN9HIXKCmiDQIVTzGGBOtDh3NZvPuQyHbv5/3CBrx6ynxtvDL9Hu/IiLXiMh8EZm/Y8eOsARnjDGRYPaanQx8aibXvr2A3NzQ1IaLipvFqvqiqqapalpKSqEjpI0xJqZkHM7iznHpjHj5GxIE7j0nlYSE0MwQ6meJia3kmw8WN2/qVp9iMcaYiJGTq5z/v9ms23GA/+vdkpv7t6ViucSQHc/PRDAeuEFERuOmP8xQ1R98jMcYY3y15+BRalYuR2KCcNsZ7WhYsyKdG9cM+XFDlghEZBTQB0gWkS24eV/LAajq88BnuLlX1wCHgKtCFYsxxkQyVeWjRVu5/5Nl3DGwPZd0b8rATvXDdvyQJQJVvaSE1xW4PlTHN8aYaLBt72Hu+XAxU1fuoFvTmqQ1qxX2GKKuDLUxxsSKjxdt5Z4Pl5CTq/ztnFSu6NGcxBDdEC6OJQJjjPFJjUrl6NqkJo8MO44mtSv7FoclAmOMCZPsnFxe+Xo9WTm53NC3DX3a1aV32xREwn8VkJ8lAmOMCYNl2/Zxx7h0Fm/N4OzODVBVRMT3JACWCIwxJqQys3N45qs1/G/aWmpWLsdzI4/nrE71IyIB5LFEsHkebJgJzXtCk+5+R2OMiTEbdh7i+elrGdy1IfeenUqtKuX9Duk34jsRbJ4Hrw4EzYHECnDFJ9D0JL+jMsZEuYOZ2Uxe9hNDuzWiXf1qTLmlD03r+HczuCRRUWsoZNbPBM11z3MyYfQImP1fOGCF7Ywxx2bm6h2c+dQMbh67iDXb9wNEdBKAeE8ELXpCUkWQREgoB1VSYNJf4Yn2MOYyWD0ZcnP8jtIYEwUyDmXxl/e/57JX5lE+MYEx15xC67rV/A4rIPHdNNSkO1wx/tf3CLavgIVvwfejYPl4qN4Iuo6EbiOhVnO/IzbGRKCcXOX852ezfudB/tinFTf1axPSInHBJq7SQ/RIS0vT+fPnh/5A2Udh5WcuKayZAii07APHXw7tz4GkCqGPwRgT0XYfPErNSuVISBAmLv2RRjUr0alRDb/DKpSILFDVtEJfs0QQgL2bYdG7sPBtyNgElWpB54tdUqjXMbyxGGN8p6p88N1WHvjUFYkbcVJTv0MqUXGJIL6bhgJVswn0uQN63Q7rp8F3b8L8V+Gb56Hh8S4hdDofKlb3O1JjTIht2XOIuz9cwoxVOzihWS26t6jtd0hlZlcEx+rgLkgf45qOti+DcpWh43kuKTQ5CSJosIgxJjg+XLiFv364BAXuGNiey05uFrJZw4LNrghCoUodOOWPcPJ1sHWBu0pYMg4WvQN12sDxl0GXS6BqXb8jNcYESe0qFTiheW0ePq8TjWtFdpfQ0rArgmDKPADLPnJJYfM3kJAEbQfC8VdA636QED29CIwxkJWTy0sz15Gdo9zUrw3AzzWCoo1dEYRLharQ7VL32LHSJYTvR8OKT6FaQ9cFtdul1g3VmCiwZGsGd4xLZ+m2fZzbpWFEFYkLNrsiCLXso7Dqc/juLVg7xY1kbtH7l26o5Sr6HaExJp8jWTn8Z8pqXpixjlqVy/OPoR0Z2KmB32GVmV0R+CmpPKQOcY+MLa4b6ndvwbiroWLNX7qh1u/kd6TGGGDjrkO8NHMdw7o14q9np1Kjcjm/Qwo5uyLwQ24urJ/umo5WfAo5R6FhN68b6gXWDdWYMDuYmc3EpT8y7PjGAGzefcjXGcNCwa4IIk1CArQ63T0O7Yb0sS4pfHozTLwHUoe6XkdNT7FuqMaE2PRVO7j7g8VsyzhM58Y1aF23WswlgZLYFUGkUIVt37mEsHgcHN0PdVpDiz5QqYbrfWTzJRgTNHsOHuXBCcv44LuttEqpwj/P70xa8+gfHFYUKzERbY4ehKUfwZxnYftSty6popsvwZKBMWWWk6sMeHI6G3cd4rrerbihb+uoKhJ3LKxpKNqUr+K6mh74EaYsB3Jd76MNMy0RGFMGuw5kUqtyeRIThDsHtqdRrUp0bBiZReLCKb7nI4h0zXv+UuVUvOVYtHkezHzc/TQmBFSVsfM3c/q/pzHq200AnNGxviUBj10RRLK8+RKmPerGIFSu43dEwbd5Hrw2CHKzveav8XbVY4Jq8+5D3P3hYmau3kn35rU5pWUM/j8qI7siiHRNusOQZ0AS3BiEWLP2K8jNAhSyM13zlzFB8sF3WzjzqRl8t3EPDw7txOhrTqZlSlW/w4o4lgiiQfWG0KqvmzUt1qbO/FVfhVw3XagxQZJctQLdW9Rm0i29o6pSaLhZIogWXUfCvq2wfobfkQTXtgVQOQV63gpV68HUh2HfD35HZaJUVk4uz3y1mqe/XA1Ar7YpvH5VdxrVrORzZJHNEkG0aDcIKtZwZa5jxYEdbhrQbiOh39/gsg/hyD4YMxKyDvsdnYkyS7ZmMPiZWfx70irW7TxAtHWN95MlgmhRrqIrP7H8EziS4Xc0wbH0A9AcV28J3LSfw1508zt88ic3yM6YEhzJyuHRz1cw5NlZ7DyQyQuXncDTw7vFZJXQUAlpIhCRgSKyUkTWiMidhbzeVESmishCEUkXkUGhjCfqdRsJ2Udg6Yd+RxIc6WOh3nFQL/WXdR3OgdP/6mZ/m/0f/2IzUWPT7kO88vU6Lji+MV/e3JszO9b3O6SoE7JEICKJwLPAWUAqcImIpBbY7K/AWFXtBgwHngtVPDGh4fGQ0h4WxkDz0K61sHU+dL7wt6/1ug06DoPJf4dVE8Mfm4l4+49k8d78zQC0rVeNqbf14Z8XdI6LSqGhEMorgu7AGlVdp6pHgdHAkALbKJBXarMGsC2E8UQ/Eeg6ArbMg52r/Y6mbBa/B4hr7ipIBIY8C/WPg/evdpP8GOOZumI7Zz45gzvGpbNm+36AmJo20g+hTASNgM35lrd46/K7D7hURLYAnwE3FrYjEblGROaLyPwdO3aEItbo0flikMToHlOg6pp+mp8GNQr+SXjKV4ZLRkG5SvDuxa5Kq4lruw8e5eYxi7jq9W+pUiGJ96/rQeu61fwOKyb4fbP4EuB1VW0MDALeEpHfxKSqL6pqmqqmpaTEeT/zavWhdX83BWa0jinYugB2r/vlJnFRajSG4e+4brPvXQk52WEJz0SenFzlgv/N5pPvt3FTvzZ8etNpHN+0lt9hxYxQJoKtQJN8y429dfldDYwFUNU5QEUgOYQxxYauI2D/Nlg31e9Ijk36WEisAKmDS962SXc450k3kc+ke0Ifm4koO/ZnkpurJCYIdw/qwCc3nsYtA9pSISm2K4WGWygTwbdAGxFpISLlcTeDxxfYZhPQD0BEOuASQZy3/QSg3VlQqVZ0Ng/lZMGScdBuoBsXEYhul8LJ18M3z8OCN0Ibn4kIqsqYbzfR9/FpvDvPFYnrn1qPDg1s9r5QCFnROVXNFpEbgIlAIvCqqi4VkQeA+ao6HrgVeElEbsbdOL5SbRRIyZIqwHEXug/Fw3tcUogW66bBoZ0lNwsVNOAB2LECJtwKyW2gWY+QhGf8t2nXIe78IJ3Za3dxUovanNbaGglCzSamiVbbFsKLfeDsJ+DEq/2OJnDjfg+rJ8NtqyGpfOnee3gvvNzP/bxmKtRsGooIjY/eX7CFez9aQmKCcNeg9lxyYlOrDxQkxU1M4/fNYnOsGnSFuh2jq3ko8wCsmAAdzyt9EgCoVBMuGe2al0aNcPszMaVe9Qr0aFWHybf0YuRJViQuXCwRRKu8MQVb50dPP/sVEyDrUOmbhfJLbgMXvuqm8PzoOsjNDV58JuyOZufy9JereXLyKgB6tknhlStPpEENKxIXTpYIolnni7wxBVEy0jh9DNRoCk1OKtt+WveHAQ/C8vEw47HgxGbC7vvNezn3v1/z5Jer2Lz7kBWJ85ElgmhWtS60PRO+HxP5fewPbHfdXTtfCAlB+LM75XpXmnvaI7Ds47Lvz4TN4aM5PDRhGec9N4uMw1m8fHkaT1zc1YrE+cgSQbTrOsJNcr/2K78jKd6ScaC5cNxFwdmfiBtf0Lg7fHgt/JAenP2akNu85xBvzN7I8O5NmXRLL/qn1vM7pLhniSDatTnTzWUc6c1D6WOhfmeo2z54+0yqABe/7brPjh7h5jcwEWnfkSzG5isSN+32Pjx83nFUr2hF4iKBJYJol1Tefcte+Vnk1uPZuRq2fVe2m8RFqVbPlaE4uBPGXArZR4N/DFMmX634iTOemMGd49JZs9319GpoM4ZFlBITgYg8LiIdwxGMOUZdR0DOUdf8EonSx+IqjZ4fmv037AZDn4XNc2HCLTahTYTYdSCTP41eyO9en0+NSuX44I+n0rquTRwfiQIZWbwceFFEkoDXgFGqGiNTZMWIBp3dBC+L3oHuf/A7ml9ThcVjoWVvqN4gdMfpdD78tAxm/hvqdYKTrw3dsUyJcnKVC5+fw+Y9h7i5f1uu69OK8knWABGpSvyXUdWXVfVU4HKgOZAuIu+KyOmhDs6UQreRbrTxT8v8juTXtnwLezYE7yZxcU6/B9qdDRPvivyb5zFq+/4jPxeJu+fsDnx6Y0/+1L+NJYEIF9C/jjfbWHvvsRP4HrhFREaHMDZTGsddCAlJkXfTOH0sJFWEDueG/lgJCTDsBTeL23tXuVnQTFjk5irvfLORvv+ezjtekbh+HerRrr7NFxANArlH8CSwAjdfwMOqeoKq/lNVzwW6hTpAE6AqydB2oPvgzcnyOxrn50qjg6BimKpGVqjmJrSRBBg1HI5YK2apbJ4HMx93PwO0YedBRrw8l3s+XELnxjXo3SbO5wyJQoFcEaQDXVX1/1S14F9H9xDEZI5V15FwcDusmeJ3JM6aKXB4txsBHU61msNFb7rJb8b9Pnon8Am3tdPg1YHw1T/gjcEBJYOx8zdz5lMzWLp1H48OO453fn8STevYtJHRJpBEsJd8N5VFpKaIDAWwm8YRps0AqJwMi972OxJn8VioVBta9Qv/sVv0hLMeg9WTYMr94T9+NJr1FGiOG/iXcxQ2zCzxLY1qVqJX2xQm39Kb4d2b2ujgKBVIIvh7/g98Vd0L/D1kEZljl1jO9dVf+QUc3OVvLJn7YcVnx15pNBhOvBrSroZZT7upPU3RDu3+9RVAYnlo3vM3m2Vm5/Dk5FU8MckVOjy1dTIvXZ5G/RoVwxWpCYFAEkFh24RsQhtTRl1HQG4WLHnf3ziWfwrZh0MziKw0zvqn+0AbfxNssXksijT7P64ybF7vroGPumlC81m4aQ/n/vdrnp6ymq17j1iRuBgSSCKYLyJPiEgr7/EEsCDUgZljVL8TNOgCC31uHkofAzWb/ebDJOwSy8GFb0C1+jB6JOzb5m88kWj/TzD3edfzbNC/IKEc7Fr988uHjmbz4KfLGPa/2ew/ks2rV6bx+EVdrBkohgSSCG4EjgJjvEcmcH0ogzJl1HUk/JgOPy725/j7f3STzXe+yBWH81uVOm5Cm6MHXE2irMN+RxRZZv7b3RPoc6eb/KfV6a7Et/eNf+uew7w1dyMjT2rKpJt70be9FYmLNYEMKDuoqneqapr3uEtVD4YjOHOMjrvQfatbNMqf4we70mgw1EuFYS/BtkUw/kYrQ5Fnz0aY/xocfxnUaeXWdRgMezcx8csvAGhTrxrTb+/DP4YeRzUrEheTAhlH0FZEXhSRSSLyVd4jHMGZY1S5NrQ7yzXP+DGmIH2Mm0ozpW34j12c9oOg719h8Xvw9ZN+RxMZpj/mxlz0+svPq6aSRjYJrJ0x6ucicTZjWGwL5Kbve8DzwMuAdciOFl1Husv71ZOg/dnhO+6OlfDD93DmI+E7Zmn0vBW2L4MpD0DdDi5hxqsdq+D7d+HkP0KNRuw8kMl945fyafoPfFC1M1dXT6dCShW/ozRhEMg9gmxV/Z+qzlPVBXmPkEdmyqZ1f6hSN/yT26ePdd8wQ1VptKxEYPAz7ob6uN/D9uV+R+SfqQ9Bucpw2s3k5CoX/G82k5b+xG1ntKXLGZdRYd96lzRNzAskEXwiIn8UkQYiUjvvEfLITNkkJkGXi2HVF+GbsOXnSqN93DwBkap8ZRj+rvsQHDU8cudxCKVti2DZRxzodg25leqQmCD8/dyOTLjpNG7o24bEDucCAsvG+x2pCYNAEsEVwO3AbFy30QWAdciOBl1HQm62axMPh83fwN5N/o8dCESNRi4Z7NsGYy+PnPpMYaJf/YPMctXpP7cz73yzEYDT29elTT2vSFy1etCsh2teNDEvkF5DLQp5tAxHcKaM6naAhseHr3kofQwkVQrvPYmyaHIinPu0K6Uw8W6/owmbrd9/hayZzBOHzqZ1k4b0aVe38A07DHZNQztXF/66iRmB9BqqLCJ/FZEXveU2InJO6EMzQdF1BPy02N3ADaXso7D0Q5cEKkRR6eGuI+CUG2Dei64bZYwbM28j2z64ix3UpO3gW3jr6u40qV1Ekbi80uHLPg5fgMYXgTQNvYYbUNbDW94K/CNkEZng6nS+qxsT6quCNV/C4T3R0SxU0IAH3M31z26DDbP8jiakOh1ZwImygop97+T8k9oWPzq4RiNofKI1D8WBQBJBK1V9DMgCUNVDQAQMFzUBqVzbfUtPHxvaid0Xj4XKddyo1GiTkAjnvwK1Wribx5PuLVU9/kiWmZ3Dvyeu5N8TV4IqHZc/BTWbUq3H1YHtoMNgdzW5e31I4zT+CiQRHBWRSoACiEgrXJkJEy26jnTzAqyeGJr9H8mAlZ97Vx9ROvK0Uk3oczdk7nMF2N44N+qTwYKNuxn09EyembqG7fuPoMs+dh/qfe4KvCJs6mD3c/knoQvU+C6gMtTAF0ATEXkHmAL8pfi3mIjS8nSoWh8Whmgay+WfQPaRyCopcSz2rufn/xLZgdXjj0QHM7O5b/xSLnh+Dkeycnnjd915bFgnZOpDkNyudM13tZq7MRfWPBTTAuk1NBkYBlwJjALSVHVaIDsXkYEislJE1ojInUVsc5GILBORpSIS5tFPcSIxCboMd6OMD2wP/v7Tx7pmlcZpwd93ODXvCUkV3HOh0Hr80WDb3sO8O28Tl5/cjIk396J32xTXo2vnKuh7j2sKK40Og2HLt5CxNTQBG98F0muoF9AR2A/sA1K9dSW9LxF4FjgLSAUuEZHUAtu0Ae4CTlXVjsCfS3sCJkBdR7jZp9LHBne/+7bB+hmRU2m0LJp0hyvGQ5OT3ejolHZ+RxSwjENZvPuNmzS+Tb1qzPzL6dw/pBNVKyS5q5tpj7j6Tx0Gl37nqUPcT2seilmBNA3dnu9xL/AJcF8A7+sOrFHVdap6FBgNDCmwzR+AZ1V1D4CqhuDrqgHch1qjNFj0TnArby4ZB2j0NwvladLd9SLKzYZVIbqnEmRfLPmR/k9O596Pl7B2hysSV696vhnDvnvDDfTrd++xJevkNlA31ZqHYlggTUPn5nsMADoBewLYdyNgc77lLd66/NoCbUVklojMFZGBhe1IRK4RkfkiMn/HjjCVS4hF3Ua6AUI/LArePtPHQKMTILl18Pbpt8YnQrWGEd9/fvv+I/zxnQVc+/YCUqpW4OPrT6VVStVfb3T0EMz4FzQ7tWxzR3cYDBtnu0lsTMwJ5IqgoC1AhyAdPwloA/QBLgFeEpGaBTdS1Rfz5kNISUkJ0qHjUMdhkFgheGMKti93k9/EytVAnoQE11tm9WQ393IEyslVLnp+Dl8u387tZ7bj4xtOpVOjGr/dcN6LcOAn6HuMVwN5UgcDCis+PfZ9mIgVyD2C/4rIf7zHM8BM4LsA9r0VaJJvubG3Lr8twHhVzVLV9cAqXGIwoVCpJnQ4x9Ueyg5CD+D0sSCJ0GlY2fcVaVKHQE5mxDUP/ZBxmNxcdUXiBnfks5t6cv3prSmXWMh/5SMZbt6F1gOg2SllO3DdVKjT2pqHYlRAcxbzS7G5OcAdqnppAO/7FmgjIi1EpDwwHCj4V/QR7moAEUnGNRWtCyhyc2y6jnQjgFd+Xrb95Oa6hNLqdKhaRK2aaNbkZNfldtlHfkcCQG6u8vqs9fR7fDpv5xWJa1eX1nWrFv2m2c/Akb1uMp6yEnHNQ+tnxme11hgXyD2CN/I93lHVgMbgq2o2cAMwEVgOjFXVpSLygIjkdV2YCOwSkWXAVOB2Vd11bKdiAtKyj2v/Lmvz0Oa5kLE5OktKBCIhwdXaWT0ZMg/4Gsqa7Qe46IU53PfJMtKa16Zv+wAS78GdMPc5SB0KDbsGJ5DUwa7n2YoJwdmfiRglzlAmIovxRhUXfAlQVe1c1HtV9TPgswLr/pbvuQK3eA8TDgmJbkzBrKfcJPPV6h/bftLHQLkq0VNp9FikDoFvX4I1k6Hjeb6EMHreJv42fimVyiXy+IVdGHZ8o+LrA+X5+knIOgSn3xO8YBp0hZpNXfPQ8ZcFb7/Gd4E0DX2OG1k80nvkfbifA5wbutBMyHQd6SaXTx9zbO/PzoSlH7kkUD6GpzJs1gOqpLhz9UnTOpXp36EuX97Sm/NPaBxYEsjYCvNegi4jgjtvdF7z0NqpcHhv8PZrfBdIIhigqn9R1cXe407gDFXdqKobQx2gCYHk1tDkJNc8dCxjClZPdm3PsdoslCch0WsemuS6YYbBkawcHvtiBY99sQKAHq2SeW7kCaRUqxD4TmY85hJ97xBUgkkdCrlZEXcT3ZRNIIlAROTUfAs9AnyfiWRdR8COFbA1kA5gBaSPcd+UW/YJelgRJ3Woa2JZMznkh5q/YTeD/jOT56atZffBo+ixJOlda+G7tyDtKqjVLPhBNjrB3WOy3kMxJZAP9KuB50Rkg4hsAJ4DfhfSqEzodTzPzSa2qJSF6A7vdd8GO53vahjFumanuvLaIRxcdiAzm79/vIQLX5jD0exc3vxddx49v3NgzUAFTXvEzT/R87bgBwq/3ERf86XvN9FN8ATSa2iBqnYBugBdVLWrqh7D10gTUSrWcP+hl7wPWUcCf9/y8a5/fecYG0RWlMQkaH+OS35Zh0NyiB8zDjP6281ccUpzJv65F73aHuOgyZ+WwuL34eRr3ZzDoZI6xFWbXT0pdMcwYRXIgLJ6IvIKMFpVM0QkVUQCnNXCRLSuI7y5BErRHTB9LNRu5eZCjhcdh8LRA7BmStB2uefgUd6a626xta7risTdN7gjVSqU4Srrq4egQnXocVOQoixC05Nd06A1D8WMQJqGXsf192/oLa/CqoTGhha9oXrjwMcUZGyBDV+7m8TRXmm0NJr3hEq1gtI8pKp8tvgHBjw5nfvHL/25SFzd/EXijsWW+S6hn3qjm5UulBISvaukSSG7SjLhFUgiSFbVsUAu/DxQLCekUZnwSEiArpfA2q9cOemSLH4fV2n0gpCHFlESy7kPvpWfl6k0x/Z9R7j27QX88Z3vaFCjEuNvOO23ReKO1ZQHoHIynHRdcPZXktQhkHUwqFdJxj+BJIKDIlKHX6aqPBnICGlUJny6XOK6Gn4/uuRtF7/nKnPWaRX6uCJN6lA4ut8lzWOQk6tc+MIcpq3cwV1ntefDP/YgtWH14MS2bhqsnw49b4UKQUosJWl+mrtKsuahmBBIg+QtuBpBrURkFpACxNlXwhhWpxU07eGah067uegmn5+Wwk9LYNC/wxtfpGjRy91gX/YxtDsr4Ldt23uY+tUrkpggPDCkE01qVaJlsK4CwI0DmfIgVG8EaWHszJdYDtqd7RJBduYvM7uZqFTsFYE3y1hv79ED+D+go6qmhyE2Ey5dR8Cu1a6duSh5lUZ9KrXgu6TyrnloxWcBNQ/l5CqvFSgS17ttSnCTAMCqL2DrfOh9B5Qr432G0kodApn7YN308B7XBF2xiUBVc4BLVDVbVZeq6hJVzQpTbCZcOg6FcpVh0duFv56b6+4PtO4PVZLDGlpESR0CmRklfvCt2b6fC5+fzf2fLOOklrXp1yFEXTlzc93VQO1WLpmHW8verpfS8siewMeULJB7BLNE5BkR6Skix+c9Qh6ZCZ8K1dyH3JIPCu8Fsmk27NsSP2MHitKyD1SoUWzvoXe/2cSgp79m/c6DPHlxF1678kQa1awUmniWfgDbl8Lpd7ummnBLqgBtB7pqpDn2/TCaBZIIuuImr38AeNx7xGlDcQzrOsJd5hdWYjh9DJSvCu0GhT+uSJJUwd0fWPFpkR98zZMrc0bHeky+pTfndQuwSNyxyMmCqQ9B3Y5u5jm/pA5x81ts+Nq/GEyZFZkIROQGAFU9HbhBVU/P9+gbtghNeDQ7zZUYXligeSjrCCz92I1CLl/Zn9giSeoQV3BvvWseOpKVwyOfL+fRz38pEvfMiONJrhrim6eL3oHd69yE9Ak+lv5q3c+VI7feQ1GtuL+g/F0Q3gp1IMZnCQmubPG6aW7gWJ7Vk1y7+HEX+hZaRGnVF8pXg6Uf8c26XZz19ExemL6O/Ueyjq1I3LHIOgLTH3NdedsODM8xi1KuErQZAMs/gVwbXhStAv0qEUfDSONYl+GA/npMQfoYqFrPjUI2UK4iWa3P4FD6x4x88WtycpV3f38SD513XOiagQqa/yrs2wr9/hYZI7xTh8DBHbBprt+RmGNUXCKoKSLnicj5QHURGZb/Ea4ATRjVbuGaiBa94/qnH97jrgjipdJogHY2HUTlnH3c33kPX/y5Jz1ah7EnVeZ+mPm4u3Hdolf4jlucNmdAUkVrHopixSWC6cBg3ExkM3CzkeU9zgl9aMYX3Ua6tufN37jeMTlHrbcQsPvgUd6aswGABiecjZarzMhqC6lcPswJcu7zcGgn9P1byduGS4Wq0KofLBvvurSaqFPkX7GqXhXOQEyE6DAYJtzmrgp2rYXktm6u2jilqnya/gP3jV/KviNZnNo6mZYpVZG2A2H5pzDo8fBdLR3aDbP/40b0Nj4hPMcMVOoQV/Ru6wJocqLf0ZhSspnGzK9VqOoGmC1+HzbOguMuiox2aB/8tO8If3hzATeOWkijWpX45MbTfhkZ3HGo+2a+cVb4Apr1tGsa6hvECemDpe2ZkFDOBpdFKUsE5re6jnTTMwIkt/E3Fp/k5CoXvTCHmat3cM+gDnxwXQ/a189XJK71ADcaO4Qzl/3K/h/hmxdc7616HcNzzNKoVNPdt1j28bHNg218ZYnA/JYk8nNHsQ+vhc3zfA0nnLbsOUROrpKYIDw4pBMT/9yLP/RqSVJigf8q5Su7m6Th6jY583E3aXyfO0N/rGOVOgT2boIfvvc7ElNKgcxQVllE7hWRl7zlNiJiN4tj2aZZvzQH5RyFDTP9jScMcnKVl2euo/8T03nbmzmsV9sUmidXKfpNqUPg4HbYNCe0we3ZCPNfg26XRXYJ8PZnuy8R4bpKMkETyBXBa0AmcIq3vBX4R8giMv5r3hMSK7j/1Inl3XIMW/njfob9bzb/mLCcU1slc0bHAIvE5XWbDPUH3/R/giRAr9tDe5yyqlzbzVOwfLw1D0WZQBJBK1V9DMgCUNVD2ACz2NakO1wx3t2UvGK8W45Rb8/dyDn/ncnm3Yd4enhXXr4ijQY1AiwSV6GqG1Ubym6TO1bC96Og+x+gRqPQHCOYUofArjWwfbnfkZhSCCQRHBWRSvwyQ1kr3BWCiWVNursZr2I0CeSVg2hdtyqDjmvA5Jt7MaRro9KPDk4dCgd+dOMuQmHqQ+6m9Gk3h2b/wdb+HECseSjKBJII7gO+AJqIyDvAFOAvoQzKmFA5fDSHhyYs49EvXJG4k1vW4enh3ahzrEXi2p7pmtFC8cG3bZHb7ynXR888ENXqQdNTbJRxlCkxEajqJGAYcCUwCkhT1WmhDcuY4JuzdhcDn57BSzPXcygzJzhF4ipUcxP2LPs4+M1DX/3DzQt8yvXB3W+opQ6B7ctg52q/IzEBCqTX0CfAGcA0Vf1UVXeGPixjgmffkSzu+mAxl7zkiqK9+4eTeHBop+AViUsdAvu3uSkjg2XjbFgz2TUJVawRvP2GQ4dz3U9rHooagTQN/RvoCSwTkfdF5AIRCfPkqMYcu+37Mvlo4Vau6dWSL/7Uix6tgtzM0m6g610VrA8+VZjygKv6euIfgrPPcKrRCBqlWfNQFAmkaWi6qv4RaAm8AFwEbA9k5yIyUERWisgaESlyJIyInC8iKiJpgQZuTHF2Hcjk9VnrAXdD+Os7TufuQR2oVD4x+AerWMPNUxCsUbVrprixCb1uj97JgFKHuIFlezb4HYkJQEAji71eQ+cD1wInAm8E8J5E4FngLCAVuEREUgvZrhrwJyBE3S5MPFFVPl60lf5PTOehz5azbscBgGO/GRyo1KGQsRm2fle2/eTmwpT73Wxxx18RlNB8kTrY/VxmVwXRIJB7BGOB5UBf4BncuIIbA9h3d2CNqq5T1aPAaGBIIds9CPwTOBJw1MYUYtvew1z9xnz+NHoRzepUYcJNPX8pEhdq7Qa6omvLPizbfpaPhx/Toc/dkFQ+OLH5oVZzqN/ZmoeiRCBXBK/gPvyvVdWpqhpo14hGwOZ8y1u8dT8TkeOBJqpayIzpv9ruGhGZLyLzd+zYEeDhTTzJzsll+ItzmbN2F/eek8q463rQtl618AVQqVbZi67l5rhxA8ntYmMOiNQhsOVbyNjqdySmBMVNXp83QX0VYEiwZygTkQTgCeDWkrZV1RdVNU1V01JSUsp6aBNDNu92ReKSEhN4+LzjmPjnXlx9WgsSE3wY/N5xqFd0bdGxvT99DOxcBX3/CgkhuJcRbqleA8DyT/yNw5SouCuCvElqzy3kEUjRua1Ak3zLjb11eaoBnYBpIrIBOBkYbzeMTSCyc3J5ccZa+j8x/eeZw05rk0zTOj7eXG03CBKSYOlHpX9vdiZMfcRNApTX/TLaJbeBlA7WPBQFipuh7O/e0wdUdX3+10SkRQD7/hZo4227FRgOjMi3/wzg5358IjINuE1Vg9gZ28Si5T/s445x6aRvyWBAaj3OOq6B3yE5lWtDi96ueaj/faWb0Oe7NyFjE5z7ZGxNBJQ6xBXNO7Adqtb1OxpThEDuEYwrZN37Jb1JVbOBG4CJuJvNY1V1qYg8ICKDSxemMc5bczZw7n+/ZuuewzwzohsvXnYC9apH0LCW1CGwZ7274Ruoowdh+mPQ7FQ3928sSR0MqDUPRbgirwhEpD3QEahR4J5AdSCg/3mq+hnwWYF1hc66rap9AtmniU+qiojQtl41zu3SkHvPSaV2lQjsVdP+HPj0ZndV0KBLYO+Z96Kb1+CiN2PragCgbirUbuWah0682u9oTBGKm3W7He5eQE3cfYE8+4EoHO5ootGho9n8e+IqkhKFuwd14KSWdTipZR2/wypalTrQoqe7T9D33pI/2A/vha+fcnMbNDul+G2jkYi7Spr1NBza7ZrPTMQpsmlIVT9W1auAc1T1qnyPm1R1dhhjNHFq1pqdnPnUDF6dtZ6j2bnBKRIXDqlDYPda+GlpydvOeRaO7HU9hWJV6mDQHFhRbC9x46PirgjyLBSR63HNRD83Canq70IWlYlrGYezeHjCcsbM30yL5CqM/b9T6N4iir5Jtj8XJtzqmofqdyp6uwM7XCJIHRp4M1I0atDVjZRePh6Ov8zvaEwhArlZ/BZQHzgTmI7rBro/lEGZ+LbzQCafpG/j2t6t+PxPPaMrCQBUTXE3fksqQvf1k5B9GE6/Jzxx+UUEOgyGtVPhSIbf0ZhCBJIIWqvqvcBBVX0DOBs4KbRhmXizY38mr37teim3SqnK13f05c6z2lOxXJQOrEodAjtXFj1lY8YW+PZl6DICUtqGNzY/pA6B3CxY+YXfkZhCBJIIsryfe0WkE1ADsA7BJihUlQ8XbmHAk9N59PMVrN95ECAyewSVRofBFDtl4/THQHOhzx1hDcs3jdKgWgMbXBahAkkEL4pILeBeYDywDHgspFGZuLB172Guev1bbh7zPS2Tq/DZn06jRXIVv8MKjmr1oFmPwkcZ71oLC9+GtN+5tvN4kJDgkuOaLyHzgN/RmAICmY/gZVXd481L0FJV66rq8+EIzsQuVyRuDvPW7+a+c1N579oetK4bxiJx4ZA6FHYshx0rf71+2iNuIpueJZbZii2pgyH7CKye5HckpoDiBpTdUtwbVfWJ4IdjYt2mXYdoVKsSSYkJPDqsM01rV6ZJ7SidfKUkHc6Fz293Nfl73+7W/bgEFr8Pp/3ZXTXEk6anQJUU1zzUqcx1K00QFXdFUK2EhzEBy87J5X/T1tL/yem86RWJO7V1cuwmAYDqDaDJybDso1/WTX0IKlSHHjf5FpZvEhLdyOtVkyDrsN/RmHyKKzp3fzgDMbFr6bYM7hiXzpKt+zizYz3OjpQiceHQcSh8cSfsXAOH98DKz9zgsXgdYZs6GBa85qbj7BBIEWMTDoHMUNZWRKaIyBJvubOIxPAwSBNMb8zewJBnZvFjRib/G3k8L1yWRt1IKhIXah3ypmz8CL56AConw0nX+RqSr5r3hIo1rfdQhAmk19BLwF143UhVNR1XUtqYIuWVg2hfvxpDujbiy1t6RU656HCq0Qgad3f1hNbPgOMuhAphmj4zEiWWc81DK79wczCYiBBIIqisqvMKrMsORTAm+h3MzOa+8Ut5+DM3kOqklnV4/KIu1Kwc5eMCyqJhNzjqDcZf8BpsLvjfKc6kDobMDFg33e9IjCeQRLBTRFoBCiAiFwA/hDQqE5VmrNrBGU/O4I05G8jK0egpEhdq5Sr98jwnCzbM9C+WSNCyj7thvryEEhwmbAJJBNcDLwDtRWQr8Gfg2lAGZaJLxqEsbnvvey5/dR4VyiUw9v9O4b7BHZFYq61/rNqfDUkVQRLd+IHmPf2OyF9JFaDtQFeNNCer5O1NyJVYfVRV1wH9RaQKLnEcwt0j2Bji2EyU2Hkwk88X/8Af+7Tipn5torc+UKg06Q5XfOKuBJr3dMvxLnUwLB4LG76GVqf7HU3cK25AWXXc1UAj4GPgS2/5ViAdeCccAZrItH3/EcYv2sbve7b8uUhcrWivDxRKTbpbAsivVT8oV9n1HrJE4Lvimobews1Sthg3I9lU4ELgPFUdEobYTARSVd5fsIUBT8zgsYkrfy4SZ0nAlEr5ym5WtuWfQm6O39HEveKahlqq6nEAIvIy7gZxU1U9EpbITMTZvPsQd3+4mJmrd5LWrBaPnt85dorEmfBLHezGV2yaC81P9TuauFZcIvj5Lo6q5ojIFksC8Ss7J5dLXprLnoNHeXBIR0ae1IyEBLsZbMqgzRmQWME1D1ki8FVxiaCLiOzzngtQyVsWQFW1esijM77bsPMgTWpXJikxgccucEXiGteK4fpAJnwqVIPW/WH5J3DmI65UtfFFcZPXJ6pqde9RTVWT8j23JBDjsnJyeXbqGs54csbPReJ6tEq2JGCCK3Uw7NsKn91qA+18ZCnY/MaSrRkMeWYW/5q4kgGp9Tinc0O/QzKxqkqK+zn/NXhjsCUDn5Q4jsDEl9dmrecfE5ZTu0p5nr/0BAZ2qu93SCaW/bDIe6KQk+nGWlg327CzRGAA1y1UROjYsAbDujXir2enUqNyOb/DMrGueU836jr7iJvDuVYLvyOKSxJt9WDS0tJ0/vz5focRMw5kZvPYFyson5jAX89J9TscE482z4NlH8N3b0LlOvD7L6FKst9RxRwRWaCqaYW9ZvcI4ti0lds588kZvDV3IwpWJM74o0l3OPMhuHQc7NsGoy6xGczCzBJBHNpz8Ci3jF3Ela99S6Xyibx/bQ/uPSfVisQZfzXpDsNehC3z4MNrITfX74jiht0jiEN7Dh1l0tKfuKlva67v25oKSVYkzkSIjkNh74Mw+V6Y0gwGPOB3RHEhpFcEIjJQRFaKyBoRubOQ128RkWUiku5Nh9kslPHEs+37jvDijLWoKi1TqjLrjr7cckY7SwIm8vS4EdKuhllPw/xX/Y4mLoQsEYhIIvAscBaQClwiIgXvRi4E0lS1M/A+8Fio4olXqsrYbzfT74npPD5pFRt2HQKwHkEmconAWY+5EhQTboPVk/2OKOaF8oqgO7BGVdep6lFgNPCrqqWqOlVVD3mLc4HGIYwn7mzefYjLXpnHX8al06FBdT7/U08rEmeiQ2ISXPAa1OsI710JP6T7HVFMC2UiaARszre8xVtXlKuBzwt7QUSuEZH5IjJ/x44dQQwxduUViVu0eS//GNqJ0X84mZYpcTxpuok+FarCiLFQsQa8exFkbPU7opgVEb2GRORSIA34V2Gvq+qLqpqmqmkpKSnhDS7KrN95kJxcJSkxgX9d0IVJN/fi0pOtUqiJUtUbwMj3IPOASwZH9pX8HlNqoUwEW4Em+ZYbe+t+RUT6A/cAg1U1M4TxxLSsnFz+O2U1Zz45gzdmbwDglFZ1aFizUvFvNCbS1esIF70B25e7ZiKb5zjoQpkIvgXaiEgLESmPm+d4fP4NRKQb8AIuCWwPYSwxLX3LXs7979c8PnkVZ3aqz+CuViTOxJjW/eCcJ2HtFJhwC9jgx6AK2TgCVc0WkRuAiUAi8KqqLhWRB4D5qjoe1xRUFXjPG8y0SVUHhyqmWPTq1+v5x4RlpFSrwEuXpzEgtZ7fIRkTGidcAXs3wszHXU2inrf4HVHMCOmAMlX9DPiswLq/5XveP5THj2V5ReI6N67BxSc24c6zOlCjknUJNTHu9L/Cno0w5X6o1Qw6ne93RDHBRhZHmf1Hsnj08xVUSErkb+emkta8NmnNa/sdljHhkZAAQ59zNYk+vA6qNYRmp/gdVdSLiF5DJjBTV2znjCdnMGreJpISxYrEmfiUVAGGvwM1GsPoS2DXWr8jinqWCKLA7oNH+fPohVz1+rdUq5jEuOt6cPegDlYkzsSvyrVdt1JJgLfPh4M7/Y4oqlkiiAIZh7OYsnw7f+rXhk9v7Em3prX8DskY/9VpBZeMttLVQWCJIEL9mHGE56e7InEtkqvw9Z19uXlAW8on2T+ZMT+z0tVBYZ8qEUZVGTVvEwOemM5TX65iY16ROOsRZEzhOg6FAQ/Cso9cbyJTatZrKIJs3HWQO8ctZs66XZzcsjaPDutMcysSZ0zJetwIezbArKdct9K03/kdUVSxRBAhsnNyGfHSN2QczuLh845j+IlNrD6QMYHKK12dsdmVrq7RBNoM8DuqqGGT1/ts7Y4DNKtdmaTEBOau20WzOpVpUMPqAxlzTDL3w2tnwe71cNXn0KCz3xFFDJu8PgIdzc7lqS9XMfCpGbw5ZyMAJ7esY0nAmLKoUM1KVx8DSwQ+WLTZFYl76svVDDquAUO7FTdNgzGmVKo3dMnASlcHzBJBmL3y9XqGPTeLjMNZvHJFGk8P70btKuX9DsuY2FK/k5WuLgVLBGGSdy+ma5MaDO/elEm39KJfB6sUakzI/Kp09a1WuroY1msoxPYdyeKRz1ZQsVwCfz+3Iyc0q80JzaxInDFhccIVrlvp109A7RZw2s1+RxSR7IoghL5c9hMDnpjOmG83UT4pwYrEGeOHvve6ctVf3gdLxvkdTUSyK4IQ2HUgk/s/Wcb477fRvn41XrwsjS5NavodljHxKSEBhljp6uLYFUEI7D+SzdSV27m5f1vG33CaJQFj/FauIgx/10pXF8ESQZBs23uYZ6euQVVpnlyFWXf25U/921iROGMiRf7S1e9cAAd3+R1RxLBPqTLKzVXenruRM56cwTNfrfm5SFz1ilYkzpiIk1e6OmOruzLIOuJ3RBHB7hGUwfqdB7lzXDrfrN/Nqa3r8Mh5nWlap7LfYRljipNXuvq9K+DdC6FFb2jRy62PU5YIjlF2Ti6XvvwN+45k8dj5nbkwrbHNGGZMtOg4FDb8Hr59GdbPAATqpkKdllClLlT1HgWfl4/NL3qWCEppzfb9NK9ThaTEBJ68uCvN6lSmXvWKfodljCmtag0BAdQ9jh6Enathwyw4vLvw95SvClVSoGo9qJriJYr8z6MzaVgiCFBmdg7PTl3Lc1PXcNegDlx9Wgu6t7CBYcZErRY9Iaki5ByFxPJw/ku/NA/lZMHBHXBgu/fzp98+37kaNnwNh/cUvv/yVfNdVRSTNPZsgq3fQvOevjVPWSIIwHeb9nDH++ms3n6AYd0aMcyKxBkT/Zp0hyvGw4aZv/0QTiznitdVb1jyfrKPwqGdLjkc2A4Ht//2+Y5VxScNxCWlK8b7kgwsEZTgpRnrePjz5TSoXpHXrjqR09vV9TskY0ywNOle9g/epPKlSxoHd3gJYgcsfAuWfwKouzLZMNMSQSTJzVUSEoTjm9Vk5ElNuWNge6pZl1BjTFkklYcajdwDoFJNWD35l+ap5j39CcuXo0awjMNZPDRhGZXKJXL/kE5WJM4YEzrFNU+FkSWCfCYu/ZF7P1rCroNH+b9eLVFV6xJqjAmtYDRPlZElAmDngUz+/vFSJiz+gdQG1Xn1yhPp1KiG32EZY0xYWCIADhzJZubqHdx+Zjuu6dWScolWecMYEz/iNhFs3XuYD7/bwvWnt6Z5chVm39WPqhXi9tdhjIljIf3qKyIDRWSliKwRkTsLeb2CiIzxXv9GRJqHMh5wvYHemrOBM56YzrNT1/5cJM6SgDEmXoXs009EEoFngQHAFuBbERmvqsvybXY1sEdVW4vIcOCfwMWhimntjgPcNW4x8zbspmebZB4+7zia1I6eYeDGGBMKofwa3B1Yo6rrAERkNDAEyJ8IhgD3ec/fB54REdEQzOmYnZPL5a/MY/+RLP51QWcuOMGKxBljDIQ2ETQCNudb3gKcVNQ2qpotIhlAHWBn/o1E5BrgGoCmTZseUzBJiQk8NbwrzWpXpq4ViTPGmJ9FRfcYVX1RVdNUNS0lJeWY93Ni89qWBIwxpoBQJoKtQJN8y429dYVuIyJJQA3A5o8zxpgwCmUi+BZoIyItRKQ8MBwYX2Cb8cAV3vMLgK9CcX/AGGNM0UJ2j8Br878BmAgkAq+q6lIReQCYr6rjgVeAt0RkDbAblyyMMcaEUUg7z6vqZ8BnBdb9Ld/zI8CFoYzBGGNM8aLiZrExxpjQsURgjDFxzhKBMcbEOUsExhgT5yTaemuKyA5g4zG+PZkCo5bjgJ1zfLBzjg9lOedmqlroiNyoSwRlISLzVTXN7zjCyc45Ptg5x4dQnbM1DRljTJyzRGCMMXEu3hLBi34H4AM75/hg5xwfQnLOcXWPwBhjzG/F2xWBMcaYAiwRGGNMnIvJRCAiA0VkpYisEZE7C3m9goiM8V7/RkSa+xBmUAVwzreIyDIRSReRKSLSzI84g6mkc8633fkioiIS9V0NAzlnEbnI+7deKiLvhjvGYAvgb7upiEwVkYXe3/cgP+IMFhF5VUS2i8iSIl4XEfmP9/tIF5Hjy3xQVY2pB67k9VqgJVAe+B5ILbDNH4HnvefDgTF+xx2Gcz4dqOw9vy4eztnbrhowA5gLpPkddxj+ndsAC4Fa3nJdv+MOwzm/CFznPU8FNvgddxnPuRdwPLCkiNcHAZ8DApwMfFPWY8biFUF3YI2qrlPVo8BoYEiBbYYAb3jP3wf6SXTPZF/iOavqVFU95C3Oxc0YF80C+XcGeBD4J3AknMGFSCDn/AfgWVXdA6Cq28McY7AFcs4KVPee1wC2hTG+oFPVGbj5WYoyBHhTnblATRFpUJZjxmIiaARszre8xVtX6Daqmg1kAHXCEl1oBHLO+V2N+0YRzUo8Z++SuYmqTghnYCEUyL9zW6CtiMwSkbkiMjBs0YVGIOd8H3CpiGzBzX9yY3hC801p/7+XKKQT05jIIyKXAmlAb79jCSURSQCeAK70OZRwS8I1D/XBXfXNEJHjVHWvn0GF2CXA66r6uIicgpv1sJOq5vodWLSIxSuCrUCTfMuNvXWFbiMiSbjLyV1hiS40AjlnRKQ/cA8wWFUzwxRbqJR0ztWATsA0EdmAa0sdH+U3jAP5d94CjFfVLFVdD6zCJYZoFcg5Xw2MBVDVOUBFXHG2WBXQ//fSiMVE8C3QRkRaiEh53M3g8QW2GQ9c4T2/APhKvbswUarEcxaRbsALuCQQ7e3GUMI5q2qGqiaranNVbY67LzJYVef7E25QBPK3/RHuagARScY1Fa0LY4zBFsg5bwL6AYhIB1wi2BHWKMNrPHC513voZCBDVX8oyw5jrmlIVbNF5AZgIq7HwauqulREHgDmq+p44BXc5eMa3E2Z4f5FXHYBnvO/gKrAe9598U2qOti3oMsowHOOKQGe80TgDBFZBuQAt6tq1F7tBnjOtwIvicjNuBvHV0bzFzsRGYVL5snefY+/A+UAVPV53H2QQcAa4BBwVZmPGcW/L2OMMUEQi01DxhhjSsESgTHGxDlLBMYYE+csERhjTJyzRGCMMXHOEoHxhVcN9O18y0kiskNEPvUzrtISkQ1ef31EZHYJ214pIg1Luf/mhVWhFJF1ItKuwLqnROSOQGI1Jj9LBMYvB4FOIlLJWx5AGUdHBos32rzUVLVHCZtcCZQqERRjNPnGv3glNS7w1htTKpYIjJ8+A872nl8CjMp7QUSqeHXZ53l15od465uLyEwR+c579PDW9xGRaSLyvoisEJF3Cqso623ztIgsEpElItLdW3+fiLwlIrNwgw1TRGSciHzrPU71tqsjIpPE1fp/GVcKOG/fB/I9v0NEFovI9yLyqIhcgKvx9I537EoicoKITBeRBSIyMa+CpLf+exH5Hri+iN/dKODifMu9gI2qulFEPvL2uVRErinkd/CrqwwRuU1E7vOetxKRL7z3zxSR9kUc38QSv2tv2yM+H8ABoDOuDHhFYBFuNOWn3usPA5d6z2viauZUASoDFb31bXCjS/Hem4Gru5IAzAFOK+S404CXvOe98Gq+4ypYLgAqecvv5r0faAos957/B/ib9/xs3EjW5Lxz8n6eBczml/kfauc7dpr3vJy3TYq3fDFu1CxAOtDLe/4viq5LvwTo4j1/HrihwPEqedvU8ZY34GrwNM+/T+A24D7v+RSgjff8JFz5Fd//XuwR2kfMlZgw0UNV08XNDncJ7uogvzOAwSJym7dcEfeBvA14RkS64kootM33nnmqugVARBbhPvC+LuTQo7zjzxCR6iJS01s/XlUPe8/7A6n5Liqqi0hVXPIY5r1/gojsKWT//YHX1Jv/QVULqy3fDlcUb7J3jETgBy+Wmupq0gO8hUsshRkFDBeRpcBQXCkCgJtE5DzveRNcwiyxzIR3fj34pQwJQIWS3meinyUC47fxwL9x3+jzzwkhwPmqujL/xl4Txk9AF9w3//wTzuSvqJpD0X/fBeuq5C0fzLcuAThZVX81oU0hrU3HSoClqnpKgf3XLMU+RgOTgOlAuqr+JCJ9cInoFFU9JCLTcEk0v2x+3Syc93oCsFdVu5YiBhMD7B6B8durwP2qurjA+onAjXnt/OKqp4IrGf6Dulrzl+G+SZfWxd4+T8NVbswoZJtJ5JvgxLsCATft5Qhv3VlArULeOxm4SkQqe9vV9tbvx5XHBlgJpIirn4+IlBORjurmDdjrxQYwsqiTUNW1wE7gUX65v1ID2OMlgfa48tsF/QTU9e53VADO8fa3D1gvIhd6MYmIdCnq+CZ2WCIwvlLVLar6n0JeehDXjp7uNX086K1/DrjCu5Hanl9/iw/UERFZiGtXv7qIbW4C0sRNDr4MuNZbfz/Qy4tpGK4EcsFz+gJ3pTPfa6LKa956HXjeW5eI6+XzT+9cFuGaZcBVk3zW266kS5BRuN/DB97yF0CSiCzHJYi5hcSXBTwAzMMlrRX5Xh4JXO3FtJTCp/80Mcaqj5q44jWV3KbRPS+BMUFlVwTGGBPn7IrAGGPinF0RGGNMnLNEYIwxcc4SgTHGxDlLBMYYE+csERhjTJz7f7cXhF74ExwTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reliability diagram\n",
    "fop, mpv = calibration_curve(testy, probs_test, n_bins=10, normalize=True)\n",
    "# plot perfectly calibrated\n",
    "pyplot.plot([0, 1], [0, 1], linestyle=\"--\")\n",
    "# plot model reliability\n",
    "pyplot.plot(mpv, fop, marker=\".\")\n",
    "pyplot.xlabel(\"Mean predicted Value\")\n",
    "pyplot.ylabel(\"Relative Frequency\")\n",
    "pyplot.title(\"SVM Reliability Diagram\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74a9dd24",
   "metadata": {},
   "source": [
    "In the above graph, the solid line represents the calibration of the SVMs predicted probabilities and the dotted line represents a comparison to a perfectly calibrated model. The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot. Here, we see that calibration probabilities are much further away from the ideal prediction.\n",
    "\n",
    "The position of the points or the curve relative to the diagonal can help to interpret the probabilities; for example:\n",
    "\n",
    "* *Below the diagonal*: The model has over-forecast; the probabilities are too large.\n",
    "* *Above the diagonal*: The model has under-forecast; the probabilities are too small.\n",
    "\n",
    "Since this graph indicates that the probabilities calculated by SVM are too small, we shall move forward and look in the Calibrated Classification Model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46e1744e",
   "metadata": {},
   "source": [
    "#### Calibrated Classification Model\n",
    "\n",
    " - CalibratedClassifierCV class using 5-fold cross-validation, to calibrate the predicted probabilities.\n",
    " - Since the over sampled data resulted in a better accuracy, we will consider this dataset for training the Calibrated Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9ba588fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create X = test_duration and y = pass_or_fail and calculate probabilty for over sampled data.\n",
    "X, y = np.array([df_os[\"test_duration\"].values]), np.array(\n",
    "    [df_os[\"pass_or_fail\"].values]\n",
    ")\n",
    "X = X.reshape(-1, 1)\n",
    "y = y.reshape(-1, 1)\n",
    "\n",
    "# split into train/test sets.\n",
    "trainx, testx, trainy, testy = train_test_split(\n",
    "    X, y, train_size=0.8, test_size=0.2, random_state=42\n",
    ")\n",
    "\n",
    "# fit a Calibrated Classification Model.\n",
    "model = SVC(probability=False)\n",
    "calibrated = CalibratedClassifierCV(model, method=\"isotonic\", cv=5)\n",
    "calibrated.fit(trainx, trainy)\n",
    "\n",
    "# Evaluate model\n",
    "y_pred = calibrated.predict(testx)\n",
    "\n",
    "# predict probabilities.\n",
    "probs_train = calibrated.predict_proba(trainx)[:, 0]\n",
    "probs_test = calibrated.predict_proba(testx)[:, 0]\n",
    "probs = np.append(probs_train, probs_test, axis=None)\n",
    "df_os[\"prob\"] = probs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef3269a",
   "metadata": {},
   "source": [
    "*NOTE* : The probabilities are not normalized, but can be normalized when calling the `calibration_curve()` function by setting the `normalize` argument to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0b9a663f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CCM Model accuracy : 0.9146124523506989\n",
      "CCM Brier Score 0.08538754764930115\n",
      "CCM Roc Score 0.9143269806101052\n"
     ]
    }
   ],
   "source": [
    "# Mean accuracy on the given test data and labels.\n",
    "cc_accuracy = calibrated.score(testx, testy)\n",
    "print(\"CCM Model accuracy :\", cc_accuracy)\n",
    "cc_brier = brier_score_loss(testy, y_pred, pos_label=1)\n",
    "print(\"CCM Brier Score\", cc_brier)\n",
    "cc_roc = roc_auc_score(testy, y_pred)\n",
    "print(\"CCM Roc Score\", cc_roc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd89927e",
   "metadata": {},
   "source": [
    "Brier Score is a distance in the probability domain. Which means: the lower the value of this score, the better the prediction. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47a4de4a",
   "metadata": {},
   "source": [
    "The above calculated score is a harsh metric since we require for each sample that each label set be correctly predicted. This does not give us a clear indication of a better model so we will use the reliability diagram to find out the better suited model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51138c75",
   "metadata": {},
   "source": [
    "##### Reliability Diagram\n",
    "A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency  (x-axis). These plots are commonly referred to as ‘reliability‘ diagrams in forecast literature, although may also be called ‘calibration‘ plots or curves as they summarize how well the forecast probabilities are calibrated.\n",
    "\n",
    "The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "45f042b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABKf0lEQVR4nO3dd3hUZfbA8e9J6L0l1EDoCdhFRBRERQQLYAcb7ura1rKyuuq69v25rt2194IiNpSoKFgAASmChRZ6B+kQakLK+f3x3sAQUybJzNxM5nyeZ55MuXPvuZOZ+9633POKqmKMMSZ2xfkdgDHGGH9ZQWCMMTHOCgJjjIlxVhAYY0yMs4LAGGNinBUExhgT46wg8JmIqIh08O6/JCL3ePf7iMhaf6M7lIhMFJGry/jeZG9fq4Q6Lm/9/xSR1wIenysia0Rkt4gcLSLzRaRPGLb7lYgMC/V6y0NE7heRd737rb3PID6I9xX7nSvu+xmuz7eYWA75f5vyCcuPMtaIyCXAcCAF2AX8Cvyfqk4pzXpU9brQR+eISDKwAqiqqjlh2kYn4P+AU4CqwCrgLeCZcGwvkKo+XOCpx4EbVXWM97hrebchIvcDHVT1soDtDijveovY1lvAJcB+7zYbuElVF5ZmPaq6GqgTipiK+36q6oHPt7DPqTREZCLQA8gGFFgCfAQ8papZ3vYK/r9NOViNoJxEZDjwNPAw0BRoDbwADIpwHCWe8YV5++2BGcAa4HBVrQ9cCHQD6voQUhtgvg/bDaVHVbUO0BJYB7zuczyRdKOq1gWaA38HhgBjRUTCudFw1VgrPFW1WxlvQH1gN3BhMct0B6YBO4DfgeeAagGvK+7sCdzZ87+9+32AtcA/gS3ASuDSgPe9BbwIjAX2AH2Bs4BfgJ24A/L9Acuv9ra127ud4D3/ZyAd2A6MA9oEvOd0YCGQ4cU9Cbi6iP18F/iymM8h2dt+Fe/xn7zt7gKWA9cGLNsE+ML7zLYBk4E477U7cAfFXcAi4DTv+fu9GKp7+6fe57LMe30l0Ne7H+99rsu89cwGkrzXnvE+u53e87285/vjzsyzvfX/5j0/Mf8zwZ1Y/QtXE9oEvAPUL7D/w7z/xRbg7mI+rwPfBe/xmcCegMctgE+Azbia3s0Br90PvFuGz70PJX/nDvl+Bry2Evcd/MPnhDshmF1g/4YDY4rY9wOfacBzrYG9wNkF99F7/BGwAfdd/QHoGvBaY+Bz73/6E/BvYEqB3+BfcTWPFcV9DwK2/RHu+7YLmAt0Au7y/u9rgH5+H59Kc7MaQfmcANQAPi1mmVzgVtzB7QTgNOCGINffzHtfS9wB5BUR6Rzw+iW4ppi6wBTcge8KoAGuULheRAZ7y/b2/jZQ1TqqOk1EBuF+9OcBCbgD7vsAItIEGI07sDXBHTRPLCbWvsDHQe4XuB/M2UA93MHpKRE5xnvt77gDUgKulvVPQL19vxE4Tt3Z4hm4A9ABqpql7iwa4EhVbV/ItocDQ3EH13q4wnCv99pPwFFAI2Ak8JGI1FDVr3G1vg+8z+/IQtZ7pXc7BWiHa5J5rsAyJwGdcd+De0UktfCP5yARqe3Fu9R7HIc7sP2G+26cBvxNRM4oaV0U/7lDyd+5YhXxOaUBbQvs6+W4gjLY9a4GZgG9iljkK6AjkAj8DLwX8NrzuN9GM9w+FdanMxg4HujiPS70exCw/DnACKAh7uRrHO5EoCXwIPBysPtWEVhBUD6NgS1aTJu7qs5W1emqmqOqK3FfkJNLsY17vIPbJOBL4KKA18ao6lRVzVPVTFWdqKpzvcdzcAf14rZ1HfAfVU339uFh4CgRaYM7SM5X1Y9VNRvX/LWhmHU1xtV4gqKqX6rqMnUmAeM5+CPPxjUJtFHVbFWdrO5ULBd3xt9FRKqq6kpVXRbsNgNcDfxLVRd52/9NVbd6cb2rqlu9/9cT3vaCPRBeCjypqstVdTfuDHFIgeaGB1R1n6r+hjuQF1ag5LtNRHbgzjpPwh08AY4DElT1QVXdr6rLgVdxzSfFKuFzz1fcd67U1LXrfwBcBiAiXXE1lS9Kuar1uANzYdt4Q1V3edu6HzhSROp7TabnA/ep6l5VXQC8Xcgq/qOq21R1n7e+kr4Hk1V1nPe7+Qh30vKI91sZBSSLSINS7p9vrCAon61Ak+LaFUWkk4h8ISIbRGQn7mDbJMj1b1fVPQGPV+GaBPKtKbCt40VkgohsFpEM3IG+uG21AZ4RkR3eAWcbILizmhaB6/cOxGsKW4lnK+7gHRQRGSAi00Vkm7ftMwNifQx39jteRJaLyJ1eDEuBv+F+6JtEZJSItPjDykuWhKvhFBbXbSKSLiIZXlz1Cf7/1QL3P8q3Cjcgo2nAc4GF6V6K78h9XFUb4A6a+zh4IGoDtMj/v3lx/rPAdgpVwucOJX/nyupt4BKvjf9y4EPvoF0aLXHf0UOISLyIPCIiy7zf2ErvpSa4A3QVDv3uFvY9LvhbKul7sDHg/j7cCWFuwGMIUSd9JFhBUD7TgCxctbIoL+La2Tuqaj3cDzbYDq+GXrNAvta4s6J8BVPHjsRVw5PUdda+FLCtwtLMrsG1ETcIuNVU1R9xZ/dJ+Qt6P+CkQtaR71vcmVeJRKQ6rn37caCpd7Abmx+rd2b3d1VtBwwEhovIad5rI1X1JNzBUIH/BrPNAtYAf2gyEpFewD9wZ8ANvbgyKP4zDLTeiytfayCHQw8apeY1i9yCK7RrevGvKPB/q6uqZxa3npI+d09J37mgQi5kH6bj+g564Zo0R5RmhSKSBByLa74s6BLc4Iy+uAN2cv7bcH0oOUCrgOUL+x4fiDmI70GlYwVBOahqBnAv8LyIDBaRWiJS1TvretRbrC6uw2m3iKQA15dyMw+ISDXvy3k2rhpalLrANlXNFJHuuB9Ivs1AHq7tOt9LwF1eVR2vKn2h99qXQFcROc+r8dyMa2Mtyn1ATxF5TESaeevrICLvFlJFroaram8GckRkANAv/0UROdt7r+B+gLlAnoh0FpFTvQNaJu7MK6+YmIryGvCQiHQU5wgRaYz7/HK8uKqIyL24tvR8G3FV/qJ+N+8Dt4pIWxGpw8G28nIP11XVb3AH5GuAmcAuEblDRGp6Z8SHichxJaym2M89QGm+c4Up6nN6B9dnkq1BDq32flMnA2Nw+z22kMXq4k7ItgK1cJ87AN5Z+mjgfm9dKbh+tOKU9D2odKwgKCev/XA4rlN1M+5s7UbgM2+R23AH5F24dtwPSrH6DbjRPOtxnV/XafHjyG8AHhSRXbgC6sOAOPfiOpanes0JPVT1U9wZ9SivSj0PGOAtvwU32uMR3A+sIzC1qA17bfUn4M7G5ntNU5/gOvh2FVh2F65g+dDbv0twNZl8HXE1jN24WtcLqjoBdxB7BDeiZQOuY/CuYj6PojzpbXs8rpB+HaiJ6/D7GliMaxLJ5NAmg/wD4lYR+bmQ9b6BO9P9ATeSJxO4qQzxFeUx3JlqFdwB+ihvO1twhVv94t4cxOcOpf/OFaaoz2kEcBhutE1JnvO+xxtx/VOfAP1VtbCC/x3c/2sdsACYXuD1G3GfzQYvhvdxBUdRSvoeVDrimn6NMSa8vGatTcAxqrrExzj+CzRT1Qp1RbifrEZgjImU64GfIl0IiEiK1/wnXpPpVRQ/5DvmxOZVdMaYiBKRlbjO1sE+bL4urjmoBa6p6Qlcn4PxWNOQMcbEOGsaMsaYGBd1TUNNmjTR5ORkv8MwxpioMnv27C2qmlDYa1FXECQnJzNr1iy/wzDGmKgiIquKes2ahowxJsZZQWCMMTHOCgJjjIlxVhAYY0yMs4LAGGNiXNgKAhF5Q0Q2ici8Il4XEfmfiCwVkTly6CxJxhhjIiScNYK3cPOXFmUALstkR1xq3RfDGIsx0W/NTJj8hPtrTAiF7ToCVf1BRJKLWWQQ8I4389V0EWkgIs1VNejpDo2JGatnwFtngeZCfHUYlgZJ3f2OykTI3v05bN29n6RGtcKyfj/7CFpyaI7vtd5zfyAi14jILBGZtXnz5ogEZ0yFMmcU5GWD5kHuflhZ2ERdpjL6cekW+j89mevenU1eXnhyw0VFZ7GqvqKq3VS1W0JCoVdIG1O5Ze307gjEV4PkgvPNm8omY182d34yh0tem0GcwD1ndyEuLjyzZfqZYmIdh84d2sp7LvLWzHRnWMm9rLptKh5VWPOTu1+3GVz0jn1PK7ncPOX8F39k+ebdXHtyO27t24kaVePDtj0/C4I04EYRGQUcD2T40j+wZia8fQ7kZEGVGtb2aiqeDXNgxyqonQCZGdCym98RmTDZvmc/DWpVJT5OuK1fZ1o0qMERrRqEfbvhHD76Pm6+2c4islZErhKR60TkOm+RscByYCluLt8bwhVLsVZOdoUACjn7YMl4X8IwpkgL0kDiocf1kL0XMlb7HZEJMVXl01/WcsoTExn1k+s67X9Ys4gUAhDeUUNDS3hdgb+Ga/tBS+7lagI5mYDCrDch+SRo18fvyIxx0tMg+URoc6J7vGkhNEz2NSQTOut37OPuT+cyYdFmjm7dgG5tGkY8hqhLQx1ySd1dc9DKyVCnKUx9Bt4ZBD3+CqfdC1Vr+B2hiWWbFsKWxdD9GkhIcc9tTofOxV2iY6LFmF/Xcfen88jNU+49uwvDeiYTH6YO4eJYQQCuMMjvF+h6HnxzL0x/HpZPgPNegWaH+xufiV3paYBA6jlQswHUbQ6bF/kdlQmR+jWrclRSA/5z3uFhu0YgGFExfDSiqtWCsx6HSz+BvVvh1VNh6v8gL8/vyEwsWpAGSce70ULgagWb0v2NyZRZTm4eL09axnPfLwGgT+dERlzV3ddCAKwgKFrHvnD9NOh0BnxzD7wzEHasKfl9xoTKtuWwcS50GXjwucRU11RkJyZRZ8H6nZz7wo/856uFpG/YhesmBZHINwUVZAVBcWo3hotGwKDnYf0v8OKJMOcjv6MysWJBmvubes7B5xJS3MihHUXOOmgqmKycXJ4Yv4iBz03h94x9vHDpMTw39OgKUQDks4KgJCJw9GVw3RR3Njb6avj4z7Bvu9+RmcouPQ1aHA0NWh98LjHV/d280J+YTKmt3LKXlyYtY+BRLfjm1pM58/DmFaoQACsIgteoLfxpLJx6DywY42oHyyf5HZWprDLWwrrZkDrw0OcTOru/1k9Qoe3JyuGzX1yihM7N6vLd8D48edFRNKxdzefICmcFQWnExUPv2+Cqb6BqLddv8PU/ITvT78hMZZP+uftbsCCoUR/qtrAaQQU2eclmznj6B2798FeWbtoFQOvG/nYGl8QKgrJoeQxc+wMc9xc3zPTVU2BDofPvGFM2C9IgsQs06fDH1xJt5FBFlLE3m398/BuXvz6TavFxfHDNCXRIrOt3WEGxgqCsDgwz/dgbZnqKDTM1obFrI6ye9sfaQL6E/JFDuZGNyxQpN085/6Uf+eTnddzQpz1jb+lF97aN/A4raHZBWXl1PN0NM/38ZjfMdMl4GPwiNEgq+b3GFGbhF4AeOmw0UGKKS4myYxU0ahfR0Myhtu3ZT4OaLknc7Wd0pmWDmhzWsr7fYZWa1QhCoXZjuPhdGPicDTM15ZeeBo3au6ahwiR4I4c2WT+BX1SVT2av5ZTHDyaJO6Nrs6gsBMAKgtARgWMu94aZptgwU1M2e7fBismuNlDUEMP8kUObrZ/AD2u372XYmz/x949+o0NinahqAiqKNQ2FWqO2cOVYmPIUTHoEVk93TUXtTvY7MhMNFo118xIX1T8AUKMe1GtlNQIffPrLWv716TwUeGBgVy7v0SZss4ZFktUIwiG+Cpx8+6HDTMfdbcNMTckWpEH91u5CsuIkpliNwAeNalfn2ORGjL+1N8N6JleKQgCsIAivA8NMr4Zpz7kEdjbM1BQlc6fLeJt6TtHNQvkSUmDLEhs5FGbZuXm8MHEp//vOJYk7uVMCb//pOFo1rNjXBZSWFQThVq0WnPUEXPIR7Nlsw0xN0RaPg9z9RY8WCpTgjRzavjLsYcWqeesyGPz8VB79ehFLNu2uUEniQs0Kgkjp1A9umAYd+x3MZpqx1u+oTEWSPgbqNINWQcyZnZ9zyC4sC7nM7Fwe/Xohg56fysadWbx02TE8W8GSxIWaFQSRVLuJN8z0WVj3M7zQ04aZGmf/HljyLaSeDXFB/Cxt5FDYrNq6l1cnL+e8o1vy3fCT6X9Yc79DCjsrCCJNBI65Aq6f4n7MNszUACz9FnL2FT9aKFD1ulA/yWYrC5E9WTmM/tnV0Ds3q8v3f+/DYxceSf1aVX2OLDJs+KhfGrWDP33lhplO/I8bZnriLbB/NyT3Ojh1pokNC9KgZqODE9QHIyHFhpCGwKTFm/nn6Lmsz9jHEa3q0yGxru8zhkWaFQR+yh9m2uFU+OBy+Oof7vkq1WHYF1YYxIqcLNdR3HWw+04EKzEFVvzgRg7FxYctvMpq+579PPTlAkb/vI72CbX56NroSRIXatY0VBG0PNY1F+F1RuVkwafXwpJvbHRRLFg2Afbvgi6DSve+hFTIzYJtK8ITVyWWnyRuzK/rufGUDnx5cy+6JUf/FcJlZTWCiqL9qTDlaTd8UAT27YD3LoAmneD46+DIoW4oqql80tOgen1oW8qrzxNT3N/N6YWnqzZ/sHV3Fg1rVSM+TrizfwotG9aka4vozA8USlYjqCiSusOwNDj1btd38PdFcN6r7srkL4fDk6nw7f2Qsc7vSE0o5WbDwi+hc3+oUsrZq5rkz1Zm/QQlUVU+nLWGUx6fyPs/rQagX9dmVgh4rEZQkSR1P7Rf4IiL4PALXUfy9Odh6jPw47PQZTD0uAFaHetbqCZEVk6GzB3BjxYKVL2OS0dhQ0iLtWbbXv756VwmL9lC9+RGnNCusd8hVThWEFR0ItDmBHfbvhJmvAI/vwPzPoak46HH9ZByTuk6GU3FsSANqtaGDqeV7f2JNnKoOKN/Xsu/PpuHAA8NPoxLu7euNPmBQsmahqJJw2To/zAMXwD9/wu7N8JHV8L/jnJpK/bt8Dc+Uzp5uW4Smo6nQ9WaZVtHQgpsXQK5OaGNrZJoUqc63ds2YvzwkytNptBwsIIgGtWoBz2ug5t+hiEjoUEbl7biyS4w9nbYuszvCE0wVk93+aeCyS1UlMRUN8Bgu40cApck7rnvl/DMty5JXO9OCbz1p+60bFDGgjZGWHtCNIuLh5Sz3O3332D6izDrTZj5KnTq75qN2vYuOZOl8Ud6GsRXd/mnyirBGzm0KR2adAxNXFFq3roMbv94Dum/72TQUS1Q1UqdHyiUrEZQWTQ/Es59CW6dD71vh7UzXWK7l06CX961uRAqGlVI/9z1DVQvx0VMB3IOxW4/QWZ2Lo985ZLEbdmdxcuXH8szQyp3krhQC2tBICL9RWSRiCwVkTsLeb21iEwQkV9EZI6InBnOeGJC3aZuCOqtC1xyO82DMX+Fpw+DCf+B3Zv8jtCASzq4c13ZRgsFqlbbNQ3GcBbS1dv28vqU5VxwTCu+vfVkzujazO+Qok7YCgIRiQeeBwYAXYChIlJwNu5/AR+q6tHAEOCFcMUTc6rW8JLb/QhXjIEWx7ipM5/qCp/dABvm+h1hbEsfA3FV3PUD5ZWYGnM1gl2Z2Xw0y00a36lpXSbc1of/XnBEzCSJC7Vw9hF0B5aq6nIAERkFDAIWBCyjQD3vfn1gfRjjiU0i0K6Pu21ZCjNehF9Hwq/vueR2PW5w/QnBpD42oaHqho22PRlqNiz/+hJSYOl37uK0+Mp/IJywcBN3fzqXDTszObp1Azok1q10M4ZFWjh//S2BNQGP13rPBbofuExE1gJjgZsKW5GIXCMis0Rk1ubNm8MRa2xo0sHNljZ8AfR9ALYth1FD4bljYcbLkLXb7whjw8Z5bpRPeUYLBUpIgbxs9/+sxLbt2c+tH/zKn976idrVq/Dx9T1jNklcqPl9GjgUeEtVWwFnAiNE5A8xqeorqtpNVbslJCREPMhKp2ZDOOlvcMtvcMEbUKuxy3z6ZBcYdzfsWO13hJXbgjSQOEg5OzTrSwwYOVRJ5eYpF7z4I5//tp6bT+vIFzefxDGtQ1CbMkB4m4bWAUkBj1t5zwW6CugPoKrTRKQG0ASwHs1IiK8Kh53vbmt+gukvuCGo019wE6j3uMFdvWyjL0IrPc3NO1C7SWjW16QzIJWyn2Dzriwa13ZJ4v55ZiotG9YktXm9kt9oSiWcNYKfgI4i0lZEquE6g9MKLLMaOA1ARFKBGoC1/fgh6Ti48E342xzoeRMsnwhvnAGvnuqm08zN9jvCymHzYnfALu9ooUDVakHDNpWqIFBVPvhpNac+MZGRM10NtW+XplYIhEnYCgJVzQFuBMYB6bjRQfNF5EERyf8V/B34i4j8BrwPXKmqGq6YTBDqt4LTH4Th6XDm45C1002n+fThMPkJ2LvN7wijW/oY9zc1RM1C+RJSK03OodVb93LpazO445O5dGlej5M6hKjmZIok0Xbc7datm86aNcvvMGJHXh4s/cY1Fy2fCFVqwpFD3FXLmRkue6ZNrRm8l3pBlRpw9TehXe+398OPz8Hdv0f1yKGPZ6/lns/mER8n3HVmCkOPsyRxoSIis1W1W2GvWYoJU7y4OOh0hrttnO/6EH4dCbPfdB2e4NIkDEuzwqAk21bAhjnQ79+hX3dCqhs5tHXZwc7jKNS0XnV6tm/Mv889jOb1LT9QpPg9ashEk6ZdYdBzbvhpuz7uqmXNc9Mlrpzsd3QVX/rn7m/qOaFfd+BsZVFkf04ez3y7hKe+WQxAr44JvH7lcVYIRJgVBKb0ajeBU+52TRzgCoNQXBhV2aWnuZxQDZNDv+7GHQGJqn6C39bs4Jxnp/DUt4tZs20v0dZMXZlY05Apm6TuMOxzWPQVzPkAxt8DTQ+z5qGiZKyDtT/Bqf8Kz/qr1XIFTBTUCPbtz+XJbxbx+pQVJNatwWtXdKNvl6Z+hxXTrEZgyi6pO/S9D67+Fuokwrvnw9rZfkdVMS38wv1NHRS+bSRGx8ihNdv38vaPqxjSvTXjh/e2QqACsILAlF+9FjDsC6jVCEacC+t/8TuiimdBmksFkdApfNtISIFtyyBnf/i2UUY7M7P5MCBJ3MTb+/DwuYdTr0b0jnCqTKwgMKFRv6VrKqpRH94Z7CbKMc7uzbD6x9BeRFaYxFTIy3GFQQXy/cKN9HvyB+78ZA5LN7l8Vi1sxrAKpcSCQESeEJGukQjGRLkGreHKz6FaHXhnEGyY53dEFcPCL1yHeqiSzBUloWLlHNq6O4tbRv3Cn9+aRf2aVRl9w4l0SKzjd1imEMHUCNKBV0RkhohcJyL1wx2UiWINk11hUKWmmyFt44IS31LppadBw7auMz2cmnRy13ZUgFQTuXnKhS9NY+zc37m1byc+v+kkjkpq4HdYpgglFgSq+pqqnghcASQDc0RkpIicEu7gTJRq1A6u/ALiqrrCYPMivyPyz77tsOIHVxsId/K+qjVcgeNjjWDTrkzy8pT4OOHus1L54qZe3NK3I9WqWCt0RRbUf8ebbSzFu20BfgOGe5PNGPNHjdu7wgCBt8+BLUv8jsgfi75y7fbhHC0UyKfZyvLylPdmrOLUxyfxnpck7rTUpnRuZvMFRINg+gieAhbi5gt4WFWPVdX/quo5wNHhDtBEsSYdXQey5sFbZ7v0B7FmQRrUawUtj4nM9hI6u885Jysy2wNWbtnDJa9N5+5P53FEq/qc3NHmDIk2wdQI5gBHqeq1qjqzwGt29ZApXmIKXJHm8uC8fY7LtxMrsnbBsu9dSolIzemQkAqaC1uXRmRzH85awxlP/8D8dTt55LzDee/q42nd2KaNjDbBFAQ7CLgCWUQaiMhgAFXNCE9YplJp2gWuGAPZe11hsH2V3xFFxuJxLg9TuEcLBYrwbGUtG9Skd6cEvhl+MkO6t0ZsEqOoFExBcF/gAV9VdwD3hS0iUzk1O9wVBlk74e2zYceakt8T7dLToHaim+UtUhp3DOvIoaycXJ76ZjFPjncDAE7s0IRXr+hGs/o1wrI9ExnBFASFLWM5ikzpNT8SLv8M9mW4wiCj4Myllcj+vbDkGzcBTVx85LZbtYYbtRWGguCX1ds559kpPPPdEtbtyLQkcZVIMAXBLBF5UkTae7cnAUsoY8qm5TFw+aduprO3z4adv/sdUXgs+841hYX7auLCJKSENOfQ3v05PPTFAs578Ud2ZebwxpXdeOKiI60ZqBIJpiC4CdgPfODdsoC/hjMoU8m1OhYu+wR2b3KFwa4NfkcUegvSXGru5JMiv+3EVNi2PGQjh9Zt38eI6au49PjWjL+1N6emWJK4yiaYC8r2qOqdqtrNu92lqnsiEZypxJK6w6UfuxrB2wNdoVBZ5GTB4q+h81n+TBuZkOJGDpXj2o2MfdmM8q4H6Ni0LpNu78O/Bx9OXUsSVykFcx1BJxF5RUTGi8j3+bdIBGcquTYnwKUfQcYal5tozxa/IwqN5ZNcp3gkRwsFSkx1f8vYTzB+/gZOf3ISd38270CSOJsxrHILptP3I+Al4DUgN7zhmJiTfCIMHQUjL3KFwbDPXTrraJY+BqrXc9N5+qFxB5D4Ug8h3bI7i/vT5vPFnN9JaVaX14Z1syRxMSKYgiBHVV8MeyQmdrU7GYa+DyOHuNxEV6RFb2GQmwMLx0KnM6BKdX9iqFK91COHcvOUC178kfU7MrmtXyeuPbk9VeMtP1CsCOY//bmI3CAizUWkUf4t7JGZ2NL+VBgy0iWoG3Eu7Nvhd0Rls2oK7Nvmz2ihQIkpQdUINu48mCTuvnO68uXNJ3HjqR2tEIgxwfy3hwG3Az/iho3OBmaFMygTozr2hYvfhY3z4d3zIDMKL1xfkAZVa0GHvv7GkZAK21dAdmahL+flKSOmr+K0Jybx3gx3pfcpKYl0bGpJ4mJRMKOG2hZyaxeJ4EwM6nQGXPSOm+Hs3Qtcvp5okZfnJqHp0NdNJu+nxBSX7G/rH0cOLd+8myGvTueez+ZxVFID+nRO9CFAU5EEM2qoloj8S0Re8R53FJGzwx+aiVkpZ8KFb8G62fDehZC12++IgrNmBuzeCF0ilHK6OAneyKECF5Z98NNqBjwzmYW/7+TRC45gxFXdSWpkSeJiXTBNQ2/iLijr6T1eB/w7bBEZAy5j5wWvw5qZbkTR/ii4dCX9c4ivBh37+R2JGzkUVwU2H9pP0KphLfp0TuDb4SdzUbckuzrYAMGNGmqvqheLyFAAVd0r9u0xkdD1XMjLhdF/gZEXwyUf+t/kUhRVVxC0PxVq1PM7GqhSDRq1J3djOk+NcwnibjujMyd2aMKJHZr4HJypaIKpEewXkZqAAohIe1yaCWPC7/ALYPBLsHIKjBoK2fv8jqhw63+BjNX+jxYKsL1OO35f8gvPTVjKpl2WJM4ULZgawX3A10CSiLwHnAhcGc6gjDnEkRe7lAmf3eBmOuvYD9qf4tJUVBTpaa4ppvMAvyNhT1YOj41bRMOlNbmpygZGXHEEvbok+R2WqcCCGTX0DXAe7uD/PtBNVScGs3IR6S8ii0RkqYjcWcQyF4nIAhGZLyIjgw/dxJSjLoGTboV1s2Diw26CmzUFJ8zziaobNprcq0JcCLd+xz5GzlxN8w5HEYfSq+E2v0MyFVwwo4Z6A12BXcBOoIv3XEnviweeBwYAXYChItKlwDIdgbuAE1W1K/C30u6AiSHV6wBe91ROppsBrCLYtAC2LfMvtxCQsTebkTMOJomb/I9TuOhMr9M6hCmpTeUUTNPQ7QH3a+DmKZ4NnFrC+7oDS1V1OYCIjAIGAQsClvkL8LyqbgdQ1UqUgtKEXHIvqFLDTf+oefDrSFdTaNze37gWpAECKf6Mqv563gbuGTOPbXv2c3y7RrRPqEPTejWgVvtCRw4ZU1CJBYGqnhP4WESSgKeDWHdLIHA+wrVAwTn7OnnrnArEA/er6tcFVyQi1wDXALRu3TqITZtKKak7DEuDlZOhZmP47gF4vZ8bTdTqWP/iSk+DNj2hTmQvzNq0K5P70+Yzdu4GujSvx5tXHkf7hIAkcVWquWGkmxdFNC4Tfcoy5eRaIDWE2+8I9AFaAT+IyOHevMgHqOorwCsA3bp1s6EPsSyp+8FO4uSTXCqKt86CC9/0p6N2y1LXNNT/vxHdbG6ectFL01ifkcntZ3Tmmt7tCs8PlJDirtI2phglFgQi8ize0FFcn8JRwM9BrHsdEDhUoZX3XKC1wAxVzQZWiMhiXMHwUxDrN7GuSQe4+lt3wdmoS+CsJ6DbnyMbQ/oY9zf1nOKXC5HfM/bRtG4NlyRuYFeSGtYqPlV0YiosGOPmUK6o12AY3wU1ZzEHk81NA+5Q1cuCeN9PQEcRaSsi1YAhQFqBZT7D1QYQkSa4pqLlQUVuDLjmmCu/hA6nwxe3wncPulE8kbIgDVp2g/otw7qZvDzlrakrOO2JSbybnySuc2LJ8wUkpAAKWxaHNT4T3YLpI3i7LCtW1RwRuREYh2v/f0NV54vIg8AsVU3zXusnIgtwk97crqpby7I9E8Oq1XYprL8cDpOfgIx1MPBZ10YeTttXwe+/wukPhnUzSzft5s5P5jBr1XZ6d0rg1JRS9EUEzlbW4qiwxGeiXzBNQ3M52DR0yEuAquoRRb1XVccCYws8d2/AfQWGezdjyi6+CpzzDNRPggn/ht0b4KIR4U33kP65+xvGZqFRM1dzb9p8alaN54kLj+S8Y1qWLj9Qo3YQV7XUs5WZ2BJMZ/FX3t8R3t9Lvb82a5mpWETg5NuhXgv4/GZ4c4CbE7lei/BsLz0Nmh7uDrZh0rpxLfqmJvLAwMNIqFuGGc/iq3ojh+xaAlO0YAqC01X16IDHd4rIz6pa6JXCxvju6EuhbjP48Ap47XS47OODTSShsvN3l3b6lLtDutrM7Fz+952bQ+Af/VPo2b4JPduXM0lcYgqsC2Z8h4lVwXQWi4icGPCgZ5DvM8Y/HU6DP30FeTnw+hkuaV0oLfzC/Q1hkrlZK7dx5v8m88LEZWzbsz90SeISUmHHquhI5W18EcwB/SrgBRFZKSIrgReACI/RM6YMmh8BV3/jagcjzoV5n4Ru3QvGQJNO7my7nHZn5XDfmHlc+PI09ufk8c6fu/PI+UeEbq6A/Bht5JApQjCjhmYDR4pIfe9xFE4ka2JWg9bw569h1KXw8Z9h53o44UbXn1BWe7bAqqlwUmjGOGzI2Meon9Yw7IRkbj+jM7Wrl+U6z2IEzlbW4ujilzUxKZikc01F5HVglKpmiEgXEbkqArEZExq1GsHln0KXwTD+X/D1nW7Cm7Ja+KXLdVSOJHPb9+xnxHR3PUCHRJck7v6BXUNfCIDrzI6vZjmHTJGCaRp6CzfeP3/oxWIsS6iJNlVrwAVvutrAjJfgo2Fln+QmPQ0atIFmRY6cLpKqMnbu75z+1CQeSJvPss1uPubEejXKFksw4qtA446WhdQUKZiCoImqfgjkgbtQDHfxlzHRJS4Ozvg/OOM/kP4FvDMY9pYyV/++HbB8kqsNlLJ5adPOTK57dzY3vPczzevXJO3Gkw5NEhdOiSlWIzBFCqYg2CMijTk4VWUPwPoJTPQ64Qa48C03veTr/WD7yuDfu/hryMuG1EGl2mRunnLhy9OYuGgzdw1I4dMbetKlRQTnNk5IgR2rIWt35LZpokYwDZLDcTmC2nvpohOAC8IalTHh1nWwy1P0/lB3rcGlHwbXkbogDeq2gJbBpb1ev2Mfzeq5JHEPDjqMpIY1aRepWkCghPyRQ4uCjt3EjmJrBN4sYyd7t57AtUBXVZ0TgdiMCa82PeGq8W6ymzfPgiXfFL981m5Y9p1LKRFXfGU6N095s0CSuJM7JfhTCMDBC+qsn8AUothvs6rmAkNVNUdV56vqPC9ltDGVQ0Jnd61B4/Yw8mL4+Z2il10y3k2RWcJooaWbdnHhSz/ywOcLOL5dI05LbRrioMugYVsbOWSKFEzT0FQReQ74ADhwaaKq2jXrpnKo2wz+NBY+HAZpN7nspX3u/GNncHoa1E6A1icUuaqRM1Zzf9p8aleP56mLj2TwUaVMEhcu8VXcBXA2W5kpRDAFwVHe38Bcu0rJcxYbEz2q14VLPoDP/waTHoGMtXDO0y5pG7ihpovHwxEXQlx8katJblKLfl2bcv/ArjSpU4YkceGUkAJrZvodhamAiiwIRORGVX1OVU8Rka6qOj+SgRkTcfFVYdBzUL+VKwx2b4AL34bqdWDZ95C95w+5hTKzc3nq28UIwp0DQpQkLlwSU2Dex66vo7pPfRWmQiqujyAwn9CIIpcypjIRgVPuchPbLJsAb50Juza60UI1GkDb3gcWnbF8KwOemczLk5azKzM7dEniwiU/1YQ1D5kCgr2evQI0choTQcdcAXWauSuQX+7lLiRr2xviq7IrM5v/fr2Qd6evpnWjWoy8+nh6dqigtYBAB2YrS4dWNoTUHFRcQdBARM7F1Rrqich5gS+q6uiwRmaM3zr1gwGPQtqN7vGKSbBmJhurd+Hj2Wu5+qS2DO/XiVrVwpAfKBwaJkN8dZutzPxBcd/gSUB+g+gPQOB8fApYQWAqvz2bQOJA88jLzSVu5WQ69OrO5H+cWrYZw/wUF++NHLJrCcyhiiwIVPVPkQzEmIpI25xEXlw1NHc/2RrPtgbH0hKirxDIl5gCq6b5HYWpYGymMWOKsHFnJn+ZEM+Fe+9kZK3L2Xjuh7Q8vI/fYZVPQgrsXAuZO/2OxFQgUdK4aUxk5eYpF708jQ0Zmdw2YBCXnJhMlfhKcN6U32G8ZTG06uZvLKbCsILAmABrt++lef2axMcJDw06jNaNapHcpLbfYYVOfvK5TelWEJgDgpmhrJaI3CMir3qPO4rI2eEPzZjIyc1TXpu8nL5PTuJdb+aw3p0SKlchAG7kUJUa1mFsDhFMjeBNYDaQn2BlHfAR8EW4gjImkhZt2MU/PpnDb2t2cFpKIv26VoAkceGSP3LIhpCaAMEUBO1V9WIRGQqgqnulQmTRMqb83p2+igc+n0/dGlV5ZshRDDyyRcVIEhdOCSmwaqrfUZgKJJiCYL+I1OTgDGXtgaywRmVMmKkqIkKHxDqceXhz7j27C40rWpK4cElMgbkfQmYG1KjvdzSmAgimILgf+BpIEpH3gBOBK8MYkzFhs29/Lk9+s4i4OOGuAan0aNeYHu0a+x1WZAXmHErq7m8spkIosSBQ1fEiMhvogcs5dIuqbgl7ZMaE2LRlW7lz9BxWbd3L5T3aHKgVxJzEgJFDVhAYgigIRORzYCSQpqp7SlremIpmZ2Y2/xm7kPdnrqZN41qM/MvxFTdVdCQ0SIYqNW3kkDkgmCtkHgd6AQtE5GMRuUBEaoQ5LmNCZtPOLD77ZR3X9G7H17f0ju1CANx8ywmWc8gcVGJBoKqTVPUGoB3wMnARsCmYlYtIfxFZJCJLReTOYpY7X0RUROwKFxMSW3dn8dbUFQB0SKzDlDtO4Z9nplKzWtGzi8WUhFSbyN4cENQ1896oofOB64DjgLeDeE888DwwAOgCDBWRLoUsVxe4BZgRfNjGFE5VGfPrOvo+OYn/G5vO8s27AWJnRFCwElNg13o3z4KJecFcWfwhkI6bo/g53HUFNwWx7u7AUlVdrqr7gVHAoEKWewj4L5AZdNTGFGL9jn1c9fYsbhn1K20a1+bLm3vRLsGmZCyUzVZmAgQzfPR1YKiq5pZy3S2BNQGP1wLHBy4gIscASar6pYjcXtSKROQa4BqA1q1blzIMEwtycvMY8sp0Nu/K4p6zu3Blz2Ti42JwRFCwEjq7v5vTofXxxS9rKr3iJq8/VVW/B2oDgwoOsyvvDGUiEgc8SRDXJKjqK8ArAN26davgE8OaSFqzbS8tGtSkSnwcD597OK0b1aJ141p+h1XxNWgDVWtZP4EBiq8RnAx8z6Ezk+ULZoaydUBSwONW3nP56gKHARO9QqYZkCYiA1V1VgnrNjEuJzePN6au4Inxi7lrQApXntiWkzrG+Gig0oiL82Yrs5xDpvgZyu7z7j6oqisCXxORtkGs+yego7fsOmAIcEnA+jOAA79cEZkI3GaFgClJ+u87ueOTOcxZm8HpXZoy4PDmfocUnRJTYdkEv6MwFUAwo4Y+KeS5j0t6k6rmADcC43CdzR+q6nwReVBEBhb/bmMKN2LaSs55dgrrtu/juUuO5pXLj6VpPbuspUwSUmD3Bti33e9IjM+K6yNIAboC9UXkvICX6gFB/fJUdSwwtsBz9xaxbJ9g1mliU346iE5N63LOkS245+wuNKpdze+woltiwMih1j38jcX4qrg+gs7A2UADDu0n2AX8JYwxGXPA3v05PD5uMVXihX+emcrx7RpzfKwliQuXwNnKrCCIacX1EYwBxojICao6LYIxGQPA1KVbuHP0HNZs28eVPZNjN0lcuNRPgqq1LdWECeo6gl9E5K+4ZqIDTUKq+uewRWViWsa+bB7+Mp0PZq2hbZPafHjtCXRv28jvsCqfuDh3PYHNVhbzguksHoEb2nkGMAk3DHRXOIMysW3L7iw+n7Oe605uz1e39LJCIJwSUqxGYIIqCDqo6j3AHlV9GziLAlcIG1Nem3dl8cYUN0q5fUIdptxxKncOSKFGVUsSF1aJKbB7I+zd5nckxkfBFATZ3t8dInIYUB9IDF9IJpaoKp/+spbTn5rEI18tZMUWN+WFjQiKkAM5h6xWEMuC6SN4RUQaAvcAaUAdoNAhoMaUxrod+7j707lMXLSZY1o34NELjqBtk9p+hxVbAmcra9PT31iMb4KZqvI17+4k3JwExpSbSxI3ja2793P/OV24/ARLEueL+klQrY7VCGJccReUDS/ujar6ZOjDMZXd6q17adnQJYl75LwjaN2oFkmNLEmcb0TcyCErCGJacX0EdUu4GRO0nNw8Xpy4jL5PTeKdaSsBOLFDEysEKgKbrSzmFXdB2QORDMRUXvPXZ3DHJ3OYt24nZ3RtylmWJK5iSUyBX991I4dq2VDdWBTMDGWdROQ7EZnnPT5CRP4V/tBMZfD2jysZ9NxUNmRk8eKlx/Dy5d1ItCRxFUv+yCG7sCxmBTN89FXgLrxhpKo6B5dS2pgiqbr5g1Ka1WXQUS35dnhvSxddUQXOVmZiUjDDR2up6swCOV5ywhSPiXJ7snJ4bNwiqsYLd5/VxZLERYP6raBa3fD1E6yZCSsnQ3IvSOoenm2YcgmmINgiIu1xs5IhIhcAv4c1KhOVfli8mbtGz2V9xj6GnWBJ4qJGOEcOrZkJb58NOdlQpToMS7PCoAIKpiD4K26+4BQRWQesAC4Na1QmqmTszeahLxfw8ey1tEtwSeKOS7ZOx6iSmAKLvg79eldOhpwsdz8nyz22gqDCCeaCsuVAXxGpjetT2IvrI1gV5thMlNiyJ4uv5v7ODX3ac/NpHS0/UDRKSIVf3oU9W6B2COd+btwh4EEexNtAgYqoyM5iEaknIneJyHMicjquABgGLAUuilSApmLatCuT1yYvBw4miftHf0sSF7XyU02Eunlom/uO0P1aaNQevn8IVs8I7TZMuRU3amgEbpayubgZySYAFwLnquqgCMRmKiBV5ePZazn9yR94dNyiA0niGlqSuOgWriGk80ZDy2PhzEfhz+OgXgsYeSFsnB/a7ZhyKa4gaKeqV6rqy8BQoAtwhqr+GpHITIWzZtternhjJrd99BsdE+sw9uZeliSusqjXAqrXC22NYOsy2DAHunpTntdJgCs+c7OijTgXtq0I3bZMuRRXEOSnn0ZVc4G1qpoZ/pBMRZSTm8fQV6fz86rtPDSoKx9eewIdEuv4HZYJFRE3SU0oh5DOG+3+dh188LkGreHyTyF3P4wYDLs2hG57psyKKwiOFJGd3m0XcET+fRHZGakAjb9WbtlDbp5SJT6ORy84gnG39ubyE5KJs0yhlU9C59BeVDZ/NCT1cNcpBEpMgUs/gd2bYcR5sG976LZpyqTIgkBV41W1nnerq6pVAu7Xi2SQJvKyc/N4fsJS+j31w4EkcT3bN6FVQ0sSV2klpsLere4AXV6bFsKmBXDYeYW/3upYGDoSti6BkRfD/j3l36Yps2BSTJgYM29dBoOem8pj4xZxepemnH1EC79DMpGQkD9yKAS1gvmjAYEuxYwradcHzn8d1v4EH14BOfvLv11TJlYQmEO8OXUFg56fyubdWbx02bE8f+kxJNSt7ndYJhIS80cOlbOfQNX1DySfBHWbFb9sl4Fw9tOw9Fv47DrIyyvftk2ZBHNlsYkB+ekguraoz3lHt+RfZ3Whfq2qfodlIqluc6hev/w1go3zXJNPj+uDW/7YYa6f4Nv7oGZDOPNx13ltIsYKghi3OyuHR79eSLX4OP51dhe6t21E97aWHiImibiO3M2LyreeeaNB4otvFiropL/Bvm0w9Rmo2QhOvbt8MZhSsaahGDZx0SbOeOoHRkxfhXIwdbSJYQkp7qKysn4XVF3/QNvepU9V0fcBOOYK+OFRmP5i2bZvysRqBDFo+579PPTlAkb/vI4OiXX4+LqeHNumod9hmYogMRV+fhv2bIY6iaV///pfYPtK6PX30r9XxPUX7NsOX98JNRrAUUNLvx5TalYjiEHb9+5n/PyN3HxqB768+SQrBMxB+SOHyppqYv5oiKsCKWeX7f1x8W4kUduTYcxfYeHYsq3HlEpYCwIR6S8ii0RkqYjcWcjrw0VkgYjM8abDbBPOeGLZpp2ZvPLDMlSVdgl1mHrHqQzv15nqVSxJnAmQUI7kc6ow/zNof2r55j6uUh2GvAfNj4SProSVU8q+LhOUsBUEIhIPPA8MwOUpGioiXQos9gvQTVWPAD4GHg1XPLFKVfnwpzWc9uQknhi/mJVb9wLYiCBTuLrNoEb9stUI1s6CjDUHcwuVR/W6cNkn0DAZRg6B338r/zpNkcJZI+gOLFXV5aq6HxgFHDKMQFUnqOpe7+F0oMC16KY81mzby+Wvz+Qfn8whtXk9vrrFksSZEoi4TKRlqRHMHw3x1SDlzNDEUquRy0tUs4FLRbFlaWjWa/4gnAVBS2BNwOO13nNFuQr4qrAXROQaEZklIrM2bw7B5e8xID9J3K9rdvDvwYcx6i89aJdgSeJMEBLLMHIoL881C3U43dUoQqV+S7j8M3d/xGDIWBe6dZsDKkRnsYhcBnQDHivsdVV9RVW7qWq3hISEyAYXZVYEJIl77IIjGX9rby7r0caSxJngJaRC5g7YvTH496yZDrvWF51bqDyadIDLR0NmhktfvWdr6LcR48JZEKwDkgIet/KeO4SI9AXuBgaqalYY46nUsnPzePa7JZzx1A+8/eNKAE5o35gWDWr6G5iJPmWZrWzeaKhSEzr1D09MzY+Eoe/DjlXw3gWQtSs824lR4SwIfgI6ikhbEamGm+c4LXABETkaeBlXCGwKYyyV2py1Ozjn2Sk88c1izjisGQOPsiRxphwSSplzKC8XFoyBTv2gehibH5NPggvfch3Hoy6FHDtvDJWwFQSqmgPcCIwD0oEPVXW+iDwoIgO9xR4D6gAficivIpJWxOpMEd6YsoLBz09l+979vHpFN54dejRN6liSOFMOdRJdzp9gcw6tnAJ7NoVmtFBJOg+AwS/AiknwyVWQmxP+bcaAsF5ZrKpjgbEFnrs34H7fcG6/MstPEndEq/pcfFwSdw5IpX5NGxJqQiB/5FCwNYL5o930kx37hTeufEcOOXj18Rd/g4HPWpK6crIUE1FmV2Y2j3y1kOpV4rn3nC50S25Et2RLEmdCLKGzO8CrFn+Qzc2GBWnuTL1aBCct6nE97N3m8hLVagSnPxi5bVdCFWLUkAnOhIWb6PfUD7w/czVV4sWSxJnwSUx1o3RKmlN4xSSXNTQco4VKcso/4birXcbSKU9HfvuViNUIosC2Pft58PP5fPbrejo1rcMLl/bk6NaWH8iEUeBsZfWaF73cvE+hej3o4EMrrwgMeOzQuQyOHRb5OCoBKwiiQMa+bL5L38Qtp3Xkr6d0oFoVq8iZMAucraz9qYUvk7MfFn4OKWe5/EB+iIuDwS+52ssXf3NXIZdmHgQDWNNQhbUhI5OXJrkkcW2b1GbKnady6+mdrBAwkVE7wU0QU9zIoWXfuwNwJEYLFadKNbhoBLQ6Dj65GpZN8DeeKGRHlQpGVXl/5mpOf3IST3+7mFX5SeJsRJCJJBFXKyhutrL5o92cAe36RCqqolWrBZd8AI07umsM1s7yO6KoYgVBBbJq6x4ueXUGd42eS9eW9fj6lt4kW5I445eEFNc0VNighOxMN1dA6jnujLwiqNnQpaKok+CuPi7rnAoxyAqCCiInN49LXp3B3HUZPHzu4Yy8uocVAsZfiamQlQG7fv/ja0u/gf27/BktVJy6zVySuvhqLi/R9lV+RxQVrCDw2bLNu8nJzaNKfBxPXHQk3wzvzSXHt7YkccZ/CZ3d38LOrOeNhlpNILl3ZGMKRqO2Ln119l5XGOy27DUlsYLAJ/tz8nj628X0f/oH3pnmzlp6tGtM8/qWJM5UEPk5hwomn9u/BxZ/DV0GQnwFHXjYtCtc8hHsXA/vnuc6tU2RrCDwwa9rXJK4p79dwpmHN2fw0cVN02CMT+okQK3Gf6wRLB7nzrb9Hi1UktbHw8Xvun6OkUMge5/fEVVYVhBE2OtTVnDeC1PJ2JfN68O68cyQo2lUu4J0thlTUGGzlc0fDXWaQpue/sRUGh37wrkvweppbv7j3Gy/I6qQrCCIkPx0EEcl1WdI99aMH96b01Kb+hyVMSVITHFDSPNHDmXtgiXfQJfBEBfva2hBO/wCOOtx15w15q9uNjVziArawFd57MzM5j9jF1Kjahz3ndOVY9s04tg2liTORImEFMjaCTvXQf1WsOgryMmseKOFSnLc1bB3O0z4txtm2v8Ry1gawGoEYfTtgo2c/uQkPvhpNdWqxFmSOBN9Egt0GM8bDfVaQqvu/sVUVr1vgx43wIyXYNKjfkdToViNIAy27s7igc8XkPbbelKa1eWVy7txZFIDv8MypvQCZytr2Q2WfgvHX+ty/EQbEej3f7BvB0x82KWv7v4Xv6OqEKwgCINdmTlMWLSJW/t24vo+7S0/kIletRu7vEOb02Hhl5CXXfFHCxUnLs5NZJO5A8beDnu3QnxVSO4FSVFYywkRibbmim7duumsWRUvj8j6Hfv49Jd13NCnPSLCzsxs6tWw/ECmEnjrbDf0smYD2LIEbvkt+tvXs/fB66fDhrmAuCuRL/sE2vbyO7KwEZHZqtqtsNesRlBOeXnKyJmreeSrheTmKWcd3pzkJrWtEDCVR0IK/Poe5O6HE26M/kIAoGpNN6vahrmAQm4WvH02NGrnmsMSA26NO1acfEphYgVBOazYsoc7P5nDjBXbOLFDY/5z7hG0bhzB6fqMiYTEFHcBGUTfaKHidDgdpj7rCri4eDjiYjdCalO6G2qquW45iYfGHQ4tHBJSXaFRUa+sLqXKsRc+yMnN47LXZrAzM5tHzz+CC7u1QirDmZIxBeV3GNds5LKOVhZJ3WFYGqyc/Mc+gpws1wy2eSFsWuA6y3//DRaMAbzm9Phq0KSTVzCkQGIXV2g2SI66znTrIyilpZt2kdy4NlXi45i5YhttGteiab0avsVjTNgt+RbeOx8QqFLDHTxjtWN1/17YssgVDJsWuNrD5oWQsebgMlVruYR9BZuY6rX0tVnN+ghCICsnl+cnLOOFCUu568xUrjqpLd3b2oVhJgZs+A0kDjTPNaOsnBy7BUG1WtDiaHcLlLnTXYG9acHBWsSy7+C3kQeXqV7Pqzl4BYMI7NoInfq7vEg+soIgCD+v3s4dH89hyabdnHd0S86zJHEmliT3gvjqrhCIr+Yem0PVqAdJx7lboL3bvFpDuvu7aSGkfw4/v31wmekvwLDPfS1crSAowas/LOfhr9JpXq8Gb/7pOE7pnOh3SMZEVnFt6aZ4tRpB8onulk8Vvn8IJj8F5LlEeD7XsqwgKEJenhIXJxzTpgGXHt+aO/qnUNeGhJpYldTdCoBQEXHNQdNeqDC1LCsICsjYl83/fbmAmlXjeWDQYZYkzhgTehWslmUFQYBx8zdwz2fz2LpnP9f2boeq2pBQY0x4VKBalhUEwJbdWdw3Zj5fzv2dLs3r8caVx3FYy/p+h2WMMRFhBQGwOzOHyUs2c/sZnbmmdzuqxkfXxSDGGFMeMVsQrNuxj09/XstfT+lAcpPa/HjXadSpHrMfhzEmhoX11FdE+ovIIhFZKiJ3FvJ6dRH5wHt9hogkhzMecKOBRkxbSb8nJ/H8hGWs2upyqFghYIyJVWE7+olIPPA8cDqwFvhJRNJUdUHAYlcB21W1g4gMAf4LXByumJZt3s1dn8xl5spt9OrYhIfPPZykRpYkzhgT28J5GtwdWKqqywFEZBQwCAgsCAYB93v3PwaeExHRMCRAysnN44rXZ7IrM5vHLjiCC461JHHGGAPhLQhaAgGZmFgLFEyocWAZVc0RkQygMbAlcCERuQa4BqB169ZlCqZKfBxPDzmKNo1qkWhJ4owx5oCoGB6jqq+oajdV7ZaQkFDm9RyX3MgKAWOMKSCcBcE6ICngcSvvuUKXEZEqQH1gaxhjMsYYU0A4C4KfgI4i0lZEqgFDgLQCy6QBw7z7FwDfh6N/wBhjTNHC1kfgtfnfCIwD4oE3VHW+iDwIzFLVNOB1YISILAW24QoLY4wxERTWwfOqOhYYW+C5ewPuZwIXhjMGY4wxxYuKzmJjjDHhYwWBMcbEOCsIjDEmxllBYIwxMU6ibbSmiGwGVpXx7U0ocNVyDLB9jg22z7GhPPvcRlULvSI36gqC8hCRWaraze84Isn2OTbYPseGcO2zNQ0ZY0yMs4LAGGNiXKwVBK/4HYAPbJ9jg+1zbAjLPsdUH4Exxpg/irUagTHGmAKsIDDGmBhXKQsCEekvIotEZKmI3FnI69VF5APv9RkikuxDmCEVxD4PF5EFIjJHRL4TkTZ+xBlKJe1zwHLni4iKSNQPNQxmn0XkIu9/PV9ERkY6xlAL4rvdWkQmiMgv3vf7TD/iDBUReUNENonIvCJeFxH5n/d5zBGRY8q9UVWtVDdcyutlQDugGvAb0KXAMjcAL3n3hwAf+B13BPb5FKCWd//6WNhnb7m6wA/AdKCb33FH4P/cEfgFaOg9TvQ77gjs8yvA9d79LsBKv+Mu5z73Bo4B5hXx+pnAV4AAPYAZ5d1mZawRdAeWqupyVd0PjAIGFVhmEPC2d/9j4DSJ7pnsS9xnVZ2gqnu9h9NxM8ZFs2D+zwAPAf8FMiMZXJgEs89/AZ5X1e0AqropwjGGWjD7rEA97359YH0E4ws5Vf0BNz9LUQYB76gzHWggIs3Ls83KWBC0BNYEPF7rPVfoMqqaA2QAjSMSXXgEs8+BrsKdUUSzEvfZqzInqeqXkQwsjIL5P3cCOonIVBGZLiL9IxZdeASzz/cDl4nIWtz8JzdFJjTflPb3XqKwTkxjKh4RuQzoBpzsdyzhJCJxwJPAlT6HEmlVcM1DfXC1vh9E5HBV3eFnUGE2FHhLVZ8QkRNwsx4epqp5fgcWLSpjjWAdkBTwuJX3XKHLiEgVXHVya0SiC49g9hkR6QvcDQxU1awIxRYuJe1zXeAwYKKIrMS1paZFeYdxMP/ntUCaqmar6gpgMa5giFbB7PNVwIcAqjoNqIFLzlZZBfV7L43KWBD8BHQUkbYiUg3XGZxWYJk0YJh3/wLge/V6YaJUifssIkcDL+MKgWhvN4YS9llVM1S1iaomq2oyrl9koKrO8ifckAjmu/0ZrjaAiDTBNRUtj2CMoRbMPq8GTgMQkVRcQbA5olFGVhpwhTd6qAeQoaq/l2eFla5pSFVzRORGYBxuxMEbqjpfRB4EZqlqGvA6rvq4FNcpM8S/iMsvyH1+DKgDfOT1i69W1YG+BV1OQe5zpRLkPo8D+onIAiAXuF1Vo7a2G+Q+/x14VURuxXUcXxnNJ3Yi8j6uMG/i9XvcB1QFUNWXcP0gZwJLgb3An8q9zSj+vIwxxoRAZWwaMsYYUwpWEBhjTIyzgsAYY2KcFQTGGBPjrCAwxpgYZwWB8YWXDfTdgMdVRGSziHzhZ1ylJSIrvfH6iMiPJSx7pYi0KOX6kwvLQikiy0Wkc4HnnhaRO4KJ1ZhAVhAYv+wBDhORmt7j0ynn1ZGh4l1tXmqq2rOERa4ESlUQFGMUAde/eCk1LvCeN6ZUrCAwfhoLnOXdHwq8n/+CiNT28rLP9PLMD/KeTxaRySLys3fr6T3fR0QmisjHIrJQRN4rLKOst8wzIvKriMwTke7e8/eLyAgRmYq72DBBRD4RkZ+824neco1FZLy4XP+v4VIB5697d8D9O0Rkroj8JiKPiMgFuBxP73nbrikix4rIJBGZLSLj8jNIes//JiK/AX8t4rN7H7g44HFvYJWqrhKRz7x1zheRawr5DA6pZYjIbSJyv3e/vYh87b1/soikFLF9U5n4nXvbbrF5A3YDR+DSgNcAfsVdTfmF9/rDwGXe/Qa4nDm1gVpADe/5jrirS/Hem4HLuxIHTANOKmS7E4FXvfu98XK+4zJYzgZqeo9H5r8faA2ke/f/B9zr3T8LdyVrk/x98v4OAH7k4PwPjQK23c27X9VbJsF7fDHuqlmAOUBv7/5jFJ2Xfh5wpHf/JeDGAtur6S3T2Hu8EpeDJzlwncBtwP3e/e+Ajt7943HpV3z/vtgtvLdKl2LCRA9VnSNudrihuNpBoH7AQBG5zXtcA3dAXg88JyJH4VIodAp4z0xVXQsgIr/iDnhTCtn0+972fxCReiLSwHs+TVX3eff7Al0CKhX1RKQOrvA4z3v/lyKyvZD19wXeVG/+B1UtLLd8Z1xSvG+8bcQDv3uxNFCXkx5gBK5gKcz7wBARmQ8MxqUiALhZRM717ifhCswS00x4+9eTg2lIAKqX9D4T/awgMH5LAx7HndEHzgkhwPmquihwYa8JYyNwJO7MP3DCmcCMqrkU/f0umFcl//GegOfigB6qesiENoW0NpWVAPNV9YQC629QinWMAsYDk4A5qrpRRPrgCqITVHWviEzEFaKBcji0WTj/9Thgh6oeVYoYTCVgfQTGb28AD6jq3ALPjwNuym/nF5c9FVzK8N/V5Zq/HHcmXVoXe+s8CZe5MaOQZcYTMMGJVwMBN+3lJd5zA4CGhbz3G+BPIlLLW66R9/wuXHpsgEVAgrj8+YhIVRHpqm7egB1ebACXFrUTqroM2AI8wsH+lfrAdq8QSMGl3y5oI5Do9XdUB8721rcTWCEiF3oxiYgcWdT2TeVhBYHxlaquVdX/FfLSQ7h29Dle08dD3vMvAMO8jtQUDj2LD1amiPyCa1e/qohlbga6iZscfAFwnff8A0BvL6bzcCmQC+7T17iaziyviSq/eest4CXvuXjcKJ//evvyK65ZBlw2yee95UqqgryP+xxGe4+/BqqISDqugJheSHzZwIPATFyhtTDg5UuBq7yY5lP49J+mkrHsoyameE0lt2l0z0tgTEhZjcAYY2Kc1QiMMSbGWY3AGGNinBUExhgT46wgMMaYGGcFgTHGxDgrCIwxJsb9P7xpn2qj26gXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reliability diagram\n",
    "fop, mpv = calibration_curve(testy, probs_test, n_bins=10, normalize=True)\n",
    "# plot perfectly calibrated\n",
    "pyplot.plot([0, 1], [0, 1], linestyle=\"--\")\n",
    "# plot model reliability\n",
    "pyplot.plot(mpv, fop, marker=\".\")\n",
    "pyplot.xlabel(\"Mean predicted Value\")\n",
    "pyplot.ylabel(\"Relative Frequency\")\n",
    "pyplot.title(\"Calibrated Classification Reliability Diagram\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d341c4bf",
   "metadata": {},
   "source": [
    "Here, we see that the shape of the calibrated probabilities is different, hugging the diagonal line much better, although still under-forecasting and over-forecasting i.e. the solid line is both above and below the ideal model.\n",
    "\n",
    "Visually, the plot suggests a better calibrated model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b646e340",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Store only the valuable columns.\n",
    "combined_df = df_os[[\"timestamp\", \"tab\", \"grid\", \"test\", \"prob\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7406bb19",
   "metadata": {},
   "source": [
    "We now have the probabilities for each test likely to fail stored in the column `prob`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aae618f3",
   "metadata": {},
   "source": [
    "## Save results to Ceph or locally\n",
    "* Use the following helper function to save the data frame in a parquet format on the Ceph bucket if we are running in automation, and locally if not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fe192358",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:53.427951Z",
     "start_time": "2021-04-01T15:43:51.595971Z"
    }
   },
   "outputs": [],
   "source": [
    "timestamp = datetime.datetime.now()\n",
    "\n",
    "if AUTOMATION:\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    cc.upload_to_ceph(\n",
    "        combined_df.head(1000000),\n",
    "        \"s3_path\",\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    save_to_disk(\n",
    "        combined_df.head(1000000),\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "2957126c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:53.914079Z",
     "start_time": "2021-04-01T15:43:53.429787Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-08-25 19:11:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.955633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-08-25 14:51:42</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-08-23 00:01:04</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.037275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-08-22 08:53:17</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.911741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2021-08-20 23:21:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.132383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2021-08-16 00:01:05</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.132383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1545</th>\n",
       "      <td>2021-08-16 03:07:16</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>TestInstall_test_install.start_install_and_wai...</td>\n",
       "      <td>0.955633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1547</th>\n",
       "      <td>2021-08-13 04:53:07</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>TestInstall_test_install.start_install_and_wai...</td>\n",
       "      <td>0.037275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1509</th>\n",
       "      <td>2021-08-11 20:45:05</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8141</th>\n",
       "      <td>2021-08-19 15:58:19</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>0.955633</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19674 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               timestamp                          tab  \\\n",
       "0    2021-08-25 19:11:17  \"redhat-assisted-installer\"   \n",
       "2    2021-08-25 14:51:42  \"redhat-assisted-installer\"   \n",
       "3    2021-08-23 00:01:04  \"redhat-assisted-installer\"   \n",
       "4    2021-08-22 08:53:17  \"redhat-assisted-installer\"   \n",
       "5    2021-08-20 23:21:32  \"redhat-assisted-installer\"   \n",
       "...                  ...                          ...   \n",
       "12   2021-08-16 00:01:05  \"redhat-assisted-installer\"   \n",
       "1545 2021-08-16 03:07:16  \"redhat-assisted-installer\"   \n",
       "1547 2021-08-13 04:53:07  \"redhat-assisted-installer\"   \n",
       "1509 2021-08-11 20:45:05  \"redhat-assisted-installer\"   \n",
       "8141 2021-08-19 15:58:19  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                   grid  \\\n",
       "0     periodic-ci-openshift-release-master-nightly-4...   \n",
       "2     periodic-ci-openshift-release-master-nightly-4...   \n",
       "3     periodic-ci-openshift-release-master-nightly-4...   \n",
       "4     periodic-ci-openshift-release-master-nightly-4...   \n",
       "5     periodic-ci-openshift-release-master-nightly-4...   \n",
       "...                                                 ...   \n",
       "12    periodic-ci-openshift-release-master-nightly-4...   \n",
       "1545  periodic-ci-openshift-release-master-nightly-4...   \n",
       "1547  periodic-ci-openshift-release-master-nightly-4...   \n",
       "1509  periodic-ci-openshift-release-master-nightly-4...   \n",
       "8141  periodic-ci-openshift-release-master-nightly-4...   \n",
       "\n",
       "                                                   test      prob  \n",
       "0                                               Overall  0.955633  \n",
       "2                                               Overall  0.000000  \n",
       "3                                               Overall  0.037275  \n",
       "4                                               Overall  0.911741  \n",
       "5                                               Overall  0.132383  \n",
       "...                                                 ...       ...  \n",
       "12                                              Overall  0.132383  \n",
       "1545  TestInstall_test_install.start_install_and_wai...  0.955633  \n",
       "1547  TestInstall_test_install.start_install_and_wai...  0.037275  \n",
       "1509                                            Overall  0.000000  \n",
       "8141                                            Overall  0.955633  \n",
       "\n",
       "[19674 rows x 5 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "if AUTOMATION:\n",
    "    sanity_check = cc.read_from_ceph(\n",
    "        \"s3_path\",\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    sanity_check = read_from_disk(\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24483935",
   "metadata": {},
   "source": [
    "## Conclusion : \n",
    "\n",
    "In this notebook, we calculated for a given moment in time, what is the probability that a test will fail. Using this information, we plan to build a model that uses the probability to fail value to predict the time to failure, that is how many days, weeks, months, hours, minutes or seconds do you have until the test in question stops working. As next steps, we plan to work on this [issue](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/226)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}