examples/od_mahalanobis_kddcup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mahalanobis outlier detection on KDD Cup '99 dataset\n",
    "\n",
    "## Method\n",
    "\n",
    "The Mahalanobis online outlier detector aims to predict anomalies in tabular data. The algorithm calculates an outlier score, which is a measure of distance from the center of the features distribution ([Mahalanobis distance](https://en.wikipedia.org/wiki/Mahalanobis_distance)). If this outlier score is higher than a user-defined threshold, the observation is flagged as an outlier. The algorithm is online, which means that it starts without knowledge about the distribution of the features and learns as requests arrive. Consequently you should expect the output to be bad at the start and to improve over time.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The outlier detector needs to detect computer network intrusions using TCP dump data for a local-area network (LAN) simulating a typical U.S. Air Force LAN. A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Each connection is labeled as either normal, or as an attack.\n",
    "\n",
    "There are 4 types of attacks in the dataset:\n",
    "\n",
    "- DOS: denial-of-service, e.g. syn flood;\n",
    "- R2L: unauthorized access from a remote machine, e.g. guessing password;\n",
    "- U2R: unauthorized access to local superuser (root) privileges;\n",
    "- probing: surveillance and other probing, e.g., port scanning.\n",
    "\n",
    "The dataset contains about 5 million connection records.\n",
    "\n",
    "There are 3 types of features:\n",
    "\n",
    "- basic features of individual connections, e.g. duration of connection\n",
    "- content features within a connection, e.g. number of failed log in attempts\n",
    "- traffic features within a 2 second window, e.g. number of connections to the same host as the current connection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import confusion_matrix, f1_score\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder\n",
    "\n",
    "from alibi_detect.od import Mahalanobis\n",
    "from alibi_detect.datasets import fetch_kdd\n",
    "from alibi_detect.utils.data import create_outlier_batch\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.utils.mapping import ord2ohe\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_instance_score, plot_roc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "We only keep a number of continuous (18 out of 41) features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(494021, 18) (494021,)\n"
     ]
    }
   ],
   "source": [
    "kddcup = fetch_kdd(percent10=True)  # only load 10% of the dataset\n",
    "print(kddcup.data.shape, kddcup.target.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume that a machine learning model is trained on *normal* instances of the dataset (not outliers) and standardization is applied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100000, 18) (100000,)\n",
      "0.0% outliers\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "normal_batch = create_outlier_batch(kddcup.data, kddcup.target, n_samples=100000, perc_outlier=0)\n",
    "X_train, y_train = normal_batch.data.astype('float'), normal_batch.target\n",
    "print(X_train.shape, y_train.shape)\n",
    "print('{}% outliers'.format(100 * y_train.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean, stdev = X_train.mean(axis=0), X_train.std(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load or define outlier detector\n",
    "\n",
    "The pretrained outlier and adversarial detectors used in the example notebooks can be found [here](https://console.cloud.google.com/storage/browser/seldon-models/alibi-detect). You can use the built-in ```fetch_detector``` function which saves the pre-trained models in a local directory ```filepath``` and loads the detector. Alternatively, you can initialise a detector from scratch. \n",
    "\n",
    "Be aware that `Mahalanobis` is an online, stateful outlier detector. Saving or loading a Mahalanobis detector therefore also saves and loads the state of the detector. This allows the user to *warm up* the detector before deploying it into production."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_outlier_detector = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:alibi_detect.od.mahalanobis:No threshold level set. Need to infer threshold using `infer_threshold`.\n"
     ]
    }
   ],
   "source": [
    "filepath = 'my_path'  # change to directory where model is downloaded\n",
    "if load_outlier_detector:  # load initialized outlier detector\n",
    "    detector_type = 'outlier'\n",
    "    dataset = 'kddcup'\n",
    "    detector_name = 'Mahalanobis'\n",
    "    od = fetch_detector(filepath, detector_type, dataset, detector_name)\n",
    "    filepath = os.path.join(filepath, detector_name)\n",
    "else:  # initialize and save outlier detector\n",
    "    threshold = None  # scores above threshold are classified as outliers   \n",
    "    n_components = 2  # nb of components used in PCA\n",
    "    std_clip = 3  # clip values used to compute mean and cov above \"std_clip\" standard deviations\n",
    "    start_clip = 20  # start clipping values after \"start_clip\" instances\n",
    "    \n",
    "    od = Mahalanobis(threshold, \n",
    "                     n_components=n_components,\n",
    "                     std_clip=std_clip,\n",
    "                     start_clip=start_clip)\n",
    "    \n",
    "    save_detector(od, filepath)  # save outlier detector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `load_outlier_detector` equals False, the warning tells us we still need to set the outlier threshold. This can be done with the `infer_threshold` method. We need to pass a batch of instances and specify what percentage of those we consider to be normal via `threshold_perc`. Let's assume we have some data which we know contains around 5% outliers. The percentage of outliers can be set with `perc_outlier` in the `create_outlier_batch` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0% outliers\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "perc_outlier = 5\n",
    "threshold_batch = create_outlier_batch(kddcup.data, kddcup.target, n_samples=1000, perc_outlier=perc_outlier)\n",
    "X_threshold, y_threshold = threshold_batch.data.astype('float'), threshold_batch.target\n",
    "X_threshold = (X_threshold - mean) / stdev\n",
    "print('{}% outliers'.format(100 * y_threshold.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New threshold: 10.670934046719518\n"
     ]
    }
   ],
   "source": [
    "od.infer_threshold(X_threshold, threshold_perc=100-perc_outlier)\n",
    "print('New threshold: {}'.format(od.threshold))\n",
    "threshold = od.threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detect outliers\n",
    "\n",
    "We now generate a batch of data with 10% outliers, standardize those with the `mean` and `stdev` values obtained from the normal data (*inliers*) and detect the outliers in the batch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 18) (1000,)\n",
      "10.0% outliers\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "outlier_batch = create_outlier_batch(kddcup.data, kddcup.target, n_samples=1000, perc_outlier=10)\n",
    "X_outlier, y_outlier = outlier_batch.data.astype('float'), outlier_batch.target\n",
    "X_outlier = (X_outlier - mean) / stdev\n",
    "print(X_outlier.shape, y_outlier.shape)\n",
    "print('{}% outliers'.format(100 * y_outlier.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "od_preds = od.predict(X_outlier, return_instance_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now save the *warmed up* outlier detector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_detector(od, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F1 score and confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.9693877551020408\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXPklEQVR4nO3de5QcVbXH8e8PQnhJCBDAkHB5GTsCQoAgQVAQ5EoQIYpwg6IQkUFFRa+KoF4RNN6lC0RYIjA8wysgKBJZ8YFoUESBEGJ4pS+RxyIRiAIJBDDJTO/7R9dIGzM9NTPdc6Zrfh9Wrek+VX1qNxN2NqdOnVJEYGZmA2+d1AGYmQ1VTsBmZok4AZuZJeIEbGaWiBOwmVkiTsBmZokMSx1AqymVSqcCJwECLi2Xy9/vZ3/HA1/L3n6rXC7PKJVKGwE3ATsBncDPyuXy6f05jw1KVwCHA0uBXRPHYgm4Au6FUqm0K9Xk+zZgd+DwUqn0ppyfnVMqlbZfo21z4Exgn6zPM0ul0mbZ7nPK5fJ4YA9gv1KpNLkx38IGkauAQ1MHYem4Au6dtwD3lMvlVwFKpdKdwAdKpdKPgQuBLYFXgZPK5fLCHP29B7i9XC6/kPV3O3BouVyeCfwWoFwuryqVSvOAsQ3/Npba74DtUwdh6dRNwJL2rLc/IuY1NpxB7yFgeqlU2gJ4DTgMmAu0A58ol8uPlUqlfYAfAgfl6G8M8HTN+8VZ2z+VSqWRwPuA8/sfvpkNJj1VwOfW2RfkSzKFUS6XHy2VSt8BfgW8AswHNgTeDtxUKpW6Dl0foFQqTQNOzdreBMwulUqrgCfK5fL7ezpfqVQaBswELiiXy4838ruYWXpqxloQktqANoBLLrlkr2kfeHfDzzEYfP/iqxi1xWZcds2NzJl1fd1jT/j0aUz/6hcYM3rrf7bNvn0O9z2wgDNP+ywAZ333AvbeYzcOO+RAAL727e+x0YYb8pXPf7Jp3yGF9UbtCMCw4WN6OLL4tttuLLf+dAYT9jg4dSjJdaxaAtWL2/2y+u+P505q643asd/n64/cF+Ek7SrpGEkf7dq6OzYi2iNiYkRMbGtra0ykg8TzLy4D4Jlnl3LHnX/giEMPZszoN/LL3/wegIhg4WP5itX99tmLu++dx/KXXmb5Sy9z973z2G+fvQC4oH0GK1a8yumnntycL2JWVJXO/FtiuS7CSToTOBDYGZgNTAbuAq5uWmSD1Oe/8i2WvfQSw4YN46tf+BQjNnkD3znzNL55zg+4ZMZMOjo6mHzwAYwft2OPfW06YhNOPuFYpn68OkrxiWkfYtMRm/Ds0r/RPuMGdthuW46e9hkAjj3qfXzwCF8wL5Jrr7mQA965L6NGbc6Tj8/lrLPP4cqrbkgdVuuLSuoIcss1BCHpQarTrh6IiN0lbQ1cGxGH5DhHrP67hy/tdR6CsLVp2BDEM4/mH4IY/ZakQxB5p6G9FhEVSR2SRlCdOL5tE+MyM+uTaKEKOG8CnitpJHApcD+wAvhj06IyM+urSsEScER8Knt5saRfACMiYkHzwjIz66MCVsBI2o3qXTvDsvdvioifNCkuM7O+6VydOoLc8s6CuALYDXgY6PrrJQAnYDMbXIo2BAFMioidmxqJmVkDtNJFuLw3YvxRkhOwmQ1+lUr+LbG8FfDVVJPws8BKqnP1IiJ2a1pkZmZ90UIVcN4EfDnwEeBBXh8DNjMbfIp2EQ74W0TMamokZmaNMAiGFvLKm4AfkHQ98DOqQxAAeBqamQ06BRyC2JBq4v3PmjZPQzOzwadIFbCkdYEFEXHeAMRjZtYvEemXmcyrx2loUf02xw5ALGZm/ReV/FtieYcg/iDpB8CNVB/FAwzJZ8KZ2WDX2ZE6gtzyJuAJ2c+za9qG3DPhzKwFDIInXeSVdzW0dzU7EDOzhhgEQwt55V2MZ1PgTOCdWdOdwNkRsbxZgZmZ9UkLzYLIuxbEFcDLwDHZ9hJwZbOCMjPrswJehNspIo6qeX+WpPnNCMjMrF9aqALO/Uw4SftHxF0AkvYDXmteWGZmfRMFXAvik8CMbCwY4EXg+OaEZGbWDwWsgB8FvgvsBIwElgNTAD8XzswGl0EwtptX3gR8K7AMmAcsaV44Zmb9VMAKeGxEHNrUSMzMGqGAFfDdkt4aEQ82NRozs/4q4K3I+wMnSHoCP5LIzAazAg5BTG5qFGZmjVK0BBwRTzU7EDOzhmjQGLCkEtUVILvsCHyd6kywk4C/Ze1fiYjZ2WfOAE4EOoHPRsQv650jbwVsZtYaGlQBR0SZbCXI7MEUS4BbgGnAeRFxTu3xknYGpgK7ANsAv5b05qizQnzetSDMzFpDc9aCOBj4Sw+jAUcCN0TEyoh4AlgEvK1ep07AZlYsnR35t/ymAjNr3n9a0gJJV0jaLGsbAzxdc8zirK1bTsBmViyVSu5NUpukuTVb25rdSRoOHAHclDVdRPWu4AnAM8C5fQ3VY8BmViy9GAOOiHagvYfDJgPzIuK57DPPde2QdClwW/Z2CbBtzefG0sOdw66AzaxYIvJv+RxLzfCDpNE1+94PPJS9ngVMlbS+pB2AccC99Tp2BWxmxdLAecCSNgYOAU6uaf6upAlUn4v5ZNe+iHhY0o+AR4AO4JR6MyDACdjMiqaBCTgiXgG2WKPtI3WOnw5Mz9u/E7CZFUsB14IwM2sN+cd2k3MCNrNiKdpaEGZmLcMJ2MwskQIuyG5m1hKio+7Mr0HFCdjMisUVsJlZIhXPgjAzS8MX4czMEnECNjNLxDdimJkl4lkQZmaJeBaEmVkingVhZpZG+CKcmVkiroDNzBLxGLCZWSKeBWFmloiHIMzMEvEQhJlZIq6AzczS8DQ0M7NUOpyAzczS8BiwmVkiHgM2M0sjnIDNzBJxAjYzS6SFZkGskzoAM7OG6qjk33ogaaSkmyUtlPSopH0lbS7pdkmPZT83y46VpAskLZK0QNKePfXvBGxmhRIRubcczgd+ERHjgd2BR4HTgTsiYhxwR/YeYDIwLtvagIt66twJ2MyKpRL5tzokbQq8E7gcICJWRcQy4EhgRnbYDGBK9vpI4Oqo+hMwUtLoeudwAjazYmlQAgZ2AP4GXCnpAUmXSdoY2DoinsmOeRbYOns9Bni65vOLs7ZuOQGbWaFEJXJvktokza3Z2mq6GgbsCVwUEXsAr/D6cEP1XNVxjD5Pu/AsCDMrll5MQ4uIdqC9m92LgcURcU/2/maqCfg5SaMj4plsiGFptn8JsG3N58dmbd1yBWxmhRIdkXur20/Es8DTkkpZ08HAI8As4Pis7Xjg1uz1LOCj2WyIScDymqGKtXIFbGbF0tgbMT4DXCdpOPA4MI1q4fojSScCTwHHZMfOBg4DFgGvZsfW5QRsZsXSwPswImI+MHEtuw5ey7EBnNKb/p2AzaxQvBaEmVkqrXMnshOwmRVLTxfXBhMnYDMrlBZaj90J2MwKxgnYzCwNV8BmZqk4AZuZpeEK2MwskUpH6gjycwI2s2IJpY4gNydgMysUD0GYmSUSFVfAZmZJuAI2M0uk0ukK2MwsCQ9BmJklku9p84ODE7CZFYorYDOzRJyAzcwS8RCEmVkilc7Wedi7E7CZFYrnAZuZJVLxWhBmZmmEE7CZWRqeBWFmlohnQZiZJdLpWRBmZmm00hhw6/xVYWaWQ0T+LQ9J60p6QNJt2furJD0haX62TcjaJekCSYskLZC0Z099uwI2s0JpwjS0U4FHgRE1bV+KiJvXOG4yMC7b9gEuyn52yxWwmRVKhHJvPZE0FngvcFmOUx8JXB1VfwJGShpd7wMDUgGvN2rHgTiNtZiOVUtSh2AF1NnYaWjfB04DNlmjfbqkrwN3AKdHxEpgDPB0zTGLs7ZnuuvcFbCZFUpvKmBJbZLm1mxtXf1IOhxYGhH3r3GKM4DxwN7A5sCX+xrrwFTAw8cMxGmsRazOKt/xW+2dOBIbTBYuva8h/fRmDDgi2oH2bnbvBxwh6TBgA2CEpGsj4rhs/0pJVwJfzN4vAbat+fzYrK1broDNrFCiF1vdfiLOiIixEbE9MBX4TUQc1zWuK0nAFOCh7COzgI9msyEmAcsjotvhB/AsCDMrmAFYjOc6SVsCAuYDn8jaZwOHAYuAV4FpPXXkBGxmhdKMGzEiYg4wJ3t9UDfHBHBKb/p1AjazQumkde6EcwI2s0KpeDEeM7M0Kq6AzczSCCdgM7M0WuiRcE7AZlYsroDNzBLpSB1ALzgBm1mhuAI2M0ukhZ7J6QRsZsXiaWhmZom00H0YTsBmViwdcgVsZpaEK2Azs0R8I4aZWSKeBWFmlohnQZiZJeIxYDOzRDpapwB2AjazYnEFbGaWiC/CmZkl4mloZmaJOAGbmSXShKfSN40TsJkVihdkNzNLxLMgzMwS8SwIM7NEWuki3DqpAzAza6RKL7Z6JG0g6V5Jf5b0sKSzsvYdJN0jaZGkGyUNz9rXz94vyvZv31OsTsBmViidyr/1YCVwUETsDkwADpU0CfgOcF5EvAl4ETgxO/5E4MWs/bzsuLqcgM2sUBpVAUfViuztetkWwEHAzVn7DGBK9vrI7D3Z/oOl+o/ncAI2s0KJXmw9kbSupPnAUuB24C/Asojomu22GBiTvR4DPA2Q7V8ObFGvfydgMyuUCpF7k9QmaW7N1lbbV0R0RsQEYCzwNmB8I2P1LAgzK5TezIKIiHagPcdxyyT9FtgXGClpWFbljgWWZIctAbYFFksaBmwKPF+vX1fAZlYojRqCkLSlpJHZ6w2BQ4BHgd8CH8wOOx64NXs9K3tPtv83EVH3NK6AzaxQGrgg+2hghqR1qRarP4qI2yQ9Atwg6VvAA8Dl2fGXA9dIWgS8AEzt6QROwGZWKJUG3YwcEQuAPdbS/jjV8eA12/8BHN2bczgBm1mheC0IM7NEWulWZCdgMyuURg1BDAQnYDMrlM7UAfSCE7CZFYorYDOzRFon/ToBm1nB+CKcmVki0UI1sBOwmRWKK2Azs0Q6XQGbmaXhWRBmZol4CMLMLBFfhDMzS8QVsJlZIq6AzcwS6aj/EIpBxQnYzAqlddKvE7CZFYynoZmZJeIxYDOzRDwLwswskc4WSsFOwGZWKK2Tfp2AzaxgwtPQzMzS8CwIM7NEPARhZpaIp6GZmSXSGa1TAzsBm1mhtE76hXVSB2Bm1kjRi396IukKSUslPVTT9g1JSyTNz7bDavadIWmRpLKk9/TUvytgMyuUBs+CuAr4AXD1Gu3nRcQ5tQ2SdgamArsA2wC/lvTmiOjsrnNXwGZWKBGRe8vR1++AF3Ke+kjghohYGRFPAIuAt9X7gBOwmRVKhci9SWqTNLdma8t5mk9LWpANUWyWtY0Bnq45ZnHW1i0nYDMrlM6o5N4ioj0iJtZs7TlOcRGwEzABeAY4t6+xOgGbWaFEL7Y+9R/xXER0RkQFuJTXhxmWANvWHDo2a+uWE7CZFUpvhiD6QtLomrfvB7pmSMwCpkpaX9IOwDjg3np9eRaEmRVKI2dBSJoJHAiMkrQYOBM4UNIEqkX0k8DJABHxsKQfAY8AHcAp9WZAgBOwmRVMI1dDi4hj19J8eZ3jpwPT8/bvBGxmheIF2c3MEvF6wGZmiXg9YDOzRFwBm5kl4grYzCwRL8huZpaIF2Q3M0uk4jFgM7M0PARhZpaIK2Azs0RcAZuZJeIK2MwskUr9BcgGFSdgMysU34hhZpaIb0U2M0vEFbCZWSKugM3MEvGtyGZmibgCNjNLxGPAZmaJuAI2M0vEd8KZmSXiCtjMLBHPgjAzS8RDEGZmiXg5Svs3j/3fn1ixYgWdnRU6OjqYtO9hqUOyBD5y0lSOPm4Kkrjp2p9ydftMPv2lkzj6uCm88PwyAM6bfiG/u+PuxJG2LlfAtlbvPuRonn/+xdRhWCLjxu/E0cdN4ZhDj2f1qg4uvfEC5tz+ewBmXDKTK354beIIi6FQF+EkrQNMigj/lWzWDzuO254F8x7iH6+tBOC+u+dxyHvflTiq4qm00EW4dXo6ICIqwIUDEEuhRQQ/nz2Te/70cz5+4odTh2MJPLbwL0ycNIGRm23KBhuuzwHvfjujt9kagA9/7GhunXM907//P4zYdJPEkba2iMi9paY8QUg6B/gj8JPofdSx3vAxfYmtULbZ5o389a/PsuWWW/CLn9/AqZ/7GnfddU/qsJJYvWoJAOO32jtxJAPvqA8dwYemfZBXX32NReXHWbVyNe0XXMWLzy8jIjj19E+w5daj+Ornvpk61AG3cOl9AOpvP8OGj8mdozpWLen3+fojbwJ+GdgY6AReo/ovKSJiRDfHtwFt2dv2iGhvTLitTVJb9u/iG8AK4Jy0EVli3wYWS+qo+W9ke+A2YNdkUdmAyZWArd82BtaR9NuIOAC4HTgb+EXasCyBrYClwH8AvwImbbfddnc+9dRTu2f7Pw/sA0xNFJ8NoFyzICQJ+DCwQ0R8U9K2wOiIuLep0RXH1sAtCxcuHAfcC1yPk+9Q9WNgC2A1cAqw7Pzzzx8LPAgE8CRwcrLobEDlHYK4CKgAB0XEWyRtBvwqIobeIF4/SJobERNTx2GDi/9cDF155wHvExF7SnoAICJelDS8iXEVlcfCbW3852KI6nEaWma1pHWp/i8SkrakWhFbL/hipK2N/1wMXXkT8AXALcBWkqYDd1G9gmtmZn2UKwFHxHXAacD/As8AUyLipmYGZv9K0hxJHicsAEknSNqm5v0/f7eSZksamS46G0h1x4AljYiIlyRtTnXqzMyafZtHxAvNDrAIJA2LiI7UcdigcQLwEPDXNXdERK9WaZK0bkR0NiguG2A9XYS7HjgcuJ/q+K/W+LljU6MbRCRtD/yc6vDL24ElwJFACbgY2Aj4C/Cx7CLlHGA+sD8wU9Jbqd7EsgfVuaAfAz4K7AvcExEnZOe5CNgb2BC4OSLOHJAvaP0i6b+p/k4BLgN+CtwWEbtm+78IvIFq4p0IXCfpNaq//9p+ngQmRsTfJR0HfBYYDtwDfCoiOiWtAC4B3k11KttdTf561iR1hyAi4vDs5w4RseOaPwcmxEFlHHBhROwCLAOOAq4GvhwRu1Gdy1mbMIdHxMSIODd7vxnV/+A+D8wCzgN2Ad4qaUJ2zFezKUm7AQdI2q3ZX8r6R9JewDSqN1BMAk6i+rv+NxFxMzAX+HBETIiI17rp8y3AfwH7RcQEqnehdi0isjHVv7R3jwgn3xbW0xDEnvX2R8S8xoYz6D0REfOz1/cDOwEjI+LOrG0GUDs2fuMan/9ZRISkB4HnIuJBAEkPU70FdT5wTHYr9zBgNLAzsKAZX8YaZn/gloh4BUDST4B39LPPg4G9gPuq90GxIdVhQKgm4x/3s38bBHoagji3zr4ADmpgLK1gZc3rTqCniyWvdPP5yhp9VYBhknYAvgjsnQ1jXAVs0PdwLaGR/Ov/Yfb29yhgRkScsZZ9//C4bzH0NATxroh4FzC563VNmx/pAMuBFyV1VTsfAe6sc3xPRlBN2sslbQ1M7md8NjB+D0yRtJGkjYH3U71esJWkLSStT/VaSpeXgZ7WnLwD+KCkraB60VvSdk2I3RLKeyfc3cCawxFraxuKjgculrQR8DjVscA+iYg/Z3cbLgSeBv7QmBCtmSJiXvZ/K11ro1wWEfdJOjtrW0L1d9rlKqp/Zv7tIlxNn49I+hrwq+yhCF1rRzzVnG9hKdRdC0LSG4ExwLXAh3h9rc4RwMURMb7pEZqZFVRPFfB7qM5ZHAt8r6b9ZeArTYrJzGxIyLsa2lER4auuZmYNlHcMeFdJu6zZGBFnNzgeM7MhI28CXlHzegOqV3QfbXw4ZmZDR58eSZRNq/llRBzY8IjMzIaIvMtRrmkjqhfmzMysj/I+E67reVVQTdpbAUPvudlmZg2UdxbEdlQXF3kH1VssZ0fE/U2Ozcys0PIOQRwJXAOMAtYDrpT0maZFZWY2BOStgBcA+9as9rQx8MdsCUYzM+uDvBWwqK7+1aWT129LNjOzPsg7D/hK4B5Jt2TvpwCXNyckM7OhIfc84Gxx9v2zt7+PiAeaFpWZ2RDQpxsxzMys//p6I4aZmfWTE7CZWSJOwGZmiTgBm5kl4gRsZpbI/wP0BOLFUNIagwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "labels = outlier_batch.target_names\n",
    "y_pred = od_preds['data']['is_outlier']\n",
    "f1 = f1_score(y_outlier, y_pred)\n",
    "print('F1 score: {}'.format(f1))\n",
    "cm = confusion_matrix(y_outlier, y_pred)\n",
    "df_cm = pd.DataFrame(cm, index=labels, columns=labels)\n",
    "sns.heatmap(df_cm, annot=True, cbar=True, linewidths=.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot instance level outlier scores vs. the outlier threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5wU9ZX38c+ZcYARlQFDlIsGvMEi4KAD6GJWDFm84SVKMMaskBgNPhpFXSM+m1WSJ9lINIpmXRNIBDXESNSAskZWiZdoFB0CDiKy3lDBC4oMCo4wwHn+qOqhZ+ju6enp6u7p/r5fr3lNd3V11amqmdO//tWvTpm7IyIipaMs3wGIiEhuKfGLiJQYJX4RkRKjxC8iUmKU+EVESowSv4hIidkjyoWb2RrgU2AHsN3da8ysB3Av0A9YA0xw941RxiEiIrvkosV/vLtXu3tN+HwqsNjdDwUWh89FRCRH8tHVczpwZ/j4TuCMPMQgIlKyLMord83sTWAj4MCv3X2mmdW7e1X4ugEbY89bvPdC4EKArl27HjVw4MDI4hTpSOo/a2RdfQM74/53y8zoU1VJ1Z4VeYxMCs3SpUs/cveeLadH2scPHOvu68zsi8CjZvZK/Ivu7maW8JPH3WcCMwFqamq8trY24lBFOo75y9Zxw6LVvFvfQO+qSq46YQBnDOuT77CkwJjZW4mmR5r43X1d+Hu9mf0JGAF8YGa93P09M+sFrI8yBpFidMawPkr0krHI+vjNrKuZ7R17DIwFXgIeBCaGs00EFkQVg4iI7C7KFv9+wJ+Cbnz2AH7v7o+Y2QvAPDM7H3gLmBBhDCIi0kJkid/d3wCOSDB9AzAmqvWKSGFrbGxk7dq1fP755/kOpWh06dKFvn37UlGR3sn9qE/uiog0s3btWvbee2/69etH2CMg7eDubNiwgbVr19K/f/+03qOSDSKSU59//jn77ruvkn6WmBn77rtvm75BKfGLSM4p6WdXW/enEr+ISIlR4hcRybHRo0eTz4tSdXJXRApaoV2lvH37dvbYo2Onzo4dvYgUtfnL1nHNAytoaNwBwLr6Bq55YAVAu5L/mjVrOOmkkzj22GP529/+Rp8+fViwYAGrV69m8uTJfPbZZxx88MHccccddO/endGjR1NdXc3TTz/NOeecw4oVK6isrGTZsmWsX7+eO+64g7vuuotnn32WkSNHMmfOHAAuuugiXnjhBRoaGhg/fjw/+tGP2r1PskFdPSJSsG5YtLop6cc0NO7ghkWr273sV199lYsvvpiVK1dSVVXF/fffz3nnncf06dOpq6tjyJAhzRL1tm3bqK2t5corrwRg48aNPPvss9x8882cdtppXH755axcuZIVK1awfPlyAH76059SW1tLXV0dTz75JHV1de2OOxuU+EWkYL1b39Cm6W3Rv39/qqurATjqqKN4/fXXqa+v57jjjgNg4sSJPPXUU03zn3322c3ef+qpp2JmDBkyhP32248hQ4ZQVlbG4Ycfzpo1awCYN28eRx55JMOGDWPlypW8/PLL7Y47G9TVIyIFq3dVJesSJPneVZXtXnbnzp2bHpeXl1NfX59y/q5duyZ8f1lZWbNllZWVsX37dt58801uvPFGXnjhBbp3786kSZMK5mpltfhFpGBddcIAKivKm02rrCjnqhMGZH1d3bp1o3v37vz1r38F4O67725q/Wfik08+oWvXrnTr1o0PPviAP//5z9kKtd3U4heRghU7gZurUT133nln08ndgw46iNmzZ2e8rCOOOIJhw4YxcOBADjjgAEaNGpXFSNsn0jtwZYtuxCJSPFatWsU//MM/5DuMopNov5rZ0rj7nTdRV4+ISIlR4hcRKTFK/CIiJUaJX0SkxCjxi4iUGCV+EZESo8QvItKKOXPm8O677zY9jy+rfPLJJ7d61W+hUeIXkcJWNw9uHgzTqoLfdfNyHkLLxB/v4YcfpqqqKu1l7dixo/WZIqbELyKFq24ePHQpbHoH8OD3Q5dmJfnfdNNNDB48mMGDBzNjxgzWrFnD4MGDm16/8cYbmTZtGvfddx+1tbWce+65VFdX09DQvHZQv379+OijjwD43e9+x4gRI6iuruZ73/teU5Lfa6+9uPLKKzniiCN49tln2x17eynxi0jhWvxjaGxRpK2xIZjeDkuXLmX27NksWbKE5557jlmzZrFx48aE844fP56amhrmzp3L8uXLqaxMXCBu1apV3HvvvTzzzDMsX76c8vJy5s6dC8CWLVsYOXIkL774Iscee2y7Ys8G1eoRkcK1aW3bpqfp6aef5mtf+1pTxc0zzzyzqThbphYvXszSpUsZPnw4AA0NDXzxi18EguqfZ511VruWn01K/CJSuLr1Dbt5EkzPsvr6enbu3Nn0vK0llN2diRMn8rOf/Wy317p06UJ5eXmCd+WHunpEpHCNuRYqWnStVFQG09vhy1/+MvPnz+ezzz5jy5Yt/OlPf+Kkk05i/fr1bNiwga1bt7Jw4cKm+ffee28+/fTT1KGOGcN9993H+vXrAfj4449566232hVnVNTiF5HCNXRC8Hvxj4PunW59g6Qfm56hI488kkmTJjFixAgAvvvd7zJ8+HCuvfZaRowYQZ8+fRg4cGDT/JMmTWLy5MlUVlYmPTk7aNAgfvKTnzB27Fh27txJRUUFt912G1/60pfaFWsUVJZZRHJKZZmjobLMIiKSlBK/iEiJUeIXESkxSvwiIiVGiV9EpMQo8YuIlBglfhEpKRs2bKC6uprq6mr2339/+vTpQ3V1NVVVVQwaNCjr63viiScYN25cm94TX/Y53pw5c7jkkkvaHVPkid/Mys1smZktDJ/3N7MlZvaamd1rZp2ijkFEJGbfffdl+fLlLF++nMmTJ3P55Zc3PS8raz0lbt++PQdRRisXLf7LgFVxz6cDN7v7IcBG4PwcxCAi0qodO3ZwwQUXcPjhhzN27NimEsyjR49mypQp1NTUcMstt/Dhhx9y1llnMXz4cIYPH84zzzwDwJNPPtn0bWLYsGFNZR42b97M+PHjGThwIOeeey6xC2cXL17MsGHDGDJkCN/5znfYunXrbjHNnj2bww47jBEjRjStp70iLdlgZn2BU4CfAleYmQFfAb4ZznInMA24Pco4RKQwTXlkCsvfX57VZVbvX82ME2dk9N5XX32Ve+65h1mzZjFhwgTuv/9+vvWtbwGwbdu2pu6Xb37zm1x++eUce+yxvP3225xwwgmsWrWKG2+8kdtuu41Ro0axefNmunTpAsCyZctYuXIlvXv3ZtSoUTzzzDPU1NQwadIkFi9ezGGHHcZ5553H7bffzpQpU5riee+997juuutYunQp3bp14/jjj2fYsGHt3EPRt/hnAD8AYiXv9gXq3T32XWkt0CfRG83sQjOrNbPaDz/8MOIwRUSgf//+VFdXA3DUUUexZs2aptfOPvvspsePPfYYl1xyCdXV1Zx22ml88sknbN68mVGjRnHFFVdw6623Ul9fzx57BG3rESNG0LdvX8rKyqiurmbNmjWsXr2a/v37c9hhhwEwceJEnnrqqWbxLFmyhNGjR9OzZ086derULIb2iKzFb2bjgPXuvtTMRrf1/e4+E5gJQa2eLIcnIgUg05Z5VDp37tz0uLy8vNndtmK1+wF27tzJc88919Sij5k6dSqnnHIKDz/8MKNGjWLRokUJl5vv8wRRtvhHAaeZ2RrgDwRdPLcAVWYW+8DpC6yLMAYRkawbO3Ysv/zlL5ueL18edFe9/vrrDBkyhKuvvprhw4fzyiuvJF3GgAEDWLNmDa+99hoAd999N8cdd1yzeUaOHMmTTz7Jhg0baGxs5I9//GNW4o8s8bv7Ne7e1937Ad8A/uLu5wKPA+PD2SYCC6KKQUQkCrfeeiu1tbUMHTqUQYMG8atf/QqAGTNmMHjwYIYOHUpFRQUnnXRS0mV06dKF2bNn8/Wvf50hQ4ZQVlbG5MmTm83Tq1cvpk2bxjHHHMOoUaOyVtU0J2WZw66ef3X3cWZ2EME3gB7AMuBb7r77qew4KsssUjxUljkabSnLnJMbsbj7E8AT4eM3gBG5WK+IiOxOV+6KiJQYJX4RybmOcOe/jqSt+1OJX0RyqkuXLmzYsEHJP0vcnQ0bNuw2tDQV3WxdRHKqb9++rF27Fl2YmT1dunShb9++ac+vxC8iOVVRUUH//v3zHUZJU1ePiEiJUeIXESkxSvwiIiVGiV9EpMQo8YuIlJhWE7+Z7Wlm/25ms8Lnh4Yll0VEpANKp8U/G9gKHBM+Xwf8JLKIREQkUukk/oPd/edAI4C7fwZYpFGJiEhk0kn828ysEnAAMzuY4BuAiIh0QOlcuXsd8AhwgJnNJbiz1qQogxIRkeikTPxmZsArwJnA0QRdPJe5+0c5iE1ERCKQMvG7u5vZw+4+BPjvHMUkIiIRSqeP/+9mNjzySEREJCfS6eMfCZxrZm8BWwi6e9zdh0YamYiIRCKdxH9C5FGIiEjOtNrV4+5vAVXAqeFPVThNREQ6oHRKNlwGzAW+GP78zsy+H3VgIiISjXS6es4HRrr7FgAzmw48C/wyysBERCQa6YzqMWBH3PMdqGSDiEiHlU6LfzawxMz+FD4/A/htdCGJiEiUWk387n6TmT0BHBtO+ra7L4s0KhERiUyrid/MjgZWuvvfw+f7mNlId18SeXQiIpJ16fTx3w5sjnu+OZwmIiIdUFond93dY0/cfSfpnRsQEZEClE7if8PMLjWzivDnMuCNqAMTEZFopJP4JwP/SHDLxXUEtXsujDIoERGJTjqjetYD38hBLCIikgNJW/xmdoGZHRo+NjO7w8w2mVmdmR2ZuxBFRCSbUnX1XAasCR+fAxwBHARcAdwSbVgiIhKVVIl/u7s3ho/HAXe5+wZ3fwzo2tqCzayLmT1vZi+a2Uoz+1E4vb+ZLTGz18zsXjPr1P7NEBGRdKVK/DvNrJeZdQHGAI/FvVaZxrK3Al9x9yOAauDE8GKw6cDN7n4IsJGgCJyIiORIqsR/LVBL0N3zoLuvBDCz40hjOKcHYhd+VYQ/DnwFuC+cfidB7R8REcmRpKN63H2hmX0J2NvdN8a9VAucnc7CzawcWAocAtwGvA7Uu/v2cJa1QJ8k772QcNjogQcemM7qREQkDSnH8bv79hZJH3ffEteST8ndd7h7NdAXGAEMTDcwd5/p7jXuXtOzZ8903yYiIq1I5wKudnP3euBx4Bigysxi3zT6ElwUJiIiORJZ4jeznmZWFT6uBP4ZWEXwATA+nG0isCCqGEREZHdJ+/hbu0grVqY5hV7AnWE/fxkwLzxv8DLwBzP7CbAM3dRFRCSnUpVs+EWK12Kjc5LP4F4HDEsw/Q2C/n4REcmDVKN6js9lICIikhut9vGb2Z5m9kMzmxk+P9TMxkUfmoiIRCGdk7uzgW0EpZkhGIXzk8giEhGRSKWT+A92958DjQDu/hlgkUYlIiKRSSfxbwuHYzqAmR1MUIdHREQ6oHTunTsNeAQ4wMzmAqOASRHGJCKlom4eLP4xbFoL3frCmGth6IR8R1X0LO4+6slnMtsXOJqgi+c5d/8o6sDi1dTUeG1tbS5XKaVISah1dfPgz1dDw8fB88oecNL05Psp1fx18+ChS6GxIe4NBjh0O6B4938O/87MbKm71+w2vbXEb2YPAb8nqNC5JZLoWqHEL5Grmwfz/w/sbNw1rawCzviv3f8po/jH7QgfOon2UbyWyTrZ/OWd4PTbwu19J/U6O3WFcTMKb19kKtGHXUUlnHprJNvYnsR/HEE1zlOAF4A/AAvd/fOsR5mEEr9Ebnr/Xa3SeBVd4d/e3fU81QcEZJa8k7V8a74DBx7dfJmHjoVX/yf5OuI/QCq7B9MaNrY+bzrx3jy49UQdn8RSzd/tgGC9tN7j0EzsGwPs+uCwcvAdbfuWkGrb27JfsrUPux0Al7/UetxtlHHij1tAOcHVuhcAJ7r7PtkNMTklfonctG7JXztz1q5/5lQfEOxsnrzLKoLWbWP4RTlZt0iqBFneCXZsSxF4XNfIoWPhxd+3+ABpw7yttTynVZFWoo4lsZTzW5AoW/sgSfjWcigrT7Jfwm1M9WHQ2gdtom8piY5dJq33pPvEYFp9ys3ORLLEn1aRtnBUz1nAZGA4wQ1URKJVNy9IitOqgt918/ITx+If73qcKOlDkNxbJtydjbuSfuy9Cy7efTs2rU2+7pRJH5qSyKZ3oPaOFEk/jXkbG5pva0vd+rYSSyi2Panmj7WOK9K5mV8LviPFfvFd80CwrQ9d2nyfL/5xgv3kwT55aErirqyGj9NbTqb7MN19myXpXLk7j6Cq5leA/yQY1//9qAOTEhdrTW16B/Dg9wMXBi3zKD4EKnskfy1VYm6rHdt2TwxZ+6dvS7dJknlTbeuYa4NvMa2JbU+y+cs77WqFn3pr0CqPUstknHQbvfkHdabLaW0ftvywq6gMpudQOsM5fwuc4x77CJWS17Jfs7V+52TvTdUHnaxVBrtacZC9E2InTYcHLkj8WnxiruyRuNVvZeA701tXy8Qw5trk6861VB9CsX0dP0qnpfgklmj+ll0mQyfserzwCqiNqFhv/D7PtIsp3eWksw/zfCI/nZO7ewJXAAe6+4VmdigwwN0X5iJAUB9/QUnYP5pEolEeqd4b3z+aqs89fvnZPCG28Irg6358a7hln23dvKC7Jr6robwTDPuXVvrXW4n7P3rDtmwMmgv7uDOZN5PRJdkejdRy+GciKfv4k4jf53Xzgm+PifZTax/gLZeTwxE6mWhPH79q9ZSiZP3rCVviSbTsX23tvfFfpa08jeVnsQsGYNxNcObMsOvBgt8t/4mHTgiGIsbPc/ptwXubui0saNkm2oZYN8du654RvNZy3przm6+r6TnsVjmlojI4QRkfQ1MXVivzJtrWdAydEJ7IrQ9+tzfhDZ0AV78J0zYFP2fOat4NV9kDvvaruGNA3H5OUkmmZVfK0AnBtifaJ0d9e/fjkGo58cc8032YB+m0+GvdvcbMlrn7sHDai+5+RE4iRC3+nEvVkknWUkolrVEeMeHohny0+LMtk4ud2tJ6jnLYYUfUtI1pDvFMtk/aupwC1p5x/H8DxgDPuPuRYa2ee9w9ZzdTUeJvIep/4lRjjSGD/tEwmaczDjyWzFubt8C+UosUovZ09VxH81o9i4EfZDk+SVey0S4Lr8jeOlKNVshkCF78KI9U743/Kp1w3vCreQf6Si1SiFod1ePuj5rZ39lVq+cyIEknWAmIurXd2vKTjkH+LSydHZyYau/X0lSjFVqOSqjsDls3wc4kg74SjfJIZ1RPgYx+EClGaV+52+xNZm+7+4ERxJNQwXT1JByVUhYOjtgJGHTaMxidkell5K2NEkj36sn2dIW0dbRCW8oEiEjOtLtkQ4uFvePuEV91sUvBJP50+qhbk+oEXzp1PNoSQ3tOfpbCyUApLPqby7pkiT+dC7gSafunRT4k+kOCzC8+ysbwwdhl+7D7etK5EnDMtemPrGlPvPEX1khqSljt1/JbZhQX6UmTpInfzH5J0mpCVEUWUbYk+kN64ILgwo9Yf/Smd5pfKdjaH1t7rviLF7tsv+U60rkScOgEePu53S80SiTH9T9KkhJWdqSqe6P9mHWpRvXUAksT/NQChV+rJ9nFQslOQsakKrKUaVGpRBK1xtOt4xG70ChVfZk81P8oSZkU6pLdZVL3RjKWtMXv7h27Amd7/mCSvbflSJOKPVMXdUolUWu8LSNZYl0xRXSxSYekhJUdmdS9kYxl2sdf+NrTLdNakaX4hLrwClg6JywDm2BUTyLJLttPtPzWqC8+v5SwsmPMtYlHkulbaySKN/G35SRovLb+sY27KfhJpq2X7UvHooSVHbpuI6cyGs6ZaxkP50xUbbGsAjrvvWuceVtG9YgkolE9UqAyHs5pZocBtwP7uftgMxsKnObuhV+hc9xNu9+zVP+Ukm3qbpMOJp2unlnAVcCvAdy9zsx+T0cpzax/ShGRZtIp0ranuz/fYtr2KIIRKXiFch9gkXZIp8X/UViK2QHMbDzwXqRRiRQiXawlRSKdFv/FBN08A81sHTAFuCjSqEQKkS7WkiKRTlnmN4CvmllXoMzdP40+LJECpIu1pEi02uI3s/8wsyp33+Lun5pZdzPrGCd2RbIp2UVZulhLOph0unpOcvf62BN33wic3NqbzOwAM3vczF42s5Vmdlk4vYeZPWpmr4a/u2cevkgOpVtLSaTApZP4y82sc+yJmVUCnVPMH7MduNLdBxHcvetiMxsETAUWu/uhBLdxnNr2sEXyYOiE4GY03Q4ATLeAlA4rnVE9c4HFZjY7fP5toNUCbu7+HuHon7CLaBXQBzgdGB3OdifwBHB1m6IWyRddFyJFIJ2Tu9PNrA4YE076f+6+qC0rMbN+wDBgCcEVwLHhoO8D+yV5z4XAhQAHHpizuzyKiBS9tIq0ufufgT9nsgIz2wu4H5ji7p+YWfxy3cwSFgty95nATAhq9WSybhER2V06o3rODE/EbjKzT8zsUzP7JJ2Fm1kFQdKf6+4PhJM/MLNe4eu9gPWZBi8iIm2XzsndnxMUZevm7vu4+97uvk9rb7Kgaf9bYJW7x9ctfhCYGD6eCCxoa9AiIpK5dLp6PnD3VRksexTwL8AKM1seTvu/wPXAPDM7H3gL0JkyEZEcSifx15rZvcB8YGtsYlzXTULu/jTBjdkTGZNkuoiIRCydxL8P8BkwNm6aAykTv4iIFKZ0hnN+OxeBiIhIbqRzB64uwPnA4UCX2HR3/06EcYmISETSGdVzN7A/cALwJNAXUIVOEZEOKp3Ef4i7/zuwxd3vBE4BRkYbloiIRCWdxN8Y/q43s8FAN+CL0YUkIiJRSmdUz8ywdPIPCS6+2gv490ijEhGRyKST+BeHNfifAg4CMLP+kUYlIiKRSaer5/4E0+7LdiAiIpIbSVv8ZjaQYAhnNzM7M+6lfYgb1ikiIh1Lqq6eAcA4oAo4NW76p8AFUQYlIiLRSZr43X0BsMDMjnH3Z3MYk4iIRCidPv6vmdk+ZlZhZovN7EMz+1bkkYmISCTSSfxj3f0Tgm6fNcAhwFVRBiUiItFJJ/FXhL9PAf7o7psijEdERCKWzjj+h8zsFaABuMjMegKfRxuWiIhEpdUWv7tPBf4RqHH3RmALcHrUgYmISDTSafEDDAT6mVn8/HdFEI+IiEQsnXr8dwMHA8uBHeFkR4lfRKRDSqfFXwMMcnePOhgREYleOqN6XiK4EYuIiBSBdFr8XwBeNrPnga2xie5+WmRRiYhIZNJJ/NOiDkJERHKn1cTv7k/mIhAREcmNVGWZPyUYvbPbS4C7+z6RRSUiIpFJVZ1z71wGIiIiuZHOqB4RESkiSvwiIiVGiV9EpMQo8YuIlBglfhGREqPELyJSYpT4RURKTLr1+DukKY9MYfn7y/MdhohIRqr3r2bGiTOyvtzIWvxmdoeZrTezl+Km9TCzR83s1fB396jWLyIiiVlUZfbN7J+AzcBd7j44nPZz4GN3v97MpgLd3f3q1pZVU1PjtbW1kcQpIlKszGypu9e0nB5Zi9/dnwI+bjH5dODO8PGdwBlRrV9ERBLL9cnd/dz9vfDx+8B+yWY0swvNrNbMaj/88MPcRCciUgLyNqonvJVj0n4md5/p7jXuXtOzZ88cRiYiUtxynfg/MLNeAOHv9Tlev4hIyct14n8QmBg+nggsyPH6RURKXmTj+M3sHmA08AUzWwtcB1wPzDOz84G3gAlRrV9ESsf8Zeu4YdFq3q1voHdVJVedMIAzhvXJd1gFK7LE7+7nJHlpTFTrFJHSM3/ZOq55YAUNjTsAWFffwDUPrABQ8k9CJRtEpEO7YdHqpqQf09C4gxsWrc5TRIVPiV9EOrR36xvaNF2U+EWkg+tdVdmm6aLELyId3FUnDKCyorzZtMqKcq46YUCeIip8RV2dU0SKX+wErkb1pE+JX0Q6vDOG9VGibwN19YiIlBglfhGREqPELyJSYpT4RURKjBK/iEiJUeIXESkxGs4pkkeqKin5oMQvkieqKin5oq4ekTxRVUnJFyV+kTxRVUnJFyV+kTxRVUnJFyV+kTzJtKrk/GXrGHX9X+g/9b8Zdf1fmL9sXZRhShHSyV2RPMmkqqROCEs2KPGL5FFbq0qmOiGsxC/pUuKXgqOx7cnphLBkg/r4paDEujLW1Tfg7OrKUD92QCeEJRuU+KWgaGx7arrNoGSDunqkoBRLV0ZU3VW6zaBkgxK/FJTeVZWsS5DkO1JXRtQjb3SbQWkvdfVIQSmGrgx1V0mhU4tfCkoxdGUUS3eVFC8lfik4Hb0roxi6q6S4qatHJMuSdVcdP7CnSi1IQVCLX0pWLkfeHD+wJ/cvXadSC3mgCwJ3Z+6e7xhaVVNT47W1tfkOQ4pIy5E3ELTKf3bmkEiSwqjr/5Kw+6dPVSXPTP1K1tcngVwf50JjZkvdvabldLX4I6SWRuGKquZNsmOuE775kc3jXEz/z0r8EcnmWO5i+oMrFFEk4lTHPNUJ30I9voUaV1tk6zgXW1VUJf6IZKulUWx/cIUiWyNv4pNjmRk7WnSdxo75VScMSNjlcPzAngV5fHP9dxfVh0y2jnOxVUUt6j7++cvWMe3BldQ3NEYQVf6cVvY0P9hjHr3tI971L/Dz7RN4cOexOV1PrmLINL58KcSYpOMbdXAP5l5wTJvfl6yPPy+J38xOBG4ByoHfuPv1qebPJPGfO+tZnnn948yDLFCnlT3N9RW/YU/b1jTtM+/E1MbvZjXBpFoPkJMYMo0vX4m2EGOS4pFJ8k+W+HM+jt/MyoHbgJOAQcA5ZjYom+v44fwVRZn0AX6wx7xmiQVgT9vGD/aYl7P15CqGTOPLl0KMSYpHNnNaPvr4RwCvufsbAGb2B+B04OVsreCeJe9ka1EFp7d9lGT6hhyuJ/G3xGzHkEqu9kNbFGJMIonkI/H3AeIz81pgZMuZzOxC4MLw6WYzS7vCVaf9DzmqXREWsGr7nE5s3236NvbgPZ+Sk/UAOYkhlVzth7YoxJikuNj0cUvb+JYvJZpYsKN63H0mMLO9yzGz2kR9XMVM21watOuERBcAAAgoSURBVM2lIYptzketnnXAAXHP+4bTREQkB/KR+F8ADjWz/mbWCfgG8GAe4hARKUk57+px9+1mdgmwiGA45x3uvjLCVba7u6gD0jaXBm1zacj6NneIC7hERCR7VI9fRKTEKPGLiJSYok78Znaima02s9fMbGq+48kGMzvAzB43s5fNbKWZXRZO72Fmj5rZq+Hv7uF0M7Nbw31QZ2ZH5ncLMmdm5Wa2zMwWhs/7m9mScNvuDQcLYGadw+evha/3y2fcmTKzKjO7z8xeMbNVZnZMsR9nM7s8/Lt+yczuMbMuxXaczewOM1tvZi/FTWvzcTWzieH8r5rZxLbEULSJPxelIfJkO3Cluw8CjgYuDrdrKrDY3Q8FFofPIdj+Q8OfC4Hbcx9y1lwGrIp7Ph242d0PATYC54fTzwc2htNvDufriG4BHnH3gcARBNtetMfZzPoAlwI17j6YYPDHNyi+4zwHOLHFtDYdVzPrAVxHcPHrCOC62IdFWty9KH+AY4BFcc+vAa7Jd1wRbOcC4J+B1UCvcFovYHX4+NfAOXHzN83XkX4IrvdYDHwFWAgY8BGwR8vjTTBi7Jjw8R7hfJbvbWjj9nYD3mwZdzEfZ3Zd1d8jPG4LgROK8TgD/YCXMj2uwDnAr+OmN5uvtZ+ibfGTuDRExyucnUL41XYYsATYz93fC196H9gvfFws+2EG8ANgZ/h8X6De3WM1EuK3q2mbw9c3hfN3JP2BD4HZYffWb8ysK0V8nN19HXAj8DbwHsFxW0pxH+eYth7Xdh3vYk78Rc3M9gLuB6a4+yfxr3nQBCiacbpmNg5Y7+5trVPSke0BHAnc7u7DgC3s+voPFOVx7k5QsLE/0Bvoyu5dIkUvF8e1mBN/0ZaGMLMKgqQ/190fCCd/YGa9wtd7AevD6cWwH0YBp5nZGuAPBN09twBVZha7CDF+u5q2OXy9G9DRSmSuBda6+5Lw+X0EHwTFfJy/Crzp7h+6eyPwAMGxL+bjHNPW49qu413Mib8oS0OYmQG/BVa5+01xLz0IxM7sTyTo+49NPy8cHXA0sCnuK2WH4O7XuHtfd+9HcBz/4u7nAo8D48PZWm5zbF+MD+fvUC1jd38feMfMBoSTxhCULi/a40zQxXO0me0Z/p3Htrloj3Octh7XRcBYM+seflMaG05LT75PckR8AuVk4H+B14F/y3c8WdqmYwm+BtYBy8Ofkwn6NhcDrwKPAT3C+Y1gdNPrwAqCERN53452bP9oYGH4+CDgeeA14I9A53B6l/D5a+HrB+U77gy3tRqoDY/1fKB7sR9n4EfAK8BLwN1A52I7zsA9BOcwGgm+2Z2fyXEFvhNu+2vAt9sSg0o2iIiUmGLu6hERkQSU+EVESowSv4hIiVHiFxEpMUr8IiIlRolfcs7M3Mx+Eff8X81sWpaWPcfMxrc+Z7vX8/WwYubjLab3i6+62MZlTjKz3tmJUCQ5JX7Jh63AmWb2hXwHEi/u6tB0nA9c4O7HZzGESQSlCkQipcQv+bCd4D6il7d8oWWL3cw2h79Hm9mTZrbAzN4ws+vN7Fwze97MVpjZwXGL+aqZ1ZrZ/4Z1fmK1/G8wsxfCuubfi1vuX83sQYKrRFvGc064/JfMbHo47VqCC+l+a2Y3JNvIsAX/gJk9EtZM/3lcLHPCZa6woAb9eKAGmGtmy82s0syuDeN9ycxmhlezYmZPmNn0cNv/18y+HLfcG8P568zs++H0o8J9t9TMFsWVBrjUgvs61JnZH9I9eFIE8n0Vm35K7wfYDOwDrCGor/KvwLTwtTnA+Ph5w9+jgXqCkrSdCeqS/Ch87TJgRtz7HyFo1BxKcGVkF4Ja5j8M5+lMcEVs/3C5W4D+CeLsTVBGoCdB0bS/AGeErz1BgqtjiSu3S9CCfyPcxi7AWwT1VY4CHo17T1WiZRJevRk+vhs4NW6+X4SPTwYeCx9fRFDTJ1bCuAdQAfwN6BlOOxu4I3z8Lruugq3K99+FfnL3oxa/5IUHFUXvIrjxRrpecPf33H0rwSXs/xNOX0GQcGPmuftOd3+VIPEOJKhlcp6ZLScoY70vwQcDwPPu/maC9Q0HnvCgaNh2YC7wT22IF4Kba2xy988JvlF8KYzpIDP7pZmdCHyS5L3HW3BnqRUEhekOj3stVpxvKbu2/asENdq3A7j7x8AAYDDwaLjtPyQo6AVBKYi5ZvYtgm9hUiLa0qcpkm0zgL8Ds+OmbSfsgjSzMqBT3Gtb4x7vjHu+k+Z/yy3rkDhBzZPvu3uzQlZmNpqgxR+V+Jh3ELTGN5rZEQQ3GZkMTCCouxIfVxfgvwi+AbwTnvzukmC5O0j9f2zASnc/JsFrpxB8kJ0K/JuZDfFdde+liKnFL3kTtkjnsetWehB0/xwVPj6NoKuirb5uZmVhv/9BBHctWgRcZEFJa8zsMAtubJLK88BxZvYFC27leQ7wZAbxNBOe1C5z9/sJWuCx+6h+CuwdPo4l+Y8suPdCOiOVHgW+FztJbcHt+VYDPc3smHBahZkdHn6oHuDujwNXE3RH7dXebZOOQS1+ybdfAJfEPZ8FLDCzFwn66jNpjb9NkLT3ASa7++dm9huCLpG/hydJPwTOSLUQd3/PzKYSlAU24L/dfUGq96SpD8GdtWINr2vC33OAX5lZA8EtBmcRVKl8n6DMeGt+AxwG1JlZIzDL3f8zPHF8q5l1I/ifn0FQtfZ34TQDbnX3+ixsm3QAqs4pIlJi1NUjIlJilPhFREqMEr+ISIlR4hcRKTFK/CIiJUaJX0SkxCjxi4iUmP8PkvhhUIeL7vsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_instance_score(od_preds, y_outlier, labels, od.threshold, ylim=(0,50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the ROC curve for the outlier scores of the detector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAFNCAYAAABbpPhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXhU1f3H8fcXCCCLIIpVQSUhGCOyuIEsyqIgi0uxFrDuWhRbXAoqaG211lYTUBRRZFEpVkRkMwiKaxAp/GSHsAiBoKAoCgJZgATy/f2RkaYIYQiZzEzyeT3PPM69c+beTzxP4Ms5995j7o6IiIiIRJYK4Q4gIiIiIr+kIk1EREQkAqlIExEREYlAKtJEREREIpCKNBEREZEIpCJNREREJAKpSBMRERGJQCrSRCRimNlGM9ttZllm9p2ZjTWzGge1aW1mn5hZppntNLPpZnbOQW2ON7PnzOzrwLHWB7ZPKt2fSESk+FSkiUikucrdawDNgfOAh3/+wMxaAR8A7wCnAbHAMmCumcUF2lQGPgYaA12A44FWwDagRahCm1mlUB1bRMonFWkiEpHc/TtgFgXF2s+SgXHu/ry7Z7r7dnd/FJgPPB5oczNwBtDD3Ve5e767b3X3v7v7zEOdy8wam9mHZrbdzL43s0cC+8ea2ZOF2rU3s82Ftjea2UAzWw5kB95POujYz5vZsMD7Wmb2ipltMbNvzOxJM6t4jP+rRKSMUpEmIhHJzOoDXYH0wHY1oDXw9iGaTwQ6Bd5fDrzv7llBnqcm8BHwPgWjc/EUjMQF63qgO1AbmAB0CxyTQAHWExgfaDsW2Bc4x3lAZ+D3R3EuESlHVKSJSKSZZmaZwCZgK/BYYH8dCv7M2nKI72wBfr7e7MTDtDmcK4Hv3P0Zd98TGKH7v6P4/jB33+Tuu939K2Ax0CPwWUcgx93nm9mvgG7A/e6e7e5bgaFA76M4l4iUIyrSRCTS/NrdawLtgbP5b/H1E5APnHqI75wK/Bh4v+0wbQ7ndGB9sZIW2HTQ9ngKRtcAfsd/R9HOBGKALWa2w8x2ACOBk4/h3CJShqlIE5GI5O6zKZgeHBLYzgbmAb89RPOe/HeK8iPgCjOrHuSpNgFxh/ksG6hWaPuUQ0U9aPttoH1gurYH/y3SNgF7gZPcvXbgdby7Nw4yp4iUMyrSRCSSPQd0MrNmge1BwC1mdq+Z1TSzEwIX9rcC/hZo8zoFBdFkMzvbzCqY2Ylm9oiZdTvEOd4FTjWz+82sSuC4LQOfLaXgGrM6ZnYKcP+RArv7D0Aq8BqQ4e6rA/u3UHBn6jOBR4RUMLOGZtauGP9fRKQcUJEmIhErUPCMA/4a2P4cuAK4loLrzr6i4AL8tu6+LtBmLwU3D6wBPgR2AV9QMG36i2vN3D2TgpsOrgK+A9YBHQIfv07BIz42UlBgvRVk9PGBDOMP2n8zUBlYRcH07SSObmpWRMoRcz94pF5EREREwk0jaSIiIiIRSEWaiIiISARSkSYiIiISgVSkiYiIiEQgFWkiIiIiEahSuAMcrdq1a3t8fHy4Y0gxZWdnU716sM8YlUiivotu6r/opb6LbosWLfrR3esW57tRV6T96le/YuHCheGOIcWUmppK+/btwx1DikF9F93Uf9FLfRfdzOyr4n5X050iIiIiEUhFmoiIiEgEUpEmIiIiEoFUpImIiIhEIBVpIiIiIhFIRZqIiIhIBFKRJiIiIhKBQlakmdmrZrbVzNIO87mZ2TAzSzez5WZ2fqiyiIiIiESbUI6kjQW6FPF5V6BR4HUnMCKEWURERESiSsiKNHf/DNheRJNrgHFeYD5Q28xODVUeERERkWgSzmvS6gGbCm1vDuwTERERiXpr1649pu9HxdqdZnYnBVOi1K1bl9TU1PAGkmLLyspS/0WprKwsGv9lBtl54U4ixfb+jHAnkOJS30WN/Lw9ZK/4iMqnnU2lWr86pmOFs0j7Bji90Hb9wL5fcPdRwCiAhIQE10Kz0UsLBUev1NRUsvOy2fh093BHkWLQ7170Ut9Fh+zsbJ599lmGDx/OJa1a8eQDf+Tcc8/Fhl1f7GOGs0hLAfqZ2QSgJbDT3beEMY+IiIjIUfn666/ZsWMHZ511Fj/88AOpqakkJiaWyLFDVqSZ2ZtAe+AkM9sMPAbEALj7y8BMoBuQDuQAt4Uqi4gUT7O/fcDO3f+d36x1XEwY04iIRI60tDSSk5OZMWMG//znP2natCnDhg0r0XOErEhz9yLH99zdgT+G6vwicux27s47ML2pKRcRKe/cnW3btnHiiSdy77330qlTJ4YNG0bt2rVDcr6ouHFARIp28IhXSdHImYgI5Ofnk5KSQlJSEnXq1GHGjBl88sknIT+vijSRMqDwiJeIiJQMd8fMuPXWW1m9ejUDBw6kR48epXZ+FWkiIiIihezcuZORI0fy1ltvMX/+fJ5//nlq166NmZVqDhVpElVCNa0X7TQtKSJSMt544w3uvfdeunTpwquvvkpMTAwnnHBCWLKoSJOoomk9EREpaWvXruW5557j73//Oy1atGDhwoXExsaGO5aKtFDTyM8hHMOTszViJCIiJWXt2rU8/PDDzJkzhz/84Q9UqlSJRo0ahTvWASrSQkwjP/9Lj3EQEZFwcndmzZrFOeecQ4UKFWjXrh3jxo2jevXq4Y72C+FcYF1ERESkVOTl5fHGG2/QvHlzHnroIb777jvi4+O59957I7JAgygdSYumKURNz4mIiIRPdnY2u3btolKlSowbN46nn36aLl26lPqdmsURlUWaphBFRESkKD/++CPDhw/npZdeYtCgQfTv359Zs2aFO9ZRicoiTURERORQ9u/fD0Dr1q1p164dc+bMISEhIcypikdFmoiIiES9ZcuWkZycTE5ODlOnTmXFihVUqVIl3LGOiW4cEBERkajWt29funbtSrNmzRg7dixA1BdooCJNREREosz+/fuZPHkyN910E+7OfffdR0ZGBg899BC1atUKd7wSoyJNREREokZKSgqJiYkMHjyYa6+9FncnMTGxTIycHUxFmoiIiES0HTt2MHToUHJzc6lTpw6jR49m3rx59OjRgwoVym4pU3Z/MhEREYlqW7Zs4cEHH6Rhw4YsXryYXbt20bZtW9q1axcVzzk7Vrq7U0RERCLKmjVrOPXUU9mwYQO5ubksXryYM888M9yxSp2KNBEREYkI8+bNIykpiXnz5jF16lTatGlDmzZtwh0rbDTdKSIiImHj7uzevZuMjAxuvPFGOnXqREZGBq1btw53tLBTkSYiIiKlLi8vj3HjxtG0aVNeeOEFYmNjWbduHX/84x+pVq1auONFBE13ioiISKnKy8ujcePGnH766TzzzDN06tQJoEzfqVkcKtJEREQk5LZu3coLL7zArl27eP755/noo48444wzwh0roqlkFRERkZB65JFHOPvss/nhhx+45557AFSgBUFFmoiIiJS4JUuW8PTTTwPQoUMHVq1axcsvv0x8fHyYk0UPFWkiIiJSYj7//HM6d+7MVVddRUxMDO5Op06dOOWUU8IdLeqoSBMREZFjsn//fmbOnIm7s2nTJn73u9+xYcMGBgwYUC5WBggV3TggIiIixbJ7927Gjh3LkCFDOOWUU7j44ou5/vrrwx2rzFCRJiIiIkflp59+okaNGkyfPp333nuPcePGleuVAUJF050iIiISlE2bNjFgwAAaNmzIwoUL6dmzJykpKSrQQkRFmoiIiBTJ3Vm1ahXNmjUDYNmyZbRq1SrMqco+TXeKiIjIIX3++eckJydzxRVX8Ic//IENGzZQu3btcMcqN1SkiYiIyP/YvXs3nTp14rvvvuOBBx7glltuwcxUoJUyFWkiIiJCbm4u48ePJzs7mz/+8Y88+eSTXHLJJVSsWDHc0cotXZMmIiJSzg0bNoyGDRsyfvx4GjduDED79u1VoIWZRtJERETKoe+//565c+dy7bXXUrlyZd555x3OP//8cMeSQjSSJiIiUo6kp6fTt29fEhMTmTNnDu5O3759VaBFII2kiYiIlAPp6enEx8czYcIE6taty5o1azj55JPDHUuKoCJNRESkjHJ3PvjgA5KSkli/fj1paWk8+uij4Y4lQdJ0p4iISBmzb98+3J1XXnmFAQMGcOutt5Kenk7NmjXDHU2Ogoo0ERGRMiInJ4fhw4fTqFEjFi9ezM0338yKFSu4+eabiYmJCXc8OUoq0kRERMqA5cuXExsby8cff8z48eO54IILqFy5MmYW7mhSTLomTUREJEp99dVXDB06lI4dO9KlSxdmz57N2WefHe5YUkJCOpJmZl3M7EszSzezQYf4/Awz+9TMlpjZcjPrFso8IiIiZUFmZiY33XQT559/PjExMVx44YVUrlxZBVoZE7KRNDOrCLwIdAI2AwvMLMXdVxVq9igw0d1HmNk5wEygQagyiYiIRCt357PPPiM7O5uuXbvSpk0bXnjhBa2nWYaFciStBZDu7hvcPReYAFxzUBsHjg+8rwV8G8I8IiIiUcfdmTp1Kq1ateL3v/892dnZmBl9+/ZVgVbGhfKatHrApkLbm4GWB7V5HPjAzO4BqgOXhzCPiIhI1Ni7dy+rVq3CzEhNTeXBBx/k17/+tdbTLEfCfePA9cBYd3/GzFoBr5vZue6eX7iRmd0J3AlQt25dqgGpqamlHlaOXVZWlvouSqnvopv6L3pkZWUxffp0pkyZQrNmzbj//vvp0aMHAHPmzAlzOilNoSzSvgFOL7RdP7CvsDuALgDuPs/MqgInAVsLN3L3UcAogISEBN8LtG/fPjSpJaRSU1PVd1FKfRfd1H+RLzs7m+rVq9OnTx9ycnL46KOPaNasmfquHAtlkbYAaGRmsRQUZ72B3x3U5mvgMmCsmSUCVYEfQphJREQkonz55ZcMGTKEmTNnkp6ezsiRI6lQQY8xlRDeOODu+4B+wCxgNQV3ca40syfM7OpAswFAHzNbBrwJ3OruHqpMIiIikeSVV17hkksuoV69eixbtozjjjtOBZocENJr0tx9JgWP1Si876+F3q8C2oQyg4iISKRwd95//30GDx7MyJEj6dGjB71796Z69erhjiYRKNw3DoiIiJQLK1as4IYbbgBg4MCBNGjQQOtpSpFUpImIiIRIdnY2r7zyCq1btyYuLo6kpCS6dOmi9TQlKJr4FhERKWFZWVk89thjxMbGMnv2bKpVq0adOnXo2rWrCjQJmkbSRERESsjGjRvZuXMnCQkJ7Ny5kzlz5pCQkBDuWBKlNJImIiJyjJYtW8YNN9zABRdcwBdffEHVqlV57rnnVKDJMVGRJiIiUgzuztatW3F3BgwYQPPmzdmwYQN9+vQJdzQpIzTdKSIichT279/P1KlTSU5O5le/+hXTp0/no48+CncsKYNUpImIiATB3TEzbr75ZjZs2MAjjzzC1VdffeQvihSTijQREZEi7NixgxEjRjBx4kS++OILXnrpJY4//njdpSkhp2vSREREDuPf//43DRs2ZPXq1YwbN46YmBhq1aqlAk1KhYo0ERGRQlavXs1dd93F9u3bufjii1myZAnjxo2jSZMm4Y4m5YyKNBEREeDLL7/k17/+Ne3bt6d+/fpUqlSJ+Ph4zjjjjHBHk3JK16SJiEi5lZ+fz8yZM2nSpAkVK1akU6dOjB8/nmrVqoU7moiKNBERKX/y8vJ48803SU5OJiYmhtGjR3PhhRcSHx8f7mgiB6hIExGRciMrK4tdu3YRExPDhAkTePbZZ+nUqZNuBJCIpGvSRESkzNu6dSt/+ctfiI2N5e2336Zu3brMnDmTzp07q0CTiKWRNBERKbP27duHmXHJJZfQoUMH5s2bpylNiRoq0kREpMxZvHgxycnJ7N27l6lTp7JixQoqV64c7lgiR0XTnSIiUqb06dOHq6++mosuuohx48YBqECTqKQiTUREotq+fft46623+N3vfoe788ADD7BhwwYGDBhAzZo1wx1PpNhUpImISNSaNm0aCQkJDBs2jN69ewOQkJCgkTMpE4Iq0sysspnpSksREQm77du3M2TIEHJzc6lbty7jxo1j7ty5XH311bpTU8qUIxZpZtYdWAF8GNhubmZTQx1MRESksG+//Zb+/fsTHx/PypUryczMpE2bNrRp0ybc0URCIpi7O58AWgKfArj7Uo2qiYhIaVm5ciX169cnIyMDM2P58uXUr18/3LFEQi6YIi3P3XccNITsIcojIiICwOeff05SUhILFixg6tSpGjWTcieYa9JWm1lPoIKZxZrZUGB+iHOJiEg5lJ+fT05ODhs2bOC2226je/fuZGRk0KpVq3BHEyl1wRRp/YALgHxgCrAXuC+UoUREpHzJzc3ltdde49xzz+Wll14iLi6OL7/8kr59+3LccceFO55IWAQz3XmFuw8EBv68w8yupaBgExEROSZ5eXmcc845xMXF8cILL9CxY0cAKlTQU6KkfAumSHuUXxZkfz7EPhERkaB8//33PP/882RlZTFs2DBSU1N1M4DIQQ77zxQzuyJw/Vk9M3u20GsMBVOfIiIiR23QoEEkJiayc+dO7r//fgAVaCKHUNRI2lYgDdgDrCy0PxMYFMpQIiJStixcuJAPPviARx55hMsvv5z+/ftz8sknhzuWSEQ7bJHm7kuAJWb2hrvvKcVMIiJSRnz22Wf87W9/Y926dQwYMAB35/LLLw93LJGoEMxVmfXMbIKZLTeztT+/Qp5MRESi0r59+5g+fTruzrfffsstt9zC+vXrue+++7Rsk8hRCObGgbHAk8AQoCtwG3qYrYiIHCQnJ4dXX32VZ555htNPP502bdocWPRcRI5eMCNp1dx9FoC7r3f3Ryko1kRERNi2bRt5eXm8++67fPTRR4wfP57PPvuMOnXqhDuaSFQLpkjba2YVgPVm1tfMrgJqhjiXiIhEuK+++or77ruPRo0asWjRInr27Mm0adO0OoBICQmmSPsTUB24F2gD9AFuD2UoERGJXO7OypUrOf/886lSpQppaWlcfPHF4Y4lUuYc8Zo0d/+/wNtM4CYAM6sXylAiIhJZ3J3PPvuMpKQkunfvzh/+8Ac2bNhArVq1wh1NpMwqskgzs4uAesDn7v6jmTWmYHmojoCePCgiUg7k5ORw2WWXsW3bNh588EFuuukmzEwFmkiIHbZIM7OngN8Ay4BHzexd4A9AEtC3dOKJiEg47N27l9dff509e/bQr18/kpKSaNOmDRUrVgx3NJFyo6hr0q4Bmrn7b4HOwIPAxe7+jLvnlEo6EREpdUOHDiU2NpbJkyfTrFkzAC699FIVaCKlrKjpzj3uvhvA3beb2Vp331BKuUREpBRt2bKFuXPnct1111GtWjXee++9AwWaiIRHUSNpcWY2JfCaCsQW2p5SWgFFRCR0vvzyS/r06UPjxo2ZP38+AHfddZcKNJEIUNRI2m8O2h5+tAc3sy7A80BFYIy7P32INj2BxylYxWCZu//uaM8jIiJHZ+3atZx11llMmjSJevXqsXbtWk466aRwxxKRQopaYP3jYzmwmVUEXgQ6AZuBBWaW4u6rCrVpBDwMtHH3n8zs5GM5p4iIHJ67895775GcnMxXX33F8uXL+fOf/xzuWCJyGME8zLa4WgDp7r7B3XOBCRTcjFBYH+BFd/8JwN23hjCPiEi5tG/fPtydMWPGMGjQIPr06cPatWupWVOLx4hEsmAWWC+uesCmQtubgZYHtTkLwMzmUjAl+ri7v3/wgczsTuBOgLp161INSE1NDUFkCbWsrCz1XZRS30Wf3bt3M2PGDCZNmsTDDz9M48aNef755zEz5s6dG+54EiT97pVfQRdpZlbF3feG4PyNgPYUPBz3MzNr4u47Cjdy91HAKICEhATfC7Rv376Eo0hpSE1NVd9FKfVddFm2bBm//e1vufTSS0lJSSEnJ0f9F6X0u1d+HXG608xamNkKYF1gu5mZvRDEsb8BTi+0XT+wr7DNQIq757l7BrCWgqJNRESOUkZGBvfccw/vvPMOiYmJzJ07l8mTJ9OiRYtwRxORYgjmmrRhwJXANgB3XwZ0COJ7C4BGZhZrZpWB3kDKQW2mUTCKhpmdRMH0p57FJiJyFHbt2sXvfvc7LrzwQqpXr06LFi2oXLkyZ511VrijicgxCGa6s4K7f2VmhfftP9KX3H2fmfUDZlFwvdmr7r7SzJ4AFrp7SuCzzma2KnDMB91921H/FCIi5Yy7k5qaSk5ODt26daN9+/aMGDFC62mKlCHBFGmbzKwF4IHHatxDwbTkEbn7TGDmQfv+Wui9A/0DLxEROQJ3Z/LkySQlJZGZmck//vEPzIw777wz3NFEpIQFU6TdTcGU5xnA98BHgX0iIlJK9uzZw8qVK7nggguYO3cuf/7zn7n66qupUCGUT1ISkXAKpkjb5+69Q55ERER+YceOHYwYMYJhw4bRqVMnxo0bx9ChQ8MdS0RKQTBF2gIz+xJ4C5ji7pkhziQiUu5lZWVRo0YNHnjgAXJzc/nggw9o0qRJuGOJSCk6YpHm7g3NrDUFd2f+zcyWAhPcfULI04mIlDOrV69m8ODBzJo1i/Xr1zN69GgOunFLRMqJoC5mcPf/uPu9wPnALuCNkKYSESmHRo8eTfv27YmNjWX58uVUrVpVBZpIORbMw2xrmNkNZjYd+AL4AWgd8mQiImVcfn4+7777Lu3atWPdunVcd911ZGRk8Je//IUTTzwx3PFEJMyCuSYtDZgOJLv7nBDnEREpF5YtW8YNN9xATEwMAwcOJDY2lkqVQrmcsohEm2D+RIhz9/yQJxERKeOysrIYPXo0l1xyCXFxcTz77LN06tRJU5oickiHne40s2cCbyeb2ZSDX6WUT0Qk6mVmZvLoo48SGxvL/PnzqVGjBnXq1KFz584q0ETksIoaSXsr8N/hpRFERKSs2bBhA7t27eLss89mz549zJs3j/j4+HDHEpEocdiRNHf/IvA20d0/LvwCEksnnohI9Fm8eDG9e/emRYsWLFq0iKpVqzJkyBAVaCJyVIJ5BMfth9h3R0kHERGJZu7O999/j7szaNAgWrRoQUZGBnfcoT8uRaR4DjvdaWa9KHiAbexB16DVBHaEOpiISDTYt28fkydPJjk5mfr16/POO+/wwQcfhDuWiJQBRV2T9gWwDagPvFhofyawJJShREQinbtjZtx0001s2rSJxx9/nO7du4c7loiUIYct0tw9A8gAPiq9OCIikW379u289NJLTJo0iQULFjBy5EiOP/74cMcSkTKoqEdwzA789ycz217o9ZOZbS+9iCIikWHcuHHEx8ezfv16xo8fT0xMjAo0EQmZoqY7OwT+e1JpBBERiUQrV65k6NChJCcn06ZNG5YvX079+vXDHUtEyoGiHsHx8yoDpwMV3X0/0Aq4C6heCtlERMJmzZo1XHXVVVx22WXExcVRqVIlGjZsqAJNREpNMMtCTQMuMrOGwGvAu8B44MpQBhMRKW0/L3jerFkzYmJiuPLKK5k4cSLHHXdcuKOJSDkUTJGW7+55ZnYt8IK7DzMz3d0pImVGbm4ub7zxBoMHD6ZatWqMHDmSCy64gIYNG4Y7moiUY8EUafvM7LfATcCvA/tiQhdJRKR07Nq1i8zMTCpXrsyUKVN44YUX6Nixo9bTFJGIEOyKAx2AZHffYGaxwJuhjSUiEjrfffcdjzzyCHFxcUyZMoW6desyffp0LrvsMhVoIhIxjjiS5u5pZnYvEG9mZwPp7v6P0EcTESlZeXl5VKhQgXbt2tGpUycWLFhAbGxsuGOJiBzSEYs0M7sEeB34BjDgFDO7yd3nhjqciEhJWLBgAUlJSeTn5zNlyhTS0tKIidFVGyIS2YKZ7hwKdHP3Nu7eGugOPB/aWCIiJeOOO+7guuuu45JLLmHcuHEAKtBEJCoEU6RVdvdVP2+4+2qgcugiiYgU3759+xg/fjy9e/fG3Rk0aBDp6encd9991KhRI9zxRESCFkyRttjMXjaztoHXCLTAuohEoClTptCoUSNGjhzJTTfdBECjRo00ciYiUSmYIq0vsAF4KPDaQMGqAyIiYbdt2zaSkpLIzc3llFNO4c0332T27Nl0795dd2qKSFQr8sYBM2sCNASmunty6UQSETmyzZs3M3jwYF5//XWuvfZasrKyaN26dbhjiYiUmMMWaWb2CHAHsJiCZaGecPdXSy2ZiMghLF++nDPPPJOvv/6aKlWqkJaWxmmnnRbuWCIiJa6okbQbgKbunm1mdYGZgIo0ESl17s5nn31GUlISS5YsYerUqbRu3VojZyJSphV1Tdped88GcPcfjtBWRKTE7d+/n+zsbDZs2MBdd91Fjx49yMjI4OKLLw53NBGRkCtqJC3OzKYE3hvQsNA27n5tSJOJSLm1d+9exo0bx5AhQ7jrrrvo378/q1ev1o0AIlKuFFWk/eag7eGhDCIiApCbm0tiYiIJCQmMHDmSdu3aAahAE5Fy57BFmrt/XJpBRKT8+vbbb3n++efJzs5m+PDhfP7557oZQETKPV1nJiJh9eCDD3Luueeye/duHnjgAQAVaCIiqEgTkTCYP38+f//73wHo2rUra9euZdiwYTRo0CC8wUREIkjQRZqZVQllEBEp+1JTU2nXrh29e/emdu3auDsdO3bkpJNOCnc0EZGIc8QizcxamNkKYF1gu5mZvRDyZCJSJuTl5TFt2jTcne+//5677rqLdevWcc899+hmABGRIhS5LFTAMOBKYBqAuy8zsw4hTSUiUS87O5sxY8bw7LPPEhcXR7t27ejVq1e4Y4mIRI1girQK7v7VQf/i3R+iPCIS5X744QdOOOEEZsyYwZw5c3j77bdp0aJFuGOJiESdYK5J22RmLQA3s4pmdj+wNsS5RCTKZGRk0K9fP8466ywWLVpEz549mTRpkgo0EZFiCqZIuxvoD5wBfA9cHNh3RGbWxcy+NLN0MxtURLvfmJmb2YXBHFdEIoe7k5aWxkUXXUTNmjVZtWoVLVu2DHcsEZGod8TpTnffCvQ+2gObWUXgRaATsBlYYGYp7r7qoHY1gfuA/zvac4hIeLg7n376KUlJSVxzzTXcfffdZGRkULNmzXBHExEpM45YpH1iiwsAAB9fSURBVJnZaMAP3u/udx7hqy2AdHffEDjOBOAaYNVB7f4OJAEPBhNYRMIrJyeH9u3bk5mZyYMPPsgNN9yAmalAExEpYcHcOPBRofdVgR7ApiC+V++gdpuB/5kDMbPzgdPdfYaZqUgTiVB79uwhJSWF5cuXc++99zJ06FBatWpFhQp6HraISKgEM935VuFtM3sd+PxYT2xmFYBngVuDaHsncCdA3bp1qUbBQzEl+mRlZanvosxbb73FxIkTiYuLIy4u7kD/ffbZZ+ENJkdFv3vRS31XfgUzknawWOBXQbT7Bji90Hb9wL6f1QTOBVIDj/c4BUgxs6vdfWHhA7n7KGAUQEJCgu8F2rdvX4zoEm6pqanquyjwzTffMHfuXHr27Mm6dev44x//yI8//qi+i2L63Yte6rvyK5gVB34ys+2B1w7gQ+DhII69AGhkZrFmVpmCmw9Sfv7Q3Xe6+0nu3sDdGwDzgV8UaCJSelavXs3tt99OkyZNWLx4MQB9+vTh3HPPDXMyEZHyp8iRNCsY4mrGf0fA8t39FzcRHIq77zOzfsAsoCLwqruvNLMngIXunlL0EUSktKxevZrExESmTZtGbGws6enp1KlTJ9yxRETKtSKLNHd3M5vp7sX6Z7S7zwRmHrTvr4dp27445xCR4snPz2fGjBkkJSXx3XffsXTpUh5+OJhBchERKQ3B3Jq11MzOC3kSESkVeXl5uDtjxozhscce45577mHNmjXUqFEj3NFERKSQwxZpZvbzKNt5FDyI9kszW2xmS8xscenEE5GSkpmZeWCx82XLlnHbbbexaNEievXqRaVKxbmHSEREQqmoP5m/AM4Hri6lLCISIkuWLKFz58507NiRadOm0bx583BHEhGRIyiqSDMAd19fSllEpAStX7+eZ555hs6dO9OtWzfmzZtHfHx8uGOJiEiQiirS6ppZ/8N96O7PhiCPiByjXbt20adPHz7++GP69u1L69atqVy5sgo0EZEoU1SRVhGoQWBETUQil7vz0UcfsWfPHq688ko6d+7MmDFjtJ6miEgUK6pI2+LuT5RaEhE5au7OxIkTSU5OZs+ePfzjH//AzLjjjjvCHU1ERI7REa9JE5HIs3v3btLS0rjooov44osvePzxx+nevbsWPBcRKUOK+hP9slJLISJB2b59O08++SSxsbGMGDECgGeeeYarrrpKBZqISBlz2JE0d99emkFE5PAyMzOpWbMmAwcOZP/+/XzyySecc8454Y4lIiIhpCdYikSwlStXkpyczMcff0x6ejqjRo2iYEldEREp6zQ/IhKhRo4cyWWXXUZCQgIrVqygatWqKtBERMoRFWkiESI/P5933nmHtm3bkp6eTs+ePcnIyOCRRx7hhBNOCHc8EREpZZruFIkAS5cu5frrr6d69eoMHDiQ2NhYKlasGO5YIiISRirSRMJk165djBo1inbt2hEfH8+LL75Ihw4dNKUpIiKApjtFSt2uXbt4+OGHiYuLY/HixRx//PGccMIJdOzYUQWaiIgcoJE0kVKybt06srKySExMZP/+/SxYsIDY2NhwxxIRkQilkTSREFuwYAG//e1vadOmDcuWLaNq1aokJyerQBMRkSKpSBMJAXdny5YtuDuPPvoobdu2ZcOGDdx6663hjiYiIlFC050iJWjfvn0HFjyPi4tjypQpzJo1K9yxREQkCqlIEykB7o6ZceONN7Jlyxb++c9/0rVr13DHEhGRKKYiTeQYbNu2jeHDhzN58mQWL17M6NGjqVmzZrhjiYhIGaBr0kSKaezYsTRq1IhNmzbx9ttvU6lSJRVoIiJSYlSkiRyF5cuXc9ttt7F9+3YuvfRS0tLSGDNmDAkJCeGOJiIiZYyKNJEgrF69mm7dutGlSxfOPvtsKleuTFxcHKeddlq4o4mISBmla9JEDmP//v288847XHDBBVSuXJkePXowZcoUqlatGu5oIiJSDqhIEznI3r17GTduHEOGDKF27dq8/PLLnHfeeTRs2DDc0UREpBxRkSYSsHPnTrKysqhcuTLvvvsuI0eOpF27dlpPU0REwkLXpEm59+233/LQQw8RFxfHtGnTqFu3Lu+88w7t27dXgSYiImGjkTQpt/Ly8qhQoQLt27ena9euLF68mDPPPDPcsURERAAVaVIOzZ8/n+TkZACmTJnCypUriYmJCXMqERGR/xV1051fZ+ZT6zj9hSrFc+utt3L99dfTsWNHXn/9dQAVaCIiEpGirkjLd1j2WOdwx5AokZeXx+uvv07Pnj3Jz8/n0UcfZd26dfTr14/q1auHO56IiMhhRV2RJhKst99+m4YNG/Laa69x++23Y2bEx8dTqZJm+UVEJPKpSJMy5YcffuCpp54iNzeX+vXrM2nSJD755BO6dOmiOzVFRCSqqEiTMuHrr7+mX79+JCQksHHjRrKzs2nVqhUtWrQIdzQREZFi0byPRLUlS5YQFxfH5s2bqVmzJqtWreKUU04JdywREZFjpiJNoo6788knn5CcnExaWhpTp06ldevWtG7dOtzRRERESoymOyVq7N+/n+zsbNavX88999xDr1692LBhg6Y0RUSkTFKRJhFv9+7dvPzyyyQkJDB69Gji4+NZuXIlt99+O1WqVAl3PBERkZDQdKdEtNzcXBITEzn33HN57bXXaNu2LYDu1BQRkTJPRZpEnM2bN/Pcc8+xe/duXnzxRebPn6+bAUREpNzRdKdElP79+9O0aVP279/PwIEDAVSgiYhIuRTSIs3MupjZl2aWbmaDDvF5fzNbZWbLzexjMzszlHkkMv3nP//hiSeeAOCqq64iPT2doUOHcsYZZ4Q5mYiISPiErEgzs4rAi0BX4BzgejM756BmS4AL3b0pMAlIDlUeiTwff/wxbdu25cYbb6Ru3bq4Ox06dKBOnTrhjiYiIhJ2oRxJawGku/sGd88FJgDXFG7g7p+6e05gcz5QP4R5JALk5eUxZcoU3J1t27bRr18/1q5dy913362bAURERAoJ5Y0D9YBNhbY3Ay2LaH8H8F4I80gYZWZmMnr0aJ5++mmaNWtGhw4d6NmzZ7hjiYiIRKyIuLvTzG4ELgTaHebzO4E7ASqfEk9qamrphZNj8tNPP1GzZk0+//xzZs+ezSOPPELz5s1ZtmxZuKPJUcrKytLvXhRT/0Uv9V35Ze4emgObtQIed/crAtsPA7j7Uwe1uxx4AWjn7luPdNwqpzbyvVvWhSCxlKT169czZMgQ3nrrLd5///0DqwKkpqbSvn378IaTYlHfRTf1X/RS30U3M1vk7hcW57uhvCZtAdDIzGLNrDLQG0gp3MDMzgNGAlcHU6BJ5HN3VqxYQcuWLTnxxBNZs2aNlm0SEREphpBNd7r7PjPrB8wCKgKvuvtKM3sCWOjuKcBgoAbwduCi8a/d/epQZZLQcHc++ugjkpOT6dGjB3fffTcbN26kRo0a4Y4mIiIStUJ6TZq7zwRmHrTvr4XeXx7K80vo5eTkcMkll7Bnzx4eeughrr/+esxMBZqIiMgxiogbByS67N69m9dee428vDzuu+8+hg8fTsuWLalQQQtYiIiIlBT9rSpHJSkpiQYNGjBr1ixatix4okqrVq1UoImIiJQwjaTJEW3atIn//Oc/9OrVi5NOOonU1FQSExPDHUtERKRM0/CHHFZaWhq33HLL/zzX7I477lCBJiIiUgpUpMn/cHdWrlwJwLvvvsvZZ59Neno6//znP8OcTEREpHzRdKcAkJ+fT0pKCsnJyfz4448sXryYQYMGhTuWiIhIuaWRtHIuNzcXd2fUqFH84x//oH///qxevVqP0BAREQkzFWnl1K5duxg8eDCxsbEsX76c3//+93zxxRdcd911VKxYMdzxREREyj0VaeXQ4sWLiYuLY8mSJcyYMYNmzZpRqVIlAqs+iIiISATQNWnlxNq1axkyZAhdu3ale/fuLFiwgNjY2HDHEhERkcPQSFoZt3PnTq677jratm3LqaeeStu2balcubIKNBERkQinkbQyyN2ZNWsWubm5XHXVVVx55ZX861//onr16uGOJiIiIkHSSFoZ4u688cYbNG/enIceeggAM+PWW29VgSYiIhJlNJJWBmRnZ5OWlkbLli1ZunQpTz31FF27dtWNACIiIlFMI2lR7Mcff+Txxx8nNjaW0aNHAzB48GC6deumAk1ERCTKaSQtCu3cuZNatWrx8MMPAzBnzhwSEhLCnEpERERKkoq0KLJs2TKSk5OZPXs26enpjBo1SiNmIiIiZZSmO6PEiBEj6Nq1K02bNmXlypVUrVpVBZqIiEgZpiItQu3fv5/JkyfTqlUr0tPTuf7668nIyGDgwIHUqlUr3PFEREQkxDTdGYGWLFlCr169OOGEExg4cCCxsbFaT1NERKScUZEWIXbs2MHLL7/MZZddRnx8PKNHj+bSSy/VlKaIiEg5penOMNu1axcPPfQQDRs2JC0tjdq1a3PCCSfQrl07FWgiIiLlmEbSwmTNmjXk5ORwzjnnULFiRRYvXsyZZ54Z7lgiIiISITSSVsrmzZtHjx49uPTSS0lLS6Nq1ao89dRTKtBERETkf2gkrRS4O99++y2nnnoqTzzxBN27d+eNN96gWrVq4Y4mIiIhlpeXx+bNm9mzZ0+xvl+rVi1Wr15dwqmkpFWtWpX69esTExNTYsdUkRZCeXl5TJgwgeTkZBISEpg0aRLvvfdeuGOJiEgp2rx5MzVr1qRBgwbFutY4MzOTmjVrhiCZlBR3Z9u2bWzevJnY2NgSO66KtBDIz8+nQoUK3HDDDfz4448MGTKEzp07hzuWiIiEwZ49e4pdoEl0MDNOPPFEfvjhhxI9roq0EvTDDz8wbNgwpk6dytKlS3nttdeoXr16uGOJiEiYqUAr+0LRx7pxoIS8+uqrnHXWWWzdupWpU6dSqVIlFWgiIhIRzIwbb7zxwPa+ffuoW7cuV155JQBjx46lbt26nHfeeTRq1IgrrriC//znP0Ef//7776devXrk5+cf2Pf4448zZMiQ/2nXoEEDfvzxRwC+++47evfuTcOGDbngggvo1q0ba9euDep8GRkZtGzZkvj4eHr16kVubu4v2uTm5nLbbbfRpEkTmjVrRmpq6oHP3nzzTZo0aULTpk3p0qXLgUw/e+aZZzCzA/sHDx5M8+bNad68Oeeeey4VK1Zk+/btQWU9FirSjsGSJUu4+eab2bZtGx06dGD16tWMHDmSRo0ahTuaiIjIAdWrVyctLY3du3cD8OGHH1KvXr3/adOrVy+WLFnCunXrGDRoENdee21QNyzk5+czdepUTj/9dGbPnh1UHnenR48etG/fnvXr17No0SKeeuopvv/++6C+P3DgQP70pz+Rnp7OCSecwCuvvPKLNqNHjwZgxYoVfPjhhwwYMID8/Hz27dvHfffdx6effsry5ctp2rQpw4cPP/C9TZs28cEHH3DGGWcc2Pfggw+ydOlSli5dylNPPUW7du2oU6dOUFmPhYq0Yli5ciWdO3fmqquuomnTplSpUoXY2FhOOeWUcEcTERE5pG7dujFjxgygYCTp+uuvP2zbDh06cOeddzJq1KgjHjc1NZXGjRtz99138+abbwaV5dNPPyUmJoa+ffse2NesWTMuueSSI37X3fnkk0+47rrrALjllluYNm3aL9qtWrWKjh07AnDyySdTu3ZtFi5ciLvj7mRnZ+Pu7Nq1i9NOO+3A9/70pz+RnJx82OnLI/2/K0kq0oK0f/9+Jk6cyFdffUXVqlXp3bs369ev54EHHqBGjRrhjiciIlKk3r17M2HCBPbs2cPy5ctp2bJlke3PP/981qxZA8DLL7/Myy+/fMh2PxctPXr0YMaMGeTl5R0xS1paGhdccMEhP8vMzDwwtXjwa9WqVWzbto3atWtTqVLBZfX169fnm2+++cVxmjVrRkpKCvv27SMjI4NFixaxadMmYmJiGDFiBE2aNOG0005j1apV3HHHHQC888471KtXj2bNmh0yW05ODu+//z6/+c1vjvgzlgTdOHAEe/bsYezYsQwZMoSTTz6Zl156iebNm9OwYcNwRxMRkSjUYNCMEj/mxqe7H7FN06ZN2bhxI2+++SbdunU7Ynt3P/C+8IhXYbm5ucycOZNnn32WmjVr0rJlS2bNmsWVV1552JGoI11gX7NmTZYuXXrYzw++fuxwbr/9dlavXs2FF17ImWeeSevWralYsSJ5eXmMGDGCJUuWEBcXxz333MNTTz1F//79+ec//8kHH3xw2GNOnz6dNm3alMpUJ6hIO6yffvqJ7OxsqlSpwgcffMDYsWNp27ZtuGOJiEiUC6agKqwkn5N29dVX88ADD5Camsq2bduKbLtkyRISExOLbDNr1ix27NhBkyZNgIKRpuOOO44rr7ySE088kS1btvxP+8zMTGrXrk3jxo2ZNGnSIY+ZmZl52GnP8ePHk5iYyI4dO9i3bx+VKlVi8+bNv7i+DqBSpUoMHTr0wHbr1q0566yzDhSAPw+29OzZk6effpprrrmGjIyMA6Nomzdv5vzzz+eLL744cDnThAkTSm2qEzTd+QubN29mwIABNGzYkOnTp1O3bl2mTJmiAk1ERKLe7bffzmOPPXagqDqc2bNnM2rUKPr06VNkuzfffJMxY8awceNGNm7cSEZGBh9++CE5OTlceumlpKSkkJmZCcCUKVNo1qwZFStWpGPHjuzdu/d/rnlbvnw5c+bMOTCSdqjXOeecg5nRoUOHA0Xev/71L6655ppfZMvJySE7OxsouFGiUqVKnHPOOdSrV49Vq1YdeKbZhx9+SGJiIk2aNGHr1q0Hfpb69euzePHiAwXazp07mT179iHPFSoaSQvIzc2lQoUKXHbZZXTv3p1ly5Zx+umnhzuWiIhIialfvz733nvvIT976623+Pzzz8nJySE2NpbJkycfGEn7+Xq0wtOeP1+fVfhaterVq9O2bVumT59Or1696NevH23btsXMOPnkkxkzZgxQMOU5depU7r//fpKSkqhatSoNGjTgueeeC+rnSEpKonfv3jz66KOcd955B64pS0lJYeHChTzxxBNs3bqVK664ggoVKlCvXj1ef/11AE477TQee+wxLr30UmJiYjjzzDMZO3bsEc85depUOnfuXKqP17LCc87RoMqpjXzvlnUldry5c+eSlJRETEwMkydPPjB8KqGRmppK+/btwx1DikF9F93Uf+GzevXqI04bFkXLQkWPQ/W1mS1y9wuLc7xyPd150003cfPNN9O1a1f+/e9/A6hAExERkYhQriqS3Nxcxo8fz7vvvsvEiRP529/+xhlnnKHCTERERCJOuRlJmzhxIg0bNuSNN96gb9++mBlxcXEq0ERERCQileki7fvvv+fJJ59k7969nHHGGUybNo0PP/yQyy+/XIvdiohIqYm267/l6IWij8tkkfbVV19x9913k5iYyLfffktOTg4XX3zxYZ9uLCIiEipVq1Zl27ZtKtTKMHdn27ZtVK1atUSPW6bm+hYtWkR8fDzffvstJ554ImvWrOHkk08OdywRESnH6tevz+bNmw88l+to7dmzp8T/8peSV7VqVerXr1+ixwxpkWZmXYDngYrAGHd/+qDPqwDjgAuAbUAvd994NOdwdz788EOSkpJYu3YtU6ZMoVWrVrRq1apkfggREZFjEBMTQ2xsbLG/n5qaynnnnVeCiSRahGy608wqAi8CXYFzgOvN7JyDmt0B/OTu8cBQICnY4+/bt4+srKwDi5zfcsstrF+/nosuuqikfgQRERGRsAnlNWktgHR33+DuucAE4OC1FK4B/hV4Pwm4zI5wRb+789JLL3HWWWfx6quvEh8fz7Jly7j55pupXLlyif8QIiIiIuEQyunOesCmQtubgZaHa+Pu+8xsJ3AicNgl7vN++IpZs2bx73//m9atWwPoTk0REREpc6LixgEzuxO4M7C5NyUlJS0lJSWckaT4TqKIIlwimvouuqn/opf6LrolFPeLoSzSvgEKr1BeP7DvUG02m1kloBYFNxD8D3cfBYwCMLOFxV0DS8JP/Re91HfRTf0XvdR30c3MFhb3u6G8Jm0B0MjMYs2sMtAbOHj4KwW4JfD+OuAT14NkREREREI3kha4xqwfMIuCR3C86u4rzewJYKG7pwCvAK+bWTqwnYJCTkRERKTcC+k1ae4+E5h50L6/Fnq/B/jtUR52VAlEk/BR/0Uv9V10U/9FL/VddCt2/5lmF0VEREQiT5lcu1NEREQk2kVskWZmXczsSzNLN7NBh/i8ipm9Ffj8/8ysQemnlEMJou/6m9kqM1tuZh+b2ZnhyCmHdqT+K9TuN2bmZqa7ziJIMP1nZj0Dv4MrzWx8aWeUQwviz84zzOxTM1sS+POzWzhyyi+Z2atmttXM0g7zuZnZsEDfLjez84M5bkQWaaFeUkpCJ8i+WwJc6O5NKVhpIrl0U8rhBNl/mFlN4D7g/0o3oRQlmP4zs0bAw0Abd28M3F/qQeUXgvzdexSY6O7nUXCj3Uulm1KKMBboUsTnXYFGgdedwIhgDhqRRRohWlJKSsUR+87dP3X3nMDmfAqeoSeRIZjfPYC/U/APoz2lGU6OKJj+6wO86O4/Abj71lLOKIcWTN85cHzgfS3g21LMJ0Vw988oeErF4VwDjPMC84HaZnbqkY4bqUXaoZaUqne4Nu6+D/h5SSkJr2D6rrA7gPdCmkiOxhH7LzBMf7q7zyjNYBKUYH7/zgLOMrO5ZjbfzIr617+UnmD67nHgRjPbTMGTE+4pnWhSAo7270YgSpaFkrLJzG4ELgTahTuLBMfMKgDPAreGOYoUXyUKplzaUzCK/ZmZNXH3HWFNJcG4Hhjr7s+YWSsKnjN6rrvnhzuYhEakjqQdzZJSFLWklJS6YPoOM7sc+DNwtbvvLaVscmRH6r+awLlAqpltBC4GUnTzQMQI5vdvM5Di7nnungGspaBok/AKpu/uACYCuPs8oCoF63pK5Avq78aDRWqRpiWlotcR+87MzgNGUlCg6XqYyFJk/7n7Tnc/yd0buHsDCq4pvNrdi702nZSoYP7snEbBKBpmdhIF058bSjOkHFIwffc1cBmAmSVSUKT9UKoppbhSgJsDd3leDOx09y1H+lJETndqSanoFWTfDQZqAG8H7vX42t2vDltoOSDI/pMIFWT/zQI6m9kqYD/woLtrFiLMguy7AcBoM/sTBTcR3KrBichgZm9S8I+fkwLXDD4GxAC4+8sUXEPYDUgHcoDbgjqu+ldEREQk8kTqdKeIiIhIuaYiTURERCQCqUgTERERiUAq0kREREQikIo0ERERkQikIk1ESpSZ7TezpYVeDYpo28DM0krgnKlm9qWZLQssd5RQjGP0NbObA+9vNbPTCn025lALzR9jzgVm1jyI79xvZtWO9dwiEn1UpIlISdvt7s0LvTaW0nlvcPdmwL8oeBbfUXH3l919XGDzVuC0Qp/93t1XlUjK/+Z8ieBy3g+oSBMph1SkiUjIBUbM5pjZ4sCr9SHaNDazLwKjb8vNrFFg/42F9o80s4pHON1nQHzgu5eZ2RIzW2Fmr5pZlcD+p81sVeA8QwL7HjezB8zsOgrWlH0jcM7jAiNgFwZG2w4UVoERt+HFzDmPQgssm9kIM1toZivN7G+BffdSUCx+amafBvZ1NrN5gf+Pb5tZjSOcR0SilIo0ESlpxxWa6pwa2LcV6OTu5wO9gGGH+F5f4Hl3b05BkbQ5sPRNL6BNYP9+4IYjnP8qYIWZVQXGAr3cvQkFK6zcbWYnAj2Axu7eFHiy8JfdfRKwkIIRr+buvrvQx5MD3/1ZL2BCMXN2oWCJpp/92d0vBJoC7cysqbsPA74FOrh7h8AyTv/f3t28RBXFYRz/PouEiBJcFEFgRZBE4CIKoV25iRZBiCAh0aYW1aKoRWR/QqskJAg0yIIgCUQiiQgSe6W0oBfB2kW4kAjRTf5anDMwDSPObOQqz2c3d+4959y7GJ4551x+PUB7fpZvgYvL9GNmq1Qhy0KZ2ao2n4NKuXVAb96D9ZdUL7LSOHBV0jbgYURMSToM7APe5BJi60mBr5q7kuaBH8B5YDfwPSK+5e8HgLNAL7AA3JY0DAzXemMRMSNpOtfemwJagLHcbj3jbCCVRit/Tp2STpN+l7cCe4DJimvb8vGx3E8D6bmZ2RrkkGZmK+EC8AtoJc3gL1SeEBGDkl4BR4ERSWcAAQMRcaWGPk6UF3qX1FTtpFwj8QCpUHUHcA44VMe93Ac6gS/AUESEUmKqeZzAO9J+tBvAcUk7gEvA/oiYldRPKp5dScBoRHTVMV4zW6W83GlmK6ER+BkRi0A3qYD0fyTtBKbzEt8j0rLfU6BD0uZ8TpOk5hr7/Apsl7Qrf+4Gnuc9XI0RMUIKj61Vrv0DbFyi3SHgGNBFCmzUO85cFPsa0CapBdgEzAG/JW0BjiwxlpfAwdI9SdogqdqspJmtAQ5pZrYSbgInJU2QlgjnqpzTCXyS9AHYC9zJb1T2AE8kTQKjpKXAZUXEAnAKeCDpI7AI9JECz3Bu7wXV93T1A32lFwcq2p0FPgPNEfE6H6t7nHmv23XgckRMAO9Js3ODpCXUklvAY0nPImKG9ObpvdzPOOl5mtkapPSHzszMzMyKxDNpZmZmZgXkkGZmZmZWQA5pZmZmZgXkkGZmZmZWQA5pZmZmZgXkkGZmZmZWQA5pZmZmZgXkkGZmZmZWQP8AmysivLe287sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "roc_data = {'MD': {'scores': od_preds['data']['instance_score'], 'labels': y_outlier}}\n",
    "plot_roc(roc_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Include categorical variables\n",
    "\n",
    "So far we only tracked continuous variables. We can however also include categorical variables. The `fit` step first computes pairwise distances between the categories of each categorical variable. The pairwise distances are based on either the model predictions (*MVDM method*) or the context provided by the other variables in the dataset (*ABDM method*). For MVDM, we use the difference between the conditional model prediction probabilities of each category. This method is based on the Modified Value Difference Metric (MVDM) by [Cost et al (1993)](https://link.springer.com/article/10.1023/A:1022664626993). ABDM stands for Association-Based Distance Metric, a categorical distance measure introduced by [Le et al (2005)](http://www.jaist.ac.jp/~bao/papers/N26.pdf). ABDM infers context from the presence of other variables in the data and computes a dissimilarity measure based on the Kullback-Leibler divergence. Both methods can also be combined as ABDM-MVDM. We can then apply multidimensional scaling to project the pairwise distances into Euclidean space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and transform data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_cols = ['protocol_type', 'service', 'flag']\n",
    "num_cols = ['srv_count', 'serror_rate', 'srv_serror_rate',\n",
    "            'rerror_rate', 'srv_rerror_rate', 'same_srv_rate', \n",
    "            'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', \n",
    "            'dst_host_srv_count', 'dst_host_same_srv_rate', \n",
    "            'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate',\n",
    "            'dst_host_srv_diff_host_rate', 'dst_host_serror_rate', \n",
    "            'dst_host_srv_serror_rate', 'dst_host_rerror_rate', \n",
    "            'dst_host_srv_rerror_rate']\n",
    "cols = cat_cols + num_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(494021, 21) (494021,)\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "kddcup = fetch_kdd(keep_cols=cols, percent10=True)\n",
    "print(kddcup.data.shape, kddcup.target.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dictionary with as keys the categorical columns and values the number of categories for each variable in the dataset. This dictionary will later be used in the `fit` step of the outlier detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 3, 1: 66, 2: 11}\n"
     ]
    }
   ],
   "source": [
    "cat_vars_ord = {}\n",
    "n_categories = len(cat_cols)\n",
    "for i in range(n_categories):\n",
    "    cat_vars_ord[i] = len(np.unique(kddcup.data[:, i]))\n",
    "print(cat_vars_ord)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit an ordinal encoder on the categorical data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrdinalEncoder(categories='auto', dtype=<class 'numpy.float64'>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc = OrdinalEncoder()\n",
    "enc.fit(kddcup.data[:, :n_categories])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine scaled numerical and ordinal features. `X_fit` will be used to infer distances between categorical features later. To make it easy, we will already transform the whole dataset, including the outliers that need to be detected later. This is for illustrative purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(494021, 21)\n"
     ]
    }
   ],
   "source": [
    "X_num = (kddcup.data[:, n_categories:] - mean) / stdev  # standardize numerical features\n",
    "X_ord = enc.transform(kddcup.data[:, :n_categories])  # apply ordinal encoding to categorical features\n",
    "X_fit = np.c_[X_ord, X_num].astype(np.float32, copy=False)  # combine numerical and categorical features\n",
    "print(X_fit.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize and fit outlier detector\n",
    "\n",
    "We use the same threshold as for the continuous data. This will likely not result in optimal performance. Alternatively, you can infer the threshold again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_components = 2\n",
    "std_clip = 3\n",
    "start_clip = 20\n",
    "    \n",
    "od = Mahalanobis(threshold,\n",
    "                 n_components=n_components, \n",
    "                 std_clip=std_clip, \n",
    "                 start_clip=start_clip,\n",
    "                 cat_vars=cat_vars_ord,\n",
    "                 ohe=False)  # True if one-hot encoding (OHE) is used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set `fit` parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "d_type = 'abdm'  # pairwise distance type, 'abdm' infers context from other variables\n",
    "disc_perc = [25, 50, 75]  # percentiles used to bin numerical values; used in 'abdm' calculations\n",
    "standardize_cat_vars = True  # standardize numerical values of categorical variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply `fit` method to find numerical values for categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "od.fit(X_fit,\n",
    "       d_type=d_type,\n",
    "       disc_perc=disc_perc,\n",
    "       standardize_cat_vars=standardize_cat_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The numerical values for the categorical features are stored in the attribute `od.d_abs`. This is a dictionary with as keys the columns for the categorical features and as values the numerical equivalent of the category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat = 0  # categorical variable to plot numerical values for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEWCAYAAACaBstRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdLElEQVR4nO3debwcVZn/8c+XLGwJhCUihISwubBIxPxAXBAUlGUgoCggMuyRARz4iUscHaMMo4yMgguKqGyOsqgsATLiggRFcBKQNcAYMZCELQkQwiaEPPPHOZdUmu6+5ya3b3fu/b5fr/u6Xae2p05X9VN1TnWXIgIzM7MSq7U7ADMzW3U4aZiZWTEnDTMzK+akYWZmxZw0zMysmJOGmZkVc9LoI5LeLemBXljObEl79EZMNcsNSVv19nK7WefpkhZIeqwv19tpJD0raYtOX2YnkzQmb/OglVzOjZKO7a24ulnXvZJ2K5y24fEp6UhJf+jV4JroN0kjf5g+IWntStmxkm5sY1iviojfR8Qb2x1Hp5A0BjgV2CYiXt/mWPo8YVZFxLCIeLDTl9mMpLG5Hgf31TqrIuLhvM2vtGP9KyIito2IG9sZg6TVJZ0v6RlJj0n6ZHfz9JukkQ0CTm53ELXadSB1uDHAwoh4oqcz9pf67C/b0W6rWj12WLxfArYGNgN2Bz4jaa9mM/S3pHEm8ClJI2pH1DsTql6K5ku8myWdJelpSQ9Kekcun5OvYo6ozLu6pP+U9LCkxyWdK2nNPG43SXMlfTY3vVzQVVaZf7SkKyTNl7RQ0ndy+ZaSbshlCyT9pN721Nm+nfOZwqBK2YGS7sqvd5J0S962RyV9R9LQBsta7hK99vJX0psk/VrSk5IekPSRyrh9JM2UtFjSPEmfqrP8PYBfA5vkJoULc/n++ZL96RzDmyvzzM71eRfwXL0DT9K2lbgel/Qv3W27pJvy7HfmWA7O5f8g6Y48zx8lvaWynh0l/Tlv488kXSbp9Mr44yTNynFMkbRJZVxIOlHSX4C/VMq2yq+b7VcbSro2x/SkpN9LqnsM1yzzQknnSLoux/wnSVvWmy9P/668zU/nff/IXL5v3u5ncvmXKrN11ePTuR53yfMcLek+SU9Jul7SZpX1vD/vP4skfVfSNC07HleT9AVJDykdexdLWjeP6zqWj5H0MHCDao5vSetLukDSI3ndV+Xy9XIdzs/l10ratFFdVGLdRNILktavlL1V6Rgdom6O23r7rypNzc320Yp9lD6XFkg6s8l73/D4rOMI4N8i4qmIuA/4AXBk08qIiH7xB8wG9gCuAE7PZccCN+bXY4EABlfmuRE4Nr8+ElgCHEW6YjkdeBg4B1gdeD+wGBiWpz8LmAKsDwwHrgG+msftlpf1H3neNXPZ3Dx+EHBnXsbawBrAu/K4rYA983wjSQfj2bXb2aAO/grsWRn+GTApv34b8HZgcK6L+4BTKtMGsFVtvVTq5g/59drAnFxPg4G3AgtIzUwAjwLvzq/XA3ZsEOur9ZGH3wA8l7d9CPAZYBYwtLLddwCjgTXrLG94XvepuT6HAzv3dNvz8FuBJ4Cd83t1RF7/6sBQ4CHSFe0Q4IPASyzb596b62PHPP23gZtq1vVr0n6zZp26b7ZffRU4N693CPBuQA3qt7rMC4GFwE65Dn4CXNpgvs1I+/mheR0bAOMq79n2pJPNtwCPAwc0Ob4m5PfwzXm9XwD+mMdtCDyT629wrs+XWXY8Hp3n3QIYRjquf1yzrotJ++OatesHrgMuI+2DQ4D35PINgA8Ba+X6/RlwVb3PhDp1cwNwXGX4TODcHhy3y+2/VI5lyvbR3+X9Ygzwvyz/2VV0fNZsz3p5uRtVyg4C7m76WdtXH+qt/mNZ0tgOWJTfuJ4mjb9Uxm1fp0IXAuMAkT7gtqyM2wX4W+XgeglYozJ+N5YljV2A+dVYmmzXAcCfa7ezwbSnA+fn18NzjJs1mPYU4MqanbIkaRwM/L5mWd8HJufXDwMfB9bpZrterY88/K/A5ZXh1YB5wG6V7T66yfIOrdZTN+tuuO15+Huks6/qPA8A7wF2zXGpMu4PLEsaPwK+Vhk3jPRhOLayrvfWLDtIHzrd7VenAVdXY22yjbVJ44eVcfsA9zeY73PVuulmHWcDZzU5vv4bOKbmPX2elJj+EbilMk6kD7uu4/G3wAmV8W/M9Ti4sq4tKuNfXT+wMbAUWK9gG8YBT1WGb6Rx0jgWuKEm3l0bTFvvuD26ZprZND6W6+2je1WGTwB+29Pjs6Z8dF5u9XNqT2B2szrrb81TRMQ9wLXApBWY/fHK6xfy8mrLhpES0lrAbfly8mngl7m8y/yIeLHBekYDD0XEktoRkjaSdKlS084zwH+RzspK/BT4oKTVSWdwt0fEQ3m5b8iX4o/l5X6lB8ut2gzYuWu787YfBnR1Zn+I9KH0UG5u2KVwuZuQzuABiIilpINyVGWaOU3mH0260nqNFdj2zYBTa7ZxdI5xE2Be5COsTly12/Es6WSjZDu626/OJJ19/yo3U/RkH6/eofY8aT+up1k97izpd7lpZxFwPN3X4zcr2/Ik6cN2FKmeXq2HXJ9zK/MuV4/59WBgo0pZo3ocDTwZEU/V2Ya1JH0/N3s9Q7oiGKGyu65+AewiaWPSycNS4Pd5uSXHbcP9t3Afrc7/EKmOanV3fFY9m/+vUylbh3Sl2VC/SxrZZOA4lj9Qn8v/16qUrehdOwtICWTbiBiR/9aNiOqBGA3mhfTmj1H9DrGv5Hm3j4h1gI+RDrRuRcRM0s60N/BRUhLp8j3gfmDrvNx/abLc52hcT3OAaZXtHhHprpV/yjFMj4gJwOuAq4DLS2IHHiHt8ABIEungn1fdxCbzzyE1ZdTTk23vWta/12zjWhFxCakJbFSOr8voJtuxNqlJpGQ7mu5XEbE4Ik6NiC2A/YFPSnpfk+1YEXOARv0dPyU1nY2OiHVJTWVd9VBvm+YAH6+pxzUj4o+keny1LyHXZ7VvYbl6JDXJLGH5E7tG9TgHWF/1+wJPJV217Jz3hV27QmiwrGUrS0noV6Sz+Y+Smvi6Yig5bpvtvyX7aHU/G0Oqo1pNj8862/MosEOleAfg3iZx9s+kERGzSO2Z/1wpm086cD8maZCko2l8cHS3/KWkDqOzJL0OQNIoSR8oXMT/kN6sMyStLWkNSe/M44aTzgAWSRoFfLqH4f2U1D68K6m9tstwUhvys5LeBLxmJ6q4g3TFspZSZ+oxlXHXAm+QdHjuABwi6f9JerOkoZIOk7RuRLyc17e0MO7LgX0lvU/SENLB/Xfgj4XzXwtsLOkUpc7k4ZJ2Ltz2x1k+4fwAOD6fWSu/R/tKGg7cArwCnJQ7MyeQ+gq6XAIcJWlcvuL7CvCniJjd3QZ0t18pdc5vlT9gF+U4Suu31E+APSR9JG/fBpLG5XHDSWfwL0raifTB2WV+jqVaj+cCn5O0bY5/XUkfzuOuA7aXdEA+eTqR5U9OLgH+v6TNJQ0j1eNl9a7Oa0XEo6Smse8qdXwPkdSVHIaTEvPTSp3akwvrpctPSU1rB7H8SdnKHrclx+en8/aMJh3jl9WZpuHx2WC9FwNfyMt9E+lk+8JmgfbLpJGdRuoUqjqO9GYuBLal/AOpns+SmgpuzZeTvyGdwXQr0r3k+5HasR8mXZYfnEd/mdSJuoh0YF3Rw7guIbW93xARCyrlnyId5ItJH0z1drguZ5H6ZB4HLiJ9kHTFvph0U8AhpDOdx1jW4Q9wODA718nxpEvjbkXEA6Szs2+Tzrj3A/aLiJcK519Mao/dL8f0F9IthND9tn8JuChfzn8kImaQ9pXvAE+R3ucj83peIjX9HQM8nWO+lpTgiIjfkPpnfkE6MdiSVFelmu1XW+fhZ0nJ67sR8bseLLtbEfEwqXnxVFJz0h0sOxM9AThN0mLgi1SuIiPieeDfgZtzPb49Iq4k7RuX5m25h3QVTN43Pwx8jXQ8bgPMINcjcD7wY1Lz0d+AF4FP9GBTDif1gdxPuqnhlFx+NqnjfAFwK6n5ryemkN6HxyLizkr5yh63Jcfn1cBtpPfkOlL/2XIKjs9ak0nNkQ8B04AzI6JpnWj5plkz6ylJfyLdRXNBu2NZVSndPjoXOKy3E6H1rv58pWHWEpLeI+n1ufnmCNLtpz09Yx3wJH1A0ojcjNfVhn9rm8OybnTSNxPNVhVvJDXNrA08CByU29GtZ3Yh9QsMBWaSvvPxQntDsu64ecrMzIq5ecrMzIr1u+apDTfcMMaOHdvuMMzMVim33XbbgogY2d10/S5pjB07lhkzZrQ7DDOzVYqkh7qfys1TZmbWA04aZmZWzEnDzMyKOWmYmVkxJw0zMyvmpGFmZsWcNMzMrJiThpmZFet3X+6zgWXspOvaHUK/NfuMfdsdgnUgX2mYmVkxJw0zMyvmpGFmZsWcNMzMrJiThpmZFXPSMDOzYk4aZmZWzEnDzMyKOWmYmVkxJw0zMyvmpGFmZsXamjQknS/pCUn3NBgvSd+SNEvSXZJ27OsYzcxsmXZfaVwI7NVk/N7A1vlvIvC9PojJzMwaaGvSiIibgCebTDIBuDiSW4ERkjbum+jMzKxWu680ujMKmFMZnpvLliNpoqQZkmbMnz+/z4IzMxtoOj1pFImI8yJifESMHzlyZLvDMTPrtzo9acwDRleGN81lZmbWBp2eNKYA/5jvono7sCgiHm13UGZmA1VbH/cq6RJgN2BDSXOBycAQgIg4F5gK7APMAp4HjmpPpGZmBm1OGhFxaDfjAzixj8IxM7NudHrzlJmZdRAnDTMzK+akYWZmxZw0zMysmJOGmZkVc9IwM7NiThpmZlbMScPMzIo5aZiZWTEnDTMzK+akYWZmxZw0zMysmJOGmZkVc9IwM7NiThpmZlbMScPMzIo5aZiZWTEnDTMzK+akYWZmxZw0zMysmJOGmZkVc9IwM7NiThpmZlbMScPMzIo5aZiZWTEnDTMzK+akYWZmxZw0zMysmJOGmZkVc9IwM7NiThpmZlbMScPMzIq1NWlI2kvSA5JmSZpUZ/yRkuZLuiP/HduOOM3MLBncrhVLGgScA+wJzAWmS5oSETNrJr0sIk7q8wDNzOw12nmlsRMwKyIejIiXgEuBCW2Mx8zMutHOpDEKmFMZnpvLan1I0l2Sfi5pdL0FSZooaYakGfPnz29FrGZmRud3hF8DjI2ItwC/Bi6qN1FEnBcR4yNi/MiRI/s0QDOzgaSdSWMeUL1y2DSXvSoiFkbE3/PgD4G39VFsZmZWRzuTxnRga0mbSxoKHAJMqU4gaePK4P7AfX0Yn5mZ1Si+e0rSWhHxfG+tOCKWSDoJuB4YBJwfEfdKOg2YERFTgH+WtD+wBHgSOLK31m9mZj3XbdKQ9A5S09AwYIykHYCPR8QJK7vyiJgKTK0p+2Ll9eeAz63seszMrHeUNE+dBXwAWAgQEXcCu7YyKDMz60xFfRoRMaem6JUWxGJmZh2upE9jTm6iCklDgJNxh7SZ2YBUcqVxPHAi6Yt384BxedjMzAaYbq80ImIBcFgfxGJmZh2u5O6pC4CoLY+Io1sSkZmZdaySPo1rK6/XAA4EHmlNOGZm1slKmqd+UR2WdAnwh5ZFZGZmHWtFfkZka+B1vR2ImZl1vpI+jcWkPg3l/48Bn21xXGZm1oFKmqeG90UgZmbW+RomDUk7NpsxIm7v/XDMzKyTNbvS+HqTcQG8t5djMTOzDtcwaUTE7n0ZiJmZdb6i52lI2g7YhvQ9DQAi4uJWBWVmZp2p5O6pycBupKQxFdib9D0NJw0zswGm5HsaBwHvAx6LiKOAHYB1WxqVmZl1pJKk8UJELAWWSFoHeAIY3dqwzMysE5X0acyQNAL4AXAb8CxwS0ujMjOzjlTy5b6uZ4GfK+mXwDoRcVdrwzIzs07UbfOUpCmSPipp7YiY7YRhZjZwlfRpfB14FzBT0s8lHSRpje5mMjOz/qekeWoaME3SINK3wI8DzgfWaXFsZmbWYUq/3LcmsB9wMLAjcFErgzIzs85U8uW+y4GdgF8C3wGm5VtwzcxsgCm50vgRcGhEvNLqYMzMrLOV9Glc3xeBmJlZ51uRx72amdkA5aRhZmbF/OQ+M+tTYydd1+4Q+q3ZZ+zb8nX4yX1mZlasrU/uk7QX8E1gEPDDiDijZvzqpOd2vA1YCBwcEbNbHZeZmdXXtif35W+YnwPsCcwFpkuaEhEzK5MdAzwVEVtJOgT4D9IXDM3MrA1KfrBwMvDt/Lc78DVg/15Y907ArIh4MCJeAi4FJtRMM4Fl3z7/OfA+SeqFdZuZ2QooudI4iPS0vj9HxFGSNgL+qxfWPQqYUxmeC+zcaJqIWCJpEbABsKA6kaSJwESAMWPGrFRQ7qRrnVZ00vVFx5/1Lr9nq7Z+8eS+iDgvIsZHxPiRI0e2Oxwzs36rnU/um8fyyWfTXFZvmrmSBpOeTb6wF9ZtZmYroJ1P7psObC1pc1JyOAT4aM00U4AjSEnqIOCGiIheWLeZma2Ako7wAyWtC5Bvd31Y0gEru+KIWAKcBFwP3AdcHhH3SjpNUldH+4+ADSTNAj4JTFrZ9ZqZ2YoraZ6aHBFXdg1ExNP5jqqrVnblETEVmFpT9sXK6xeBD6/seszMrHeUdITXm6bo+x1mZta/lCSNGZK+IWnL/PcNUoe4mZkNMCVJ4xPAS8Bl+e/vwImtDMrMzDpTyd1Tz+EOaDMzo/lPo58dEadIuob0q7bLiYje+CkRMzNbhTS70vhx/v+ffRGImZl1vmY/jX5b/iXaiRFxWB/GZGZmHappR3hEvAJsJmloH8VjZmYdrOT7Fg8CN0uaAjzXVRgR32hZVGZm1pFKksZf899qwPDWhmNmZp2s5JbbLwNIWisinm99SGZm1qlKfrBwF0kzgfvz8A6SvtvyyMzMrOOUfCP8bOAD5OdYRMSdwK6tDMrMzDpTSdIgIubUFL3SgljMzKzDlXSEz5H0DiAkDQFOJj3/wszMBpiSK43jST9QOIr0hL1x+AcLzcwGpJK7pxYA/ka4mZl1nzTyM7w/AYytTu8fLDQzG3hK+jSuIj2r+xpgaWvDMTOzTlaSNF6MiG+1PBIzM+t4JUnjm5ImA78iPbUPgIi4vWVRmZlZRypJGtsDhwPvZVnzVORhMzMbQEqSxoeBLSLipVYHY2Zmna3kexr3ACNaHYiZmXW+kiuNEcD9kqazfJ+Gb7k1MxtgSpLG5JZHYWZmq4SSb4RP64tAzMys85V8I3wx6W4pgKHAEOC5iFinlYGZmVnnKbnSePURr5IETADe3sqgzMysMxU9T6NLJFeRHspkZmYDTEnz1Acrg6sB44EXWxaRmZl1rJK7p/arvF4CzCY1UZmZ2QBT0qdxVG+vVNL6wGWkn1ufDXwkIp6qM90rwN158GF/N8TMrL0aJg1JX2wyX0TEv63EeicBv42IMyRNysOfrTPdCxExbiXWY2ZmvahZR/hzdf4AjqH+B3xPTAAuyq8vAg5YyeWZmVkfaHilERFf73otaThwMnAUcCnw9UbzFdooIh7Nrx8DNmow3RqSZpD6Us7Id269hqSJwESAMWPGrGRoZmbWSNM+jdz38EnSM8IvAnas1/fQYN7fAK+vM+rz1YGICElRZzqAzSJinqQtgBsk3R0Rf62dKCLOA84DGD9+fKNlmZnZSmrWp3Em8EHSh/H2EfFsTxYcEXs0WfbjkjaOiEclbQw80WAZ8/L/ByXdCLwVeE3SMDOzvtGsT+NUYBPgC8Ajkp7Jf4slPbOS650CHJFfHwFcXTuBpPUkrZ5fbwi8E5i5kus1M7OV0KxPo0ffFu+hM4DLJR0DPAR8BEDSeOD4iDgWeDPwfUlLScntjIhw0jAza6OSL/f1uohYCLyvTvkM4Nj8+o+kR82amVmHaOXVhJmZ9TNOGmZmVsxJw8zMijlpmJlZMScNMzMr5qRhZmbFnDTMzKyYk4aZmRVz0jAzs2JOGmZmVsxJw8zMijlpmJlZMScNMzMr5qRhZmbFnDTMzKyYk4aZmRVz0jAzs2JOGmZmVsxJw8zMijlpmJlZMScNMzMr5qRhZmbFnDTMzKyYk4aZmRVz0jAzs2JOGmZmVsxJw8zMijlpmJlZMScNMzMr5qRhZmbFnDTMzKxYW5KGpA9LulfSUknjm0y3l6QHJM2SNKkvYzQzs9dq15XGPcAHgZsaTSBpEHAOsDewDXCopG36JjwzM6tncDtWGhH3AUhqNtlOwKyIeDBPeykwAZjZ8gDNzKyuTu7TGAXMqQzPzWWvIWmipBmSZsyfP79PgjMzG4hadqUh6TfA6+uM+nxEXN2b64qI84DzAMaPHx+9uWwzM1umZUkjIvZYyUXMA0ZXhjfNZWZm1iad3Dw1Hdha0uaShgKHAFPaHJOZ2YDWrltuD5Q0F9gFuE7S9bl8E0lTASJiCXAScD1wH3B5RNzbjnjNzCxp191TVwJX1il/BNinMjwVmNqHoZmZWROd3DxlZmYdxknDzMyKOWmYmVkxJw0zMyvmpGFmZsWcNMzMrJiThpmZFXPSMDOzYk4aZmZWzEnDzMyKOWmYmVkxJw0zMyvmpGFmZsWcNMzMrJiThpmZFWvL8zQ62ewz9m13CGZmHctXGmZmVsxJw8zMijlpmJlZMScNMzMr5qRhZmbFnDTMzKyYk4aZmRVz0jAzs2JOGmZmVkwR0e4YepWk+cBD7Y6jj2wILGh3ENYjfs9WPQPlPdssIkZ2N1G/SxoDiaQZETG+3XFYOb9nqx6/Z8tz85SZmRVz0jAzs2JOGqu289odgPWY37NVj9+zCvdpmJlZMV9pmJlZMScNMzMr5qSxipK0l6QHJM2SNKnd8Vhzks6X9ISke9odi3VP0mhJv5M0U9K9kk5ud0ydwn0aqyBJg4D/BfYE5gLTgUMjYmZbA7OGJO0KPAtcHBHbtTsea07SxsDGEXG7pOHAbcABPsZ8pbGq2gmYFREPRsRLwKXAhDbHZE1ExE3Ak+2Ow8pExKMRcXt+vRi4DxjV3qg6g5PGqmkUMKcyPBfv0GYtIWks8FbgT+2NpDM4aZiZNSBpGPAL4JSIeKbd8XQCJ41V0zxgdGV401xmZr1E0hBSwvhJRFzR7ng6hZPGqmk6sLWkzSUNBQ4BprQ5JrN+Q5KAHwH3RcQ32h1PJ3HSWAVFxBLgJOB6Ugfd5RFxb3ujsmYkXQLcArxR0lxJx7Q7JmvqncDhwHsl3ZH/9ml3UJ3At9yamVkxX2mYmVkxJw0zMyvmpGFmZsWcNMzMrJiThpmZFXPSMGtA0uslXSrpr5JukzRV0hsaTDtC0gl9HaNZX3PSMKsjf7nrSuDGiNgyIt4GfA7YqMEsI4CWJw1Jg1u9DrNmnDTM6tsdeDkizu0qiIg7gT9L+q2k2yXdLanr14XPALbMXwI7E0DSpyVNl3SXpC93LUfSv+ZnofxB0iWSPpXLx0m6NU9/paT1cvmNks6WNAP4vKS/5Z+4QNI61WGzVvNZi1l925GeoVDrReDAiHhG0obArZKmAJOA7SJiHICk9wNbk37GXsCU/EyNF4APATsAQ4DbK+u5GPhEREyTdBowGTgljxsaEePzsscC+wJXkX5C5oqIeLkXt92sIScNs54R8JWcAJaSfpK+XpPV+/Pfn/PwMFISGQ5cHREvAi9KugZA0rrAiIiYlqe/CPhZZXmXVV7/EPgMKWkcBRzXC9tlVsRJw6y+e4GD6pQfBowE3hYRL0uaDaxRZzoBX42I7y9XKJ1SZ9oSz3W9iIibJY2VtBswKCL8CFnrM+7TMKvvBmB1SRO7CiS9BdgMeCInjN3zMMBi0lVEl+uBo/PzGJA0StLrgJuB/SStkcf9A0BELAKekvTuPP/hwDQauxj4KXDBSm6nWY/4SsOsjogISQcCZ0v6LKkvYzbwJeBbku4GZgD35+kXSrpZ0j3Af0fEpyW9Gbgl3YjFs8DHImJ67gO5C3gcuBtYlFd7BHCupLWAB0lNT438BDgduKQXN9usW/6VW7M+JmlYRDybk8NNwMSu51H3YBkHARMi4vCWBGnWgK80zPreeZK2IfWFXLQCCePbwN6An+9gfc5XGmZmVswd4WZmVsxJw8zMijlpmJlZMScNMzMr5qRhZmbF/g9zfOAUXnSf3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(np.arange(len(od.d_abs[cat])), od.d_abs[cat])\n",
    "plt.xticks(np.arange(len(od.d_abs[cat])))\n",
    "plt.title('Numerical values for categories in categorical variable {}'.format(cat))\n",
    "plt.xlabel('Category')\n",
    "plt.ylabel('Numerical value')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option would be to set `d_type` to `'mvdm'` and `y` to `kddcup.target` to infer the numerical values for categorical variables from the model labels (or alternatively the predictions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run outlier detector and display results\n",
    "\n",
    "Generate batch of data with 10% outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 21) (1000,)\n",
      "10.0% outliers\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "outlier_batch = create_outlier_batch(kddcup.data, kddcup.target, n_samples=1000, perc_outlier=10)\n",
    "data, y_outlier = outlier_batch.data, outlier_batch.target\n",
    "print(data.shape, y_outlier.shape)\n",
    "print('{}% outliers'.format(100 * y_outlier.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocess the outlier batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 21)\n"
     ]
    }
   ],
   "source": [
    "X_num = (data[:, n_categories:] - mean) / stdev\n",
    "X_ord = enc.transform(data[:, :n_categories])\n",
    "X_outlier = np.c_[X_ord, X_num].astype(np.float32, copy=False)\n",
    "print(X_outlier.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "od_preds = od.predict(X_outlier, return_instance_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F1 score and confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.9375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXqElEQVR4nO3deZxcVZn/8c83hLBJNsISE0YCYiEgBAiCgiOCCGFkE4YJiiSMY+OI4jijjKgj44K/n75kUH7yA5o1gCwGZYyZqCDbsE1iSGJYQg1h8ZVkCEEDASQGuvuZP+5tLUNSfbu7qk/X7e+b13l13XNv3XqKrjx5cu65pxQRmJnZwBuWOgAzs6HKCdjMLBEnYDOzRJyAzcwScQI2M0vECdjMLJHhqQNoNZVK5TPAxwEBl1er1e/283zTgS/nm9+oVqszK5XK1sAsYDegE/hptVr9Qn9exwalnYFrgR2BANqB7yWNyAaUPA+4uEqlsjdwE/BO4DXg58AnqtXqsgLPvRuYUa1Wn6npGwssAKaQ/QF8CDgAWA8cVK1W76pUKiOAO4BvVqvVnzX0DVlq4/O2ENiW7Pd/AvBYyqBs4LgC7p23A/Oq1eqrAJVK5R7gQ5VK5UfAxcD2wKvAx6vV6uMFzncUcHu1Wl2Tn+924OhqtXojcBdAtVp9rVKpLAQmNvzdWGrP5g3gZWApMAEn4CGjbgKWtH+9/RGxsLHhDHqPAOdXKpXtgHXAMWQVbDtZJfxEpVI5CPj/wOEFzjcBWF6zvSLv+6NKpTIaOBb/07TsdgH2A+YljsMGUE8V8AV19gXFkkxpVKvVpZVK5VvAbcDvgcXAVsC7gVmVSqX70C0AKpXKGcBn8r63AnMrlcprwNPVavXEnl6vUqkMB24ELqpWq0818r3YoPIm4EfAPwAvJY7FBlBTxoAltQFtAJdddtkBZ3zo/Q1/jcHgu5dew7jtxnDFdTdz9+wb6h4741PncP6X/okJ43f8Y9/c2+/mV4uWcN45ZwPw1W9fxIH77cMxRx4GwJe/+W9svdVWfPGzf9+095DC5uN2BWD4iAk9HFl+w4cPZ/a/z+S22+/hu99rTx1OUh2vrYTs4na/vP7bpwontc3H7drv1+uPwtPQJO0t6RRJp3e3TR0bEe0RMSUiprS1tTUm0kHidy+8CMCzq1Zzxz33c9zRRzBh/E784s57AYgIHn+iWLF6yEEH8MD8hax96WXWvvQyD8xfyCEHHQDARe0zeeWVV/nCZ85szhuxQeHy9gtY+viyIZ98G6qrs3hLrNBFOEnnAYcBewJzganAfWRTaIaUz37xG7z40ksMHz6cL/3TJxm57Zv41nnn8PXvfJ/LZt5IR0cHU494L3vsvmuP5xo1clvOnHEq0/4uG6X4xBkfZtTIbVm1+nnaZ97EpLfszF+f8WkATj3pWE4+7uimvjcbWIe8+0A+etrJLHn4MRb86jYA/uVf/i8/+/mdiSNrcdGVOoLCCg1BSHoY2BdYFBH7StoRuD4ijizwGvH6bz18aX/iIQjbmIYNQTy7tPgQxPi3Jx2CKDoNbV1EdEnqkDQSWE02idzMbFCJFqqAiybgBZJGA5eTTRZ/BXiwaVGZmfVVV8kScER8Mn94qaSfAyMjYknzwjIz66MSVsBI2odssvjwfPutEfHjJsVlZtY3na+njqCworMgrgL2AR4Fuv96CcAJ2MwGl7INQQAHR8SeTY3EzKwBWukiXNEbMR6U5ARsZoNfV1fxlljRCvhasiS8imypRAEREfs0LTIzs75ooQq4aAK+Evgo8DB/GgM2Mxt8ynYRDng+ImY3NRIzs0YYBEMLRRVNwIsk3QD8lGwIAgBPQzOzQaeEQxBbkSXeD9T0eRqamQ0+ZaqAJW0GLImICwcgHjOzfolIv8xkUT1OQ4vs3Zw6ALGYmfVfdBVviRUdgrhf0veBm8m+igcYkt8JZ2aDXWdH6ggKK5qAJ+c/v1bTN+S+E87MWsAg+KaLooquhva+ZgdiZtYQg2Booaiii/GMAs4D/jLvugf4WkSsbVZgZmZ90kKzIIquBXEV8DJwSt5eAq5uVlBmZn1Wwotwu0XESTXbX5W0uBkBmZn1SwtVwIW/E07SoRFxH4CkQ4B1zQvLzKxvooRrQfw9MDMfCwZ4AZjenJDMzPqhhBXwUuDbwG7AaGAtcALg74Uzs8FlEIztFlU0Af8EeBFYCKxsXjhmZv1Uwgp4YkQc3dRIzMwaoYQV8AOS3hERDzc1GjOz/irhrciHAjMkPY2/ksjMBrMSDkFMbWoUZmaNUrYEHBG/aXYgZmYN0aAxYEkVshUgu+0KfIVsJtjHgefz/i9GxNz8OecCHwM6gbMj4hf1XqNoBWxm1hoaVAFHRJV8Jcj8iylWArcCZwAXRsR3ao+XtCcwDdgLeDPwS0lvizorxBddC8LMrDU0Zy2II4AnexgNOB64KSLWR8TTwDLgnfVO6gRsZuXS2VG8FTcNuLFm+1OSlki6StKYvG8CsLzmmBV53yY5AZtZuXR1FW6S2iQtqGltG55O0gjgOGBW3nUJ2V3Bk4FngQv6GqrHgM2sXHoxBhwR7UB7D4dNBRZGxHP5c57r3iHpcmBOvrkS2LnmeRPp4c5hV8BmVi4RxVsxp1Iz/CBpfM2+E4FH8sezgWmStpA0CdgdmF/vxK6AzaxcGjgPWNI2wJHAmTXd35Y0mex7MZ/p3hcRj0r6IfAY0AGcVW8GBDgBm1nZNDABR8Tvge026PtonePPB84ven4nYDMrlxKuBWFm1hqKj+0m5wRsZuVStrUgzMxahhOwmVkiJVyQ3cysJURH3Zlfg4oTsJmViytgM7NEujwLwswsDV+EMzNLxAnYzCwR34hhZpaIZ0GYmSXiWRBmZol4FoSZWRrhi3BmZom4AjYzS8RjwGZmiXgWhJlZIh6CMDNLxEMQZmaJuAI2M0vD09DMzFLpcAI2M0vDY8BmZol4DNjMLI1wAjYzS8QJ2MwskRaaBTEsdQBmZg3V0VW89UDSaEm3SHpc0lJJ75I0VtLtkp7If47Jj5WkiyQtk7RE0v49nd8J2MxKJSIKtwK+B/w8IvYA9gWWAl8A7oiI3YE78m2AqcDueWsDLunp5E7AZlYuXVG81SFpFPCXwJUAEfFaRLwIHA/MzA+bCZyQPz4euDYy/wWMljS+3ms4AZtZufQiAUtqk7SgprXVnGkS8DxwtaRFkq6QtA2wY0Q8mx+zCtgxfzwBWF7z/BV53yb5IpyZlUpvpqFFRDvQvondw4H9gU9HxDxJ3+NPww3dzw9JfZ524QrYzMqlQUMQZBXsioiYl2/fQpaQn+seWsh/rs73rwR2rnn+xLxvk5yAzaxUoiMKt7rniVgFLJdUybuOAB4DZgPT877pwE/yx7OB0/PZEAcDa2uGKjbKQxBmVi6NvRHj08APJI0AngLOICtcfyjpY8BvgFPyY+cCxwDLgFfzY+tyAjazcmngfRgRsRiYspFdR2zk2ADO6s35nYDNrFS8FoSZWSqtcyeyE7CZlUtPF9cGEydgMyuVFlqP3QnYzErGCdjMLA1XwGZmqTgBm5ml4QrYzCyRro7UERTnBGxm5RJKHUFhTsBmVioegjAzSyS6XAGbmSXhCtjMLJGuTlfAZmZJeAjCzCyRYt82Pzg4AZtZqbgCNjNLxAnYzCwRD0GYmSXS1dk6X/buBGxmpeJ5wGZmiXR5LQgzszTCCdjMLA3PgjAzS8SzIMzMEun0LAgzszRaaQy4df6qMDMrIKJ4K0LSZpIWSZqTb18j6WlJi/M2Oe+XpIskLZO0RNL+PZ3bFbCZlUoTpqF9BlgKjKzp+3xE3LLBcVOB3fN2EHBJ/nOTXAGbWalEqHDriaSJwF8BVxR46eOBayPzX8BoSePrPWFAKuDNx+06EC9jLabjtZWpQ7AS6mzsNLTvAucA227Qf76krwB3AF+IiPXABGB5zTEr8r5nN3VyV8BmViq9qYAltUlaUNPaus8j6YPA6oh4aIOXOBfYAzgQGAv8c19jHZAKeMQWEwfiZaxFvLZ+BQBv235K4khsMPnv5xc05Dy9GQOOiHagfRO7DwGOk3QMsCUwUtL1EXFavn+9pKuBz+XbK4Gda54/Me/bJFfAZlYq0YtW9zwR50bExIjYBZgG3BkRp3WP60oScALwSP6U2cDp+WyIg4G1EbHJ4QfwLAgzK5kBWIznB5K2BwQsBj6R988FjgGWAa8CZ/R0IidgMyuVZtyIERF3A3fnjw/fxDEBnNWb8zoBm1mpdNI6d8I5AZtZqXR5MR4zszS6XAGbmaURTsBmZmm00FfCOQGbWbm4AjYzS6QjdQC94ARsZqXiCtjMLJEW+k5OJ2AzKxdPQzMzS6SF7sNwAjazcumQK2AzsyRcAZuZJeIbMczMEvEsCDOzRDwLwswsEY8Bm5kl0tE6BbATsJmViytgM7NEfBHOzCwRT0MzM0vECdjMLJEmfCt90zgBm1mpeEF2M7NEPAvCzCwRz4IwM0uklS7CDUsdgJlZI3X1otUjaUtJ8yX9WtKjkr6a90+SNE/SMkk3SxqR92+Rby/L9+/SU6xOwGZWKp0q3nqwHjg8IvYFJgNHSzoY+BZwYUS8FXgB+Fh+/MeAF/L+C/Pj6nICNrNSaVQFHJlX8s3N8xbA4cAtef9M4IT88fH5Nvn+I6T6X8/hBGxmpRK9aD2RtJmkxcBq4HbgSeDFiOie7bYCmJA/ngAsB8j3rwW2q3d+J2AzK5UuonCT1CZpQU1rqz1XRHRGxGRgIvBOYI9GxupZEGZWKr2ZBRER7UB7geNelHQX8C5gtKTheZU7EViZH7YS2BlYIWk4MAr4Xb3zugI2s1Jp1BCEpO0ljc4fbwUcCSwF7gJOzg+bDvwkfzw73ybff2dE1H0ZV8BmVioNXJB9PDBT0mZkxeoPI2KOpMeAmyR9A1gEXJkffyVwnaRlwBpgWk8v4ARsZqXS1aCbkSNiCbDfRvqfIhsP3rD/D8Bf9+Y1nIDNrFS8FoSZWSKtdCuyE7CZlUqjhiAGghOwmZVKZ+oAesEJ2MxKxRWwmVkirZN+nYDNrGR8Ec7MLJFooRrYCdjMSsUVsJlZIp2ugM3M0vAsCDOzRDwEYWaWiC/CmZkl4grYzCwRV8BmZol01P8SikHFCdjMSqV10q8TsJmVjKehmZkl4jFgM7NEPAvCzCyRzhZKwU7AZlYqrZN+nYDNrGTC09DMzNLwLAgzs0Q8BGFmloinoZmZJdIZrVMDOwGbWam0TvqFYakDMDNrpOjFfz2RdJWk1ZIeqen7V0krJS3O2zE1+86VtExSVdJRPZ3fFbCZlUqDZ0FcA3wfuHaD/gsj4ju1HZL2BKYBewFvBn4p6W0R0bmpk7sCNrNSiYjCrcC5/hNYU/Cljwduioj1EfE0sAx4Z70nOAGbWal0EYVbP3xK0pJ8iGJM3jcBWF5zzIq8b5OcgM2sVDqjq3CT1CZpQU1rK/ASlwC7AZOBZ4EL+hqrx4DNrFR6U9dGRDvQ3qvzRzzX/VjS5cCcfHMlsHPNoRPzvk1yBWxmpdLsIQhJ42s2TwS6Z0jMBqZJ2kLSJGB3YH69c7kCNrNSaeQsCEk3AocB4yStAM4DDpM0mazYfgY4EyAiHpX0Q+AxoAM4q94MCHACNrOSaeRqaBFx6ka6r6xz/PnA+UXP7wRsZqXiBdnNzBLxesBmZol4PWAzs0RcAZuZJeIK2MwsES/IbmaWiBdkNzNLpMtjwGZmaXgIwswsEVfAZmaJuAI2M0vEFbCZWSJd9RcgG1ScgM2sVHwjhplZIr4V2cwsEVfAZmaJuAI2M0vEtyKbmSXiCtjMLBGPAZuZJeIK2MwsEd8JZ2aWiCtgM7NEPAvCzCwRD0GYmSXSSstRDksdQFm1X/YdVixfzKKFv/xj35gxo5k79wYeffRe5s69gdGjRyWM0FI4vW0ac/7zZv7j3puZfuapAIwaPZKrZ13MbfN+zNWzLmbkqG0TR9nauiIKt9ScgJvk2utm8cFjT/uzvnM+fxZ33Xk/e+31Hu66837O+fxZiaKzFHbfYzdOOe1ETj7qdI477MO878hD+YtJE2k7ewYP3jufDxz0IR68dz5tZ89IHWpLi4jCLbUeE7CkYZLePRDBlMl9983jhRde/LO+Y4/9ANddPwuA666fxXHHHZUiNEtkt7ftwq8XPsIf1q2ns7OT+Q8s5AN/dThHTH0vt948B4Bbb57D+485LG2gLa4rugq31HpMwBHRBVw8ALGU3g47jGPVqtUArFq1mh12GJc4IhtITyx9kikHT2b0mFFsudUWvPf9hzB+wo6M234szz/3OwCef+53jNt+bOJIW1srVcBFL8LdIekk4McxGKIuCf+vHFqefOIZLv9/13LVrO+z7tV1LH3kv+nsfOO3N/hz0T+t9H9PRX7Zkl4GtgE6gXWAgIiIkZs4vg1oyzfbI6K9MeG2nF2AOcDeAGvWrFk1duzY/YBngfHA3UAlVXCW3DeBFWvWrPmKPxdDU6GLcBGxbUQMi4jNI2Jkvr3R5Jsf3x4RU/I2VJPvG8yaNQtger45HfhJumgskR3yn38BfAi4wZ+LoatoBSzgI8CkiPi6pJ2B8RExv9kBtrAbgcOAccBzwHk77bTTp1atWrWG7A/fb4BTgDXJIrQU7gW2A14H/hG4Y6eddlrkz8XQVDQBXwJ0AYdHxNsljQFui4gDmx1gmUhaEBFTUsdhg4s/F0NX0YtwB0XE/pIWAUTEC5JGNDGusvJwjG2MPxdDVNEbMV6XtBn5BUZJ25NVxNYLHg+3jfHnYugqmoAvAm4FdpB0PnAf2RVcMzPro6KzIH4AnAP8H7KpMidExKxmBmZ/TtLdkjxOWAKSZkh6c832H3+3kuZKGp0uOhtIdceAJY2MiJckjQVWk13Z7943NiJ8pbYAScMjoiN1HDZozAAeAf5nwx0RcUxvTiRps4h4490c1hJ6ugh3A/BB4CGy8V9t8HPXpkY3iEjaBfgZ2fDLu4GVwPFkE+YvBbYGngT+Nr9IeTewGDgUuFHSO8huYtmPbC7o3wKnA+8C5kXEjPx1LgEOBLYCbomI8wbkDVq/SPpHst8pwBXAvwNzImLvfP/ngDeRJd4pwA8krSP7/dee5xlgSkT8VtJpwNnACGAe8MmI6JT0CnAZ8H7gLLLPpLWgukMQEfHB/OekiNh1w58DE+KgsjtwcUTsBbwInARcC/xzROwDPAzUJswR+c0oF+TbY8j+wH0WmA1cCOwFvEPS5PyYL+VTkvYB3itpn2a/KesfSQcAZwAHAQcDHyf7Xb9BRNwCLAA+EhGTI2LdJs75duBvgEMiYjLZXagfyXdvQ/aX9r4R4eTbwnoagti/3v6IWNjYcAa9pyNicf74IWA3YHRE3JP3zQRqx8Zv3uD5P42IkPQw8FxEPAwg6VGy25YXA6fkt3IPJ7stdU9gSTPejDXMocCtEfF7AEk/Bt7Tz3MeARwA/Cq7D4qtyIYBIUvGP+rn+W0Q6GkI4oI6+wI4vIGxtIL1NY87gZ4ulvx+E8/v2uBcXcBwSZOAzwEH5sMY1wBb9j1cS2g0f/4vzN7+HgXMjIhzN7LvDx73LYeehiDeFxHvA6Z2P67p69XFgpJaC7wgqbva+ShwT53jezKSLGmvlbQjMLWf8dnAuBc4QdLWkrYBTiS7XrCDpO0kbUF2LaXby0BPX3txB3CypB0gu+gt6S1NiN0SKnon3APAhsMRG+sbiqYDl0raGniKbCywTyLi1/ndho8Dy4H7GxOiNVNELMz/tdK9NsoVEfErSV/L+1aS/U67XUP2mXnDRbiacz4m6cvAbZKGka0dcRbZWhFWEnXXgpC0EzABuB74MNk/iyCr1C6NiD2aHqGZWUn1VAEfRTZncSLwbzX9LwNfbFJMZmZDQtHV0E6KCF91NTNroKJjwHtL2mvDzoj4WoPjMTMbMoom4FdqHm9JdkV3aePDMTMbOgoNQbzhSdm0ml9ExGENj8jMbIgouhzlhrYmuzBnZmZ9VGgIIr91trtUHka2mMzXmxWUmdlQUHQWxFvIFhd5D9ktlnMj4qEmx2ZmVmpFhyCOB64j+4bfzYGrJX26aVGZmQ0BRSvgJcC7alZ72gZ4MF+C0czM+qBoBSyy1b+6dfKn25LNzKwPis4DvhqYJ+nWfPsE4MrmhGRmNjQUngecL85+aL55b0QsalpUZmZDQJ9uxDAzs/7r640YZmbWT07AZmaJOAGbmSXiBGxmlogTsJlZIv8LPPFUBLygJXQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = od_preds['data']['is_outlier']\n",
    "f1 = f1_score(y_outlier, y_pred)\n",
    "print('F1 score: {}'.format(f1))\n",
    "cm = confusion_matrix(y_outlier, y_pred)\n",
    "df_cm = pd.DataFrame(cm, index=labels, columns=labels)\n",
    "sns.heatmap(df_cm, annot=True, cbar=True, linewidths=.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot instance level outlier scores vs. the outlier threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3wU9b3/8dcnECGiJoLUKqEFrVcuBg2IB/vTlh68X0ux1h6hWi1Wq6Bt1dMepP70tFbrhR5rD17AWmql3vWotM0p2vpT2yAYRbSgooIXEAEFIyTw+f0xs8sm7G5mN3vL5v18PPLIzndmZz6zszufme935jvm7oiIiABUFDsAEREpHUoKIiISp6QgIiJxSgoiIhKnpCAiInE9ix1AZ+y2224+aNCgYochItKlLFiw4AN3759sXJdOCoMGDaKxsbHYYYiIdClm9maqcao+EhGROCUFERGJU1IQEZG4Lt2mICLlpaWlhRUrVvDpp58WO5Sy0Lt3b2pra6msrIz8HiUFESkZK1asYOedd2bQoEGYWbHD6dLcnTVr1rBixQoGDx4c+X2qPhKRkvHpp5/Sr18/JYQcMDP69euX8VmXkoKIlBQlhNzJ5rNUUhARkTglBRGREnHkkUcW/YZcNTSLSJf14MKVXDvvVd5Z18yeNVX84Kj9OHnEgKLE0traSs+eXX+XmrczBTO7w8xWmdlLScZdYmZuZruFw2ZmM8xsmZk1mdnB+YpLRMrDgwtXcvn9L7JyXTMOrFzXzOX3v8iDC1dmPc/ly5dzwAEHcM455zBkyBDGjRtHc3MzixYtYvTo0QwfPpxTTjmFtWvXAsGR/ZQpU6ivr+emm25i0qRJnHfeeYwePZq99tqL+fPnc9ZZZ3HAAQcwadKk+HLOO+886uvrGTJkCFdccUUnP4ncymf10Wzg6PaFZjYQGAe8lVB8DLBP+HcucEse4xKRMnDtvFdpbtnSpqy5ZQvXznu1U/NdunQp559/PosXL6ampob77ruPM888k2uuuYampiaGDRvGT37yk/j0mzdvprGxkUsuuQSAtWvX8swzz3DDDTdw4oknMnXqVBYvXsyLL77IokWLALj66qtpbGykqamJJ598kqampk7FnEt5Swru/hTwYZJRNwA/BBIfDn0S8BsPPAvUmNke+YpNRLq+d9Y1Z1Qe1eDBg6mrqwPgkEMO4bXXXmPdunUcccQRAEycOJGnnnoqPv1pp53W5v0nnHACZsawYcPYfffdGTZsGBUVFQwZMoTly5cDMHfuXA4++GBGjBjB4sWLefnllzsVcy4VtKHZzE4CVrr7C+1GDQDeThheEZYlm8e5ZtZoZo2rV6/OU6QiUur2rKnKqDyqXr16xV/36NGDdevWpZ2+T58+Sd9fUVHRZl4VFRW0trbyxhtvcN1119HQ0EBTUxPHHXdcSd3BXbCkYGY7Av8OTOvMfNx9prvXu3t9//5JuwMXkW7gB0ftR1VljzZlVZU9+MFR++V0OdXV1ey666789a9/BeCuu+6KnzVk46OPPqJPnz5UV1fz/vvv8/jjj+cq1JwoZFP53sBg4IXwhopa4HkzGwWsBAYmTFsblomIJBW7yqgQVx/deeedTJ48mU8++YS99tqLWbNmZT2vgw46iBEjRrD//vszcOBAxowZk8NIO8/cveOpsp252SDgUXcfmmTccqDe3T8ws+OAC4BjgUOBGe4+qqP519fXe7Gv6RWR3FmyZAkHHHBAscMoK8k+UzNb4O71yabP5yWpdwPPAPuZ2QozOzvN5I8BrwPLgFuB7+YrLhERSS1v1UfufnoH4wclvHbg/HzFIiIi0aibCxERiVNSEBGROCUFERGJU1KQzDXNhRuGwvSa4H/T3GJHJCI5oqQgmWmaC49cCOvfBjz4/8iFSgzSLc2ePZt33nknPpzY9fWxxx7b4d3QpUhJQTLTcCW0tOtbpqU5KBcptCKftbZPCokee+wxampqIs9ry5YtHU9UAEoKkpn1KzIrF8mXPJ21Xn/99QwdOpShQ4dy4403snz5coYO3Xb/7XXXXcf06dO59957aWxs5IwzzqCuro7m5rYHS4MGDeKDDz4A4Le//S2jRo2irq6O73znO/EEsNNOO3HJJZdw0EEH8cwzz3Qq7lxRUpDMVNdmVi6SL3k4a12wYAGzZs3iueee49lnn+XWW2+NPzuhvfHjx1NfX8+cOXNYtGgRVVXJO+JbsmQJ99xzD08//TSLFi2iR48ezJkzB4CNGzdy6KGH8sILL3D44YdnHXcudf3HBElhjZ0WHI0l/hgrq4JykULKw1nr3/72N0455ZR4z6ennnpqvCO8bDU0NLBgwQJGjhwJQHNzM5/5zGeAoBfWr371q52af64pKUhmhk8I/jdcGfz4qmuDhBArFymU6tqw6ihJeQ6tW7eOrVu3xocz7eba3Zk4cSI//elPtxvXu3dvevTokeRdxaPqI8nc8Akw9SWYvi74r4QgxTB2WnCWmqiTZ61f/OIXefDBB/nkk0/YuHEjDzzwAMcccwyrVq1izZo1bNq0iUcffTQ+/c4778zHH3+cPsyxY7n33ntZtWoVAB9++CFvvvlm1jHmm84URKRrysNZ68EHH8ykSZMYNSropPnb3/42I0eOZNq0aYwaNYoBAwaw//77x6efNGkSkydPpqqqKmVD8YEHHshVV13FuHHj2Lp1K5WVldx88818/vOfzzrOfMpr19n5pq6zRcqLus7OvZLpOltERLoeJQUREYlTUhARkTglBRERiVNSEBGROCUFERGJU1IQEQHWrFlDXV0ddXV1fPazn2XAgAHU1dVRU1PDgQcemPPlzZ8/n+OPPz6j9yR2zZ1o9uzZXHDBBTmJK29JwczuMLNVZvZSQtm1ZvaKmTWZ2QNmVpMw7nIzW2Zmr5rZUfmKS0QkmX79+rFo0SIWLVrE5MmTmTp1any4oqLjXWVra2sBosy/fJ4pzAaOblf2J2Couw8H/glcDmBmBwJfB4aE7/mVmZVWhyAi0m1t2bKFc845hyFDhjBu3Lh4N9lHHnkkU6ZMob6+nptuuonVq1fz1a9+lZEjRzJy5EiefvppAJ588sn4WciIESPiXWNs2LCB8ePHs//++3PGGWcQu5m4oaGBESNGMGzYMM466yw2bdq0XUyzZs1i3333ZdSoUfHl5ELeurlw96fMbFC7sj8mDD4LjA9fnwT83t03AW+Y2TJgFFAaHYyLSMFNeWIKi95blNN51n22jhuPvjHj9y1dupS7776bW2+9lQkTJnDffffxzW9+E4DNmzfHq3S+8Y1vMHXqVA4//HDeeustjjrqKJYsWcJ1113HzTffzJgxY9iwYQO9e/cGYOHChSxevJg999yTMWPG8PTTT1NfX8+kSZNoaGhg33335cwzz+SWW25hypQp8XjeffddrrjiChYsWEB1dTVf+tKXGDFiRA4+oeK2KZwFPB6+HgAkdne4Iizbjpmda2aNZta4evXqPIcoIgKDBw+mrq4OgEMOOYTly5fHx5122mnx13/+85+54IILqKur48QTT+Sjjz5iw4YNjBkzhosvvpgZM2awbt06evYMjsdHjRpFbW0tFRUV1NXVsXz5cl599VUGDx7MvvvuC8DEiRN56qmn2sTz3HPPceSRR9K/f3922GGHNjF0VlE6xDOzHwGtwJxM3+vuM4GZEPR9lOPQRKREZHNEny+9evWKv+7Ro0ebp6zFnr0AsHXrVp599tn4mUDMZZddxnHHHcdjjz3GmDFjmDdvXtL5lkK7RMHPFMxsEnA8cIZv641vJTAwYbLasExEpMsYN24cv/zlL+PDixYF1V+vvfYaw4YN49JLL2XkyJG88sorKeex3377sXz5cpYtWwbAXXfdxRFHHNFmmkMPPZQnn3ySNWvW0NLSwh/+8IecrUNBk4KZHQ38EDjR3T9JGPUw8HUz62Vmg4F9gL8XMjYRkc6aMWMGjY2NDB8+nAMPPJBf//rXANx4440MHTqU4cOHU1lZyTHHHJNyHr1792bWrFl87WtfY9iwYVRUVDB58uQ20+yxxx5Mnz6dww47jDFjxuS0Z9m8dZ1tZncDRwK7Ae8DVxBcbdQLWBNO9qy7Tw6n/xFBO0MrMMXdH28/z/bUdbZIeVHX2bmXadfZ+bz66PQkxbenmf5q4Op8xSMiIh3THc0iIhKnpCAiJaUrPw2y1GTzWSopiEjJ6N27N2vWrFFiyAF3Z82aNdtdHtuRotynICKSTG1tLStWrEA3puZG7969qa2tzeg9SgoiUjIqKysZPHhwscPo1lR9JCIicUoKIiISp6QgIiJxSgoiIhKnpCAiInEdJgUz29HM/sPMbg2H9zGzzB4sKiIiXUKUM4VZwCbgsHB4JXBV3iISEZGiiZIU9nb3nwMtAGGX15bXqEREpCiiJIXNZlYFOICZ7U1w5iCSnaa5cMNQmF4T/G+aW+yIRCQU5Y7mK4AngIFmNgcYA0zKZ1BSxprmwiMXQkv4OMP1bwfDAMMnFC8uEQE6OFMwMwNeAU4lSAR3A/XuPj/vkUl5arhyW0KIaWkOykWk6NKeKbi7m9lj7j4M+J8CxSTlbP2KzMpFpKCitCk8b2Yj8x6JdA/VKXpsTFUuIgUVJSkcCjxjZq+ZWZOZvWhmTfkOTMrU2GlQWdW2rLIqKBeRoovS0HxU3qOQ7iPWmNxwZVBlVF0bJAQ1MouUhA6Tgru/aWYHAV8Mi/7q7i909D4zuwM4Hljl7kPDsr7APcAgYDkwwd3Xhg3aNwHHAp8Ak9z9+cxXR7qE4ROUBERKVJRuLi4C5gCfCf9+a2bfizDv2cDR7couAxrcfR+gIRwGOAbYJ/w7F7glSvAiIpJbUdoUzgYOdfdp7j4NGA2c09Gb3P0p4MN2xScBd4av7wROTij/jQeeBWrMbI8oKyAiIrkTJSkYsCVheAvZd3Oxu7u/G75+D9g9fD0AeDthuhVh2fbBmJ1rZo1m1qjnuIqI5FaUhuZZwHNm9kA4fDJwe2cXHN4D4Vm8byYwE6C+vj7j94uISGpRGpqvN7P5wOFh0bfcfWGWy3vfzPZw93fD6qFVYflKYGDCdLVhmYiIFFCUhubRwFJ3n+HuM4DXzOzQLJf3MDAxfD0ReCih/EwLjAbWJ1QziYhIgURpU7gF2JAwvIEIVweZ2d3AM8B+ZrbCzM4Gfgb8q5ktBb4SDgM8BrwOLANuBb4beQ1ERCRnorQpmLvH6+7dfauZRal2Oj3FqLFJpnXg/AixiIhIHkU5U3jdzC40s8rw7yKCo3oRESkzUZLCZOBfCBp+VxL0hXRuPoMSEZHiiFINtAr4egFiERGRIkt5pmBm55jZPuFrM7M7zGx92FPqwYULUURECiVd9dFFBJ3WAZwOHATsBVxM0HmdiIiUmXRJodXdW8LXxxP0TbTG3f8M9Ml/aCIiUmjpksJWM9vDzHoTXEb654RxVSneIyIiXVi6huZpQCPQA3jY3RcDmNkR6JJUEZGylDIpuPujZvZ5YGd3X5swqhE4Le+RiYhIwaW9JNXdW4G17co25jUiEREpmig3r4mISDehpCAiInEpq486ukHN3Z/PfTgiIlJM6doUfpFmnANfznEsIiJSZOmuPvpSIQMREZHii/LktR3N7MdmNjMc3sfMjs9/aCIiUmhRGppnAZsJus+GoPvsq/IWkYiIFE2UpLC3u/8caAFw908Ay2tUIiJSFFGSwmYzqyJoXMbM9gY25TUqEREpiijPaJ4OPAEMNLM5wBhgUh5jku6kaS40XAnrV0B1LYydBsMnFDsqkW4rypPX/mhmC4DRBNVGF7n7B51ZqJlNBb5NcPbxIvAtYA/g90A/YAHwb+6+uTPLkRLXNBceuRBamoPh9W8Hw6DEIFIkUa4+egQYB8x390dzkBAGABcC9e4+lKAX1q8D1wA3uPsXCPpbOrszy5EuoOHKbQkhpqU5KBeRoojSpnAd8EXgZTO718zGh89Y6IyeQJWZ9QR2BN4luBnu3nD8ncDJnVyGlLr1KzIrF5G86zApuPuT7v5dgkdx/jcwAViV7QLdfSVBonmLIBmsJ6guWhf2ygqwAhiQ7P1mdq6ZNZpZ4+rVq7MNQ0pBdW1m5SKSd5E6xAuvPvoqMBkYSXAknxUz2xU4CRgM7EnwaM+jo77f3We6e7271/fv3z/bMKQUjJ0Gle0e4ldZFZSLSFF02NBsZnOBUQRXIP0X8KS7b+3EMr8CvOHuq8P5309wRVONmfUMzxZqCW6Sk3IWa0zW1UciJSPKJam3A6e7+5YcLfMtYLSZ7Qg0Ezz/uRH4CzCe4AqkicBDOVqelLLhE5QEREpIlOqjvwKX56rvI3d/jqBB+XmCy1ErgJnApcDFZraM4LLU27NdhoiIZCfKmcIsgobgxL6P/gA8mu1C3f0K4Ip2xa8TVFOJiEiRqO8jERGJU99HIiISF6X66ArU95GUCvWVJJJXUfo++pOZPU9C30fADvkOTGQ76itJJO8i3bzm7mvc/X8S+j56Ns9xiWxPfSWJ5F2kpJCEGpql8NRXkkjeZZsUPKdRiEShvpJE8i5lm4KZ/ZLkO38DavIWkUgqY6e1bVOA7tFXkhrXpYDSNTQ3ZjlOJD+6Y19JalyXAjP3rlsTVF9f742Nyk8SQVc92r5haJAI2qseCFNfKnw8UhbMbIG71ycbF+U+BZGurSsfbatxXQos24Zmka6jK1/KqsZ1KTAlBSl/XfloWw8ikgLrMCmY2b5m1mBmL4XDw83sx/kPTSRHuvLR9vAJcMKMoA0BC/6fMCN5tVfT3KANYnpN8L9pbsHDla4vSpvCrcAPCJ7PjLs3mdnvgKvyGZhIznT1S1mjPIioK7ebSEmJUn20o7v/vV1Zaz6CEcmLTI62u6qu3G4iJSXKmcIHYXfZsa6zxwPv5jUqkVwr98d+duV2EykpUZLC+QSPy9zfzFYCbwDfzGtUIpKZ6toU9zN0gXYTKSkdVh+5++vu/hWgP7C/ux/u7svzHpmIRKerlCRHolx99J9mVuPuG939YzPb1czUyCxSSrpDu4kURIfdXJjZQncf0a7seXc/OOuFmtUAtwFDCdoqzgJeBe4BBgHLgQnuvjbdfNTNhYhI5tJ1cxHl6qMeZtYrYWZVQK8000dxE/CEu+8PHAQsAS4DGtx9H6AhHBbJD13TL5JUlIbmOUCDmc0Kh78F3JntAs2sGvg/hM95dvfNwGYzOwk4MpzsTmA+cGm2yxFJSdf0i6QUpaH5GuBq4IDw7/+6+887sczBwGpglpktNLPbzKwPsLu7xy51fQ/YPdmbzexcM2s0s8bVq1d3Igzptsr9mn6dBUknROol1d0fBx7P4TIPBr7n7s+Z2U20qypydzezpI0d7j6T4BJZ6uvru26/31I85XxNv86CpJOiXH10qpktNbP1ZvaRmX1sZh91YpkrgBXu/lw4fC9BknjfzPYIl7kHsKoTyxBJrSv3hdSRcj8LkryL0tD8c+BEd692913cfWd33yXbBbr7e8DbZrZfWDQWeBl4GJgYlk0EHsp2GSJplfM1/eV8FiQFEaX66H13X5Lj5X4PmGNmOwCvEzReVwBzzexs4E1A57qSH+X8WE/d2SydFOU+hZuAzwIPApti5e5+f35D65juUxBpp32bAgRnQQd9A5b+sfySoGSls4/j3AX4BBiXUOZA0ZOCiLST7Cxon3Hwwu/U+CyRdHimUMp0piASwQ1DU1QpDYSpLxU+Him6Tp0pmFlv4GxgCNA7Vu7uZ+UsQhHJn0wan5vmlmdbi0QW5eqjuwjaFI4CngRqgY/zGZSI5FDUS3Bj7RHr3wZ8WzWTbn7rVqIkhS+4+38AG939TuA44ND8hiUiORP1Etxs7nHQ3dNlJ0pSaAn/rzOzoUA18Jn8hSQiORW1W+1M73Eo1pmFElFeRbn6aKaZ7Qr8mOAGs52A/8hrVCKSW1EeR5rpPQ7pzizy1Q6hbjzyLsqZQoO7r3X3p9x9L3f/DPDHfAcmIgWW6Z3eKc8s3s7fkby68ci7KEnhviRl9+Y6EBEpskyf3pbyLmnLX5VSR4lIVUmdlrL6yMz2J7gMtdrMTk0YtQsJl6aKSBmJUs0UM3ba9ndPYwT3tibIZZVSqiouUFVSjqQ7U9gPOB6oAU5I+DsYOCf/oYlISUt2ZtE+IcTkqkO+ZFVciVSV1GlR+j46zN2fKVA8GdEdzSIlphB3T8dvsEtxxgAESUo336XS2Wc0n2Jmu5hZpZk1mNlqM/tmjmMUkXJQiG7Jh08IEkz1wDQTRWzP0OWt24mSFMa5+0cEVUnLgS8AP8hnUCLSRWXaWN0ZHVUlQfrqJN3BnVSU+xQqw//HAX9w9/VmlseQRKRLy6SxurPLgW19NWXantHR5a3dtA+oKEnhETN7BWgGzjOz/sCn+Q1LRCSCxASUsj0jxaWz6S5v7cY3yHVYfeTulwH/AtS7ewuwETgp34GJiGQk0/aMVMnCeiQ/g7j/nG7R7hClTQFgf+A0MzsTGE/bB+6IiBRfpu0ZqZKIb0m9jHTtDmXSaB3leQp3AXsDi4DYp+XAb/IYl4hI5jJpz0j1rO6OLndNdjNeGfXJFKVNoR440LvyI9pERJJJlUS2u1O7nfbtEcXoHDBPolQfvUTwkJ2cMrMeZrbQzB4Nhweb2XNmtszM7jGzHXK9TBGRDrWphkqhfXtEukbrLiZKUtgNeNnM5pnZw7G/HCz7ImBJwvA1wA3u/gVgLcEjQEVECi92g9ypt0ZrvE7XOeCjF3eptoYo1UfTc71QM6sluO/hauBiC258+DLwjXCSO8Pl3pLrZYuIRJaq3aF9ldDYaXD/uWx/r4RD4x3byrtAW0OHfR/lZaFm9wI/BXYGvg9MAp4NzxIws4HA4+4+NMl7zwXOBfjc5z53yJtvvlmosEVEUpteHX3aqr5w6Rv5i6UDWfV9ZGYfm9lHSf4+NrOPOhHM8cAqd1+Qzfvdfaa717t7ff/+/bMNQ0Qkt9L2xdRO84clW42UsvrI3XfO0zLHACea2bEEz2XYBbgJqDGznu7eCtQCK/O0fBGR3Iv6fImYByYH/0usGinqzWs54+6Xu3utuw8Cvg78r7ufAfyF4MY4gInAQ4WOTUQka8lunqs/K/X0vqUkO+CL0tBcKJcCvzezq4CFwO1FjkdEJDPJ7ntY/EBQXZRMCd7LUPAzhUTuPt/djw9fv+7uo9z9C+7+NXffVMzYRERy4phr0nfxnaun0uVIKZ0piIiUn9hZwAOTk/erlPIeh+Io6pmCiEi3MHwCnPLr/D+VLgeUFERECqGQT6XrBFUfiYgUSqGeStcJOlMQEZE4JQUREYlTUhARkTglBRERiVNSEBGROCUFERGJU1IQEZE4JQUREYlTUhARkTglBRERiVNSEBGROCUFERGJU1IQEZE4JQUREYlTUhARkTglBRERiSt4UjCzgWb2FzN72cwWm9lFYXlfM/uTmS0N/+9a6NhERLq7YpwptAKXuPuBwGjgfDM7ELgMaHD3fYCGcFhERAqo4EnB3d919+fD1x8DS4ABwEnAneFkdwInFzo2EZHurqhtCmY2CBgBPAfs7u7vhqPeA3ZP8Z5zzazRzBpXr15dkDhFRLqLoiUFM9sJuA+Y4u4fJY5zdwc82fvcfaa717t7ff/+/QsQqYhI91GUpGBmlQQJYY673x8Wv29me4Tj9wBWFSO2rDXNhRuGwvSa4H/T3GJHJLmibSvdSDGuPjLgdmCJu1+fMOphYGL4eiLwUKFjy1rTXHjkQlj/NuDB/0cu7H47j3LceTbNhQe/23bbPvjd1OtWjp+BdCsW1NQUcIFmhwN/BV4EtobF/07QrjAX+BzwJjDB3T9MN6/6+npvbGzMY7QR3TA03Gm0Yz3At0J1LYydBsMnFD62XGuaCw1XwvoVbdcrlhhbmrdNW1kFB30Dlv5x++nzEUM+XDMYmpN8DSv7wI/e2T6uB78LW1u2lVVUwsm/Ko9tL2XDzBa4e33ScYVOCrlUMklheg0pmkC2qayCE2Z07Z1Dqh3/CTPCnXSSxJhMVV845prtP4vEnX1VeJtK89qOk09M9cDcJ4jp1anHnXpr22WlSiCx9U1ct9ZN0LKx7fhkcTfNhccv3TbfdNMWUyETtXSakkK+pTpTaK96IEx9KX9x5PuHmWo9qwcGy+woMSZqnyTT7ewTp+8o+eQ6+aZLCu23Z7ppK6tSrxtAjx3gpJvbxp3szCOm/mz43OjkSTRdQt1u+g/DM9ot25IqZPY9SrXtrCI8U06SrDtKdu1jjZpEJRIlhXzraIeWaPr6AsZgUH8WHH99yrdlJOUZkQU7j6hnCjGJO9UoiTVq8sll8k119A+AwfR12wbTJYUo2sfd0WfSYwfYsrnj+caq8V74Xcff0YpKMGs7344SbZRtlziPVMkulhgh2u8plnSSJbXEhFPZB3r22j5JJpPuwCrTg65sDtIKdMaVLimo76NcGD4h+MJXDwQs+JKmcs3g7BofO2rAbLgyyY/IofH27JfZXnVt6vKx04IffibWr0j+Ot30qWJINd/OOuaa1OMSY2maG+ykOqN93B2tR5SEAMH3YsHsaActW1u2n29Lc/D9SiXK5504j4Yrk5/9bNkcjEv6XU7CwyZJ3xLGEV4E8MDktom8ZWM4HF4ocP85yX8TyS4Yuf9cePTi5BccpJpPqnl1dPFJiVywoqSQK8MnBEd509fBKb8GLPl0zR8GX7Tp1dGvTonyZUn3w2z+MP0XOKpkO/7Kqm1HM4mJsXog7NAn/fwSd6pRdvax5JPqs81kXlENnxBU1bRfZmy9Ydv2ie2kstU+7hyuh8d2nNlK9/2KGmdsHunmtX5F55L61pZtSSKd5g+3/w2lPLC6Ax6ZkjyRJZtPqnl1lFyzeU8eKCnkw/AJpK/iCMfFjkQ6ShBRvixRfpipvsBRJdvxJ1YrJCbGqS/B8TcGVQLJJO5UoeMzjcTkU38WKRND+/nmwvHXw6kzU6931CPbdHrssH3cY6cF1Tk5sKWzP/V036+oZ4mxeaSbV3VtbpN6Ou1/QymTkW9rz4gyn3Tz6ighZvqePFBSSKcz15xXD4w4YbsE8ejF208S5csS5QgaOn/k0X7Hn66+cxdopyUAAA1nSURBVPiEoI449lnEqtXa71Rj0yYmnKq+wV+ynXCbnXQH882VdOud8Y/WgnrumKq+2zcyx5Z58q/aThtTUZk64bbzie/A71q/HG3HnWy+HSXa2Lar6pt6msR5pEp2scSYTVVkthK3XWeSUfvvQLqq1lSyeU8edMuG5ilPTGHRe4vST7RxNXywtG2VgFXAbvtAnwjdayR7f1T992u7jBX/CK6+aK9nL6gduW14zWvw8bvbT5fMoMMzj0uSS7V9KnoG2z/b71Cijath7fJgOT17wa6DgvJYWUXPYHhrKy30wICebGETlbzt/fm4R19G9GtJOj1m4J58vrGyqPEmxtl+vonz2Lg6+L5ubd32WfXbe9s0beZD9AvbLDwwirpfS/wNbVwNq19NMd8OYmj/W8xm/5Hhe+o+W8eNR9+YJqjU0jU098xqjuUq2Rc6kW8Nxkf5gcSmSfziR9V+GbsOSv5lif2AY/rtDb136XiZPXtlFo+kl2r79Ns7eJ3tDjZRn/7J39eu7IMNm3j9g41s3brtu1tRYezVtwr61ERfdjYxposz0+kSx6c6wIrtqJMltcTfQOxMsn1bQ/vfUJ/+8OlH2x9YWQXstDtseC95skn2W4zFnsm2z+Y9edAtk0LS7BprLGztCfRMfVTQajBpfmYLjF9m9jaRDnuSLSOby+ESL8uLKYeb6EpRCd289eDClVw771XeWdfMnjVV/OCo/Th5xICixJIzufh8o84j3V37sd9x+8tgy+j31C2rj5Iq1A1obRJEnpaRdHnF31mJSGlQ9VEUURoLc3Fly/AJ23bKj14cXO6WeOaQ66tnEpcnItIBXX0Uk6qF33qQ9AqYXOjoUkcRkQLTmULM2GmpO3vL505aR/IiUkJ0phDT0Y1ZIiLdQPc9U0jVAKskIFI2yvJKrDzrnkmhfY+isb6EQElBpEw8uHAll9//Is0twf0JK9c1c/n9LwIoMaTRPauPSqTjKRHJn2vnvRpPCDHNLVu4dl6Ku5YF6K5JoUQ6nhKR/HlnXfJOClOVS6B7Vh+leiBMgTueksx0p/rhbNe1O31GHdmzpoqVSRLAnjUF6myvi+qedzQneUrZJ74Dl7V8m4e3Bh3FGfAve/dl+ZrmpD+wzvz4Mnlvumm70w6gff0wQFVlD3566rC8rHM+PtsHF65k+sOLWdcc9Mu/646VXHHCEE4eMaDN8qqrKtm4uZWWLdt+mwacMfpzXHXysLTzL8Rn1FW+d+k+j8Y3P2TOs2/Fbxvts0MPrj4l2ueUbP2BNtsWoMJgq8OAFJ9Rqs+xEJ9vl3ocp5kdDdwE9ABuc/efpZo2m6Tw4MKV/O2BX3E5s+lrGwBYy05MbzkznhC6qhMr/sYPe85lT/uAd3w3ft46IS/rlG45hYoh2/iKpRRjkq6vZ4Vx3dcOyjhpdJluLsysB3Az8K/ACuAfZvawu7+ci/k/uHAl8//wX/xn5W3saNseOdjbIz7WsISdWPE3fpawXrX2AT+rvA1ayOnOJ91ygILEkG18xdoJl2JMUh5atzpT7wkeA5Crs4lSa2geBSxz99fdfTPwe+CkXM382nmv8v2ec9skBIAdbTM/7FnY56Dm2g8LtF7pllOoGLKNr1hKMSYpHw45vaKqpKqPzGw8cLS7fzsc/jfgUHe/IGGac4Fzw8H9gMifxg6f/cIhw+z1VEvnRR+cXeAloFDrlW45qbsEL9xnW4rbtxRjkvKz+b1lCzKY/PPunvRBDSVVfRSFu88EZnZ2PmbWmKpOrVxpnbsHrXP3kK91LrXqo5VA4sONa8MyEREpgFJLCv8A9jGzwWa2A/B14OEixyQi0m2UVPWRu7ea2QXAPIJLUu9w98V5Wlynq6C6IK1z96B17h7yss4l1dAsIiLFVWrVRyIiUkRKCiIiEtctk4KZHW1mr5rZMjO7rNjx5IqZDTSzv5jZy2a22MwuCsv7mtmfzGxp+H/XsNzMbEb4OTSZ2cHFXYPsmFkPM1toZo+Gw4PN7Llwve4JL1rAzHqFw8vC8YOKGXdnmFmNmd1rZq+Y2RIzO6yct7OZTQ2/0y+Z2d1m1rsct7OZ3WFmq8zspYSyjLermU0Mp19qZhMziaHbJYWErjSOAQ4ETjezA4sbVc60Ape4+4HAaOD8cN0uAxrcfR+gIRyG4DPYJ/w7F7il8CHnxEXAkoTha4Ab3P0LwFrg7LD8bGBtWH5DOF1XdRPwhLvvDxxEsP5luZ3NbABwIVDv7kMJLkL5OuW5nWcDR7cry2i7mllf4ArgUIJeIq6IJZJI3L1b/QGHAfMShi8HLi92XHla14cI+pF6FdgjLNsDeDV8/d/A6QnTx6frKn8E97I0AF8GHiW4tfoDoGf77U1wVdth4eue4XRW7HXIYp2rgTfax16u2xkYALwN9A2326PAUeW6nYFBwEvZblfgdOC/E8rbTNfRX7c7U2DbFyxmRVhWVsJT5hHAc8Du7v5uOOo9YPfwdTl8FjcCPwS2hsP9gHXu3hoOJ65TfH3D8evD6buawcBqYFZYbXabmfWhTLezu68ErgPeAt4l2G4LKP/tHJPpdu3U9u6OSaHsmdlOwH3AFHf/KHGcB4cOZXEdspkdD6xy90z6fCkHPYGDgVvcfQSwkW1VCkDZbeddCTrGHAzsCfRh+yqWbqEQ27U7JoWy7krDzCoJEsIcd78/LH7fzPYIx+8BrArLu/pnMQY40cyWE/So+2WCuvYaM4vdmJm4TvH1DcdXA2sKGXCOrABWuPtz4fC9BEmiXLfzV4A33H21u7cA9xNs+3LfzjGZbtdObe/umBTKtisNMzPgdmCJu1+fMOphIHYFwkSCtoZY+ZnhVQyjgfUJp6klz90vd/dadx9EsB3/193PAP4CjA8na7++sc9hfDh9lzuadvf3gLfNbL+waCzwMmW6nQmqjUab2Y7hdzy2vmW9nRNkul3nAePMbNfwLGtcWBZNsRtVitSQcyzwT+A14EfFjieH63U4wallE7Ao/DuWoD61AVgK/BnoG05vBFdivQa8SHB1R9HXI8t1PxJ4NHy9F/B3YBnwB6BXWN47HF4Wjt+r2HF3Yn3rgMZwWz8I7FrO2xn4CfAK8BJwF9CrHLczcDdBu0kLwRnh2dlsV+CscP2XAd/KJAZ1cyEiInHdsfpIRERSUFIQEZE4JQUREYlTUhARkTglBRERiVNSkJJiZm5mv0gY/r6ZTc/RvGeb2fiOp+z0cr4W9lz6l3blgxJ7v8xwnpPMbM/cRCiSmpKClJpNwKlmtluxA0mUcOdsFGcD57j7l3IYwiSCLh5E8kpJQUpNK8GzZ6e2H9H+SN/MNoT/jzSzJ83sITN73cx+ZmZnmNnfzexFM9s7YTZfMbNGM/tn2HdS7HkM15rZP8J+6b+TMN+/mtnDBHfQto/n9HD+L5nZNWHZNIKbCG83s2tTrWR45H+/mT0R9nn/84RYZofzfNGC5wiMB+qBOWa2yMyqzGxaGO9LZjYzvNMXM5tvZteE6/5PM/tiwnyvC6dvMrPvheWHhJ/dAjObl9CdwoUWPJejycx+H3XjSRko9h18+tNf4h+wAdgFWE7QZ833genhuNnA+MRpw/9HAusIug3uRdDPy0/CcRcBNya8/wmCg6F9CO4Y7U3QF/2Pw2l6EdwpPDic70ZgcJI49yTofqE/QQd1/wucHI6bT5K7hknoEpngyP/1cB17A28S9FdzCPCnhPfUJJsn4V2t4eu7gBMSpvtF+PpY4M/h6/MI+kiKdTXdF6gE/h/QPyw7DbgjfP0O2+4Qrin290J/hfvTmYKUHA96dv0NwYNVovqHu7/r7psIbvv/Y1j+IsHOOGauu29196UEO+X9CfqGOdPMFhF0Nd6PIGkA/N3d30iyvJHAfA86aWsF5gD/J4N4IXhwynp3/5TgTOTzYUx7mdkvzexo4KMU7/2SBU8Ve5GgI8AhCeNiHSEuYNu6f4Wgj/1WAHf/ENgPGAr8KVz3HxN0ngZB9xlzzOybBGdv0k1kUk8qUkg3As8DsxLKWgmrPM2sAtghYdymhNdbE4a30vZ73r5fFyfoQ+Z77t6m0zAzO5LgTCFfEmPeQnAUv9bMDiJ4iMxkYAJBPzaJcfUGfkVw5vB22BDfO8l8t5D+N27AYnc/LMm44wiS3AnAj8xsmG97doGUMZ0pSEkKj2Tnsu0RixBUKR0Svj6RoPojU18zs4qwnWEvgqdVzQPOs6DbccxsXwseWpPO34EjzGw3Cx7xejrwZBbxtBE2sFe4+30ER+6x5+5+DOwcvo4lgA8seHZGlCuq/gR8J9ZgbsEjG18F+pvZYWFZpZkNCRPuQHf/C3ApQRXXTp1dN+kadKYgpewXwAUJw7cCD5nZCwRtA9kcxb9FsEPfBZjs7p+a2W0E1SzPhw22q4GT083E3d81s8sIum824H/c/aF074loAMET1WIHbJeH/2cDvzazZoJHT95K0GPoewTdwXfkNmBfoMnMWoBb3f2/wkbsGWZWTbA/uJGgB+HfhmUGzHD3dTlYN+kC1EuqiIjEqfpIRETilBRERCROSUFEROKUFEREJE5JQURE4pQUREQkTklBRETi/j+DOpFv7s7feAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_instance_score(od_preds, y_outlier, labels, od.threshold, ylim=(0, 150))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use OHE instead of ordinal encoding for the categorical variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we will apply one-hot encoding (OHE) on the categorical variables, we convert `cat_vars_ord` from the ordinal to OHE format. `alibi_detect.utils.mapping` contains utility functions to do this. The keys in `cat_vars_ohe` now represent the first column index for each one-hot encoded categorical variable. This dictionary will later be used in the counterfactual explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 3, 3: 66, 69: 11}\n"
     ]
    }
   ],
   "source": [
    "cat_vars_ohe = ord2ohe(X_fit, cat_vars_ord)[1]\n",
    "print(cat_vars_ohe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit a one-hot encoder on the categorical data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OneHotEncoder(categorical_features=None, categories='auto', drop=None,\n",
       "              dtype=<class 'numpy.float64'>, handle_unknown='error',\n",
       "              n_values=None, sparse=True)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc = OneHotEncoder(categories='auto')\n",
    "enc.fit(X_fit[:, :n_categories])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transform `X_fit` to OHE:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(494021, 98)\n"
     ]
    }
   ],
   "source": [
    "X_ohe = enc.transform(X_fit[:, :n_categories])\n",
    "X_fit = np.array(np.c_[X_ohe.todense(), X_fit[:, n_categories:]].astype(np.float32, copy=False))\n",
    "print(X_fit.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize and fit outlier detector\n",
    "\n",
    "Initialize:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "od = Mahalanobis(threshold,\n",
    "                 n_components=n_components, \n",
    "                 std_clip=std_clip, \n",
    "                 start_clip=start_clip,\n",
    "                 cat_vars=cat_vars_ohe,\n",
    "                 ohe=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply fit method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "od.fit(X_fit,\n",
    "       d_type=d_type,\n",
    "       disc_perc=disc_perc,\n",
    "       standardize_cat_vars=standardize_cat_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run outlier detector and display results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transform outlier batch to OHE:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 98)\n"
     ]
    }
   ],
   "source": [
    "X_ohe = enc.transform(X_ord)\n",
    "X_outlier = np.array(np.c_[X_ohe.todense(), X_num].astype(np.float32, copy=False))\n",
    "print(X_outlier.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "od_preds = od.predict(X_outlier, return_instance_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F1 score and confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.9375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXqElEQVR4nO3deZxcVZn/8c83hLBJNsISE0YCYiEgBAiCgiOCCGFkE4YJiiSMY+OI4jijjKgj44K/n75kUH7yA5o1gCwGZYyZqCDbsE1iSGJYQg1h8ZVkCEEDASQGuvuZP+5tLUNSfbu7qk/X7e+b13l13XNv3XqKrjx5cu65pxQRmJnZwBuWOgAzs6HKCdjMLBEnYDOzRJyAzcwScQI2M0vECdjMLJHhqQNoNZVK5TPAxwEBl1er1e/283zTgS/nm9+oVqszK5XK1sAsYDegE/hptVr9Qn9exwalnYFrgR2BANqB7yWNyAaUPA+4uEqlsjdwE/BO4DXg58AnqtXqsgLPvRuYUa1Wn6npGwssAKaQ/QF8CDgAWA8cVK1W76pUKiOAO4BvVqvVnzX0DVlq4/O2ENiW7Pd/AvBYyqBs4LgC7p23A/Oq1eqrAJVK5R7gQ5VK5UfAxcD2wKvAx6vV6uMFzncUcHu1Wl2Tn+924OhqtXojcBdAtVp9rVKpLAQmNvzdWGrP5g3gZWApMAEn4CGjbgKWtH+9/RGxsLHhDHqPAOdXKpXtgHXAMWQVbDtZJfxEpVI5CPj/wOEFzjcBWF6zvSLv+6NKpTIaOBb/07TsdgH2A+YljsMGUE8V8AV19gXFkkxpVKvVpZVK5VvAbcDvgcXAVsC7gVmVSqX70C0AKpXKGcBn8r63AnMrlcprwNPVavXEnl6vUqkMB24ELqpWq0818r3YoPIm4EfAPwAvJY7FBlBTxoAltQFtAJdddtkBZ3zo/Q1/jcHgu5dew7jtxnDFdTdz9+wb6h4741PncP6X/okJ43f8Y9/c2+/mV4uWcN45ZwPw1W9fxIH77cMxRx4GwJe/+W9svdVWfPGzf9+095DC5uN2BWD4iAk9HFl+w4cPZ/a/z+S22+/hu99rTx1OUh2vrYTs4na/vP7bpwontc3H7drv1+uPwtPQJO0t6RRJp3e3TR0bEe0RMSUiprS1tTUm0kHidy+8CMCzq1Zzxz33c9zRRzBh/E784s57AYgIHn+iWLF6yEEH8MD8hax96WXWvvQyD8xfyCEHHQDARe0zeeWVV/nCZ85szhuxQeHy9gtY+viyIZ98G6qrs3hLrNBFOEnnAYcBewJzganAfWRTaIaUz37xG7z40ksMHz6cL/3TJxm57Zv41nnn8PXvfJ/LZt5IR0cHU494L3vsvmuP5xo1clvOnHEq0/4uG6X4xBkfZtTIbVm1+nnaZ97EpLfszF+f8WkATj3pWE4+7uimvjcbWIe8+0A+etrJLHn4MRb86jYA/uVf/i8/+/mdiSNrcdGVOoLCCg1BSHoY2BdYFBH7StoRuD4ijizwGvH6bz18aX/iIQjbmIYNQTy7tPgQxPi3Jx2CKDoNbV1EdEnqkDQSWE02idzMbFCJFqqAiybgBZJGA5eTTRZ/BXiwaVGZmfVVV8kScER8Mn94qaSfAyMjYknzwjIz66MSVsBI2odssvjwfPutEfHjJsVlZtY3na+njqCworMgrgL2AR4Fuv96CcAJ2MwGl7INQQAHR8SeTY3EzKwBWukiXNEbMR6U5ARsZoNfV1fxlljRCvhasiS8imypRAEREfs0LTIzs75ooQq4aAK+Evgo8DB/GgM2Mxt8ynYRDng+ImY3NRIzs0YYBEMLRRVNwIsk3QD8lGwIAgBPQzOzQaeEQxBbkSXeD9T0eRqamQ0+ZaqAJW0GLImICwcgHjOzfolIv8xkUT1OQ4vs3Zw6ALGYmfVfdBVviRUdgrhf0veBm8m+igcYkt8JZ2aDXWdH6ggKK5qAJ+c/v1bTN+S+E87MWsAg+KaLooquhva+ZgdiZtYQg2Booaiii/GMAs4D/jLvugf4WkSsbVZgZmZ90kKzIIquBXEV8DJwSt5eAq5uVlBmZn1Wwotwu0XESTXbX5W0uBkBmZn1SwtVwIW/E07SoRFxH4CkQ4B1zQvLzKxvooRrQfw9MDMfCwZ4AZjenJDMzPqhhBXwUuDbwG7AaGAtcALg74Uzs8FlEIztFlU0Af8EeBFYCKxsXjhmZv1Uwgp4YkQc3dRIzMwaoYQV8AOS3hERDzc1GjOz/irhrciHAjMkPY2/ksjMBrMSDkFMbWoUZmaNUrYEHBG/aXYgZmYN0aAxYEkVshUgu+0KfIVsJtjHgefz/i9GxNz8OecCHwM6gbMj4hf1XqNoBWxm1hoaVAFHRJV8Jcj8iylWArcCZwAXRsR3ao+XtCcwDdgLeDPwS0lvizorxBddC8LMrDU0Zy2II4AnexgNOB64KSLWR8TTwDLgnfVO6gRsZuXS2VG8FTcNuLFm+1OSlki6StKYvG8CsLzmmBV53yY5AZtZuXR1FW6S2iQtqGltG55O0gjgOGBW3nUJ2V3Bk4FngQv6GqrHgM2sXHoxBhwR7UB7D4dNBRZGxHP5c57r3iHpcmBOvrkS2LnmeRPp4c5hV8BmVi4RxVsxp1Iz/CBpfM2+E4FH8sezgWmStpA0CdgdmF/vxK6AzaxcGjgPWNI2wJHAmTXd35Y0mex7MZ/p3hcRj0r6IfAY0AGcVW8GBDgBm1nZNDABR8Tvge026PtonePPB84ven4nYDMrlxKuBWFm1hqKj+0m5wRsZuVStrUgzMxahhOwmVkiJVyQ3cysJURH3Zlfg4oTsJmViytgM7NEujwLwswsDV+EMzNLxAnYzCwR34hhZpaIZ0GYmSXiWRBmZol4FoSZWRrhi3BmZom4AjYzS8RjwGZmiXgWhJlZIh6CMDNLxEMQZmaJuAI2M0vD09DMzFLpcAI2M0vDY8BmZol4DNjMLI1wAjYzS8QJ2MwskRaaBTEsdQBmZg3V0VW89UDSaEm3SHpc0lJJ75I0VtLtkp7If47Jj5WkiyQtk7RE0v49nd8J2MxKJSIKtwK+B/w8IvYA9gWWAl8A7oiI3YE78m2AqcDueWsDLunp5E7AZlYuXVG81SFpFPCXwJUAEfFaRLwIHA/MzA+bCZyQPz4euDYy/wWMljS+3ms4AZtZufQiAUtqk7SgprXVnGkS8DxwtaRFkq6QtA2wY0Q8mx+zCtgxfzwBWF7z/BV53yb5IpyZlUpvpqFFRDvQvondw4H9gU9HxDxJ3+NPww3dzw9JfZ524QrYzMqlQUMQZBXsioiYl2/fQpaQn+seWsh/rs73rwR2rnn+xLxvk5yAzaxUoiMKt7rniVgFLJdUybuOAB4DZgPT877pwE/yx7OB0/PZEAcDa2uGKjbKQxBmVi6NvRHj08APJI0AngLOICtcfyjpY8BvgFPyY+cCxwDLgFfzY+tyAjazcmngfRgRsRiYspFdR2zk2ADO6s35nYDNrFS8FoSZWSqtcyeyE7CZlUtPF9cGEydgMyuVFlqP3QnYzErGCdjMLA1XwGZmqTgBm5ml4QrYzCyRro7UERTnBGxm5RJKHUFhTsBmVioegjAzSyS6XAGbmSXhCtjMLJGuTlfAZmZJeAjCzCyRYt82Pzg4AZtZqbgCNjNLxAnYzCwRD0GYmSXS1dk6X/buBGxmpeJ5wGZmiXR5LQgzszTCCdjMLA3PgjAzS8SzIMzMEun0LAgzszRaaQy4df6qMDMrIKJ4K0LSZpIWSZqTb18j6WlJi/M2Oe+XpIskLZO0RNL+PZ3bFbCZlUoTpqF9BlgKjKzp+3xE3LLBcVOB3fN2EHBJ/nOTXAGbWalEqHDriaSJwF8BVxR46eOBayPzX8BoSePrPWFAKuDNx+06EC9jLabjtZWpQ7AS6mzsNLTvAucA227Qf76krwB3AF+IiPXABGB5zTEr8r5nN3VyV8BmViq9qYAltUlaUNPaus8j6YPA6oh4aIOXOBfYAzgQGAv8c19jHZAKeMQWEwfiZaxFvLZ+BQBv235K4khsMPnv5xc05Dy9GQOOiHagfRO7DwGOk3QMsCUwUtL1EXFavn+9pKuBz+XbK4Gda54/Me/bJFfAZlYq0YtW9zwR50bExIjYBZgG3BkRp3WP60oScALwSP6U2cDp+WyIg4G1EbHJ4QfwLAgzK5kBWIznB5K2BwQsBj6R988FjgGWAa8CZ/R0IidgMyuVZtyIERF3A3fnjw/fxDEBnNWb8zoBm1mpdNI6d8I5AZtZqXR5MR4zszS6XAGbmaURTsBmZmm00FfCOQGbWbm4AjYzS6QjdQC94ARsZqXiCtjMLJEW+k5OJ2AzKxdPQzMzS6SF7sNwAjazcumQK2AzsyRcAZuZJeIbMczMEvEsCDOzRDwLwswsEY8Bm5kl0tE6BbATsJmViytgM7NEfBHOzCwRT0MzM0vECdjMLJEmfCt90zgBm1mpeEF2M7NEPAvCzCwRz4IwM0uklS7CDUsdgJlZI3X1otUjaUtJ8yX9WtKjkr6a90+SNE/SMkk3SxqR92+Rby/L9+/SU6xOwGZWKp0q3nqwHjg8IvYFJgNHSzoY+BZwYUS8FXgB+Fh+/MeAF/L+C/Pj6nICNrNSaVQFHJlX8s3N8xbA4cAtef9M4IT88fH5Nvn+I6T6X8/hBGxmpRK9aD2RtJmkxcBq4HbgSeDFiOie7bYCmJA/ngAsB8j3rwW2q3d+J2AzK5UuonCT1CZpQU1rqz1XRHRGxGRgIvBOYI9GxupZEGZWKr2ZBRER7UB7geNelHQX8C5gtKTheZU7EViZH7YS2BlYIWk4MAr4Xb3zugI2s1Jp1BCEpO0ljc4fbwUcCSwF7gJOzg+bDvwkfzw73ybff2dE1H0ZV8BmVioNXJB9PDBT0mZkxeoPI2KOpMeAmyR9A1gEXJkffyVwnaRlwBpgWk8v4ARsZqXS1aCbkSNiCbDfRvqfIhsP3rD/D8Bf9+Y1nIDNrFS8FoSZWSKtdCuyE7CZlUqjhiAGghOwmZVKZ+oAesEJ2MxKxRWwmVkirZN+nYDNrGR8Ec7MLJFooRrYCdjMSsUVsJlZIp2ugM3M0vAsCDOzRDwEYWaWiC/CmZkl4grYzCwRV8BmZol01P8SikHFCdjMSqV10q8TsJmVjKehmZkl4jFgM7NEPAvCzCyRzhZKwU7AZlYqrZN+nYDNrGTC09DMzNLwLAgzs0Q8BGFmloinoZmZJdIZrVMDOwGbWam0TvqFYakDMDNrpOjFfz2RdJWk1ZIeqen7V0krJS3O2zE1+86VtExSVdJRPZ3fFbCZlUqDZ0FcA3wfuHaD/gsj4ju1HZL2BKYBewFvBn4p6W0R0bmpk7sCNrNSiYjCrcC5/hNYU/Cljwduioj1EfE0sAx4Z70nOAGbWal0EYVbP3xK0pJ8iGJM3jcBWF5zzIq8b5OcgM2sVDqjq3CT1CZpQU1rK/ASlwC7AZOBZ4EL+hqrx4DNrFR6U9dGRDvQ3qvzRzzX/VjS5cCcfHMlsHPNoRPzvk1yBWxmpdLsIQhJ42s2TwS6Z0jMBqZJ2kLSJGB3YH69c7kCNrNSaeQsCEk3AocB4yStAM4DDpM0mazYfgY4EyAiHpX0Q+AxoAM4q94MCHACNrOSaeRqaBFx6ka6r6xz/PnA+UXP7wRsZqXiBdnNzBLxesBmZol4PWAzs0RcAZuZJeIK2MwsES/IbmaWiBdkNzNLpMtjwGZmaXgIwswsEVfAZmaJuAI2M0vEFbCZWSJd9RcgG1ScgM2sVHwjhplZIr4V2cwsEVfAZmaJuAI2M0vEtyKbmSXiCtjMLBGPAZuZJeIK2MwsEd8JZ2aWiCtgM7NEPAvCzCwRD0GYmSXSSstRDksdQFm1X/YdVixfzKKFv/xj35gxo5k79wYeffRe5s69gdGjRyWM0FI4vW0ac/7zZv7j3puZfuapAIwaPZKrZ13MbfN+zNWzLmbkqG0TR9nauiIKt9ScgJvk2utm8cFjT/uzvnM+fxZ33Xk/e+31Hu66837O+fxZiaKzFHbfYzdOOe1ETj7qdI477MO878hD+YtJE2k7ewYP3jufDxz0IR68dz5tZ89IHWpLi4jCLbUeE7CkYZLePRDBlMl9983jhRde/LO+Y4/9ANddPwuA666fxXHHHZUiNEtkt7ftwq8XPsIf1q2ns7OT+Q8s5AN/dThHTH0vt948B4Bbb57D+485LG2gLa4rugq31HpMwBHRBVw8ALGU3g47jGPVqtUArFq1mh12GJc4IhtITyx9kikHT2b0mFFsudUWvPf9hzB+wo6M234szz/3OwCef+53jNt+bOJIW1srVcBFL8LdIekk4McxGKIuCf+vHFqefOIZLv9/13LVrO+z7tV1LH3kv+nsfOO3N/hz0T+t9H9PRX7Zkl4GtgE6gXWAgIiIkZs4vg1oyzfbI6K9MeG2nF2AOcDeAGvWrFk1duzY/YBngfHA3UAlVXCW3DeBFWvWrPmKPxdDU6GLcBGxbUQMi4jNI2Jkvr3R5Jsf3x4RU/I2VJPvG8yaNQtger45HfhJumgskR3yn38BfAi4wZ+LoatoBSzgI8CkiPi6pJ2B8RExv9kBtrAbgcOAccBzwHk77bTTp1atWrWG7A/fb4BTgDXJIrQU7gW2A14H/hG4Y6eddlrkz8XQVDQBXwJ0AYdHxNsljQFui4gDmx1gmUhaEBFTUsdhg4s/F0NX0YtwB0XE/pIWAUTEC5JGNDGusvJwjG2MPxdDVNEbMV6XtBn5BUZJ25NVxNYLHg+3jfHnYugqmoAvAm4FdpB0PnAf2RVcMzPro6KzIH4AnAP8H7KpMidExKxmBmZ/TtLdkjxOWAKSZkh6c832H3+3kuZKGp0uOhtIdceAJY2MiJckjQVWk13Z7943NiJ8pbYAScMjoiN1HDZozAAeAf5nwx0RcUxvTiRps4h4490c1hJ6ugh3A/BB4CGy8V9t8HPXpkY3iEjaBfgZ2fDLu4GVwPFkE+YvBbYGngT+Nr9IeTewGDgUuFHSO8huYtmPbC7o3wKnA+8C5kXEjPx1LgEOBLYCbomI8wbkDVq/SPpHst8pwBXAvwNzImLvfP/ngDeRJd4pwA8krSP7/dee5xlgSkT8VtJpwNnACGAe8MmI6JT0CnAZ8H7gLLLPpLWgukMQEfHB/OekiNh1w58DE+KgsjtwcUTsBbwInARcC/xzROwDPAzUJswR+c0oF+TbY8j+wH0WmA1cCOwFvEPS5PyYL+VTkvYB3itpn2a/KesfSQcAZwAHAQcDHyf7Xb9BRNwCLAA+EhGTI2LdJs75duBvgEMiYjLZXagfyXdvQ/aX9r4R4eTbwnoagti/3v6IWNjYcAa9pyNicf74IWA3YHRE3JP3zQRqx8Zv3uD5P42IkPQw8FxEPAwg6VGy25YXA6fkt3IPJ7stdU9gSTPejDXMocCtEfF7AEk/Bt7Tz3MeARwA/Cq7D4qtyIYBIUvGP+rn+W0Q6GkI4oI6+wI4vIGxtIL1NY87gZ4ulvx+E8/v2uBcXcBwSZOAzwEH5sMY1wBb9j1cS2g0f/4vzN7+HgXMjIhzN7LvDx73LYeehiDeFxHvA6Z2P67p69XFgpJaC7wgqbva+ShwT53jezKSLGmvlbQjMLWf8dnAuBc4QdLWkrYBTiS7XrCDpO0kbUF2LaXby0BPX3txB3CypB0gu+gt6S1NiN0SKnon3APAhsMRG+sbiqYDl0raGniKbCywTyLi1/ndho8Dy4H7GxOiNVNELMz/tdK9NsoVEfErSV/L+1aS/U67XUP2mXnDRbiacz4m6cvAbZKGka0dcRbZWhFWEnXXgpC0EzABuB74MNk/iyCr1C6NiD2aHqGZWUn1VAEfRTZncSLwbzX9LwNfbFJMZmZDQtHV0E6KCF91NTNroKJjwHtL2mvDzoj4WoPjMTMbMoom4FdqHm9JdkV3aePDMTMbOgoNQbzhSdm0ml9ExGENj8jMbIgouhzlhrYmuzBnZmZ9VGgIIr91trtUHka2mMzXmxWUmdlQUHQWxFvIFhd5D9ktlnMj4qEmx2ZmVmpFhyCOB64j+4bfzYGrJX26aVGZmQ0BRSvgJcC7alZ72gZ4MF+C0czM+qBoBSyy1b+6dfKn25LNzKwPis4DvhqYJ+nWfPsE4MrmhGRmNjQUngecL85+aL55b0QsalpUZmZDQJ9uxDAzs/7r640YZmbWT07AZmaJOAGbmSXiBGxmlogTsJlZIv8LPPFUBLygJXQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = od_preds['data']['is_outlier']\n",
    "f1 = f1_score(y_outlier, y_pred)\n",
    "print('F1 score: {}'.format(f1))\n",
    "cm = confusion_matrix(y_outlier, y_pred)\n",
    "df_cm = pd.DataFrame(cm, index=labels, columns=labels)\n",
    "sns.heatmap(df_cm, annot=True, cbar=True, linewidths=.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot instance level outlier scores vs. the outlier threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5wU1Zn/8c8zMMKIOqASg4ABXS6LgIMOqItGDFkV4z0EJSZCNBKyuiq6ifrb/JC4yW9jJGrIumYxAdQQV6LGW7zPRoyut0HJICIBFXUQBcEBwRFm4Pn9UdVNz9DdUz3Tl5me7/v16ld3naqueqprpp+uc06dMndHREQEoKTQAYiISPuhpCAiInFKCiIiEqekICIicUoKIiISp6QgIiJxOUsKZtbfzP5sZm+Y2XIzuzws39/MnjKzVeFzr7DczGyOma02sxozOzJXsYmISHK5PFNoBK5y92HAMcAlZjYMuAaocvdBQFU4DTABGBQ+pgG35TA2ERFJImdJwd3Xufur4etPgRVAX+BM4I5wsTuAs8LXZwJ3euBFoKeZ9clVfCIisqeu+diImQ0ARgEvAQe5+7pw1ofAQeHrvsD7CW+rDcvWJZRhZtMIziTo0aPHUUOHDs1Z3EWl/hOoew981+4yK4Geh0BZr8LFJSJ5t2TJko/dvXeyeTlPCma2D3AfcIW7bzGz+Dx3dzPLaJwNd58LzAWorKz06urqbIZb3GoWQdX1sLkWyvvB+JkwclKhoxKRPDOzd1PNy2lSMLNSgoSw0N3vD4s/MrM+7r4urB5aH5avBfonvL1fWCbZMnKSkoCIpJXL3kcG/BZY4e43Jcx6CJgSvp4CPJhQfkHYC+kYYHNCNZOIiORBLs8UxgLfBpaZ2dKw7P8APwMWmdlFwLtA7Kfro8CpwGrgM+A7OYxNRESSyFlScPfnAEsxe3yS5R24JFfxiEj719DQQG1tLZ9//nmhQykK3bt3p1+/fpSWlkZ+T156H4mIRFFbW8u+++7LgAEDSOyUIplzdzZu3EhtbS0DBw6M/D4NcyEi7cbnn3/OAQccoISQBWbGAQcckPFZl5KCiLQrSgjZ05rPUklBRETilBRERNqJcePGUegLctXQLCId1gOvreXGJ1byQV09B/cs4wcnD+GsUX0LEktjYyNdu3b8r9SOvwci0ik98Nparr1/GfUNOwFYW1fPtfcvA2h1YlizZg0TJkzguOOO43//93/p27cvDz74ICtXrmT69Ol89tlnHHbYYcybN49evXoxbtw4KioqeO6555g8eTLLli2jrKyM1157jfXr1zNv3jzuvPNOXnjhBY4++mgWLFgAwPe//31eeeUV6uvrmThxIj/+8Y+z8plkg6qPRKRDuvGJlfGEEFPfsJMbn1jZpvWuWrWKSy65hOXLl9OzZ0/uu+8+LrjgAm644QZqamoYMWJEky/xHTt2UF1dzVVXXQXAJ598wgsvvMDNN9/MGWecwYwZM1i+fDnLli1j6dLgOt6f/vSnVFdXU1NTw+LFi6mpqWlTzNmkpCAiHdIHdfUZlUc1cOBAKioqADjqqKN46623qKur44QTTgBgypQpPPvss/Hlzz333CbvP/300zEzRowYwUEHHcSIESMoKSnh8MMPZ82aNQAsWrSII488klGjRrF8+XLeeOONNsWcTao+EpEO6eCeZaxNkgAO7lnWpvV269Yt/rpLly7U1dWlXb5Hjx5J319SUtJkXSUlJTQ2NvLOO+8we/ZsXnnlFXr16sXUqVPb1RXcOlMQkQ7pBycPoay0S5OystIu/ODkIVndTnl5Ob169eIvf/kLAHfddVf8rKE1tmzZQo8ePSgvL+ejjz7isccey1aoWaEzBRHpkGKNyfnofXTHHXfEG5oPPfRQ5s+f3+p1HXHEEYwaNYqhQ4fSv39/xo4dm8VI286Cceg6Jt1kR6S4rFixgr//+78vdBhFJdlnamZL3L0y2fKqPhIRkTglBRERiVNSEBGROCUFERGJU1IQEZG4nCUFM5tnZuvN7PWEsnvMbGn4WBO7d7OZDTCz+oR5v85VXCIiklouzxQWAKckFrj7ue5e4e4VwH3A/Qmz34rNc/fpOYxLRCQrFixYwAcffBCfThz6+tRTT23xauj2KGdJwd2fBTYlm2fB7YAmAXfnavsi0gnULIKbh8OsnsFzzaK8br55Ukj06KOP0rNnz8jr2rlzZ8sL5UGh2hSOBz5y91UJZQPN7DUzW2xmxxcoLhHpKGoWwcOXweb3AQ+eH76szYnhpptuYvjw4QwfPpxbbrmFNWvWMHz48Pj82bNnM2vWLO69916qq6s5//zzqaiooL6+6ThMAwYM4OOPPwbgd7/7HWPGjKGiooLvfe978QSwzz77cNVVV3HEEUfwwgsvtCnubClUUphM07OEdcAh7j4KuBL4vZntl+yNZjbNzKrNrHrDhg15CFVE2qWq66Gh2YB4DfVBeSstWbKE+fPn89JLL/Hiiy9y++2388knnyRdduLEiVRWVrJw4UKWLl1KWVnygfhWrFjBPffcw/PPP8/SpUvp0qULCxcuBGDbtm0cffTR/PWvf+W4445rddzZlPexj8ysK3AOcFSszN23A9vD10vM7C1gMLDHGBbuPheYC8EwF/mIWUTaoc21mZVH8Nxzz3H22WfHRz4955xz4gPhtVZVVRVLlixh9OjRANTX1/OFL3wBCEZh/frXv96m9WdbIQbE+yrwprvHj5yZ9QY2uftOMzsUGAS8XYDYRKSjKO8XVh0lKc+iuro6du3aFZ/OdJhrd2fKlCn8+7//+x7zunfvTpcuXZK8q3By2SX1buAFYIiZ1ZrZReGs89izgfnLQE3YRfVeYLq7J22kFhEBYPxMKG1WZVNaFpS30vHHH88DDzzAZ599xrZt2/jjH//IhAkTWL9+PRs3bmT79u088sgj8eX33XdfPv300/Rhjh/Pvffey/r16wHYtGkT7777bqtjzLWcnSm4++QU5VOTlN1H0EVVRCSakZOC56rrgyqj8n5BQoiVt8KRRx7J1KlTGTNmDADf/e53GT16NDNnzmTMmDH07duXoUOHxpefOnUq06dPp6ysLGVD8bBhw/jJT37CSSedxK5duygtLeXWW2/lS1/6UqvjzCUNnS0i7YaGzs4+DZ0tIiKtpqQgIiJxSgoiIhKnpCAiInFKCiIiEqekICIicUoKIiKhjRs3UlFRQUVFBV/84hfp27cvFRUV9OzZk2HDhmV9e8888wynnXZaRu9JHJ470YIFC7j00kvbHJOSgohI6IADDmDp0qUsXbqU6dOnM2PGjPh0SUnLX5eNjY15iDK3lBRERCLYuXMnF198MYcffjgnnXRSfKjscePGccUVV1BZWckvf/lLNmzYwNe//nVGjx7N6NGjef755wFYvHhx/Cxk1KhR8eExtm7dysSJExk6dCjnn38+sQuKq6qqGDVqFCNGjODCCy9k+/bte8Q0f/58Bg8ezJgxY+LbaatCDIgnItKiKx6/gqUfLs3qOiu+WMEtp9zSqveuWrWKu+++m9tvv51JkyZx33338a1vfQuAHTt2xKt0vvnNbzJjxgyOO+443nvvPU4++WRWrFjB7NmzufXWWxk7dixbt26le/fuALz22mssX76cgw8+mLFjx/L8889TWVnJ1KlTqaqqYvDgwVxwwQXcdtttXHHFFfF41q1bx3XXXceSJUsoLy/nxBNPZNSoUW38hHSmICISycCBA6moqADgqKOOYs2aNfF55557bvz1008/zaWXXkpFRQVnnHEGW7ZsYevWrYwdO5Yrr7ySOXPmUFdXR9euwW/yMWPG0K9fP0pKSqioqGDNmjWsXLmSgQMHMnjwYACmTJnCs88+2ySel156iXHjxtG7d2/22muvJjG0hc4URKRdau0v+lzp1q1b/HWXLl2a3Gktdv8FgF27dvHiiy/GzwRirrnmGr72ta/x6KOPMnbsWJ544omk6y10u4TOFEREsuikk07iV7/6VXx66dKgCuytt95ixIgRXH311YwePZo333wz5TqGDBnCmjVrWL16NQB33XUXJ5xwQpNljj76aBYvXszGjRtpaGjgD3/4Q1biV1IQEcmiOXPmUF1dzciRIxk2bBi//vWvAbjlllsYPnw4I0eOpLS0lAkTJqRcR/fu3Zk/fz7f+MY3GDFiBCUlJUyfPr3JMn369GHWrFkce+yxjB07Nmujy2robBFpNzR0dvZp6GwREWk1JQUREYlTUhCRdqUjV2m3N635LHOWFMxsnpmtN7PXE8pmmdlaM1saPk5NmHetma02s5VmdnKu4hKR9qt79+5s3LhRiSEL3J2NGzfu0TW2Jbm8TmEB8B/Anc3Kb3b32YkFZjYMOA84HDgYeNrMBrv7zhzGJyLtTL9+/aitrWXDhg2FDqUodO/enX79+mX0npwlBXd/1swGRFz8TOC/3X078I6ZrQbGAC/kKDwRaYdKS0sZOHBgocPo1ArRpnCpmdWE1Uu9wrK+wPsJy9SGZXsws2lmVm1m1fo1ISKSXflOCrcBhwEVwDrgF5muwN3nunulu1f27t072/GJiHRqeU0K7v6Ru+90913A7QRVRABrgf4Ji/YLy0REJI/ymhTMrE/C5NlArGfSQ8B5ZtbNzAYCg4CX8xmbiIjksKHZzO4GxgEHmlktcB0wzswqAAfWAN8DcPflZrYIeANoBC5RzyMRkfxrcewjM9sbuAo4xN0vNrNBwBB3fyQfAaajsY9ERDLX1rGP5gPbgWPD6bXAT7IUm4iItCNRksJh7v5zoAHA3T8DLKdRiYhIQURJCjvMrIygHQAzO4zgzEFERIpMlIbm64DHgf5mthAYC0zNZVAiIlIYaZOCmRnwJnAOcAxBtdHl7v5xHmITEZE8S5sU3N3N7FF3HwH8KU8xiYhIgURpU3jVzEbnPBIRESm4KG0KRwPnm9m7wDaCKiR395E5jUxERPIuSlLQDW9ERDqJFquP3P1doCdwevjoGZaJiEiRaTEpmNnlwELgC+Hjd2b2z7kOTERE8i9K9dFFwNHuvg3AzG4guCPar3IZmLRzNYug6nrYXAvl/WD8TBg5qdBRiUgbRUkKBiSOWLoTDXPRudUsgocvg4b6YHrz+8E0KDGIdHBRksJ84CUz+2M4fRbw29yFJO1e1fW7E0JMQ31QrqQg0qG1mBTc/SYzewY4Liz6jru/ltOopH3bXJtZuYh0GC0mBTM7Blju7q+G0/uZ2dHu/lLOo5P2qbxfUGWUrFxEOrQoVzTfBmxNmN4alklnNX4mlJY1LSstC8pFpEOLkhTME27P5u67yOFtPKUDGDkJTp8D5f0BC55Pn6P2BJEiEOXL/W0zu4zdZwf/BLzd0pvMbB5wGrDe3YeHZTcSXAC3A3iLoH2izswGACuAleHbX3T36Rnsh+TbyElKAiJFKMqZwnTgHwhuw7mWYCykaRHetwA4pVnZU8DwcNykvwHXJsx7y90rwocSgohIAUTpfbQeOC/TFbv7s+EZQGLZkwmTLwITM12viIjkTsozBTO72MwGha/NzOaZ2WYzqzGzI7Ow7QuBxxKmB5rZa2a22MyOTxPXNDOrNrPqDRs2ZCEMERGJSVd9dDmwJnw9GTgCOBS4EvhlWzZqZv8KNBKMqQSwDjjE3UeF6/+9me2X7L3uPtfdK929snfv3m0JQ0REmkmXFBrdvSF8fRpwp7tvdPengR6t3aCZTQ3Xd36sV5O7b3f3jeHrJQSN0INbuw0REWmddElhl5n1MbPuwHjg6YR5ZSnek5aZnQL8EDjD3T9LKO9tZl3C14cCg4jQw0lERLIrXUPzTKAa6AI85O7LAczsBKJ1Sb0bGAccaGa1wHUEvY26AU+ZGezuevpl4HozawB2AdPdfVNrd0pERFrHEq5L23OmWVdgX3f/JKGsR/i+rSnfmCeVlZVeXV1d6DBERDoUM1vi7pXJ5qXtkurujcAnzcq2ZTE2ERFpR6JcvCYiIp2EkoKIiMSlrD5q6QK12FDaIiJSPNK1KfwizTwHvpLlWEREpMBSJgV3PzGfgYiISOG12KZgZnub2Y/MbG44PcjMTst9aCIikm9RGprnE9z/4B/C6bXAT3IWkYiIFEyUpHCYu/8caAAIh6ewnEYlIiIFESUp7DCzMoLGZczsMGB7TqMSEZGCiHI7zlnA40B/M1sIjAWm5jAmEREpkCh3XnvSzJYAxxBUG13u7h/nPDIREcm7FpOCmT0M/J5gpFSNeyQiUsSitCnMBo4H3jCze81sYniPBZHWqVkENw+HWT2D55pFhY5IREJRqo8WA4vDm+B8BbgYmAckvV2mSFo1i+Dhy6ChPpje/H4wDTByUuHiEhEg4oB4Ye+jrwPTgdHAHbkMSopY1fW7E0JMQ31QLiIFF6VNYREwhqAH0n8Ai919V64DkyK1uTazchHJqyhdUn8LTHb3nbkORjqB8n5BlVGychEpuCjVR38Brm3N2EdmNs/M1pvZ6wll+5vZU2a2KnzuFZabmc0xs9VmVtPS0N3SQY2fCaVlTctKy4JyESm4XI99tAA4pVnZNUCVuw8CqsJpgAnAoPAxDbgt4jakIxk5CU6fA+X9AQueT5+jRmaRdiJK9dFh7n6umU2GYOwjM4s09pG7P2tmA5oVnwmMC1/fATwDXB2W3+nuDrxoZj3NrI+7r4uyLelARk5SEhBppwox9tFBCV/0HwIHha/7AomVzbVhWRNmNs3Mqs2sesOGDW0IQ0REmouSFK6j6dhHVcAPs7Hx8KzAM3zPXHevdPfK3r17ZyMMEREJRbl47Skze5WEsY+AvdqwzY9i1UJm1gdYH5avBfonLNcvLBMRkTyJdPGau2909z+5+yPhYHgvtmGbDwFTwtdTgAcTyi8IeyEdA2xWe4KISH5FaWhOJlJDs5ndTdCofKCZ1RJURf0MWGRmFwHvArEWx0eBU4HVwGfAd1oZm4iItFJrk0KkdgB3n5xi1vgkyzpwSSvjERGRLEiZFMzsVyT/8jegZ84iEhGRgkl3plDdynkiItJBpUwK7q6RUEVEOplIvY9ERKRzUFIQEZE4JQUREYlrMSmY2WAzq4oNf21mI83sR7kPTURE8i3KmcLtwLVAA4C71wDn5TIoEREpjChJYW93f7lZWWMughERkcKKkhQ+DofLjg2dPRHQmEQiIkUoyjAXlwBzgaFmthZ4B/hWTqMSEZGCiDJ09tvAV82sB1Di7p/mPiwRESmEKL2P/p+Z9XT3be7+qZn1MrOo92gWSa9mEdw8HGb1DJ5rFhU6IpFOLUqbwgR3r4tNuPsnBENci7RNzSJ4+DLY/D7gwfPDlykxiBRQlKTQxcy6xSbC+zV3S7O8SDRV10NDfdOyhvqgXEQKIkpD80Kgyszmh9PfATRYnrTd5trMykUk56I0NN9gZjXsvjHOv7n7E7kNSzqF8n5h1VGSchEpiEh3XnP3x4DHchyLdDbjZwZtCIlVSKVlQbmIFESU3kfnmNkqM9tsZlvM7FMz29LaDZrZEDNbmvDYYmZXmNksM1ubUK7G7GI3chKcPgfK+wMWPJ8+JygXkYKw4NbIaRYwWw2c7u4rsr5xsy7AWuBograKre4+O+r7KysrvbpaN4ETEcmEmS1x98pk86L0PvooFwkhNB54y93fzdH6RUQkA1HaFKrN7B7gAWB7rNDd78/C9s8D7k6YvtTMLiC4B/RV4TURTZjZNGAawCGHHJKFEEREJCZK9dH8JMXu7he2acNmewEfAIe7+0dmdhDwMcHAe/8G9GlpG6o+6oRqFgXXMWyuDXopjZ+pNgiRDKWrPorSJfU72Q8JgAnAq+7+Ubidj2IzzOx24JEcbVc6qtgV0LHeSrEroEGJQSRLWkwKZtYduAg4HOgeK2/rmQIwmYSqIzPr4+6xIbnPBl5v4/ql2KS7AlpJQSQrojQ03wV8ETgZWAz0A9o0Umo44uo/AontEj83s2XhhXInAjPasg0pQp31CmgNGih5FKWh+e/c/Rtmdqa732Fmvwf+0paNuvs24IBmZd9uyzqlE+iMV0CrykzyLMqZQkP4XGdmw4Fy4Au5C0kkhfEzgyueE0W9Arqj/trWoIGSZ1HOFOaaWS/gR8BDwD7A/81pVCLJxH4ZZ9r7qCP/2u6sVWZSMFGSQlV4vcCzwKEAZjYwp1GJpDJyUuZf5B25gTqTKjN115UsiFJ9dF+SsnuzHYhIznTkX9tRq8x0wyLJkpRnCmY2lKAbarmZnZMwaz8SuqaKtHsduYE6apVZRz4bknYlXfXREOA0oCdwekL5p8DFuQxKJKs6+hDdUarMOvLZkLQrKZOCuz8IPGhmx7r7C3mMSSS7WttA3ZF05LMhaVeiNDSfbWbLgXrgcWAkMMPdf5fTyESyqTUN1B1JRz8bknYjSkPzSe6+haAqaQ3wd8APchmUSM511OsWUtENiyRLopwplIbPXwP+4O6bzSyHIYnkWEe+biGdYj8bkryIcqbwsJm9CRwFVJlZb+Dz3IYlkkPFfpVwsZ0FSV5FGTr7GjP7ObDZ3Xea2TbgzNyHJpIjxdxTp1jPgiRvolQfAQwFBphZ4vJ35iAekdwr5p46ul5B2qjF6iMzuwuYDRwHjA4fSe/YI9IhtGVgvfaumM+CJC+inClUAsO8pft2inQUxXzdQqqzoLJeQftCse2vZF2UpPA6wU121rW0oEiHUaw9dZJdr1BSCju2Qv2mYFrtDJJGlKRwIPCGmb0MbI8VuvsZOYtKRFon2VnQjm27E0JMqnYGjbTa6UVJCrNyHYSIZFHzs6BZPZMv17ydoTU9l5REik6ULqmLc7FhM1tDMLjeTqDR3SvNbH/gHmAAwdXTk8J7OYhIa0XtbZVpz6VCdn9VMsqZlL2PzOxTM9uS5PGpmW3J0vZPdPcKd4/1ZrqG4KY+g4CqcFpE2iJqb6tMey4V6iJA3Tsip1ImBXff1933S/LY1933y1E8ZwJ3hK/vAM7K0XZEOo+o4yKluk4jVXnKJPJ+bq+oLvYr0gss6sVrueDAk2bmwH+5+1zgIHeP9XL6EDio+ZvMbBowDeCQQw7JV6wiHVuU3laZjrSaqloK212eiyqldMmoZpGqkdooythHuXKcux8JTAAuMbMvJ84Mr4vY49oId5/r7pXuXtm7d+88hSrSCWQ60mqyaimMPf5ts/0rPt2V56pGajNrD9ekmdksYCvBHd3Gufs6M+sDPOPuQ1K9r7Ky0qurq/MUpYjsoXmDb9IzBwCDWXXZ22bzM5pEZfvDXj2iNUJ30gZrM1uS0JbbREGqj8ysB1Di7p+Gr08CrgceAqYAPwufHyxEfCISUfNqqZuH535cqdj27k9xV+D6TdEu1NPggUkVqvroIOA5M/sr8DLwJ3d/nCAZ/KOZrQK+Gk6LSEeRr3GlRk4Kq7kiSFV9la7BuhMPP16QMwV3fxs4Ikn5RmB8/iMSkazI57hSyRrGU0nWOJ2uwboTn0EUsveRiBSjfI0rFXVID0hefZWqDcS6JD+D+ON0uH9a6kRXJO0TSgoi0nE1T0DJGqFTVV+l6oKb6szDdwbPyc4ciqh9opBdUkVEsiuTbrWplo3SVtG8nSJV+8RjV7dlbwpCZwoiUlwyqb5KtWyUtorENolU7RP1m+CRK2HVkx2mWklnCiIiiZqfQViX5MsltlOk63JbPa9DjdOkpCAi0tzISTDj9eCCu7N/3XI327RdbpNc4d2Oq5WUFERE0onSTjFyUnAldVT1m+CGge3yjEFtCiIiLYnSTjHhhiRtEUnGgoqp39QueyjpTEFEJBuSnVFUXpj+Pe1wyG+dKYiIZEuyM4rlf0x+QV1Mqp5LBaIzBRGRXJpwQ5IhxhNkc7DALFBSEBHJpVi1UrKG6FwMFthGSgoiIrk2chJc/Q6cc3v0mxgViNoURETyJV+DBbaBzhRERCROSUFEROKUFEREJE5JQURE4vKeFMysv5n92czeMLPlZnZ5WD7LzNaa2dLwcWq+YxMR6ewK0fuoEbjK3V81s32BJWb2VDjvZnefXYCYRESEAiQFd18HrAtff2pmK4C++Y5DRET2VNA2BTMbAIwCXgqLLjWzGjObZ2a9ChaYiEgnVbCkYGb7APcBV7j7FuA24DCgguBM4hcp3jfNzKrNrHrDhg15i1dEpDMoSFIws1KChLDQ3e8HcPeP3H2nu+8CbgfGJHuvu89190p3r+zdu3f+ghYR6QQK0fvIgN8CK9z9poTyPgmLnQ28nu/YREQ6u0L0PhoLfBtYZmZLw7L/A0w2swqC2xStAb5XgNhERDq1QvQ+eo7gHnXNPZrvWEREpCld0SwiInFKCtlSswhuHg6zegbPNYsKHZFki46tdCLm7oWOodUqKyu9urq60GEEXxIPXxbchDumpBS67Qv1nwS32xs/s92Po95mNYuCm5Bvrt29z7BnWUf6HJIdW4CBJ8CUh5Iv35H3VzoFM1vi7pVJ5ykpZMHNw2Hz++mXKS1rl3dZyliqL71UidF3ge9suo6y/YP71jb/LBLXXRZeu5gsqdYsgseu3n0z9FTry4Z0x7byIjjtpt3TNYvggX+CXQ27y0pK4az/DF5H2bfm8rmvbaFk2KEoKeTarJ4EnaZaUN4fZuSwp22u/zGTffHHkl3V9S0nxkTNk2SqX+TNl4c9v3gTlffP7n6nO7bWBa7btHv6hoG7v7wTlfYAdqXeNwwqL2yaYCB5kompvAgOOSZ5okmVdJIm3U3BfvjOpp9dJn9LqY6dlQQ/CpIdkyjJLjGG0r2hsT5YX0y2j3UnoqSQa1HOFGJmbW79dtL9o6b7Us3Wr8tU+1neP4gpSmJs/r5YkozyGZb3D57zeVbWUlyx41mzCO6/uA0bMjhnbtOYW9p2l71g544W19zYpTtdR50Pf/19msQUKi2DI76557LpPtNMz5RTJbsue8GZt6Y+80ylefKBpgmntAd07ZY+YcakS1aPXAlLFoRnvgZ77Q07PkudNFvzIy1PZ1xKCrmWyR9w4h9oJgc93a/0kZOi/WO2NTmk/NVswb5kcqYQe9+suhbW3Wx5iLAc2TsrS/dlHztTyOT4p9M85qhnoBHsshJKEn9lpxM7c2gpvphMz5TT/a1GWSadVFWWqbLCNIQAAA24SURBVERNVoccC+8sjrYeaPn/NZnWvKeV0iUF9T7KhpGTggNX3h+w4Ms3lYZt4a8QD/7o758Gs8pb7tVSdf2eXzoN9UE5hL/UW1C/Kfija23vmfJ+qcvHzwz+gFu7vlTrbr58lOUg2ucRxchJQaNyMkdNDZ6THZvWaB5z1H2NwKImBEj9hZrqM830mKQ7NlGWSWdXQ/SEAE3/h6quT15Vt3NH+oTQfD2xdaX7f02mNe/JASWFdDLpijhyUvALZ1YdXP1OBhsJf2HFEsQjVyZfLNU/Saw86j9mW/7Ikn3xl5btPttJTIzl/YN67y57JV9X7H3p1p1s+fEzg1+DLcniFypTHgr2xboE09alaSNzthJQ85ij7msEOz2Df/XYfjaX6jON+oMg9v50xybKMtnW1kTUfD3p1hUlIWbynhzotNVHVzx+BUs/XNq0cNsG+GQNNG6Hkq7BL47Ez8dK4MBB0CPCQHy1rwTraY3eQ/bcRqr1de0G/UYHsX+8qmlDXDoDjmtdbImfUddu0GtA+s8jcXmz4PNM9b7mnz/ArsY9l9+2ATa+FcxLJpPjlA1tOdYxqWLetgE+Xr3nr18Lq9Ii/P/uwthovehNXct/H1YC+xwEWz9qumxLn2kmx2TbBvj4b3vGbgYHDk5YJoO/57aI/Q+19TjG1gMt/78mk+F7Kr5YwS2n3NKqUNNVHxVi7KP2qfkfYbI/bt8VfGlF+bLpNaD1f9TJtpFsfVYSlMPu5dP9Y8Z07ZZ5TDE9emf2ZZvJ8lGXbb5cpokq21Idm3RfrhAt5ti+JtvHxHUkJNFGgl/6XdnJdkqp5QuUH3Aw2JbkSTdZsu6+X2afabI4U/0ISPa3WtIVDjhsz2Vi64oqg2QZLJ/wP9RrQOpk1a0cPq+Ltp74utL8vybTmvfkQKdNCntk2JuHg0c4BW40mPpMtI0078kQVaptRO2ZkG67xXK9RHuT7vqNPPfff+C1tdz4xEreravn4J5l/OrkIZw1qgPf3DDtxaFJutSCeh+1QaetPtpDLq81SNU/PJvbiLJtXVQkHZX+jrNKXVKjKMRVyY9cCdXzaJKM9EteRHJMXVKTad6zaNBJe/agKCkNu5eGvWmy/WV92k3BBUuJPXaUEESkgDrnmUKqi0SO+CaselKnqCJFIta+8kHYvvKDjt6+kiXqfdRcqotEVj2Z27GJRCRvHnhtLdfev4z6hqA779q6eq69fxmAEkManTMptJOLRERS0S/ctrvxiZXxhBBT37CTG59Yqc8yjc6ZFFKM01O76wCOu+ZPAPTN4T+i/uFbJ5+fWy629cBra5n10HLq6oOhFHrtXcp1px/OWaP6NtleeVkp23Y00rAzqNpdW1fPjHuWUv3uJn5y1oi8x91RfVCXfOiRxPJsfV7F9Lm3uzYFMzsF+CXQBfiNu/8s1bKtbVN48obz+epnj1CScKfoz3wvrmn4Lg/tauWVvu3AGSXP8cOuizjYPuYDP5CfN07Kyf60tJ18xdHa+AqhPcYkxeFbxxzS4o+F5jpMm4KZdQFuBf4RqAVeMbOH3P2NbG3j1lt+ync+e7JJQtjl8IedX+7Q/6RnlDzHz0p/w94WDKXczz7mZ6W/gQayul8tbSdfcbQ2vkJojzFJ8fjdi+8BZJwYUmlvXVLHAKvd/W133wH8N3BmNjdw5qZ58X/OmBKD8SVLU7yjY/hh10V77NfetoMfds3u/YRb2k6+4mhtfIXQHmOS4nL3S60YZjyFdlV9ZGYTgVPc/bvh9LeBo9390oRlpgHTwskhwMpMtnFUn5KjUmydZT6wFVG3DyPs7RRzsrtfLW0nX3GkUujtJ9MeY5Lis+PD1UsyWPxL7p50MKt2VX0UhbvPBeZmY11mVp2qXq0Ydbb9Be1zZ6F9zp72Vn20FuifMN0vLBMRkTxob0nhFWCQmQ00s72A84CHChyTiEin0a6qj9y90cwuBZ4g6JI6z92X53CTWamG6kA62/6C9rmz0D5nSbtqaBYRkcJqb9VHIiJSQEoKIiIS1ymTgpmdYmYrzWy1mV1T6Hiyxcz6m9mfzewNM1tuZpeH5fub2VNmtip87hWWm5nNCT+HGjM7srB70Dpm1sXMXjOzR8LpgWb2Urhf94SdFjCzbuH06nD+gELG3RZm1tPM7jWzN81shZkd2wmO84zw7/p1M7vbzLoX27E2s3lmtt7MXk8oy/i4mtmUcPlVZjYlkxg6XVJIGEpjAjAMmGxmwwobVdY0Ale5+zDgGOCScN+uAarcfRBQFU5D8BkMCh/TgNvyH3JWXA6sSJi+AbjZ3f8O+AS4KCy/CPgkLL85XK6j+iXwuLsPBY4g2P+iPc5m1he4DKh09+EEHVHOo/iO9QLglGZlGR1XM9sfuA44mmCUiOtiiSQSd+9UD+BY4ImE6WuBawsdV4729UGCcaRWAn3Csj7AyvD1fwGTE5aPL9dRHgTXslQBXwEeAQz4GOja/HgT9Go7NnzdNVzOCr0PrdjncuCd5rEX+XHuC7wP7B8eu0eAk4vxWAMDgNdbe1yBycB/JZQ3Wa6lR6c7U2D3H1dMbVhWVMLT5VHAS8BB7r4unPUhcFD4uhg+i1uAHwK7wukDgDp3bwynE/cpvr/h/M3h8h3NQGADMD+sNvuNmfWgiI+zu68FZgPvAesIjt0Siv9YQ+bHtU3HuzMmhaJnZvsA9wFXuPuWxHke/HQoin7IZnYasN7dMxnzpRh0BY4EbnP3UcA2dlcpAMV1nAHC6o8zCRLiwUAP9qxmKXr5OK6dMSkU9VAaZlZKkBAWuvv9YfFHZtYnnN8HWB+Wd/TPYixwhpmtIRhR9ysEde09zSx2YWbiPsX3N5xfDmzMZ8BZUgvUuvtL4fS9BEmiWI8zwFeBd9x9g7s3APcTHP9iP9aQ+XFt0/HujEmhaIfSMDMDfguscPebEmY9BMR6IEwhaGuIlV8Q9mI4BticcJra7rn7te7ez90HEBzH/3H384E/AxPDxZrvb+xzmBgu3+F+Tbv7h8D7ZjYkLBoPvEGRHufQe8AxZrZ3+Hce2+eiPtahTI/rE8BJZtYrPMM6KSyLptCNKgVqyDkV+BvwFvCvhY4ni/t1HMGpZQ2wNHycSlCXWgWsAp4G9g+XN4KeWG8Bywh6dhR8P1q57+OAR8LXhwIvA6uBPwDdwvLu4fTqcP6hhY67DftbAVSHx/oBoFexH2fgx8CbwOvAXUC3YjvWwN0EbSYNBGeEF7XmuAIXhvu+GvhOJjFomAsREYnrjNVHIiKSgpKCiIjEKSmIiEickoKIiMQpKYiISJySgrQrZuZm9ouE6X8xs1lZWvcCM5vY8pJt3s43wpFL/9ysfEDi6JcZrnOqmR2cnQhFUlNSkPZmO3COmR1Y6EASJVw1G8VFwMXufmIWQ5hKMLyDSE4pKUh700hw79kZzWc0/6VvZlvD53FmttjMHjSzt83sZ2Z2vpm9bGbLzOywhNV81cyqzexv4dhJsfsx3Ghmr4Tj0n8vYb1/MbOHCK6ebR7P5HD9r5vZDWHZTIKLCH9rZjem2snwl//9ZvZ4OOb9zxNiWRCuc5kF9xCYCFQCC81sqZmVmdnMMN7XzWxueJUvZvaMmd0Q7vvfzOz4hPXODpevMbN/DsuPCj+7JWb2RMJwCpdZcF+OGjP776gHT4pAoa/g00OPxAewFdgPWEMwXs2/ALPCeQuAiYnLhs/jgDqCYYO7EYzz8uNw3uXALQnvf5zgx9AggitGuxOMRf+jcJluBFcKDwzXuw0YmCTOgwmGXuhNMEDd/wBnhfOeIclVwyQMiUzwy//tcB+7A+8SjFdzFPBUwnt6Jlsn4VWt4eu7gNMTlvtF+PpU4Onw9fcJxkiKDTO9P1AK/C/QOyw7F5gXvv6A3VcH9yz034Ue+XvoTEHaHQ9Gdr2T4KYqUb3i7uvcfTvBZf9PhuXLCL6MYxa5+y53X0XwpTyUYGyYC8xsKcFQ4wcQJA2Al939nSTbGw0848EAbY3AQuDLGcQLwY1TNrv75wRnIl8KYzrUzH5lZqcAW1K890QL7ii2jGAgwMMT5sUGQlzC7n3/KsEY+40A7r4JGAIMB54K9/1HBIOnQTB8xkIz+xbB2Zt0EpnUk4rk0y3Aq8D8hLJGwipPMysB9kqYtz3h9a6E6V00/TtvPq6LE4wh88/u3mTQMDMbR3CmkCuJMe8k+BX/iZkdQXADmenAJIJxbBLj6g78J8GZw/thQ3z3JOvdSfr/cQOWu/uxSeZ9jSDJnQ78q5mN8N33LZAipjMFaZfCX7KL2H17RQiqlI4KX59BUP2RqW+YWUnYznAowd2qngC+b8Gw45jZYAtuWpPOy8AJZnagBbd4nQwsbkU8TYQN7CXufh/BL/fYfXc/BfYNX8cSwMcW3DsjSo+qp4DvxRrMLbhl40qgt5kdG5aVmtnhYcLt7+5/Bq4mqOLap637Jh2DzhSkPfsFcGnC9O3Ag2b2V4K2gdb8in+P4At9P2C6u39uZr8hqGZ5NWyw3QCclW4l7r7OzK4hGLrZgD+5+4Pp3hNRX4I7qsV+sF0bPi8Afm1m9QS3nbydYLTQDwmGg2/Jb4DBQI2ZNQC3u/t/hI3Yc8ysnOD74BaCEYR/F5YZMMfd67Kwb9IBaJRUERGJU/WRiIjEKSmIiEickoKIiMQpKYiISJySgoiIxCkpiIhInJKCiIjE/X8j7tKw/wTPPQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_instance_score(od_preds, y_outlier, labels, od.threshold, ylim=(0,200))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}