doc/source/examples/od_vae_adult.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VAE outlier detection for income prediction\n",
    "\n",
    "## Method\n",
    "\n",
    "The Variational Auto-Encoder ([VAE](https://arxiv.org/abs/1312.6114)) outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The VAE detector tries to reconstruct the input it receives. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is measured as the mean squared error (MSE) between the input and the reconstructed instance.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The instances contain a person's characteristics like age, marital status or education while the label represents whether the person makes more or less than $50k per year. The dataset consists of a mixture of numerical and categorical features. It is originally not an outlier detection dataset so we will inject artificial outliers. It is fetched using the [Alibi](https://github.com/SeldonIO/alibi) library, which can be installed with pip. We also use `seaborn` to visualize the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install alibi seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import alibi\n",
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, f1_score, precision_score, recall_score\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "import tensorflow as tf\n",
    "tf.keras.backend.clear_session()\n",
    "from tensorflow.keras.layers import Dense, InputLayer\n",
    "\n",
    "from alibi_detect.od import OutlierVAE\n",
    "from alibi_detect.utils.perturbation import inject_outlier_tabular\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_instance_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_seed(s=0):\n",
    "    np.random.seed(s)\n",
    "    tf.random.set_seed(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load adult dataset\n",
    "\n",
    "The ```fetch_adult``` function returns a ```Bunch``` object containing the features, the targets, the feature names and a mapping of the categories in each categorical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult = alibi.datasets.fetch_adult()\n",
    "X, y = adult.data, adult.target\n",
    "feature_names = adult.feature_names\n",
    "category_map_tmp = adult.category_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shuffle data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(0)\n",
    "Xy_perm = np.random.permutation(np.c_[X, y])\n",
    "X, y = Xy_perm[:,:-1], Xy_perm[:,-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reorganize data so categorical features come first, remove some features and adjust ```feature_names``` and ```category_map``` accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Education', 'Marital Status', 'Relationship', 'Age', 'Capital Gain', 'Capital Loss', 'Hours per week']\n"
     ]
    }
   ],
   "source": [
    "keep_cols = [2, 3, 5, 0, 8, 9, 10]\n",
    "feature_names = feature_names[2:4] + feature_names[5:6] + feature_names[0:1] + feature_names[8:11]\n",
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32561, 7)\n"
     ]
    }
   ],
   "source": [
    "X = X[:, keep_cols]\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "category_map = {}\n",
    "i = 0\n",
    "for k, v in category_map_tmp.items():\n",
    "    if k in keep_cols:\n",
    "        category_map[i] = v\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess data\n",
    "\n",
    "Normalize numerical features or scale numerical between -1 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "minmax = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_num = X[:, -4:].astype(np.float32, copy=False)\n",
    "if minmax:\n",
    "    xmin, xmax = X_num.min(axis=0), X_num.max(axis=0)\n",
    "    rng = (-1., 1.)\n",
    "    X_num_scaled = (X_num - xmin) / (xmax - xmin) * (rng[1] - rng[0]) + rng[0]\n",
    "else:  # normalize\n",
    "    mu, sigma = X_num.mean(axis=0), X_num.std(axis=0)\n",
    "    X_num_scaled = (X_num - mu) / sigma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit OHE to categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OneHotEncoder()"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_cat = X[:, :-4].copy()\n",
    "ohe = OneHotEncoder(categories='auto')\n",
    "ohe.fit(X_cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine numerical and categorical data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.c_[X_cat, X_num_scaled].astype(np.float32, copy=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define train, validation (to find outlier threshold) and test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 7) (25000,) (5000, 7) (5000,) (2561, 7) (2561,)\n"
     ]
    }
   ],
   "source": [
    "n_train = 25000\n",
    "n_valid = 5000\n",
    "X_train, y_train = X[:n_train,:], y[:n_train]\n",
    "X_valid, y_valid = X[n_train:n_train+n_valid,:], y[n_train:n_train+n_valid]\n",
    "X_test, y_test = X[n_train+n_valid:,:], y[n_train+n_valid:]\n",
    "print(X_train.shape, y_train.shape,\n",
    "      X_valid.shape, y_valid.shape,\n",
    "      X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create outliers\n",
    "\n",
    "Inject outliers in the numerical features. First we need to know the features for each kind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2] [3, 4, 5, 6]\n"
     ]
    }
   ],
   "source": [
    "cat_cols = list(category_map.keys())\n",
    "num_cols = [col for col in range(X.shape[1]) if col not in cat_cols]\n",
    "print(cat_cols, num_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numerical\n",
    "\n",
    "Now we can add outliers to the validation (or threshold) and test sets. For the numerical data, we need to specify the numerical columns (```cols```), the percentage of outliers (```perc_outlier```), the strength (```n_std```) and the minimum size of the perturbation (```min_std```). The outliers are distributed evenly across the numerical features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.62% outliers\n"
     ]
    }
   ],
   "source": [
    "perc_outlier = 10\n",
    "data = inject_outlier_tabular(X_valid, num_cols, perc_outlier, n_std=8., min_std=6.)\n",
    "X_threshold, y_threshold = data.data, data.target\n",
    "X_threshold_, y_threshold_ = X_threshold.copy(), y_threshold.copy()  # store for comparison later\n",
    "outlier_perc = 100 * y_threshold.sum() / len(y_threshold)\n",
    "print('{:.2f}% outliers'.format(outlier_perc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect an instance that was changed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hours per week changed by -11.79.\n"
     ]
    }
   ],
   "source": [
    "outlier_idx = np.where(y_threshold != 0)[0]\n",
    "vdiff = X_threshold[outlier_idx[0]] - X_valid[outlier_idx[0]]\n",
    "fdiff = np.where(vdiff != 0)[0]\n",
    "print('{} changed by {:.2f}.'.format(feature_names[fdiff[0]], vdiff[fdiff[0]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same thing for the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.76% outliers\n"
     ]
    }
   ],
   "source": [
    "data = inject_outlier_tabular(X_test, num_cols, perc_outlier, n_std=8., min_std=6.)\n",
    "X_outlier, y_outlier = data.data, data.target\n",
    "print('{:.2f}% outliers'.format(100 * y_outlier.sum() / len(y_outlier)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply one-hot encoding\n",
    "\n",
    "OHE to train, threshold and outlier sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 17) (5000, 17) (2561, 17)\n"
     ]
    }
   ],
   "source": [
    "X_train_ohe = ohe.transform(X_train[:, :-4].copy())\n",
    "X_threshold_ohe = ohe.transform(X_threshold[:, :-4].copy())\n",
    "X_outlier_ohe = ohe.transform(X_outlier[:, :-4].copy())\n",
    "print(X_train_ohe.shape, X_threshold_ohe.shape, X_outlier_ohe.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 21) (5000, 21) (2561, 21)\n"
     ]
    }
   ],
   "source": [
    "X_train = np.c_[X_train_ohe.toarray(), X_train[:, -4:]].astype(np.float32, copy=False)\n",
    "X_threshold = np.c_[X_threshold_ohe.toarray(), X_threshold[:, -4:]].astype(np.float32, copy=False)\n",
    "X_outlier = np.c_[X_outlier_ohe.toarray(), X_outlier[:, -4:]].astype(np.float32, copy=False)\n",
    "print(X_train.shape, X_threshold.shape, X_outlier.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load or define outlier detector\n",
    "\n",
    "The pretrained outlier and adversarial detectors used in the example notebooks can be found [here](https://console.cloud.google.com/storage/browser/seldon-models/alibi-detect). You can use the built-in ```fetch_detector``` function which saves the pre-trained models in a local directory ```filepath``` and loads the detector. Alternatively, you can train a detector from scratch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_outlier_detector = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.\n",
      "WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No threshold level set. Need to infer threshold using `infer_threshold`.\n"
     ]
    }
   ],
   "source": [
    "filepath = './models/'  # change to directory where model is downloaded\n",
    "if load_outlier_detector:  # load pretrained outlier detector\n",
    "    detector_type = 'outlier'\n",
    "    dataset = 'adult'\n",
    "    detector_name = 'OutlierVAE'\n",
    "    od = fetch_detector(filepath, detector_type, dataset, detector_name)\n",
    "else:  # define model, initialize, train and save outlier detector\n",
    "    n_features = X_train.shape[1]\n",
    "    latent_dim = 2\n",
    "\n",
    "    encoder_net = tf.keras.Sequential(\n",
    "      [\n",
    "          InputLayer(input_shape=(n_features,)),\n",
    "          Dense(25, activation=tf.nn.relu),\n",
    "          Dense(10, activation=tf.nn.relu),\n",
    "          Dense(5, activation=tf.nn.relu)\n",
    "      ])\n",
    "\n",
    "    decoder_net = tf.keras.Sequential(\n",
    "      [\n",
    "          InputLayer(input_shape=(latent_dim,)),\n",
    "          Dense(5, activation=tf.nn.relu),\n",
    "          Dense(10, activation=tf.nn.relu),\n",
    "          Dense(25, activation=tf.nn.relu),\n",
    "          Dense(n_features, activation=None)\n",
    "      ])\n",
    "\n",
    "    # initialize outlier detector\n",
    "    od = OutlierVAE(threshold=None,  # threshold for outlier score\n",
    "                    score_type='mse',  # use MSE of reconstruction error for outlier detection\n",
    "                    encoder_net=encoder_net,  # can also pass VAE model instead\n",
    "                    decoder_net=decoder_net,  # of separate encoder and decoder\n",
    "                    latent_dim=latent_dim,\n",
    "                    samples=5)\n",
    "\n",
    "    # train\n",
    "    od.fit(X_train,\n",
    "           loss_fn=tf.keras.losses.mse,\n",
    "           epochs=5,\n",
    "           verbose=True)\n",
    "\n",
    "    # save the trained outlier detector\n",
    "    save_detector(od, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The warning tells us we still need to set the outlier threshold. This can be done with the ```infer_threshold``` method. We need to pass a batch of instances and specify what percentage of those we consider to be normal via ```threshold_perc```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New threshold: 1.505036223053932\n"
     ]
    }
   ],
   "source": [
    "od.infer_threshold(X_threshold, threshold_perc=100-outlier_perc, outlier_perc=100)\n",
    "print('New threshold: {}'.format(od.threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s save the outlier detector with updated threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n"
     ]
    }
   ],
   "source": [
    "save_detector(od, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detect outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "od_preds = od.predict(X_outlier,\n",
    "                      outlier_type='instance',\n",
    "                      return_feature_score=True,\n",
    "                      return_instance_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Display results\n",
    "\n",
    "F1 score and confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.89 -- Accuracy: 0.98 -- Precision: 0.90 -- Recall: 0.88\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAD4CAYAAAAw/yevAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAa9ElEQVR4nO3deZwU1bn/8c/DLiibyM5PxaBGTSSKu7nRSECMBoyIJC6gIl6FqD93c6MoJuJNwKhxBUUwUVxwARMBEfcVUAiKoiJuIPu+RZjp5/7RNdjALDUz3dNnar5vX/Xq6lOn65zW8ZkzT506Ze6OiIiEpVa+OyAiIjtTcBYRCZCCs4hIgBScRUQCpOAsIhKgOlXQhqaDiEhcVtkTbF2xIHbMqduiY6Xby5WqCM5sXbGgKpqRaqRui47Uqdcu392QwBRsWVT5k6QKK3+OAFRJcBYRqTKeyncPskLBWUSSJaXgLCISHNfIWUQkQIUF+e5BVig4i0iy6IKgiEiAlNYQEQmQLgiKiIRHFwRFREKkkbOISIAKt+a7B1mh4CwiyaK0hohIgJTWEBEJkEbOIiIB0shZRCQ8ntIFQRGR8GjkLCISIOWcRUQCpIWPREQCpJGziEiAlHMWEQmQFtsXEQmQRs4iIuFxT8YFwVr57oCISFalUvG3UphZBzN72cw+MrO5ZnZpVN7czKaa2WfRa7Oo3MzsTjObb2ZzzOyQjHP1i+p/Zmb94nwNBWcRSRZPxd9KVwBc4e4HAEcCg8zsAOBaYJq7dwKmRe8BegCdom0gcC+kgzkwBDgCOBwYUhTQS6PgLCLJkqWRs7svdvf3o/31wMdAO6AnMDaqNhboFe33BB72tHeApmbWBugOTHX3Ve6+GpgKnFjW11DOWUSSpRyzNcxsIOlRbpGR7j6ymHp7AT8B3gVaufvi6NASoFW03w74JuNjC6OykspLpeAsIslSjptQokC8UzDOZGa7Ak8Bl7n7OjPL/LybmVewp6VSWkNEkiVLaQ0AM6tLOjA/4u5PR8VLo3QF0euyqHwR0CHj4+2jspLKS6XgLCLJkr3ZGgY8CHzs7rdlHJoIFM246AdMyCg/J5q1cSSwNkp/TAG6mVmz6EJgt6isVEpriEiyZG9tjWOAs4EPzGx2VPZ74FbgCTM7H/gK6BMdex44CZgPbALOBXD3VWZ2MzAjqjfU3VeV1biCs4gkS5Zu33b3NwAr4fAJxdR3YFAJ5xoNjC5P+wrOIpIsun1bRCRAWjJURCRAGjmLiARIwVlEJECek3tCqpyCs4gkS0ENWGw/Wk2pRHHm6omIVKkackHwPcApfq6fAx2z3iMRkcqoCTlnd9+7qjoiIpIVNS3nHN0T3gloUFTm7q/lolMiIhVWE0bORcxsAHAp6dWUZpN+KsDbwM9z1jMRkYpISHCOuyrdpcBhwFfufjzpRafX5KpTIiIV5YWFsbeQxU1r/Mfd/2NmmFl9d59nZvvltGciIhWRkJFz3OC80MyaAs8CU81sNeml8kREwlJDptIB4O6nRrs3mtnLQBNgcs56JSJSUamaOVujA7A+2g4C3s9Rv0REKqYmpTWiVfz7AwuAom/uaLaGiIQm8At9ccUdOfcB9nH3LbnsTEgWL13O728ezsrVqzGM3j17cHafXtvVeen1t/nbqIepZbWoXbs21146kEMOPqhS7a5dt54rrh/Gt0uW0rZ1K0bcfB1NGu+Wk7ak6rVv35Yxo++gZasWuDsPPPAIf7vrQW668SpOOaUbqZSzfNkKzhvw/1m8eGm+u1s9JWTkbB7jbhozewq4yN2XlVl5Z751xYIKfCy/lq9YxfKVqzhgvx+wceMm+px/CXcOu5599t5zW51Nmzazyy4NMDM+mf8FV15/C8+NGxXr/NPfn8OE56fypz9csV35iLsfpEnj3Rhwdh8e+PsTrFu/nssvPr9SbYWobouO1KnXLt/dqHKtW7ekTeuWzJr9Ibvu2ojp707mtN7nsXDhYtav3wDA4EHn8cMf7sugwdfmubdVr2DLopIeCxXbpuEDYiedG175QKXby5W485yHAbPMbIqZTSzactmxfNujRXMO2O8HADRq1JCOe3Zg6fKV29Vp2HAX0g/ohc3/+Q/Y9/+dRz8ynjPOv4RTz7mIux74e+x2X379bXr26ApAzx5deem1t8tsS6qPJUuWMWv2hwBs2LCRefM+o13b1tsCM6R/3uIMmqQEnoq/BSxuWmMs8L/AB3yfc64xFi1eyseffc6PD9x5aveLr77JHfeNYeXqNdwzfCgAb777Hl8vXMRjD9yBuzP4mpuYOfsDunT+UZltrVy9hj1apBcDbLF7M1auXlNqW1J97blnezoffBDvTp8FwM1Dr+GsM3uzdt06uv7i9Dz3rhpLyGyNuGmNGe5+WOyTmg0EBgLcf//9h577664V72Gebdq0mf6Dr+aCc/ryi+OOKbHezNkfcN9Dj/LAHcP4y12jmPryG+y2667pc2zezICzz+C0U7rzmwsuY8uWrWzavJm169bTplVLAC6/+DyOOeJQjurem7enjN923qNPPJ23Jj9ZYlvVVU1NaxRp1KghL017imG33smzz07a7tg1Vw+mQYP63DR0RJ56lz/ZSGtsHNYvdnRudN3YYP8EjTtyft3MhgETge+KCt292Kl07j4SGFn0tjrmnAG2FhRw2f/8kV92O77UwAzQpfOPWPjtElavWQsOA84+gz69Ttqp3rhRtwMl55x3b9aU5StWsUeL5ixfsYrmTZuU2lazYo5L2OrUqcOTj49i3LhndgrMAI+Oe5rnJv69RgbnrEjIbI24OeefkF7s6BZgRLQNz1WnQuDu3DDsdjru2YF+fX9dbJ2vF367LTf40Sfz2bJlK02bNOboww/hmX+9wKZNmwFYunzFdumJ0hx37JFMmPQiABMmvcjxPz2q1Lak+hk1cgQfz5vP7XeM3Fb2gx98vzrvr07pzieffJ6PriVDyuNvAStz5GxmtYGJ7v7XKuhPMGbNmctzk6fRaZ+9OK3fIAAuvbAfi5cuB+CMU3/J1FfeYOKkadSpU4cG9esxfOi1mBnHHHEoC776hjMvvByAhrs0YNgNV7F7s6Zltjvg7D5ccf0tPP3PKbRt3ZIRN/8eoMS2pHo55ujDOPus3sz54CNmzngBgOuvv5Vzz+3LvvvuQyqV4uuvF3HxoJo3UyNrathUuunufngF26i2aQ3JnZqec5biZSXnfEPf+DnnoY8FO8KJm3N+08zuAh4HNhYVlpRzFhHJm8CnyMUVNzh3jl4z52/p9m0RCU/gueS44q5Kd3yuOyIikg1eUINma5hZEzO7zcxmRtsIM9McLhEJT0Jma8SdSjea9DKhfaJtHfBQrjolIlJhNez27X3c/bSM9zeZ2ewc9EdEpHICHxHHFXfkvNnMji16Y2bHAJtz0yURkYrzlMfeQhZ35HwRMDYjz7wa6JebLomIVEJCLgjGDc4fA38G9gGaAmuBXsCcnPRKRKSiAh8RxxU3OE8A1pB+ZuCinPVGRKSyalhwbu/uJ+a0JyIiWZCUBxXEvSD4lpmVvVK8iEi+JWSec9yR87FAfzP7gvR6zga4u/84Zz0TEamIwINuXHFHzj2ATkA34BTg5OhVRCQoXpCKvZXFzEab2TIz+zCj7EYzW2Rms6PtpIxj15nZfDP7xMy6Z5SfGJXNN7NY68HGXVvjqzj1RETyLrs3/o0B7gIe3qH8r+6+3QNHzOwAoC9wINAWeNHM9o0O3w38AlgIzDCzie7+UWkNx01riIhUC9m8ucTdXzOzvWJW7wk85u7fAV+Y2XygaB38+e6+AMDMHovqlhqc46Y1RESqh3JcEDSzgRkLus2MHk4dx2AzmxOlPZpFZe2AbzLqLIzKSiovlYKziCRLKv7m7iPdvUvGNrKk02a4l/QNeZ2BxaSfqZp1SmuISKLkes0Md19atG9mo4B/Rm8XAR0yqrbn+5v2SiovkUbOIpIoXuCxt4owszYZb08FimZyTAT6mll9M9ub9Ay36cAMoJOZ7W1m9UhfNJxYVjsaOYtIsmRxtoaZjQOOA1qY2UJgCHCcmXUm/ai+L4ELAdx9rpk9QfpCXwEwyN0Lo/MMBqYAtYHR7j63zLar4FZHPX1bdqKnb0txsvH07ZWn/Cx2UNv9uVer/dO3RUSqh7AfcBKbgrOIJErgT5+KTcFZRBLFC/Ldg+xQcBaRRNHIWUQkQArOIiIh8mAnYJSLgrOIJIpGziIiAfKURs4iIsFJFSo4i4gER2kNEZEAKa0hIhKg3C8XVDUUnEUkUTRyFhEJkC4IiogESCNnEZEAue4QFBEJj6bSiYgEKKWRs4hIeJTWEBEJkGZriIgESLM1REQCpJyziEiAlHMWEQmQ1tYQEQmQ0hoiIgFK6YKgiEh4NHIuh7otOlZFM1LNFGxZlO8uSALpgmB5GqnXriqakWqkYMsiOrb4Sb67IYFZsGJWpc+hkbOISIASMllDwVlEkqUwVSvfXcgKBWcRSZSErBiq4CwiyeIo5ywiEpxUQpLOCs4ikigpjZxFRMKjtIaISIAKFZxFRMKj2RoiIgFKSnBOxmxtEZGIY7G3spjZaDNbZmYfZpQ1N7OpZvZZ9NosKjczu9PM5pvZHDM7JOMz/aL6n5lZvzjfQ8FZRBIlZfG3GMYAJ+5Qdi0wzd07AdOi9wA9gE7RNhC4F9LBHBgCHAEcDgwpCuilUXAWkURJYbG3srj7a8CqHYp7AmOj/bFAr4zyhz3tHaCpmbUBugNT3X2Vu68GprJzwN+JgrOIJEphOTYzG2hmMzO2gTGaaOXui6P9JUCraL8d8E1GvYVRWUnlpdIFQRFJlJTFn0rn7iOBkRVty93dzHJyT6JGziKSKF6OrYKWRukKotdlUfkioENGvfZRWUnlpVJwFpFESZVjq6CJQNGMi37AhIzyc6JZG0cCa6P0xxSgm5k1iy4EdovKSqW0hogkSjaf72pm44DjgBZmtpD0rItbgSfM7HzgK6BPVP154CRgPrAJOBfA3VeZ2c3AjKjeUHff8SLjThScRSRRsnn7trv/poRDJxRT14FBJZxnNDC6PG0rOItIomRz5JxPCs4ikihJuX1bwVlEEiUha+0rOItIsiitISISIKU1REQCVKiRs4hIeDRyFhEJkIKziEiANFtDRCRAmq0hIhIgpTVERAJUmO8OZImCs4gkitIaIiIBUlpDRCRAmq0hIhKgVELCs4KziCSKLgiKiARIOWcRkQAlZbZGmU/fNrNaZnZ0VXRGRKSyUnjsLWRlBmd3TwF3V0FfREQqzcuxhazM4ByZZmanmVlC/mAQkaRKlWMLWdyc84XA5UChmW0GjPSTwBvnrGciIhVQGPyYOJ5Ywdndd8t1R0REsiH0EXFcsdIalnaWmV0fve9gZofntmsiIuVXYy4IRu4BjgJ+G73fgC4SikiAknJBMG7O+Qh3P8TMZgG4+2ozq5fDfomIVEhS0hpxg/NWM6tN9MvGzPYgOf8ORCRBatQFQeBO4BmgpZn9CegN/CFnvRIRqaDQc8lxxco5u/sjwNXAMGAx0Mvdn8xlx6q7+vXr8/ab/+S9mVP59+yXGHLDFQBcfFF/5n30BgVbFrH77s3y3EsprzZtW/HIsyOZ8uZTTH5jPP0H/manOj179+D5Vx9n0mtP8OTzY9j/wH0r3W69enW584FbeWn6BJ6e8jDtOrQB4NifHcGEaY8w6bUnmDDtEY766WGVbqu6qxE5ZzNr7O7rzKw5sAwYl3GsubuvynUHq6vvvvuOrt36sHHjJurUqcNrrzzD5Mkv89bbM/jX8y8yber4fHdRKqCgsJBbbriNuXPm0WjXhkyc9ihvvPIu8z9dsK3ON199S99fDWDd2vX87IRjuOW2P/Dr7ufEOn+7Dm34y11D+W3PC7Yr73NmL9atWc/PD+/Jyad255ohl3LJgGtZtWoNF5x5GcuWLGff/fdhzJP3cPSPumf1O1c3SRk5l5XWeBQ4GXiP9C8a2+G1Y057V81t3LgJgLp161Cnbl3cndmz5+a5V1IZy5euYPnSFQBs3LCJ+Z9+Qes2e2wXnN+f8e9t+7NmzqF121bb3vc8/ST6X/Ab6taty+z3P+CGq4aRSpV9+aZrj+O448/3AzBp4ovceOs1AHz0wSfb6nw673MaNKhPvXp12bJla+W+aDWWlIthpaY13P3k6HVvd++442vVdLH6qlWrFjNnvMDiRXOYNu01ps+Yle8uSRa169CGA3+0H7Pf+7DEOn3O6sWr094EYJ9Oe3Nyr26cftK5nHx8X1KFKXr2PilWW63atGTxoiUAFBYWsn7dBpo1b7pdnR6ndGXunHk1OjADeDn+CVlZaY1DSjvu7u+X8LmBwECA+++/v8Kdq+5SqRRdDutGkyaNeerJBznwwP2YO/eTsj8owWvYaBfuGTOcm/9nOBs2bCy2zpHHdqHPmb3o88vzADj6vw7noIMP4Nmp/wCgwS71WbkinRm8d+wIOvy/dtStV5e27Vrzz5cfA2DMyEcZP25imf3ptF9Hrr7hEvqdfnE2vl61VlNma4wo5ZgDPy/2gPtIYGTR24sH31SBriXH2rXreOXVN+ne7TgF5wSoU6cO9zw0nInjJzHlXy8VW2f/Azox7K83cF7fwaxZvRYAM+Ppx57jL3/82071L+qXvmBcUs556eJltGnXmiWLl1G7dm12a7wrq1etAaB1m5bc9/BtXDnoer7+cmEWv2n1VFPSGse7+/FAj6L9jLJ4f4/VUC1aNKdJk/S6UA0aNKDrCf/FJ598nudeSTbcescQPv/0Cx689x/FHm/brjX3jBnOFRdfzxeff72t/K3XptPjV13ZvUV6lk6Tpo1p275NrDanTX6V0/qeAkCPX3Xl7ddnALBb4115cNzf+PPQO3lv+r9LO0WNkXKPvYUs7jznt4AdUxzFlUmkTZtWjH7wdmrXrkWtWrUYP/45/vX8iwwedB5XXnExrVvvwaz3XmTS5Je48L+vynd3JaYuR3Tm12eczLy5n25LPQz/0120bd8agEfHjOd3Vw2kWfOmDP3zdUA6R9yz65nM/3QBI265m7FP3kutWsbWggKGXH0r3y5cXGa7jz/yLLfd80demj6BtWvWcckF1wJwzoC+7Ll3B3535UB+d+VAAPqdfhErV6zOxdevFsIOufGZl/Lbw8xaA+2Af5BeV6NoPefGwH3uvn+MNrxOvXaV7ackTMGWRXRs8ZN8d0MCs2DFrEqvGf/bPU+NHZ8f/eqZYNeoL2vk3B3oD7QHbssoXw/8Pkd9EhGpsNBnYcRVanB297HAWDM7zd2fqqI+iYhUWEEWg7OZfUl6MFoIFLh7l+imvMeBvYAvgT7RYnAG3EH6etwmoH9JM9riiJtzPsjMDtyx0N2HVrRhEZFcyMHI+Xh3X5Hx/lpgmrvfambXRu+vAXoAnaLtCODe6LVC4q7nvAHYGG2FUSf2qmijIiK5UgXPEOwJjI32xwK9Msof9rR3gKZmFm86TjHiPqZqu/nOZjYcmFLRRkVEcqW0SQ47yrxhLjIyuk9j2+mAF8zMgfujY63cvWiKzRKg6P78dsA3GZ9dGJWVPR2nGHHTGjtqSPoioYhIUMqz8NEON8wV51h3X2RmLYGpZjZvh897FLizLlZwNrMP+H76YC2gJXBzLjokIlIZ2bx9290XRa/LzOwZ4HBgqZm1cffFUdpiWVR9EdAh4+Pto7IKiZtzPhk4BxhF+iplD3ff+R5UEZE8y9YDXs2skZntVrQPdAM+BCYC/aJq/YAJ0f5E4JzogdhHAmsz0h/lFjet0RO4AHia9I0oD5nZKAVoEQlNeXLOZWgFPJOeIUcd4FF3n2xmM4AnzOx84CugT1T/edLT6OaTnkp3bmUajxucBwBHuvtGADP7X+BtQMFZRIKSrYWP3H0BcHAx5SuBE4opd2BQlpqPHZyN9BS6IoV8fyu3iEgwasQdghkeAt6NEuKQntf3YE56JCJSCTXlMVUAuPttZvYKcGxUdK6767EeIhKcQk/Gis6x5zlH94hX+D5xEZGqUNPSGiIi1ULoi+jHpeAsIomSjNCs4CwiCVOjLgiKiFQXCs4iIgGqcbM1RESqA83WEBEJUBbX1sgrBWcRSRTlnEVEAqSRs4hIgAqzti5dfik4i0ii6A5BEZEAabaGiEiANHIWEQmQRs4iIgHSyFlEJEC6fVtEJEBKa4iIBMg1chYRCY9u3xYRCZBu3xYRCZBGziIiASpMKecsIhIczdYQEQmQcs4iIgFSzllEJEAaOYuIBEgXBEVEAqS0hohIgJTWEBEJkJYMFREJkOY5i4gESCNnEZEApbRkqIhIeHRBUEQkQArOIiIBSkZoBkvKb5nqwMwGuvvIfPdDwqKfCylOrXx3oIYZmO8OSJD0cyE7UXAWEQmQgrOISIAUnKuW8opSHP1cyE50QVBEJEAaOYuIBEjBWUQkQArO1YiZfWlmLfLdD6kcM+tvZm0z3r9iZl2i/efNrGneOifBUHCuImamuzGlSH+gbXEH3P0kd18T90RmVjtLfZLAKDiXg5ntZWYfm9koM5trZi+Y2S5m1tnM3jGzOWb2jJk1i+q/Yma3m9lM4NLo/V/NbGZ0nsPM7Gkz+8zM/pjRzrNm9l7Uhm5QqAbM7HIz+zDaLot+Vj7MOH6lmd1oZr2BLsAjZjbbzHbZ4Tzb/joys7PMbHpU7/6iQGxmG8xshJn9GziqCr+mVCEF5/LrBNzt7gcCa4DTgIeBa9z9x8AHwJCM+vXcvYu7j4jeb3H3LsB9wARgEHAQ0N/Mdo/qnOfuh5L+n/iSjHIJkJkdCpwLHAEcCVwANCuurruPB2YCZ7p7Z3ffXMI5fwicARzj7p2BQuDM6HAj4F13P9jd38jmd5Fw6E/t8vvC3WdH++8B+wBN3f3VqGws8GRG/cd3+PzE6PUDYK67LwYwswVAB2Al6YB8alSvA+lfCCuz+SUkq44FnnH3jQBm9jTw00qe8wTgUGCGmQHsAiyLjhUCT1Xy/BI4Befy+y5jvxBoWkb9jSV8PrXDuVJAHTM7DugKHOXum8zsFaBBBfsq+dOU7f8yLe9/QwPGuvt1xRz7j7sXVrRjUj0orVF5a4HVZlY0UjobeLWU+mVpAqyOAvP+pP9MlrC9DvQys4Zm1gg4FZgEtDSz3c2sPnByRv31wG5lnHMa0NvMWgKYWXMz2zMHfZdAaeScHf2A+8ysIbCAdP6xoiYD/21mHwOfAO9koX+SQ+7+vpmNAaZHRQ+4+wwzGxqVLQLmZXxkDOmfl82UcEHP3T8ysz8AL5hZLWAr6esTX+XmW0hodPu2iEiAlNYQEQmQgrOISIAUnEVEAqTgLCISIAVnEZEAKTiLiARIwVlEJED/B9M5P3y7JLdMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "labels = data.target_names\n",
    "y_pred = od_preds['data']['is_outlier']\n",
    "f1 = f1_score(y_outlier, y_pred)\n",
    "acc = accuracy_score(y_outlier, y_pred)\n",
    "prec = precision_score(y_outlier, y_pred)\n",
    "rec = recall_score(y_outlier, y_pred)\n",
    "print('F1 score: {:.2f} -- Accuracy: {:.2f} -- Precision: {:.2f} -- Recall: {:.2f}'.format(f1, acc, prec, rec))\n",
    "cm = confusion_matrix(y_outlier, y_pred)\n",
    "df_cm = pd.DataFrame(cm, index=labels, columns=labels)\n",
    "sns.heatmap(df_cm, annot=True, cbar=True, linewidths=.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot instance level outlier scores vs. the outlier threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8jklEQVR4nO2dfZxUVf34358ddpcBlAVB5UEETVFAQMGnQsXMxxKBDCv9qt9KtCzTytTyZ2iZomZKX/sWPldWEiKiaaTkQ/rNdFFcQCVSSVhQBNkVYWFnd87vj3tnmZm9986dmXvn8fN+vXZn5tynz7nnns8953M+53PEGIOiKIpSPdQUWwBFURSlsKjiVxRFqTJU8SuKolQZqvgVRVGqDFX8iqIoVYYqfkVRlCojNMUvIvuIyNMi8rqIrBSRb9vps0SkWUSW2X+nhSWDoiiK0h0Jy49fRAYBg4wxr4jIbsBSYCowA/jYGHNLKBdWFEVRPOkR1omNMRuADfb3rSLyBjAkrOspiqIo/gitxZ9yEZHhwHPAGOA7wPnAR0Aj8F1jzBaHY2YCMwF69+494aCDDgpdTkVRlEpi6dKlm4wxA9PTQ1f8ItIHeBa43hizQET2AjYBBvgxljnoK17nmDhxomlsbAxVTkVRlEpDRJYaYyamp4fq1SMitcBDwAPGmAUAxpj3jTGdxpg4cCdwRJgyKIqiKKmE6dUjwN3AG8aYW5PSByXtNg1YEZYMiqIoSndCG9wFPgX8F7BcRJbZaT8AviQi47FMPWuAC0OUQVEURUkjTK+e5wFx2PR4EOePxWKsW7eOHTt2BHG6qqdnz54MHTqU2traYouiKErIhNniD5V169ax2267MXz4cCyrkpIrxhg2b97MunXrGDFiRLHFURQlZMo2ZMOOHTvYY489VOkHgIiwxx57aO9JUaqEslX8gCr9ANF7qSjVQ1krfkVRFCV7VPGXMcOHD2fTpk3FFkNRlDKjbAd3s2Xhq83cvHgV61vaGNwQ5fKTRzL10OKFDuro6KBHj6q5/YqilBBV0eJf+GozVy1YTnNLGwZobmnjqgXLWfhqc17nXbNmDQcffDAXXHABo0eP5qSTTqKtrY1ly5Zx1FFHMXbsWKZNm8aWLVYoosmTJ3PppZcyceJEbr/9diZPnsxll13GxIkTOfjgg3n55ZeZPn06BxxwAFdffXXXdaZOncqECRMYPXo0c+fOzUtmRVGUqlD8Ny9eRVusMyWtLdbJzYtX5X3u1atXc/HFF7Ny5UoaGhp46KGHOPfcc5k9ezZNTU0ccsghXHvttV37t7e309jYyHe/+10A6urqaGxs5KKLLuKMM87gjjvuYMWKFdx3331s3rwZgHvuuYelS5fS2NjInDlzutIVRVFyoSoU//qWtqzSs2HEiBGMHz8egAkTJvDWW2/R0tLCcccdB8B5553Hc88917X/WWedlXL8lClTADjkkEMYPXo0gwYNor6+nv3224+1a9cCMGfOHMaNG8dRRx3F2rVrWb16dd5yK4pSvVSFkXlwQ5RmByU/uCGa97nr6+u7vkciEVpaWjz37927t+PxNTU1Keeqqamho6ODZ555hqeeeop//OMf9OrVi8mTJ6u/vaIoeVEVLf7LTx5JtDaSkhatjXD5ySMDv1bfvn3p168ff//73wH47W9/29X6z4XW1lb69etHr169ePPNN3nxxReDElVRlCqlKlr8Ce+dQnn13H///Vx00UVs376d/fbbj3vvvTfnc51yyin86le/4uCDD2bkyJEcddRRAUqqKEo1UpAVuPLFaSGWN954g4MPPrhIElUmek8VpbIoykIsiqIoSumhil9RFKXKUMWvKIpSZajiVxRFqTJU8SuKolQZqvgVRVGqDFX8BeK+++5j/fr1Xb8nT55MwkX1tNNOyzjjV1EUJSiqR/E3zYOfj4FZDdZn07yCXj5d8Sfz+OOP09DQ4PtcnZ2dmXdSFEVxoToUf9M8ePQSaF0LGOvz0UvyVv633norY8aMYcyYMdx2222sWbOGMWPGdG2/5ZZbmDVrFvPnz6exsZGzzz6b8ePH09aWGjcoeUGV3/3udxxxxBGMHz+eCy+8sEvJ9+nTh+9+97uMGzeOf/zjH3nJrShKdVMdin/JdRBLC9IWa7PSc2Tp0qXce++9/POf/+TFF1/kzjvv7Iq7n86ZZ57JxIkTeeCBB1i2bBnRqHNwuDfeeIMHH3yQF154gWXLlhGJRHjggQcA2LZtG0ceeSSvvfYakyZNylluRVGUqojVQ+u67NJ98PzzzzNt2rSuaJvTp0/vCsyWK0uWLGHp0qUcfvjhALS1tbHnnnsCVuTPz3/+83mdX1EUBapF8fcdapt5HNIDpKWlhXg83vU72/DJxhjOO+88brjhhm7bevbsSSQScThKURQlO6rD1HPCNVCbZl6pjVrpOXLMMcewcOFCtm/fzrZt23j44Yc59dRT2bhxI5s3b2bnzp089thjXfvvtttubN261VvME05g/vz5bNy4EYAPP/yQ//znPznLqCiK4kR1tPjHzrA+l1xnmXf6DrWUfiI9Bw477DDOP/98jjjiCAC+9rWvcfjhh3PNNddwxBFHMGTIEA466KCu/c8//3wuuugiotGo6+DsqFGj+MlPfsJJJ51EPB6ntraWO+64g3333TdnORVFUdLRsMxKF3pPFaWy0LDMiqIoCqCKX1EUpepQxa8oilJlqOJX3ClymAtFUcKhOrx6lOxJhLlIzHhOhLmAvLyhFEUpPtriV5wJIcyFoiilgSr+HNm8eTPjx49n/Pjx7L333gwZMoTx48fT0NDAqFGjAr/erFmzuOWWW7I6pk+fPo7p559/PvPnz/c+OIQwF4qilAahKX4R2UdEnhaR10VkpYh8207vLyJPishq+7NfWDKEyR577MGyZctYtmwZF110EZdddlnX75qazLe1o6OjAFLmgVs4i4DDXCiKUnjCbPF3AN81xowCjgIuFpFRwJXAEmPMAcAS+3dF0dnZyQUXXMDo0aM56aSTusIwT548mUsvvZSJEydy++23s3TpUo477jgmTJjAySefzIYNGwCYM2cOo0aNYuzYsXzxi1/sOu/rr7/O5MmT2W+//ZgzZ05Xenp46HSMMXzzm99k5MiRfOYzn+kKCeFJCGEuFEUpDUIb3DXGbAA22N+3isgbwBDgDGCyvdv9wDPAFflc69K/XMqy95blc4pujN97PLedcltOx65evZo//OEP3HnnncyYMYOHHnqIc845B4D29nYaGxuJxWIcd9xxPPLIIwwcOJAHH3yQH/7wh9xzzz3ceOONvPPOO9TX16eszPXmm2/y9NNPs3XrVkaOHMnXv/51mpqausJDG2M48sgjOe644zj00EO7jnv44YdZtWoVr7/+Ou+//z6jRo3iK1/5incmQghzoShKaVAQrx4RGQ4cCvwT2Mt+KQC8B+zlcsxMYCbAsGHDCiBlcIwYMYLx48cDMGHCBNasWdO17ayzzgJg1apVrFixghNPPBGwegmDBg0CYOzYsZx99tlMnTqVqVOndh372c9+lvr6eurr69lzzz15//33XcNDJyv+5557ji996UtEIhEGDx7Mpz/9aX8ZGTtDFb2iVCChK34R6QM8BFxqjPlIRLq2GWOMiDgGCzLGzAXmghWrx+saubbMw6K+vr7reyQSSVlxK6GgjTGMHj3aMWDbn//8Z5577jkeffRRrr/+epYvX+543pIfJ1AUpSQJ1atHRGqxlP4DxpgFdvL7IjLI3j4I8GFwrjxGjhzJBx980KX4Y7EYK1euJB6Ps3btWo4//nhmz55Na2srH3/8set5nMJDH3PMMSn7HHvssTz44IN0dnayYcMGnn766VDzpihKaRNai1+spv3dwBvGmFuTNi0CzgNutD8fCUuGUqauro758+dzySWX0NraSkdHB5deeikHHngg55xzDq2trRhjuOSSSzwXYncKD51s5gGYNm0af/vb3xg1ahTDhg3j6KOPDjNriqKUOKGFZRaRScDfgeVAYlmqH2DZ+ecBw4D/ADOMMR96nUvDMhcGvaeKUlm4hWUO06vneUBcNp8Q1nUVRVEUb3TmrqIoSpVR1oq/HFYPKxf0XipK9VC2ir9nz55s3rxZFVYAGGPYvHkzPXv2LLYoiqIUgLINyzx06FDWrVvHBx98UGxRKoKePXsydKjG4VGUaqBsFX9tbS0jRowothiKoihlR9maehRFUZTcUMWvKIpSZajiVxRFqTJU8SuKolQZvhS/iOwrIp+xv0dFZLdwxVIURVHCIqPiF5ELgPnAr+2kocDCEGVSFEVRQsRPi/9i4FPARwDGmNXAnmEKpShVQdM8+PkYmNVgfTbNK7ZESpXgx49/pzGmPbGAioj0AHS6bBA0zdOlDauVpnnw6CUQsxfpaV1r/QZ9BpTQ8dPif1ZEfgBEReRE4E/Ao+GKVQUkKn7rWsDsqvja6qsOlly3S+kniLVZ6YoSMn4U/xXAB1hx9S8EHgeuDlOoqkArfnXTui67dEUJEE9Tj4hEgJXGmIOAOwsjUpWgFb+66TvU7u05pCtKyHi2+I0xncAqERlWIHmqB7cKrhW/OjjhGqiNpqbVRq10RQkZP6aefsBKEVkiIosSf2ELVvFoxa9uxs6A0+dA330AsT5Pn6MDu0pB8OPV8/9Cl6IaSVRw9eqpXsbO0PJWikJGxW+MeVZE9gIOt5NeMsZsDFesKkErvqIoRcDPzN0ZwEvAF4AZwD9F5MywBVMURVHCwY+p54fA4YlWvogMBJ7CCuOgKIqilBl+Bndr0kw7m30epyiKopQgflr8fxGRxcAf7N9nAU+EJ5KiKIoSJn4Gdy8XkenAJDtprjHm4XDFUhRFUcIio+IXkRHA48aYBfbvqIgMN8asCVs4RVEUJXj82Or/BMSTfnfaaYqiKEoZ4kfx9zDGtCd+2N/rwhNJURRFCRM/iv8DEZmS+CEiZwCbwhNJURRFCRM/Xj0XAQ+IyP8AAqwFzg1VKkVRFCU0/Hj1vAUcJSJ97N8fhy6VoiiKEhquph4ROV1E9k1K+g7wgh2dc0T4oimKoihh4GXjvx5r5S1E5HPAOcBXgEXAr8IXTVEURQkDL8VvjDHb7e/TgbuNMUuNMXcBA8MXTVEURQkDL8UvItJHRGqAE4AlSdt6ZjqxiNwjIhtFZEVS2iwRaRaRZfbfabmLriiKouSCl+K/DVgGNAJvGGMaAUTkUGCDj3PfB5zikP5zY8x4++/xrKRVFEVR8sbVq8cYc48dnG1P4LWkTe8B/53pxMaY50RkeN4SKopSWTTN05XnikymxdabjTGvGmPiSWkbjDHv5nHNb4pIk20K6ue2k4jMFJFGEWn84IMP8ricoiglQ9M8ePQSaF0LGOvz0UusdKVgFDqu/v8C+wPjscxFP3Pb0Rgz1xgz0RgzceBAHUtWlIpgyXUQa0tNi7VZ6UrBKKjiN8a8b4zptHsQdwJHFPL6iqIUmdZ12aUroeBq4xeR/l4HGmM+zPZiIjLIGJMYGJ4GrPDaX1GUCqPvUNvM45CuFAyvkA1LAYMVnycdA+zndWIR+QMwGRggIuuAHwGTRWS8ffwa4MKsJVaqEx0QrAxOuMay6Sebe2qjVrpSMLy8evIKy2CM+ZJD8t35nFOpUhIDggllkRgQhGCUv75UCkfivoZ9v7VMPRFjjPcOIgKcDYwwxvxYRIYBextjXiqEgAATJ040jY2NhbqcUmr8fIyLeWAfuCxPa2H6SwWsFujpc1RRlCtapl2IyFJjzMT0dD+Du78Ejga+bP/eCtwRoGyK4k2YA4LqZVJ5aJlmxI/iP9IYczGwA8AYswVdgUspJG4Df0EMCKqXSeVRbmXaNM/q1c5qsD4LMKfBj+KPiUgEa0AWERlI6hq8ihIuJ1xjddWTCWpAMMyXilIcyqlMizShzY/inwM8DOwpItcDzwM/DVUqRUlm7AzLPtt3H0Csz6DstWG+VJTiUE5lWiSzlJ8VuB4QkaVYEToFmGqMeSNUqRQlnbEzwhmYK5SXiVI4yqlMi2SWyqj4RWQO8EdjjA7oKpVJWC8VpXiUS5kWaUKbH1PPUuBqEXlLRG4RkW6uQYqiKEoOFMkslVHxG2PuN8acBhwOrAJmi8jqUKVSFEWpBsIcv/Igo6kniU8ABwH7AmrjV5Sw0Fmn1UURzFJ+bPw3YQVUewv4I/BjY0xLyHIpSnUSdngKRcFfi/8t4GhjzKawhVGUqsfLvU8VvxIQfgZ37wROEZFrAERkmIhoHH1FCYNym3WqlCV+FP8dWLF6EtE2NVaPooRFOc06VcoWjdWjKKVEOc06VcoWjdWjKKVEkdz7lOrCz+BueqyeM4GrQ5VKUaqZcpl1qpQtOcXqAVpDlktRFEUJCV8TuIwxbwJvJn6LyLvAsLCEUhRFUcLDj43fCacF2BVFUZQyIFfF771Qr6IoilKyuJp6ROQXOCt4ARrCEkhRFEUJFy8bf2OO2xRFUZQSxlXxG2PuL6QgRUWjISqKUkVkE5a5MtFoiIqiVBm5Du5WDkVa7FhRFKVYqOLXaIiKolQZGRW/iBwoIktEZIX9e6yIVE7IBo2GqChKleE3Hv9VQAzAGNMEfDFMoQqKRkNUikHTPPj5GJjVYH02zSu2REoV4Wdwt5cx5iWRlMm6HSHJU3gSA7il4tWjHkaVjzoUKEXGj+LfJCL7syss85nAhlClKjSlEg1RFUJ1oMsrKkXGj6nnYuDXwEEi0gxcCnw9TKFKikJ2ydXDqDpQhwKlyPgJy/w28BkR6Q3UGGO2hi9WiVDoFrhfhaDmoPKm71DrWXJKV5QC4Mer56ci0mCM2WaM2Soi/UTkJ4UQrugUugXux8Mo8TJqXQuYXS8jHRwsH/w6FOgAsBISfkw9pxpjWhI/7DV3TwtNolKi0F1yPwpBzUHlj5/lFfUFr4SIn8HdiIjUG2N2AohIFKjPdJCI3AN8DthojBljp/UHHgSGA2uAGfaLpDQpdJfcj4eR2ocrg0wOBToArISInxb/A8ASEfmqiHwVeBLwE8DtPuCUtLQrgSXGmAOAJfbv0qUYPv5jZ8BlK2BWi/WZXsl1wll1oC94JUQyKn5jzGzgeuBg++/HxpibfBz3HPBhWvIZ7Hpp3I+1fm/p4qdLXmh0wll1oC94JUT8rrn7BPBEANfbyxiTmAPwHrCX244iMhOYCTBsWBGX983Vxz8sz5tSm3CmhMMJ16R6lIG+4JXAEGO8V1EUkenAbGBPrNW3BDDGmN0znlxkOPBYko2/xRjTkLR9izGmX6bzTJw40TQ2ltHaL+luoGBV2mL3FpTyQt12LfQ+5IyILDXGTExP99Pivwk43RjzRgByvC8ig4wxG0RkELAxgHOWHjowpwRBqcwoLyY6mz0U/Azuvh+Q0gdYBJxnfz8PeCSg85YWOjCnKMGg7suh4KfF3ygiDwILgZ2JRGPMAq+DROQPwGRggIisA34E3AjMs72D/gMU/pVdiG5jhc3MXPhqMzcvXsX6ljYGN0S5/OSRTD10SLHFUqoBbUSFgh/FvzuwHTgpKc0AnorfGPMll00n+BMtBArVbXQZmHt5/29x6Y1/KysFuvDVZq5asJy2WCcAzS1tXLVgOUDJy66UGLk0uiqsEVUq+InV89+FEKQgFMr27uB58/L+3+Lcl/elzb5+uSjQmxev6lL6Cdpindy8eFVJy62UGLk2utS7KRQyKn4R6Ql8FRgN9EykG2O+EqJc4eDVbQzaBJQ2MHfpjX/rUvoJykGBrm9pyypdURxxaXS9t+AHHP373u494LEzeHnNFvZ55Wb2NJvYKANYe8jlHK4Du3nhx9TzW+BN4GTgOuBsIKjB3sLi0m3cWbs78QXfJJoYwgjBBBS0Ai2U3X1wQ5RmBxkHN0Qd9lbyoaLHUlwaXXuaTRjce8ALX23mqpf3pS12e1da9OUIN+zTXNL3ptTL0o9XzyeMMf8P2GaMuR/4LHBkuGKFhMOs145IT7a1x3cp/QQBew64KcpcFGjC7t7c0pZSaRa+2pynlN25/OSRRGsjKWnR2giXnzwy8GtVBS4RNwtZpkXBxSa/3uzR9T3RA07Gy9RYqpRDWfpR/DH7s0VExgB9sSZzlR8OIRh+IhfRgMsSAwF6DgSpQAtZGaYeOoQbph/CkIYoAgxpiHLD9ENKqvVSNnhE3Ay1TEshvLNDo2u7qeOmjtQedXoPuBxNjeXwsvJj6pkrIv2Aq7H88PsA/y9UqcIkzfZ+/5V/5mt1Axgqm7rv6+Y5kMN4QEJRBtH9K3RlmHroEFX0QeDhXLC+ZbbjIXmXaalMgEpzeHiPAfw09gUWxSel7JbeAy5HU2M5vKz8KP4ldujk54D9AERkRKhSFZDBDVFu+mgGN9beRS9p70pvo56ok+dAHhUpKAVajpVBwdO5ILQyLaVZ5EmNrhdfbebJBcshvqtl7NQDvvzkkSnuxG77lRLlUD/9mHoeckibH7QgxeLyk0fyZOQ4rox9jXXxAcSN0GwGsOKwHztXjEwzCQvQrVa7e5niEXEztDIt0QlQfk2I5WhqLIf66driF5GDsFw4+9qB2hLsTpJbZ7mzywRTxzEtkzKbYDK5hBagWx2k2UgpIB4+6VPHhlSmJTwBym8PuCRNjR7m3nKon67ROUXkDKx4+VOwbPsJtgJ/NMb8X+jS2ZRUdM6fj3GpSPtYn27bLlsRrlzljltFqrTIjIXOj0aKDZ4yuqdu0Tn9hGU+2hjzj9Ak80FJKX6vQl8wEyuaRTpirailOON2T8d9GV77fVlUsJLG62VTaS/WQuDV+Mu3gRdwebgpfj82/mkisruI1IrIEhH5QETOyVmScsdrVS5dNSk33MZNlt6nkRmDwG05T13QPTfCGjdxKo8FM+Gx7+R3Xgf8KP6TjDEfYS2cvgb4BHB54JKUE24VSZdFzA23CmM6ndM1MmMwaMjj3AirgedUHhhovCfwl7EfxV9rf34W+JMxpjVQCSqJUlyjtxxwqzAScU7XHlQwlKjHT8kTVgPP9b6bwF/Gfvz4HxWRN4E24OsiMhDYEagUlYSumpQ9bt4ubjZ+7UEFQ7QftH3YPT2sF2uljCeEte61mwcWBP4y9hOW+UoRuQloNcZ0isg24IxApVCqG6+KNOyoylAWpUbTPNjpEKokUhfOi7VUZhAHRRgNvBOucXcQCfhlnNGrB0BEPgkMJ+lFYYz5TaCSeFBSXj2KUgm4eaZE+8MV7xTueurqnMpj37Fs+snKPw9PtpwXWxeR3wL7A8uAxGibAQqm+BVFCRg300HblsJeT8cTUvncrQXp5fqx8U8ERhk/XQNFUcqDQs/oLeEZxCVHAcYJ/Xj1rAD2DlUKRVEKS6Fdj9XV2aIUQmTjr8U/AHhdRF6CXauVGGOmhCaVoijhEpZnSjLpXjzjvgyr/1q9A/UlNMDtJ2TDcU7pxphnQ5HIAR3cVZQyo4zi2RSMIgxw5zy4W0gFryhKhVBK6wCUCiU0wO1q4xeRrSLykcPfVhH5qJBCVj352AWLdWwulIj9UwmAElJyJUMJxfJybfEbY3YrpCCKC/nYBYt1bC44XW/hN+CJKywXw2q0CZcz6sXTHY/1GAqNH68epZjkE0irWMfmgtP14jE7pEAWkSO9eg3aoygcTl48AO3bqu++J567BTOhR9SaJFfkWF5+vHqUYuLaZV5rPVBeD00+3e1Cd9X9nDeTjdirlwIl41Hhi3KPa5OQ9YkrUuMBtX1Y2vc9mSDKIP2ZbPvQeiFOn1vU/GuLv9Tx6hpnagHnY1MstD3S73m9XhBevZRyCkFcKXHyx86Aut7d00v1vicTVBmU6HOnir/UcesyQ+YHKJ9JM6UwwccJrxeEVy+lnAYbS1RZ5GQqK6f7nkxQZVCi+VfFX+okYvy74fUA5bM+QKHXFkhcL9rffZ9MLx6vXkoJeVRkpBSVRa4t4HK678kEVQYlmn9V/OXA2Bm7FnNPJ9MD5LZamN/r5npsrnSkr0Ak1oefF49XLyWsHkwYA8alqCxybQGXa6iGoMqgRPOvir9cKNEHKFDclp5LzGzM9OLx6qWE0YMJyxZfimWdawu4XFelC6oMSjT/vuLxFxsN2WBT7p4emZjVgOMiFIjV6yg1wpyCX2plXY3x9EutDHIg55ANSglR6cs6ltuknzBt8aVW1iU0+ahglFoZBEhRTD0iskZElovIMhHRprxiUYomDi9K0RYfFvmYLHTiXMlRzBb/8caYTUW5cgV04SqSfEMFF7pcq60VnEsLuBChP8q1PhdR7uoz9ZRQTGzFgVy718Uo12LEtC8XpZYg7Cid5Vqfiyx3UQZ3ReQdYAvWSN6vjTFzHfaZCcwEGDZs2IT//Oc/wVy8GgepqoFClmuhlHEpx7T3ew/CHLBvmgcPXwSms/u2MOtzPuXfdazDswqBy+02uFssd85JxpjDgFOBi0Xk2PQdjDFzjTETjTETBw4cGNyVS3FyjJI/hSrXQoZTCHoGb1C29mzuQVjjIAkZnJQ+hFef8yn/lGNdKJAeKoriN8Y0258bgYeBIwp28VIckNPBr/wpVLkWMpxCkC+zIF9Y2dyDsAbsHed8JBFWfQ464m06BdJDBVf8ItJbRHZLfAdOwlrQPXya5llhYdNxehALpYwrJSBXsSmUR1AQytjvsxXkyyzIF1Y29yCsCUxe9zvIck8vK7fWej4RbxMU0DGgGIO7ewEPi0ji+r83xvwl9Ks62UvBig1z6uzUB7GQAy/5DH6V+8BfkBRioBXyn2uQ6dlKLtNoP4jUQWf7ruNzVQ5B9h6yvQdh+MO7ySCR4MZAnMoKwXHMItov8/ncZAbrhVjA+lvwFr8x5m1jzDj7b7Qx5vqCXNitm1XXu/vNLofuvPYUulOI2EL59iy8nq30Mm37EIwJZuGOIHsPJ1wDNbWpaTW1hXVjdSuHab8KrtzdQog40f5x5rrnJvP0OwsXC8umemL1ZFKwyV26fLpz2ZJrhSzEy0nHHrqTr+nC6zl0W4Wsrnf+L7OgTWEiqb/jMVhwQeGek0LEwMmmvne2Z657JRS3p3r8+L26p25mIKd9gybXSUBhe7GUq390IcjHdOH1HIYdAgKCMYUtuS7V/JRMIZ+TsEMqeJlmnPBTTn5lDtmMWz0tfq8Wj5/R9rAGXnJtBYTtxVKqi4GUO17PYdhlGpQpLJOCq5TnxK2s3NaMCKqcCmDGrR7F76VgPR/kAi1Ckm2FzNR1z9dMU+z5DtnIn0tei2XG8noOCxmrKJ/8+1FwlTAvxq2sTp0dbjkVoNFVPaYecO9muXa/S3g2r1fXPQgzTTEjZWYjfy55LbYZy+059GuOydcMkG/+ncyT6ZR7oLr0e+y0OHpYppgCNLo0Hj8ENzW+VNwr3fyNo/2tgUI/8uVzT/K9D9mEX5g9wvJ+8bNvLucvlTJNliffZzWI8BYpoQfSXBxrozDuy7D6r6Vz37Kh2KEyAgw/UmohG0oLv3Z2r+5xKblXurUM2j70L1+uYw9B3Ae/XlVN85yVvtO+frY5nb9UyjRBEGaAIFqUXebJVqs1nPycjPsyvPb70rpv2VDs8a0CmPwq39Tjt8WWabQ9U/c47CiE2eDXGyGTfLl4TeR7H5rm4TpJJt184FURvUwNfs1YpVSmCYJQ2n7zn2vd+fmY0rtv2VCI8S2ve1uAyYiV3eIvZHySYg+GJuPUYnAjaPnyvQ9LrsM1mmN6i8frnF6tI78tKte8rC3e3IYgPH/85D+fulNKdSEXwvau8nNvQ56MWNmKv5DxSUop+JuTmcbNBQ0TrALL9z64KgfT/eF3O2e0v3dF8TJjJZvzxKt6BGzCyGRGTGxr32aFcUgmWzOAHzNePnWnVOpCrp5LYZtaim1KotJNPYWMT1JqqzGld7+9JqkF6dWS733w8rDye61TZ2e+jpMZK/0euYX8TSbWZsWEXzBzV5ccsuume5kRIXVb24dWeIRof2jbkrsZIJMZL5e64xVrvtB1IR/PpbBNLV49yaZ5BTGHVbbiD8ol0S2qZ6Ru18NcqCBhuZIin8M9CcoGm+99yObFEfQ9d5vIJxEwcVzjtCReEK1rYeE3rHAGiZmtfhROphagWxiHK97JmKWcybbueM5+F2vAt5B1Id/xmTBnBUf7uTslFMituLIVfxCtcK8HOt0VNuwp5PmSkM9tVaSgbLB+7oPb4Fa2yjx9/4SyzKUc3PJv4pat1Sssb4J4rHtaJoWTS+s6bHt5tnXHc/a7sVw7C0mpjjM0zYOdW923F2gQvLJt/AA9kmx10f7Z++J6PdDxWHlOTXcLIesntGwQZBrcSsxiTcSvSUSuzOVc2ZDJNp3NoHk6XgrH67rFspdn686bSaEWWuGGufJXPjO+l1zn3DhIpgD3qnIVf0IhJHepOtrg3RezK7hsHuigwwBkOl+5Rs/MZNrIRpkHOVCWaVAvXRlKxP+5vRSO13XdXjbt28Iv72w8SzIp1Gi/wj6rYQzQBjJHxYdSL8AgeOWaetwUQuPdu377sb9m8olPFFKQYQCa5sETV6S+tJwW7Mj1em1bskv3kjPZJHPASf5ma2bqhmdjnw2yS+/HzJRsjnIyA9bUptr4IbPC8XPd9Oeh7cPu5V3MWcZeYRxqaq149Qn5W9dag+HvvgifuzUceYIe/2ma57ywe7ammUz6pECD4JUbssHNju2E11RoLxt/8jTuoKZZZwoRHe1vDer5uN7CV5u5efEq1re0MbghyuUnj2TqoUOCm7KfKV6L2zT3TNd3LTuxWp/ZnCtsnF5+Kx/epeScVnjLhUz5DCPMQLYvkmSvHolYSrLvPlbvxHEwU5xj4AQtV75kfNYdnsuszmVPWOy7j//Gk0+qL2RDNt0lr9ZhSveeXd37dJtnUC3PTCGi2z6Ex76TMazBwlebuWrBcppb2jBAc0sbVy1YzsJXm62HKd0XPNlDyY8JyU8oa68FuL1WcMrGPuvUpa+ptZRNIcwKyeaQE66xQhWktMy35L5ASTaLA+Vr8kov88e+k71ZIzmMw48+tD4vW+HRkzTZm+SCGtPJxkwa5MLuTmMn0+da9yrx/BQg1EXlmnoOOAka78FXqz+xGEtay237ysfp2fYe6+N7cFfdOYw/Y6bVYnY7R7auo04tFz8visZ7vPMC3Lx4FW2x1G5pW6yTmxevYuppdPdISvz2a0Ly+0JLXuEseS1Z4qn7Ja/olI9LZ7Rfd7NCoSJvei3V5zdqaHo+3BY8SZB4vrJoeKT3BG8btZrDl/8otcyd6k66WcNvy9vLvOEkt9d5n7gi/3AQ2ZpJg17Y3c3rrYAhQirT1OPHDJEgEUnwtd+n7G+wOmAJtps6rjEzmTTtGwDdTSiRF7Lramcjo28EJn4FPncrI678s5uxhHf2usK5InZF7/SxILQf18bkY/zkNeEvnz5ekPA2atsCtb2sQXoTt/afcH6qnThM008mRefXvOgkdy7PQ0oUTJeySJgGbRI9weRGwQv1lzBENvm8qG3WyMa01DTPsuk7xl9KK5fHvpM6DgdWb/SMO6zvCy7ILJdXGbnZ6m22Rwfx47YvsK29g+/3mMfgms0gQo2Jd99ZIt5r/LrJ4paejYnTJ26mnspU/BmVUpJNLbEClw8lZgw8KCdzbedXOLHzWX7U4zf0l48BaK9roH7c593tc+mF7Wr3zBO78n3q8QE0t3RXIkMaorywYzq+xz+czj/uy6l2bDdqamHqL7sPTPq9zulzrAHATD23iV/dpURDqDyAu23WftEC/l+GCep6w9gveituVwRGHAvrXvJ+WSTGGOxnvIMaIiZOsxnATR3Ws3l77S+7LaHrSqLOuCnP9DqVsPXX9oZY2iTI9BdF0zx3xe63UeL1MvL5ct1pIghCnXR0paU3BDOOn3hcK/1cHZGe9DjjF+56KI9GS3Upfo+W1/boIG6KncX9Hx/BeX1e4vu1D9KrbYPvUydulwFq0itLomWSKRRAXrhErkym7z4snLy4W8vuzLr/47reD3nmt9sDno0MUmPPcLWJ1MGh/9W9BeeXRFiCTPmViGVTBu8Wv62QTOs63mcAN7R/gcbdT7RMHW/9wttk4arUbeXfpbx9lE+hqY06PntOCs6TRHmm9Y6TMcAO6omys/vGmlqo38091ITfBpsT0f7QudNxhv326CB6XfFm9i/mbpdP6pE69STyaNhtjw5i5cGXMWbp1URll3mvzdSxYsJPOHzKhbmJXFWK36WAt0cHMeHj22iLdTKl5nlurL2LXkk3ORCc3s75PnAJov1h9DTPipcsR7KC69erjqvNr+jRuSN/OUqR6Xe6t+pczHnbTR1/6jyWGZHnUiqbY2vO04zjppBK4CWQaHEHwM7avtT32t3zWe4wNfQQB7NIAq/WayZTWd99cqpHcQM7eg3KqoHnjEuvMYCGXRzhmJ4LmPDRk5aJSTaz3uzBTR0zWLr7ibxw5adzk9hF8Vfm4K7d5WuPbaerLdMjyvXt09lmdze/1eOPIDvZHvS1W9dCzDrro8vW8/On/sVTO97t3jvIhdh2GHyo9bfoYo8dBVrfBWB3NnJDr3ugR0/ad7QR8GuudFj0LVjzd/j3Eus+JRTe7kNh8lXwzA1d5dKF7GRaZAlG4qnPQWw7PDULDv7crrTdB8NHHpFDuyFw2H9Z8rgeFzI9otZ4SEDE21vpiLUibr1pU0eUdu9nLKl+dMPrHvfsB8ddAY9/L+s8xYGatvV51/XtPfeil5PsT81yz5NPmuP9WdeyhbVMZGF7qp6Wlo/zOrcTldniB2iax8WPXcQvY1lOSlIURSkh9tx5LRccMY2fTD0k62Orq8UPMHYGU3v3Y9/3Xu1K+uXT/6Z1hxUn4xuRRfQVh4ibedBhavgLR1E7ZByvvNvSlT5K1nBqzUvUSVKXO1ILB0+xvq96oqvF0E4POkwNUdodB9wMwuyOLzKux7ucYF5MOaexDfROnYvMtnuB/iOIt7xLTTxpUMukelr6GwPIjlbTm2fiY5lS8w//g4x+6dkAx3wH/n4r7GjptjluhBpxaPwkjrNZ2dxK+8pFjGd1FjIKnHgtK5tbib/xGKPjqxCX8vGktheMPNUaUHfwLokbQQCJ9oVPnACDxu3auOE1WLnQv7knUguDxtO+9pWUZ6vdRHgifgRAt2c5se11M9z5We/CNn31bEiRc2VzK8/+6wM+2hFjQt1ajpfGXSbJ2l6s3XMyj74/gI92xNi9Zy3HHTiQ0W/f7Vie6aQ/v8l0RKJEOtqc65mBj+jN7mznI3rxTHwsr5vh9O1ZyzeO/0SKzBf3WMRu5KZLjIF3zF4MlU2u97SHGcLvXnyXifv2d3cnz5LKVfzAifufyIn7n9j1+8Beu1zZtsb35od52vhN1z/YQh9mxc7l3fgk5G3om7RfM7Cx5vku211Ng7vPc8IN8/m6Sxjq4GK3Lj6A/+04k3c6Us+ZsAd+v8c8htZ0P25zvA9RaU/Lr1URuwa811sD3t+O/IG+sY3EjXjba9OIGWErvWlgG3Gcj02viNtNHVfGvkZzfBJnu+S5w9RQg0Ew2b8YduyAT30fdhvezQ6bsPF/IfJcyn0xADvaaF78a2v+xmdn8sjbq2je+UkG1jzv3wum7z4s7HU29y5fTlvsk/wdmJJUZhtlANv2PYH9W15w991PHm84cB7bH7o4RdbE/fuzOYa3rj2tuwxN8+hcuZgILoHBurxlUge2Z/3kR5zf/ruUZ6s5Pgmwnrtkj7Yt9GFj51E0xyelPeubiFNDROIYk+QMsaONthVPsqLuWJr3+Rz3Ll8OsU52B1Z3wOok8Xp11hBbY6DTsDvAx/DsKzC4Zw0/jsz1HLPaburYQR39cTaVtHXWUoch4mC66qCGT+z8TUpaX0A+hgN7jU+RuTW+Nz+ovSt1nMiDhN5IeFZd3WMeQ6UHKepY4Hx5l0mx73Ul3bx4VWCKv3JNPckkjbhvj+7t4NXzXpfvuHn5bt/KJW5gv52/z0qUIQ1Rz4GaT934N5pb2hwHnxOVfJFdAZ3wOg7oUjob2IObYjN4tv54trV3EOvc9RxEayPcMP0QpjwymposBifbTQ++F5vJovgkptQ8z+y0ypBQtCfULEtRKIn8ZMrz83WXOL7UwLbjOm2wPZxuXryKiR89yVV1f2IvNvE+A/hp+xe6ZE0oKkj11krM35jf/smutHfqv5z5GcngVpvI7xW1Vnm02c/llu3tXTKKg/fIJT+4qtvLPnH/1tz42e4X8XAsaKOe6PT/cWyALHy1mcv/9BqxePfy9/tsCtAjIjwd+ZZjuTWbAXwu8r9s2Z4hWqULZ9Q8z6zeD9EvtrGr/q576ZGUewN4OnG49QiMgREOdXtIgzVLPL1Mr+1xD//V46luz6BTQ+eq2Nd4JOk+vV3/ZccxwLgR9tv5QNdvAd5xKmMPqsurJxmHEfd2InxsojTINnZE96bXqbtmxr036xPszQe+Tt1sBvCpnXOyEue2s8Z7vrWTJ9hMcWjReyn9BF3H1Wxmfdz/cckMaYjy4PYLHCts3Di4stqsiw9gUvucLjl+YCuxZp9yTKl5nivr5jGYzbyXpJwT29IrsTFWi/PRzqO6tdw7Ij15ddx1nPvyvilurbU1Qp+ePbopHLcXy7r4AI6L/YJOu64srZvJHjXdW5GdRhAh5Zlym0g3peZ5bq79NfVJ3fudJsLlsQtZFJ/U9fJNf1YSDYN0GqK19K7v0S0uk5nV19n0Z+DS2De4/ac3OGy1OPS6vzoqZa/7lCh7AXrVRdjW3ulbseXKOUcN67J/O8k8peZ5bq/7pet9cFL8yXlJIMAn9+/PC291d9N0uydb4/W0sptnHfZzPyFzo9GJ6lX8PlwpuyZQjJ3By4t+3c2X1snf2U/rO51obQ03TB/bNeu3b7SW9o5Otscsk0i/XrX86PTRAMxatJKWttxaQgkEGNwQdW1xZjr2vD4v8f3YL7u17P7UeSznRp5yrDBOlbm2Rhxbjm7UCNw6YzxAt7kIXi/D9G0/i5/FM3WTfbcoMymoRD4yKW3YVZY3L17leP/dXh6b432Y0D4XcK7oTjNvnRDg9Jrnua32l4552hzvwxGxO/nZjHEpL5eFrzZnfPayVeR+FVuuCPDzs8YzZO1jDF56E4PYxHrbjLIoPonaiPDa7t9xdOd0MoHmUrfzebn56UHVRoSbzxyXtamn+gZ3E/iIKdOjc0dXPIzDp1zIy8CQV25ib5PaZcyl9Z2gtkb4/IShKZU2vXJt2R7j8vmvcdbh+7Czw79t3Y3BDVGOP2ggD7z4btbe5IMboow/eSbXPNzBpeaP3fJ9Qs0yR3v8erNHym+BrJQ+WD2KqxY00b93PW2xTiIiXa3tRfFJLGp3vu+O2zr8vzzXmwGeeUqUyKL4JIh5Pw9btse4asFyPj9hCA8tbe6mqBP28XSS09c7vDASFT+TcjZY8rn1zHaTHXxW/s7l83ed18u8k0ym+5TOTR0zHBVbol7liwGemX8HN9Te2dVgGyqbuLH2LojBo/FJXNk6zVGGazvOBZzLMptZGNnek2TSn6cN7MHs2K7nqXddhOunde/95YO2+G0Sb+YagS8fOYyn3/wgp5ayE2GcM0G/XrWOLdpobcRV6fgh0X12awH6aaWUwPSlrMh1XMULt3vgNk7gZluGXb2IqYcOcTXDJOPWCk2Qa4s7l/uUq9nSL356FWHKEMazk6C2Rrj5C9m39qEKTT1XL1zO71581/cM3aC6ndVE2JW5GBQqT6/Uz3Rs9X9o+nDYzrmux9UIRARiPjqEXoPhkJ+NvdTKPuxxBD+EeU+itTW88eNTsz6uqhR/QuknSC6QLaY3u8mOvO31ipIPU2qe55bauSnPYbJXVFDX8Gr0VFJjJ+xxhFIgeRDbL1W1EMsf/plq2lkUn8Sk9jnst/MBJrTP5XuxmayLDyBuhHXxAar0lYKzKD6p23MYpNJPXOPK2NfYHO/TbfmFIG3spcBNHTPYblIXF6q0PP7uxXethZQCoCgtfhE5BbgdiAB3GWNu9No/2xb/8Cv/nJ+AilJhlJppJgyqIY/ZunSWjFePiESAO4ATgXXAyyKyyBjzelDXSPYCURTF2xuqUqiGPDp5euVCMUw9RwD/Nsa8bYxpB/4InBHkBb505D5Bnk5RFKUkGNwQzbyTD4rhxz8ESDbCrwOOTN9JRGYCM+2fH4vIqmwuUrf3JybkLKGiKEoJsrZ14zty1UfZLGe3r1NiyU7gMsbMBdz92nwiIo1ONq5KpZryW015herKr+Y1XIph6mkGkm0xQ+00RVEUpQAUQ/G/DBwgIiNEpA74IrCoCHIoiqJUJQU39RhjOkTkm8BiLHfOe4wxK0O8ZN7mojKjmvJbTXmF6sqv5jVEymLmrqIoihIcFTlzV1EURXFHFb+iKEqVUdGKX0ROEZFVIvJvEbmy2PIEgYisEZHlIrJMRBrttP4i8qSIrLY/+9npIiJz7Pw3ichhxZU+MyJyj4hsFJEVSWlZ509EzrP3Xy0i5xUjL5lwyessEWm2y3eZiJyWtO0qO6+rROTkpPSSf85FZB8ReVpEXheRlSLybTu9UsvWLb+lUb7GmIr8wxo4fgvYD6gDXgNGFVuuAPK1BhiQlnYTcKX9/Upgtv39NOAJrLDwRwH/LLb8PvJ3LHAYsCLX/AH9gbftz372937FzpvPvM4Cvuew7yj7Ga4HRtjPdqRcnnNgEHCY/X034F92niq1bN3yWxLlW8kt/tBDQ5QQZwD329/vB6Ympf/GWLwINIjIoCLI5xtjzHNA+szEbPN3MvCkMeZDY8wW4EnglNCFzxKXvLpxBvBHY8xOY8w7wL+xnvGyeM6NMRuMMa/Y37cCb2DN4q/UsnXLrxsFLd9KVvxOoSGCW7useBjgryKy1A5rAbCXMSaxoOh7wF7290q5B9nmr9zz/U3bvHFPwvRBBeVVRIYDhwL/pArKNi2/UALlW8mKv1KZZIw5DDgVuFhEjk3eaKx+Y8X66FZ6/oD/BfYHxgMbgJ8VVZqAEZE+wEPApcaYj5K3VWLZOuS3JMq3khV/RYaGMMY0258bgYexuoLvJ0w49udGe/dKuQfZ5q9s822Med8Y02mMiQN3YpUvVEBeRaQWSwk+YIxZYCdXbNk65bdUyreSFX/FhYYQkd4islviO3ASsAIrXwnvhvOAR+zvi4BzbQ+Jo4DWpG51OZFt/hYDJ4lIP7srfZKdVvKkjcFMwypfsPL6RRGpF5ERwAHAS5TJcy4iAtwNvGGMuTVpU0WWrVt+S6Z8iz36HeYflmfAv7BGxX9YbHkCyM9+WKP6rwErE3kC9gCWAKuBp4D+drpgLXrzFrAcmFjsPPjI4x+wusAxLHvmV3PJH/AVrAGyfwP/Xex8ZZHX39p5abIr+KCk/X9o53UVcGpSesk/58AkLDNOE7DM/jutgsvWLb8lUb4askFRFKXKqGRTj6IoiuKAKn5FUZQqQxW/oihKlaGKX1EUpcpQxa8oilJlqOJXioKIGBH5WdLv74nIrIDOfZ+InBnEuTJc5wsi8oaIPJ2WPlySIm5mec7zRWRwMBIqijOq+JVisROYLiIDii1IMiKSzXKkXwUuMMYcH6AI5wOq+JVQUcWvFIsOrLVGL0vfkN5iF5GP7c/JIvKsiDwiIm+LyI0icraIvCTWGgX7J53mMyLSKCL/EpHP2cdHRORmEXnZDpJ1YdJ5/y4ii4DXHeT5kn3+FSIy2067BmuSzt0icrNbJu0W/AIR+YtY8eNvSpLlPvucy0XkMjvPE4EHxIrVHhWRa2x5V4jIXHtGKCLyjIjMtvP+LxE5Jum8t9j7N4nIt+z0Cfa9Wyoii5PCJFwiVsz4JhH5o9/CU8qcYs9w07/q/AM+BnbHWl+gL/A9YJa97T7gzOR97c/JQAtWrPN6rJgl19rbvg3clnT8X7AaNgdgzYrtCcwErrb3qQcasWKfTwa2ASMc5BwMvAsMBHoAfwOm2tuewWE2NDAcO8Y+Vgv+bTuPPYH/YMVemYAVXjhxTIPTObFnstrffwucnrTfz+zvpwFP2d+/DswHeiSOB2qB/wMG2mlnAffY39cD9cky6F/l/2mLXykaxopW+BvgkiwOe9lYsc53Yk1h/6udvhxL4SaYZ4yJG2NWYyneg7DiupwrIsuwQuTugfViAHjJWHHQ0zkceMYY84ExpgN4AGsBlWxYYoxpNcbswOpR7GvLtJ+I/EJETgE+cjn2eBH5p4gsBz4NjE7algh0tpRdef8M8GtbVowxHwIjgTHAk3ber8YK9gVW6IAHROQcrF6YUgVkY89UlDC4DXgFuDcprQPbDCkiNVgrDyXYmfQ9nvQ7TurznB6LxGDFf/mWMSYlqJeITMZq8YdFssydWK3xLSIyDmthkYuAGVgxaJLl6gn8EqsHsNYe/O7pcN5OvOuyACuNMUc7bPss1ovsdOCHInJI4qWhVC7a4leKit0inYc1UJpgDZYpBGAKlqkiW74gIjW23X8/rMBXi4GvixUuFxE5UKwop168BBwnIgNEJAJ8CXg2B3lSsAe1a4wxD2G1wBNrym7FWqoPdin5TWLFdffjqfQkcGFikFpE+mPlfaCIHG2n1YrIaPuluo8x5mngCixzVJ9886aUPtriV0qBnwHfTPp9J/CIiLyGZavPpTX+LpbS3h24yBizQ0TuwjKJvGIPkn7ArqX+HDHGbBBrgeunsVrOfzbGPOJ1jE+GAPfayhfgKvvzPuBXItIGHI11L1ZgrU71so/z3gUcCDSJSAy40xjzP/bA8RwR6YtV72/Divj4OztNgDnGmJYA8qaUOBqdU1EUpcpQU4+iKEqVoYpfURSlylDFryiKUmWo4lcURakyVPEriqJUGar4FUVRqgxV/IqiKFXG/wf2FUUjSl1fzAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_instance_score(od_preds, y_outlier.astype(int), labels, od.threshold, ylim=(0, 25))"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "ipynb,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}