doc/source/examples/od_llr_genome.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Likelihood Ratio Outlier Detection on Genomic Sequences\n",
    "\n",
    "## Method\n",
    "\n",
    "The outlier detector described by [Ren et al. (2019)](https://arxiv.org/abs/1906.02845) in [Likelihood Ratios for Out-of-Distribution Detection](https://arxiv.org/abs/1906.02845) uses the likelihood ratio between 2 generative models as the outlier score. One model is trained on the original data while the other is trained on a perturbed version of the dataset. This is based on the observation that the likelihood score for an instance under a generative model can be heavily affected by population level background statistics. The second generative model is therefore trained to capture the background statistics still present in the perturbed data while the semantic features have been erased by the perturbations.\n",
    "\n",
    "The perturbations are added using an independent and identical Bernoulli distribution with rate $\\mu$ which substitutes a feature with one of the other possible feature values with equal probability. Each feature in the genome dataset can take 4 values (one of the ACGT nucleobases). This means that a perturbed feature is swapped with one of the other nucleobases. The generative model used in the example is a simple LSTM network.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The bacteria genomics dataset for out-of-distribution detection was released as part of the [Likelihood Ratios for Out-of-Distribution Detection](https://arxiv.org/abs/1906.02845) paper. From the original *TL;DR*: *The dataset contains genomic sequences of 250 base pairs from 10 in-distribution bacteria classes for training, 60 OOD bacteria classes for validation, and another 60 different OOD bacteria classes for test*. There are respectively 1, 7 and again 7 million sequences in the training, validation and test sets. For detailed info on the dataset check the [README](https://storage.cloud.google.com/seldon-datasets/genome/readme.docx?organizationId=156002945562)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook requires the `seaborn` package for visualization which can be installed via `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Dense, Input, LSTM\n",
    "\n",
    "from alibi_detect.od import LLR\n",
    "from alibi_detect.datasets import fetch_genome\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_roc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load genome data\n",
    "\n",
    "*X* represents the genome sequences and *y* whether they are outliers ($1$) or not ($0$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000000, 250) (1000000,) (6999774, 250) (6999774,) (7000000, 250) (7000000,)\n"
     ]
    }
   ],
   "source": [
    "(X_train, y_train), (X_val, y_val), (X_test, y_test) = \\\n",
    "        fetch_genome(return_X_y=True, return_labels=False)\n",
    "print(X_train.shape, y_train.shape, X_val.shape, y_val.shape, X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no outliers in the training set and a majority of outliers (compared to the training data) in the validation and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fraction of outliers in train, val and test sets: 0.00, 0.86 and 0.86\n"
     ]
    }
   ],
   "source": [
    "print('Fraction of outliers in train, val and test sets: '\n",
    "      '{:.2f}, {:.2f} and {:.2f}'.format(y_train.mean(), y_val.mean(), y_test.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define model\n",
    "\n",
    "We need to define a generative model which models the genome sequences. We follow the paper and opt for a simple LSTM. Note that we don't actually need to define the model below if we simply load the pretrained detector later on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "genome_dim = 249  # not 250 b/c we use 1->249 as input and 2->250 as target\n",
    "input_dim = 4  # ACGT nucleobases\n",
    "hidden_dim = 2000\n",
    "\n",
    "inputs = Input(shape=(genome_dim,), dtype=tf.int8)\n",
    "x = tf.one_hot(tf.cast(inputs, tf.int32), input_dim)\n",
    "x = LSTM(hidden_dim, return_sequences=True)(x)\n",
    "logits = Dense(input_dim, activation=None)(x)\n",
    "model = tf.keras.Model(inputs=inputs, outputs=logits, name='LlrLSTM')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to define our loss function which we can utilize to evaluate the log-likelihood for the outlier detector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_fn(y, x):\n",
    "    y = tf.one_hot(tf.cast(y, tf.int32), 4)  # ACGT on-hot encoding\n",
    "    return tf.nn.softmax_cross_entropy_with_logits(y, x, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def likelihood_fn(y, x):\n",
    "    return -loss_fn(y, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load or train the outlier detector\n",
    "\n",
    "We can again either fetch the pretrained detector from a [Google Cloud Bucket](https://console.cloud.google.com/storage/browser/seldon-models/alibi-detect/od/LLR/genome) or train one from scratch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_pretrained = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/seldon-models/alibi-detect/od/LLR/genome/model/model_s.h5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:alibi_detect.od.llr:No threshold level set. Need to infer threshold using `infer_threshold`.\n"
     ]
    }
   ],
   "source": [
    "filepath = os.path.join(os.getcwd(), 'my_path')  # change to download directory\n",
    "detector_type = 'outlier'\n",
    "dataset = 'genome'\n",
    "detector_name = 'LLR'\n",
    "filepath = os.path.join(filepath, detector_name)\n",
    "if load_pretrained:  # load pretrained outlier detector\n",
    "    od = fetch_detector(filepath, detector_type, dataset, detector_name)\n",
    "else:\n",
    "    # initialize detector\n",
    "    od = LLR(threshold=None, model=model, log_prob=likelihood_fn, sequential=True)\n",
    "    \n",
    "    # train\n",
    "    od.fit(\n",
    "        X_train,\n",
    "        mutate_fn_kwargs=dict(rate=.2, feature_range=(0,3)),\n",
    "        mutate_batch_size=1000,\n",
    "        loss_fn=loss_fn,\n",
    "        optimizer=tf.keras.optimizers.Adam(learning_rate=5e-4),\n",
    "        epochs=20,\n",
    "        batch_size=100,\n",
    "        verbose=False\n",
    "    )\n",
    "    \n",
    "    # save the trained outlier detector\n",
    "    save_detector(od, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the log likelihoods\n",
    "\n",
    "Let's compare the log likelihoods of the inliers vs. the outlier test set data under the semantic and background models. We randomly sample $100,000$ instances from both distributions since the full test set contains $7,000,000$ genomic sequences. The histograms show that the generative model does not distinguish well between inliers and outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100000, 250) (100000, 250)\n"
     ]
    }
   ],
   "source": [
    "idx_in, idx_ood = np.where(y_test == 0)[0], np.where(y_test == 1)[0]\n",
    "n_in, n_ood = idx_in.shape[0], idx_ood.shape[0]\n",
    "n_sample = 100000  # sample 100k inliers and outliers each\n",
    "sample_in = np.random.choice(n_in, size=n_sample, replace=False)\n",
    "sample_ood = np.random.choice(n_ood, size=n_sample, replace=False)\n",
    "X_test_in, X_test_ood = X_test[idx_in[sample_in]], X_test[idx_ood[sample_ood]]\n",
    "y_test_in, y_test_ood = y_test[idx_in[sample_in]], y_test[idx_ood[sample_ood]]\n",
    "X_test_sample = np.concatenate([X_test_in, X_test_ood])\n",
    "y_test_sample = np.concatenate([y_test_in, y_test_ood])\n",
    "print(X_test_in.shape, X_test_ood.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# semantic model\n",
    "logp_s_in = od.logp_alt(od.dist_s, X_test_in, batch_size=100)\n",
    "logp_s_ood = od.logp_alt(od.dist_s, X_test_ood, batch_size=100)\n",
    "logp_s = np.concatenate([logp_s_in, logp_s_ood])\n",
    "# background model\n",
    "logp_b_in = od.logp_alt(od.dist_b, X_test_in, batch_size=100)\n",
    "logp_b_ood = od.logp_alt(od.dist_b, X_test_ood, batch_size=100)\n",
    "logp_b = np.concatenate([logp_b_in, logp_b_ood])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAeRklEQVR4nO3dfZAV9Z3v8fdHHiSKyoMTVhlwSMSosa6II6gRZcMGEY1oKvEhJqJLir0b725SFStikr26PtQ1d2sNet31rqUYzJNyTVSibpRgrKiJyBCJiSCBGAhDeAqgUVET5Hv/6N+w7TjDnDNz5pzD9OdVdep0//rX3d9umG//+tcPRxGBmZkVw361DsDMzKrHSd/MrECc9M3MCsRJ38ysQJz0zcwKxEnfzKxAnPRtnyTpEkmP1zqOapC0VtLfdHPeJyV9rpNpoyW9Lqlf+7pd7V9JkySt6k5MVltO+gUn6TRJP5P0qqTtkp6RdFKt48qT1CQpJPVvK4uI70TE1G4s65uSbqhshO9Zx2RJu1NCfU3SKkmX9+Y6uyMifh8RgyPinQ6mvWv/pv1/ZG76UxHxoWrFapXTv+sq1ldJOhh4GPh7YAEwEJgEvF3LuPqIP0REoyQBM4D7JS2JiBX5SpL6R8Su2oRoReSWfrEdBRAR34uIdyLizYh4PCJeaKsg6W8lrZS0Q9Jjko7ITQtJn5e0OrVor5f0wXTm8CdJCyQNTHWHSnpY0ta0rIclNeaW9WSa/5m0rMclHZom/zR9v5Jaz6dIukzS07n5PyxpUTpb2SzpK+XuDEmnSlqaznqWSjo1N22MpJ+m2H4s6d8kfburZUbmQWAHcGzurGWWpN8DT6TlnyvpRUmvpH1xTLtFnSRpRdp3d0saVMp+TT4o6bn0b/KQpGFp3vecQeW2d8/+ldS2/3+Z9v+F6WymNVf/cEnfT3H8TtI/5qZNkNSS1r9Z0s1d7TfrPU76xfYb4B1J8yWdJWlofqKkGcBXgE8ADcBTwPfaLeNM4ETgZODLwB3AZ4BRwHHAxanefsDdwBHAaOBN4LZ2y/o0cDnwfrKzjitT+enpe0jqjvh5uzgPAn4M/Ag4HDgSWFzyXsiWMQx4BLgVGA7cDDwiaXiq8l3guTTtWuCzJS53P0nnA0OAX+UmnQEcA5wp6Siy/fpFsv38KPDDtgNmcgnZvv4g2cH6a6m8lP16KfC3wGHArrSNJYuItv1/fNr/97XfRuCHwC+BkcAU4IuSzkxVbgFuiYiDU/wLylm/VVhE+FPgD1ni+SbQSpYQFgIj0rT/BGbl6u4H7ASOSOMBfCQ3fRlwVW78X4G5nax3HLAjN/4k8LXc+OeBH6XhprSu/rnplwFPp+GLgedL3N5vAjd0UP5Z4Ll2ZT9P6xmd9s0BuWnfBr7dyTomA7uBV4DtwHLgonbb8oFc/X8CFrTbzxuAyWl8LfDfc9OnA78tY7/elBs/Fvgz0K/9fk11P9d+/+b+rY9st42taXgi8Pt2cVwN3J2Gfwr8M3Borf+/+xNu6RddRKyMiMsiopGsZX44MDdNPgK4JXU5tCUwkbXm2mzODb/ZwfhgAEkHSPoPSesk/YksEQxRunMk2ZQb3tk2bwlGAb8tsW5nDgfWtStbR7athwPbI2Jnbtr6Lpb3h4gYEhHDImJcRNzbbnp+/netOyJ2p+kjO6m/Ls1T6n5tP+8A4FAq5wjg8Lb/J+n/yleAEWn6LLKzk5dSt9k5FVy3lclJ3/aIiJfIWsLHpaL1wN+l5NX2eV9E/Kwbi/8S8CFgYmSn+W1dBioltC6mrwc+0I2Y8v5AlrzyRpO1uDcCwyQdkJs2qofry2/Tu9YtSWn5GzpZ3+g0D5S2X9vP+xfgjz0Jvp31wO/a/T85KCKmA0TE6oi4mKzb7utkF7UPrOD6rQxO+gUm6WhJX2q78CdpFFlXybOpyv8Frpb04TT9EEmf6ubqDiJr+b+S+s+vKWPerWTdJZ0l9oeBwyR9UdL+kg6SNHEvy+snaVDuM5CsH/0oSZ+W1F/ShWRdIQ9HxDqgBbhW0kBJpwAfLyP+riwAzpY0RdIAskT+NpA/uF4hqTHtu68Cbf3qpezXz0g6Nh20rgPujw5u0+zCZjrf/88Br0m6StL7JPWTdJzSrb+SPiOpIZ3BvJLm2V3m+q1CnPSL7TWy/tglkt4gS/a/Jks6RMQDZC2ze1PXwa+Bs7q5rrnA+8hamM+SXXQtSepWuRF4JnUfnNxu+mvAx8gS8SZgNfDXe1nkHLJE2fZ5IiK2AeeQbfs2sovS50REW4v4EuCUNO0GsqRbkVtbI2IV2cXv/0O2fz4OfDwi/pyr9l3gceBlsq6stmcNStmv3yI7g9sEDAL+sYM6XbkWmJ/2/wXt4n+HbN+NA36XYrkTOCRVmQa8KOl1sou6F0XEm92IwSpAEf4RFbNySboPeCkiyjljMas5t/TNSiDpJGXPIOwnaRrZA1cP1jous3L5iVyz0vwV8AOy+/Rbgb+PiOdrG5JZ+dy9Y2ZWIO7eMTMrkLru3jn00EOjqamp1mGYme1Tli1b9seIaOhoWl0n/aamJlpaWmodhpnZPkVS+6fL93D3jplZgTjpm5kViJO+mVmB1HWfvplZpfzlL3+htbWVt956q9ahVMygQYNobGxkwIABJc/jpG9mhdDa2spBBx1EU1MT2YtM920RwbZt22htbWXMmDElz+fuHTMrhLfeeovhw4f3iYQPIInhw4eXfebipG9mhdFXEn6b7myPk76ZWYG4T9/MCqlpziMVXd7am87uss6pp57Kz37WnR+eqxwn/b7g2kNyw6/WLg4z26taJ3xw907fc+0h//Uxs7oyePBgAJ588kkmT57MJz/5SY4++mguueQSqvXGYyd9M7MaeP7555k7dy4rVqzg5Zdf5plnnqnKep30zcxqYMKECTQ2NrLffvsxbtw41q5dW5X1OumbmdXA/vvvv2e4X79+7Nq1qyrr9YXcvswXeM2snS6TvqQPAfflij4A/E/gnlTeBKwFLoiIHcqeFrgFmA7sBC6LiF+kZc0EvpaWc0NEzK/MZpiZlaeUWyz7oi6TfkSsAsYBSOoHbAAeAOYAiyPiJklz0vhVwFnA2PSZCNwOTJQ0DLgGaAYCWCZpYUTsqPhWmZnVoddffx2AyZMnM3ny5D3lt912W9ViKLdPfwrw24hYB8wA2lrq84Hz0vAM4J7IPAsMkXQYcCawKCK2p0S/CJjW4y0wM7OSlZv0LwK+l4ZHRMTGNLwJGJGGRwLrc/O0prLOyt9F0mxJLZJatm7dWmZ4Zma2NyUnfUkDgXOB/9d+WmRPFVTkyYKIuCMimiOiuaGhw9/1NTOzbiqnpX8W8IuI2JzGN6duG9L3llS+ARiVm68xlXVWbmZmVVJO0r+Y/+raAVgIzEzDM4GHcuWXKnMy8GrqBnoMmCppqKShwNRUZmZmVVLSffqSDgQ+BvxdrvgmYIGkWcA64IJU/ijZ7ZpryG7ZvBwgIrZLuh5YmupdFxHbe7wFZmZWspKSfkS8AQxvV7aN7G6e9nUDuKKT5cwD5pUfpplZhVX6pYS9+ADktddey+DBg7nyyit7vCy/hsHMrECc9M3MquTmm2/muOOO47jjjmPu3LmdlgHceOONHHXUUZx22mmsWrWqYjH43TtmZlWwbNky7r77bpYsWUJEMHHiRCZNmvSesjPOOIPdu3dz7733snz5cnbt2sX48eM58cQTKxKHk76ZWRU8/fTTnH/++Rx44IEAfOITn+iw7KmnnmL37t2cf/75HHDAAQCce+65FYvD3TtmZgXipG9mVgWTJk3iwQcfZOfOnbzxxhs88MADnHbaae8pmzRpEqeffjoPPvggb775Jq+99ho//OEPKxaHu3f2ZeXccuZ365u9W5X/DsaPH89ll13GhAkTAPjc5z7HiSee+J6yE044AYALL7yQ448/nve///2cdNJJFYtD1fox3u5obm6OlpaWWodRv7p7n7GTvhXQypUrOeaYY2odRsV1tF2SlkVEc0f13b1TQE1zHql1CGZWI076ZmYF4qRvZoVRz93Z3dGd7XHSN7NCGDRoENu2besziT8i2LZtG4MGDSprPt+9Y2aF0NjYSGtrK33pF/kGDRpEY2NjWfM46ZtZIQwYMIAxY8bUOoyac/eOmVmBOOmbmRWIu3cKKn+v/tqbzq5hJGZWTU76BbR20Kf3DDe99d0aRmJm1ebuHTOzAikp6UsaIul+SS9JWinpFEnDJC2StDp9D011JelWSWskvSBpfG45M1P91ZJm9tZGmZlZx0pt6d8C/CgijgaOB1YCc4DFETEWWJzGAc4CxqbPbOB2AEnDgGuAicAE4Jq2A4WZmVVHl0lf0iHA6cBdABHx54h4BZgBzE/V5gPnpeEZwD2ReRYYIukw4ExgUURsj4gdwCJgWkW3xszM9qqUlv4YYCtwt6TnJd0p6UBgRERsTHU2ASPS8EhgfW7+1lTWWfm7SJotqUVSS196cs7MrB6UkvT7A+OB2yPiBOAN/qsrB4DIXmZRkRdaRMQdEdEcEc0NDQ2VWKSZmSWlJP1WoDUilqTx+8kOAptTtw3pe0uavgEYlZu/MZV1Vm5mZlXSZdKPiE3AekkfSkVTgBXAQqDtDpyZwENpeCFwabqL52Tg1dQN9BgwVdLQdAF3aiqzGlo76NPZL3B191e4zGyfUurDWf8AfEfSQOBl4HKyA8YCSbOAdcAFqe6jwHRgDbAz1SUitku6Hlia6l0XEdsrshVmZlaSkpJ+RCwHOvq9xSkd1A3gik6WMw+YV06AZmZWOX4i18ysQJz0zcwKxEnfzKxAnPTNzArESd/2yL9j38z6Jr9P3/bI7tlPI9e+WstQzKyXuKVvZlYgTvpmZgXipG9mViBO+mZmBeILudax/AvYfFHXrM9wS9/MrECc9M3MCsRJ38ysQNynv6/xj52YWQ+4pW9mViBO+mZmBeKkb2ZWIE76ZmYFUlLSl7RW0q8kLZfUksqGSVokaXX6HprKJelWSWskvSBpfG45M1P91ZJm9s4mmZlZZ8q5e+evI+KPufE5wOKIuEnSnDR+FXAWMDZ9JgK3AxMlDQOuIfuB9QCWSVoYETsqsB3Wm/x0rlmf0ZPunRnA/DQ8HzgvV35PZJ4Fhkg6DDgTWBQR21OiXwRM68H6zcysTKUm/QAel7RM0uxUNiIiNqbhTcCINDwSWJ+btzWVdVZuZmZVUmr3zmkRsUHS+4FFkl7KT4yIkBSVCCgdVGYDjB49uhKLNDOzpKSWfkRsSN9bgAeACcDm1G1D+t6Sqm8ARuVmb0xlnZW3X9cdEdEcEc0NDQ3lbY2Zme1Vl0lf0oGSDmobBqYCvwYWAm134MwEHkrDC4FL0108JwOvpm6gx4CpkoamO32mpjIzM6uSUrp3RgAPSGqr/92I+JGkpcACSbOAdcAFqf6jwHRgDbATuBwgIrZLuh5YmupdFxHbK7YlZmbWpS6TfkS8DBzfQfk2YEoH5QFc0cmy5gHzyg/TzMwqwU/kmpkViJO+mVmB+H36Vh4/nWu2T3NL38ysQJz0zcwKxEnfzKxAnPTNzArESd/MrECc9M3MCsRJ38ysQJz0zcwKxA9nWff5QS2zfY5b+mZmBeKkb2ZWIE76ZmYF4qRvZlYgTvpmZgXipG9mViBO+mZmBeKkb2ZWICUnfUn9JD0v6eE0PkbSEklrJN0naWAq3z+Nr0nTm3LLuDqVr5J0ZqU3xszM9q6clv4XgJW58a8D34iII4EdwKxUPgvYkcq/keoh6VjgIuDDwDTg3yX161n4ZmZWjpKSvqRG4GzgzjQu4KPA/anKfOC8NDwjjZOmT0n1ZwD3RsTbEfE7YA0woRIbYWZmpSm1pT8X+DKwO40PB16JiF1pvBUYmYZHAusB0vRXU/095R3Ms4ek2ZJaJLVs3bq1jE0xM7OudJn0JZ0DbImIZVWIh4i4IyKaI6K5oaGhGqs0MyuMUt6y+RHgXEnTgUHAwcAtwBBJ/VNrvhHYkOpvAEYBrZL6A4cA23LlbfLzmJlZFXTZ0o+IqyOiMSKayC7EPhERlwA/AT6Zqs0EHkrDC9M4afoTERGp/KJ0d88YYCzwXMW2xMzMutST9+lfBdwr6QbgeeCuVH4X8C1Ja4DtZAcKIuJFSQuAFcAu4IqIeKcH6zczszKVlfQj4kngyTT8Mh3cfRMRbwGf6mT+G4Ebyw3SzMwqw0/kmpkViH8u0SrDP51otk9wS9/MrECc9M3MCsRJ38ysQJz0zcwKxBdy9wX5i6RmZj3glr6ZWYE46ZuZFYiTvplZgTjpm5kViJO+mVmBOOmbmRWIb9m0yvN7eMzqllv6ZmYF4qRvZlYgTvpmZgXipG9mViBO+mZmBdJl0pc0SNJzkn4p6UVJ/5zKx0haImmNpPskDUzl+6fxNWl6U25ZV6fyVZLO7K2NMjOzjpXS0n8b+GhEHA+MA6ZJOhn4OvCNiDgS2AHMSvVnATtS+TdSPSQdC1wEfBiYBvy7pH6V3BgzM9u7LpN+ZF5PowPSJ4CPAven8vnAeWl4RhonTZ8iSan83oh4OyJ+B6wBJlRkK8zMrCQl9elL6idpObAFWAT8FnglInalKq3AyDQ8ElgPkKa/CgzPl3cwT35dsyW1SGrZunVr+VtkZmadKinpR8Q7ETEOaCRrnR/dWwFFxB0R0RwRzQ0NDb21GjOzQirr7p2IeAX4CXAKMERS22scGoENaXgDMAogTT8E2JYv72AeMzOrglLu3mmQNCQNvw/4GLCSLPl/MlWbCTyUhhemcdL0JyIiUvlF6e6eMcBY4LlKbYiZmXWtlBeuHQbMT3fa7AcsiIiHJa0A7pV0A/A8cFeqfxfwLUlrgO1kd+wQES9KWgCsAHYBV0TEO5XdHDMz25suk35EvACc0EH5y3Rw901EvAV8qpNl3QjcWH6YZmZWCX4i18ysQJz0zcwKxD+iYr3LP6hiVlfc0jczKxAnfTOzAnHSNzMrECd9M7MCcdI3MysQJ30zswJx0jczKxDfp29V0zTnkT3Da286u4aRmBWXW/pmZgXipG9mViBO+mZmBeKkb2ZWIE76ZmYF4qRvZlYgTvpmZgXipG9mViBdJn1JoyT9RNIKSS9K+kIqHyZpkaTV6XtoKpekWyWtkfSCpPG5Zc1M9VdLmtl7m2X1rmnOI3s+ZlY9pbT0dwFfiohjgZOBKyQdC8wBFkfEWGBxGgc4CxibPrOB2yE7SADXABPJflD9mrYDhZmZVUeXr2GIiI3AxjT8mqSVwEhgBjA5VZsPPAlclcrviYgAnpU0RNJhqe6iiNgOIGkRMA34XgW3x+rY2kGf3jPc9NZ3axiJWXGV9e4dSU3ACcASYEQ6IABsAkak4ZHA+txsramss/L265hNdobA6NGjywmvb8n/tqyZWYWUfCFX0mDg+8AXI+JP+WmpVR+VCCgi7oiI5ohobmhoqMQizcwsKSnpSxpAlvC/ExE/SMWbU7cN6XtLKt8AjMrN3pjKOis3M7MqKeXuHQF3ASsj4ubcpIVA2x04M4GHcuWXprt4TgZeTd1AjwFTJQ1NF3CnpjIzM6uSUvr0PwJ8FviVpOWp7CvATcACSbOAdcAFadqjwHRgDbATuBwgIrZLuh5Ymupd13ZR18zMqqOUu3eeBtTJ5Ckd1A/gik6WNQ+YV06A1vf5x1XMqsdP5JqZFYiTvplZgTjpm5kViJO+mVmBOOlbXfFL2Mx6l5O+mVmBOOmbmRWIk76ZWYE46ZuZFYiTvplZgTjpm5kViJO+mVmBlPXLWWaV4p9ONKsNt/TNzArESd/MrEDcvWN1ye/YN+sdbumbmRWIk76ZWYE46ZuZFUiXSV/SPElbJP06VzZM0iJJq9P30FQuSbdKWiPpBUnjc/PMTPVXS5rZO5tjfVHb65b9ymWzniulpf9NYFq7sjnA4ogYCyxO4wBnAWPTZzZwO2QHCeAaYCIwAbim7UBhZmbV02XSj4ifAtvbFc8A5qfh+cB5ufJ7IvMsMETSYcCZwKKI2B4RO4BFvPdAYmZmvay7ffojImJjGt4EjEjDI4H1uXqtqayzcjMzq6IeX8iNiACiArEAIGm2pBZJLVu3bq3UYs3MjO4n/c2p24b0vSWVbwBG5eo1prLOyt8jIu6IiOaIaG5oaOhmeGZm1pHuJv2FQNsdODOBh3Lll6a7eE4GXk3dQI8BUyUNTRdwp6YyMzOroi5fwyDpe8Bk4FBJrWR34dwELJA0C1gHXJCqPwpMB9YAO4HLASJiu6TrgaWp3nUR0f7isBVUOW/c9OsZzHqmy6QfERd3MmlKB3UDuKKT5cwD5pUVnZmZVZSfyDUzKxAnfTOzAvGrlevJtYfUOgIz6+Oc9G2f5Yu6ZuVz946ZWYE46ZuZFYi7d6xPcFePWWnc0jczKxC39K2ulPN0rpmVzy19M7MCcUvf+hz375t1zknf+jQfAMzezd07ZmYF4pa+FUa+1d/GrX8rGid9q1ttd/L05l08HR0I9sYHCdvXOemblaGzg0TbwcDXEKzeOenXmt+s2Sd0dDAo9yyiMz54WCU56VvdK/oDW+UcPDo7QPgMxNo46Zv1IaUcILp7BpI/WPggsu9S9rO29am5uTlaWlpqHUbvcvdOtxWx1V80PqB0j6RlEdHc0bSqt/QlTQNuAfoBd0bETdWOwcz2DZU4K7F3q2rSl9QP+DfgY0ArsFTSwohYUc04as6t+4rI9/V3xmcDxdSbXViVuEBfy4NSVbt3JJ0CXBsRZ6bxqwEi4n91VL9Pde840e9z8geMUg4wnc1rVqpKHQzqqXtnJLA+N94KTMxXkDQbmJ1GX5e0qgpxHQr8sQrrKVe9xgWFiO2cPUPqwbztFGC/9Yp6ja2icenrlVoSR3Q2oe7u3omIO4A7qrlOSS2dHRVrqV7jAsfWXY6te+o1tnqNa2+q/cK1DcCo3HhjKjMzsyqodtJfCoyVNEbSQOAiYGGVYzAzK6yqdu9ExC5J/wN4jOyWzXkR8WI1Y+hEVbuTylCvcYFj6y7H1j31Glu9xtWpun44y8zMKss/omJmViBO+mZmBVLIpC/pU5JelLRbUqe3W0kaIul+SS9JWpkeLqt5XKluP0nPS3q4N2MqJzZJoyT9RNKKVPcL9RJbqjdN0ipJayTNqVJswyQtkrQ6fQ/tpN7/TtuwUtKtksp/NKD3Yhst6fEU2wpJTfUSW6p7sKRWSbfVQ1ySxkn6efr3fEHShb0dVzkKmfSBXwOfAH7aRb1bgB9FxNHA8cDKOokL4Av0fjx5pcS2C/hSRBwLnAxcIenYeogt9wqQs4BjgYurFNscYHFEjAUWp/H2sZ0KfAT4b8BxwEnAGfUQW3IP8C8RcQwwAdhSR7EBXE9pfzOVUEpcO4FLI+LDwDRgrqQhVYqvS4VM+hGxMiL2+qSvpEOA04G70jx/johXah1Xiq0ROBu4szfjySsltojYGBG/SMOvkR2URtZDbGTJak1EvBwRfwbuBWb0dmxpHfPT8HzgvA7qBDAIGAjsDwwANtdDbOnA2D8iFgFExOsRsbMeYkvxnQiMAB6vQkwlxRURv4mI1Wn4D2QHyYYqxdelQib9Eo0BtgJ3p26UOyUdWOugkrnAl4HdtQ6kM6kL4ARgSW0j2aOjV4D0+gEJGBERG9PwJrIE9S4R8XPgJ8DG9HksIqpxFtdlbMBRwCuSfpD+Dv4lnTXVPDZJ+wH/ClxZhXhKjitP0gSyg/lvezuwUtXdaxgqRdKPgb/qYNJXI+KhEhbRHxgP/ENELJF0C9mp3D/VMi5J5wBbImKZpMk9iaXSseWWMxj4PvDFiPhTPcXWG/YWW34kIkLSe+6RlnQkcAzZE+oAiyRNioinah0b2d/BJLID+O+B+4DLSGfANY7t88CjEdFayUsgFYirbTmHAd8CZkZE3TTQ+mzSj4i/6eEiWoHWiGhrqd7P3vsVS1KBuD4CnCtpOlmXwMGSvh0Rn6mD2JA0gCzhfyciftDT5bWpQGy99gqQvcUmabOkwyJiY0oCHfWHnw88GxGvp3n+EzgF6HHSr0BsrcDyiHg5zfMg2fWaHif9CsR2CjBJ0ueBwcBASa9HRI/+TisQF5IOBh4ha5Q825N4Ks3dO52IiE3AekkfSkVTgJq/9z8iro6IxohoInuNxROVSPiVkO44uQtYGRE31zqedmr1CpCFwMw0PBPo6Kzk98AZkvqng+YZVOcifSmxLQWGSGrrk/4o1fk76DK2iLgkIkanv4UrgXt6mvArEVf6//VAiuf+Xo6nfBFRuA9Zy6oVeJvsgtljqfxwstPFtnrjgBbgBeBBYGg9xJWrPxl4uF72GXAa2UXJF4Dl6TO9HmJL49OB35D1r361SvttONldHquBHwPDUnkz2S/HQfZKkv8gS/QrgJvrJbY0/rH0b/or4JvAwHqJLVf/MuC2eogL+Azwl9zfwHJgXDX+TUv5+DUMZmYF4u4dM7MCcdI3MysQJ30zswJx0jczKxAnfTOzAnHSNzMrECd9M7MC+f/nSOAv7KvLTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAc1ElEQVR4nO3de5QW1Z3u8e8jIKg4goCoNKY5EzQSV1REMAqRaILXiGaMGnNi4+jh5ByTY87SFTXREW9zzGSOIZ5MnCGKwSsyjheMJoZBTWKMRoh4g7ggBkMjCgJeEDFBfueP2s0q2276fZv30t31fNZ6V1ft2lW1d73wq127dtWriMDMzIphh3oXwMzMasdB38ysQBz0zcwKxEHfzKxAHPTNzArEQd/MrEAc9K1TJP1E0tX1Lke5JE2U1FzvclTK9tRHUqOkkNS7neXflnRjW3kl/UxS0za2/a+SLutMuay6HPR7OEnLJb0naYOk9ZIelDS83uXqqlJg+3iV9/ETSX9J38k6SfMkfaKa++yMiPjHiDi3nWXHRcQsAElTJD3eavnXIuKqWpTTyuOgXwxfiIj+wF7A68D/q3N5tmqvlVkA/5S+kwZgNfCTtjIV+PhYlTjoF0hEbALuBka1pEk6QdIzkt6WtELStPw6ksZLekLSm2n5lNbblbSrpEclXa/MIEkPpG0+LenqfEswtabPk7QUWJrSDk9530p/D8/lXy7pc7n5aZJuS9Mt3Q5Nkv4s6Q1J38nl3Sm1rNdLWgwc2pljJ2kHSZdKekXSakm3SNott/ystGytpMtal7k9EbERuAM4IFe3uyXdJultYIqkvpKmS3o1faZL6tuqfN9OdV8u6Su59G1+v8nfp+2uknRhbt2tx7mN4/GYpHMl7Q/8K/DpdOXyZlr+oe4/SSdKWpT+HT0h6VO5ZRdJWinpHUkvSTq6o+NmneegXyCSdgZOB57MJb8LnAUMAE4A/oekk1P+jwE/I7syGAIcBCxqtc1BwHzgNxHxvyJ7r8e/pO3uCTSlT2snA+OAUZJ2Bx4ErgcGAdcBD6Ztl2o8sB9wNPAPKRgBXA78bfoc005ZSjElfT4L/BegP/BDAEmjgB8BXyG7mtoNGFbKRiX1T+s9k0ueTHZyHgDcDnwHOIzs+B8IjAUuzeXfExic9tkEzJC0X1rW7veb81lgJDAJuKiUk1WLiFgCfA34bUT0j4gBbdTxYGAm8N/Jvt9/A+amk9l+wNeBQyNiV7LvaHmp+7fyOegXw32pBfYW8Hngey0LIuKxiHg+IrZExHPAncCRafGZwH9GxJ0R8deIWBsR+aC/N/BL4N8j4lIASb2AvwMuj4iNEbEYmNVGmf5PRKyLiPfIgtHSiLg1IjZHxJ3AH4AvlFHHKyLivYh4FniWLDgCnAZck/a1guzE0hlfAa6LiJcjYgNwCXBG6n45FXggIh6PiL8A/wB09FKrC9N3sozsBDIlt+y3EXFf+k7eS/u+MiJWR8Qa4Argq622d1lEvB8RvyQ7gZ4GHX6/La6IiHcj4nngZuDLZRyXUkwF/i0inoqID9K9gPfJTmQfAH3JTv59ImJ5RPyxwvu3HAf9Yjg5tcD6kbWqfilpTwBJ41LXzBpJb5G12gan9YYD2/oPeAKwE9nlfYshQG9gRS4tP91W2t7AK62Wv0KJreXktdz0RrJA2rLt/L5a76dUrcv4Clk9h7beR+qyWdvB9v45IgZExJ4RcVKrQNf6eLW1771z8+sj4t22lnfw/ba1v9bbroSPARekrp0308luOLB3RCwDvglMA1ZLmi2p0vu3HAf9AkmtrHvIWlfjU/IdwFxgeETsRhbAlZatIOsWac+PgZ8DD0naJaWtATaT3aBs0dZooXxL+FWywJC3D7AyTb8L7Jxbtuc2ytTaqlb736eMdfNal3Efsnq+nvaxtb6SdiLrxuis1lcJbe371dz8wNzxb718W99vi9bH51XK09FVzQqyq60Buc/O6YqOiLgjIsaT1TGA75a5fyuDg36BpJusk4GBwJKUvCuwLiI2SRpL1qXT4nbgc5JOk9Rb2Q3ag1pt9uvAS8ADknaKiA+Ae4BpknZWNhTxrA6K9hCwr6Qz035OJ7vZ/NO0fBFZV0ofSWPIulNKNQe4RNJASQ3AN0pYZ0dJ/XKfXmTdIv9b0ojUD/+PwF0RsZms//0Lym5G70jWam0dWLfHncClkoZIGkzWfdT6BusVknaUNAE4Efj3lL6t77fFZem7+iRwNnBXmeV7HWhIdW/Lj4GvpasOSdol3WDeVdJ+ko5KN6Y3Ae8BW8rcv5XBQb8YHpC0AXgbuAZoiogX07L/CVwp6R2yYDKnZaWI+DNwPHABsI4s+B6Y33C6cTsVaAbul9TShbQbWZfLrWRB6/32ChcRa8kC1QVk3SLfAk6MiDdSlsvIrjjWk/Vn31FG3a8g67L4E/CLVJ6OvEgWfFo+Z5PdiLwV+FXa1ibSCSQdy28As8la/RvIhmG2W+cyXQ0sAJ4Dngd+n9JavEZ2bF4lO1F/LSL+kJa1+/3m/JLs3sJ8sm6nX5RZvkfIjtlrkt5ovTAiFgD/jezG9/q0rylpcV/gWuCNVI89yO6XWJXIP6Ji1Sbpu8CeEdHZkTPdSroSeBMYGRF/qnd5zPLc0reKk/QJSZ9Kl/JjgXOAe+tdrmqS9IXURbIL8M9kLfLl9S2V2Uc56Fs17ErWr/8uWf/w/wXur2uJqm8yWffKq2Rj3s8IX0ZbF+TuHTOzAnFL38ysQLr0y5wGDx4cjY2N9S6GmVm3snDhwjciYkhby7p00G9sbGTBggX1LoaZWbciqd0nz929Y2ZWIA76ZmYF4qBvZlYgXbpP38ysUv7617/S3NzMpk2b6l2UiunXrx8NDQ306dOn5HUc9M2sEJqbm9l1111pbGxEquT78OojIli7di3Nzc2MGDGi5PXcvWNmhbBp0yYGDRrUIwI+gCQGDRpU9pWLg76ZFUZPCfgtOlMfB30zswJxn76ZFVLjxQ9WdHvLrz2hwzyHH344TzzxREX3Wy4H/QrK/yMq5R+AmRVLvQM+OOhXjU8AZtZa//792bBhA4899hjTpk1j8ODBvPDCCxxyyCHcdtttNbnn4D59M7M6eOaZZ5g+fTqLFy/m5Zdf5je/+U1N9uugb2ZWB2PHjqWhoYEddtiBgw46iOXLl9dkvw76ZmZ10Ldv363TvXr1YvPmzTXZr4O+mVmBlHQjV9IA4EbgACCAvwdeIvv900ayH4A+LSLWK7sT8QPgeGAjMCUifp+20wRcmjZ7dUTMqlhNzMzKUNQBFqWO3vkB8POIOFXSjsDOwLeB+RFxraSLgYuBi4DjyH4YeiQwDrgBGCdpd+ByYAzZiWOhpLkRsb6iNTIz66I2bNgAwMSJE5k4ceLW9B/+8Ic1K0OH3TuSdgM+A9wEEBF/iYg3gclAS0t9FnBymp4M3BKZJ4EBkvYCjgHmRcS6FOjnAcdWtDZmZrZNpbT0RwBrgJslHQgsBM4HhkbEqpTnNWBomh4GrMit35zS2kv/EElTgakA++yzT8kVqZdKP9VnZlZNpdzI7Q2MBm6IiIOBd8m6craKiCDrstluETEjIsZExJghQ9r8XV8zM+ukUlr6zUBzRDyV5u8mC/qvS9orIlal7pvVaflKYHhu/YaUthKY2Cr9sc4Xvfvw07lm1lV02NKPiNeAFZL2S0lHA4uBuUBTSmsC7k/Tc4GzlDkMeCt1Az0MTJI0UNJAYFJKMzOzGil19M43gNvTyJ2XgbPJThhzJJ0DvAKclvI+RDZccxnZkM2zASJinaSrgKdTvisjYl1FamFmZiUpKehHxCKyoZatHd1G3gDOa2c7M4GZ5RTQzKwqpu1W4e29Vdnt5Tc9bRr9+/fnwgsv3O5t+YlcM7MCcdA3M6uR6667jgMOOIADDjiA6dOnt5sGcM0117Dvvvsyfvx4XnrppYqVwe/TNzOrgYULF3LzzTfz1FNPERGMGzeOCRMmfCTtyCOPZMuWLcyePZtFixaxefNmRo8ezSGHHFKRcjjom5nVwOOPP84pp5zCLrvsAsAXv/jFNtN+/etfs2XLFk455RR23nlnAE466aSKlcNBv8Y8Zt/M6sl9+mZmNTBhwgTuu+8+Nm7cyLvvvsu9997L+PHjP5I2YcIEPvOZz3Dffffx3nvv8c477/DAAw9UrBxu6ZtZMVVxiGVbRo8ezZQpUxg7diwA5557LocccshH0g4++GAATj/9dA488ED22GMPDj300IqVQ9mw+q5pzJgxsWDBgnoXY5u254Vr7t4xq50lS5aw//7717sYFddWvSQtjIi2nq1y946ZWZG4e6cT/DplM+uu3NI3s8Loyt3ZndGZ+rilX0cevmlWO/369WPt2rUMGjSI7Ke8u7eIYO3atfTr16+s9Rz0zawQGhoaaG5uZs2aNfUuSsX069ePhoaGstZx0DezQujTpw8jRoyodzHqzn36ZmYF4qBvZlYgDvpmZgXioG9mViAO+mZmBeKgb2ZWIA76ZmYF4qBvZlYgDvpmZgXioG9mViAlBX1JyyU9L2mRpAUpbXdJ8yQtTX8HpnRJul7SMknPSRqd205Tyr9UUlN1qmRmZu0pp6X/2Yg4KPdrLBcD8yNiJDA/zQMcB4xMn6nADZCdJIDLgXHAWODylhOFmZnVxva8cG0yMDFNzwIeAy5K6bdE9qLnJyUNkLRXyjsvItYBSJoHHAvcuR1l6DH8mmUzq4VSW/oB/ELSQklTU9rQiFiVpl8DhqbpYcCK3LrNKa299A+RNFXSAkkLetIrUM3MuoJSW/rjI2KlpD2AeZL+kF8YESGpIj9JExEzgBmQ/TB6JbZpZmaZklr6EbEy/V0N3EvWJ/966rYh/V2dsq8EhudWb0hp7aWbmVmNdBj0Je0iadeWaWAS8AIwF2gZgdME3J+m5wJnpVE8hwFvpW6gh4FJkgamG7iTUpqZmdVIKd07Q4F7029K9gbuiIifS3oamCPpHOAV4LSU/yHgeGAZsBE4GyAi1km6Cng65buy5aaumZnVRodBPyJeBg5sI30tcHQb6QGc1862ZgIzyy+mmZlVgp/INTMrEAd9M7MCcdA3MyuQ7Xki16rET+eaWbW4pW9mViBu6Zco3/o2M+uu3NI3MysQB30zswJx0DczKxAHfTOzAnHQNzMrEI/e6eI8Zt/MKsktfTOzAnHQNzMrEAd9M7MCcdA3MysQB30zswJx0DczKxAHfTOzAnHQNzMrEAd9M7MCcdA3MysQB30zswJx0DczK5CSg76kXpKekfTTND9C0lOSlkm6S9KOKb1vml+WljfmtnFJSn9J0jGVroyZmW1bOS3984ElufnvAt+PiI8D64FzUvo5wPqU/v2UD0mjgDOATwLHAj+S1Gv7il8sjRc/uPVjZtYZJQV9SQ3ACcCNaV7AUcDdKcss4OQ0PTnNk5YfnfJPBmZHxPsR8SdgGTC2EpUwM7PSlNrSnw58C9iS5gcBb0bE5jTfDAxL08OAFQBp+Vsp/9b0NtbZStJUSQskLVizZk0ZVTEzs450GPQlnQisjoiFNSgPETEjIsZExJghQ4bUYpdmZoVRyi9nHQGcJOl4oB/wN8APgAGSeqfWfAOwMuVfCQwHmiX1BnYD1ubSW+TXMTOzGuiwpR8Rl0REQ0Q0kt2IfSQivgI8CpyasjUB96fpuWmetPyRiIiUfkYa3TMCGAn8rmI1MTOzDm3Pb+ReBMyWdDXwDHBTSr8JuFXSMmAd2YmCiHhR0hxgMbAZOC8iPtiO/ZuZWZnKCvoR8RjwWJp+mTZG30TEJuBL7ax/DXBNuYU0M7PK8BO5ZmYF4qBvZlYgDvpmZgXioG9mViAO+mZmBbI9QzZ7PL/YzMx6Grf0zcwKxC39bip/FbL82hPqWBIz607c0jczKxAHfTOzAnHQNzMrEAd9M7MCcdA3MysQB30zswJx0DczKxCP0+8BPGbfzErllr6ZWYE46JuZFYiDvplZgTjom5kViIO+mVmBOOibmRWIg76ZWYE46JuZFUiHQV9SP0m/k/SspBclXZHSR0h6StIySXdJ2jGl903zy9Lyxty2LknpL0k6plqVMjOztpXS0n8fOCoiDgQOAo6VdBjwXeD7EfFxYD1wTsp/DrA+pX8/5UPSKOAM4JPAscCPJPWqZGXMzGzbOgz6kdmQZvukTwBHAXen9FnAyWl6cponLT9aklL67Ih4PyL+BCwDxlakFmZmVpKS+vQl9ZK0CFgNzAP+CLwZEZtTlmZgWJoeBqwASMvfAgbl09tYJ7+vqZIWSFqwZs2a8mtkZmbtKinoR8QHEXEQ0EDWOv9EtQoUETMiYkxEjBkyZEi1dmNmVkhlvWUzIt6U9CjwaWCApN6pNd8ArEzZVgLDgWZJvYHdgLW59Bb5daxC/MZNM9uWUkbvDJE0IE3vBHweWAI8CpyasjUB96fpuWmetPyRiIiUfkYa3TMCGAn8rlIVMTOzjpXS0t8LmJVG2uwAzImIn0paDMyWdDXwDHBTyn8TcKukZcA6shE7RMSLkuYAi4HNwHkR8UFlq2NmZtvSYdCPiOeAg9tIf5k2Rt9ExCbgS+1s6xrgmvKLaWZmleAncs3MCsRB38ysQBz0y7S835ks73dmvYthZtYpDvpmZgVS1jh9+7CWFn/jpjs+lJafryeP2Tez1tzS76R8F4+7e8ysu3BLv0Ic+M2sO3BLvwp8s9fMuioHfTOzAnHQL4Nb72bW3Tnom5kViIN+FfnKwMy6Go/eKYGDt5n1FG7pV5lPGGbWlbilXxB+OtfMwC39mvC4fTPrKhz0zcwKxEHfzKxAHPQ74G4ZM+tJHPTNzArEo3dayY9yqYa23sFfax7JY1ZcbunXkLuKzKzeHPTryEM5zazWHPS7CJ8AzKwWOgz6koZLelTSYkkvSjo/pe8uaZ6kpenvwJQuSddLWibpOUmjc9tqSvmXSmqqXrXMzKwtpbT0NwMXRMQo4DDgPEmjgIuB+RExEpif5gGOA0amz1TgBshOEsDlwDhgLHB5y4miiNr7jd2Wabf8zawaOgz6EbEqIn6fpt8BlgDDgMnArJRtFnBymp4M3BKZJ4EBkvYCjgHmRcS6iFgPzAOOrWhteqBSAr9PDmZWqrL69CU1AgcDTwFDI2JVWvQaMDRNDwNW5FZrTmntpbfex1RJCyQtWLNmTTnFq7h6BVMHejOrlpLH6UvqD/wH8M2IeFvS1mUREZKiEgWKiBnADIAxY8ZUZJs9SXvdQp3lMftmxVJSS19SH7KAf3tE3JOSX0/dNqS/q1P6SmB4bvWGlNZeunXA/ftmVimljN4RcBOwJCKuyy2aC7SMwGkC7s+ln5VG8RwGvJW6gR4GJkkamG7gTkppVqJtBX6fGMysFKV07xwBfBV4XtKilPZt4FpgjqRzgFeA09Kyh4DjgWXARuBsgIhYJ+kq4OmU78qIWFeRWpiZWUk6DPoR8TigdhYf3Ub+AM5rZ1szgZnlFNDKl2/x1/MdP2bW9fiJ3B7GXTxmti1+y6Zt5ZE8Zj2fW/pmZgXioF8AHtljZi0c9Hu41sHeJwCzYnOffht6alDsqfUys9K5pW9mViAO+mZmBeLuHWuTh2+a9Uxu6ZuZFYiDvplZgTjom5kViIO+mVmBOOi3UpSx7EWpp5l9mIO+mVmBOOgXmF/JYFY8DvrmwG9WIA76ZmYF4qBvZlYgfg2DdcivZDDrOdzSN8A3dc2KwkHfzKxA3L1jZXFXj1n35pa+fYi7eMx6tg6DvqSZklZLeiGXtrukeZKWpr8DU7okXS9pmaTnJI3OrdOU8i+V1FSd6piZ2baU0tL/CXBsq7SLgfkRMRKYn+YBjgNGps9U4AbIThLA5cA4YCxwecuJosuYtlv2MTPrwToM+hHxK2Bdq+TJwKw0PQs4OZd+S2SeBAZI2gs4BpgXEesiYj0wj4+eSMzMrMo626c/NCJWpenXgKFpehiwIpevOaW1l/4RkqZKWiBpwZo1azpZPDMza8t238iNiACiAmVp2d6MiBgTEWOGDBlSqc2amRmdD/qvp24b0t/VKX0lMDyXryGltZduXZBH8Jj1XJ0N+nOBlhE4TcD9ufSz0iiew4C3UjfQw8AkSQPTDdxJKa1LyI89t9I1Xvzg1o+ZdQ8dPpwl6U5gIjBYUjPZKJxrgTmSzgFeAU5L2R8CjgeWARuBswEiYp2kq4CnU74rI6L1zWHrQlpa+42b7qhzScyskjoM+hHx5XYWHd1G3gDOa2c7M4GZZZXOzMwqyk/kmpkViIO+bZNv6pr1LH7hmlWEX8Rm1j24pW9mViAO+mZmBeKgj/utzaw4HPStQ/4pRbOew0HfzKxAHPTNzArEQzat4jx806zrctC3kuX79f1OHrPuyd07ZmYF4qBvZlYgxe7e8Q+hm1nBFDvoW9X5pq5Z1+LuHesUP6xl1j25pW8141a/Wf25pW+d5tczmHU/hQ36/jFvMysid+9YXbirx6w+Chv03S1ROX5S16z7KGz3jplZERW2pW/VsbzfmWW39t3VY1Y7xQv6fgq36joT+Fv4BGBWXTUP+pKOBX4A9AJujIhra10Gq76Wfv7t6eP3CcCs8moa9CX1Av4F+DzQDDwtaW5ELK5lOax2KnWT1ycAs8qodUt/LLAsIl4GkDQbmAxUN+i7S6dLKHXEVEcnh678jEX+hFTKiaq9upS7HbNSKSJqtzPpVODYiDg3zX8VGBcRX8/lmQpMTbP7AS9VqTiDgTeqtO3uoOj1Bx8D17/n1v9jETGkrQVd7kZuRMwAZlR7P5IWRMSYau+nqyp6/cHHwPUvZv1rPU5/JTA8N9+Q0szMrAZqHfSfBkZKGiFpR+AMYG6Ny2BmVlg17d6JiM2Svg48TDZkc2ZEvFjLMuRUvQupiyt6/cHHwPUvoJreyDUzs/ryu3fMzArEQd/MrEAKE/QlfUnSi5K2SNrmMC1JvSQ9I+mntSpftZVSf0nDJT0qaXHKe36ty1lNpf4bkHSspJckLZN0cS3LWE2Sdpc0T9LS9HdgO/n+KR2nJZKul6Ral7Uayqj/PpJ+keq/WFJjbUtaXYUJ+sALwBeBX5WQ93xgSXWLU3Ol1H8zcEFEjAIOA86TNKoWhauRDo9B7lUhxwGjgC/3oGNwMTA/IkYC89P8h0g6HDgC+BRwAHAocGQtC1lFHdY/uQX4XkTsT/YWgdU1Kl9NFCboR8SSiOjw6V5JDcAJwI3VL1XtlFL/iFgVEb9P0++QnfiG1aJ8tVDiv4GtrwqJiL8ALa8K6QkmA7PS9Czg5DbyBNAP2BHoC/QBXq9J6aqvw/qnE3zviJgHEBEbImJj7YpYfYUJ+mWYDnwL2FLvgtRTuqQ9GHiqviWpuWHAitx8Mz3nxDc0Ilal6deAoa0zRMRvgUeBVenzcET0lKveDusP7Au8Keme1MX7vXT112N0udcwbA9J/wns2cai70TE/SWsfyKwOiIWSppY6fJV2/bWP7ed/sB/AN+MiLcrVb5aqNQx6K62Vf/8TESEpI+M15b0cWB/sqflAeZJmhARv654Yatge+tPFhMnkDV4/gzcBUwBbqpsSeunRwX9iPjcdm7iCOAkSceTXeL+jaTbIuK/bn/pqq8C9UdSH7KAf3tE3LP9paqtChyDbv2qkG3VX9LrkvaKiFWS9qLtvupTgCcjYkNa52fAp4FuEfQrUP9mYFHuTcD3kd3f6jFB3907ORFxSUQ0REQj2SsiHukuAb8S0iiNm4AlEXFdvctTJz35VSFzgaY03QS0deXzZ+BISb1TA+BIes6ghlLq/zQwQFLLGyqPotqvfq+xwgR9SadIaiZrtTwo6eGUvrekh+pbuuorsf5HAF8FjpK0KH2Or1ORK66UYxARm4GWV4UsAebU8VUhlXYt8HlJS4HPpXkkjZHUMnDhbuCPwPPAs8CzEfFAPQpbBR3WPyI+AC4E5kt6HhDw4zqVtyr8GgYzswIpTEvfzMwc9M3MCsVB38ysQBz0zcwKxEHfzKxAHPTNzArEQd/MrED+P5v5fJTE6evzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show histograms\n",
    "plt.hist(logp_s_in, bins=100, label='in');\n",
    "plt.hist(logp_s_ood, bins=100, label='ood');\n",
    "plt.title('Semantic Log Probabilities')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "plt.hist(logp_b_in, bins=100, label='in');\n",
    "plt.hist(logp_b_ood, bins=100, label='ood');\n",
    "plt.title('Background Log Probabilities')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is because of the background-effect which is in this case the GC-content in the genomic sequences. This effect is partially reduced when taking the likelihood ratio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "llr_in = logp_s_in - logp_b_in\n",
    "llr_ood = logp_s_ood - logp_b_ood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAXLElEQVR4nO3df7RdZX3n8ffH8CMKCAiBAgEv7UCFSUeEGKhDNBaLAUcQy0LwB4FBGao4dVXXGKfTISM6xVkjRcdfQy0CbQWpCkbBwYhSFQQJQuVXkYixXORHDIggoKR854/zXHoM9+aeJPeec3Pzfq111t3n2c/e+3tOcu/nPM/e55xUFZKkzdtzBl2AJGnwDANJkmEgSTIMJEkYBpIkDANJEoaBNjFJ5ie5s+v+yiSv2oD9LEnyt215rySPJZnR7l+d5K0TV/WYNZyU5DuTfZyu430qyZ/363jatBgGmpLG+iNfVd+uqt+dyGNV1T9X1bZV9S8Tud+NkWQoSbWQeqw9H4vXY/tnBU1VnVZVZ058tZoOthh0AZLWaYeqWpNkLvAPSW6sqmWDLkrTjyMDbVKSLEgyPMa6/ZL8OMkJ7f7uSb6QZFVr/89jbDfyKrz7xdELk1yT5NEkX0uyc1f/o5LcluTnbUppv7VquLqtuy3JUV3rdkqyNMkvknwP+J1eH3dVLQduAw7o2t/iJD9qNd6e5JiRGoBPAb/fRhU/b+3nJ/lA1/ZvS7IiyUOtrt17rUfTj2GgaSHJgcCVwDur6qIkzwG+DPwjsAdwGPCuJK/ucZdvBE4GdgG2At7TjrMvcBHwLmAWcAXw5SRbJdmyHfNrbbt3An+XZGRa6+PAk8BuwH9st14f3yHAHGBFV/OPgPnA9sD/AP42yW5VdQdwGvDdNv21wyj7+wPgL4DjWj0/AS7utR5NP4aBpoP5wFLgxKr6Smt7KTCrqt5fVb+uqruBvwKO73Gfn6mqH1bVE8Al/Osr8jcAl1fVsqp6CvjfwHOBlwGHANsCZ7VjfgP4CnBCOzn9R8B/r6pfVtWtwAU91PGzJE8A3wU+AVw2sqKq/r6qflpVT1fV54C7gHk9Pr43AedV1fer6lfA++iMJIZ63F7TjOcMNB2cBvxDVV3d1fZCYPeRKZJmBvDtHvd5f9fy43T+yAPsTudVNABV9XSSe+iMPtYA91TV013b/qStm0Xn9+2etdaNZ2eggD+hM1rZEvg1QJITgT8FhlrfbVv/XuwOfL/rcTyWZHWrdWWP+9A04shA08FpwF5J/rKr7R7gx1W1Q9dtu6o6ciOP9VM6QQNAkgB7Ave2dXu2KaoRe7V1q+iExZ5rrRtXVf1LVZ1NZ4rp7e24L6Qz0jkd2KlNBd0KZGSz9Xwc2wA7tVq1GTIMNJVtmWRm122skeyjwELg5UnOam3fAx5N8t4kz00yI8mcJC/dyJouAV6T5LB2juDdwK+Aa4Hr6Ywi/kuSLZMsAF4LXNwuW/0isCTJ85LsDyxaz2Of1fY9E9iGzh/8VQBJTqZzTmHEA8DsJFuNsa+LgJOTHJBka+B/AtdX1cr1rEnThGGgqewK4Imu25KxOlbVz4E/BI5Icmb74/sf6Mz1/xj4GfBpOidbN1hV3Qm8Gfg/bZ+vBV7bzhH8ut0/oq37BJ3zGP/UNj+dzlTO/cD5wGfW8/CXAw8Db6uq24EP0zmX8ADwe8A1XX2/Qefqo/uT/GyUx/F14M+BLwD30bmyqdfzKZqG4pfbSJIcGUiSDANJkmEgScIwkCSxCb/pbOedd66hoaFBlyFJm5Qbb7zxZ1U1a+32TTYMhoaGWL58+aDLkKRNSpJR3/nuNJEkyTCQJBkGkiQ24XMGkjQRnnrqKYaHh3nyyScHXcqEmjlzJrNnz2bLLbfsqb9hIGmzNjw8zHbbbcfQ0BCdD6Hd9FUVq1evZnh4mL333runbZwmkrRZe/LJJ9lpp52mTRAAJGGnnXZar9GOYSBpszedgmDE+j4mw0CS5DkDSeo2tPjyCd3fyrNeM26fl73sZVx77bUTetz1ZRhMB0u6vq9lySODq0PSBhl0EIDTRJI0cNtuuy0AV199NQsWLODYY4/lRS96EW9605vo1xeQGQaSNIXcdNNNnHPOOdx+++3cfffdXHPNNeNvNAEMA0maQubNm8fs2bN5znOewwEHHMDKlSv7clzDYLpZsv1vnkOQtEnZeuutn1meMWMGa9as6ctxDQNJklcTSVK3Xi4FnY4MA0kasMceewyABQsWsGDBgmfaP/axj/WtBqeJJEmGgSTJMJAkYRhIkjAMJEkYBpIkvLRUkn7TRL+Df5I/SXjJkiVsu+22vOc979mo/TgykCQZBpI0FZx99tnMmTOHOXPmcM4554zZBvDBD36Qfffdl0MPPZQ777xzQo7vNJEkDdiNN97IZz7zGa6//nqqioMPPpj58+c/q+0Vr3gFTz/9NBdffDE333wza9as4cADD+Sggw7a6BoMA0kasO985zscc8wxbLPNNgC8/vWvH7Xt29/+Nk8//TTHHHMMz3ve8wA46qijJqQGp4kkSYaBJA3a/Pnzueyyy3j88cf55S9/yaWXXsqhhx76rLb58+fz8pe/nMsuu4wnnniCRx99lC9/+csTUoPTRJLUbZIvBR3NgQceyEknncS8efMAeOtb38pBBx30rLaXvOQlALzhDW/gxS9+MbvssgsvfelLJ6SG9OvLlifa3Llza/ny5YMuY2oY7broAfyHljZFd9xxB/vtt9+gy5gUoz22JDdW1dy1+zpNJEkyDCRJhoEksalOl6/L+j4mw0DSZm3mzJmsXr16WgVCVbF69WpmzpzZ8zZeTSRpszZ79myGh4dZtWrVoEuZUDNnzmT27Nk99x83DJLsCVwI7AoUcG5VfSTJC4DPAUPASuC4qno4SYCPAEcCjwMnVdX3274WAf+t7foDVXVBaz8IOB94LnAF8Cc1nWJa0pS15ZZbsvfeew+6jIHrZZpoDfDuqtofOAR4R5L9gcXAVVW1D3BVuw9wBLBPu50KfBKghccZwMHAPOCMJDu2bT4JvK1ru4Ub/9AkSb0aNwyq6r6RV/ZV9ShwB7AHcDRwQet2AfC6tnw0cGF1XAfskGQ34NXAsqp6qKoeBpYBC9u651fVdW00cGHXviRJfbBeJ5CTDAEvAa4Hdq2q+9qq++lMI0EnKO7p2my4ta2rfXiU9tGOf2qS5UmWT7f5PUkapJ7DIMm2wBeAd1XVL7rXtVf0kz7HX1XnVtXcqpo7a9asyT6cJG02egqDJFvSCYK/q6ovtuYH2hQP7eeDrf1eYM+uzWe3tnW1zx6lXZLUJ+OGQbs66K+BO6rq7K5VS4FFbXkR8KWu9hPTcQjwSJtOuhI4PMmO7cTx4cCVbd0vkhzSjnVi174kSX3Qy/sM/j3wFuCWJDe3tv8KnAVckuQU4CfAcW3dFXQuK11B59LSkwGq6qEkZwI3tH7vr6qH2vLb+ddLS7/abpKkPhk3DKrqO0DGWH3YKP0LeMcY+zoPOG+U9uXAnPFqkSRNDj+OQpJkGEiSDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJArYYdAGaJEu271p+ZHB1SNokODKQJBkGkiTDQJKEYSBJoocwSHJekgeT3NrVtiTJvUlubrcju9a9L8mKJHcmeXVX+8LWtiLJ4q72vZNc39o/l2SriXyAkqTx9TIyOB9YOEr7X1bVAe12BUCS/YHjgX/btvlEkhlJZgAfB44A9gdOaH0BPtT29W+Ah4FTNuYBSZLW37hhUFXfAh7qcX9HAxdX1a+q6sfACmBeu62oqrur6tfAxcDRSQL8AfD5tv0FwOvW8zFIkjbSxrzP4PQkJwLLgXdX1cPAHsB1XX2GWxvAPWu1HwzsBPy8qtaM0v9ZkpwKnAqw1157bUTp00D3+wgkaSNt6AnkTwK/AxwA3Ad8eMIqWoeqOreq5lbV3FmzZvXjkJK0WdigkUFVPTCynOSvgK+0u/cCe3Z1nd3aGKN9NbBDki3a6KC7vySpTzZoZJBkt667xwAjVxotBY5PsnWSvYF9gO8BNwD7tCuHtqJzknlpVRXwTeDYtv0i4EsbUpMkacONOzJIchGwANg5yTBwBrAgyQFAASuB/wRQVbcluQS4HVgDvKOq/qXt53TgSmAGcF5V3dYO8V7g4iQfAG4C/nrCHp0kqSfjhkFVnTBK85h/sKvqg8AHR2m/ArhilPa76VxtJEkaEN+BLEkyDCRJhoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJLYwK+91KZlaPHlzyyvPOs1A6xE0lTlyECSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEj2EQZLzkjyY5NauthckWZbkrvZzx9aeJB9NsiLJD5Ic2LXNotb/riSLutoPSnJL2+ajSTLRD1KStG69jAzOBxau1bYYuKqq9gGuavcBjgD2abdTgU9CJzyAM4CDgXnAGSMB0vq8rWu7tY8lSZpk44ZBVX0LeGit5qOBC9ryBcDrutovrI7rgB2S7Aa8GlhWVQ9V1cPAMmBhW/f8qrquqgq4sGtfkqQ+2dBzBrtW1X1t+X5g17a8B3BPV7/h1rau9uFR2keV5NQky5MsX7Vq1QaWLkla20afQG6v6GsCaunlWOdW1dyqmjtr1qx+HFKSNgsbGgYPtCke2s8HW/u9wJ5d/Wa3tnW1zx6lXZLURxsaBkuBkSuCFgFf6mo/sV1VdAjwSJtOuhI4PMmO7cTx4cCVbd0vkhzSriI6sWtfkqQ+2WK8DkkuAhYAOycZpnNV0FnAJUlOAX4CHNe6XwEcCawAHgdOBqiqh5KcCdzQ+r2/qkZOSr+dzhVLzwW+2m6SpD4aNwyq6oQxVh02St8C3jHGfs4DzhulfTkwZ7w6JEmTx3cgS5IMA0mSYSBJoodzBppehhZf/szyyrNeM8BKJE0ljgwkSYaBJMkwkCThOYPNwsqZb3xmeejJzw6wEklTlSMDSZJhIEkyDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJArYYdAEanKHFlz+zvPKs1wywEkmD5shAkmQYSJI2MgySrExyS5KbkyxvbS9IsizJXe3njq09ST6aZEWSHyQ5sGs/i1r/u5Is2riHNP11T+9I0kSYiJHBK6vqgKqa2+4vBq6qqn2Aq9p9gCOAfdrtVOCT0AkP4AzgYGAecMZIgEiS+mMyTiAfDSxoyxcAVwPvbe0XVlUB1yXZIclure+yqnoIIMkyYCFw0STUtmlbsj0AK2cOuA5J087GjgwK+FqSG5Oc2tp2rar72vL9wK5teQ/gnq5th1vbWO3PkuTUJMuTLF+1atVGli5JGrGxI4NDq+reJLsAy5L8U/fKqqoktZHH6N7fucC5AHPnzp2w/UrS5m6jRgZVdW/7+SBwKZ05/wfa9A/t54Ot+73Anl2bz25tY7VLkvpkg8MgyTZJthtZBg4HbgWWAiNXBC0CvtSWlwIntquKDgEeadNJVwKHJ9mxnTg+vLVJkvpkY6aJdgUuTTKyn89W1f9LcgNwSZJTgJ8Ax7X+VwBHAiuAx4GTAarqoSRnAje0fu8fOZksSeqPDQ6DqrobePEo7auBw0ZpL+AdY+zrPOC8Da1FkrRxfAeyJMkwkCQZBpIkDANJEn6fwWZn5cw3PrM89ORnB1iJpKnEkYEkyTCQJBkGkiQ8Z6DG70OWNm+ODCRJhoEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGGgUQ4sv/43PKpI0/RkGkiTDQJJkGEiS8PsMNmt+H7KkEY4MNhGe0JU0mRwZaEx++5m0+XBkIEkyDCRJThNNfUu2B2DlzMGW4ZSRNL05MpAkOTLQ+hvryiZHDNKmyzDQhHEqSdp0GQYCJv4NaI4epE2LYaC+Wp83z3UHh6MOaXKlqgZdAwBJFgIfAWYAn66qs9bVf+7cubV8+fK+1DZQ7WqiQdmcP6ZiokLHINNUkuTGqpr7rPapEAZJZgA/BP4QGAZuAE6oqtvH2mZzCIOhxZf/xvTNVLE5B8R0Mt7IyxCbnqZ6GPw+sKSqXt3uvw+gqv5irG2mXRgMeATQL91B4gflabL0El5jhd1I+1j7GG+7DdHPsJ3qYXAssLCq3truvwU4uKpOX6vfqcCp7e7vAnf2tVDYGfhZn4/ZK2vbMNa2/qZqXWBtvXhhVc1au3GTOoFcVecC5w7q+EmWj5aoU4G1bRhrW39TtS6wto0xVd6BfC+wZ9f92a1NktQHUyUMbgD2SbJ3kq2A44GlA65JkjYbU2KaqKrWJDkduJLOpaXnVdVtAy5rNAObouqBtW0Ya1t/U7UusLYNNiVOIEuSBmuqTBNJkgbIMJAkGQbrkuQFSZYluav93HGUPgck+W6S25L8IMkbJrmmhUnuTLIiyeJR1m+d5HNt/fVJhiaznvWs7U+T3N6ep6uSvHAq1NXV74+SVJK+Xf7XS21JjmvP221J+vbuvB7+PfdK8s0kN7V/0yP7VNd5SR5McusY65Pko63uHyQ5sB919Vjbm1pNtyS5NsmL+1XbuKrK2xg34H8Bi9vyYuBDo/TZF9inLe8O3AfsMEn1zAB+BPw2sBXwj8D+a/V5O/Cptnw88Lk+PVe91PZK4Hlt+Y/7UVsvdbV+2wHfAq4D5k6h52wf4CZgx3Z/lylU27nAH7fl/YGVfart5cCBwK1jrD8S+CoQ4BDg+n7U1WNtL+v6tzyin7WNd3NksG5HAxe05QuA163doap+WFV3teWfAg8Cz3p33wSZB6yoqrur6tfAxa3GsWr+PHBYkkxSPetVW1V9s6oeb3evo/N+koHX1ZwJfAh4sg81rU9tbwM+XlUPA1TVg1OotgKe35a3B37aj8Kq6lvAQ+vocjRwYXVcB+yQZLepUFtVXTvyb0n/fgd6Yhis265VdV9bvh/YdV2dk8yj8yrqR5NUzx7APV33h1vbqH2qag3wCLDTJNWzvrV1O4XOq7fJNm5dbRphz6ra8A+X2TC9PGf7AvsmuSbJde3TfadKbUuANycZBq4A3tmf0sa1vv8XB6VfvwM9mRLvMxikJF8HfmuUVX/WfaeqKsmY1+G2Vx5/AyyqqqcntsrpJcmbgbnAK6ZALc8BzgZOGnApY9mCzlTRAjqvIr+V5Peq6ucDrarjBOD8qvpw+7DJv0kyx///40vySjphcOigaxmx2YdBVb1qrHVJHkiyW1Xd1/7YjzpET/J84HLgz9qwdLL08rEdI32Gk2xBZ/i+ehJrWp/aSPIqOkH7iqr61RSoaztgDnB1m037LWBpkqOqarI/FreX52yYzrzyU8CPk/yQTjjcMAVqOwVYCFBV300yk86HsfVrKmssU/rjbZL8O+DTwBFV1Y/fzZ44TbRuS4FFbXkR8KW1O7SPz7iUzhzl5ye5nl4+tqO75mOBb1Q7WzXo2pK8BPi/wFF9nPteZ11V9UhV7VxVQ1U1RGcetx9BMG5tzWV0RgUk2ZnOtNHdU6S2fwYOa7XtB8wEVvWhtvEsBU5sVxUdAjzSNd07UEn2Ar4IvKWqfjjoen7DoM9gT+Ubnbn2q4C7gK8DL2jtc+l8GxvAm4GngJu7bgdMYk1H0vkioB/RGYkAvJ/OHzDo/EL+PbAC+B7w2318vsar7evAA13P09KpUNdafa+mT1cT9fichc401u3ALcDxU6i2/YFr6FxpdDNweJ/quojOVXtP0Rk5nQKcBpzW9Zx9vNV9S5//Pcer7dPAw12/A8v7Vdt4Nz+OQpLkNJEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJ+P/nIZLesPBeYQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(llr_in, bins=100, label='in');\n",
    "plt.hist(llr_ood, bins=100, label='ood');\n",
    "plt.title('Likelihood Ratio')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAFNCAYAAABbpPhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3hUZeL28e+TRgiEHor0UCJNFGnSQUHAtlgQVFZFUVcRFVCK7q5tdyU0RUUpgos/URBDUUAQFREFpUNCKKGGEkogvWee949ElxcpoUzOTHJ/rivXzJk5M3PjkczNKc9jrLWIiIiIiGfxcTqAiIiIiPyZSpqIiIiIB1JJExEREfFAKmkiIiIiHkglTURERMQDqaSJiIiIeCCVNBEREREPpJImIh7DGLPfGJNujEkxxsQZYz42xpQ+a512xpjvjTHJxphEY8xXxpjGZ61TxhjztjHmYP577clfrlS4fyIRkcunkiYinuYOa21p4HrgBmDU708YY24ClgMLgWuAusAW4GdjTGj+OgHAd0AToCdQBrgJiAdauyu0McbPXe8tIsWTSpqIeCRrbRywjLyy9rtwYJa19h1rbbK19pS19hVgLfBq/jp/BWoBfay12621LmvtcWvtG9baJef6LGNME2PMt8aYU8aYY8aY0fmPf2yMefOM9boYYw6dsbzfGDPCGLMVSM2/P++s937HGDMp/35ZY8xHxpijxpjDxpg3jTG+V/ifSkSKKJU0EfFIxpgaQC8gJn85CGgHfHGO1ecC3fPv3wJ8Y61NKeDnBAMrgG/I2ztXn7w9cQXVH7gNKAd8DvTOf0/yC1hfYHb+uh8DOfmfcQPQA3j8Ej5LRIoRlTQR8TQLjDHJQCxwHPhn/uMVyPuddfQcrzkK/H6+WcXzrHM+twNx1trx1tqM/D10v17C6ydZa2OttenW2gPARqBP/nPdgDRr7VpjTBWgN/C8tTbVWnscmAj0u4TPEpFiRCVNRDzNX6y1wUAX4Fr+V75OAy6g2jleUw04mX8//jzrnE9NYM9lJc0Te9bybPL2rgE8wP/2otUG/IGjxpgEY0wCMAWofAWfLSJFmEqaiHgka+2P5B0eHJe/nAqsAe47x+p9+d8hyhXArcaYUgX8qFgg9DzPpQJBZyxXPVfUs5a/ALrkH67tw/9KWiyQCVSy1pbL/yljrW1SwJwiUsyopImIJ3sb6G6MaZ6/PBJ42BgzxBgTbIwpn39i/03Aa/nrfEJeIfrSGHOtMcbHGFPRGDPaGNP7HJ/xNVDNGPO8MaZE/vu2yX9uM3nnmFUwxlQFnr9YYGvtCWAlMBPYZ62Nzn/8KHlXpo7PHyLExxhTzxjT+TL+u4hIMaCSJiIeK7/wzAL+kb+8GrgVuJu8884OkHcCfgdr7e78dTLJu3hgB/AtkAT8Rt5h0z+da2atTSbvooM7gDhgN9A1/+lPyBviYz95BWtOAaPPzs8w+6zH/woEANvJO3w7j0s7NCsixYix9uw99SIiIiLiNO1JExEREfFAKmkiIiIiHkglTURERMQDqaSJiIiIeCCVNBEREREP5Od0gEtVrlw5W79+fadjyGVKTU2lVKmCjjEqnkTbzrtp+3kvbTvvtmHDhpPW2pDLea3XlbQqVaqwfv16p2PIZVq5ciVdunRxOoZcBm0776bt57207bybMebA5b5WhztFREREPJBKmoiIiIgHUkkTERER8UAqaSIiIiIeSCVNRERExAOppImIiIh4IJU0EREREQ/ktpJmjJlhjDlujIk8z/PGGDPJGBNjjNlqjGnhriwiIiIi3sade9I+Bnpe4PleQIP8nyeAD9yYRURERMSruK2kWWtXAacusMpdwCybZy1QzhhTzV15RERERLyJk9NCVQdiz1g+lP/YUWfiiIiIiKew1uKykJ3rItdlyXHZ/FsXObl593NdllxrceXf5rosLhf/u3/GrctF3rK1WGvJzV92ufI+54/3yV/f2vzn82/tGfddNi9fckYOJfx9sJY/MrrOyHQsdt8V/Tfwirk7jTFPkHdIlJCQEFauXOlsILlsKSkp2n5eStvOu2n7eS9P2nY5Lku2C9JzLOk5ebdZuZCVa8nMhcxcS2KmBfhjOTMXsvNvs1yQnJVXcgCyXZbUrLz39PeFXBfk2v/dAvga8PHJu/U14GNM/u25fkzeLWDOes6Qf5u/zu/Lf7pvwAfzx33z+3vx52UMZOVCkJ/BxyfvsdysDKJ/Wc419RpRPqTqFf33drKkHQZqnrFcI/+xP7HWTgWmAoSFhVlNNOu9NFGw99K2827aft7LHdsuK8fF6bQs4lOySEjLIj41i1OpWZxMySQxPfuP+8kZOfk/2SRn5OCylpL+vpQq4UdwoB+lA/0J8velZIAvJf19KVnCF/+ShlIl/ChX0p+SAb4EBfhRMsCHQD9fAgN8CfTzxcdA6UA/Svj5EODri6+voYSfD34+Bj/fvFt/Xx98fcxV/XO7U2pqKhMmTOC9997jpptuYkTfp2jatCnvDrn893SypC0CBhtjPgfaAInWWh3qFBERKaBclyUlI4eE9CwS07M5nZbNyeRMTqVmcSoti9P55SsxPZvE9LyilZieTUZ2LuWCAqhYKoByQf5UKl2C8qX8qViqBKGVSnFj7fJUKl2CMoH+lA70o0ygH6UD/Qjw9cEY7ylOheHgwYMkJCTQsGFDTpw4wcqVK2nUqNFVeW+3lTRjzGdAF6CSMeYQ8E/AH8Ba+yGwBOgNxABpwKPuyiIiIuINrLUkZ+ZwLDGDE8mZxCVl8Ou+bNakRXMyJYukjP/t5TqdmkVKZg6lSvhRtqQ/ZUv6Uz4ogEqlA6hQqgQVSwdQs3wQ5YP8KRvkT5nAvHXKlPSnTKCfytYVioyMJDw8nMWLF/Pvf/+b6667jkmTJl3Vz3BbSbPW9r/I8xZ4xl2fLyIi4inSsnI4kZzJ8eRMjiZm/LG362hiBvGpmRxLyiQ+JZOE9Gz8fQxVygQSElyCymUCycpwUScogHohpSlT0p/yQf6EBJegfFAAZUv64+NFhwS9nbWW+Ph4KlasyJAhQ+jevTuTJk2iXLlybvk8r7hwQERExFOlZ+Vy6HQacUkZHDqdzuHT6RxJSOdoYgbHkjKIS8og12XzSldwCaqWDaRycCDlgwJoU7cClYIDCCmdV8rK5p/Hdaa8c9LqOfSnEwCXy8WiRYsYM2YMFSpUYPHixXz//fdu/1yVNBERkQtwuSxHEtOJPZXOwVOp7DuZRuypNGJPp3EkIYPkjGyqlytJlTKB1ChfkurlS9K2XkWqlQ2kaplAqpYNpHQJHV70RtZajDE88sgjREdHM2LECPr06VNon6+SJiIixZ61llOpWRw4lcaB+LwidiA+lYOn0og5lkJQCV9qVQiiZoUgQiuVokeTKtSsEET1ciUJKV1ChxyLmMTERKZMmcKcOXNYu3Yt77zzDuXKlSv0oq2SJiIixYa1lvjULPafTGXviVSijiQSHZfMzrhkrLXUqVSKWvlFrFODEGqUL0mDKsFUKBXgdHQpJJ9++ilDhgyhZ8+ezJgxA39/f8qXL+9IFpU0EREpkuJTMtl7MpUD8WnsOZHCltgEth9NwlqoU6kU9SqV4tpqwdzSuAqNqpWhYqkAHZIspnbt2sXbb7/NG2+8QevWrVm/fj1169Z1OpZKmoiIeLesHBfbDiew6WACm2ITOHQqjf3xabispX7l0tSuEETdSqUZ1DGUJtXLEFK6hMqYAHnlbNSoUfz00088/fTT+Pn50aBBA6dj/UElTUREvEZ6Vi6bYxNYt/8U248ksfNYMkcS0qlUugRdwkJoUas8j7arQ51KpbRnTM7JWsuyZcto3LgxPj4+dO7cmVmzZlGqVCmno/2JSpqIiHikrBwXu44ls+1wIlFHEtl0MIG9J1IJqxpM67oV6H1dNYZWbUitCkEE+vte/A2lWMvOzmbu3LmEh4djrWX69Om0bt2aIUOuYN4mN1NJExERjxB7Ko1f9pxky6FEIg8nsjMumZoVgriuelkaX1OGPjdUp8k1ZVXI5JKkpqaSlJSEn58fs2bN4q233qJnz55esZdVJU1ERArd6dQsth5OZPPBBDYePM3OuGSyc110aFCJ5jXK0eeG6jSrrkIml+/kyZO89957TJ48mZEjRzJ06FCWLVvmdKxLopImIiJuZa1lz4kU1uyJZ8OB02w8mMCp1Cyuq1GWJteU4aG2tbm2ajA1ypf0ir0b4tlyc3MBaNeuHZ07d+ann34iLCzM4VSXRyVNRESuquxcF5GHE9l4MIH1+0+x/sBpAnx9aFevIm1DK/JM1/rUCymtAWDlqtqyZQvh4eGkpaUxf/58tm3bRokSJZyOdUVU0kRE5IpYa9l3MpXvdxznx10n2BybQNUygVxfsxy3NqnKqF6NqFUxyOmYUoQ99dRTLFq0iOeff54nn3wSwOsLGqikiYjIJcoblyyRLbEJ/LYvb0+Zn4+hc8MQHmpbm7fvv56Kpb3/C1I8V25uLgsWLGDBggXMmjWL5557jnfeeadIFLMzqaSJiMgFWWvZH5/Gmj3x/LDzOCt3Hqdm+SDahFbk1qZVePm2RjqfTArNokWLGD58OBUqVGDEiBFYa2nUqJHTsdxCJU1ERP4kJTOHH3Yc56fdJ/g5Jp4cl4t29SrRs0lV3rq7mfaUSaFKSEhg5syZPPPMM1SoUIFp06bRqVOnIv8PA5U0EREB4HhSBt9GH+Pb7cdYv/80LeuUp0vDEJ7oFEq9kNJF/gtRPM/Ro0eZMGECM2bMoHfv3iQlJdGhQwenYxUalTQRkWIqLSuH3/adYllUHOv3n+ZYUgZdwipzT4saTOp/A2UC/Z2OKMXUjh07qFatGnv37iUrK4uNGzdSu3Ztp2MVOpU0EZFi4vfxyn7cdZKfY07y275TNKoWTJewyjzQujaNqgXj5+vjdEwpxtasWcOYMWNYs2YN8+fPp3379rRv397pWI5RSRMRKcKstWw/msS8XVm8uXEVqZk5dAkL4Z4WNZjQtznlggKcjijFnLWWjIwM4uLieOihhxg6dCizZ88mKEjDtqikiYgUMVk5LjYcOM2K6GMs3x6HwdC4DITfex031Cync8vEI2RnZ/PZZ58xduxYBgwYwEsvvcTu3bvx8dHe3N+ppImIFAGnU7P4NvoY30cf5+c9J6lTsRTdrq3MBw/eSJNryvDjjz/SolZ5p2OKAHkFrUmTJtSsWZPx48fTvXt3ABW0s6ikiYh4qeNJGczfdJgfd51g26FE2tWvSOeGlXn9L02oHBzodDyR/8/x48d59913SUpK4p133mHFihXUqlXL6VgeTSVNRMSLpGbm8OOuE0RsPMxv++K5pXEVHm5Xh84NQwj093U6nsg5jR49mg8//JC+ffsyfPhwABW0AlBJExHxcDm5LjbHJvDF+kMs2XaUptXLcnvzarzT73pKldCvcfFMmzZtYtmyZYwcOZKuXbsyZMgQqlat6nQsr6K/3SIiHig9K5cfd51gRf7gstXKBnJbs2p8P7wLIcEa7V881+rVq3n99dfZvn07L7zwAtbaP845k0ujkiYi4iEysnP5Lvo430TF8cOO4zSvWZabr63CsB4NqVa2pNPxRM4rNzeXZcuW0atXL2JjY3nggQd44IEHCAjQEC9XQiVNRMRB1lo2xSYw+9eDfLv9GM2ql+XWplV59Y7Gmh9TPF56ejoff/wx48aNo2rVqrRt25b+/fs7HavIUEkTEXHA4YR0IjYcImLTYTKyc3m4XR2G9wijalldlSme7/Tp05QuXZqvvvqKpUuXMmvWrGI9M4C7qKSJiBSSxLRsFm09whfrY9l9LIU+Laozvm9zDTArXiM2Npa3336bmTNnsnjxYvr27Uvfvn2djlVkqaSJiLhRTq6LH3ae4Iv1sazZG0+nBiEM7xHGTfUq4q95MsVLWGuJjo6mQ4cOPProo2zZsoWaNWs6HavIU0kTEXGDHXFJfL3lKPM2HKJ6+ZLc06IG4fdep7kyxausXr2a8PBwbr31Vp5++mn27t1LuXLlnI5VbKikiYhcJWlZOSzcfITZvx7kRHImdzSvxsxHW9GoWhmno4lckvT0dLp3705cXBzDhw/n4YcfxhijglbIVNJERK6AtZbIw0l8tu4gX285Qqs6FRh+axgd61fCx0fnmYn3yMrKYvbs2aSmpvLMM8/w5ptv0rFjR3x9NZOFU1TSREQuQ0pmDnPXxfLZbwfJyMml7401+XZoZ6qU0dWZ4n0mTZrE2LFjadSoEaNHjwagS5cuzoYSlTQRkUsRczyZj3/Zz6LNR+jQoBJv/KUpbepW0NWZ4nWOHTvGzz//zN13301AQAALFy6kRYsWTseSM6ikiYhcRGpmDj/tPsGnvx4k+mgyD7SpxfIXOmtMM/FKMTExjBs3jrlz5/Lwww/Tp08fnnrqKadjyTmopImInIO1lt/2nWLO+li+jTpGsxpluadFDaY/XI0SfjpHR7xPTEwM9evX5/PPPyckJIQdO3ZQuXJlp2PJBaikiYicITUzh4hNh5n1y34A7m5Rg5d7N9IUTeKVrLUsX76cMWPGsGfPHiIjI3nllVecjiUFpJImIgLEnkrj41/2E7HxEK3rVuC1u5pwU2hFnWsmXiknJwdfX18++ugj3n77bV566SX69++Pv7+/09HkEqikiUixtuHAaT5avZdf9sRzf8uaLBrcgZoVgpyOJXJZ0tLSmDFjBuPHj2fevHn89a9/5bHHHtM/NryUSpqIFDs5uS6WRR1j+uq9xKdk8Wj7OoTf25zSJfQrUbzX1q1b6d69O+3atWP27NnceOONTkeSK6TfSCJSbCRlZDPrl/3M/vUg1cuX5MlO9ejeuAq+GnRWvNSBAweYOHEi3bp1o2fPnvz4449ce+21TseSq8Sts/saY3oaY3YaY2KMMSPP8XwtY8wPxphNxpitxpje7swjIsVPrsvyw47jPP3pBtr953t2HUth6l9b8sVT7ejZtKoKmnil5ORkBgwYQIsWLfD396dly5YEBASooBUxbtuTZozxBd4HugOHgHXGmEXW2u1nrPYKMNda+4ExpjGwBKjjrkwiUnycTs1i7vpY/u/XA5Qt6c/9rWrxnz7XUTZIJ06Ld7LWsmrVKlJTU+nVqxft27fn3Xff1XyaRZg7D3e2BmKstXsBjDGfA3cBZ5Y0C/w+83BZ4Igb84hIEZfrsqzafYIvNxzix10n6N64Cu/2b8H1NfUlJt7LWsv8+fMZM2YM8fHx/Pvf/8YYowFoiwF3lrTqQOwZy4eANmet8yqw3BjzLFAKuMWNeUSkiErLyuHTtQeZ/dtBSpXwpX/rWrx+V1MqlApwOprIZcvMzGT79u0YY1i5ciUvvvgif/nLXzTheTFirLXueWNj7gV6Wmsfz18eALSx1g4+Y52h+RnGG2NuAj4CmlprXWe91xPAEwAhISE3zp071y2Zxf1SUlIoXbq00zHkMnjitkvNtnx/MJtvD2TTsLwvt9bxp345Hw03cA6euP3k3FJSUvjqq6+IiIigefPmPP/889p2Xqxr164brLUtL+e17tyTdhioecZyjfzHzvQY0BPAWrvGGBMIVAKOn7mStXYqMBUgLCzMdunSxU2Rxd1WrlyJtp938qRtF5+SyYyf9zH714N0vbYyEYPrUb9ysNOxPJonbT85t9TUVEqVKsWgQYNIS0tjxYoVNG/eXNuuGHNnSVsHNDDG1CWvnPUDHjhrnYPAzcDHxphGQCBwwo2ZRMSLHU1MZ+qqvURsPMzt11XTwLNSJOzcuZNx48axZMkSYmJimDJlCj4+bh18QbyE2/4vsNbmAIOBZUA0eVdxRhljXjfG3Jm/2jBgkDFmC/AZ8Ih11/FXEfFasafSGBWxlZ5v/4SvMSx/oRP/6tNMBU283kcffUTHjh2pXr06W7ZsoWTJkipo8ge3DmZrrV1C3rAaZz72jzPubwfauzODiHivPSdSmPzDHr7fcYwH2tTih+FddDGAeDVrLd988w1jx45lypQp9OnTh379+lGqVCmno4kH0owDIuJxdsQl8f4Pe/gl5iQPt6vDyhe7UrakxjcT77Zt2zYefPBBAEaMGEGdOnU04blckEqaiHgEay1r9sYz65cDbDh4msc71OU/dzfTfJri1VJTU/noo49o164doaGhjBkzhp49e+oKZCkQ/fYTEUe5XJZlUXG890MMWTku+reuxYT7mxMUoF9P4r1SUlIYO3YsH3zwAR07duSWW26hQoUK9OrVy+lo4kX0W1BEHJGT6+LrrUd5/4cYggJ8ef6Whtx8bWV8NJemeLH9+/eTmJhIWFgYiYmJ/PTTT4SFhTkdS7yUSpqIFKqM7FwWbT7Cez/EULVsIH+/vTEdG1TS4R/xalu2bCE8PJxvvvmGt956i+bNm/P22287HUu8nEqaiBSK9KxcPv31AFNW7SWsSjBj772ONqEVnY4lctmstZw4cYKQkBCGDRvGrbfeyuTJkylbtqzT0aSIUEkTEbc6s5y1rF2eWQNb06haGadjiVy23Nxc5s+fT3h4OFWqVOGrr75ixYoVTseSIkglTUTcIiM7lznrYnn/hxhuqFWO/z7amsbXqJyJ97LWYozhr3/9K3v37mX06NHceeedF3+hyGVSSRORqyon18Xn62J5e8VurqtRlo8ebkWzGjr8I94rISGBDz74gLlz5/Lbb78xefJkypQpo/Moxe1U0kTkqnC5LN/vOE74sh1UKBXAx4+2oml1lTPxbv/3f//Hc889x2233casWbPw9/fXOWdSaFTSROSKrd0bz5uLt+NywbAeYfRoXEV7GcRrRUdH8/bbb/Of//yHtm3bsmnTJmrVquV0LCmGVNJE5LLtiEvitUXbOZKYznM3N6DPDdVVzsRr7dy5kxEjRrBmzRoGDx6Mn58f9evXdzqWFGMqaSJyyQ6dTuPd72L4JiqO4beG0a9VTfx9fZyOJXLJXC4XS5YsoVmzZvj6+tK9e3dmz55NUFCQ09FEVNJEpOBOJGcy5psdLI+KY8BNtVn1YlfKBmmCaPE+2dnZfPbZZ4SHh+Pv78+0adNo2bKl9pyJR1FJE5GLSs3MYdGeLF5Y9SN9W9Xkp5e6qZyJV0pJSSEpKQl/f38+//xzJkyYQPfu3XWYXjySjk+IyHnluiwRGw/RddxKjqS4mP90e0b1aqSCJl7n+PHj/P3vf6du3bp88cUXhISEsGTJEnr06KGCJh5Le9JE5E9yXZZvIuOYuGIX5Ur6M/nBFqTs30qdSqWcjiZySXJycjDG0LFjR7p27cqaNWt0SFO8hkqaiPx/dsYlMypiK9m5lldua0TnhiEYY1i53+lkIgW3ceNGwsPDyczMZP78+Wzbto2AgACnY4lcEpU0EQEgMS2bcct3smTbUYbc3IAH29TCT1dsihcaNGgQS5cu5YUXXuCJJ54AUEETr6TfwCLFXE6uixmr93HzhJW4rOW7YZ15uF0dFTTxGjk5OcyZM4cHHngAay3Dhw9n7969DBs2jODgYKfjiVw27UkTKcbW7Innta+iqFAqgE8fb0tYVX2hiXdZsGABw4YNo2rVqowYMQKAsLAwh1OJXB0FKmnGmACglrU2xs15RKQQ7IxL5s3F2zkQn8awHg25s/k1usJNvMapU6eYMWMGQ4YMISQkhFmzZtG+fXunY4lcdRc9nmGMuQ3YBnybv3y9MWa+u4OJyNWXkJbFf5ZGc//UNXRvXIUVQztz1/Waykm8w5EjRxg6dCj169cnKiqK5ORk2rdvr4ImRVZB9qS9DrQBfgCw1m42xuj6ZREvkpXjYvavB3jvhz3cfG1llj3fiSplAp2OJVIgUVFR1KhRg3379mGMYevWrdSoUcPpWCJuV5CSlm2tTTjrX9rWTXlE5CpbHhXHf5buoHbFIP47sBVNrinrdCSRAlm9ejVjxoxh3bp1zJ8/X3vNpNgpSEmLNsb0BXyMMXWBIcBa98YSkSsVczyZ177azuHT6fzzjsZ0CavsdCSRi3K5XGRkZBAXF8ejjz7KsGHDmDt3LiVLlnQ6mkihK8g19oOBGwEXEAFkAs+5M5SIXL5TqVmMitjKvR+uoWtYZZa90EkFTTxeVlYWM2fOpGnTpkyePJnQ0FB27tzJU089pYImxVZB9qTdaq0dAYz4/QFjzN3kFTYR8RAul2XO+ljGLdvJHc2v4ccXu1K2pObYFM+XnZ1N48aNCQ0N5d1336Vbt24A+PhorD4p3gpS0l7hz4Xs5XM8JiIOWb//FG8ujgbgk8fa0PiaMg4nErmwY8eO8c4775CSksKkSZNYuXKlLgYQOct5S5ox5lagJ1DdGDPhjKfKkHfoU0QcFnM8hbeWRrP9SBIvdG/IPS1q4OOj4TTEs40cOZKpU6fSv39/hg0bBqCCJnIOF9qTdhyIBDKAqDMeTwZGujOUiFxYckY2734fw5x1sTzTtR7v9m9ByQBfp2OJnNf69etZvnw5o0eP5pZbbmHo0KFUrqxzJUUu5LwlzVq7CdhkjPnUWptRiJlE5DyyclzMWXeQSd/H0KlBCN++0InKGu9MPNiqVat47bXX2L17N8OGDcNayy233OJ0LBGvUJBz0qobY/4FNAb++Daw1jZ0WyoR+ZPf9p1iZMRWqpcryYyHW9GshsY7E8+Uk5PD0qVLuf322zly5AgPP/ww/fv3x99fF7KIXIqClLSPgTeBcUAv4FE0mK1IoTmdmsWbi6NZHXOC1+5sQs+m1ZyOJHJOaWlpzJgxg/Hjx1OzZk3at29Pv379nI4l4rUKcn1zkLV2GYC1do+19hXyypqIuNmK7cfo9c5PBAX48u3Qzipo4pHi4+PJzs7m66+/ZsWKFcyePZtVq1ZRoUIFp6OJeLWC7EnLNMb4AHuMMU8Bh4Fg98YSKd4OxKfy2lfb2XsihfF9m9O+fiWnI4n8yYEDB5gwYQKffPIJS5YsoW/fvvTt29fpWCJFRkH2pL0AlCJvOqj2wCBgoDtDiRRX1lrmroulz+RfaFWnAstf6KyCJh7HWktUVBQtWrSgRIkSREZG0rZtW6djiRQ5F92TZq39Nf9uMjAAwBhT3Z2hRIqj48kZvPjFVo4nZzJ7UBuuraoBacVzWEKyDXIAACAASURBVGtZtWoVY8aM4bbbbuPpp59m7969lC2rC1hE3OWCJc0Y0wqoDqy21p40xjQhb3qoboBGHhS5ClwuS8Smw7y1dAf9WtVkyM0NCPDTdDjiOdLS0rj55puJj4/nxRdfZMCAARhjVNBE3OxCMw78B7gH2AK8Yoz5GngaGAM8VTjxRIq26KNJjIrYRq7LMv3hllxfs5zTkUQAyMzM5JNPPiEjI4PBgwczZswY2rdvj6+vBk0WKSwX2pN2F9DcWptujKkAxALNrLV7CyeaSNGVmpnDlFV7+WTNfkb2upb7bqyp6ZzEY0ycOJGxY8fSvHlzRo8eDUCnTp0cTiVS/FyopGVYa9MBrLWnjDG7VNBErtz8TYf41+Jo2oRWZMlzHalWtqTTkUQ4evQoP//8M/feey9BQUEsXbqU5s2bOx1LpFi7UEkLNcZE5N83QN0zlrHW3u3WZCJFTHJGNv9cGMWGg6f578DWNLlG5/OI83bu3Mm4ceP48ssvGThwIPfeey9PPvmk07FEhAuXtHvOWn7vUt/cGNMTeAfwBaZba986xzp9gVfJm8Vgi7X2gUv9HBFP98PO4/xjYSRt61Zk6XMdCQooyBCFIu6za9cuGjZsyLx586hevTq7du2iUiUN9yLiSS40wfp3V/LGxhhf4H2gO3AIWGeMWWSt3X7GOg2AUUB7a+1pY0zlK/lMEU+TmJ7Nm19vZ+2+eF6/syldr9X/4uIcay1Lly4lPDycAwcOsHXrVl5++WWnY4nIebjzOv/WQIy1dq+1Ngv4nLyLEc40CHjfWnsawFp73I15RAqNtZb5mw7RfcKP+Pn6sPS5Tipo4picnBystUyfPp2RI0cyaNAgdu3aRXCwJo8R8WTGWvfMlW6MuRfoaa19PH95ANDGWjv4jHUWALvIm8nAF3jVWvvNOd7rCeAJgJCQkBvnzp3rlszifikpKZQuXdrpGG6VkOlidnQWh5JdPH5dCULLFo0hC4rDtitq0tPTWbx4MfPmzWPUqFE0adIEX19fjNGVxN5Ef/e8W9euXTdYa1tezmsLfGKMMaaEtTbzcj7kIp/fAOhC3uC4q4wxzay1CWeuZK2dCkwFCAsLs126dLnKMaSwrFy5kqK6/Vwuy4LNh3nj6+3c17I2/72lISUDikZBg6K97YqiLVu2cN9999GpUycWLVpEWlqatp+X0t+94uuiJc0Y0xr4CCgL1DLGNAcet9Y+e5GXHgZqnrFcI/+xMx0CfrXWZgP7jDG7yCtt6wqYX8QjxBxPYdjczVjgvwNbc10NDUorhW/fvn1MmDCBW265hV69evHzzz/TsGFDIO+LXkS8S0HOSZsE3A7EA1hrtwBdC/C6dUADY0xdY0wA0A9YdNY6C8jbi4YxphLQENBYbOI1XC7LR6v3ce+Hv3Bfy5oseLq9CpoUuqSkJB544AFatmxJqVKlaN26NQEBAX8UNBHxTgU53OljrT1w1jkMuRd7kbU2xxgzGFhG3vlmM6y1UcaY14H11tpF+c/1MMZsz3/PF6218Zf8pxBxQMzxFP6+IJKMnFwWPtOe2hVLOR1JihFrLStXriQtLY3evXvTpUsXPvjgA82nKVKEFKSkxeYf8rT5w2o8S97J/hdlrV0CLDnrsX+ccd8CQ/N/RLxCTq6LT9Ye4N3vY/hb53oM7FAXX03pJIXEWsuXX37JmDFjSE5O5l//+hfGGJ544gmno4nIVVaQkvY38g551gKOASvyHxMpdo4kpDN49kb8fH2Y+2Rb6lfWEAZSODIyMoiKiuLGG2/k559/5uWXX+bOO+/Ex8edIymJiJMKUtJyrLX93J5ExMMt3XaUfyyK4tH2dXiqUz1NiC6FIiEhgQ8++IBJkybRvXt3Zs2axcSJE52OJSKFoCAlbZ0xZicwB4iw1ia7OZOIRzmdmsUrCyOJPprEhw+14MbaFZyOJMXA72NjDR8+nKysLJYvX06zZs2cjiUiheiiJc1aW88Y0468qzNfM8ZsBj631n7u9nQiDrLW8sX6Q4z5Zgd3Xn8N4+/rSKB/0Rn3TDxTdHQ0Y8eOZdmyZezZs4dp06Zp8FmRYqpAJzNYa3+x1g4BWgBJwKduTSXisKOJ6Twycx2z1u7nvwNb8887mqigidtNmzaNLl26ULduXbZu3UpgYKAKmkgxdtGSZowpbYx50BjzFfAbcAJo5/ZkIg6w1rJg02F6v/MT19csx/yn29O0uoY0EPdwuVx8/fXXdO7cmd27d3Pvvfeyb98+/v73v1OxYkWn44mIwwpyTlok8BUQbq39yc15RByzMy6Z176K4nhyJp881kblTNxqy5YtPPjgg/j7+zNixAjq1q2Ln1+BZ+oTkWKgIL8RQq21LrcnEXFIWlYO734fw+xfDzL81jD6taqJv6+GNZCrLyUlhWnTptGxY0dCQ0OZMGEC3bt31yFNETmn834TGWPG59/90hgTcfZPIeUTcatthxLpPmEVh0+n8+0LnRjQtrYKmlx1ycnJvPLKK9StW5e1a9dSunRpKlSoQI8ePVTQROS8LrQnbU7+7XuFEUSkMGXluHhr6Q7mbzrEG39pyu3XXeN0JCmC9u7dS1JSEtdeey0ZGRmsWbOG+vXrOx1LRLzEeXcZWGt/y7/byFr73Zk/QKPCiSdy9e2IS6LP5J+JPZ3Gshc6qaDJVbdx40b69etH69at2bBhA4GBgYwbN04FTUQuSUGO6ww8x2OPXe0gIu6WmZPLxG938cC0XxnQtjZTB9xI5eBAp2NJEWGt5dixY1hrGTlyJK1bt2bfvn089ph+XYrI5Tnv4U5jzP3kDWBb96xz0IKBBHcHE7maIg8nMnTuZmpXLMVXz3agermSTkeSIiInJ4cvv/yS8PBwatSowcKFC1m+fLnTsUSkCLjQOWm/AfFADeD9Mx5PBja5M5TI1ZKd62Lcsp3M23CI0b0bcXeL6jpRW64Kay3GGAYMGEBsbCyvvvoqt912m9OxRKQIOW9Js9buA/YBKwovjsjVE3UkkWFzt3BNuZIsf6ETFUuXcDqSFAGnTp1i8uTJzJs3j3Xr1jFlyhTKlCnjdCwRKYIuNATHj/m3p40xp874OW2MOVV4EUUuTa7LMvPnfTw0/VcGdQzlo4dbqqDJVTFr1izq16/Pnj17mD17Nv7+/ipoIuI2Fzrc2TX/tlJhBBG5GvadTGX4F1swQMTT7albqZTTkcTLRUVFMXHiRMLDw2nfvj1bt26lRo0aTscSkWLgQkNw/D7LQE3A11qbC9wEPAnom088ztdbj3DPB7/Qq2lV5j55kwqaXJEdO3Zwxx13cPPNNxMaGoqfnx/16tVTQRORQlOQaaEWAK2MMfWAmcDXwGzgdncGEymoXJcl/JsdLI2MY8Yjrbi+ZjmnI4mX+n3C8+bNm+Pv78/tt9/O3LlzKVlSVwOLSOErSElzWWuzjTF3A+9aaycZY3R1p3iEE8mZvDhvC1k5LuY/3U7nnsllycrK4tNPP2Xs2LEEBQUxZcoUbrzxRurVq+d0NBEpxgpS0nKMMfcBA4C/5D/m775IIgWzevdJRny5ld7NqjL81jBK+Pk6HUm8TFJSEsnJyQQEBBAREcG7775Lt27dNEyLiHiEgs440BUIt9buNcbUBT5zbyyR87PW8uGPe3h+zib+eUdjXr6tsQqaXJK4uDhGjx5NaGgoERERhISE8NVXX3HzzTeroImIx7jonjRrbaQxZghQ3xhzLRBjrf2X+6OJ/NnRxHRGfLmNpPRsFg7WzAFyabKzs/Hx8aFz5850796ddevWUbduXadjiYic00VLmjGmI/AJcBgwQFVjzABr7c/uDidypm+3H2NUxDYeaFOLId3q4+dbkB3BIrBu3TrGjBmDy+UiIiKCyMhI/P111oaIeLaCnJM2Eehtrd0OYIxpRF5pa+nOYCK/y3VZ/r0kmiXbjjJlwI3cWLu805HEizz22GOsWLGCoUOH/jHZuQqaiHiDgpS0gN8LGoC1NtoYE+DGTCJ/OJqYzgtzNuPrY1j6XEfKBel/PbmwnJwc5s6dy6JFi/jss88YOXIkH374oYqZiHidghwv2miM+dAY0yH/5wM0wbq4mbWWT9bsp9c7P3FTaCVmDWyjgiYXFRERQYMGDZgyZQoDBgwAoEGDBipoIuKVCrIn7SlgCPBS/vJPwLtuSyTFXmJaNi99uYUjCRnMffImGlYJdjqSeLD4+HimT5/OCy+8QNWqVfnss89o27at07FERK7YBUuaMaYZUA+Yb60NL5xIUpztPZHCY/9dT/v6FZnU/wYNrSHndejQIcaOHcsnn3zC3XffTUpKCu3atXM6lojIVXPekmaMGQ08Bmwkb1qo1621MwotmRQ70UeTeHTmOgZ3q89DbWs7HUc81NatW6lduzYHDx6kRIkSREZGcs011zgdS0TkqrvQnrQHgeustanGmBBgCaCSJm6xaMsRXl0UxT/vaMxd11d3Oo54GGstq1atYsyYMWzatIn58+fTrl077TkTkSLtQiUt01qbCmCtPWGM0aBUctXl5Lr4+8JI1u49xcePtuK6GpocXf4nNzeXjIwM4uLiePLJJxk2bBgREREEBgY6HU1ExO0uVNJCjTER+fcNUO+MZay1d7s1mRR5p1OzeG7OZnJyXSwe0oGggIJcxyLFQWZmJrNmzWLcuHE8+eSTDB06lOjoaE3ZJCLFyoW+Fe85a/k9dwaR4mXboUSemb2R7o2rMKLntQT4aUet5MnKyqJRo0aEhYUxZcoUOnfuDKCCJiLFznlLmrX2u8IMIsXHN5FxjJ6/jdfubMIdzXXCt8CRI0d45513SE1N5b333mP16tW6GEBEij3tvpBCk5SRzczITF5ZsI1ZA1uroAkAL774Ik2bNiU9PZ3hw4cDqKCJiKCSJoVkc2wCt05chQG+H96FptXLOh1JHLR27VreeOMNAHr16sWuXbuYNGkSderUcTaYiIgHKXBJM8aUcGcQKZqstczbcIiBH6/jtTub8EjTEpQJ1BQ9xdXKlSvp3Lkz/fr1o1y5clhr6datG5UqVXI6moiIx7loSTPGtDbGbAN25y83N8ZoWii5qIzsXF6Ys5kPVsYwe1AbejSp6nQkcUB2djYLFizAWsuxY8d48skn2b17N88++6wuBhARuYCCjHkwCbgdWABgrd1ijOnq1lTi9fadTGXgx+u4vmY5vn62IyUDNL1TcZOamsr06dOZMGECoaGhdO7cmfvvv9/pWCIiXqMgJc3HWnvgrH/x5ropjxQBB+JTeXjGbwxsX4eH29XR3pJi5sSJE5QvX57Fixfz008/8cUXX9C6dWunY4mIeJ2CnJMWa4xpDVhjjK8x5nlgl5tziZfaEpvAfR+u4fGOdXmkfV0VtGJk3759DB48mIYNG7Jhwwb69u3LvHnzVNBERC5TQUra34ChQC3gGNA2/7GLMsb0NMbsNMbEGGNGXmC9e4wx1hjTsiDvK57pu+hjDPx4Hf/u04y/3lTH6ThSSKy1REZG0qpVK4KDg9m+fTtt2rRxOpaIiNe76OFOa+1xoN+lvrExxhd4H+gOHALWGWMWWWu3n7VeMPAc8OulfoZ4jrnrYhm3fCcfPdKK62tq/s2izlrLDz/8wJgxY7jrrrv429/+xr59+wgODnY6mohIkXHRkmaMmQbYsx+31j5xkZe2BmKstXvz3+dz4C5g+1nrvQGMAV4sSGDxPO+s2M2cdQeZPagN9SvrS7qoS0tLo0uXLiQnJ/Piiy/y4IMPYoxRQRMRucoKcuHAijPuBwJ9gNgCvK76WesdAv6/YyDGmBZATWvtYmOMSpqXSUjL4h8Lo9h+NIlFz3agUmkNpVdUZWRksGjRIrZu3cqQIUOYOHEiN910Ez4+Gg9bRMRdCnK4c86Zy8aYT4DVV/rBxhgfYALwSAHWfQJ4AiAkJISVK1de6cfLFYqOz+XDrZm0quLLi80DiFy/pkCvS0lJ0fbzMnPmzGHu3LmEhoYSGhr6x/ZbtWqVs8HkkujvnvfStiu+CrIn7Wx1gSoFWO8wUPOM5Rr5j/0uGGgKrMy/ArAqsMgYc6e1dv2Zb2StnQpMBQgLC7NdunS5jNhytSzYdJgPf4zi/Yda06HBpY0Uv3LlSrT9PN/hw4f5+eef6du3L7t37+aZZ57h5MmT2nZeTH/3vJe2XfFVkBkHThtjTuX/JADfAqMK8N7rgAbGmLrGmADyLj5Y9PuT1tpEa20la20da20dYC3wp4ImnsPlsrz2VRQTV+xizpNtL7mgieeLjo5m4MCBNGvWjI0bNwIwaNAgmjZt6nAyEZHi54J70kzeLq7m/G8PmMta+6eLCM7FWptjjBkMLAN8gRnW2ihjzOvAemvtogu/g3iS9Kxcnv50A6lZuSx6pgNlgzT/ZlESHR1No0aNWLBgAXXr1iUmJoYKFSo4HUtEpFi7YEmz1lpjzBJr7WX9M9pauwRYctZj/zjPul0u5zPE/dKycvjb/22kbEl/pv61Jf6+Olm8KHC5XCxevJgxY8YQFxfH5s2bGTWqIDvJRUSkMBTk23azMeYGtycRj7T7WDL3frCGiqUDGN+3uQpaEZCdnY21lunTp/PPf/6TZ599lh07dlC6dGmno4mIyBnO+41rjPl9L9sN5A1Eu9MYs9EYs8kYs7Fw4omTNhw4Rf9pa3mobW3G36eC5u2Sk5P/mOx8y5YtPProo2zYsIH7778fP7/LuYZIRETc6UK/mX8DWgB3FlIW8SDfRMYxKmIrE/peT9drKzsdR67Qpk2b6NGjB926dWPBggVcf/31TkcSEZGLuFBJMwDW2j2FlEU8gLWW6T/tY+pPe/n40dY01xRPXmvPnj2MHz+eHj160Lt3b9asWUP9+vWdjiUiIgV0oZIWYowZer4nrbUT3JBHHJSd62Lkl9vYdSyZ+U+3o0b5IKcjyWVISkpi0KBBfPfddzz11FO0a9eOgIAAFTQRES9zoZLmC5Qmf4+aFG0pmTk8/t91lC7hx5wn2xIUoHOUvIm1lhUrVpCRkcHtt99Ojx49mD59uubTFBHxYhf6Jj5qrX290JKIY06nZjHwv+toULk0/7n7Onx91Mu9hbWWuXPnEh4eTkZGBv/6178wxvDYY485HU1ERK7QRc9Jk6LtcEI6D8/4ja5hIYzq1QgfFTSvkJ6eTmRkJK1ateK3337j1Vdf5bbbbtOE5yIiRciFStrNhZZCHBF5OJHH/7uexzrUZVCnUKfjSAGcOnWKyZMn895779G7d29atWrF+PHjnY4lIiJucN6SZq09VZhBpHAtj4rjpS+38sZdTbmj+TVOx5GLSE5OJjg4mBEjRpCbm8v3339P48aNnY4lIiJupLPDi6Ev1scSvmwnHz/amus1xIZHi4qKIjw8nO+++46YmBimTp1K3pS6IiJS1OkElmLmhx3HGbtsJ7Mfb6OC5uGmTJnCzTffTFhYGNu2bSMwMFAFTUSkGFFJK0aWRcUx7IstvPdACxpU0dAMnsblcrFw4UI6dOhATEwMffv2Zd++fYwePZry5cs7HU9ERAqZDncWE3PXxzJm6Q6mP9ySFrX0he9pNm/eTP/+/SlVqhQjRoygbt26+Pr6Oh1LREQcpJJWxFlree/7GOasj2XuUzdRL6S005EkX1JSElOnTqVz587Ur1+f999/n65du+qQpoiIADrcWaTluix/XxjJ0sg4Iv7WTgXNQyQlJTFq1ChCQ0PZuHEjZcqUoXz58nTr1k0FTURE/qA9aUVURnYuQz7bRGpWDnOebEtwoL/TkYq93bt3k5KSQqNGjcjNzWXdunXUrVvX6VgiIuKhtCetCDqckE6/qWsJ8PNh5iOtVdActm7dOu677z7at2/Pli1bCAwMJDw8XAVNREQuSCWtiIk+msQ9k3+he+MqTOp3AwF+2sROsNZy9OhRrLW88sordOjQgb179/LII484HU1ERLyEDncWIRsOnOKJWRv4xx2Nuev66k7HKZZycnL+mPA8NDSUiIgIli1b5nQsERHxQippRcRXW47w6qIoxvdtTpewyk7HKXastRhjeOihhzh69Cj//ve/6dWrl9OxRETEi6mkFQFLtx3l9a+383+Pt6FRtTJOxylW4uPjee+99/jyyy/ZuHEj06ZNIzhYAwWLiMiV0wlLXsxaywcr9/CPRVHMeLiVCloh+/jjj2nQoAGxsbF88cUX+Pn5qaCJiMhVoz1pXspay+tfb2fNnngWDW5PtbIlnY5ULGzdupWJEycyfvx4OnXqRGRkJNdcc43TsUREpAjSnjQv5HJZRs+PZP3+03z+RFsVtEIQHR1N79696dmzJ9deey0BAQGEhoaqoImIiNtoT5qXycpx8cKczcQlZTB7UBuNgeZGubm5LFy4kBtvvJGAgAD69OlDREQEgYGBTkcTEZFiQCXNi6Rk5jDov+spHejHp4+3IdBfE3C7Q2ZmJrNmzWLcuHGUK1eODz/8kBtuuIF69eo5HU1ERIoRlTQvkZGdyxOz1lO9fEneursZfr46Un21JSYmkpKSQkBAAF9//TVTpkyhc+fOmk9TREQcoW96L/D7Ic6yJf1V0NzgyJEjvPTSS4SGhrJgwQJCQkJYuHAhXbp0UUETERHHaE+ah0tIy2LQrPWUCwrg3f43qKBdRdnZ2fj4+NClSxd69erFxo0bqV27ttOxREREAJU0j3Y6NYv7pqyhY4NKvHJbY3x9tFfnali7di3h4eEAREREEBUVhb+/LsAQERHPot0yHup4cgb3T13DzY0q8887mqigXSWPPPII/fv3p1u3bnzyyScAKmgiIuKRVNI80LGkDPpNXUvvZtUY1auR03G8WnZ2Np988gl9+/bF5XLxyiuvsHv3bgYPHkypUqWcjiciInJeKmke5tDpNO6e/Av3tKjB87c0dDqOV/viiy+oV68eM2fOZODAgRhjqF+/Pn5+OsovIiKeT99WHuRYUgYPTv+VR9vX4fGOoU7H8UonTpxg+vTpDBs2jBo1ajBv3jxat27tdCwREZFLpj1pHiI+JZMBH/3KvS1qqKBdhoMHDzJ48GDCwsLYv38/qamp3HTTTSpoIiLitbQnzQOcTs3iwem/ckujKjx7cwOn43iVTZs2ERoayqFDhwgODmb79u1UrVrV6VgiIiJXTCXNYYlp2Tw4/Ve6hFXmxVvDnI7jFay1fP/994SHhxMZGcn8+fNp164d7dq1czqaiIjIVaPDnQ46npTBfVN+oV29iozoGabR7S8iNzeX1NRU9uzZw7PPPsv999/P3r17dUhTRESKJJU0hxxJSKff1LXc1uwaXr6tkQraBaSnp/Phhx8SFhbGtGnTqF+/PlFRUQwcOJASJUo4HU9ERMQtdLjTAceT88ZBe6htLZ7oVM/pOB4tKyuLRo0a0bRpU2bOnEmHDh0AVGpFRKTIU0krZMkZ2TwxawN/uf4aFbTzOHToEG+//Tbp6em8//77rF27VhcDiIhIsaPDnYUoOSObez74hWbVy2qg2vMYOnQo1113Hbm5uYwYMQJABU1ERIolt5Y0Y0xPY8xOY0yMMWbkOZ4faozZbozZaoz5zhhT2515nJSZk8vTn27kxtoVeOMvTfHRXJx/+OWXX3j99dcBuOOOO4iJiWHixInUqlXL4WQiIiLOcVtJM8b4Au8DvYDGQH9jTOOzVtsEtLTWXgfMA8LdlcdJObkuhs3dQqC/L6/d2cTpOB7ju+++o0OHDjz00EOEhIRgraVr165UqFDB6WgiIiKOc+eetNZAjLV2r7U2C/gcuOvMFay1P1hr0/IX1wI13JjHEVk5Lp7+dCOn07J4t/8NBPgV7yPM2dnZREREYK0lPj6ewYMHs2vXLv72t7/pYgAREZEzuPPCgepA7BnLh4A2F1j/MWCpG/MUuozsXB6duY7gQD9mPtK6WBe05ORkpk2bxltvvUXz5s3p2rUrffv2dTqWiIiIx/KIqzuNMQ8BLYHO53n+CeAJgJCQEFauXFl44S6Ty1omb84E4P76Jfhl9SqHEznj9OnTBAcHs3r1an788UdGjx7N9ddfz5YtW5yOJpcoJSXFK/7uyblp+3kvbbviy50l7TBQ84zlGvmP/X+MMbcALwOdrf1/7d15dJRFusfx7yNLAoJhkxkWkUAwIltAIUQQRBgGVGBQZLkoqCAjM4gboBzn3oteHQaIOmZEETdAkLAjogiK4ngVZA0QFiEISEAUIosSQra6f3STGzGBJiTp7uT3OafP6X673qqnu06SJ1X1vuXO5FWRc24qMBUgMjLS3XzzzYUebGHKznaMWbCFyyqmMuP+NoSWK+PvkIrdnj17iI2NZc6cOXz00UeMGzcOgFWrVhHo/Sd5U98FN/Vf8FLflV5FOf+2DmhkZuFmVh7oDyzJXcDMWgKvAT2dcz8WYSzFxjnHU4sT2Z9yimn3tS51CZpzjq1btxIdHU316tXZuXOntm0SEREpgCIbSXPOZZrZCGA5UAZ4yzm3zcyeAdY755YAk4BKwDzvovHvnHM9iyqmouac47kPdvDN4ZPMGBJNxfIBMZtc5JxzfPLJJ0ycOJHevXszfPhw9u3bR6VKlfwdmoiISNAq0izCOfch8OE5x/4r1/MuRdl+cXLO8fT72/lqz1HmDIuhUkjpSNBSU1O56aabSEtLY8yYMQwYMAAzU4ImIiJyiUpHJlHE0jKyGDN/CweOpTLvwRsJq1DO3yEVqdOnT/P222+TkZHBww8/zMsvv0x0dDSXXVZ6r14VEREpbPqreomysx0Px28iIyubd4e2LfEJ2oQJE6hfvz7Lly8nOtpzR5WYmBglaCIiIoVMI2mXaMLynaT8ks6sB6IJKVsyLxI4cOAAX331Ff369aNGjRqsWrWKxo0b+zssERGREk3DH5fgXyt38/G2H3j17utLZIKWmJjI4MGDf3VfsyFDhihBExERKQZK0gro1VV7mLvhAHP+HMOVlUP8HU6hcc6xbds2KYXA+wAAFg5JREFUAJYuXcq1115LUlISf//73/0cmYiISOmi6c4CWLAhmXdW72PxX9uVmAQtOzubJUuWMHHiRI4ePcrGjRt58skn/R2WiIhIqaWRtIv0VdJRnv1gO9Pub0PNK0L9Hc4lS09PxznH1KlTee6553jsscfYsWOHbqEhIiLiZ0rSLsLKHT/w0OxNTB7Yimt+V9nf4VySkydPMmnSJMLDw9myZQtDhw5l7dq19OnThzJlSt76OhERkWCjJM1H6/f9xKh5m3lj8A3c2LCGv8O5JBs3bqRBgwZs2rSJDz74gBYtWlC2bFm8uz6IiIhIANCaNB/s+uFnhs5Yz4v9omhZr6q/wymQXbt2ERsbS/fu3bnttttYt24d4eHh/g5LRERE8qGRtAs4kZrBQ+9u4slu19Ipsqa/w7loJ06coE+fPrRv355atWrRvn17ypcvrwRNREQkwGkk7TwysrJ5YMZ62oRXo1/rq/wdjs+ccyxfvpz09HR69OjB7bffzvTp07n88sv9HZqIiIj4SCNp5/GvT5MILV+Gp3s2CYr1Ws45Zs2aRVRUFGPGjAHAzLj33nuVoImIiAQZjaTl47NvfmTuugMs+uuNXHZZYCdop06dIjExkejoaBISEhg/fjzdu3cPisRSRERE8qaRtDwk/fgLo+dt5oW+LagVVsHf4eTr6NGjjBs3jvDwcF5//XUAJk2axK233qoETUREJMhpJO0cB4+fZtCbXzOm27XcGBGYt9o4ceIEYWFhjB07FoAvvviCyMhIP0clIiIihUlJWi6p6ZkMmbaOe9vVp+8NgXehwObNm5k4cSKff/45SUlJTJ06VSNmIiIiJZSmO70ysrIZPnMjTeuE8cBNDfwdzm+8+uqrdO/enebNm7Nt2zZCQ0OVoImIiJRgStK8nl26nWznGH9Hs4BIfrKysliwYAExMTEkJSUxYMAA9u7dyxNPPEFYWJi/wxMREZEipulO4M3/3cuXe1JY8OCNlCvj/7x106ZN9OvXj6pVq/LEE08QHh6u/TRFRERKmVKfpG3Yf4xXPkvivRHtCKtYzm9xHD9+nClTptC5c2ciIiJ4/fXX6dChQ0CM6omIiEjx8/+wkR99l5LKsBnrib2rBXWrVvRLDCdPnmTMmDE0bNiQxMREqlSpQtWqVenYsaMSNBERkVKs1I6k/ZyWwYMzNzD85oZ0urb49+TcuXMnqampXHfddZQpU4aNGzdy9dVXF3scIiIiEphK5UhaVrZj5OxNNKsTxpD2xbvR+OrVq+nduzcdOnQgMTGR0NBQxo8frwRNREREfqVUjqS9tHI3v5zJZMo9xbMnp3OOQ4cOUatWLZ555hluu+02Zs2aRcWK/pliFRGR4pGRkUFycjJpaWkFriMsLIwdO3YUYlRSFEJDQ6lbty7lyhXe+vZSl6Qt2/o9c9Z9x/sPtSekbNFeMZmRkUF8fDwTJ04kMjKS+fPns2zZsiJtU0REAkdycjKVK1emfv36BR4U+Pnnn6lcuXIhRyaFyTlHSkoKycnJhIcX3gxdqUrSVu9J4anFiUy7rzU1K4cWWTvZ2dlcdtllDBw4kKNHjxIbG0vXrl2LrD0REQlMaWlpl5SgSXAwM6pXr86RI0cKtd5Sk6TtOfILI+M38ULfFjSvW6VI2jhy5AhxcXEsWrSIhIQE3n77bS6//PIiaUtERIKDErTSoSj6uVRcOLA/5RR3v/E1j3a5hpsji+ZKzrfeeotrrrmGH3/8kUWLFlG2bFklaCIi4neVKlX6zbFx48YRGxv7m+NlypQhKiqKpk2b0qNHD44fP+5TG0ePHqVcuXJMmTLlvG1PmzaNESNG5LyeMWMGTZs2pVmzZrRs2TLPmPIzfvx4IiIiiIyMZPny5XmWuemmm4iKiiIqKoratWvzpz/9CfDcYSEmJoaQkJDftPniiy/SpEkTmjZtyoABA3LWE+7du5fo6GgiIiLo168f6enpPsdaUCU+STuems59b69jWIcG/Ed0vUKte9OmTQwaNIiUlBQ6derEjh07eO2112jUqFGhtiMiIlIcKlSoQEJCAomJiVSrVo3Jkyf7dN68efNo27Yts2fP9rmtZcuW8c9//pMVK1awdetW1qxZ4/O2h9u3byc+Pp5t27bx0Ucf8Ze//IWsrKzflPviiy9ISEggISGBmJgY7rjjDgCqVatGXFwco0aN+lX5gwcPEhcXx/r160lMTCQrK4v4+HgAnnjiCR599FGSkpKoWrUqb775ps+ftaBKdJKWne0YNW8z7SJqcF+7wlvIt23bNrp27UqPHj1o3rw5ISEhhIeH8/vf/77Q2hAREfGnmJgYDh486FPZ2bNn8/zzz3Pw4EGSk5N9Omf8+PHExsZSu3ZtAEJCQnjggQd8Ove9996jf//+OX9/IyIiWLt2bb7lT548yaeffpozklazZk1at26d55WYmZmZnD59mszMTFJTU6lduzbOOT799FP69OkDwODBg1m8eLFPsV6KEp2kxX26m2OpGfzn7dddcl1ZWVnMnTuX/fv3ExoaSv/+/dmzZw+jRo3KcyhZREQkWGVlZbFy5Up69uyZcywqKirPsgcOHOD777+nTZs29O3blzlz5vjURmJiItdff32e702aNClnmjL3Y+TIkYBnxOuqq67KKV+3bt3zJpSLFy+mc+fOXHHFFeeNqU6dOowaNYp69epRq1YtwsLC6Nq1KykpKVSpUoWyZcv61F5hKbEXDny+6wgz1+zng5E3Ub5swXPRtLQ0pk2bRmxsLDVr1uSVV14hKiqKhg0bFmK0IiJSGtR/8oNCr3PfP24rtLpOnz5NVFQUBw8epHHjxvzhD3/IeS8hISHPc+bMmUPfvn0B6N+/P/fffz+PP/54vm34ssB+9OjRjB49+iKjz9/s2bMZOnToBcsdO3aM9957j71791KlShXuuusuZs6cSbdu3QotlotRIpO0xIMneHROAq/dcz2/u6Jgt9o4duwYp06dIiQkhBUrVjBt2jTat29fyJGKiEhpUpCEqjjvk3Z2TVpqaip//OMfmTx5cs7oVX5mz57N4cOHmTVrFgCHDh1i9+7dNGrUiAoVKpCenk758uUB+Omnn6hRowYATZo0YcOGDdxyyy2/qXPSpEk59eXWoUMH4uLiqFOnDgcOHMg5npycTJ06dfKM7+jRo6xdu5ZFixZd8PN/8sknhIeHc+WVVwJwxx138NVXXzFw4ECOHz9OZmYmZcuWPW97hanETXf+dCqdIdPX8UyvJrSuX+2iz09OTubxxx+nYcOGvP/++1x55ZUsXLhQCZqIiJQaFStWJC4ujueff57MzMx8y+3atYtffvmFgwcPsm/fPvbt28fYsWNzLiDo2LEjM2fOBDyjdHPnzqVTp04AjB07ltGjR3P48GEA0tPTeeONNwDPSNrZBf+5H3FxcQD07NmT+Ph4zpw5w969e9m9ezdt2rTJM8b58+dz++23Exp64UGbevXqsWbNGlJTU3HOsXLlSho3boyZ0alTJ+bPnw/A9OnT6dWrly9f5SUpUUlaemY2909bx52t6nJ789oXd256OpmZmXTu3BnnHJs3b2b48OFFFKmIiEjxSE1NpW7dujmPF154AYBnn332V8fP1bJlS5o3b56TcOW1Jm327Nn07t37V8fuvPPOnHNeeuklFi5cSFRUFG3btuWuu+6iQ4cOANx6662MGDGCLl260KRJE1q1asXJkyd9+kxNmjShb9++XHfddXTr1o3JkydTpkyZnHoPHTqUUzY+Pp4BAwb86vzDhw/nfBdnv4eTJ08SHR1Nnz59aNWqFc2aNSM7O5thw4YBMGHCBF544QUiIiJISUlhyJAhPsV6Kcw5V+SNFKbIyEj3zTff5Pne3z/cwbdHfuH1QTf4fFO5L7/8kgkTJlCuXDkWLFiQM5QpRWPVqlXcfPPN/g5DCkB9F9zUf/6xY8cOGjdufEl1aFuo4JFXf5vZBufcDQWpr8SMpL2XcJBlid/zjzub+5yg3XPPPQwaNIju3bvnDMcqQRMREZFAUCIykgM/pTJuyTbeGRJNjUoh+ZZLT0/n3XffZenSpcydO5enn36aevXqKTETERGRgBP0I2lpGVn8+Z0NjLilEU3r5H+n4rlz59KwYUNmzZrFgw8+iJnRoEEDJWgiIiISkII+SXvxk1387ooQ7m9X/zfv/fDDDzz77LOcOXOGevXqsXjxYj7++GO6dOmiDW9FRKRYBNvabymYoujnoE7S5q0/wPLEw0y6q8Wvkq79+/czfPhwGjduzKFDh0hNTaVt27b53tlYRESkKISGhpKSkqJErYRzzpGSkuLTbT4uRtDO9W3Y/xPjl+1k1tD/X4e2YcMGIiIiOHToENWrV2fnzp3UrFnTz5GKiEhpVbduXZKTkzly5EiB60hLSyv0P/5S+EJDQ/O8lcmlKNIkzcy6AS8BZYA3nHP/OOf9EGAGcD2QAvRzzu27UL0/nUrnkTkJ/E+vplz7+8qsWLGCCRMmsGvXLhYuXEhMTAwxMTGF/4FEREQuQrly5QgPD7+kOlatWkXLli0LKSIJJkU23WlmZYDJQHfgOmCAmZ270/kQ4JhzLgJ4EZjgS91/W7yVTo2q07FB5ZxNzgcPHsyePXto3bp1YX4MEREREb8oypG0NkCSc+5bADOLB3oB23OV6QWM8z6fD7xsZubOM3n/S3o2nyx4h4yEJdR45BFGjhzJ5s2bdSGAiIiIlChFmaTVAQ7kep0MROdXxjmXaWYngOrA0fwq/f7APjpes4vnZs7kxhtvBFCCJiIiIiVOUFw4YGbDgGHel2dWrViW2G7FMn+GJAVXg/Mk4RLQ1HfBTf0XvNR3wS2yoCcWZZJ2ELgq1+u63mN5lUk2s7JAGJ4LCH7FOTcVmApgZusLugeW+J/6L3ip74Kb+i94qe+Cm5mtL+i5RXmftHVAIzMLN7PyQH9gyTlllgCDvc/7AJ+ebz2aiIiISGlRZCNp3jVmI4DleG7B8ZZzbpuZPQOsd84tAd4E3jGzJOAnPImciIiISKlXpGvSnHMfAh+ec+y/cj1PA+66yGqnFkJo4j/qv+Clvgtu6r/gpb4LbgXuP9PsooiIiEjgCeq9O0VERERKqoBN0sysm5l9Y2ZJZvZkHu+HmNkc7/tfm1n94o9S8uJD3z1mZtvNbIuZrTSzq/0Rp+TtQv2Xq9ydZubMTFedBRBf+s/M+np/BreZ2bvFHaPkzYffnfXM7DMz2+T9/XmrP+KU3zKzt8zsRzNLzOd9M7M4b99uMbNWvtQbkElaUW4pJUXLx77bBNzgnGuOZ6eJicUbpeTHx/7DzCoDDwNfF2+Ecj6+9J+ZNQLGAu2cc02AR4o9UPkNH3/2/gbMdc61xHOh3SvFG6WcxzSg23ne7w408j6GAa/6UmlAJmnk2lLKOZcOnN1SKrdewHTv8/lAZ9PWA4Hggn3nnPvMOZfqfbkGzz30JDD48rMH8D94/jFKK87g5IJ86b8HgMnOuWMAzrkfizlGyZsvfeeAK7zPw4BDxRifnIdz7t947lKRn17ADOexBqhiZrUuVG+gJml5bSlVJ78yzrlM4OyWUuJfvvRdbkMAbR8ROC7Yf95h+quccx8UZ2DiE19+/q4BrjGzL81sjZmd779/KT6+9N044G4zS8Zz54SHiic0KQQX+7cRCJJtoaRkMrO7gRuAjv6ORXxjZpcBLwD3+jkUKbiyeKZcbsYziv1vM2vmnDvu16jEFwOAac65580sBs99Rps657L9HZgUjUAdSbuYLaU435ZSUux86TvMrAvwFNDTOXemmGKTC7tQ/1UGmgKrzGwf0BZYoosHAoYvP3/JwBLnXIZzbi+wC0/SJv7lS98NAeYCOOdWA6F49vWUwOfT38ZzBWqSpi2lgtcF+87MWgKv4UnQtB4msJy3/5xzJ5xzNZxz9Z1z9fGsKezpnCvw3nRSqHz53bkYzygaZlYDz/Tnt8UZpOTJl777DugMYGaN8SRpR4o1SimoJcAg71WebYETzrnvL3RSQE53akup4OVj300CKgHzvNd6fOec6+m3oCWHj/0nAcrH/lsOdDWz7UAWMNo5p1kIP/Ox7x4HXjezR/FcRHCvBicCg5nNxvPPTw3vmsH/BsoBOOem4FlDeCuQBKQC9/lUr/pXREREJPAE6nSniIiISKmmJE1EREQkAClJExEREQlAStJEREREApCSNBEREZEApCRNRAqVmWWZWUKuR/3zlK1vZomF0OYqM/vGzDZ7tzuKLEAdD5rZIO/ze82sdq733shro/lLjHOdmUX5cM4jZlbxUtsWkeCjJE1ECttp51xUrse+Ymp3oHOuBTAdz734Lopzbopzbob35b1A7VzvDXXObS+UKP8/zlfwLc5HACVpIqWQkjQRKXLeEbMvzGyj93FjHmWamNla7+jbFjNr5D1+d67jr5lZmQs0928gwntuZzPbZGZbzewtMwvxHv+HmW33thPrPTbOzEaZWR88e8rO8rZZwTsCdoN3tC0nsfKOuL1cwDhXk2uDZTN71czWm9k2M3vae2wknmTxMzP7zHusq5mt9n6P88ys0gXaEZEgpSRNRApbhVxTnYu8x34E/uCcawX0A+LyOO9B4CXnXBSeJCnZu/VNP6Cd93gWMPAC7fcAtppZKDAN6Oeca4Znh5XhZlYd6A00cc41B57NfbJzbj6wHs+IV5Rz7nSutxd4zz2rHxBfwDi74dmi6aynnHM3AM2BjmbW3DkXBxwCOjnnOnm3cfob0MX7Xa4HHrtAOyISpAJyWygRCWqnvYlKbuWAl71rsLLw7Bd5rtXAU2ZWF1jonNttZp2B64F13i3EKuBJ+PIyy8xOA/uAh4BIYK9zbpf3/enAX4GXgTTgTTNbCiz19YM5546Y2bfevfd2A9cCX3rrvZg4y+PZGi3399TXzIbh+b1cC7gO2HLOuW29x7/0tlMez/cmIiWQkjQRKQ6PAj8ALfCM4KedW8A5966ZfQ3cBnxoZn8GDJjunBvrQxsDc2/0bmbV8irk3SOxDZ6NqvsAI4BbLuKzxAN9gZ3AIuecM0/G5HOcwAY869H+BdxhZuHAKKC1c+6YmU3Ds3n2uQz42Dk34CLiFZEgpelOESkOYcD3zrls4B48G0j/ipk1AL71TvG9h2fabyXQx8xqestUM7OrfWzzG6C+mUV4X98DfO5dwxXmnPsQT/LYIo9zfwYq51PvIqAXMABPwsbFxundFPs/gbZmdi1wBXAKOGFmvwO65xPLGqDd2c9kZpebWV6jkiJSAihJE5Hi8Aow2Mw245kiPJVHmb5AopklAE2BGd4rKv8GrDCzLcDHeKYCL8g5lwbcB8wzs61ANjAFT8Kz1Fvf/5L3mq5pwJSzFw6cU+8xYAdwtXNurffYRcfpXev2PDDaObcZ2IRndO5dPFOoZ00FPjKzz5xzR/BceTrb285qPN+niJRA5vmHTkREREQCiUbSRERERAKQkjQRERGRAKQkTURERCQAKUkTERERCUBK0kREREQCkJI0ERERkQCkJE1EREQkAClJExEREQlA/weh5IYttN5LMQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "llr = np.concatenate([llr_in, llr_ood])\n",
    "roc_data = {'LLR': {'scores': -llr, 'labels': y_test_sample}}\n",
    "plot_roc(roc_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect outliers\n",
    "\n",
    "We follow the same procedure with the outlier detector. First we need to set an outlier threshold with `infer_threshold`. We need to pass a batch of instances and specify what percentage of those we consider to be normal via `threshold_perc`. Let's assume we have a  small batch of data with roughly $30$% outliers but we don't know exactly which ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 250)\n"
     ]
    }
   ],
   "source": [
    "n, frac_outlier = 1000, .3\n",
    "perc_outlier = 100 * frac_outlier\n",
    "n_sample_in, n_sample_ood = int(n * (1 - frac_outlier)), int(n * frac_outlier)\n",
    "idx_in, idx_ood = np.where(y_val == 0)[0], np.where(y_val == 1)[0]\n",
    "n_in, n_ood = idx_in.shape[0], idx_ood.shape[0]\n",
    "sample_in = np.random.choice(n_in, size=n_sample_in, replace=False)\n",
    "sample_ood = np.random.choice(n_ood, size=n_sample_ood, replace=False)\n",
    "X_thr_in, X_thr_ood = X_val[idx_in[sample_in]], X_val[idx_ood[sample_ood]]\n",
    "X_threshold = np.concatenate([X_thr_in, X_thr_ood])\n",
    "print(X_threshold.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New threshold: -0.0506670355796814\n"
     ]
    }
   ],
   "source": [
    "od.infer_threshold(X_threshold, threshold_perc=perc_outlier, batch_size=100)\n",
    "print('New threshold: {}'.format(od.threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the outlier detector with updated threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_detector(od, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let'spredict outliers on a sample of the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "od_preds = od.predict(X_test_sample, batch_size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display results\n",
    "\n",
    "F1 score, accuracy, precision, recall and confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score: 0.722 -- Accuracy: 0.646 -- Precision: 0.595 -- Recall: 0.919\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAD4CAYAAAAn3bdmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXwV1fnH8c+XhCUgEMACCihYqda9SgWrWC0VwQ2tFsUFVH5gBVut9Ve1tqJY159bqYLSioIK7lawKKQI7rIJgriUiAsgi8oSBAWSPL8/5gQvmNxMyELm8rz7mted+8yZmXNreHJyzrlzZGY455xLhjo7ugLOOefi86TtnHMJ4knbOecSxJO2c84liCdt55xLkOwauIdPT3HOxaXKXmDzl4ti55y6u+5V6fvVtJpI2ty+x7k1cRuXEFd89ggAa/ocu4Nr4mqT3HFTd3QVEsG7R5xzmaW4KP5WDkmXSnpX0gJJl4VYc0l5khaG12YhLknDJOVLmifp0JTr9AvlF0rqlxI/TNL8cM4wSeW2/D1pO+cyS1Fh/C0NSQcAA4DDgYOBkyTtDVwFTDGzjsCU8B6gJ9AxbAOBEeE6zYEhQOdwrSEliT6UGZByXo/yPp4nbedcRjErjr2V48fAdDPbYGaFwMvAr4BewOhQZjRwatjvBYyxyFtArqTdgOOBPDNbZWargTygRzjWxMzesuir6WNSrlUmT9rOucxSXBx7kzRQ0qyUbWDKld4FukpqIakhcALQDmhlZstCmeVAq7DfBliccv6SEEsXX1JKPK0aGYh0zrkaU34L+ruiZiOBkWUce1/SrcBkYD0wFyjapoxJqtEZct7Sds5lliociDSzB8zsMDM7GlgN/BdYEbo2CK8rQ/GlRC3xEm1DLF28bSnxtDxpO+cyixXH38ohqWV43YOoP3ssMB4omQHSD3gu7I8H+oZZJF2AtaEbZRLQXVKzMADZHZgUjhVI6hJmjfRNuVaZvHvEOZdRrJxZIRX0tKQWwGZgsJmtkXQL8ISk/sCnQO9QdiJRv3c+sAG4AMDMVkm6AZgZyg01s1VhfxDwEJADvBC2tDxpO+cyS3H8Pu3ymFnXUmJfAd1KiRswuIzrjAJGlRKfBRxQkTp50nbOZZYKDEQmkSdt51xmiTHAmGSetJ1zmcVb2s45lyBVOxBZ63jSds5lliociKyNPGk75zKKmfdpO+dccniftnPOJYh3jzjnXIJ4S9s55xKkaPOOrkG18qTtnMss3j3inHMJ4t0jzjmXIN7Sds65BPGk7ZxzyWE+EOmccwnifdrOOZcgGd494mtEOucyS9WuEfl7SQskvStpnKQGkjpImi4pX9LjkuqFsvXD+/xwvH3Kda4O8Q8lHZ8S7xFi+ZKuivPxPGk75zJLcXH8LQ1JbYDfAZ3M7AAgCzgLuBW4y8z2JlqhvX84pT+wOsTvCuWQtF84b3+gBzBcUpakLOBeoCewH9AnlE3Lk7ZzLrNUYUubqAs5R1I20BBYBvwCeCocHw2cGvZ7hfeE493CKuu9gMfMbKOZfUy08O/hYcs3s0Vmtgl4LJRNy5O2cy6zFBbG3iQNlDQrZRtYchkzWwrcDnxGlKzXArOBNWZWstLCEqBN2G8DLA7nFobyLVLj25xTVjwtH4h0zmWWCsweMbORwMjSjklqRtTy7QCsAZ4k6t7YoTxpO+cyS9XNHvkl8LGZfQEg6RngSCBXUnZoTbcFlobyS4F2wJLQndIU+ColXiL1nLLiZfLuEedcZqm6Pu3PgC6SGoa+6W7Ae8BU4IxQph/wXNgfH94Tjr9kZhbiZ4XZJR2AjsAMYCbQMcxGqUc0WDm+vEp5S9s5l1mqqKVtZtMlPQW8DRQCc4i6Uv4NPCbpryH2QDjlAeBhSfnAKqIkjJktkPQEUcIvBAZbWBNN0iXAJKKZKaPMbEF59fKk7ZzLLFX4jUgzGwIM2Sa8iGjmx7ZlvwV+XcZ1bgRuLCU+EZhYkTp50nbOZZbCwvLLJJgnbedcZjHb0TWoVp60nXOZJcOfPeJJ2zmXWTxpO+dcgvijWZ1zLkGKinZ0DaqVJ23nXGbx7hHnnEsQT9rOOZcg3qftnHPJYcU+T9s555LDu0eccy5BfPaIc84liLe0nXMuQTxp75yy6tflrCf/TFa9bOpkZ/HfiTN4485ntipzzLXnsMcR0eLJ2Tn1aNiiCfcceFGl7tugaSNOGn4JTdv+gLVLvmDCoL+zce2GLcdbH7QXZ/9rCM9fcg//nTizUvdy20cNG5Ez8H/JatsBMDbcfxtFC9/bcjz7sCPJ6X0BFBtWXMQ3Y+6h6MN3K3fPRo1peOm11Nm1NcVfLmfD367H1n+95XjWXvuwy9B72TBsKJtnvFKpeyWePzBq51S0cTNPnHUTmzdspE52Fn2e/gsfT32HZXM+2lJm2tBHt+z/5PzjaLl/+9jXb9flx+z/6668+Ietl6c7fPDJfPb6e8wYPoHDB51M50En88rNjwOgOuLoq8/kk1fmV+7DuUrJ6fdbCt+ZwYa7r4OsbFS//lbHC9+dzbrZrwNQZ4+9aPS7Iay7ol8pV/q+7B8fTL2f92DDfbduFa/f62wK332bjePHUf+UPtQ/5Wy+HRd+dlSHBmcPpHCe/xIHMr6l7cuNpbF5w0YA6mRnUSc7O+0v8H1POYIPxr+55f1PLzqRcycMpd+km/jZ5b+Kfc+9jzuMBU+9CsCCp15l7+6dthz7yQXd+e8LM9nwVUEFP4mrMjmNyNr3IDZNDc+tLyrENqzfuszGb7fsqn4D4LsfnPonnckufx1B41v/SYMzzo9927qH/YxNr0wCYNMrk6jb6cjvrtnjNDZPfxUrWFPhj5ORii3+lkBpk7akQ9NtNVXJHUV1RN8XbmTQnOF8+tp8ls/9qNRyTdq0oOkeLfns9WiloD27HkBuh1Y8cvK1jO5xDa0O7EDbw/eJdc+GuzZh/croH9/6lWtouGsTAHZp1YyOx3di7sNTquCTue2V1bI1VrCGhr+5kl1uHknOgCugfoPvlavb6Sga3z6aRn+8mQ333wZA9oGdqNO6LV//+WLWXTWArA4/Imvfg2Ldt07T5tiaVQDYmlXUadocADXblbo/7cqm/zyX7vSdS1FR/C0NSftImpuyFUi6TFJzSXmSFobXZqG8JA2TlC9pXmqOlNQvlF8oqV9K/DBJ88M5w8JalGmV1z1yR5pjBvyijA87EBgIcP/995dXh1rLio0xPa+hfpOG9Bp5Gbv+qC1f/nfJ98rte8oR/PffM7ZM6m9/9IG073ogfV+IVheq26gBuR1as2TGh5zz3HVk1atL3UYNaJDbaEuZV25+LG23x7HXncsrNz+W8f11tV5WFlkdfsQ3D/2doo/eJ6fvJTQ4pQ/fPvngVsU2z3qNzbNeI2vfg2jw6wtZf9MVZB/UiboHdSL75n9EhRrkkNW6LUUfzGOXG4aj7LrQIAft0pjGocw340aW2u1h4ecgp+9gvhl7v/9cpLCqWyPyQ+AQAElZRCulPwtcBUwxs1skXRXeXwn0JFq0tyPQGRgBdJbUnGjJsk5EeXO2pPFmtjqUGQBMJ1p2rAfwQrp6pU3aZnbsdn7YkUQLYALY7X9N9sDIxoINLH7zPdofc1CpSXufk7sw5S+jt7yXxPThE5j36EvfK/tor+uAsvu0N3xZQKOWuaxfuYZGLXPZ8GXUFdL6wA6cdM8lAOQ0b8xexx5McWEx+ZNnV9XHdDEUf/UFtuoLij56H4BN01+mQa+zyyxf9ME86rTcDTVuAhLfPjeWTVMmfK/c138ZBJTdp128dhXKjVrbym2OFawGogHIRr+7FgA1bkr2IZ2huIjNs16vks+bSNXT7dEN+MjMPpXUCzgmxEcD04iSdi9gTFiB/S1JuZJ2C2XzzGwVgKQ8oIekaUATM3srxMcAp1KZpJ1K0gHAfsCWvwXNbEzc85Mmp3ljiguL2Fiwgez6ddmz64HMGPH9f2zNf7gbDZo24vPZC7fEPn55HkddcQbvP/s6mzdsZJdWzSguLIrVF/1R3tvsf0ZXZgyfwP5ndCU/L0rK/zjq8i1letwxkEVT5njC3gFs7WqKv1pJnd3aUbxsMXUPOJSiJZ9sVaZOq90pXvE5AFntO6K6dbF1BRS+M5MGvS9k02t5sPFb1GzXqE88Rl/05tlvUO/o49k4fhz1jj6ezbPfAGDdpd/9wmj4myvZ/PabO3fChgo9eyS1VyAYGRqd2zoLGBf2W5nZsrC/HGgV9tsAi1POWRJi6eJLSomnFStpSxpC9NtiP6ImfE/gNSBjk3ajlrn0vPMi6mTVQXXEh89PZ9GUuRx5+eksn/8xH+W9DYQByAlvbXXup6++S4uObTj7X9cBsHn9t/z7shEQI2lPHz6Bk0f8lgPP/DkFS79kwsV/r/LP5irnm4eG0fCSa1B2NsUrlrHh/lup98uTAdj0nwnUPfxo6h19PBQWYps2sn7YUAAK589iU5s9aTz0XgDs22/YcO9NsZL2xvHjaHjpEOodcwLFX65gw9+ur74PmHQVaGlv0ytQKkn1gFOAq0s53yTVaN+ULEZfmKT5wMHAHDM7WFIr4BEzOy7GPez2Pc6tZDVdJrnis0cAWNNnu3rfXIbKHTcVoNyBuPKsv/as2Em00dDHyr1f6A4ZbGbdw/sPgWPMbFno/phmZvtIuj/sj0stV7KZ2UUhfj9Rl8o0YKqZ7RvifVLLlSXulL9vzKwYKJTUBFgJtIt5rnPO1Rwrjr/F04fvukYAxgMlM0D6Ac+lxPuGWSRdgLWhG2US0F1SszDTpDswKRwrkNQlzBrpm3KtMsXt054lKRf4BzAb+Bp4M/0pzjm3A1ThQKSkRsBxQGrr9xbgCUn9gU+B3iE+ETgByAc2ABcAmNkqSTcAJdOAhpYMSgKDgIeAHKIByLSDkBAzaZvZoLB7n6QXiUY858U51znnalJVTfkDMLP1QIttYl8RzSbZtqwBg8u4zihgVCnxWcABFalTRWaPHAS0LzlH0t5m9kzak5xzrqYl9JuOccWdPTIKOAhYAJT8GjPAk7ZzrnbxpA1AFzPbr1pr4pxzVSHDF0GIO3vkTUmetJ1ztZ4VW+wtieK2tMcQJe7lwEaiuZRmZvGeduOcczUlock4rrhJ+wHgPGA+3/VpO+dc7ZPhz9OOm7S/MLPx1VoT55yrCt7SBmCOpLHABKLuEQB8yp9zrtbxpA1E39bZSPT1yxI+5c85V+tY0U7ePRIe/j3PzO6qgfo451zlZHhLu9wpf2ZWRPTAFOecq/V8yl/kdUn3AI8DW1YxNbO3q6VWzjm3vRKajOOKm7QPCa9DU2JlrhHpnHM7TGZ3acd+yp8/rd45lwhWmNlZO9bX2CU1lXSnpFlhu0NS0+qunHPOVVhxBbYEivvskVHAOqKHffcGCoAHq6tSzjm3vXwgMvJDMzs95f31kuZWR4Wcc65SEtqCjiv2GpGSjip5I+lI4JvqqZJzzm2/qmxpS8qV9JSkDyS9L+kISc0l5UlaGF6bhbKSNExSvqR5kg5NuU6/UH6hpH4p8cMkzQ/nDAtrRaYVN2lfDNwr6RNJnwD3sPWaac45VztUbZ/234AXw4rpBwPvA1cBU8ysIzAlvAfoCXQM20BgBICk5sAQoDNwODCkJNGHMgNSzutRXoXido+8D9wG/BDIBdYCpwK+TqRzrlaxwqq5TphscTRwPoCZbQI2SeoFHBOKjQamAVcCvYAxYa3It0IrfbdQNq9kMV9JeUAPSdOI1tt9K8THEOXVtIv7xm1pPwecDHwLLCVajX192jOcc24HsOL4m6SBKbPiZkkamHKpDsAXwIOS5kj6Z1idvZWZLQtllgOtwn4bYHHK+UtCLF18SSnxtOK2tNuaWbnNduec2+EqMBBpZiOBkWUczgYOBX5rZtMl/Y3vukJKzjdJNToNJW5L+w1JB1ZrTZxzrgpUpKVdjiXAEjObHt4/RZTEV4RuD8LrynB8KdAu5fy2IZYu3raUeFpxk/ZRwGxJH4ZR0fmSvD/bOVfrVFXSNrPlwGJJ+4RQN+A9YDxQMgOkH1H3MSHeN8wi6QKsDd0ok4DukpqFAcjuwKRwrEBSlzBrpG/KtcoUt3ukZ8xyzjm3Q1lRubPmKuK3wKOS6gGLgAuIGrtPSOoPfEr0hUOAicAJQD6wIZTFzFZJugGYGcoNLRmUBAYBDxGtWfAC5QxCQvxnj3wap5xzzu1oMbo94l/LbC7QqZRD3Uopa8DgMq4ziuib5dvGZwEHVKROcVvazjmXCFZcpS3tWseTtnMuo1RlS7s28qTtnMsoZt7Sds65xPCWtnPOJUhx1c4eqXU8aTvnMooPRDrnXIJ40nbOuQSxZC5IE5snbedcRvGWtnPOJYhP+XPOuQQp8tkjzjmXHN7Sds65BPE+beecSxCfPeKccwniLW3nnEuQouK4C3Ilkydt51xGyfTukcz+leSc2+kUm2Jv5ZH0SVgTd66kWSHWXFKepIXhtVmIS9IwSflhLd1DU67TL5RfKKlfSvywcP38cG65lfKk7ZzLKGaKvcV0rJkdYmYly45dBUwxs47AlPAeorV0O4ZtIDACoiQPDAE6A4cDQ0oSfSgzIOW8HuVVpka6R6747JGauI1LmNxxU3d0FVwGqoHukV7AMWF/NDANuDLEx4S1It+SlCtpt1A2r2QxX0l5QA9J04AmZvZWiI8BTqWcxX29pe2cyygV6R6RNFDSrJRt4DaXM2CypNkpx1qZ2bKwvxxoFfbbAItTzl0SYuniS0qJp1UjLe3seuXWw+1ECjctBWDzl4t2cE1cbVJ3172q5DoVmT1iZiOBkWmKHGVmSyW1BPIkfbDN+SapRoc+vaXtnMsoVoGt3GuZLQ2vK4FnifqkV4RuD8LrylB8KdAu5fS2IZYu3raUeFqetJ1zGaWqZo9IaiSpcck+0B14FxgPlMwA6Qc8F/bHA33DLJIuwNrQjTIJ6C6pWRiA7A5MCscKJHUJs0b6plyrTD5P2zmXUarwgVGtgGfDLLxsYKyZvShpJvCEpP7Ap0DvUH4icAKQD2wALojqY6sk3QDMDOWGlgxKAoOAh4AcogHItIOQALLqH2o179N2qbxP25Um9GlXOuO+2vqM2Emt6/KnEvedd29pO+cyilU+79dqnrSdcxml0J+n7ZxzyeEtbeecS5DiHV2BauZJ2zmXUbyl7ZxzCeItbeecS5Aib2k751xyZPhqY560nXOZpdhb2s45lxwZvtqYJ23nXGbxgUjnnEuQ4vKXWUw0T9rOuYxStKMrUM08aTvnMorPHnHOuQTx2SPOOZcgPnvEOecSJNO7R3yNSOdcRimuwBaHpCxJcyQ9H953kDRdUr6kxyXVC/H64X1+ON4+5RpXh/iHko5PifcIsXxJV8Wpjydt51xGKVL8LaZLgfdT3t8K3GVmewOrgf4h3h9YHeJ3hXJI2g84C9gf6AEMD78IsoB7gZ7AfkCfUDYtT9rOuYxSlS1tSW2BE4F/hvcCfgE8FYqMBk4N+73Ce8LxbqF8L+AxM9toZh8TLfx7eNjyzWyRmW0CHgtl0/Kk7ZzLKBVJ2pIGSpqVsg3c5nJ3A3/kuxzfAlhjZoXh/RKgZOXyNsBigHB8bSi/Jb7NOWXF0/KBSOdcRqnIEpFmNhIYWdoxSScBK81stqRjqqRyVcCTtnMuo1Ths0eOBE6RdALQAGgC/A3IlZQdWtNtgaWh/FKgHbBEUjbQFPgqJV4i9Zyy4mXy7hHnXEYpqsCWjpldbWZtzaw90UDiS2Z2DjAVOCMU6wc8F/bHh/eE4y+ZmYX4WWF2SQegIzADmAl0DLNR6oV7jC/v83lL2zmXUWpgnvaVwGOS/grMAR4I8QeAhyXlA6uIkjBmtkDSE8B7QCEw2MyKACRdAkwCsoBRZragvJsr+kVQrSy7Xrl9624nUrgp+gtw85eLdnBNXG1Sd9e9gMp/B/2uPc6NndR+/9kjifsqjre0nXMZxZ+n7ZxzCeLPHnHOuQTJ9GePeNJ2zmUUXwTBOecSpDjDO0g8aTvnMooPRDrnXIJkdjvbk7ZzLsN4S9s55xKkUJnd1vak7ZzLKJmdsj1pO+cyjHePOOdcgviUP+ecS5DMTtmetJ1zGca7R5xzLkGKMryt7UnbOZdRvKXtnHMJYhne0vY1Ip1zGaW4Als6khpImiHpHUkLJF0f4h0kTZeUL+nxsL4jYQ3Ix0N8uqT2Kde6OsQ/lHR8SrxHiOVLuirO5/OkncalvxvAO3NfYu6cKTzy8L3Ur19/q+P16tVj7KMj+OC913jjtQnsuWfbSt+zfft2vPHaBD547zXGPjqCunXrAjBwwHnMefs/zJo5mZenPsuPf9yx0vdy2+fhJ/7Fqef+hl7nXMTDjz/7vePPT3qJ0/pezGnnXcw5F13OBwsrv6zapk2b+MNfbqZn7wvpM+Ayli5bsdXxZctX8tNfnsaDY5+q9L2SrhiLvZVjI/ALMzsYOAToIakLcCtwl5ntDawG+ofy/YHVIX5XKIek/YjWi9wf6AEMl5QlKQu4F+gJ7Af0CWXT8qRdht13b80lgy+kc5cTOOQn3cjKyuLM3r22KnPhBX1YvXot++53FHcP+wc333RN7Ov3Pa831/7l8u/Fb77pGu4e9g/23e8oVq9ey4UX9AFg3GPP8pNDf0mnn3bn/+4Yzu23DancB3TbZeGiT3h6/IuM++fdPD16OC+/MYPPlny+VZk2u7fmoXtu49mHR/Cb8/tw/W3DYl9/6bIVnH/JH78Xf+b5yTRpvAsvPDGK8848lTuHj9rq+G1/H0nXLp2270NlGKvAlvY6ka/D27phM+AXQMlvx9HAqWG/V3hPON5NkkL8MTPbaGYfA/nA4WHLN7NFZrYJeCyUTcuTdhrZ2dnk5DQgKyuLhjk5LFu2fKvjp5zcnYcffhKAp5/+N7849igA6tSpw603/5k33/g3b8/OY8D/nBv7nscecyRPP/1vAB5++El6nRL9JbVu3ddbyjRq1JAaWJDZlWLRJ4s5cP99yGnQgOzsLDodciD/efn1rcr85MD9aNqkMQAH7b8vK1Z+ueXYhEkvcdb/XMrp/QZz/W3DKCqK98j+l159k14n/BKA7sd0ZfrsuVt+Bqa88gZtdmvNDzvsWRUfMfEKsdibpIGSZqVsA1OvFVrEc4GVQB7wEbDGzApDkSVAycrlbYDFAOH4WqBFanybc8qKp1Vu0pZUR9LPyiuXaT7/fDl33nUfH380gyWfzWFtQQF5/3llqzK7t2nN4tDKKioqYu3aAlq0aMaFF/RhbcE6jvjZiXQ54kT69z+b9u3blXvPFi2asWbN2i3/kJcsXcbubVpvOX7xb/rx4fuvc8tNf+ayy6+twk/r4tp7rz15+50FrFlbwDfffsurb85k+Yovyiz/zPOTOCq0gD/65DNenPIyD993B0+Pvpc6derw/OSpse678ouvaN1yVwCys7PYpVFD1qwtYMOGbxj1yJMMuvCcyn+4DGEV+Z/ZSDPrlLKN3OpaZkVmdgjQlqhlvO8O+VApyp09YmbFku4FfhL3ouG31UCA+++/f/trtwPl5jbllJOPZ+8fdWHNmgIef+x+zj77V4wd+0y55x533M858MAf86tfnQhA0yaN6bh3BwoKvmbypMcBaN4sl3r16nLKKT0AOP+C37Fsm37KbY24bzQj7hvNWWedyp+uvpQL+19WyU/pKuqH7ffgwnN+zcDfX0NOgwbs03Ev6tQpve0zY/Y7PPP8ZB4ecTsA02fN5b0P8jmr/6UAbNy4kebNcgH43dVDWfr5CjYXbmbZii84vd9gAM7t3YvTTuxeZn3uHfUI5515Gg0b5lTlx0y06pjyZ2ZrJE0FjgByJWWH1nRbYGkothRoByyRlA00Bb5KiZdIPaeseJniTvmbIul04BmL8Xd5+G1V8hvLBl1yfczb1B7dunXl408+48svVwHw7L9e4IgunbZK2p8vXU67truzdOkysrKyaNq0CV99tRoJLrvsz0zOe/l71+300+gfYN/zetO+fVuG3nDnVsdzc5uSlZVFUVERbdvsxudLl3/vGo8//hz3/v3mqvy4rgJOP/l4Tj856ra6+76HtrSAU32Y/zHX3nI3991xA7lNmwBgZpzS85f8/uILvld+2M3RX05Ll63gmhvv4KF7btvqeMsftGD5yi9p3fIHFBYW8fX6DeQ2bcL8BR+SN/U17hz+AOu+Xo8k6terx9lnnFLVHzsxqmrKn6QfAJtDws4BjiMaXJwKnEHUB90PeC6cMj68fzMcf8nMTNJ4YKykO4HdgY7ADEBAR0kdiJL1WcDZ5dUrbp/2RcCTwCZJBZLWSSqIeW4iLf5sKZ07H0pOTgMAfnHsUXzwwcKtykx4fjLnnfdrAE4//USmTov6NidPfpmLLupLdnb0O7Fjx71it4SmvfwGp58etdDPO+/XjJ8wGYC99+6wpcyJJ/yShfkfV+LTucr4avUaIJqxMeXl1znhuGO2Or5s+Uou+9MN3Hzt/9J+j+9mFHXpdAh5017bcv7agnV8vjz9X1cljj2qC89N/A8Ak6e9SufDDkYSY0bczuSnRzP56dGc2/tUBvQ9c6dO2FB1U/6A3YCpkuYBM4E8M3seuBK4XFI+UZ/1A6H8A0CLEL8cuArAzBYATwDvAS8Cg0O3SyFwCTAJeB94IpRNK1ZL28waxymXSWbMnMMzz/ybmTMmUVhYyNy5C/jHPx/luiFXMGv2Ozz/fB6jHnyM0Q8N44P3XmP16jWcfe4gAB4YNZb27dsxc8aLSOLLL1bxqzMujHXfq/90I2MfGc7Q6/7I3HcWMOrBcQAMuvh8unXryubNhaxZvda7Rnag3//pr6wpKCA7O5tr/jCIJo134fFno8HjM087kREPjmVtwTr+evu9AGRlZfHEqGH8sMOe/HZAXwZedg3FVkzd7GyuuXwQu7duVe49f3XS8Vx9w//Rs/eFNG3SmP+7PtaU3p1SURUN0pvZPErpFjazRUT929vGvwV+Xca1bgRuLCU+EZhYkXopziyEMG3lHKCDmd0gqR2wm5nNiHEPy65X7oCo24kUboq67TZ/Wfn5yy5z1N11L4i6DCrl7D1Pi521x376bKXvVyNeXjAAAAkRSURBVNPido8MJ+qAL+lv+ZpoUrhzztUqFZk9kkRxByI7m9mhkuYAmNnqkq9uOudcbeIPjIpsDl+5NNgyqprp/9845xIo01euids9Mgx4Fmgp6UbgNeCmaquVc85tJ+8eAczsUUmzgW5EAwWnmtn71Voz55zbDlU1e6S2Spu0JTUxswJJzYm+ez8u5VhzM1tV3RV0zrmKyPTukfJa2mOBk4DZRP3Z2uZ1r2qtnXPOVVCmD7alTdpmdlJ47ZCunHPO1RZJ7auOq7zukUPTHTezt6u2Os45Vzk7e/fIHWmOlTwM3Dnnao1Mf9Z8ed0jx0K0Vlr4Xv0WkhpUZ8Wcc257FGV4SzvuPO03Ysacc26HqsI1Imul8vq0WxMtf5Mj6Sd89zCXJkDDaq6bc85V2E7dPQIcD5xPtKJC6tP61wF/qqY6OefcdktqCzqu8vq0RwOjJZ1uZk/XUJ2cc2677dRT/lIcIGn/bYNmNrSK6+Occ5WS6V9jjzsQ+TWwPmxFQE+gfTXVyTnntltVDURKaidpqqT3JC2QdGmIN5eUJ2lheG0W4pI0TFK+pHmp33OR1C+UXyipX0r8MEnzwznDwoIzacV9YNRW87Ul3U60rplzztUqVdinXQj8wczeltQYmC0pj2icb4qZ3SLpKqK1IK8kasx2DFtnYATQOTy7aQjQiej7LbMljTez1aHMAGA60bJjPYAX0lUqbkt7Ww2JBiedc65WMbPYWznXWVbyrW8zW0e0+G4boBcwOhQbDZwa9nsBYyzyFpAraTeiCR15ZrYqJOo8oEc41sTM3rKoMmNSrlWmWC1tSfNhy6+vOkBL4IY45zrnXE2qSEtb0kBgYEpopJmNLKVce6JFfqcDrcxsWTi0HChZmbkNsDjltCUhli6+pJR4WnEHIk8CmgFdgVxgopnNjnmuc87VmIrMHgkJ+ntJOpWkXYCngcvCo6pTzzdJNTryGbd7pBfwMLArUBd4UNJvq61Wzjm3nYqsOPZWHkl1iRL2o2b2TAivCF0bhNeVIb4UaJdyetsQSxdvW0o8rbhJ+3+ALmY2xMyuJVqZfUDMc51zrsZUVZ92mMnxAPC+maV+uXA8UDIDpB/wXEq8b5hF0gVYG7pRJgHdJTULM026A5PCsQJJXcK9+qZcq0xxu0dENNWvRBHffaXdOedqjSqcPXIkcB4wX9LcEPsTcAvwhKT+wKdA73BsInACkA9sAC4AMLNVkm4AZoZyQ1NW/RoEPATkEM0aSTtzBOIn7QeB6ZKeDe9PJfoN5JxztUpVfSPSzF6j7MZpt1LKGzC4jGuNAkaVEp8FHFCResWdp32npGnAUSF0gZnNqciNnHOuJhRn+Dci47a0S1ap8ZVqnHO1mj97xDnnEiTOrJAk86TtnMso3j3inHMJ4t0jzjmXIN7Sds65BPGWtnPOJUiRFZVfKME8aTvnMsrOvrCvc84lyk69sK9zziWNt7Sdcy5BfPaIc84liM8ecc65BPGvsTvnXIJ4n7ZzziWI92k751yCZHpLO+4akc45lwjFWOytPJJGSVop6d2UWHNJeZIWhtdmIS5JwyTlS5on6dCUc/qF8gsl9UuJHyZpfjhnmFKXei+DJ23nXEapqoV9g4eAHtvErgKmmFlHYEp4D9AT6Bi2gcAIiJI8MAToDBwODClJ9KHMgJTztr3X93jSds5llCIrjr2Vx8xeAVZtE+4FjA77o4nWzC2Jj7HIW0CupN2A44E8M1tlZquBPKBHONbEzN4K60uOSblWmbxP2zmXUSoyEClpIFGruMRIMxtZzmmtzGxZ2F8OtAr7bYDFKeWWhFi6+JJS4ml50nbOZZSKDESGBF1ekk53vkmq0ZFP7x5xzmUUq8D/ttOK0LVBeF0Z4kuBdinl2oZYunjbUuJpedJ2zmWUKh6ILM14oGQGSD/guZR43zCLpAuwNnSjTAK6S2oWBiC7A5PCsQJJXcKskb4p1ypTjXSPFG4q95eH2wnV3XWvHV0Fl4Gq8ss1ksYBxwC7SlpCNAvkFuAJSf2BT4HeofhE4AQgH9gAXABgZqsk3QDMDOWGmlnJ4OYgohkqOcALYUtfp0yfiF6bSBoYY5DD7WT858JVhHeP1KyB5RdxOyH/uXCxedJ2zrkE8aTtnHMJ4km7Znm/pSuN/1y42Hwg0jnnEsRb2s45lyCetJ1zLkE8aSeEpGmSOu3oerjKk3S+pN1T3m/5bytpoqTcHVc7V9t50q4BkvzBXC7V+cDupR0wsxPMbE3cC0nKqqpKuWTwpB2TpPaS3pf0D0kLJE2WlCPpEElvhZUqnk1ZxWKapLslzQIulfSQpBGh7CJJx4RVMd6X9FDKfUZImhXucf2O+ryuYiRdLundsF0Wfl5SVzu5QtJ1ks4AOgGPSporKWeb63wiadewf66kGaHc/SUJWtLXku6Q9A5wRA1+TFcLeNKumI7AvWa2P7AGOJ3oweVXmtlBwHyiZxOUqGdmnczsjvC+GdE/st8TPVzmLmB/4EBJh4Qy15hZJ+Ag4OeSDqruD+UqR9JhRM+Z6Ax0IVqJpFlpZc3sKWAWcI6ZHWJm35RxzR8DZwJHmtkhQBFwTjjcCJhuZgeb2WtV+mFcred/tlfMx2Y2N+zPBn4I5JrZyyE2Gngypfzj25w/ITx/dz6wwszmA0haALQH5gK9w4PZs4HdgP2AedXxYVyVOQp41szWA0h6BuhayWt2Aw4DZoZlA3P47hGgRcDTlby+SyhP2hWzMWW/CChvwGh9GecXb3OtYiBbUgfgCuCnZrY6dJs02P7quh0ol63/kq3of0cBo83s6lKOfWtmRdtdM5do3j1SOWuB1ZJKWlXnAS+nKV+eJkSJfq2kVkQLhbra71XgVEkNJTUCTiN6xGZLSS0k1QdOSim/DmhczjWnAGdIaglbVgDfsxrq7hLGW9qV1w+4T1JDYBHhGbrbw8zekTQH+IBoTbnXq6aKrjqZ2dvhr6IZIfRPM5spaWiILSX6b1riIaKfmW8oYyDRzN6T9GdgsqQ6wGZgMNHzm91OzL/G7pxzCeLdI845lyCetJ1zLkE8aTvnXIJ40nbOuQTxpO2ccwniSds55xLEk7ZzziXI/wMDlAInvARs2wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = od_preds['data']['is_outlier']\n",
    "labels = ['normal', 'outlier']\n",
    "f1 = f1_score(y_test_sample, y_pred)\n",
    "acc = accuracy_score(y_test_sample, y_pred)\n",
    "prec = precision_score(y_test_sample, y_pred)\n",
    "rec = recall_score(y_test_sample, y_pred)\n",
    "print('F1 score: {:.3f} -- Accuracy: {:.3f} -- Precision: {:.3f} '\n",
    "      '-- Recall: {:.3f}'.format(f1, acc, prec, rec))\n",
    "cm = confusion_matrix(y_test_sample, y_pred)\n",
    "df_cm = pd.DataFrame(cm, index=labels, columns=labels)\n",
    "sns.heatmap(df_cm, annot=True, cbar=True, linewidths=.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the ROC curve based on the instance level outlier scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAFNCAYAAABbpPhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3hUZeL28e+TRgiEHor0UCJNFGnSQUHAtlgQVFZFUVcRFVCK7q5tdyU0RUUpgos/URBDUUAQFREFpUNCKKGGEkogvWee949ElxcpoUzOTHJ/rivXzJk5M3PjkczNKc9jrLWIiIiIiGfxcTqAiIiIiPyZSpqIiIiIB1JJExEREfFAKmkiIiIiHkglTURERMQDqaSJiIiIeCCVNBEREREPpJImIh7DGLPfGJNujEkxxsQZYz42xpQ+a512xpjvjTHJxphEY8xXxpjGZ61TxhjztjHmYP577clfrlS4fyIRkcunkiYinuYOa21p4HrgBmDU708YY24ClgMLgWuAusAW4GdjTGj+OgHAd0AToCdQBrgJiAdauyu0McbPXe8tIsWTSpqIeCRrbRywjLyy9rtwYJa19h1rbbK19pS19hVgLfBq/jp/BWoBfay12621LmvtcWvtG9baJef6LGNME2PMt8aYU8aYY8aY0fmPf2yMefOM9boYYw6dsbzfGDPCGLMVSM2/P++s937HGDMp/35ZY8xHxpijxpjDxpg3jTG+V/ifSkSKKJU0EfFIxpgaQC8gJn85CGgHfHGO1ecC3fPv3wJ8Y61NKeDnBAMrgG/I2ztXn7w9cQXVH7gNKAd8DvTOf0/yC1hfYHb+uh8DOfmfcQPQA3j8Ej5LRIoRlTQR8TQLjDHJQCxwHPhn/uMVyPuddfQcrzkK/H6+WcXzrHM+twNx1trx1tqM/D10v17C6ydZa2OttenW2gPARqBP/nPdgDRr7VpjTBWgN/C8tTbVWnscmAj0u4TPEpFiRCVNRDzNX6y1wUAX4Fr+V75OAy6g2jleUw04mX8//jzrnE9NYM9lJc0Te9bybPL2rgE8wP/2otUG/IGjxpgEY0wCMAWofAWfLSJFmEqaiHgka+2P5B0eHJe/nAqsAe47x+p9+d8hyhXArcaYUgX8qFgg9DzPpQJBZyxXPVfUs5a/ALrkH67tw/9KWiyQCVSy1pbL/yljrW1SwJwiUsyopImIJ3sb6G6MaZ6/PBJ42BgzxBgTbIwpn39i/03Aa/nrfEJeIfrSGHOtMcbHGFPRGDPaGNP7HJ/xNVDNGPO8MaZE/vu2yX9uM3nnmFUwxlQFnr9YYGvtCWAlMBPYZ62Nzn/8KHlXpo7PHyLExxhTzxjT+TL+u4hIMaCSJiIeK7/wzAL+kb+8GrgVuJu8884OkHcCfgdr7e78dTLJu3hgB/AtkAT8Rt5h0z+da2atTSbvooM7gDhgN9A1/+lPyBviYz95BWtOAaPPzs8w+6zH/woEANvJO3w7j0s7NCsixYix9uw99SIiIiLiNO1JExEREfFAKmkiIiIiHkglTURERMQDqaSJiIiIeCCVNBEREREP5Od0gEtVrlw5W79+fadjyGVKTU2lVKmCjjEqnkTbzrtp+3kvbTvvtmHDhpPW2pDLea3XlbQqVaqwfv16p2PIZVq5ciVdunRxOoZcBm0776bt57207bybMebA5b5WhztFREREPJBKmoiIiIgHUkkTERER8UAqaSIiIiIeSCVNRERExAOppImIiIh4IJU0EREREQ/ktpJmjJlhjDlujIk8z/PGGDPJGBNjjNlqjGnhriwiIiIi3sade9I+Bnpe4PleQIP8nyeAD9yYRURERMSruK2kWWtXAacusMpdwCybZy1QzhhTzV15RERERLyJk9NCVQdiz1g+lP/YUWfiiIiIiKew1uKykJ3rItdlyXHZ/FsXObl593NdllxrceXf5rosLhf/u3/GrctF3rK1WGvJzV92ufI+54/3yV/f2vzn82/tGfddNi9fckYOJfx9sJY/MrrOyHQsdt8V/Tfwirk7jTFPkHdIlJCQEFauXOlsILlsKSkp2n5eStvOu2n7eS9P2nY5Lku2C9JzLOk5ebdZuZCVa8nMhcxcS2KmBfhjOTMXsvNvs1yQnJVXcgCyXZbUrLz39PeFXBfk2v/dAvga8PHJu/U14GNM/u25fkzeLWDOes6Qf5u/zu/Lf7pvwAfzx33z+3vx52UMZOVCkJ/BxyfvsdysDKJ/Wc419RpRPqTqFf33drKkHQZqnrFcI/+xP7HWTgWmAoSFhVlNNOu9NFGw99K2827aft7LHdsuK8fF6bQs4lOySEjLIj41i1OpWZxMySQxPfuP+8kZOfk/2SRn5OCylpL+vpQq4UdwoB+lA/0J8velZIAvJf19KVnCF/+ShlIl/ChX0p+SAb4EBfhRMsCHQD9fAgN8CfTzxcdA6UA/Svj5EODri6+voYSfD34+Bj/fvFt/Xx98fcxV/XO7U2pqKhMmTOC9997jpptuYkTfp2jatCnvDrn893SypC0CBhtjPgfaAInWWh3qFBERKaBclyUlI4eE9CwS07M5nZbNyeRMTqVmcSoti9P55SsxPZvE9LyilZieTUZ2LuWCAqhYKoByQf5UKl2C8qX8qViqBKGVSnFj7fJUKl2CMoH+lA70o0ygH6UD/Qjw9cEY7ylOheHgwYMkJCTQsGFDTpw4wcqVK2nUqNFVeW+3lTRjzGdAF6CSMeYQ8E/AH8Ba+yGwBOgNxABpwKPuyiIiIuINrLUkZ+ZwLDGDE8mZxCVl8Ou+bNakRXMyJYukjP/t5TqdmkVKZg6lSvhRtqQ/ZUv6Uz4ogEqlA6hQqgQVSwdQs3wQ5YP8KRvkT5nAvHXKlPSnTKCfytYVioyMJDw8nMWLF/Pvf/+b6667jkmTJl3Vz3BbSbPW9r/I8xZ4xl2fLyIi4inSsnI4kZzJ8eRMjiZm/LG362hiBvGpmRxLyiQ+JZOE9Gz8fQxVygQSElyCymUCycpwUScogHohpSlT0p/yQf6EBJegfFAAZUv64+NFhwS9nbWW+Ph4KlasyJAhQ+jevTuTJk2iXLlybvk8r7hwQERExFOlZ+Vy6HQacUkZHDqdzuHT6RxJSOdoYgbHkjKIS8og12XzSldwCaqWDaRycCDlgwJoU7cClYIDCCmdV8rK5p/Hdaa8c9LqOfSnEwCXy8WiRYsYM2YMFSpUYPHixXz//fdu/1yVNBERkQtwuSxHEtOJPZXOwVOp7DuZRuypNGJPp3EkIYPkjGyqlytJlTKB1ChfkurlS9K2XkWqlQ2kaplAqpYNpHQJHV70RtZajDE88sgjREdHM2LECPr06VNon6+SJiIixZ61llOpWRw4lcaB+LwidiA+lYOn0og5lkJQCV9qVQiiZoUgQiuVokeTKtSsEET1ciUJKV1ChxyLmMTERKZMmcKcOXNYu3Yt77zzDuXKlSv0oq2SJiIixYa1lvjULPafTGXviVSijiQSHZfMzrhkrLXUqVSKWvlFrFODEGqUL0mDKsFUKBXgdHQpJJ9++ilDhgyhZ8+ezJgxA39/f8qXL+9IFpU0EREpkuJTMtl7MpUD8WnsOZHCltgEth9NwlqoU6kU9SqV4tpqwdzSuAqNqpWhYqkAHZIspnbt2sXbb7/NG2+8QevWrVm/fj1169Z1OpZKmoiIeLesHBfbDiew6WACm2ITOHQqjf3xabispX7l0tSuEETdSqUZ1DGUJtXLEFK6hMqYAHnlbNSoUfz00088/fTT+Pn50aBBA6dj/UElTUREvEZ6Vi6bYxNYt/8U248ksfNYMkcS0qlUugRdwkJoUas8j7arQ51KpbRnTM7JWsuyZcto3LgxPj4+dO7cmVmzZlGqVCmno/2JSpqIiHikrBwXu44ls+1wIlFHEtl0MIG9J1IJqxpM67oV6H1dNYZWbUitCkEE+vte/A2lWMvOzmbu3LmEh4djrWX69Om0bt2aIUOuYN4mN1NJExERjxB7Ko1f9pxky6FEIg8nsjMumZoVgriuelkaX1OGPjdUp8k1ZVXI5JKkpqaSlJSEn58fs2bN4q233qJnz55esZdVJU1ERArd6dQsth5OZPPBBDYePM3OuGSyc110aFCJ5jXK0eeG6jSrrkIml+/kyZO89957TJ48mZEjRzJ06FCWLVvmdKxLopImIiJuZa1lz4kU1uyJZ8OB02w8mMCp1Cyuq1GWJteU4aG2tbm2ajA1ypf0ir0b4tlyc3MBaNeuHZ07d+ann34iLCzM4VSXRyVNRESuquxcF5GHE9l4MIH1+0+x/sBpAnx9aFevIm1DK/JM1/rUCymtAWDlqtqyZQvh4eGkpaUxf/58tm3bRokSJZyOdUVU0kRE5IpYa9l3MpXvdxznx10n2BybQNUygVxfsxy3NqnKqF6NqFUxyOmYUoQ99dRTLFq0iOeff54nn3wSwOsLGqikiYjIJcoblyyRLbEJ/LYvb0+Zn4+hc8MQHmpbm7fvv56Kpb3/C1I8V25uLgsWLGDBggXMmjWL5557jnfeeadIFLMzqaSJiMgFWWvZH5/Gmj3x/LDzOCt3Hqdm+SDahFbk1qZVePm2RjqfTArNokWLGD58OBUqVGDEiBFYa2nUqJHTsdxCJU1ERP4kJTOHH3Yc56fdJ/g5Jp4cl4t29SrRs0lV3rq7mfaUSaFKSEhg5syZPPPMM1SoUIFp06bRqVOnIv8PA5U0EREB4HhSBt9GH+Pb7cdYv/80LeuUp0vDEJ7oFEq9kNJF/gtRPM/Ro0eZMGECM2bMoHfv3iQlJdGhQwenYxUalTQRkWIqLSuH3/adYllUHOv3n+ZYUgZdwipzT4saTOp/A2UC/Z2OKMXUjh07qFatGnv37iUrK4uNGzdSu3Ztp2MVOpU0EZFi4vfxyn7cdZKfY07y275TNKoWTJewyjzQujaNqgXj5+vjdEwpxtasWcOYMWNYs2YN8+fPp3379rRv397pWI5RSRMRKcKstWw/msS8XVm8uXEVqZk5dAkL4Z4WNZjQtznlggKcjijFnLWWjIwM4uLieOihhxg6dCizZ88mKEjDtqikiYgUMVk5LjYcOM2K6GMs3x6HwdC4DITfex031Cync8vEI2RnZ/PZZ58xduxYBgwYwEsvvcTu3bvx8dHe3N+ppImIFAGnU7P4NvoY30cf5+c9J6lTsRTdrq3MBw/eSJNryvDjjz/SolZ5p2OKAHkFrUmTJtSsWZPx48fTvXt3ABW0s6ikiYh4qeNJGczfdJgfd51g26FE2tWvSOeGlXn9L02oHBzodDyR/8/x48d59913SUpK4p133mHFihXUqlXL6VgeTSVNRMSLpGbm8OOuE0RsPMxv++K5pXEVHm5Xh84NQwj093U6nsg5jR49mg8//JC+ffsyfPhwABW0AlBJExHxcDm5LjbHJvDF+kMs2XaUptXLcnvzarzT73pKldCvcfFMmzZtYtmyZYwcOZKuXbsyZMgQqlat6nQsr6K/3SIiHig9K5cfd51gRf7gstXKBnJbs2p8P7wLIcEa7V881+rVq3n99dfZvn07L7zwAtbaP845k0ujkiYi4iEysnP5Lvo430TF8cOO4zSvWZabr63CsB4NqVa2pNPxRM4rNzeXZcuW0atXL2JjY3nggQd44IEHCAjQEC9XQiVNRMRB1lo2xSYw+9eDfLv9GM2ql+XWplV59Y7Gmh9TPF56ejoff/wx48aNo2rVqrRt25b+/fs7HavIUEkTEXHA4YR0IjYcImLTYTKyc3m4XR2G9wijalldlSme7/Tp05QuXZqvvvqKpUuXMmvWrGI9M4C7qKSJiBSSxLRsFm09whfrY9l9LIU+Laozvm9zDTArXiM2Npa3336bmTNnsnjxYvr27Uvfvn2djlVkqaSJiLhRTq6LH3ae4Iv1sazZG0+nBiEM7xHGTfUq4q95MsVLWGuJjo6mQ4cOPProo2zZsoWaNWs6HavIU0kTEXGDHXFJfL3lKPM2HKJ6+ZLc06IG4fdep7kyxausXr2a8PBwbr31Vp5++mn27t1LuXLlnI5VbKikiYhcJWlZOSzcfITZvx7kRHImdzSvxsxHW9GoWhmno4lckvT0dLp3705cXBzDhw/n4YcfxhijglbIVNJERK6AtZbIw0l8tu4gX285Qqs6FRh+axgd61fCx0fnmYn3yMrKYvbs2aSmpvLMM8/w5ptv0rFjR3x9NZOFU1TSREQuQ0pmDnPXxfLZbwfJyMml7401+XZoZ6qU0dWZ4n0mTZrE2LFjadSoEaNHjwagS5cuzoYSlTQRkUsRczyZj3/Zz6LNR+jQoBJv/KUpbepW0NWZ4nWOHTvGzz//zN13301AQAALFy6kRYsWTseSM6ikiYhcRGpmDj/tPsGnvx4k+mgyD7SpxfIXOmtMM/FKMTExjBs3jrlz5/Lwww/Tp08fnnrqKadjyTmopImInIO1lt/2nWLO+li+jTpGsxpluadFDaY/XI0SfjpHR7xPTEwM9evX5/PPPyckJIQdO3ZQuXJlp2PJBaikiYicITUzh4hNh5n1y34A7m5Rg5d7N9IUTeKVrLUsX76cMWPGsGfPHiIjI3nllVecjiUFpJImIgLEnkrj41/2E7HxEK3rVuC1u5pwU2hFnWsmXiknJwdfX18++ugj3n77bV566SX69++Pv7+/09HkEqikiUixtuHAaT5avZdf9sRzf8uaLBrcgZoVgpyOJXJZ0tLSmDFjBuPHj2fevHn89a9/5bHHHtM/NryUSpqIFDs5uS6WRR1j+uq9xKdk8Wj7OoTf25zSJfQrUbzX1q1b6d69O+3atWP27NnceOONTkeSK6TfSCJSbCRlZDPrl/3M/vUg1cuX5MlO9ejeuAq+GnRWvNSBAweYOHEi3bp1o2fPnvz4449ce+21TseSq8Sts/saY3oaY3YaY2KMMSPP8XwtY8wPxphNxpitxpje7swjIsVPrsvyw47jPP3pBtr953t2HUth6l9b8sVT7ejZtKoKmnil5ORkBgwYQIsWLfD396dly5YEBASooBUxbtuTZozxBd4HugOHgHXGmEXW2u1nrPYKMNda+4ExpjGwBKjjrkwiUnycTs1i7vpY/u/XA5Qt6c/9rWrxnz7XUTZIJ06Ld7LWsmrVKlJTU+nVqxft27fn3Xff1XyaRZg7D3e2BmKstXsBjDGfA3cBZ5Y0C/w+83BZ4Igb84hIEZfrsqzafYIvNxzix10n6N64Cu/2b8H1NfUlJt7LWsv8+fMZM2YM8fHx/Pvf/8YYowFoiwF3lrTqQOwZy4eANmet8yqw3BjzLFAKuMWNeUSkiErLyuHTtQeZ/dtBSpXwpX/rWrx+V1MqlApwOprIZcvMzGT79u0YY1i5ciUvvvgif/nLXzTheTFirLXueWNj7gV6Wmsfz18eALSx1g4+Y52h+RnGG2NuAj4CmlprXWe91xPAEwAhISE3zp071y2Zxf1SUlIoXbq00zHkMnjitkvNtnx/MJtvD2TTsLwvt9bxp345Hw03cA6euP3k3FJSUvjqq6+IiIigefPmPP/889p2Xqxr164brLUtL+e17tyTdhioecZyjfzHzvQY0BPAWrvGGBMIVAKOn7mStXYqMBUgLCzMdunSxU2Rxd1WrlyJtp938qRtF5+SyYyf9zH714N0vbYyEYPrUb9ysNOxPJonbT85t9TUVEqVKsWgQYNIS0tjxYoVNG/eXNuuGHNnSVsHNDDG1CWvnPUDHjhrnYPAzcDHxphGQCBwwo2ZRMSLHU1MZ+qqvURsPMzt11XTwLNSJOzcuZNx48axZMkSYmJimDJlCj4+bh18QbyE2/4vsNbmAIOBZUA0eVdxRhljXjfG3Jm/2jBgkDFmC/AZ8Ih11/FXEfFasafSGBWxlZ5v/4SvMSx/oRP/6tNMBU283kcffUTHjh2pXr06W7ZsoWTJkipo8ge3DmZrrV1C3rAaZz72jzPubwfauzODiHivPSdSmPzDHr7fcYwH2tTih+FddDGAeDVrLd988w1jx45lypQp9OnTh379+lGqVCmno4kH0owDIuJxdsQl8f4Pe/gl5iQPt6vDyhe7UrakxjcT77Zt2zYefPBBAEaMGEGdOnU04blckEqaiHgEay1r9sYz65cDbDh4msc71OU/dzfTfJri1VJTU/noo49o164doaGhjBkzhp49e+oKZCkQ/fYTEUe5XJZlUXG890MMWTku+reuxYT7mxMUoF9P4r1SUlIYO3YsH3zwAR07duSWW26hQoUK9OrVy+lo4kX0W1BEHJGT6+LrrUd5/4cYggJ8ef6Whtx8bWV8NJemeLH9+/eTmJhIWFgYiYmJ/PTTT4SFhTkdS7yUSpqIFKqM7FwWbT7Cez/EULVsIH+/vTEdG1TS4R/xalu2bCE8PJxvvvmGt956i+bNm/P22287HUu8nEqaiBSK9KxcPv31AFNW7SWsSjBj772ONqEVnY4lctmstZw4cYKQkBCGDRvGrbfeyuTJkylbtqzT0aSIUEkTEbc6s5y1rF2eWQNb06haGadjiVy23Nxc5s+fT3h4OFWqVOGrr75ixYoVTseSIkglTUTcIiM7lznrYnn/hxhuqFWO/z7amsbXqJyJ97LWYozhr3/9K3v37mX06NHceeedF3+hyGVSSRORqyon18Xn62J5e8VurqtRlo8ebkWzGjr8I94rISGBDz74gLlz5/Lbb78xefJkypQpo/Moxe1U0kTkqnC5LN/vOE74sh1UKBXAx4+2oml1lTPxbv/3f//Hc889x2233casWbPw9/fXOWdSaFTSROSKrd0bz5uLt+NywbAeYfRoXEV7GcRrRUdH8/bbb/Of//yHtm3bsmnTJmrVquV0LCmGVNJE5LLtiEvitUXbOZKYznM3N6DPDdVVzsRr7dy5kxEjRrBmzRoGDx6Mn58f9evXdzqWFGMqaSJyyQ6dTuPd72L4JiqO4beG0a9VTfx9fZyOJXLJXC4XS5YsoVmzZvj6+tK9e3dmz55NUFCQ09FEVNJEpOBOJGcy5psdLI+KY8BNtVn1YlfKBmmCaPE+2dnZfPbZZ4SHh+Pv78+0adNo2bKl9pyJR1FJE5GLSs3MYdGeLF5Y9SN9W9Xkp5e6qZyJV0pJSSEpKQl/f38+//xzJkyYQPfu3XWYXjySjk+IyHnluiwRGw/RddxKjqS4mP90e0b1aqSCJl7n+PHj/P3vf6du3bp88cUXhISEsGTJEnr06KGCJh5Le9JE5E9yXZZvIuOYuGIX5Ur6M/nBFqTs30qdSqWcjiZySXJycjDG0LFjR7p27cqaNWt0SFO8hkqaiPx/dsYlMypiK9m5lldua0TnhiEYY1i53+lkIgW3ceNGwsPDyczMZP78+Wzbto2AgACnY4lcEpU0EQEgMS2bcct3smTbUYbc3IAH29TCT1dsihcaNGgQS5cu5YUXXuCJJ54AUEETr6TfwCLFXE6uixmr93HzhJW4rOW7YZ15uF0dFTTxGjk5OcyZM4cHHngAay3Dhw9n7969DBs2jODgYKfjiVw27UkTKcbW7Innta+iqFAqgE8fb0tYVX2hiXdZsGABw4YNo2rVqowYMQKAsLAwh1OJXB0FKmnGmACglrU2xs15RKQQ7IxL5s3F2zkQn8awHg25s/k1usJNvMapU6eYMWMGQ4YMISQkhFmzZtG+fXunY4lcdRc9nmGMuQ3YBnybv3y9MWa+u4OJyNWXkJbFf5ZGc//UNXRvXIUVQztz1/Waykm8w5EjRxg6dCj169cnKiqK5ORk2rdvr4ImRVZB9qS9DrQBfgCw1m42xuj6ZREvkpXjYvavB3jvhz3cfG1llj3fiSplAp2OJVIgUVFR1KhRg3379mGMYevWrdSoUcPpWCJuV5CSlm2tTTjrX9rWTXlE5CpbHhXHf5buoHbFIP47sBVNrinrdCSRAlm9ejVjxoxh3bp1zJ8/X3vNpNgpSEmLNsb0BXyMMXWBIcBa98YSkSsVczyZ177azuHT6fzzjsZ0CavsdCSRi3K5XGRkZBAXF8ejjz7KsGHDmDt3LiVLlnQ6mkihK8g19oOBGwEXEAFkAs+5M5SIXL5TqVmMitjKvR+uoWtYZZa90EkFTTxeVlYWM2fOpGnTpkyePJnQ0FB27tzJU089pYImxVZB9qTdaq0dAYz4/QFjzN3kFTYR8RAul2XO+ljGLdvJHc2v4ccXu1K2pObYFM+XnZ1N48aNCQ0N5d1336Vbt24A+PhorD4p3gpS0l7hz4Xs5XM8JiIOWb//FG8ujgbgk8fa0PiaMg4nErmwY8eO8c4775CSksKkSZNYuXKlLgYQOct5S5ox5lagJ1DdGDPhjKfKkHfoU0QcFnM8hbeWRrP9SBIvdG/IPS1q4OOj4TTEs40cOZKpU6fSv39/hg0bBqCCJnIOF9qTdhyIBDKAqDMeTwZGujOUiFxYckY2734fw5x1sTzTtR7v9m9ByQBfp2OJnNf69etZvnw5o0eP5pZbbmHo0KFUrqxzJUUu5LwlzVq7CdhkjPnUWptRiJlE5DyyclzMWXeQSd/H0KlBCN++0InKGu9MPNiqVat47bXX2L17N8OGDcNayy233OJ0LBGvUJBz0qobY/4FNAb++Daw1jZ0WyoR+ZPf9p1iZMRWqpcryYyHW9GshsY7E8+Uk5PD0qVLuf322zly5AgPP/ww/fv3x99fF7KIXIqClLSPgTeBcUAv4FE0mK1IoTmdmsWbi6NZHXOC1+5sQs+m1ZyOJHJOaWlpzJgxg/Hjx1OzZk3at29Pv379nI4l4rUKcn1zkLV2GYC1do+19hXyypqIuNmK7cfo9c5PBAX48u3Qzipo4pHi4+PJzs7m66+/ZsWKFcyePZtVq1ZRoUIFp6OJeLWC7EnLNMb4AHuMMU8Bh4Fg98YSKd4OxKfy2lfb2XsihfF9m9O+fiWnI4n8yYEDB5gwYQKffPIJS5YsoW/fvvTt29fpWCJFRkH2pL0AlCJvOqj2wCBgoDtDiRRX1lrmroulz+RfaFWnAstf6KyCJh7HWktUVBQtWrSgRIkSREZG0rZtW6djiRQ5F92TZq39Nf9uMjAAwBhT3Z2hRIqj48kZvPjFVo4nZzJ7UBuuraoBacVzWEKyDXIAACAASURBVGtZtWoVY8aM4bbbbuPpp59m7969lC2rC1hE3OWCJc0Y0wqoDqy21p40xjQhb3qoboBGHhS5ClwuS8Smw7y1dAf9WtVkyM0NCPDTdDjiOdLS0rj55puJj4/nxRdfZMCAARhjVNBE3OxCMw78B7gH2AK8Yoz5GngaGAM8VTjxRIq26KNJjIrYRq7LMv3hllxfs5zTkUQAyMzM5JNPPiEjI4PBgwczZswY2rdvj6+vBk0WKSwX2pN2F9DcWptujKkAxALNrLV7CyeaSNGVmpnDlFV7+WTNfkb2upb7bqyp6ZzEY0ycOJGxY8fSvHlzRo8eDUCnTp0cTiVS/FyopGVYa9MBrLWnjDG7VNBErtz8TYf41+Jo2oRWZMlzHalWtqTTkUQ4evQoP//8M/feey9BQUEsXbqU5s2bOx1LpFi7UEkLNcZE5N83QN0zlrHW3u3WZCJFTHJGNv9cGMWGg6f578DWNLlG5/OI83bu3Mm4ceP48ssvGThwIPfeey9PPvmk07FEhAuXtHvOWn7vUt/cGNMTeAfwBaZba986xzp9gVfJm8Vgi7X2gUv9HBFP98PO4/xjYSRt61Zk6XMdCQooyBCFIu6za9cuGjZsyLx586hevTq7du2iUiUN9yLiSS40wfp3V/LGxhhf4H2gO3AIWGeMWWSt3X7GOg2AUUB7a+1pY0zlK/lMEU+TmJ7Nm19vZ+2+eF6/syldr9X/4uIcay1Lly4lPDycAwcOsHXrVl5++WWnY4nIebjzOv/WQIy1dq+1Ngv4nLyLEc40CHjfWnsawFp73I15RAqNtZb5mw7RfcKP+Pn6sPS5Tipo4picnBystUyfPp2RI0cyaNAgdu3aRXCwJo8R8WTGWvfMlW6MuRfoaa19PH95ANDGWjv4jHUWALvIm8nAF3jVWvvNOd7rCeAJgJCQkBvnzp3rlszifikpKZQuXdrpGG6VkOlidnQWh5JdPH5dCULLFo0hC4rDtitq0tPTWbx4MfPmzWPUqFE0adIEX19fjNGVxN5Ef/e8W9euXTdYa1tezmsLfGKMMaaEtTbzcj7kIp/fAOhC3uC4q4wxzay1CWeuZK2dCkwFCAsLs126dLnKMaSwrFy5kqK6/Vwuy4LNh3nj6+3c17I2/72lISUDikZBg6K97YqiLVu2cN9999GpUycWLVpEWlqatp+X0t+94uuiJc0Y0xr4CCgL1DLGNAcet9Y+e5GXHgZqnrFcI/+xMx0CfrXWZgP7jDG7yCtt6wqYX8QjxBxPYdjczVjgvwNbc10NDUorhW/fvn1MmDCBW265hV69evHzzz/TsGFDIO+LXkS8S0HOSZsE3A7EA1hrtwBdC/C6dUADY0xdY0wA0A9YdNY6C8jbi4YxphLQENBYbOI1XC7LR6v3ce+Hv3Bfy5oseLq9CpoUuqSkJB544AFatmxJqVKlaN26NQEBAX8UNBHxTgU53OljrT1w1jkMuRd7kbU2xxgzGFhG3vlmM6y1UcaY14H11tpF+c/1MMZsz3/PF6218Zf8pxBxQMzxFP6+IJKMnFwWPtOe2hVLOR1JihFrLStXriQtLY3evXvTpUsXPvjgA82nKVKEFKSkxeYf8rT5w2o8S97J/hdlrV0CLDnrsX+ccd8CQ/N/RLxCTq6LT9Ye4N3vY/hb53oM7FAXX03pJIXEWsuXX37JmDFjSE5O5l//+hfGGJ544gmno4nIVVaQkvY38g551gKOASvyHxMpdo4kpDN49kb8fH2Y+2Rb6lfWEAZSODIyMoiKiuLGG2/k559/5uWXX+bOO+/Ex8edIymJiJMKUtJyrLX93J5ExMMt3XaUfyyK4tH2dXiqUz1NiC6FIiEhgQ8++IBJkybRvXt3Zs2axcSJE52OJSKFoCAlbZ0xZicwB4iw1ia7OZOIRzmdmsUrCyOJPprEhw+14MbaFZyOJMXA72NjDR8+nKysLJYvX06zZs2cjiUiheiiJc1aW88Y0468qzNfM8ZsBj631n7u9nQiDrLW8sX6Q4z5Zgd3Xn8N4+/rSKB/0Rn3TDxTdHQ0Y8eOZdmyZezZs4dp06Zp8FmRYqpAJzNYa3+x1g4BWgBJwKduTSXisKOJ6Twycx2z1u7nvwNb8887mqigidtNmzaNLl26ULduXbZu3UpgYKAKmkgxdtGSZowpbYx50BjzFfAbcAJo5/ZkIg6w1rJg02F6v/MT19csx/yn29O0uoY0EPdwuVx8/fXXdO7cmd27d3Pvvfeyb98+/v73v1OxYkWn44mIwwpyTlok8BUQbq39yc15RByzMy6Z176K4nhyJp881kblTNxqy5YtPPjgg/j7+zNixAjq1q2Ln1+BZ+oTkWKgIL8RQq21LrcnEXFIWlYO734fw+xfDzL81jD6taqJv6+GNZCrLyUlhWnTptGxY0dCQ0OZMGEC3bt31yFNETmn834TGWPG59/90hgTcfZPIeUTcatthxLpPmEVh0+n8+0LnRjQtrYKmlx1ycnJvPLKK9StW5e1a9dSunRpKlSoQI8ePVTQROS8LrQnbU7+7XuFEUSkMGXluHhr6Q7mbzrEG39pyu3XXeN0JCmC9u7dS1JSEtdeey0ZGRmsWbOG+vXrOx1LRLzEeXcZWGt/y7/byFr73Zk/QKPCiSdy9e2IS6LP5J+JPZ3Gshc6qaDJVbdx40b69etH69at2bBhA4GBgYwbN04FTUQuSUGO6ww8x2OPXe0gIu6WmZPLxG938cC0XxnQtjZTB9xI5eBAp2NJEWGt5dixY1hrGTlyJK1bt2bfvn089ph+XYrI5Tnv4U5jzP3kDWBb96xz0IKBBHcHE7maIg8nMnTuZmpXLMVXz3agermSTkeSIiInJ4cvv/yS8PBwatSowcKFC1m+fLnTsUSkCLjQOWm/AfFADeD9Mx5PBja5M5TI1ZKd62Lcsp3M23CI0b0bcXeL6jpRW64Kay3GGAYMGEBsbCyvvvoqt912m9OxRKQIOW9Js9buA/YBKwovjsjVE3UkkWFzt3BNuZIsf6ETFUuXcDqSFAGnTp1i8uTJzJs3j3Xr1jFlyhTKlCnjdCwRKYIuNATHj/m3p40xp874OW2MOVV4EUUuTa7LMvPnfTw0/VcGdQzlo4dbqqDJVTFr1izq16/Pnj17mD17Nv7+/ipoIuI2Fzrc2TX/tlJhBBG5GvadTGX4F1swQMTT7albqZTTkcTLRUVFMXHiRMLDw2nfvj1bt26lRo0aTscSkWLgQkNw/D7LQE3A11qbC9wEPAnom088ztdbj3DPB7/Qq2lV5j55kwqaXJEdO3Zwxx13cPPNNxMaGoqfnx/16tVTQRORQlOQaaEWAK2MMfWAmcDXwGzgdncGEymoXJcl/JsdLI2MY8Yjrbi+ZjmnI4mX+n3C8+bNm+Pv78/tt9/O3LlzKVlSVwOLSOErSElzWWuzjTF3A+9aaycZY3R1p3iEE8mZvDhvC1k5LuY/3U7nnsllycrK4tNPP2Xs2LEEBQUxZcoUbrzxRurVq+d0NBEpxgpS0nKMMfcBA4C/5D/m775IIgWzevdJRny5ld7NqjL81jBK+Pk6HUm8TFJSEsnJyQQEBBAREcG7775Lt27dNEyLiHiEgs440BUIt9buNcbUBT5zbyyR87PW8uGPe3h+zib+eUdjXr6tsQqaXJK4uDhGjx5NaGgoERERhISE8NVXX3HzzTeroImIx7jonjRrbaQxZghQ3xhzLRBjrf2X+6OJ/NnRxHRGfLmNpPRsFg7WzAFyabKzs/Hx8aFz5850796ddevWUbduXadjiYic00VLmjGmI/AJcBgwQFVjzABr7c/uDidypm+3H2NUxDYeaFOLId3q4+dbkB3BIrBu3TrGjBmDy+UiIiKCyMhI/P111oaIeLaCnJM2Eehtrd0OYIxpRF5pa+nOYCK/y3VZ/r0kmiXbjjJlwI3cWLu805HEizz22GOsWLGCoUOH/jHZuQqaiHiDgpS0gN8LGoC1NtoYE+DGTCJ/OJqYzgtzNuPrY1j6XEfKBel/PbmwnJwc5s6dy6JFi/jss88YOXIkH374oYqZiHidghwv2miM+dAY0yH/5wM0wbq4mbWWT9bsp9c7P3FTaCVmDWyjgiYXFRERQYMGDZgyZQoDBgwAoEGDBipoIuKVCrIn7SlgCPBS/vJPwLtuSyTFXmJaNi99uYUjCRnMffImGlYJdjqSeLD4+HimT5/OCy+8QNWqVfnss89o27at07FERK7YBUuaMaYZUA+Yb60NL5xIUpztPZHCY/9dT/v6FZnU/wYNrSHndejQIcaOHcsnn3zC3XffTUpKCu3atXM6lojIVXPekmaMGQ08Bmwkb1qo1621MwotmRQ70UeTeHTmOgZ3q89DbWs7HUc81NatW6lduzYHDx6kRIkSREZGcs011zgdS0TkqrvQnrQHgeustanGmBBgCaCSJm6xaMsRXl0UxT/vaMxd11d3Oo54GGstq1atYsyYMWzatIn58+fTrl077TkTkSLtQiUt01qbCmCtPWGM0aBUctXl5Lr4+8JI1u49xcePtuK6GpocXf4nNzeXjIwM4uLiePLJJxk2bBgREREEBgY6HU1ExO0uVNJCjTER+fcNUO+MZay1d7s1mRR5p1OzeG7OZnJyXSwe0oGggIJcxyLFQWZmJrNmzWLcuHE8+eSTDB06lOjoaE3ZJCLFyoW+Fe85a/k9dwaR4mXboUSemb2R7o2rMKLntQT4aUet5MnKyqJRo0aEhYUxZcoUOnfuDKCCJiLFznlLmrX2u8IMIsXHN5FxjJ6/jdfubMIdzXXCt8CRI0d45513SE1N5b333mP16tW6GEBEij3tvpBCk5SRzczITF5ZsI1ZA1uroAkAL774Ik2bNiU9PZ3hw4cDqKCJiKCSJoVkc2wCt05chQG+H96FptXLOh1JHLR27VreeOMNAHr16sWuXbuYNGkSderUcTaYiIgHKXBJM8aUcGcQKZqstczbcIiBH6/jtTub8EjTEpQJ1BQ9xdXKlSvp3Lkz/fr1o1y5clhr6datG5UqVXI6moiIx7loSTPGtDbGbAN25y83N8ZoWii5qIzsXF6Ys5kPVsYwe1AbejSp6nQkcUB2djYLFizAWsuxY8d48skn2b17N88++6wuBhARuYCCjHkwCbgdWABgrd1ijOnq1lTi9fadTGXgx+u4vmY5vn62IyUDNL1TcZOamsr06dOZMGECoaGhdO7cmfvvv9/pWCIiXqMgJc3HWnvgrH/x5ropjxQBB+JTeXjGbwxsX4eH29XR3pJi5sSJE5QvX57Fixfz008/8cUXX9C6dWunY4mIeJ2CnJMWa4xpDVhjjK8x5nlgl5tziZfaEpvAfR+u4fGOdXmkfV0VtGJk3759DB48mIYNG7Jhwwb69u3LvHnzVNBERC5TQUra34ChQC3gGNA2/7GLMsb0NMbsNMbEGGNGXmC9e4wx1hjTsiDvK57pu+hjDPx4Hf/u04y/3lTH6ThSSKy1REZG0qpVK4KDg9m+fTtt2rRxOpaIiNe76OFOa+1xoN+lvrExxhd4H+gOHALWGWMWWWu3n7VeMPAc8OulfoZ4jrnrYhm3fCcfPdKK62tq/s2izlrLDz/8wJgxY7jrrrv429/+xr59+wgODnY6mohIkXHRkmaMmQbYsx+31j5xkZe2BmKstXvz3+dz4C5g+1nrvQGMAV4sSGDxPO+s2M2cdQeZPagN9SvrS7qoS0tLo0uXLiQnJ/Piiy/y4IMPYoxRQRMRucoKcuHAijPuBwJ9gNgCvK76WesdAv6/YyDGmBZATWvtYmOMSpqXSUjL4h8Lo9h+NIlFz3agUmkNpVdUZWRksGjRIrZu3cqQIUOYOHEiN910Ez4+Gg9bRMRdCnK4c86Zy8aYT4DVV/rBxhgfYALwSAHWfQJ4AiAkJISVK1de6cfLFYqOz+XDrZm0quLLi80DiFy/pkCvS0lJ0fbzMnPmzGHu3LmEhoYSGhr6x/ZbtWqVs8HkkujvnvfStiu+CrIn7Wx1gSoFWO8wUPOM5Rr5j/0uGGgKrMy/ArAqsMgYc6e1dv2Zb2StnQpMBQgLC7NdunS5jNhytSzYdJgPf4zi/Yda06HBpY0Uv3LlSrT9PN/hw4f5+eef6du3L7t37+aZZ57h5MmT2nZeTH/3vJe2XfFVkBkHThtjTuX/JADfAqMK8N7rgAbGmLrGmADyLj5Y9PuT1tpEa20la20da20dYC3wp4ImnsPlsrz2VRQTV+xizpNtL7mgieeLjo5m4MCBNGvWjI0bNwIwaNAgmjZt6nAyEZHi54J70kzeLq7m/G8PmMta+6eLCM7FWptjjBkMLAN8gRnW2ihjzOvAemvtogu/g3iS9Kxcnv50A6lZuSx6pgNlgzT/ZlESHR1No0aNWLBgAXXr1iUmJoYKFSo4HUtEpFi7YEmz1lpjzBJr7WX9M9pauwRYctZj/zjPul0u5zPE/dKycvjb/22kbEl/pv61Jf6+Olm8KHC5XCxevJgxY8YQFxfH5s2bGTWqIDvJRUSkMBTk23azMeYGtycRj7T7WDL3frCGiqUDGN+3uQpaEZCdnY21lunTp/PPf/6TZ599lh07dlC6dGmno4mIyBnO+41rjPl9L9sN5A1Eu9MYs9EYs8kYs7Fw4omTNhw4Rf9pa3mobW3G36eC5u2Sk5P/mOx8y5YtPProo2zYsIH7778fP7/LuYZIRETc6UK/mX8DWgB3FlIW8SDfRMYxKmIrE/peT9drKzsdR67Qpk2b6NGjB926dWPBggVcf/31TkcSEZGLuFBJMwDW2j2FlEU8gLWW6T/tY+pPe/n40dY01xRPXmvPnj2MHz+eHj160Lt3b9asWUP9+vWdjiUiIgV0oZIWYowZer4nrbUT3JBHHJSd62Lkl9vYdSyZ+U+3o0b5IKcjyWVISkpi0KBBfPfddzz11FO0a9eOgIAAFTQRES9zoZLmC5Qmf4+aFG0pmTk8/t91lC7hx5wn2xIUoHOUvIm1lhUrVpCRkcHtt99Ojx49mD59uubTFBHxYhf6Jj5qrX290JKIY06nZjHwv+toULk0/7n7Onx91Mu9hbWWuXPnEh4eTkZGBv/6178wxvDYY485HU1ERK7QRc9Jk6LtcEI6D8/4ja5hIYzq1QgfFTSvkJ6eTmRkJK1ateK3337j1Vdf5bbbbtOE5yIiRciFStrNhZZCHBF5OJHH/7uexzrUZVCnUKfjSAGcOnWKyZMn895779G7d29atWrF+PHjnY4lIiJucN6SZq09VZhBpHAtj4rjpS+38sZdTbmj+TVOx5GLSE5OJjg4mBEjRpCbm8v3339P48aNnY4lIiJupLPDi6Ev1scSvmwnHz/amus1xIZHi4qKIjw8nO+++46YmBimTp1K3pS6IiJS1OkElmLmhx3HGbtsJ7Mfb6OC5uGmTJnCzTffTFhYGNu2bSMwMFAFTUSkGFFJK0aWRcUx7IstvPdACxpU0dAMnsblcrFw4UI6dOhATEwMffv2Zd++fYwePZry5cs7HU9ERAqZDncWE3PXxzJm6Q6mP9ySFrX0he9pNm/eTP/+/SlVqhQjRoygbt26+Pr6Oh1LREQcpJJWxFlree/7GOasj2XuUzdRL6S005EkX1JSElOnTqVz587Ur1+f999/n65du+qQpoiIADrcWaTluix/XxjJ0sg4Iv7WTgXNQyQlJTFq1ChCQ0PZuHEjZcqUoXz58nTr1k0FTURE/qA9aUVURnYuQz7bRGpWDnOebEtwoL/TkYq93bt3k5KSQqNGjcjNzWXdunXUrVvX6VgiIuKhtCetCDqckE6/qWsJ8PNh5iOtVdActm7dOu677z7at2/Pli1bCAwMJDw8XAVNREQuSCWtiIk+msQ9k3+he+MqTOp3AwF+2sROsNZy9OhRrLW88sordOjQgb179/LII484HU1ERLyEDncWIRsOnOKJWRv4xx2Nuev66k7HKZZycnL+mPA8NDSUiIgIli1b5nQsERHxQippRcRXW47w6qIoxvdtTpewyk7HKXastRhjeOihhzh69Cj//ve/6dWrl9OxRETEi6mkFQFLtx3l9a+383+Pt6FRtTJOxylW4uPjee+99/jyyy/ZuHEj06ZNIzhYAwWLiMiV0wlLXsxaywcr9/CPRVHMeLiVCloh+/jjj2nQoAGxsbF88cUX+Pn5qaCJiMhVoz1pXspay+tfb2fNnngWDW5PtbIlnY5ULGzdupWJEycyfvx4OnXqRGRkJNdcc43TsUREpAjSnjQv5HJZRs+PZP3+03z+RFsVtEIQHR1N79696dmzJ9deey0BAQGEhoaqoImIiNtoT5qXycpx8cKczcQlZTB7UBuNgeZGubm5LFy4kBtvvJGAgAD69OlDREQEgYGBTkcTEZFiQCXNi6Rk5jDov+spHejHp4+3IdBfE3C7Q2ZmJrNmzWLcuHGUK1eODz/8kBtuuIF69eo5HU1ERIoRlTQvkZGdyxOz1lO9fEneursZfr46Un21JSYmkpKSQkBAAF9//TVTpkyhc+fOmk9TREQcoW96L/D7Ic6yJf1V0NzgyJEjvPTSS4SGhrJgwQJCQkJYuHAhXbp0UUETERHHaE+ah0tIy2LQrPWUCwrg3f43qKBdRdnZ2fj4+NClSxd69erFxo0bqV27ttOxREREAJU0j3Y6NYv7pqyhY4NKvHJbY3x9tFfnali7di3h4eEAREREEBUVhb+/LsAQERHPot0yHup4cgb3T13DzY0q8887mqigXSWPPPII/fv3p1u3bnzyyScAKmgiIuKRVNI80LGkDPpNXUvvZtUY1auR03G8WnZ2Np988gl9+/bF5XLxyiuvsHv3bgYPHkypUqWcjiciInJeKmke5tDpNO6e/Av3tKjB87c0dDqOV/viiy+oV68eM2fOZODAgRhjqF+/Pn5+OsovIiKeT99WHuRYUgYPTv+VR9vX4fGOoU7H8UonTpxg+vTpDBs2jBo1ajBv3jxat27tdCwREZFLpj1pHiI+JZMBH/3KvS1qqKBdhoMHDzJ48GDCwsLYv38/qamp3HTTTSpoIiLitbQnzQOcTs3iwem/ckujKjx7cwOn43iVTZs2ERoayqFDhwgODmb79u1UrVrV6VgiIiJXTCXNYYlp2Tw4/Ve6hFXmxVvDnI7jFay1fP/994SHhxMZGcn8+fNp164d7dq1czqaiIjIVaPDnQ46npTBfVN+oV29iozoGabR7S8iNzeX1NRU9uzZw7PPPsv999/P3r17dUhTRESKJJU0hxxJSKff1LXc1uwaXr6tkQraBaSnp/Phhx8SFhbGtGnTqF+/PlFRUQwcOJASJUo4HU9ERMQtdLjTAceT88ZBe6htLZ7oVM/pOB4tKyuLRo0a0bRpU2bOnEmHDh0AVGpFRKTIU0krZMkZ2TwxawN/uf4aFbTzOHToEG+//Tbp6em8//77rF27VhcDiIhIsaPDnYUoOSObez74hWbVy2qg2vMYOnQo1113Hbm5uYwYMQJABU1ERIolt5Y0Y0xPY8xOY0yMMWbkOZ4faozZbozZaoz5zhhT2515nJSZk8vTn27kxtoVeOMvTfHRXJx/+OWXX3j99dcBuOOOO4iJiWHixInUqlXL4WQiIiLOcVtJM8b4Au8DvYDGQH9jTOOzVtsEtLTWXgfMA8LdlcdJObkuhs3dQqC/L6/d2cTpOB7ju+++o0OHDjz00EOEhIRgraVr165UqFDB6WgiIiKOc+eetNZAjLV2r7U2C/gcuOvMFay1P1hr0/IX1wI13JjHEVk5Lp7+dCOn07J4t/8NBPgV7yPM2dnZREREYK0lPj6ewYMHs2vXLv72t7/pYgAREZEzuPPCgepA7BnLh4A2F1j/MWCpG/MUuozsXB6duY7gQD9mPtK6WBe05ORkpk2bxltvvUXz5s3p2rUrffv2dTqWiIiIx/KIqzuNMQ8BLYHO53n+CeAJgJCQEFauXFl44S6Ty1omb84E4P76Jfhl9SqHEznj9OnTBAcHs3r1an788UdGjx7N9ddfz5YtW5yOJpcoJSXFK/7uyblp+3kvbbviy50l7TBQ84zlGvmP/X+MMbcALwOdrf1/7d15dJRFusfx7yNLAoJhkxkWkUAwIltAIUQQRBgGVGBQZLkoqCAjM4gboBzn3oteHQaIOmZEETdAkLAjogiK4ngVZA0QFiEISEAUIosSQra6f3STGzGBJiTp7uT3OafP6X673qqnu06SJ1X1vuXO5FWRc24qMBUgMjLS3XzzzYUebGHKznaMWbCFyyqmMuP+NoSWK+PvkIrdnj17iI2NZc6cOXz00UeMGzcOgFWrVhHo/Sd5U98FN/Vf8FLflV5FOf+2DmhkZuFmVh7oDyzJXcDMWgKvAT2dcz8WYSzFxjnHU4sT2Z9yimn3tS51CZpzjq1btxIdHU316tXZuXOntm0SEREpgCIbSXPOZZrZCGA5UAZ4yzm3zcyeAdY755YAk4BKwDzvovHvnHM9iyqmouac47kPdvDN4ZPMGBJNxfIBMZtc5JxzfPLJJ0ycOJHevXszfPhw9u3bR6VKlfwdmoiISNAq0izCOfch8OE5x/4r1/MuRdl+cXLO8fT72/lqz1HmDIuhUkjpSNBSU1O56aabSEtLY8yYMQwYMAAzU4ImIiJyiUpHJlHE0jKyGDN/CweOpTLvwRsJq1DO3yEVqdOnT/P222+TkZHBww8/zMsvv0x0dDSXXVZ6r14VEREpbPqreomysx0Px28iIyubd4e2LfEJ2oQJE6hfvz7Lly8nOtpzR5WYmBglaCIiIoVMI2mXaMLynaT8ks6sB6IJKVsyLxI4cOAAX331Ff369aNGjRqsWrWKxo0b+zssERGREk3DH5fgXyt38/G2H3j17utLZIKWmJjI4MGDf3VfsyFDhihBExERKQZK0gro1VV7mLvhAHP+HMOVlUP8HU6hcc6xbds2KYXA+wAAFg5JREFUAJYuXcq1115LUlISf//73/0cmYiISOmi6c4CWLAhmXdW72PxX9uVmAQtOzubJUuWMHHiRI4ePcrGjRt58skn/R2WiIhIqaWRtIv0VdJRnv1gO9Pub0PNK0L9Hc4lS09PxznH1KlTee6553jsscfYsWOHbqEhIiLiZ0rSLsLKHT/w0OxNTB7Yimt+V9nf4VySkydPMmnSJMLDw9myZQtDhw5l7dq19OnThzJlSt76OhERkWCjJM1H6/f9xKh5m3lj8A3c2LCGv8O5JBs3bqRBgwZs2rSJDz74gBYtWlC2bFm8uz6IiIhIANCaNB/s+uFnhs5Yz4v9omhZr6q/wymQXbt2ERsbS/fu3bnttttYt24d4eHh/g5LRERE8qGRtAs4kZrBQ+9u4slu19Ipsqa/w7loJ06coE+fPrRv355atWrRvn17ypcvrwRNREQkwGkk7TwysrJ5YMZ62oRXo1/rq/wdjs+ccyxfvpz09HR69OjB7bffzvTp07n88sv9HZqIiIj4SCNp5/GvT5MILV+Gp3s2CYr1Ws45Zs2aRVRUFGPGjAHAzLj33nuVoImIiAQZjaTl47NvfmTuugMs+uuNXHZZYCdop06dIjExkejoaBISEhg/fjzdu3cPisRSRERE8qaRtDwk/fgLo+dt5oW+LagVVsHf4eTr6NGjjBs3jvDwcF5//XUAJk2axK233qoETUREJMhpJO0cB4+fZtCbXzOm27XcGBGYt9o4ceIEYWFhjB07FoAvvviCyMhIP0clIiIihUlJWi6p6ZkMmbaOe9vVp+8NgXehwObNm5k4cSKff/45SUlJTJ06VSNmIiIiJZSmO70ysrIZPnMjTeuE8cBNDfwdzm+8+uqrdO/enebNm7Nt2zZCQ0OVoImIiJRgStK8nl26nWznGH9Hs4BIfrKysliwYAExMTEkJSUxYMAA9u7dyxNPPEFYWJi/wxMREZEipulO4M3/3cuXe1JY8OCNlCvj/7x106ZN9OvXj6pVq/LEE08QHh6u/TRFRERKmVKfpG3Yf4xXPkvivRHtCKtYzm9xHD9+nClTptC5c2ciIiJ4/fXX6dChQ0CM6omIiEjx8/+wkR99l5LKsBnrib2rBXWrVvRLDCdPnmTMmDE0bNiQxMREqlSpQtWqVenYsaMSNBERkVKs1I6k/ZyWwYMzNzD85oZ0urb49+TcuXMnqampXHfddZQpU4aNGzdy9dVXF3scIiIiEphK5UhaVrZj5OxNNKsTxpD2xbvR+OrVq+nduzcdOnQgMTGR0NBQxo8frwRNREREfqVUjqS9tHI3v5zJZMo9xbMnp3OOQ4cOUatWLZ555hluu+02Zs2aRcWK/pliFRGR4pGRkUFycjJpaWkFriMsLIwdO3YUYlRSFEJDQ6lbty7lyhXe+vZSl6Qt2/o9c9Z9x/sPtSekbNFeMZmRkUF8fDwTJ04kMjKS+fPns2zZsiJtU0REAkdycjKVK1emfv36BR4U+Pnnn6lcuXIhRyaFyTlHSkoKycnJhIcX3gxdqUrSVu9J4anFiUy7rzU1K4cWWTvZ2dlcdtllDBw4kKNHjxIbG0vXrl2LrD0REQlMaWlpl5SgSXAwM6pXr86RI0cKtd5Sk6TtOfILI+M38ULfFjSvW6VI2jhy5AhxcXEsWrSIhIQE3n77bS6//PIiaUtERIKDErTSoSj6uVRcOLA/5RR3v/E1j3a5hpsji+ZKzrfeeotrrrmGH3/8kUWLFlG2bFklaCIi4neVKlX6zbFx48YRGxv7m+NlypQhKiqKpk2b0qNHD44fP+5TG0ePHqVcuXJMmTLlvG1PmzaNESNG5LyeMWMGTZs2pVmzZrRs2TLPmPIzfvx4IiIiiIyMZPny5XmWuemmm4iKiiIqKoratWvzpz/9CfDcYSEmJoaQkJDftPniiy/SpEkTmjZtyoABA3LWE+7du5fo6GgiIiLo168f6enpPsdaUCU+STuems59b69jWIcG/Ed0vUKte9OmTQwaNIiUlBQ6derEjh07eO2112jUqFGhtiMiIlIcKlSoQEJCAomJiVSrVo3Jkyf7dN68efNo27Yts2fP9rmtZcuW8c9//pMVK1awdetW1qxZ4/O2h9u3byc+Pp5t27bx0Ucf8Ze//IWsrKzflPviiy9ISEggISGBmJgY7rjjDgCqVatGXFwco0aN+lX5gwcPEhcXx/r160lMTCQrK4v4+HgAnnjiCR599FGSkpKoWrUqb775ps+ftaBKdJKWne0YNW8z7SJqcF+7wlvIt23bNrp27UqPHj1o3rw5ISEhhIeH8/vf/77Q2hAREfGnmJgYDh486FPZ2bNn8/zzz3Pw4EGSk5N9Omf8+PHExsZSu3ZtAEJCQnjggQd8Ove9996jf//+OX9/IyIiWLt2bb7lT548yaeffpozklazZk1at26d55WYmZmZnD59mszMTFJTU6lduzbOOT799FP69OkDwODBg1m8eLFPsV6KEp2kxX26m2OpGfzn7dddcl1ZWVnMnTuX/fv3ExoaSv/+/dmzZw+jRo3KcyhZREQkWGVlZbFy5Up69uyZcywqKirPsgcOHOD777+nTZs29O3blzlz5vjURmJiItdff32e702aNClnmjL3Y+TIkYBnxOuqq67KKV+3bt3zJpSLFy+mc+fOXHHFFeeNqU6dOowaNYp69epRq1YtwsLC6Nq1KykpKVSpUoWyZcv61F5hKbEXDny+6wgz1+zng5E3Ub5swXPRtLQ0pk2bRmxsLDVr1uSVV14hKiqKhg0bFmK0IiJSGtR/8oNCr3PfP24rtLpOnz5NVFQUBw8epHHjxvzhD3/IeS8hISHPc+bMmUPfvn0B6N+/P/fffz+PP/54vm34ssB+9OjRjB49+iKjz9/s2bMZOnToBcsdO3aM9957j71791KlShXuuusuZs6cSbdu3QotlotRIpO0xIMneHROAq/dcz2/u6Jgt9o4duwYp06dIiQkhBUrVjBt2jTat29fyJGKiEhpUpCEqjjvk3Z2TVpqaip//OMfmTx5cs7oVX5mz57N4cOHmTVrFgCHDh1i9+7dNGrUiAoVKpCenk758uUB+Omnn6hRowYATZo0YcOGDdxyyy2/qXPSpEk59eXWoUMH4uLiqFOnDgcOHMg5npycTJ06dfKM7+jRo6xdu5ZFixZd8PN/8sknhIeHc+WVVwJwxx138NVXXzFw4ECOHz9OZmYmZcuWPW97hanETXf+dCqdIdPX8UyvJrSuX+2iz09OTubxxx+nYcOGvP/++1x55ZUsXLhQCZqIiJQaFStWJC4ujueff57MzMx8y+3atYtffvmFgwcPsm/fPvbt28fYsWNzLiDo2LEjM2fOBDyjdHPnzqVTp04AjB07ltGjR3P48GEA0tPTeeONNwDPSNrZBf+5H3FxcQD07NmT+Ph4zpw5w969e9m9ezdt2rTJM8b58+dz++23Exp64UGbevXqsWbNGlJTU3HOsXLlSho3boyZ0alTJ+bPnw/A9OnT6dWrly9f5SUpUUlaemY2909bx52t6nJ789oXd256OpmZmXTu3BnnHJs3b2b48OFFFKmIiEjxSE1NpW7dujmPF154AYBnn332V8fP1bJlS5o3b56TcOW1Jm327Nn07t37V8fuvPPOnHNeeuklFi5cSFRUFG3btuWuu+6iQ4cOANx6662MGDGCLl260KRJE1q1asXJkyd9+kxNmjShb9++XHfddXTr1o3JkydTpkyZnHoPHTqUUzY+Pp4BAwb86vzDhw/nfBdnv4eTJ08SHR1Nnz59aNWqFc2aNSM7O5thw4YBMGHCBF544QUiIiJISUlhyJAhPsV6Kcw5V+SNFKbIyEj3zTff5Pne3z/cwbdHfuH1QTf4fFO5L7/8kgkTJlCuXDkWLFiQM5QpRWPVqlXcfPPN/g5DCkB9F9zUf/6xY8cOGjdufEl1aFuo4JFXf5vZBufcDQWpr8SMpL2XcJBlid/zjzub+5yg3XPPPQwaNIju3bvnDMcqQRMREZFAUCIykgM/pTJuyTbeGRJNjUoh+ZZLT0/n3XffZenSpcydO5enn36aevXqKTETERGRgBP0I2lpGVn8+Z0NjLilEU3r5H+n4rlz59KwYUNmzZrFgw8+iJnRoEEDJWgiIiISkII+SXvxk1387ooQ7m9X/zfv/fDDDzz77LOcOXOGevXqsXjxYj7++GO6dOmiDW9FRKRYBNvabymYoujnoE7S5q0/wPLEw0y6q8Wvkq79+/czfPhwGjduzKFDh0hNTaVt27b53tlYRESkKISGhpKSkqJErYRzzpGSkuLTbT4uRtDO9W3Y/xPjl+1k1tD/X4e2YcMGIiIiOHToENWrV2fnzp3UrFnTz5GKiEhpVbduXZKTkzly5EiB60hLSyv0P/5S+EJDQ/O8lcmlKNIkzcy6AS8BZYA3nHP/OOf9EGAGcD2QAvRzzu27UL0/nUrnkTkJ/E+vplz7+8qsWLGCCRMmsGvXLhYuXEhMTAwxMTGF/4FEREQuQrly5QgPD7+kOlatWkXLli0LKSIJJkU23WlmZYDJQHfgOmCAmZ270/kQ4JhzLgJ4EZjgS91/W7yVTo2q07FB5ZxNzgcPHsyePXto3bp1YX4MEREREb8oypG0NkCSc+5bADOLB3oB23OV6QWM8z6fD7xsZubOM3n/S3o2nyx4h4yEJdR45BFGjhzJ5s2bdSGAiIiIlChFmaTVAQ7kep0MROdXxjmXaWYngOrA0fwq/f7APjpes4vnZs7kxhtvBFCCJiIiIiVOUFw4YGbDgGHel2dWrViW2G7FMn+GJAVXg/Mk4RLQ1HfBTf0XvNR3wS2yoCcWZZJ2ELgq1+u63mN5lUk2s7JAGJ4LCH7FOTcVmApgZusLugeW+J/6L3ip74Kb+i94qe+Cm5mtL+i5RXmftHVAIzMLN7PyQH9gyTlllgCDvc/7AJ+ebz2aiIiISGlRZCNp3jVmI4DleG7B8ZZzbpuZPQOsd84tAd4E3jGzJOAnPImciIiISKlXpGvSnHMfAh+ec+y/cj1PA+66yGqnFkJo4j/qv+Clvgtu6r/gpb4LbgXuP9PsooiIiEjgCeq9O0VERERKqoBN0sysm5l9Y2ZJZvZkHu+HmNkc7/tfm1n94o9S8uJD3z1mZtvNbIuZrTSzq/0Rp+TtQv2Xq9ydZubMTFedBRBf+s/M+np/BreZ2bvFHaPkzYffnfXM7DMz2+T9/XmrP+KU3zKzt8zsRzNLzOd9M7M4b99uMbNWvtQbkElaUW4pJUXLx77bBNzgnGuOZ6eJicUbpeTHx/7DzCoDDwNfF2+Ecj6+9J+ZNQLGAu2cc02AR4o9UPkNH3/2/gbMdc61xHOh3SvFG6WcxzSg23ne7w408j6GAa/6UmlAJmnk2lLKOZcOnN1SKrdewHTv8/lAZ9PWA4Hggn3nnPvMOZfqfbkGzz30JDD48rMH8D94/jFKK87g5IJ86b8HgMnOuWMAzrkfizlGyZsvfeeAK7zPw4BDxRifnIdz7t947lKRn17ADOexBqhiZrUuVG+gJml5bSlVJ78yzrlM4OyWUuJfvvRdbkMAbR8ROC7Yf95h+quccx8UZ2DiE19+/q4BrjGzL81sjZmd779/KT6+9N044G4zS8Zz54SHiic0KQQX+7cRCJJtoaRkMrO7gRuAjv6ORXxjZpcBLwD3+jkUKbiyeKZcbsYziv1vM2vmnDvu16jEFwOAac65580sBs99Rps657L9HZgUjUAdSbuYLaU435ZSUux86TvMrAvwFNDTOXemmGKTC7tQ/1UGmgKrzGwf0BZYoosHAoYvP3/JwBLnXIZzbi+wC0/SJv7lS98NAeYCOOdWA6F49vWUwOfT38ZzBWqSpi2lgtcF+87MWgKv4UnQtB4msJy3/5xzJ5xzNZxz9Z1z9fGsKezpnCvw3nRSqHz53bkYzygaZlYDz/Tnt8UZpOTJl777DugMYGaN8SRpR4o1SimoJcAg71WebYETzrnvL3RSQE53akup4OVj300CKgHzvNd6fOec6+m3oCWHj/0nAcrH/lsOdDWz7UAWMNo5p1kIP/Ox7x4HXjezR/FcRHCvBicCg5nNxvPPTw3vmsH/BsoBOOem4FlDeCuQBKQC9/lUr/pXREREJPAE6nSniIiISKmmJE1EREQkAClJExEREQlAStJEREREApCSNBEREZEApCRNRAqVmWWZWUKuR/3zlK1vZomF0OYqM/vGzDZ7tzuKLEAdD5rZIO/ze82sdq733shro/lLjHOdmUX5cM4jZlbxUtsWkeCjJE1ECttp51xUrse+Ymp3oHOuBTAdz734Lopzbopzbob35b1A7VzvDXXObS+UKP8/zlfwLc5HACVpIqWQkjQRKXLeEbMvzGyj93FjHmWamNla7+jbFjNr5D1+d67jr5lZmQs0928gwntuZzPbZGZbzewtMwvxHv+HmW33thPrPTbOzEaZWR88e8rO8rZZwTsCdoN3tC0nsfKOuL1cwDhXk2uDZTN71czWm9k2M3vae2wknmTxMzP7zHusq5mt9n6P88ys0gXaEZEgpSRNRApbhVxTnYu8x34E/uCcawX0A+LyOO9B4CXnXBSeJCnZu/VNP6Cd93gWMPAC7fcAtppZKDAN6Oeca4Znh5XhZlYd6A00cc41B57NfbJzbj6wHs+IV5Rz7nSutxd4zz2rHxBfwDi74dmi6aynnHM3AM2BjmbW3DkXBxwCOjnnOnm3cfob0MX7Xa4HHrtAOyISpAJyWygRCWqnvYlKbuWAl71rsLLw7Bd5rtXAU2ZWF1jonNttZp2B64F13i3EKuBJ+PIyy8xOA/uAh4BIYK9zbpf3/enAX4GXgTTgTTNbCiz19YM5546Y2bfevfd2A9cCX3rrvZg4y+PZGi3399TXzIbh+b1cC7gO2HLOuW29x7/0tlMez/cmIiWQkjQRKQ6PAj8ALfCM4KedW8A5966ZfQ3cBnxoZn8GDJjunBvrQxsDc2/0bmbV8irk3SOxDZ6NqvsAI4BbLuKzxAN9gZ3AIuecM0/G5HOcwAY869H+BdxhZuHAKKC1c+6YmU3Ds3n2uQz42Dk34CLiFZEgpelOESkOYcD3zrls4B48G0j/ipk1AL71TvG9h2fabyXQx8xqestUM7OrfWzzG6C+mUV4X98DfO5dwxXmnPsQT/LYIo9zfwYq51PvIqAXMABPwsbFxundFPs/gbZmdi1wBXAKOGFmvwO65xPLGqDd2c9kZpebWV6jkiJSAihJE5Hi8Aow2Mw245kiPJVHmb5AopklAE2BGd4rKv8GrDCzLcDHeKYCL8g5lwbcB8wzs61ANjAFT8Kz1Fvf/5L3mq5pwJSzFw6cU+8xYAdwtXNurffYRcfpXev2PDDaObcZ2IRndO5dPFOoZ00FPjKzz5xzR/BceTrb285qPN+niJRA5vmHTkREREQCiUbSRERERAKQkjQRERGRAKQkTURERCQAKUkTERERCUBK0kREREQCkJI0ERERkQCkJE1EREQkAClJExEREQlA/weh5IYttN5LMQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "roc_data = {'LLR': {'scores': od_preds['data']['instance_score'], 'labels': y_test_sample}}\n",
    "plot_roc(roc_data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}