examples/integrated_gradients_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrated gradients for MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we apply the integrated gradients method to a convolutional network trained on the MNIST dataset. \n",
    "Integrated gradients defines an attribution value for each feature of the input instance (in this case for each pixel in the image) by integrating the model's gradients with respect to the input along a straight path from a baseline instance $x^\\prime$ to the input instance $x.$\n",
    "\n",
    "A more detailed description of the method can be found [here](https://docs.seldon.io/projects/alibi/en/latest/methods/IntegratedGradients.html). Integrated gradients was originally proposed in Sundararajan et al., [\"Axiomatic Attribution for Deep Networks\"](https://arxiv.org/abs/1703.01365)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TF version:  2.3.1\n",
      "Eager execution enabled:  True\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Activation, Conv2D, Dense, Dropout\n",
    "from tensorflow.keras.layers import Flatten, Input, Reshape, MaxPooling2D\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from alibi.explainers import IntegratedGradients\n",
    "import matplotlib.pyplot as plt\n",
    "print('TF version: ', tf.__version__)\n",
    "print('Eager execution enabled: ', tf.executing_eagerly()) # True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading and preparing the MNIST data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28, 1) (60000, 10) (10000, 28, 28, 1) (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "train, test = tf.keras.datasets.mnist.load_data()\n",
    "X_train, y_train = train\n",
    "X_test, y_test = test\n",
    "test_labels = y_test.copy()\n",
    "train_labels = y_train.copy()\n",
    "                         \n",
    "X_train = X_train.reshape(-1, 28, 28, 1).astype('float64') / 255\n",
    "X_test = X_test.reshape(-1, 28, 28, 1).astype('float64') / 255\n",
    "y_train = to_categorical(y_train, 10)\n",
    "y_test = to_categorical(y_test, 10)\n",
    "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a convolutional neural network on the MNIST dataset. The model includes 2 convolutional layers and it reaches a test accuracy of 0.98. If `save_model = True`, a local folder `./model_mnist` will be created and the trained model will be saved in that folder. If the model was previously saved, it can be loaded by setting `load_mnist_model = True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_mnist_model = False\n",
    "save_model = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = './model_mnist/'  # change to directory where model is saved\n",
    "if load_mnist_model:\n",
    "    model = tf.keras.models.load_model(os.path.join(filepath, 'model.h5'))\n",
    "else:\n",
    "    # define model\n",
    "    inputs = Input(shape=(X_train.shape[1:]), dtype=tf.float64)\n",
    "    x = Conv2D(64, 2, padding='same', activation='relu')(inputs)\n",
    "    x = MaxPooling2D(pool_size=2)(x)\n",
    "    x = Dropout(.3)(x)\n",
    "    \n",
    "    x = Conv2D(32, 2, padding='same', activation='relu')(x)\n",
    "    x = MaxPooling2D(pool_size=2)(x)\n",
    "    x = Dropout(.3)(x)\n",
    "    \n",
    "    x = Flatten()(x)\n",
    "    x = Dense(256, activation='relu')(x)\n",
    "    x = Dropout(.5)(x)\n",
    "    logits = Dense(10, name='logits')(x)\n",
    "    outputs = Activation('softmax', name='softmax')(logits)\n",
    "    model = Model(inputs=inputs, outputs=outputs)\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                  optimizer='adam',\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    # train model\n",
    "    model.fit(X_train,\n",
    "              y_train,\n",
    "              epochs=6,\n",
    "              batch_size=256,\n",
    "              verbose=1,\n",
    "              validation_data=(X_test, y_test)\n",
    "              )\n",
    "    if save_model:\n",
    "        if not os.path.exists(filepath):\n",
    "            os.makedirs(filepath)\n",
    "        model.save(os.path.join(filepath, 'model.h5'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate integrated gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IntegratedGradients class implements the integrated gradients attribution method. A description of the method can be found [here](https://docs.seldon.io/projects/alibi/en/latest/methods/IntegratedGradients.html).\n",
    "\n",
    "In the following example, the baselines (i.e. the starting points of the path integral) are black images (all pixel values are set to zero). This means that black areas of the image will always have zero attribution.\n",
    "The path integral is defined as a straight line from the baseline to the input image. The path is approximated by choosing 50 discrete steps according to the Gauss-Legendre method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize IntegratedGradients instance\n",
    "n_steps = 50\n",
    "method = \"gausslegendre\"\n",
    "ig  = IntegratedGradients(model,\n",
    "                          n_steps=n_steps, \n",
    "                          method=method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate attributions for the first 10 images in the test set\n",
    "nb_samples = 10\n",
    "X_test_sample = X_test[:nb_samples]\n",
    "predictions = model(X_test_sample).numpy().argmax(axis=1)\n",
    "explanation = ig.explain(X_test_sample, \n",
    "                         baselines=None, \n",
    "                         target=predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'IntegratedGradients',\n",
       " 'type': ['whitebox'],\n",
       " 'explanations': ['local'],\n",
       " 'params': {'method': 'gausslegendre',\n",
       "  'n_steps': 50,\n",
       "  'internal_batch_size': 100,\n",
       "  'layer': 0}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Metadata from the explanation object\n",
    "explanation.meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['attributions', 'X', 'baselines', 'predictions', 'deltas', 'target'])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data fields from the explanation object\n",
    "explanation.data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get attributions values from the explanation object\n",
    "attrs = explanation.attributions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample images from the test dataset and their attributions.\n",
    "\n",
    "* The first column shows the original image. \n",
    "* The second column shows the values of the attributions.\n",
    "* The third column shows the positive valued attributions.\n",
    "* The fourth column shows the negative valued attributions.\n",
    "\n",
    "The attributions are calculated using the black image as a baseline for all samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApYAAAGaCAYAAAC12APsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABGhUlEQVR4nO3deZwcVbn/8e8zS5LJAtlYshIgbIHLZgh6AeUKAsqqoHBFjQso94J6RUFEth+LKO6IgujlCsgmghgFRWSTfVXQKEhYQhISskMmmWSWfn5/VA32qcx0T0+fme6e+bzz6lfm6aquPtN9+vQzVU+dMncXAAAAUK66SjcAAAAAAwOJJQAAAKIgsQQAAEAUJJYAAACIgsQSAAAAUZBYAgAAIIoBk1ia2c/M7ML05/3M7PlebucKMzs7butQK8zseDP7Q17sZjY94vabzWybWNtDz5nZmWb20wLLg/e+2uWPVWa2v5ktjLjtXo+hCJnZ78xsdqXb0VP5Y1T+92qkbfP9Ogj0a2JpZq+YWUvacV9PO+3I2M/j7g+4+w49aM/HzezBzGNPcvcLYrepi+e+In0dOm8bzGxNXz/vQGRm95nZKjMbmnffK2Z2YF48LU0SGwpty92vc/eDIrbrhMz2R7r7SzG2P9DFHi/c/WvufkK67Y36Q8z3vqe6akdX41JXYo5V2T+gejqGVqO03yw1sxF5951gZvf1w3OfZ2Y/z7/P3d/r7lf39XMXa0dX41FXYo1Rlfx+RWVVYo/l4e4+UtKekmZKOiu7QrEv/4Eg/YCN7LxJukHSzZVuV60xs2mS9pPkko4oc1sDvt/VoKLjxWBkZvWVbkOVq5f0+Uo3opYw/iEad++3m6RXJB2YF39T0m/Tn13SyZJekPRyet9hkv4iabWkhyXtmvfYPSQ9LWmNpJsk3SjpwnTZ/pIW5q07RdKtkpZJWiHpMkk7SVovqUNSs6TV6bo/69xOGp8oaZ6klZLmSJqYt8wlnZS2ebWkH0qyXrwuI9Lf4139+X4MhJukcyQ9JOk7eX3pWkk5SS3pe3u6pFfT96s5vb1D0sfTx3437RcXpvc9mHmPPyfpJUnL0z5bly47T9LP89adlq7fIOmitG+tT5/vsrztTU9/3lTSNWm/nK8kaerc9sclPSjpW5JWSXpZ0nvznuvjaZvWpMuOr/R70Qfv7Svqfrw4QtLc9HN3n6Sd8tb7sqRF6WvzvKQDsu9Xgf7wYLr8cknfyrTn15JOTX+eKOmW9L17WdLnCvweh0r6s6Q3JS2QdF7esq7a0d24dLmkOyStlXSg8sYqpWOepDPTfvpKfp9IX6MTMv2n83f9U9qGtelzHquNx9Cd0m2sTl/3I/KW/UzJ2Hd7+po/JmnbdJkp+XwtTX//v0rapR/6zRlKxuzR6X0nSLovb50dJd2VrvO8pA/lLRsn6Tdpe59QMi7kjwnfT9/HNyU9JWm/9P5DJLVKaktfx2fyX3tJQ9PXb5e8bW2mZJzaPI27/c7r4vfscTtUeDzKfu/mj1E/k3RF+lqtkXS/pK3SZdPSdRuy/Ux9/P0qaXraljeU9PebKj1eccvrm/36ZHlfFEqSvbmSLkhjTzvvWElNShLHpZL2VvLX5+z08UMlDVHyRfwFSY2Sjkk/RBslluljn1EyuI2QNEzSvumyjytvwEjv+1nedt6ddto90+f9gaQ/5a3rkn4rabSkqUq+ZA5Jl01NPwxTe/C6fExJklByUjrYb+mg9N+S3pb2gS2yfS2NuxoEPy6pXdJnlSSDTdk+kT7m3rRfTpX0T6Vf0CqQWKbxfcr7Ms/bXuegfY2SZGVU+th/SvpUXtvalAy89ZL+S9JrSr6oRyj5MtkhXXeCpJ0r/V70wXv71nuovPFC0vZKkqD3KPn8n572gyGSdlDyZTsx7z3pTHLeer8K9IfOZOud6XY6v8jGKEkAJio50vOUkj9qhkjaJv38HtzN77G/pH9LH7erpNclHdWTduTd9zMlX6L7pNsZpo0Ty3Ylf2ANlfSu9DXq7CNBX8w+R36/zNte5xjamL6+Z6a/77uVJBk75LVthaRZSj5H10m6MV12cPpajVbSd3eSNKE/+o2SnQmdr89biaWSz88CSZ9I27uHknF+Rrr8xvQ2XNKMdN381+ojSpLPBklflLRE0rCuxoTsay/pKkkX5S07WdLv05+7/c7r5vfsdTsy7/tb37vZvpC+t2uUfB6GKklmOz8j09RNYlmkH8f4fr1B0lf1r8/CvpUer7j961aJQ+G3mdlqJXtj7pf0tbxlF7v7SndvkfRpST9298fcvcOTGpUNkt6e3holfc/d29z9l0r+suzKLCVfBqe5+1p3X+/uReuXUsdLusrdn3b3DZK+Iukd6eHXTl9399Xu/qqSBGR3SXL3V919dHp/MbMlXePpJwY9Y2b7StpK0i/c/SlJL0r6cImbec3df+Du7Wm/68o30n75qqTvSfrPXjc6lR7KPE7SV9x9jbu/Iunbkj6at9p8d/+Ju3dIulpJArlFuiwnaRcza3L3xe4+t9w2VamuxotjJd3u7ne5e5uSvbpNkv5dyR6SoZJmmFmju7/i7i/24nkfUPLFtl8aHyPpEXd/TdJekjZz9/PdvdWTerSfKHk/N+Lu97n7X9095+7PKvlSfFcv2vRrd38o3c76btY52903uPv9SvYgfqgXz5P1dkkjlYx1re5+j5Iv/PzPwa/c/XF3b1eSWO6e3t+m5A+nHZUk6f9w98UR2tQT50j6rJltlrn/MEmvuPv/pZ/7PyvZ+/zB9HN5tKRz3X2du/9dyWfvLe7+c3dfkT7220r6W0/rUa9X2E8+nN4nFf7O20iZ7ciX/73bldvd/U/pd+BXlXwHTunF82T1+vtVSb/aSskfkKV8p6MfVCKxPCpNuLZy9//OdOYFeT9vJemLZra686Zkr8XE9LYok4jN7+b5pij5gm7vRVsn5m/X3ZuV/GU+KW+dJXk/r1MyAPeYmU1Vsnfgml60b7CbLekP7r48ja9P7yvFguKrBOvMV9IvyjVeyR9H+f12vrrpW+6+Lv1xpLuvVZJcnSRpsZndbmY7RmhTNepqvMh+LnNK3qNJ7j5P0v8o2WOz1MxuNLOS3690bLlR/0qePqwkYZLSL7TM2HSm/pX0B8xsbzO718yWmdkbSt638aW2ScX76qq0b3SK1VcnSlqQvs752y46DqZJ6GVKDmMuNbMrzWyTCG0qyt3/piQBPiOzaCtJe2fev+Mlbank0HSDwtc6eN3N7Etm9g8zeyN97Kbq+ft5r6ThaZ+YpiRR+lVeu7r7zttIme3IV6xfvbU8/Q5c2V2bSlTO9+vpSvaAP25mc83skxHag0iqbbqh/ERxgZJDBqPzbsPd/QZJiyVNMjPLW39qN9tcIGlqN4XJxfYQvqbkwy5JSs8yHKekfiuWj0p6yDlTuCRm1qRkb8y7zGyJmS1RUhqxm5ntpo3f2+7e657sJc7/63yqkn4hJYcah+ct27KEbS/Xv/7qzt92j/qWu9/p7u9RshfzOSV7zAaL7OfSlLxHiyTJ3a9398692S7pG11soyfv+w2SjjGzrZQcnrwlvX+Bknq0/LFplLu/r5vtXK+kfmyKu2+qpGatc+zqqh297atj8s+EVml9tZDXJE0xs/zvi1L66qXu/jYlh5W3l3RaCc9drnOVlJPkJysLJN2fef9Guvt/KTnc2i5pct76b33+zWw/JUnNhySNcffRSkoUCr2fb0mPPvxCyR8s/6mkZrhzNpBC33mBXrajt/0q//cfqeSw+WtK+pTUfb/qs+9Xd1/i7ie6+0RJn5H0I4s4LRzKU22JZb6fSDop/cvOzGyEmR1qZqMkPaLkw/85M2s0sw8oOeTdlceVJKJfT7cxzMz2SZe9LmmymQ3p5rE3SPqEme2eTmXzNUmPpYctY/mYkroTlOYoJYc9Zyj5q393JfVbDyh5TV9XUvvWaZmSw8e9mUPyNDMbkx7++bySk8WkpMj+nWY21cw2VXIoJ1+2DW/J+4K5yMxGpcnLqZJ+3tX6+cxsCzM7Mh2INygpjs8VedhA8gtJh5rZAWbWqKS+bIOkh81sBzN7d/p5Xa+kLrKr16Zof0gPkS6X9FNJd7r76nTR45LWmNmXzazJzOrNbBcz26ubTY2StNLd15vZLIXlGl21o9i4VMj/M7MhaeJxmP4108RfJH3AzIanX8Cfyjyu276q5GScdZJOT8fb/SUdrmSPbkFmtlc6hjcqSUTWqx/7aroH+yYlJ+B1+q2k7c3so+nv05i2c6f0c3mrpPPS12pHJeNJp1FKvnuWSWows3Mk5e+BfV3StEwSnnW9kiMOx+tfh8Glwt95Wb1pR6H3uJD3mdm+aX+8QNKj7r7A3ZcpSQI/kn4GPilp28zz9cn3q5l90Mw6k/9VSpLYwTQGVrWqTSzd/Uklf2lepqTjzFNSDCx3b5X0gTReqeRDems32+lQMghOV3IG5sJ0fUm6R8kJAUvMbHkXj/2jpLOV7KlYrORD02UdVVaabDRbcqi7u3XeoeQvY6YZKt1sSf/nSS3rks6bkv5yvKSLJZ2VHlL6Unoo+SJJD6X3dVm31I1fKzkB4S9K6tb+V5Lc/S4lX1rPpst/m3nc95Xs8VplZpd2sd3PKvmyfUlJDeH1Sor7i6lTkoS+pqT/v0vJyT2Dgrs/r+TEhR8oSfwOVzItUauSOrOvp/cvkbS5Nk74VUJ/uF7JiSDX5z22Q0nStruSM8I7k89Nu9nGf0s635J5as9RkhgXakfBcamAJUrGyteUHLY/yd2fS5d9V8mZwq8rqRm8LvPY8yRdnbYhqMtMX9fDJb03/V1/JOljedsuZBMlCdMqJYc9Vyg5u78/na/khB1JUrqH8CAlY/lrSl63byjpO5J0ipL3comSGSZuUPKHiyTdKen3Sk60m68kUc4/lNw5lq8ws6e7aoy7P6bkcz9R0u/y7u/2O68LvWlHsfGoO9cr2fO7UslJkh/JW3aikj3QKyTtrORM9k599v2qpM75MTNrVnI04PMc9asenWc8AgCADDP7hqQt3b3U+m1gUKraPZYAAPQ3M9vRzHZND0fPUlI28KtijwOQYKZ9AAD+ZZSSw98TlZQOfFtJOQyAHuBQOAAAAKLgUDgAAACiKHgo3MzYnVnD3N2Kr9X3Wlpa6Ec1rKmpqeL9iD5U26qhD0n0o1pXLf0IhbHHEgAAAFGQWAIAACAKEksAAABEQWIJAACAKEgsAQAAEAWJJQAAAKIgsQQAAEAUJJYAAACIgsQSAAAAUZBYAgAAIAoSSwAAAERBYgkAAIAoSCwBAAAQBYklAAAAoiCxBAAAQBQklgAAAIiCxBIAAABRkFgCAAAgChJLAAAAREFiCQAAgChILAEAABAFiSUAAACiILEEAABAFCSWAAAAiILEEgAAAFGQWAIAACAKEksAAABE0VDJJz/mmGOC+MQTTwzi1157LYjXr18fxNddd10QL1myJIjnzZtXbhNRA6zDg/jV9oeDePSQyUE8zMYFcVPzsCDuaKoPYq+3cpuIGrNgw0NBXKwPNWpkn7cJ1S87FpU6djQ0twcxYxFqEXssAQAAEAWJJQAAAKIgsQQAAEAU5u7dLzTrfmEEL730UhBPmzatrO2tWbMmiOfOnVvW9sq1cOHCIL7kkkuC+Mknn+zT53f3qijIaWlp6dN+9Kdl3w/iNetXB/Gz8x8N4rvufTGI/++0nwbx+bd+IYgv+OAVQTypfq8gXtzxdBCPb9whiJtWNQbxm6PD9mXr9Tb4qiAe3jY2iHND+vfvwaampor3o4HWhyY2hn1oSVvhPtSgEUHc4kuDuFgfGmbjVUnV0Iekvu9H/c1yYexFhobGVa1B3DZmSEnPV9caPuFgHItQHHssAQAAEAWJJQAAAKIgsQQAAEAUFa2xPOCAA4J41113DeJ//OMfQbzTTjsF8Z577hnE+++/fxBPmjQpiBcsWBDEU6ZM6XFbJam9PZxjbNmyZUE8YcKEgo//zne+E8Rf+tKXSnr+Ug2WGsvs3G/WXvjp2jcJax7XWFgL+/jrNwTx3c/+LohPe09YL7e+vTmIRzdMDZ9P4fyrUti+TdrCfuoN4d97HXVhXVS9wrqovi4aq4a6pmqvjWv2yvahkVZ4LMsp7EN1Kq22rlzV0Iek6h+LitVI9rVsDWV2LMq2L/umDoaxCMWxxxIAAABRkFgCAAAgChJLAAAARFHRGsvYxowZE8S77757ED/11FNBvNde4VxyxWSvVf7Pf/4ziLM1oWPHhvMPnnzyyUF8+eWXl/T8pRosNZax1WXqotoaNgRxQy68tnhdW3YyuTDMzvVmucLXE16ncI7C19Y/G8TTh4S1yX19/eBqqGuqtT6U1ZGpkazXsG7WjCM7z2W2D23bdGCfPn9WNfQhqfb6UXYsyjWEL2N2HsvYY1Gxb5Byr41eqmrpRyiMPZYAAACIgsQSAAAAUZBYAgAAIIoBVWPZ344++ugg/sUvfhHEf/vb34L4P/7jP4J45cqVfdOwFDWWtaFVq4P4hrlnBPEndghrcT1TZzUY5o6jDxVWrA/N3jmcN7O/VUMfkuhH5crWfA7GsQjFsccSAAAAUZBYAgAAIAoSSwAAAETRUOkG1JLNN988iH/0ox8FcV1dmKeff/75QdzXNZWoDdm535bnXgji5xY8F8TtO7cFcX9f5xnVb3lb4T6U27my1wpHdSp1HsrsPJpAV9hjCQAAgChILAEAABAFiSUAAACioMayBNlrfW+22WZBvGrVqiB+/vnn+7xNqD3t9eG1x/+25K4gPvvgG4K4ob0xiHN8age97LXH/7a0cB+iphJdKfXa3sWuXQ5I7LEEAABAJCSWAAAAiILEEgAAAFFQrVXAPvvsE8RnnHFGN2smjjrqqCDOXiscg1NDc3sQ/7n9liCevOnOQTzCJwQxNZXIenZ14T403MI+BEgbj0XtIwsPLpYLY2oq0RPssQQAAEAUJJYAAACIgsQSAAAAUVC9VcD73ve+IG5sDOcTvPvuu4P4kUce6fM2ofZ0NNUH8RTtFsRbtM0I4hx/7qGIKaPCPjS+fpcKtQS1JDsWZdWvD4sqc8MYjFA6eg0AAACiILEEAABAFCSWAAAAiIIayzxNTU1BfMghhwRxa2trEJ977rlB3NbW1jcNQ03JzvTWWr82iJc2vxjE40dSH4fC2tQcxEtb6EMoLjsWFbs2eAc1lYiAXgQAAIAo2GMJAAAQwYgpw7wjc3Z9uTYsb7vT3Q8pvmZ1ILEEAACIILc+p60/MD7qNp+7cnHcDfYxEss8p512WhDvscceQfz73/8+iB9++OE+bxNqzwatDuLF658N4u1HHtSPrUEtaqUPIQKvdAMGI5Pq6gb3NdVJLAEAACKxwZ1XklgCAADEYJLqBvlp0SSWAAAAMZhkHAofvA499NAgPvvss4P4zTffDOLzzz+/z9uE2lPXHlYyratfFsSbDd0uiBs9nC/VB/cYhC6s88J9qEFhHwKkjceiXEPhwcUyRZiMRXGwxxIAAABRGIklAAAAymUm1Q3ys3dILAEAACJhj+UgMm7cuCC+9NJLg7i+vj6I77jjjiB+9NFH+6ZhqDHhVRU889fp2I7pQWyt4fodw/umVagl2StzhN9Eo207AaXK1khmay43HovC7zzEQY0lAAAAymbGHksSSwAAgEi48g4AAACiGOTn7gzsxDJbM5m91vfWW28dxC+++GIQZ+e1BCSprj0cNWxDWLe0dsTqIG6qC2t7gWxNZdZ6rQjiYaIPYWMb1VBmxqKOEeF3oNVRU9nXzKixHNCJJQAAQP8xrrxT6QYAAAAMCOyxJLEEAACIhbPCB7Btt902iN/2trcVXP/UU08N4mzNJQandq0N4nnr7w3iXXRIEA9vGxPEuca+aRdqx0Z9qDnsQzuOPCyIqalET2SvBd6wPqy5rGsLay5zjYM84+kHJq68M6ATSwAAgH7DPJYklgAAALFQYwkAAICymcRZ4ZVuQExbbbVVEP/hD38ouP5pp50WxL/97W+jtwm1Jzs33JCX24N4xtaHBnF7Q2YuuSJzFGLwadCIIM7WVBa7djgGp+xYpJffDOOtNwnC9pHhV/rgTm8qhLPCB1ZiCQAAUEmD/Nwd/iwGAACIwZRcKzzmrUfPa3aImT1vZvPM7Iwulg81s5vS5Y+Z2bS8Zbua2SNmNtfM/mpmw8p5DdhjCQAAEEMFzgo3s3pJP5T0HkkLJT1hZnPc/e95q31K0ip3n25mx0n6hqRjzaxB0s8lfdTdnzGzcZLaymnPgEosP/3pTwfx1KlTC65///33B7G7d7MmBpP1DWuCuH36uiC+f9GlQXzw5C8HcZ2G9E3DUDNaFdbCtXs4j+X9i34axPQhdCU7T6W227Skx/ONVhk93csY0SxJ89z9JUkysxslHSkpP7E8UtJ56c+/lHSZmZmkgyQ96+7PSJK7ryi3MRwKBwAAiMBkqrO4N0njzezJvNunM087SdKCvHhhel+X67h7u6Q3JI2TtL0kN7M7zexpMzu93NdgQO2xBAAAqBjrkz2Wy919ZuyNphok7StpL0nrJN1tZk+5+93lbBAAAABl6jx5p58tkjQlL56c3tfVOgvTuspNJa1QsnfzT+6+XJLM7A5Je0oanInlvvvuG8Sf/exnK9QS1LKhSzcE8d+Hh9dxnjhiRrh+Q3jC3JA1YUVJ+6iIjUNNyl4LvFgfoqYS0sZj0YbNh5b0+IY14Zy77aNq+iu+ZtX1/zUdn5C0nZltrSSBPE7ShzPrzJE0W9Ijko6RdI+7u5ndKel0MxsuqVXSuyR9t5zG0OsAAABisJ5PERSLu7eb2SmS7pRUL+kqd59rZudLetLd50j6X0nXmtk8SSuVJJ9y91Vm9h0lyalLusPdby+nPSSWAAAAEVToULjc/Q5Jd2TuOyfv5/WSPtjNY3+uZMqhKEgsAQAAIqkb5JfeqenEcr/99gvikSNHFlz/xRdfDOLm5ubobULtaRvdGMYtLUHcZOODeJ8tTwhi38CsXQi1dZTWhwBp47GoVD6EsajSzKS6QX6x8JpOLAEAAKpH/9dYVhsSSwAAgBiMQ+EklgAAABFU6uSdajKgE8tnnnkmiA844IAgXrlyZX82B1WquXFZEL+w4Okg3mqTPYN4tG0fxB2lTTWHAWidvx7ELywvrQ8BkpQrs0ayY+jgru2rFtRYAgAAoHxmMg6FAwAAoFwcCiexBAAAiGawJ5bm7t0vNOt+Iaqeu1dF725paaEf1bCmpqaK9yP6UG2rhj4k0Y9qXbX0o0K2mD7Cj71kl6jb/MHRjz/l7jOjbrQPsccSAAAgCuaxJLEEAACIhHksAQAAULbkko4klt2qlho91LZaqItBdaMPIQb6EfoD81gCAACgbCbjUHilGwAAADAgcCicxBIAACAWEksAAACUzSTVGTWWAAAAKJOZqaF+cKdWg/u3BwAAiMY4K7zSDQAAABgITFK91Ve6GRU1YNJqM/uZmV2Y/ryfmT3fy+1cYWZnx20dAAAY8MxUV1cf9VZr+jWxNLNXzKzFzJrN7PU0GRwZ+3nc/QF336EH7fm4mT2YeexJ7n5B7DZ18dyzzewpM3vTzBaa2SVmxh5kAABqWJ3VR73VmkrssTzc3UdK2lPSTElnZVcYJAnWcEn/I2m8pL0lHSDpS5VsEAAA6D1Layxj3mpNxVrs7osk/U7SLpJkZm5mJ5vZC5JeSO87zMz+YmarzexhM9u18/FmtoeZPW1ma8zsJknD8pbtb2YL8+IpZnarmS0zsxVmdpmZ7STpCknvSPegrk7XfeuQehqfaGbzzGylmc0xs4l5y9zMTjKzF9I2/tCsZ1Puu/vl6Z7V1vS1uE7SPr14KQEAQDUwU73VR7317GntEDN7Ps1Xzuhi+VAzuyld/piZTcssn5rmQmXv4KpYYmlmUyS9T9Kf8+4+SsneuxlmtoekqyR9RtI4ST+WNCd9cYZIuk3StZLGSrpZ0tHdPE+9pN9Kmi9pmqRJkm50939IOknSI+4+0t1Hd/HYd0u6WNKHJE1It3FjZrXDJO0ladd0vYPTx05Nk82pPXxJ3ilpbg/XBQAAVcakfq+xTPOcH0p6r6QZkv7TzGZkVvuUpFXuPl3SdyV9I7P8O0p29pWtEonlbenewQcl3S/pa3nLLnb3le7eIunTkn7s7o+5e4e7Xy1pg6S3p7dGSd9z9zZ3/6WkJ7p5vlmSJko6zd3Xuvt6d3+wm3Wzjpd0lbs/7e4bJH1FyR7OaXnrfN3dV7v7q5LulbS7JLn7q+4+Or2/IDP7pJKygG/1sF0AAKDqmOqsLuqtB2ZJmufuL7l7q5IdYEdm1jlS0tXpz7+UdEDnEVYzO0rSy4q0c6sStYxHufsfu1m2IO/nrSTNNrPP5t03REmS6JIWubvnLZvfzTanSJrv7u29aOtESU93Bu7ebGYrlOz1fCW9e0ne+usklXQyUvqGXizpQHdf3os2AgCAKmCmvjiTe7yZPZkXX+nuV+bFkxTmTwuVHP1VV+u4e7uZvSFpnJmtl/RlSe9RpPM8qu0kmfxEcYGki9z9ouxKZvYuSZPMzPKSy6mSXuximwskTTWzhi6SS+9i/XyvKUlwO593hJLD8ouKPK5HzOwQST+RdKi7/zXGNgEAQKVYX8xjudzdZ8beaOo8Sd9Nd5xF2WA1n270E0knmdnelhhhZoea2ShJj0hql/Q5M2s0sw8o2RXclcclLZb09XQbw8ys8ySZ1yVNTms2u3KDpE+Y2e5mNlTJYfvH3P2Vcn+5tH7zOklHu/vj5W4PAABUVue1wvv5UPgiJUdnO03WxjvA3lonnXlnU0krlOzZvMTMXlEyU82ZZnZKOa9B1SaW7v6kpBMlXSZplaR5kj6eLmuV9IE0XinpWEm3drOdDkmHS5ou6VUlu4iPTRffo6SmYImZbXQYOj1kf7akW5Qkp9tKOq4n7c87w6q7k3fOVvLG3pGu12xmUQpnAQBAJVRkgvQnJG1nZlunO8qOkzQns84cSbPTn4+RdI8n9nP3ae4+TdL3JH3N3S8r6xUIyxQBAADQG9N23szPvb7LSWp67ZO7//ipYofCzex9ShLDeiUnHV9kZudLetLd55jZMCUz6eyhZIfcce7+UmYb50lqdveyTiSuthpLAACAmmQy1VfgMozufoekOzL3nZP383pJHyyyjfNitIXEEgAAIILOGsvBjMQSAAAgBrO+mG6oppBYAgAARGGqiz/dUE0pmFiaGWf21DB3jzMpVZlaWlroRzWsqamp4v2IPlTbqqEPSfSjWlct/aiQ5JKOHAoHAABA2fpkgvSaQmIJAAAQgZk4FF7pBgAAAAwMnLxDYgkAABAB0w2RWAIAAMRhlZkgvZqQWAIAAESQ7LEksQQAAEDZjEPhlW4AAADAQMEeSwAAAJTNuPIOiSUAAEAUZjISy9r1pS99KYibmpqCeNdddw3iY445puD2Lr/88iB+5JFHgvjaa68ttYmoAfUbckG8YWhLEL+ZezWIn13+uyBe07I6iA/d6qwgbmhvDOJcQ9VflQxlatfaIC63D9VpSLzGoWplx6KOoXFr9erawytaMhb1DfZYAgAAoGwmU51ILAEAABABeywBAABQNk7eqbHE8qabbgriYjWTWblcruDyz3zmM0F84IEHBvH9998fxK++GtZNoTa0K6yhfK7t7iCePvSAIB5VNyWI37757CAeqtFBnFNbELc2rAviRh8RxE6ZU83J9qF5zZk+NDJuH8rWbDYo7EMYGGLXVGZlayotLLlkLIqCk3dqKrEEAACoZtRYAgAAoGxmXHmHxBIAACASaiyrWLk1lc8991wQ33nnnUG8zTbbBPHhhx8exNtuu20QH3/88UF88cUXl9QeVEZOrUH8pyWXBfHbt/hIEP/g8TDeZvMdg/g/tjoxiOc1h7W300bOCmJXRxA3WDjfqjS4/7qtBdXWh0bRhxABNZV9gZN3qjqxBAAAqBXGyTv8mQsAABBLneqj3nrCzA4xs+fNbJ6ZndHF8qFmdlO6/DEzm5be/x4ze8rM/pr+/+5yf3/2WAIAAETR/4fCLdlF+kNJ75G0UNITZjbH3f+et9qnJK1y9+lmdpykb0g6VtJySYe7+2tmtoukOyVNKqc9VZVYzpw5M4jf//73F1x/7ty5QXzEEUcE8fLly4O4ubk5iIcMCa+/++ijjwbxbrvtFsTjxo0r2B5Uh2w9WrO/FsS7b35YEA+x0UH8xT1uDjdoYSFSLhP/26hpQdzwRjgHYcfwcJDJhZcORxUqtw99YdYtJT3fLpk+VFzhOXkxMNS1Zd7n7FiUnZcy8/jiYxEHLWMzq8jJO7MkzXP3l5I22I2SjpSUn1geKem89OdfSrrMzMzd/5y3zlxJTWY21N039LYxVZVYAgAA1K4+2WM53syezIuvdPcr8+JJkhbkxQsl7Z3ZxlvruHu7mb0haZySPZadjpb0dDlJpURiCQAAEEmfnLyz3N1nFl+t98xsZyWHxw8qd1sklgAAABGYKnLlnUWS8q8bOzm9r6t1FppZg6RNJa2QJDObLOlXkj7m7i+W25iqSiwnTJgQxJapJ8nWVB588MFBvHjx4pKe74tf/GIQz5gxo+D6t99+e0nbR2Vk6+Mefe36IJ686c5BPH5YOMdgrrG0yd1ymedTLrwAb936sE7KG8KPHXPJVZ9S+9DYkTtFfT4r+sVEbdxAVNcejh2l1kB69o6iY1G4fcaiGCpy5Z0nJG1nZlsrSSCPk/ThzDpzJM2W9IikYyTd4+5uZqMl3S7pDHd/KEZjqiqxBAAAqFWVOHknrZk8RckZ3fWSrnL3uWZ2vqQn3X2OpP+VdK2ZzZO0UknyKUmnSJou6RwzOye97yB3X9rb9pBYAgAARFGZK++4+x2S7sjcd07ez+slfbCLx10o6cKYbSGxBAAAiKR4KcvAVlWJ5W9+85sgnj59ehCvWbMmiFeuXFnW8x133HFB3NjIBIMDQZ3C+UkP3vIr4QrZueDKLIdpXBPWMWXrltpHhR+zDq0P4noNK68BiC7bhw6adGafPl+pX0T0odpkHZkqyCLzUpaqYU17EBcbixCfca3w6kosAQAAaheJJYklAABAJDbIZ20gsQQAAIhmcM/bVNWJ5fz586Nu77TTTgvi7bffvuD6jz32WMEYtcHr437Ic2oN78jMWdY6MpwrLntQpCFTD7fR3HMY8LJ9KFvTWWw5NZW1KfZYtJHM5ttHNhRavBHGohiMPZaVbgAAAMBAYEpSy8GMxBIAACAa9lgCAAAgAvZYDmCHHXZYEJ9//vlBPGRIWLe0dGl4BaOvfCWc/3DdunURW4daka13G7YiXN4xMvzrtL4jU1WZ+eOV6/EOPsVqJrOKLQckaciKsF91ZGsqs/Nm1oWDD2NRXzBZ/18rvKoM6MQSAACgfw3ujJ3EEgAAIBLOCgcAAEAERo1lpRvQl2bOnBnE2ZrKrJtuuimI77///uhtQu2p7wivIb/+5WVBPHSb0UHcMTb7sRrcgwx6UzOZy8SDew8IEtmayfUvrw7ijccianUrY3B/Xgd0YgkAANBfmMeSxBIAACASk210vbXBhcQSAAAgEk7eGUBuu+22ID7ooIMKrn/NNdcE8VlnnRW7SahBDW+2BXFdc3u4wts2D8I2ox4O5aIPYWOlj0V93SL0BIfCAQAAEIFpsP9hSGIJAAAQQXLyDoklAAAAIuBQeA2bMGFCEP/7v/97EA8dOjSIly9fHsQXXnhhEDc3N0dsHWpFXXs4N9wbd80L4jHv2iaIrS2sh3OmigMQQflj0eDeU1YVzCSuFQ4AAIAY2GMJAACACGzQ11gO7t8eAAAgKot868Ezmh1iZs+b2TwzO6OL5UPN7KZ0+WNmNi1v2VfS+583s4N7+Uu/pab3WN5yyy1BPG7cuILr//znPw/iF198MXqbUHuW2twgbvhAWJu7zsK54xo7skWVg/uwB6TlHX8L4oa6sA+NtInhco3IbIG/8SHlGsKxZMTROwVxa2b97LXDUR36e4+lmdVL+qGk90haKOkJM5vj7n/PW+1Tkla5+3QzO07SNyQda2YzJB0naWdJEyX90cy2d/eO3raH0QwAACCCzmuFx/zXA7MkzXP3l9y9VdKNko7MrHOkpKvTn38p6QAzs/T+G919g7u/LGleur1eI7EEAACIIvZhcJOk8Wb2ZN7t05knnSRpQV68ML2vy3XcvV3SG5LG9fCxJanpQ+EAAABVw9NbXMvdfWb0rfaRmkosjzjiiCDec889C65/3333BfG5554bu0moQfUbwrnf5rc8HcQzNj0kiE31mQ1QU4nQ/DUl9iFAG49FHUNLPIjIWFSFXOb9Xvu6SNKUvHhyel9X6yw0swZJm0pa0cPHloRD4QAAALF45FtxT0jazsy2NrMhSk7GmZNZZ46k2enPx0i6x909vf+49KzxrSVtJ+nxkn/nPDW1xxIAAKCq9fMOS3dvN7NTJN0pqV7SVe4+18zOl/Sku8+R9L+SrjWzeZJWKkk+la73C0l/l9Qu6eRyzgiXSCwBAADi6f9D4XL3OyTdkbnvnLyf10v6YDePvUjSRbHaUtWJZXZeyjPPPDOIGxsbCz7+L3/5SxBzLfDByTKf8YeafxrEM8YeFK6fqRDJxswch0dXXBnEpfYhDE7ZsajkmsoMxqIq5Bu/z4NNVSeWAAAANYXEEgAAAFFU4FB4NSGxBAAAiGVw55XVnVh+8YtfDOK99tqr4Pq33XZbEDNvJSTJ2sO54r75i+8FcUPj94P45o/9NYhzQ6iPQ6hYH7rphPD684C08VjkjYXHlrrWcH3GItSCqk4sAQAAaoarEhOkVxUSSwAAgFgGd15JYgkAABANiWX1OvXUU0ta/5RTTgli5q2EJDU3vB7El594QxD/5OEvhw+o4/q7CK3zxUFctA8BXcgVqancCGNRbeJQOAAAAGJggnQAAACUz8Wh8Eo3AAAAYMAgsRw4xo4dG8RtbW1lbe+NN94ouL3stco33XTTgtsbPXp0EJdaQ9rR0RHEX/5yWNe1bt26krY3WDTYiCDevGG3IP7MPt8K4o76cO44U1jn5JlRo87rwycsUhaVHXM2Wj2zQnYuu+zy1UMXZLYXtmeYjQviDb4qiEdqYrh5yro2UmofcoWf1WJ9KPue9bc1Pj+IS+1DIyzsQ4gj11Dah3GjQ7D9PBZ1DCtvns1s+2tzLHJqLCvdAAAAgIHARI0l0/gDAAAgCvZYAgAAxMKh8IHj2Wefjbq9m2++OYgXLw7nsttiiy2C+Nhjj436/MUsWbIkiC+66KJ+ff5aMVSbBHGrwvlNN6mfHMRDVod1RIuu+FMQb3bG/kG8TkuDuCW3IohH100P4jqFtbnKlC29sP4PQfyrZ34QxCvfWBPEFx1yVxCv95Vh+zycx7M1l5nfNXPcYoSol8sakulDbUX6UPGaybAGs8UzfchL7EMZ89ZVuA9Rc9mlbMlgsfSjcXVY158dizY/44Bwe+XWJGZLKDPva8fQvj3IWZs1lRmcFT6wEksAAIBKGuw1liSWAAAAsXAoHAAAAFEM7rxS5gUya7PK7tC99dZbg/jII4+sUEt6p729PYhzuVw3aybmzJkTxE8++WTB9R944IEgfvTRR4PYvToqVlpaWirajxrWhO9Dx/Cw/i3by+tawvq39aM2ZLZYl4kaMnG4/TaF84u2+dogblcYL1n3fBC/0RrW0v5x7i+CePWacPufPzCcU3FC454F25cdBLO9pqmpqeL9qNJ9qFzZ97jW+1Cp825WQx+SKt+Pyh2L2kcNrH1BxebNrMaxqJiZ/7aHPz7nnqjbrN9m7FPuPjPqRvvQwOqlAAAAFeMqtMNuMCCxBAAAiKXwwckBj8QSAAAgAnfJc4N7j2VV11hmnX766UGcvVZ3MTvvvHMQlzrv5FVXXRXEr7zySsH1b7nlliB+7rnnSnq+clFj2bVWvRnE2es6N2pkENdn5wzM/DZrFF6re7iF85tm69Gy142uaw9jawv/3M0NDR+fnVuur1VDXVOt96Fi8042e3l9qNovolYNfUiqvn5UqlJrEktV1565hj1jUcnetsvu/tgtf4y6zcYdN+t1jaWZjZV0k6Rpkl6R9CF3X9XFerMlnZWGF7r71WY2XNLNkrZVMvnub9z9jGLPWd2jEQAAQA3xnEe9lekMSXe7+3aS7k7jQJp8nitpb0mzJJ1rZmPSxd9y9x0l7SFpHzN7b7EnJLEEAACIwSXlPO6tPEdKujr9+WpJR3WxzsGS7nL3lenezLskHeLu69z9Xkly91ZJT0ua3MXjA9RYAgAARNEnZ4WPN7P8+QevdPcre/jYLdy983rUSyRt0cU6k6Sgpmthet9bzGy0pMMlfb/YE9ZUYnnJJZdE3d6HP/zhqNtDbche97mYjYaITJXPSE0pqz257KewobQ5AtH/Su1DxYy08voQBodiY1G5cg2ZDTIW9U78s8KXF6qxNLM/Stqyi0VfzQ/c3Xtz7oyZNUi6QdKl7v5SsfVrKrEEAACoZv09j6W7H9jdMjN73cwmuPtiM5sgaWkXqy2StH9ePFnSfXnxlZJecPfv9aQ91FgCAADEUH01lnMkzU5/ni3p112sc6ekg8xsTHrSzkHpfTKzCyVtKul/evqEJJYAAACRVNlZ4V+X9B4ze0HSgWksM5tpZj+VJHdfKekCSU+kt/PdfaWZTVZyOH2GpKfN7C9mdkKxJ+RQOAAAQCxVdElHd18h6YAu7n9S0gl58VWSrsqss1C9qOQlsQQAAIjAPcpexppGYgkAABAL1woHAABADP19Vni1IbEEAACIofOs8EGMxBIAACAWEksAAADEwKFwAAAAlM/FyTuVbgAAAMDAwHRDJJYAAACxcCgcAAAAZXOxx7LSDQAAABgwSCwBAABQLhdnhZNYAgAAxOAub+uodCsqisQSAAAgBmosSSwBAADicHnH4J7IsmBi6e7WXw3BwNXU1EQ/QlnoQ4iBfoQ+55JyJJYAAAAok0vyDg6FAwAAoFzucvZYAgAAIAZqLAEAAFA+aixJLAEAAOJwphuqdAMAAAAGBOdQOIklAABABC4N+pN36irdgFjM7GdmdmH6835m9nwvt3OFmZ0dt3UAAGDAc5c6cnFvNaZfE0sze8XMWsys2cxeT5PBkbGfx90fcPcdetCej5vZg5nHnuTuF8RuUxfPPdTMvmtmr5nZKjP7kZk19vXzAgCAvuM5j3orh5mNNbO7zOyF9P8x3aw3O13nBTOb3cXyOWb2t548ZyX2WB7u7iMl7SlppqSzsiuY2WA4RH+Gkt9/F0nbK3k9NnotAABAjUhrLGPeynSGpLvdfTtJd6dxwMzGSjpX0t6SZkk6Nz8BNbMPSGru6RNW7FC4uy+S9DsliZXMzM3sZDN7QdIL6X2HmdlfzGy1mT1sZrt2Pt7M9jCzp81sjZndJGlY3rL9zWxhXjzFzG41s2VmtsLMLjOznSRdIekd6R7U1em6bx1ST+MTzWyema1MM/aJecvczE5KM/zVZvZDM+vpJcMOl3Spu69092WSLpX0yRJfRgAAUDWSCdJj3sp0pKSr05+vlnRUF+scLOmuNB9ZJekuSYdIUnpU+VRJF3bxuC5VLLE0symS3ifpz3l3H6UkY55hZntIukrSZySNk/RjSXPSQ8hDJN0m6VpJYyXdLOnobp6nXtJvJc2XNE3SJEk3uvs/JJ0k6RF3H+nuo7t47LslXSzpQ5ImpNu4MbPaYZL2krRrut7B6WOnpsnm1EIvQ+bnyWa2aYH1AQBAtXJJHR73Jo03syfzbp8uoUVbuPvi9OclkrboYp1JkhbkxQvT+yTpAknflrSup09YiUPOt5lZu6Q3JN0u6Wt5yy5295WSlL5wP3b3x9JlV5vZmZLeruSta5T0PXd3Sb80s1O7eb5ZkiZKOs3d29P7Huxm3azjJV3l7k+nbfqKpFVmNs3dX0nX+bq7r5a02szulbS7pN+7+6uSRhfY9u8lfT59TL2kz6X3D1fy2gAAgBrTB2eFL3f3md0tNLM/Stqyi0VfDdrl7mbW46JNM9td0rbu/gUzm9bTx1UisTzK3f/YzbL8jHkrSbPN7LN59w1RkiS6pEVpUtlpfjfbnCJpfl5SWYqJkp7uDNy92cxWKMnkX0nvXpK3/jpJPT0Z6SIliedfJG2Q9BNJe0h6vRftBAAAFebu/T6Ppbsf2N2y9ETpCe6+2MwmSFraxWqLJO2fF0+WdJ+kd0iaaWavKMkXNzez+9x9fxVQbdMN5SeKCyRd5O6j827D3f0GSYslTcrUM3Z3yHmBpKndnBBULHN/TUmCK0kysxFKDssvKvaLFOPuLe5+irtPcvdtJK2Q9JS7197cAgAAQJKqrcZyjqTOs7xnS/p1F+vcKekgMxuTnrRzkKQ73f1yd5/o7tMk7Svpn8WSSqn6Est8P5F0kpntbYkRZnaomY2S9IikdkmfM7PG9IylWd1s53EliejX020MM7N90mWvK6lrHNLNY2+Q9Akz293Mhio5bP9Y3mHwXjOzSWY2Mf3d3i7pbCVnZQEAgFrUNzWW5fi6pPekJ0YfmMYys5lm9lNJSksQL5D0RHo7v7MssTeqdlofd3/SzE6UdJmk7SS1KKmN/JO7t6bJ5E+UnKl0h6Rbu9lOh5kdruSs61eVvO3XS3pI0j2S5kpaYmY5dx+feewfLZks/RZJYyQ9LOm4nrQ/PWnn75JmpPWWWdtKukbS5kr2qp7h7n/oybYBAEA18qq68o67r5B0QBf3PynphLz4KiUnTHe3nVeUzuJTjIVligAAAOiNXSdu57858btRtznt/MOfKnTyTrWp2j2WAAAAtaW69lhWAoklAABADJ01loMYiSUAAEAErj6Zx7KmkFgCAADE4C5vJ7HsVikztKP6uHtPr1vep1paWuhHNaypqani/Yg+VNuqoQ9J9KNaVy39qCBXv0+QXm3YYwkAABCDiz2WlW4AAADAwND/l3SsNiSWAAAAETh7LEksAQAAouDkHRJLAACAWJx5LAEAAFA2DoWTWAIAAEThnLxDYgkAABABJ++QWAIAAMTByTsklgAAALFwKBwAAADl41A4iSUAAEAUHAonsQQAAIjBxaFwEksAAIAY2GNJYgkAABCFs8eSxBIAACAGTt4hsQQAAIiDQ+EklgAAABG4S97hlW5GRdVVugEAAAADQnooPOatHGY21szuMrMX0v/HdLPe7HSdF8xsdt79Q8zsSjP7p5k9Z2ZHF3tOEksAAIAoXN6Ri3or0xmS7nb37STdncYBMxsr6VxJe0uaJencvAT0q5KWuvv2kmZIur/YE3IoHAAAIIbqO3nnSEn7pz9fLek+SV/OrHOwpLvcfaUkmdldkg6RdIOkT0raUZLcPSdpebEnJLEEAACIoW/msRxvZk/mxVe6+5U9fOwW7r44/XmJpC26WGeSpAV58UJJk8xsdBpfYGb7S3pR0inu/nqhJySxBAAAiMD7Zh7L5e4+s7uFZvZHSVt2seirYdvczayUM4saJE2W9LC7n2pmp0r6lqSPFnsQAAAAIujvCdLd/cDulpnZ62Y2wd0Xm9kESUu7WG2R/nW4XEqSyfskrZC0TtKt6f03S/pUsfZw8g4AAEAUrpznot7KNEdS51nesyX9uot17pR0kJmNSU/aOUjSne7ukn6jfyWdB0j6e7EnZI8lAABABO5SzqtqHsuvS/qFmX1K0nxJH5IkM5sp6SR3P8HdV5rZBZKeSB9zfueJPEpO9LnWzL4naZmkTxR7QhJLAACASDrK38sYjbuvULKnMXv/k5JOyIuvknRVF+vNl/TOUp5zUCWW06dPD+Lx48cH8fvf//4g3n///YM4lws7yxVXXBHEDz30UBDPmzevN81ElavfEPaD3JCwoqTN1gbxdXO/GMRHzgjjMb5dwedzClYGnXaV1odGW+E+hIGp2FjkVtr2rEg+xFhUnKeHwgezQZVYAgAA9KUqOxTe70gsAQAAImGPJQAAAMrmzqHwAZVY7rLLLkF8yimnBPEHPvCBIM7WWJZq7733DuL29vYgfv7554P4wQcfDOLPf/7zQdza2lpWexDHRtPH5sI75uXCS6V2tLQF8aSm3YP4oztfGq6v8H3eYG8G8TOrbgniHUa/K4jHrp0abm9E+DEuta4K/e/llnuDuMPL60OtKq0PbWLb9LitqJxiY1HH0LhFj6XWUDY0h995jEUJDoUDAACgbC4OhZNYAgAAROFVNd1QJZBYAgAARJDsseRQeM3Yddddg/jkk08O4mOPPTaIN9lkk4LbW7RoURA/8MADQfzyyy8H8emnnx7ETz31VBDPmjUriMeOHRvE73vf+4L4mWeeCeLsvJjoG9k5ArN+8/KFQbzT5vsE8bUPfyeID97jyCDeommHIHbvCOLhNiGI56//UxBvNWrPzPpbBPGvV10UxIeNPFfoX0X70EvV3YduX/D/gvjQKfSharRRjWJ9dRUtto8MU4jqal2FOIfCayqxBAAAqFYuV3uuo/iKAxiJJQAAQCTssQQAAEDZnJN3ZF6gyNRso1m0+tWPf/zjIM5ey7vYPJR33313EP/1r38N4jPPPDOI169fX3B7994bzj33X//1X0F81VXh9dt33333IH799deDeOrUcD7CLbfcMoiXLVtWsD3FuFfHLGItLS0V7Ud17eHTr6lfHMRrO8LXefHafwTxng0fDGLrCAeNjeZuK1IHteHOsHa36YBp4QqZz2R7Y3hYpV0tQTysbWTm+TPXCy5zqrumpqaK96NK96Gstf5aGBfpQ7ttelyft6mQnMJ5MrN9aIgK16OXqxr6kFT5fpQdi3INpb0s9WvDsaC/x6JcY+HBpK4tbM9AHIuK2bpuvJ877LCo2/xEy9VPufvMqBvtQ+yxBAAAiICzwkksAQAAIuGSjiSWAAAAEXDlnQonlsOGDQvi7DyRJ5xwQhCbheUV2RrEyy+/PIi/+c1vBvHatYXnnitm3LhxQVxfXx/E5513XhD//ve/D+KtttqqrOdHzwxduiGIX/rWnUE8+eLDg3jM0jFBvPnE3YI4c3leuYXve6mGHbx1ELdnrvtc39EYxB0Kf5+mdSOCeMPwsF6uQeFyxDfCJoZxQxhvvmnYh/paLtOH6jQkiLN9KFtTmZ2Xkz4UR6ljUePrYZ1/68SmIM4ND8ee2GNRNh2yIrPm1K8LV+gYXl57BgoOhQMAAKBs7pwVTmIJAAAQCYfCAQAAEAWHwito//33D+LTTjstiLM1ldlrex999NFB/Pjjj5fVnmzN5JQpU4L4mmuuCeI77rgjiMeMCWv1srK/z7XXXhvEq1ev7kkzkdGq1UH8aP0vgvjtl3w6iO9f9v0gPmDEfxfcfrmzgXYorJvKqT2IW3xpEI9bE/a7hrXh5G8dm4UNGtoS1irnwrIsDe4hrmeyfejpFZk+NC7sQ3/K9KF3bvb5PmlXp1L70Ca2TRA3KpzrNIuayr6xYfOhQTzpkiOCOPvZ9BGFv5Jjz0xcbGxoWBP2M60N50Pt2Cwce+pbwprLXFOmJrSk1tUm56xw9lgCAADEsFCr7zzVby189ZbSLY+8vT5FYgkAABCBux9S6TZUWpkXWAIAAAASFd1jma1p7OgoPGlWe3tY77H33nsH8THHHBPEO+64Y8HttbSE8//ttNNOBePly8O90VtssUXB7WdlrxV+4YUXBnFbW1i/gp4ZsXJ4EL+z7WNBfMv8c4L4nG9fF8TPfOdzQfxc82+DeOKIGUH87Iqwtna/MSeHy9f+Mohb2t8M4n1bjw/ixi3DeriOZW+E8ZvhHIUNGzLXDt8m/P2r/mK6VWiIRgdxtqbyN0X60J8vDWssS+1D+44/JYj/+ubNQZztQ7PGfiqIG61wDWUxrsw1qMV8hL3RuDL8rFprWGvXumVYk5iVvdZ3qawjrGIsdq3wIUsy82Zm2texbF0YFxmLOrYZ1aN2YmBjjyUAAACiILEEAABAFCSWAAAAiKKiNZb33HNPEN97771BfOCBBwbx1KlTg/jSSy8NYi8yKWm2hjNb41lMsZrKXC6sp/nVr34VxJ/7XFjLt3jx4pKeH11rHRu+j22ZOf+mvhHWtw0bHtYdra5/OYg3dDQXfL59x2fnvQy3t/0mB6qQVoV1TNnrNNdtF05E2dAeXtc5Vxc+HzWV5cvWGLZl3pOpowv3oTf9pSAuvQ+FivWhrI36kBozcXjt8CxqKuNoG1v4dS4m11Dmp7lITWVWsZrP3HabhptvD79js2MRILHHEgAAAJGQWAIAACAKEksAAABEYYXqEs2sopf2HD16dBCfccYZQbzPPvsE8YoVK4L41VdfDeKhQ8Prtu62225BPGvWrN408y1XXHFFEJ955plB3N/XAnePfWXZ3mlpaamqS8Q2NIfzoZ771KFBPHncVkF89M7h+9igsB9tsnazIK7bENbadmSul1v/Zjhf6bp5K4N46Mwtg7htWNjeYvVysTU1NVW8H1VbH8q64P6Dg7jUPjTcJvRNw1I5hfMPDsY+JFVfP8qORe0jyzvtIbu92GNRx7DK7ouqln6EwthjCQAAgChILAEAABAFiSUAAACiqOoay752zTXXBPFHPvKRguuvWbMmiE899dQg/tnPfhbExa593teosewf9evDOqYNw8Jr0DconJcyp7AOatGGJ4J4ytCwdrjSqqGuaaD3oazsvJT0oTgGej/KjkWVromMrVr6EQobWL0OAAAAFUNiCQAAgChILAEAABBFRa8V3t9OP/30ID7uuONKevxJJ50UxDfccEPZbULtqWsN65isLYwbhoX1cBv0RhA3akQQV1s9HCovW1NJH0JXio1FGmA1lqgN9DoAAABEQWIJAACAKEgsAQAAEMWArrE84YQTgviss84K4oaGwr/+3Llzg/jWW2+N0zDUlPrM9XbVEU6F1zEq7Ec5hdffNWWuz5u5TvOAnlgPvZJTOAdutg/197W+UR1KHYuKyU4KyViEGNhjCQAAgChILAEAABAFiSUAAACiGFA1lrNmzQrib3/720E8cuTIgo9vbm4O4uy8lRs2bCijdagVnqlvax6yIog71BrEm7RPCpc3hP2kQcMy28dAl+1D671wHxppkzPLC/chDE4dQwvvC6prD0cXbyh8aW3GIvQF9lgCAAAgChJLAAAAREFiCQAAgCgGVI3l4YcfHsSjRo0quP7atWuD+Igjjgjihx56KE7DUFMaWsJ4+LDNg7jd1gdxa0MYD/GwltcLlzlhAMrOO9lkYR/qUNhn2hV2ukYVrgfH4FDfEtbq5oaF/So7tuQyNZWWKaJkLEJ/YI8lAAAAoiCxBAAAQBQklgAAAIiipmssszWUp59+ekmPv+6664L4vvvuK7dJqEGWufxuXWv2jrAwae2QJUG8qU0LYuqYUMxaD/vQJpk+hMGp1LEoO6/lRtf+ZixCBbDHEgAAAFGQWAIAACAKEksAAABEYe7dXy3ULDsLVmVlr/X9j3/8I4gnTQqv2Zz17LPPBvHb3/72IF6/Ppxbrta5V0eFTUtLS1X1IymsW1re8fcgHlY3OohH2IQgrm8P/x7LXo+3yn7ZsjU1NVW8Hw20PpSd53Kgq4Y+JFVjPypPsWuDD6hfVtXTj1AYeywBAAAQBYklAAAAoiCxBAAAQBQ1NY/lu9/97iCePHlyEBeqF5WkL3zhC0E80Goq0TOWC/+eWtO6LIiHDRsdxBt8VRAP17ggHmh1TOiJ8vrQMBvfJ61CbcnOW+ll7uphLEI1YI8lAAAAoiCxBAAAQBQklgAAAIiipmosL7jggiAuVlP5zW9+M4jvvffe6G1C7Wmx5UG8oWNtEM95/pIgPmqHrwRxroGp1Aa79V5eHwKk8msqGYtQjdhjCQAAgChILAEAABAFiSUAAACiqKkay7FjxwaxWVhfsnTp0iD+3ve+19dNQg1q1ZtBPHHEjCAesfWYMM5c55m54lBqHxqe6UNAb2QrKhmLUI3YYwkAAIAoSCwBAAAQBYklAAAAoqipGsvvfOc7BePsPJeLFy/u8zah9gzRJkG8aP1fgnj6kAOCmDomZBXrQ9s2HdiPrcFAZR3h6OP1zFuJ6sceSwAAAERBYgkAAIAoSCwBAAAQhRW63raZUV5Ww9y9KgpyWlpa6Ec1rKmpqeL9iD5U26qhD0n0o1pXLf0IhbHHEgAAAFGQWAIAACAKEksAAABEUbDGEgAAAOgp9lgCAAAgChJLAAAAREFiCQAAgChILAEAABAFiSUAAACiILEEAABAFP8fRah0Zg5vZqIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x504 with 13 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=3, ncols=4, figsize=(10, 7))\n",
    "image_ids = [0, 1, 9]\n",
    "cmap_bound = np.abs(attrs[[0, 1, 9]]).max()\n",
    "\n",
    "for row, image_id in enumerate(image_ids):\n",
    "    # original images\n",
    "    ax[row, 0].imshow(X_test[image_id].squeeze(), cmap='gray')\n",
    "    ax[row, 0].set_title(f'Prediction: {predictions[image_id]}')\n",
    "    \n",
    "    # attributions\n",
    "    attr = attrs[image_id]\n",
    "    im = ax[row, 1].imshow(attr.squeeze(), vmin=-cmap_bound, vmax=cmap_bound, cmap='PiYG')\n",
    "    \n",
    "    # positive attributions\n",
    "    attr_pos = attr.clip(0, 1)\n",
    "    im_pos = ax[row, 2].imshow(attr_pos.squeeze(), vmin=-cmap_bound, vmax=cmap_bound, cmap='PiYG')\n",
    "    \n",
    "    # negative attributions\n",
    "    attr_neg = attr.clip(-1, 0)\n",
    "    im_neg = ax[row, 3].imshow(attr_neg.squeeze(), vmin=-cmap_bound, vmax=cmap_bound, cmap='PiYG')\n",
    "    \n",
    "ax[0, 1].set_title('Attributions');\n",
    "ax[0, 2].set_title('Positive attributions');\n",
    "ax[0, 3].set_title('Negative attributions');\n",
    "\n",
    "for ax in fig.axes:\n",
    "    ax.axis('off')\n",
    "\n",
    "fig.colorbar(im, cax=fig.add_axes([0.95, 0.25, 0.03, 0.5]));"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}