examples/integrated_gradients_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrated gradients for MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we apply the integrated gradients method to a convolutional network trained on the MNIST dataset. \n",
    "Integrated gradients defines an attribution value for each feature of the input instance (in this case for each pixel in the image) by integrating the model's gradients with respect to the input along a straight path from a baseline instance $x^\\prime$ to the input instance $x.$\n",
    "\n",
    "A more detailed description of the method can be found [here](https://docs.seldon.io/projects/alibi/en/latest/methods/IntegratedGradients.html). Integrated gradients was originally proposed in Sundararajan et al., [\"Axiomatic Attribution for Deep Networks\"](https://arxiv.org/abs/1703.01365)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TF version:  2.2.0\n",
      "Eager execution enabled:  True\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Activation, Conv2D, Dense, Dropout\n",
    "from tensorflow.keras.layers import Flatten, Input, Reshape, MaxPooling2D\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from alibi.explainers import IntegratedGradients\n",
    "import matplotlib.pyplot as plt\n",
    "print('TF version: ', tf.__version__)\n",
    "print('Eager execution enabled: ', tf.executing_eagerly()) # True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading and preparing the MNIST data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28, 1) (60000, 10) (10000, 28, 28, 1) (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "train, test = tf.keras.datasets.mnist.load_data()\n",
    "X_train, y_train = train\n",
    "X_test, y_test = test\n",
    "test_labels = y_test.copy()\n",
    "train_labels = y_train.copy()\n",
    "                         \n",
    "X_train = X_train.reshape(-1, 28, 28, 1).astype('float64') / 255\n",
    "X_test = X_test.reshape(-1, 28, 28, 1).astype('float64') / 255\n",
    "y_train = to_categorical(y_train, 10)\n",
    "y_test = to_categorical(y_test, 10)\n",
    "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a convolutional neural network on the MNIST dataset. The model includes 2 convolutional layers and it reaches a test accuracy of 0.98. If `save_model = True`, a local folder `./model_mnist` will be created and the trained model will be saved in that folder. If the model was previously saved, it can be loaded by setting `load_mnist_model = True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_mnist_model = False\n",
    "save_model = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = './model_mnist/'  # change to directory where model is saved\n",
    "if load_mnist_model:\n",
    "    model = tf.keras.models.load_model(os.path.join(filepath, 'model.h5'))\n",
    "else:\n",
    "    # define model\n",
    "    inputs = Input(shape=(X_train.shape[1:]), dtype=tf.float64)\n",
    "    x = Conv2D(64, 2, padding='same', activation='relu')(inputs)\n",
    "    x = MaxPooling2D(pool_size=2)(x)\n",
    "    x = Dropout(.3)(x)\n",
    "    \n",
    "    x = Conv2D(32, 2, padding='same', activation='relu')(x)\n",
    "    x = MaxPooling2D(pool_size=2)(x)\n",
    "    x = Dropout(.3)(x)\n",
    "    \n",
    "    x = Flatten()(x)\n",
    "    x = Dense(256, activation='relu')(x)\n",
    "    x = Dropout(.5)(x)\n",
    "    logits = Dense(10, name='logits')(x)\n",
    "    outputs = Activation('softmax', name='softmax')(logits)\n",
    "    model = Model(inputs=inputs, outputs=outputs)\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                  optimizer='adam',\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    # train model\n",
    "    model.fit(X_train,\n",
    "              y_train,\n",
    "              epochs=6,\n",
    "              batch_size=256,\n",
    "              verbose=1,\n",
    "              validation_data=(X_test, y_test)\n",
    "              )\n",
    "    if save_model:\n",
    "        if not os.path.exists(filepath):\n",
    "            os.makedirs(filepath)\n",
    "        model.save(os.path.join(filepath, 'model.h5'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate integrated gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IntegratedGradients class implements the integrated gradients attribution method. A description of the method can be found [here](https://docs.seldon.io/projects/alibi/en/latest/methods/IntegratedGradients.html).\n",
    "\n",
    "In the following example, the baselines (i.e. the starting points of the path integral) are black images (all pixel values are set to zero). This means that black areas of the image will always have zero attribution.\n",
    "The path integral is defined as a straight line from the baseline to the input image. The path is approximated by choosing 50 discrete steps according to the Gauss-Legendre method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize IntegratedGradients instance\n",
    "n_steps = 50\n",
    "method = \"gausslegendre\"\n",
    "ig  = IntegratedGradients(model,\n",
    "                          n_steps=n_steps, \n",
    "                          method=method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate attributions for the first 10 images in the test set\n",
    "nb_samples = 10\n",
    "X_test_sample = X_test[:nb_samples]\n",
    "predictions = model(X_test_sample).numpy().argmax(axis=1)\n",
    "explanation = ig.explain(X_test_sample, \n",
    "                         baselines=None, \n",
    "                         target=predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'IntegratedGradients',\n",
       " 'type': ['whitebox'],\n",
       " 'explanations': ['local'],\n",
       " 'params': {'method': 'gausslegendre',\n",
       "  'n_steps': 50,\n",
       "  'internal_batch_size': 100}}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Metadata from the explanation object\n",
    "explanation.meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['attributions', 'X', 'baselines', 'predictions', 'deltas', 'target'])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data fields from the explanation object\n",
    "explanation.data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get attributions values from the explanation object\n",
    "attrs = explanation.attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample images from the test dataset and their attributions.\n",
    "\n",
    "* The first column shows the original image. \n",
    "* The second column shows the values of the attributions.\n",
    "* The third column shows the positive valued attributions.\n",
    "* The fourth column shows the negative valued attributions.\n",
    "\n",
    "The attributions are calculated using the black image as a baseline for all samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApYAAAGaCAYAAAC12APsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZwcVbn/8e8zayaZhJAEIiFAkOhlkzWAAl5REJF9FRSV6yUgV3BDQFQUhCCKolx/sopcJAgCotyoKIIKCgKyXFDCImHNCoQkZJskszy/P6pG+1RmuqenT093T3/evPrFPF3VVSfdp888c+qpKnN3AQAAAKVqqHQDAAAAMDyQWAIAACAKEksAAABEQWIJAACAKEgsAQAAEAWJJQAAAKIYFomlmU0xMzezpjT+jZmdMIjtbG5mK82sMX4rUe2yn7+Z3WNm0yNuf1D9EnEM5PudLn/rULZrsMzs3Wb2bE78kpntF3H7s81sn1jbq1e19r3P/Q6Y2XVmNiPitq80s6/G2h6q05Allumg15F22lfTDttejn25+wfd/ccDbNM/B2J3f8Xd2929uxztytlv7y+43Ieb2RfKud/hKE3+lppZa85z6w2GA/mlG/PzN7PzzOyGzPYH1C+RiD1mZD/fvv5wSJe/UGrbi9FXO9LxYGq+17n7n9393yK1Yb3vjLtv5+73xNj+UEr7zWtmNirnuelmds8Q7Lsqvvd9tWOgfyjH+g6Y2X+Y2X2ZbZ/i7heUum1Ut6GesTzE3dsl7SJpmqRzsitYYljMpPYn5xdce/p+vENSj6TbKty0mmJmUyS9W5JLOrTEbTVFaBLiKzhm1CP6a0GNkj5b6UbUEvoUonH3IXlIeknSfjnxtyX9Kv35HkkXSrpfUoekqZI2kPQjSQslzZc0Q1Jjun6jpO9IWizpBUmnKkkumnK2Nz1nXydJelrSCklPKfklNVNJMtchaaWksyRNyWxnkqRZkpZImiPppJxtnifpFknXp9udLWnaIN+bcyX9cag+i+HykPS1tM98N6cvnSypU9K69HP9ZYHP+kRJr0j6Ux+f/z2SLpL0V0nLJf2vpHHpsn0kzeurj0s6IN1/Z7q/J7L9UskfdedIelnSa2k/2iBd1tuOE9K2LZb0lZz97C7pkbRNr0r6bqU/izJ9vi+p/zEj33ezz/cn9/NVMt50S1qTfkY/SNdxJePPHpIWKR1z0mVHSPpbzud3tqTnJb2RjgXj+vl3bCjpV5Jel7Q0/Xlyumy9dqR90SWtSp87tre/Sfpi2q6Z2T6Yvl9fUjLGLZX0P5JGpMv+Q9J9mXb1/lvX+85k339JrZIulbQgfVwqqTX3uyDpC2lfXijpEzn7OTBt0wolY/kZQ9Bvzk77xtj0uemS7slZZ2tJd6XrPCvpQznLxisZN5ZLeljJ7577cpb/t6S56fJHJb07fT7v9z59D5dJ2j5nWxspGZc2TuODJT2ervcXSTvk+XcOuB199bOcPnCqpOckvZjbL9Kfr5N0ZfperZB0r6Qtst+nnDb1/lu3SffVne5vWc72ZuSsf5KS7+8SJd/nSZn+eUratmWSLpNk6bKpaVveVDI+3lzp8YpHTt8csh2Fg9RmShKxC9L4HiW/QLdTMug3S/qFpKskjZK0sZJf7p9M1z9F0jPpdsZJ+qP6SSwlHaNkMNtNkqUdcotsm9J4SmY7f5J0uaQRknZS8ovhfemy89IvzoFKEt2LJD2Ys63LJV0+gPfFlPxy+o9Kd4Zae6QD0qck7apkEJ2YPh8MXgU+6+vTPtbWx+d/T9p3tk/XuU3SDemyfdRPYpnTP27ILM/tl/+Ztv+tktol/VzSzEzbfpi2a0dJayVtky5/QNLH0p/bJb2z0p9FmT7f3PczO2bk+272+f708/lOz+wz95fq85Len7PsVklnpz9/VtKDkiYrSRiuknRTP/+O8ZKOkjRS0uh0O7f31S/6akdOf+uS9K10f23ZPpi+X0/qX+Pi/Uq/B8qTWA7kOyPp/PTfu7GSZOgvOZ9Fb9vOVzJ2HyhptaQN0+UL9a+kZ0NJuwxFv1Hyner99/8zsVTyXZ4r6RNKft/srCQ52TZd/tP0MVLStum6uYnlR9PPtElJMr1I/0rgz1P+7/21ki7MWXaqpN+mP++sJDHfQ8nvlBPSf0trP//OQbcj0wfuSvtLWz/9YoWkf1fS7/67971QnsQyT5/7Zz+T9L70fd8l3fb/k/SnTNt+JWmspM2VfMcPSJfdJOkrSv7AGyFp76Een3j0/xjqQ863m9kySfcp+WvjGznLrnP32e7epaSTHyjpc+6+yt1fk/Q9Scel635I0qXuPtfdlyhJ6vozXdLF7v6wJ+a4+8uFGmpmm0naS9IX3X2Nuz8u6RpJH89Z7T53v8OTmq2ZShIASZK7f8rdP1VoP5L2ljRR0s8GsC5SZra3pC0k3eLujypJAj4yiE2dl/axjn6Wz3T3J919laSvSvpQpJO7jlcyk/aCu69UMtN0XOZw1NfdvcPdn1Ay69DbvzolTTWzCe6+0t0fjNCearXemDGA72as9+cmSR+WJDMbrWRMuilddoqSWeR57r5WyS/yo/s6nOjub7j7be6+2t1XKJk9es8g2tMj6Vx3X5unv/4gZ1y8sLf9ERwv6Xx3f83dX5f0dUkfy1nemS7vdPc7lMxS/VvOsm3NbIy7L3X3xyK1qZCvSfq0mW2Uef5gSS+5+/+4e5e7/5+SPxqPSb/bRyl5n1e7+1OSgvpId78h/Uy73P0SJUnRQGtdb9S/fo9JyZh1Y/rzyZKucveH3L3bk7rMtZLe2deGSmxHrovcfUmePvVrd/9T2s+/Iuld6XewVMdLutbdH0u3/aV021Ny1vmmuy9z91eUTCDtlD7fqWT8n5SOAUEtJyprqBPLw919rLtvkSZeuR15bs7PWyj5y3ehmS1Lf7FcpeSvZSk5DJa7fr5EcTMlSUexJklakv4iyN3PpjnxopyfV0saMYg6lRMk3ZYmFxi4EyT9zt0Xp/GN6XPFmlvE8peV9MsJg9hP1iSF/fZlJTMPE3Oey/av3hNXTpT0dknPmNnDZnZwhPZUq77GjELfzVjvz42SjkxPDDtS0mM5f5RuIekXOePT00oO+03MbsTMRprZVWb2spktVzLbOnYQf6C87u5rCqyT7a+TitxHf/rqr7nbfiOdFOiV21+PUpKUv2xm95rZuyK1KS93f1LJjNfZmUVbSNqj97NLP7/jJb1FyWxsk8L3MRgjzOwMM3vazN5MX7uBBj4m/FHSSDPbI02gdlJydK63XV/ItGsz9fMZltiOXAMeA9PfU0v6a1ORgj6VbvsN5f8d29unzlJytO+v6dUL/jNCexBJNRXres7Pc5X8pTYhM1j1WqjkC9dr8zzbnStpqwHsM2uBpHFmNjrnF9jmSg6NRmFmbUoO1R8Ra5v1IH3fPiSp0cx6B55WJb+sd1Tfn2t/n3W+PiCt3886lRy+WaXkUFlvmxqV/FIa6HYXKPlFkrvtLiU1gZPzvdDdn5P04fQktyMl/czMxqezqvUg73ezv/enj+3k/Yzc/Skze1nSBxXOLEnJuPKf7n7/ANr7BSUzSXu4+yIz20nS/yn5xViwHQNtbyrbXxekP2f761uK3HZvf53dx7bzcveHJR1mZs2STlNSjxpjxmsgzpX0mKRLcp6bK+led39/duX0e9yl5Dv4j/TpzXKWv1tJUrOvpNnu3mNmSzXAz9Ldu83sFiUzya8qqRnu7cNzlRwmv7DQP2qQ7Sh5DEyvyjBOyWff+0fOSCV1nlKSnA90u8EYmJ7FP14D+B3r7ouU1Gf2Hr2628z+5O5zCr0W5VeVZ1+7+0JJv5N0iZmNMbMGM9vKzHoPH90i6TNmNtnMNtT6f5HmukbSGWa2a3rG+VQz6+3MryqpceurDXOV1BFdZGYjzGwHJTMhN/S1/iAdoaTI/o8Rt1kPDlcyO7Stkr/4d1JSLP5nJYdD+/pc+/2sC/iomW1rZiOV1JD9LC19+IeSGeqD0l+Y5yhJbnP3NyXPFQ5ukvR5M9syHay/oaQAva8/pAJm9lEz28jde5QUtUvJYdK6UOi7WcT7M5A+caOSesp/V1Ib2etKSRf2jiVmtpGZHdbPNkYrOUFjmZmNU5LsFGrHYPvrqem4OE7JYcub0+efkLSdme1kZiOUHLovZn83STon/XdOUHKYueBYaGYtZna8mW3g7p1KEpAh66tponGzpM/kPP0rSW83s4+ZWXP62M3Mtkm/2z+XdF4607y1wvKn0UoSz9clNZnZ1ySNyVle6HsvJX3qWCWzpLl/rPxQ0inpbKaZ2ah0fBndxzYG047B9qkDzWxvM2uRdIGScwnmpiUR85WMkY3prGHuJM6rkianr+vLTZI+kfbJViVj4EPu/lKhBpnZMWbW+wf4UiVJbN2MgdWuKhPL1McltehfZzj+TNIm6bIfSrpTyWD5mJKBoE/ufquSWqMblRQh367kLy4pqc08Jz3scEYfL/+wkgLlBUoOV5zr7ncPpPGWXAj2ygKrnaCkhm+gMxZInCDpfzy5bNOi3oeSM2qPV3I1gW3Tz/X29DWFPuv+zFRScL5ISZH4ZyTJ3d9UcuLQNUoG11VKzozt1ZuEvGFmfdWUXZtu+0+SXlTy1/+nB9imAyTNNrOVSorpj8tTHzVc5ftuDvT9+W8ldZFLzez7/eznJiX1kH/IKbvofe0sSb8zsxVKTmzZo59tXKrkZJvF6Xq/HUA7zpP047S/fqif7fblRiV/lL+gpARohiS5+z+U/GF0t5KzbLM1aX19Z3LNUHKm/d8k/V3JuDvQC2d/TNJLlpQBnKLkOzqUzldywo4kKZ0h3F9JreMCJd/t3pOipGRWdQP96+z7m5QcQZOS3zu/VfKH5ctKvre5h5ILfe/l7g8pGS8mSfpNzvOPKJmF+4GS33lzlJwA05fBtGMg/b0vNyr5Y2iJkhMlP5qz7CRJZyo5hL2dkj/4ev1ByQz3IjPL/e5IktLv61eV1LcuVJKUHpddrx+7SXoo/Y7PkvRZH+Lrz6J/Rk4DAEDfzOxbkt7i7oOp4QbqTjXPWAIAMKTMbGsz2yE9HL27kjKLXxR6HYBENZ28AwBApY1Wcvh7kpI6wUuU3BwBwABwKBwAAABRcCgcAAAAUeQ9FG5mTGfWMHe3wmuVX0dHB/2ohrW1tVW8H9GHals19CGJflTrqqUfIT9mLAEAABAFiSUAAACiILEEAABAFCSWAAAAiILEEgAAAFGQWAIAACAKEksAAABEQWIJAACAKEgsAQAAEAWJJQAAAKIgsQQAAEAUJJYAAACIgsQSAAAAUZBYAgAAIAoSSwAAAERBYgkAAIAoSCwBAAAQBYklAAAAoiCxBAAAQBQklgAAAIiCxBIAAABRkFgCAAAgChJLAAAAREFiCQAAgChILAEAABAFiSUAAACiILEEAABAFE2V3PnRRx8dxCeddFIQL1iwIIjXrFkTxD/5yU+CeNGiRUE8Z86cUpuIGtCjziBe0v1sEI9s2CiMbWIQN3R5EHujhXEYYhgqtQ8BMTAWYThgxhIAAABRkFgCAAAgChJLAAAARGHu3v9Cs/4XRvDCCy8E8ZQpU0ra3ooVK4J49uzZJW2vVPPmzQviiy++OIgfeeSRsu7fvToqcjo6Osrajx5ddn0Qf/WabwTxjtu/JYiXrVgdxJ/+4FfC5WvC2t6mhpYg3nrD9wZxg5qDuMVG512+1t9UPq22Qbh/jcq7frm1tbVVvB/Rh+hDMZS7H6G8qqUfIT9mLAEAABAFiSUAAACiILEEAABAFBWtsdx3332DeIcddgjip59+Ooi32WabIN5ll12CeJ999gniTTfdNIjnzp0bxJttttmA2ypJXV1dQfz6668H8SabbJL39d/97neD+Iwzzihq/8WqlxrLlsVr8y73pvDvp46x4ef44OIfBfGt990YxFMmhZ/r1pN2CuKr/zdc/z8OOjKId9nkwCDu6lkXxGu7VwXxlu17BfGqnvD6rO0NYXtaNFblVA11TdVWG9eljiCmD+VXDX1Iqr6xqHNscz9roi/V0o+QHzOWAAAAiILEEgAAAFGQWAIAACCKitZYxrbhhhsG8U47hXVMjz76aBDvtttuRW0/e6/yf/zjH0GcrQkdN25cEJ966qlBfMUVVxS1/2LVS41lbOvdr7chfButJ1zeuDKs2ewa3RRuMHO/3+z63aPC9Zfr5SCev+rvQbzNqIPD9pX5U66GuqZa60OVtsLz96Gt28M+VG7V0Iek2utHlR6LCo0t2QyhHsYiFMaMJQAAAKIgsQQAAEAUJJYAAACIYljVWA61o446KohvueWWIH7yySeD+L3vDe8PvGTJkvI0LEWNZW3o1MogfnHVn4N4yqh3BXGrh9ccrIe6JvpQfsX2oXJftzKrGvqQRD+KjRpL9IUZSwAAAERBYgkAAIAoSCwBAAAQRVPhVdBr4403DuLLL788iBsawjz9/PPPD+Jy11SiNlh3WJj04pqwHu7+F38RxFO33y+Iq6NyFtUkW1NZqA8B0vpjkTcWN7gwFqEvzFgCAAAgChJLAAAAREFiCQAAgCiosSxC9l7fG220URAvXbo0iJ999tmytwm1p7NxdRC3t0wI4mO3+3oQN3U2BnFPc3nahdrRpVVBXKgPNYhOg/UVW1PZ0NkTxD3NzE1hffQKAAAAREFiCQAAgChILAEAABAFNZZ57LXXXkF89tln513/8MMPD+LsvcJRnxrWhXVJy5teCeKRTRsE8QgbH8TUMSFreU9xfQiQ1h+LelqKG1sYizAQ9BIAAABEQWIJAACAKEgsAQAAEAU1lnkceOCBQdzcHF4L7ve//30QP/DAA2VvE2pPV0tXEM9fHtbe7tRyVBD3jAj/3gvv5ot61KN1QTx/ZdiH3jHmmKFsDmpUoZrKxjWZGkzGIgwCM5YAAACIgsQSAAAAUZBYAgAAIApqLHO0tbUF8QEHHBDE69aFdU7nnntuEHd2dpanYagp2bvvLun+RxC/vvqlIO4ZFb6COiZkFepDGjN0bUHtyI5FhcaWnmbGIpSOGUsAAABEwYwlAABABKM2G+HdmbPrS7V2ceed7n5A4TWrA4klAABABD1rerTlkROibvOZqxfG3WCZkVjmOPPMM4N45513DuLf/va3QfyXv/yl7G1C7enOXHOwx7uDeM+Jnwhit2wlFOpd9rqVhfoQ0JdiayS9kbGoZCY1NNT3+0hiCQAAEEm9zxWQWAIAAERgkhrq/LRoEksAAIAYTDIOhdevgw46KIi/+tWvBvHy5cuD+Pzzzy97m1B7ehRevzRbY7lB0+ZB3KoNy94m1Bb6EDB8MGMJAACAKIzEEgAAAKUykxrq/OwdEksAAIBImLGsI+PHjw/i73//+0Hc2NgYxHfccUcQP/jgg+VpGIYVV1cQj1yzQRB3jxjK1qAWZfsQNZWIoTFzR5juEXWeAZUJNZYAAAAomRkzliSWAAAAkXDnHQAAAERR5+fuDO/EMlszmb3X95ZbbhnEzz//fBBnr2uJ+pQdI6wrvANv05pwjcaRY4PYm4q9Yy/qTYOag7hFY/tZE/Ws0FjUsCa8p3z3yPBXvDfVecYzBMyosRzWiSUAAMDQMe68U+kGAAAADAvMWJJYAgAAxMJZ4cPYVlttFcS77rpr3vVPP/30IM7WXKI+dWfu47zIHw/iCX99SxC3vmdyEPdQ11T3svcCX9QZ9qFJzbsNZXNQo7LV2tmayTV/XRjEjEVDz8Sdd4Z1YgkAADBkuI4liSUAAEAs1FgCAACgZCZxVnilGxDTFltsEcS/+93v8q5/5plnBvGvfvWr6G1C7bHusJJpbcPSIN64edsgbt2rLYi7G+t7UMH61nr+PgT0JTsWeYGxpW2vTYOYsagCOCt8eCWWAAAAlVTn5+6QWAIAAMRg4l7hJJYAAAAxcFb48EosTz755CDefPPN865/7733BrE793SGtLIhvBZcR8+SIL77+WuC+Li3X1r2NqG2rPIFQUwfwmAUqqnM6m6t84ymSjBjCQAAgJKZjAukV7oBAAAAw4IxY0liCQAAEAEn79R4Yrn33nsH8ac//ekKtQS1rHnpuiCe3/JEEG88cmoQv/jas0HcMKUniHtaqHOqd/NXF9eH9PZytwi1IDsWdW7YUtTrG9YxFlWDhjo/e6emE0sAAICqYcaMZaUbAAAAMBxwKJzEEgAAIBrOCq9h7373u4O4vb097/rPP/98EK9cuTJ6m1B7vDmsh2ltHBXEizteDOIT3/nNILau8rQLtavYPgRI649FxbKewuugvMykhjq/WXhNJ5YAAADVgxrL+k6rAQAAYrHkUHjMx4B2a3aAmT1rZnPM7Ow+lrea2c3p8ofMbEpm+eZmttLMzij1LWDGEgAAIIJKnLxjZo2SLpP0fknzJD1sZrPc/amc1U6UtNTdp5rZcZK+JenYnOXflfSbGO0Z1onlE0+E15Lbd999g3jJkvD+vahPb456LYgXLnkmiNuaxgTx1JH7B3H3sP4WYSCy9wZfuKq4PgRIUld7aYNJ9wgOQlaDCtRY7i5pjru/IElm9lNJh0nKTSwPk3Re+vPPJP3AzMzd3cwOl/SipFUxGkMvBAAAiMFMFvkhaYKZPZLzODmz100lzc2J56XP9bmOu3dJelPSeDNrl/RFSV+P9RYw1wIAABBBmQ6FL3b3abE3mjpP0vfcfaVFukwSiSUAAEAkFTgrfL6kzXLiyelzfa0zz8yaJG0g6Q1Je0g62swuljRWUo+ZrXH3Hwy2Mebu/S80638hqp67V8U1Dzo6OuhHNaytra3i/Yg+VNuqoQ9J9KNaVy39KJ+JU0f5sRdvH3Wb/++ovz6ab8YyTRT/IWlfJQnkw5I+4u6zc9Y5VdI73P2U9OSdI939Q5ntnCdppbt/p5T2MmMJAAAQxdBfx9Ldu8zsNEl3SmqUdK27zzaz8yU94u6zJP1I0kwzmyNpiaTjytUeEksAAIBIKnFLR3e/Q9Idmee+lvPzGknHFNjGeTHaQmIJAAAQQXJLx6o/Yl9WeRPLaqnRQ22rhboYVDf6EGKgH2EocK9wAAAAlMw08NswDlcklgAAADFwKJzEEgAAIBYSSwAAAJTMJDUYNZYAAAAokZmpqbG+U6v6/tcDAABEY5wVXukGAAAADAcmqdEaK92MihoWabWZTTEzT++XKTP7jZmdMIjtbG5mK83qvFcAAIDimamhoTHqo9YMWWJpZi+ZWUeauL1qZteZWXs59uXuH3T3Hw+wTfvlvO4Vd2939+5ytCtnvxub2U1mtsDM3jSz+81sj3LuEwAAlF+DNUZ91JqhnrE8xN3bJe0iaZqkc7IrWGJYzKTm0S7pYUm7Shon6ceSfl2uRBsAAJSfpTWWMR+1piItdvf5kn4jaXtJMrN7zOxCM7tf0mpJbzWzDczsR2a20Mzmm9mM3kPUZtZoZt8xs8Vm9oKkg3K3n25vek58kpk9bWYrzOwpM9vFzGZK2lzSL9NZ1LP6OKQ+ycxmmdkSM5tjZiflbPM8M7vFzK5PtzvbzKYN8N//grt/190Xunu3u18tqUXSv5XwtgIAgEoyU6M1Rn3UmoqcvGNmm0k6UNLPc57+mKQPSnpWSf3rLZJekzRV0ihJv5I0V9JVkk6SdLCknSWtknRbnn0dI+k8SYdLekTSVpI63f1jZvZuSdPd/e503SmZl/9U0pOSJknaWtJdZva8u/8hXX6opCMlfULSDEk/kPTOdFuXS5K7f2oA78dOShLLOYXWBQAA1cmkmqyLjGmoE8vbzaxL0puSfi3pGznLrnP32ZJkZhOVJJ5j3b1D0ioz+56kk5Uklh+SdKm7z03Xv0jSPv3sc7qki9394TQeUPKWJr97STrI3ddIetzMrpH0cUm9ieV97n5Huv5MSZ/rff1AEsr0dWMkzZT0dXd/cyCvAQAA1ci4QPoQ7+/w3tnBPszN+XkLSc2SFtq/bubekLPOpMz6L+fZ52aSni++qZokaYm7r8jsJ/dw96Kcn1dLGmFmTe7eNZAdmFmbpF9KetDdLxpEGwEAQJUwY8aymq5j6Tk/z5W0VtKEfpK0hUoSxl6b59nuXCWHvwvtM2uBpHFmNjonudxc0vw8rxkwM2uVdLukeZI+GWObAACgkqwm6yJjqsr5WndfKOl3ki4xszFm1mBmW5nZe9JVbpH0GTObbGYbSjo7z+aukXSGme2annE+1cy2SJe9Kumt/bRhrqS/SLrIzEaY2Q6STpR0Q6n/PjNrlvQzSR2STnD3nlK3CQAAKqv3XuExH7Wmmlv8cSUntDwlaamSRGyTdNkPJd0p6QlJjyk8CSjg7rdKulDSjZJWKJklHJcuvkjSOWa2zMzO6OPlH5Y0Rcns5S8knZvnUH7AzK40syv7WbynkpOP9pe0LD0rfWV6MhEAAKhJXCDd3PMdDQYAAMBATNluIz/3xqOibvM/d7rqUXcf0OUMq0E11VgCAADULJOpsQZnGWMisQQAAIigt8aynpFYAgAAxGBWk3WRMZFYAgAARGFqqPPLDeVNLM2MM3tqmLtb4bXKr6Ojg35Uw9ra2irej+hDta0a+pBEP6p11dKP8klu6cihcAAAAJSMC6STWAIAAERgJg6FV7oBAAAAwwMn75BYAgAARMDlhkgsAQAA4jAukE5iCQAAEEEyY0liCQAAgJIZh8Ir3QAAAIDhghlLAAAAlMy4847qe74WAAAgFjOZNUZ9DGy3doCZPWtmc8zs7D6Wt5rZzenyh8xsSvr8+83sUTP7e/r/95X6FtT0jOUZZ5wRxG1tbUG8ww47BPHRRx+dd3tXXHFFED/wwANBPHPmzGKbiBq0TsuCuNvXBfGfF1wTxHNenR3E03e5LIhbNCazB/6eG+7oQ0D9GuoZS0uyz8skvV/SPEkPm9ksd38qZ7UTJS1196lmdpykb0k6VtJiSYe4+wIz217SnZI2LaU9NZ1YAgAAVAuTqUFDfih8d0lz3P0FSTKzn0o6TFJuYnmYpPPSn38m6QdmZu7+fznrzJbUZmat7r52sI0hsQQAAIikDDOWE8zskZz4ane/OifeVNLcnHiepD0y2/jnOu7eZWZvShqvZMay11GSHi5TdAEAACAASURBVCslqZRILAEAAKIo08k7i919WuyN5jKz7ZQcHt+/1G3VVGJ58803B3Ghmsmsnp6evMs/+clPBvF+++0XxPfee28Qv/LKK0XtH9WhW2uC+PXOp4J4ZNO4IB5hYfzvm4b9ZK9J4R93Pd4VxOtsZRC3elgv51agwag6le5D69dcAsUzD2PGohhswCfcRDRf0mY58eT0ub7WmWdmTZI2kPSGJJnZZEm/kPRxd3++1MbUVGIJAABQzSpQY/mwpLeZ2ZZKEsjjJH0ks84sSSdIekDS0ZL+4O5uZmMl/VrS2e5+f4zGkFgCAABEYDb0d95JayZPU3JGd6Oka919tpmdL+kRd58l6UeSZprZHElLlCSfknSapKmSvmZmX0uf29/dXxtse0gsAQAAIqnEBdLd/Q5Jd2Se+1rOz2skHdPH62ZImhGzLVWdWJZaU/nMM88E8Z133hnEb33rW4P4kEMOCeKtttoqiI8//vggvuiii4pqDyqjS6uCeLW/HsQTmrcJ4kfeCK9Xev9zwXdVx+/6tSC+YNangvjgPQ4K4vdsOj2IszWWoq6p6lVbH2oxaiwRQabGkrEoBu68U9WJJQAAQK2wypy8U1VILAEAACKpwMk7VYXEEgAAIAoOhVdVYjltWnj9zyOOOCLv+rNnh/fXPfTQQ4N48eLFQbxyZeZacC0tQfzggw8G8Y477hjE48ePz9seVAdXdxCv8aVB3GKjw7hrRBDvNSqsZ9tz2klB3NMUFiJdfkTYbxpXh9cg7OnODjJhYZM3UNhUbYrtQ01qC+J3jj85b5x12VEP5l2O+tDQFY4Nlok9M/ZkxyLLXKp5vbFoRP6Eh7GodGaVOXmnmlRVYgkAAFC7mLEksQQAAIiCk3dILAEAACIwcfJOVSWWm2yySRCbhfUe2ZrKD3zgA0G8cOHCovb3hS98IYi33XbbvOv/+te/Lmr7qIxOrQji7/3pxCA+YpfwPs3vGHVkEPeMKK7OqKEzLGzK1jFl65Z8aG/KgEEotg9tPzrsQ8BgZMcKxqJaNPR33qk2VZVYAgAA1CpO3iGxBAAAiISTd0gsAQAAIjFqLKvHL3/5yyCeOnVqEK9YEdY9LVmypKT9HXfccUHc3Nxc0vZQHVo0NojPfddvgtgb49YZ2brMxeMy15brDi9xqAbP1D1x6biqk+1DX33PnRVqSSJ7Xc16/8VVqxoyY8V6Y1FjaYNBobGopy3sN5a5VzhjUemMGcvqSiwBAABqF4kliSUAAEAkJs4KBwAAQBT1XVNQ1Ynlyy+/HHV7Z555ZhC//e1vz7v+Qw89lDdGbehpifvXY7YuSZ55oiHcX5dWB3GztUdtD4a/LnUEcbPoQ7Wo0mNRFjWV5WDMWFa6AQAAAMOBKUkt6xmJJQAAQDTMWAIAACACZiyHsYMPPjiIzz///CBuaWkJ4tdeey2Iv/SlLwXx6tVhrRzqg2UuDde8eG0Qd7eHX6Psteioh0Op6EOQSh+LMBRMxr3CAQAAEEd9J/QklgAAAJFwVjgAAAAiMGosK92Acpo2bVoQZ2sqs26++eYgvvfee6O3CbXHesJrw3UtDa8p2DBiVBB3j6zv23kBKA/GolrBjCUAAABKxHUsSSwBAAAiMZnqe6aYxBIAACASTt4ZRm6//fYg3n///fOuf/311wfxOeecE71NqD3ZgxiNq7qCuGfy6CBePWpVELeqOYizt/MFstZpeRC3aEyFWoJqUuxY1DUqnCnLvp6xaGhwKBwAAAARmDh5BwAAACVLTt4hsQQAAEAEHAqvYZtsskkQ77nnnkHc2toaxIsXLw7iGTNmBPHKlSsjtg61IjsENKzqDuKu0WHNZPYFLZn7OFPHhGJl+xDqU6ljURZjUQWYSdwrHAAAADEwYwkAAIAIrO5rLOv7Xw8AABCVRX4MYI9mB5jZs2Y2x8zO7mN5q5ndnC5/yMym5Cz7Uvr8s2b2gcH9m/+lpmcsb7vttiAeP3583vVvuOGGIH7++eejtwm1J1vHtGzkvCBuUlsQj+reKIi9ib/PEFrhLwdxtg+12caZV9CHsP5Y1D0q/x1cGrrCKkpvqu9DsNViqGcszaxR0mWS3i9pnqSHzWyWuz+Vs9qJkpa6+1QzO07StyQda2bbSjpO0naSJkm628ze7u5hZywCoxkAAEAEvfcKj/nfAOwuaY67v+Du6yT9VNJhmXUOk/Tj9OefSdrXzCx9/qfuvtbdX5Q0J93eoJFYAgAARBH7MLhJ0gQzeyTncXJmp5tKmpsTz0uf63Mdd++S9Kak8QN8bVFq+lA4AABA1XCV4zpPi919WvStlklNJZaHHnpoEO+yyy5517/nnnuC+Nxzz43dJNQgy3zpszWVr615LointO0dxNQxIStbU1moDwHS+mNRoZrKLMaiauQyH/IriM6XtFlOPDl9rq915plZk6QNJL0xwNcWhUPhAAAAsXjkR2EPS3qbmW1pZi1KTsaZlVlnlqQT0p+PlvQHd/f0+ePSs8a3lPQ2SX8t+t+co6ZmLAEAAKraEE9YunuXmZ0m6U5JjZKudffZZna+pEfcfZakH0maaWZzJC1RknwqXe8WSU9J6pJ0ailnhEsklgAAAPEM/aFwufsdku7IPPe1nJ/XSDqmn9deKOnCWG2p6sQye13KL3/5y0Hc3Jy5b2rG448/HsTcC7w+ZeuYHlhydRCPbZ0UxK91hNc3fVvzvkHcU9XfGgyFB98org9t1bZf2duE6pcdi7zIEsnsdSt7qLGsPr7+51xv+BUJAAAQC4klAAAAoqjAofBqQmIJAAAQS33nldWdWH7hC18I4t122y3v+rfffnsQc91KSJJ1h9/y2QseCuLO7nVB/JEdLwhiH9gttVBHiu1DgLT+WFTsdSi9kbEI1a+qE0sAAICa4arEBdKrCoklAABALPWdV5JYAgAARENiWb1OP/30otY/7bTTgpjrVkKS5vaE9XBtzaOCePSIsUE8tmvzIO7Jf7lU1IH56x4M4kJ9aIxNKXOLUIuKve5kQ2dP+Ppm7sJcEzgUDgAAgBi4QDoAAABK5+JQeKUbAAAAMGyQWA4f48aNC+LOzs6Stvfmm2/m3V72XuUbbLBB3u2NHRvWYRVbQ9rd3R3EX/ziF4N49erVRW2vXkxseUcQH/pvZwbxoo6ng7hQTWXBKqkS7wecPYySvfZd1rqm8HPv9FVB3GxhPaCpMYibNCKI63xM7FOxfUjqycTVXRvXpbDPFNuHGjN9CHEUqqmstrGo1HuXZ19dm2ORU2NZ6QYAAAAMByZqLKv7z2gAAADUDGYsAQAAYuFQ+PDxt7/9Ler2br311iBeuHBhEE+cODGIjz322Kj7L2TRokVBfOGFFw7p/mtFth5slE0K4i1HbhTEzcvCWtbOsDRWKz1837sV3ic6q13h/hq7M0WcDWFl0bOrfxPEn/7B54L4iA/sHsQf3fEb4fbVGsQdvjiIl3dm+nGmfrBJYT0diu9D6x8MCmsuVxXbhzL7a1BL3vX/sYo+NBw0Lwvr+jvHhmNHwfSlQMnjejWTmbEoW5NZ7L3NizUs0jHOCh9eiSUAAEAl1XuNJYklAABALBwKBwAAQBT1nVdWd2J5xx13BPFhhx02pPs/5phjSnp9V1dXEPf0ZK9tF5o1a1YQP/LII3nX//Of/zy4htWZ1pVhN+9pDeuEGhrDa/B1j8pfD7eiK4zf0rxTEHtmVMnWpzU1tgXx8u5XgrixIayjOvHIg4P40Tlhv3jvzH2C+LKvfDmIdxn3kSBuaRkTxM09YT2cc62I9RS6TmNjpgYzK3ofsurqQ9RUDkzTyszvhNbwy+aNYdw9Kn+/KpU3lrdmsliW+RVZk2ORS+qp78yyqhNLAACA2uFyDoUDAAAgivwHJ4c9EksAAIAI3CWv80Phlm/K1qy6Tpo/66yzgjh7r+5CtttuuyAu9rqT1157bRC/9NJLede/7bbbgviZZ54pan+lci/2zrDl0dHRUVX9aJ2WBXGPwutWNmlkELeuylwzMFOX9HrLs0HcYqMzcViP5pn9tShTr5a55bu3hIVGpd6Pt1htbW0V70e13oeaFNZEZi3pCe81Xmofyl53s9KqoQ9J1dePitW4qjvzRPi2do+IW5TYuDrcH2NRYbtuv5M/dNvdUbfZvPVGj7r7tKgbLSNmLAEAACKp9xlLEksAAIAYOCucxBIAACAOzgqvqRpLFIcaS8RQDXVN9KHaVg19SKIf1bpq6Uf57Lrdjv7ATb+Lus3WHd9CjSUAAEA9qvcZSxJLAACAGKixJLEEAACIhbPCAQAAEAeHwgEAAFAqd6/7Gcu4l+kHAACoZz2RHyUws3FmdpeZPZf+f8N+1jshXec5MzshfW6kmf3azJ4xs9lm9s2B7JPEEgAAIBJ3j/oo0dmSfu/ub5P0+zQOmNk4SedK2kPS7pLOzUlAv+PuW0vaWdJeZvbBQjsksQQAAIih96zwmI/SHCbpx+nPP5Z0eB/rfEDSXe6+xN2XSrpL0gHuvtrd/yhJ7r5O0mOSJhfaITWWAAAAscSvsZxgZo/kxFe7+9UDfO1Ed1+Y/rxI0sQ+1tlU0tyceF763D+Z2VhJh0j670I7JLEEAACIpAwXSF+c7847Zna3pLf0segrmXb5YO6oaGZNkm6S9H13f6HQ+iSWAAAAMbhKPuGm6F2679ffMjN71cw2cfeFZraJpNf6WG2+pH1y4smS7smJr5b0nLtfOpD2UGMJAAAQRXK5oZiPEs2SdEL68wmS/rePde6UtL+ZbZietLN/+pzMbIakDSR9bqA7JLEEAACIxT3uozTflPR+M3tO0n5pLDObZmbXJM31JZIukPRw+jjf3ZeY2WQlh9O3lfSYmT1uZtML7ZBD4QAAADF4dd3S0d3fkLRvH88/Iml6TnytpGsz68yTZMXuk8QSAAAglipKLCuBxBIAACACV1nOCq8pJJYAAAAxuMs7uyvdiooisQQAAIihymosK4HEEgAAIAqXdw/xhSyrTN7E0t2LPhsIyGpra6MfoST0IcRAP0LZuaQeEksAAACUyCV5N4fCAQAAUCp3OTOWAAAAiIEaSwAAAJSOGksSSwAAgDicyw1VugEAAADDgnMonMQSAAAgApfq/uSdhko3IAYzm2JmbmZNafwbMzthENvZ3MxWmllj/FYCAIBhzV3q7on7qDFDllia2Utm1pEmbq+a2XVm1l6Ofbn7B939xwNs0345r3vF3dvdvew3+jSzPc3sr2a2wsz+ZmZ7l3ufAACgvLzHoz5qzVDPWB7i7u2SdpE0TdI52RUsMSxmUvtjZuMk/VLStyWNlXSxpF+a2YYVbRgAABi8tMYy5qPWVCSBc/f5kn4jaXtJMrN7zOxCM7tf0mpJbzWzDczsR2a20Mzmm9mM3kPUZtZoZt8xs8Vm9oKkg3K3n25vek58kpk9nc4OPmVmu5jZTEmbK0noVprZWX0cUp9kZrPMbImZzTGzk3K2eZ6Z3WJm16fbnW1m0wb4FuwpaZG73+ru3e5+g6TXJR05yLcUAABUXHKB9JiPWlORxNLMNpN0oKT/y3n6Y5JOljRa0suSrpPUJWmqpJ0l7S+pN1k8SdLB6fPTJB2dZ1/HSDpP0scljZF0qKQ33P1jkl5ROovq7hf38fKfSponaVK6j2+Y2ftylh+arjNW0ixJP8jZ7+Vmdnm+t6GPePs86wMAgGrmkro97qPGDPVZ4bebWZekNyX9WtI3cpZd5+6zJcnMJipJPMe6e4ekVWb2PSWJ51WSPiTpUnefm65/kaR9+tnndEkXu/vDaTxnIA1Nk9+9JB3k7mskPW5m1yhJUP+Qrnafu9+Rrj9T0ud6X+/un8qz+QckTTKzD0v6maSPSNpK0siBtA0AAFSnWpxljGmoE8vD3f3ufpbNzfl5C0nNkhaa/XNiryFnnUmZ9V/Os8/NJD1ffFM1SdISd1+R2U/u4e5FOT+vljTCzJrcvSvfht39DTM7TNJ3JF0m6U5JdyuZHQUAADXI3WuyLjKmarqOZe5871xJayVN6CdJW6gkYey1eZ7tzlUyG1hon1kLJI0zs9E5yeXmkubnec2Aufu9knaTpLSm8wVJl8TYNgAAqIx6n7GsyrOv3X2hpN9JusTMxphZg5ltZWbvSVe5RdJnzGxyeib12Xk2d42kM8xs1/SM86lmtkW67FVJb+2nDXMl/UXSRWY2wsx2kHSipBsi/BNlZjubWbOZjVEycznX3e+MsW0AAFAB1FhWZ2KZ+rikFklPSVqqpBZxk3TZD5UcPn5C0mOSft7fRtz9VkkXSrpR0gpJt0saly6+SNI5ZrbMzM7o4+UfljRFyezlLySdm+dQfsDMrjSzK/OscpakxUpmVDeRdMRAtgsAAKoVZ4Wbe+1lwwAAANVmh0lv81+e9L2o25xy/iGPuvtAL2dYcdVUYwkAAFDDvCZnGWMisQQAAIiht8ayjpFYAgAARODirHASSwAAgBjc5V0klv0ys/qez61x7p69bWRFdHR00I9qWFtbW8X7EX2otlVDH5LoR7WuWvpRXi4ukF7pBgAAAAwLLmYsK90AAACA4YFbOpJYAgAARODMWJJYAgAARMHJOySWAAAAsXidX8eymu8VDgAAUDvSQ+ExH6Uws3FmdpeZPZf+f8N+1jshXec5Mzuhj+WzzOzJgeyTxBIAACAGT07eifko0dmSfu/ub5P0+zQOmNk4SedK2kPS7pLOzU1AzexISSsHukMSSwAAgAi8ymYsJR0m6cfpzz+WdHgf63xA0l3uvsTdl0q6S9IBkmRm7ZJOlzRjoDukxhIAACCG8py8M8HMHsmJr3b3qwf42onuvjD9eZGkiX2ss6mkuTnxvPQ5SbpA0iWSVg+0sSSWAAAAkZThOpaL3X1afwvN7G5Jb+lj0VeCdrl7MXdUNLOdJG3l7p83sykDfR2JJQAAQAwVuI6lu+/X3zIze9XMNnH3hWa2iaTX+lhtvqR9cuLJku6R9C5J08zsJSX54sZmdo+776M8qLEEAACIIT0UXkU1lrMk9Z7lfYKk/+1jnTsl7W9mG6Yn7ewv6U53v8LdJ7n7FEl7S/pHoaRSYsYSAAAgCldZDoWX4puSbjGzEyW9LOlDkmRm0ySd4u7T3X2JmV0g6eH0Nee7+5LB7pDEEgAAIIYqu/OOu78had8+nn9E0vSc+FpJ1+bZzkuSth/IPkksAQAAYvCqm7EcciSWAAAAMVTg5J1qQ2IJAAAQRXUdCq8EEksAAIAI3CXvHvClIoclEksAAIAYOBROYgkAABCHc/JOpRsAAAAwLDBjSWIJAAAQRZVdx7ISSCwBAAAicK5jSWIJAAAQC4klAAAAInD1OIklAAAASuQu9TjXsQQAAEAE3cxY1o+pU6cG8YQJE4L4iCOOCOJ99tkniHt6ws5y5ZVXBvH9998fxHPmzBlMM1HlLHNXBesJ49VNS4L4wdevD+JpG30oiNss7IdN3a1B7I02qHaidq3xxUFcbB9q1IjyNAxVpdBY1NPcEHX7jEWFOYfC6yuxBAAAKCcOhQMAACAKZiwBAABQMncOhQ+rxHL77bcP4tNOOy2IjzzyyCDO1lgWa4899gjirq6uIH722WeD+L777gviz372s0G8bt26ktqDOCxzFKPHuoN4WcPzQTxv9ZNBvFX73kG858bTg7htTXsQL219KYhfXPnXIG5tDNffdsRBQexNYd1TfR+EqU6usA+96Zk+tLK4PtSiMUG83F8I4heX5+9DW7cfXKDFqAbZscgzJY7ZmsdiayAb14QJUPeIsCaz0PYaujI1mIxFkjgUPqwSSwAAgEpxcSicxBIAACAK53JDlW4AAADAcJDMWHIovGbssMMOQXzqqacG8bHHHhvEY8aEdUhZ8+fPD+I///nPQfziiy8G8VlnnRXEjz76aBDvvvvuQTxu3LggPvDAA4P4iSeeCOLsdTFRHuu0PIibMtf8yy7v8jWZ9UcG8RbtuwbxojWzg3j8iC2DeKEeD+Kn5t8bxAuWvhTEn3hH2C+WeHh91DHaIogb1CKUV8E+5Jk+pMh9aG2mDy0urg8t8+eCeIzRh6pRtqYytmxNZbF6mriu5XqcQ+E1lVgCAABUK5erq6e78IrDGIklAABAJMxYAgAAoGTOyTvVnVheddVVQZy9l3eh61D+/ve/D+K///3vQfzlL385iNesCeugsvbcc88g/q//+q8gvvbaa4N4p512CuJXX301iC+77LIgvu2224L49ddfz9seDExDZ/gln9sVXuNv4oitg3h518IgnmzTgtjWhdvrGdEYxKPbwnq1rI5Lwvq2Qz7zpSD2CWHd05u+IIhHWFi727IiXL97JPf3Lbe5HcX1oUnNuxW1/UJ9aMyIKUG8xeR/D1eYHIYrfV4QZ/sQNZVDIzsWFXsv7+x1IwuNRV5g869eeFcQb/qZ94avz7SvUE1m04rwWs7dIzPtqYexiBrL6k4sAQAAagVnhZNYAgAARMItHUksAQAAIuDOOxVOLEeMCK/9lr1O5PTp4f1xzcL6jGwN4hVXXBHE3/72t4N41apVg2pnr/HjxwdxY2NYP3LeeecF8W9/+9sg3mKL/HVTGJxs1U63OoO4ZXl4WOLJFX8M4nF3jg7iySeG1yPN3v+2pyn83Iv1lrP3C7fXnf+wySjbJIibVofLu+eG10z0bTccfOMgSerJ9KEGNQfxk6+GfWirKeFn2t6cKXKssGwfMpXWh9G37FiU/WY3LQ9rENeND2tbl171UBCPr7KxKKtxdXhZne65K4K4XsciDoUDAACgZO6cFU5iCQAAEAmHwgEAABAFh8IraJ999gniM888M4izNZXZe3sfddRRQfzXv4bXlitWtmZys802C+Lrr78+iO+4444g3nDD/PUk2X/PzJkzg3jZsmUDaicyMnVBM5/5dBB/4O0nB/E1M8PrhR41/etB3FXi/W+zr7bMtee6msL6vdUN4fVNR1vY71oXdATxmknh17Zx6/qsYyqnmbML9KHbwz502OcuLHubcvVoXRCv9rAPtWf6ULamslANKQYpW6OYuW5jtqYya6Pjw3vGl3ssyt7rO3udyezrWzJj0bpJbeH6jEVyzgpnxhIAACCGeVp25+n+8/x3byne4sjbKysSSwAAgAjc/YBKt6HSirufFAAAANCPis5YZmsau7u7+1kz0dUVXgNsjz32COKjjz46iLfeOrx/b1ZHR1gvss022+SNFy8OZ6MnTpyYd/tZ2XuFz5gxI4g7O8O6JwxM9v60B219WhC/vOKxIJ6wUVgXtK49rId5YfXvgvjXs38YxHu/7ZAg3q39Y0F8y8tfDOIxI8YG8X6bhrXEIyw8atKwJmzPusWZC1fOC/+9vntx/RCFFduHsjWPL6y+J4gL9aFdx348iH/2fHhN32L7UCHUVJZHdizqHFvc+9zVXtqv5Oy9xNeroSyyZrPQWNQzL7yGbgNjEcSMJQAAACIhsQQAAEAUJJYAAACIwjzPhTzNrKxX+WxrC+uUbrzxxiDeb7/wPqYjR44M4oaGMC/O92+R1q/hzNZ4lqqnJ6xH+cUvfhHEn/nMZ4J44cKFUfef5e6lXQQtko6OjjJfLTZ839doaRD/cd7lQdzWMiqId5jwwXBr3p03bm1sD+K13SvD7TeG9XDZerbR3eF9m7N1T2/6S0E8wsLttXZvEL6+sbwfc1tbW8X7EX0o7EOjbJLyWV6gD7UojMutGvqQNBT9qLZkazKzY1GhN8sy1+2sh7EIhTFjCQAAgChILAEAABAFiSUAAACiqGiNZSFjx4Z1QGeffXYQ77XXXkH8xhtvBPErr7wSxK2trUG84447BvHuu+8+qHb2uvLKK4P4y1/+chAP9b3A66fGspCwfq4x05yP/3xaED/+UHhttlu/eUUQnz3zc0F8yycfD+JOrQrikWvDmsisNQ8sCOKmDcJ+6jvnr7crt2qoa6q2PpT9m/wTP9k5iGP3oVaVdg/mSt8bvBr6kFQN/SjU2BHW3na3xa37X29/a/Pfw7rQWGQ7bxS9TcWoln6E/JixBAAAQBQklgAAAIiCxBIAAABRVHWNZbldf/31QfzRj3407/orVqwI4tNPPz2Ir7vuuiAudO/zcqPGcmhk7w+8anTYT7L1cZ0Kr1n4t2U/D+LsfaMrrRrqmoZ7H8pam7mOJn0ojuHej7JjUdfo0u49Xm2qpR8hP2YsAQAAEAWJJQAAAKIgsQQAAEAUdVVjedZZZwXxjBkzgripKX89yvHHHx/EN910U5yGlQk1luWS/1pw3VoXxOs8vKZhq4X1ckN9TcFiVUNdE32IPhTD8OtH9aVa+hHyY8YSAAAAUZBYAgAAIAoSSwAAAEQxvC5ylTF9+vQgPuecc4K4UE3l7Nmzg/jnP/95P2tiOGtaHt5nuWdk2G/WNa0J11d4f92R2jjv9in6qkfh3/Rd6gjibB9qs/x9CPWh0FjU05S/BHG9syYyqzMWIQZmLAEAABAFiSUAAACiILEEAABAFMOqxnL33XcP4ksuuSSI29vb875+5crw/runnHJKEK9du7aE1qFWZKuUsnVMnqljytbDrfQFQTzaJoevL615GIYK9aH2TB9CfSh2LCp2g4xFKAdmLAEAABAFiSUAAACiILEEAABAFMOqxvKQQw4J4tGjR+ddf9WqVUF86KGHBvH9998fp2GoKZ2Zawq2eGtmjWxdU/j32WjP1FRyd9u6s/51KdsKvCLsQ9RUQlq/BtK8wIUos0szt6R3ppIwBOhmAAAAiILEEgAAAFGQWAIAACCKmq6xzNZQnnXWWUW9/ic/+UkQ33PPPaU2CTUoW6XU3D0iiNc0rwji5d2vBPHYxi2DuKlhVLS2oTZlayrXaXkQF+xDog/Vo/UqJrvDmsqe5uLmgqipRCXQ7QAAABAFiSUAAACiILEEAABAFObrXRcrZ6FZVd1KNHuv76effjqIN91007yv/9vf/hbEvSuFQwAAAiRJREFU73znO4N4zZo1JbSu+rhXxxUUOzo6qqofZd+ULoWfe4e/HsTttlkQN3SGF4fzTN1TVf1jI2hra6t4P6q2PpTVXWQfqjfV0Iek6utH2Tel2MYxFqEaMWMJAACAKEgsAQAAEAWJJQAAAKKoqetYvu997wviyZMz92TOUy8qSZ///OeDeLjVVGJwGhVet3JlV1gf19Q8MojbGsYF8XCrY0Lxiu1DIzS+7G3C8OcNYckhYxGqATOWAAAAiILEEgAAAFGQWAIAACCKmqqxvOCCC4K4UE3lt7/97SD+4x//GL1NqD2u8Npvb/qLQfzk4ruC+L2bfDZ8fSOXUkNpfQiQSq+JZCxCNWLGEgAAAFGQWAIAACAKEksAAABEUVM1luPGhdcPNAvrS1577bUgvvTSS8veJtSeNb4kiFd2htccXNGxLIibMtco5FpxKLYPZa9zCQxGqfcWB4YCM5YAAACIgsQSAAAAUZBYAgAAIIqaqrH87ne/mzfOXudy4cKFZW8Tak+XwnvE3//yrUF89Nu+GcTUMSGr2D4ExMBYhFrAjCUAAACiILEEAABAFCSWAAAAiMLy3W/bzCjpqGHuXhU3ku3o6KAf1bC2traK9yP6UG2rhj4k0Y9qXbX0I+THjCUAAACiILEEAABAFCSWAAAAiCJvjSUAAAAwUMxYAgAAIAoSSwAAAERBYgkAAIAoSCwBAAAQBYklAAAAoiCxBAAAQBT/H62M5/KPXXuxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x504 with 13 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=3, ncols=4, figsize=(10, 7))\n",
    "image_ids = [0, 1, 9]\n",
    "cmap_bound = np.abs(attrs[[0, 1, 9]]).max()\n",
    "\n",
    "for row, image_id in enumerate(image_ids):\n",
    "    # original images\n",
    "    ax[row, 0].imshow(X_test[image_id].squeeze(), cmap='gray')\n",
    "    ax[row, 0].set_title(f'Prediction: {predictions[image_id]}')\n",
    "    \n",
    "    # attributions\n",
    "    attr = attrs[image_id]\n",
    "    im = ax[row, 1].imshow(attr.squeeze(), vmin=-cmap_bound, vmax=cmap_bound, cmap='PiYG')\n",
    "    \n",
    "    # positive attributions\n",
    "    attr_pos = attr.clip(0, 1)\n",
    "    im_pos = ax[row, 2].imshow(attr_pos.squeeze(), vmin=-cmap_bound, vmax=cmap_bound, cmap='PiYG')\n",
    "    \n",
    "    # negative attributions\n",
    "    attr_neg = attr.clip(-1, 0)\n",
    "    im_neg = ax[row, 3].imshow(attr_neg.squeeze(), vmin=-cmap_bound, vmax=cmap_bound, cmap='PiYG')\n",
    "    \n",
    "ax[0, 1].set_title('Attributions');\n",
    "ax[0, 2].set_title('Positive attributions');\n",
    "ax[0, 3].set_title('Negative attributions');\n",
    "\n",
    "for ax in fig.axes:\n",
    "    ax.axis('off')\n",
    "\n",
    "fig.colorbar(im, cax=fig.add_axes([0.95, 0.25, 0.03, 0.5]));"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}