examples/linearity_measure_fashion_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linearity measure applied to fashion MNIST\n",
    " \n",
    "## General definition\n",
    "\n",
    "The model linearity module in alibi provides metric to measure how linear an ML model is. Linearity is defined based on how much the linear superposition of the model's outputs differs from the output of the same linear superposition of the inputs. \n",
    "\n",
    "Given $N$ input vectors $v_i$, $N$ real coefficients $\\alpha_i$ and a predict function $\\text{M}(v_i)$, the linearity of the predict function is defined as\n",
    "\n",
    "$$L = \\Big|\\Big|\\sum_i \\alpha_i M(v_i) - M\\Big(\\sum_i \\alpha_i v_i\\Big) \\Big|\\Big| \\quad \\quad \\text{If M is a regressor}$$\n",
    "\n",
    "$$L = \\Big|\\Big|\\sum_i \\alpha_i \\log \\circ M(v_i) - \\log \\circ M\\Big(\\sum_i \\alpha_i v_i\\Big)\\Big|\\Big| \\quad \\quad \\text{If M is a classifier}$$\n",
    "\n",
    "Note that a lower value of $L$ means that the model $M$ is more linear.\n",
    "\n",
    "\n",
    "## Alibi implementation\n",
    "* Based on the general definition above, alibi calculates the linearity of a model in the neighboorhood of a given instance $v_0$. \n",
    "\n",
    "## Fashion MNIST data set\n",
    "\n",
    "* We train a convolutional neural network to classify the images in the fashion MNIST dataset. \n",
    "\n",
    "* We investigate the correlation between the model's linearity associated to a certain instance and the class the instance belong to. \n",
    "\n",
    "* We also calculate the linearity measure for each internal layer of the CNN and show how linearity propagates through the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from time import time\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from alibi.confidence.model_linearity import linearity_measure, LinearityMeasure\n",
    "from alibi.confidence.model_linearity import _infer_feature_range\n",
    "\n",
    "from tensorflow.keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPooling2D, Input, Activation\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras import backend as K"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data fashion mnist \n",
    "The fashion MNIST data set consists of 60000 images of shape $28 \\times 28$ divided in 10 categories. Each category corresponds to a different type of clothing piece, such as \"boots\", \"t-shirts\", etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28) y_train shape: (60000,)\n"
     ]
    }
   ],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()\n",
    "print('x_train shape:', x_train.shape, 'y_train shape:', y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample instance from the MNIST data set.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAUFElEQVR4nO3da2yc1ZkH8P8z4/ElzjiJk+CE4BIuoZDCEqhJuIlSKDREVQOli4gQCxLaoF3otl0+gGhXZb+sEFpAaNntroEsYVWoWhUERREFzCULlDQmpOS2ITeHxDi2ExPbcTz2XJ794Bdqgs/zmnnnRs7/J1kezzNn5njGf78zc+acI6oKIjr+xcrdASIqDYadyBMMO5EnGHYiTzDsRJ6oKuWNVUuN1qK+lDdJ5JUUhjCqIzJRLVLYRWQpgEcAxAE8rqr3W5evRT2WyJVRbpKIDOu0zVnL+2m8iMQB/DuAawAsBLBCRBbme31EVFxRXrMvBrBTVXer6iiAXwNYXphuEVGhRQn7PAD7xv28Pzjvc0RkpYi0i0h7GiMRbo6Ioij6u/Gq2qqqLarakkBNsW+OiByihL0TQPO4n08KziOiChQl7OsBLBCRU0SkGsCNAF4oTLeIqNDyHnpT1YyI3AngDxgbelulqlsK1jMiKqhI4+yqugbAmgL1hYiKiB+XJfIEw07kCYadyBMMO5EnGHYiTzDsRJ5g2Ik8wbATeYJhJ/IEw07kCYadyBMMO5EnGHYiT5R0KWkqA5lwVeG/iLixZ3xmo1n/5LtnOGsNT78b6bbDfjepSjhrmh6NdttRhT0uljwfMx7ZiTzBsBN5gmEn8gTDTuQJhp3IEww7kScYdiJPcJz9OCfxuFnXTMasxxbZe3Vuu32q3X7YXUsMLTbbVg3nzHri5XazHmksPWwMP+R+hdjH0Sh9kyojtsbDySM7kScYdiJPMOxEnmDYiTzBsBN5gmEn8gTDTuQJjrMf58wxWYSPs+/77nSzftNF/2vW3+491VnbWzPHbKt1ZhlV37nIrJ/xH53OWqbjI/vKQ+aMh91vYeIzZriL2azZNjsw4C4a3Y4UdhHpADAIIAsgo6otUa6PiIqnEEf2b6vqwQJcDxEVEV+zE3kiatgVwMsi8p6IrJzoAiKyUkTaRaQ9jZGIN0dE+Yr6NP5SVe0UkRMAvCIi/6eqa8dfQFVbAbQCQIM0RlvdkIjyFunIrqqdwfceAM8BsKcxEVHZ5B12EakXkeSnpwFcDWBzoTpGRIUV5Wl8E4DnZGzebxWAp1X1pYL0igoml0pFaj963hGz/sNp9pzy2ljaWXszZs9X73yt2axn/8ru296Hks5a7v2LzbYzN9tj3Q3vd5n1g5fNM+u933S/om0KWU5/xqu7nDXpc0c677Cr6m4A5+bbnohKi0NvRJ5g2Ik8wbATeYJhJ/IEw07kCdGIW/Z+GQ3SqEvkypLdnjesZY9DHt8jN1xo1q/5+Rtm/azaj836YK7WWRvVaB/gfHT7t8z60O5pzlpsNGTL5JBytsleClrT9nF0xgb37163vNtsK4/NdtY+aHsER/r2Tdh7HtmJPMGwE3mCYSfyBMNO5AmGncgTDDuRJxh2Ik9wnL0ShGwPHEnI43v2e/b/+x/MsKewhokbaxsPabXZ9nC2PtJt92bcU1zTIWP8j++wp8AeMcbwASCWsR/Tq779vrN2feN6s+0Dp53jrK3TNgxoH8fZiXzGsBN5gmEn8gTDTuQJhp3IEww7kScYdiJPcMvmSlDCzzoca8eRE8z6oYapZv1Axt7SeWbcvdxzMjZstp2fsPcL7c26x9EBIJ5wL1U9qnGz7T9/4/dmPXVWwqwnxF6K+mJjHYC/3vo3Ztt67DbrLjyyE3mCYSfyBMNO5AmGncgTDDuRJxh2Ik8w7ESe4Di752bX2Nse14p7y2UAqJaMWf84PcNZ2zH8dbPthwP2ZwCWNm0x62ljLN2aZw+Ej5OfmPjErKfUHoe37tVLmuxx9I1m1S30yC4iq0SkR0Q2jzuvUUReEZEdwXf3I0pEFWEyT+OfBLD0mPPuAdCmqgsAtAU/E1EFCw27qq4F0HfM2csBrA5OrwZwbYH7RUQFlu9r9iZV7QpOHwDQ5LqgiKwEsBIAajElz5sjoqgivxuvYytWOt/tUNVWVW1R1ZYEaqLeHBHlKd+wd4vIXAAIvvcUrktEVAz5hv0FALcEp28B8HxhukNExRL6ml1EngFwOYBZIrIfwC8A3A/gNyJyG4C9AG4oZiePeyHrxkvcnnutGfdYd3yGPSr6rembzHpvtsGsH87a78NMjx911gYz7r3bAaBv2L7uM2u6zPqGo/OdtdnV9ji51W8A6BidZdYX1Bww6w90u/dPaK499v3wz8tceZmzpuv+6KyFhl1VVzhK3O2B6CuEH5cl8gTDTuQJhp3IEww7kScYdiJPcIprJQhZSlqq7IfJGnrbd9tZZtsrpthLJr+TmmfWZ1cNmnVrmuncmn6zbbIpZdbDhv0aq9zTdwezdWbbKbERsx72e59fbS+D/dNXz3fWkmcfMts2JIxjtDGKyyM7kScYdiJPMOxEnmDYiTzBsBN5gmEn8gTDTuQJjrNXAElUm/Vcyh5vtszaNGrWD2btJY+nx+ypntUhSy5bWyNf3LjHbNsbMha+YfgUs56Mu7eEnh2zx8mbE/ZY96ZUs1lfM3S6Wb/te686a8+0XmW2rX7pHWdN1P148chO5AmGncgTDDuRJxh2Ik8w7ESeYNiJPMGwE3niqzXObiy5LFX2eLHEQ/6vxex6LmXMb87ZY81hNG2PhUfxyH89atb3Zaab9QNpux625HLWmGD97vA0s21tzN4uenbVgFkfyNnj9JbBnL3MtTVPHwjv+90zdzhrz/Z/x2ybLx7ZiTzBsBN5gmEn8gTDTuQJhp3IEww7kScYdiJPVNQ4e5T10cPGqtUe9iyr4eWLzfq+a+1x/JvO+5OzdiCTNNu+b2xrDADTjDnhAFAfsr56St2ff/h41N5OOmys2loXHgBOMMbhs2of5zrTdt/ChH3+YH/GWNP++/Zc++lP5dWl8CO7iKwSkR4R2TzuvPtEpFNENgZfy/K7eSIqlck8jX8SwNIJzn9YVRcFX2sK2y0iKrTQsKvqWgB9JegLERVRlDfo7hSRD4Kn+c4XOCKyUkTaRaQ9Dfv1HREVT75h/yWA0wAsAtAF4EHXBVW1VVVbVLUlgZo8b46Iosor7KrarapZVc0BeAyA/XYyEZVdXmEXkbnjfrwOwGbXZYmoMoSOs4vIMwAuBzBLRPYD+AWAy0VkEQAF0AHg9kJ0xhpHj6pq7hyznj6lyaz3neXeC/zoHGNTbACLlm0z67c2/bdZ7802mPWEGPuzp2eabc+b0mHWX+tfaNYPVk0169Y4/cX17jndAHA4Z++/fmLVJ2b97p0/dNaapthj2Y+fbA8wpTVn1ren7Zes/Tn3fPh/WPi62fY5zDbrLqFhV9UVE5z9RF63RkRlw4/LEnmCYSfyBMNO5AmGncgTDDuRJypqiuvINReY9RN+tttZW9Sw32y7sO4ts57K2UtRW9Mttw7PM9sezdlbMu8YtYcF+zP2EFRc3MNAPaP2FNcH99jLFrct/k+z/vOPJ5oj9RexOnXWDmXtYbvrp9pLRQP2Y3b719Y6a6dW95htXxyaa9Y/DpkC25ToN+vzE73O2g+SH5pt8x1645GdyBMMO5EnGHYiTzDsRJ5g2Ik8wbATeYJhJ/JEacfZxV4uesm/rDebX5nc4qwdVXtKYdg4eti4qWValb1s8Ejavpt70vYU1jBn1Bxw1q5r2Gi2XfvoErN+aepHZn3XFfb03LZh91TO3oz9e9+45wqzvuGjZrN+4fw9zto5yU6zbdhnG5LxlFm3ph0DwFDO/ff6bsr+/EG+eGQn8gTDTuQJhp3IEww7kScYdiJPMOxEnmDYiTwhqu75xoVWN6dZT7v5H5311jv+zWz/dN+Fzlpzrb0d3cnVB836zLi9/a8lGbPHXL+esMdcXxw6yay/cfhMs/7NZIezlhB7u+fLp+w067f+9C6znqm1l9EemO8+nmTq7b+9hnMPmfUfnf6aWa82fvfDWXscPex+C9uSOYy1BkEyZm+T/eCy65y1P3Y8if7hrgkfFB7ZiTzBsBN5gmEn8gTDTuQJhp3IEww7kScYdiJPlHQ+eywNTOl2jy++OLDIbH9qnXut7YNpe330Pxw5x6yfVGdv/2ttPXy6MZ8cADamppv1l3q/YdZPrLPXT+9OT3PWDqXrzbZHjXnVAPDEww+Z9Qe77XXnr2vc4KydW22Pox/O2ceirSHr7Q/map21lNrrG/SHjMMnjb8HAEirHa24seXz9Jg9hj9wjnsb7my3+3ZDj+wi0iwir4vIVhHZIiI/Ds5vFJFXRGRH8D3/1R+IqOgm8zQ+A+AuVV0I4EIAd4jIQgD3AGhT1QUA2oKfiahChYZdVbtUdUNwehDANgDzACwHsDq42GoA1xark0QU3Zd6g05E5gM4D8A6AE2q2hWUDgBocrRZKSLtItKeGRmK0FUiimLSYReRqQB+B+Anqvq5d4x0bDbNhLMaVLVVVVtUtaWqxn6ziIiKZ1JhF5EExoL+K1V9Nji7W0TmBvW5AOxtMYmorEKH3kREADwBYJuqjh+HeQHALQDuD74/H3Zd8dEckvtGnPWc2tMlXzvonurZVDtotl2U3GfWtx+1h3E2DZ/orG2o+prZti7u3u4ZAKZV21Nk66vc9xkAzEq4f/dTauz/wdY0UABYn7J/t7+b/YZZ/yjjHqT5/dAZZtutR933OQDMCFnCe9OAu/3RjL2N9kjWjkYqYw/lTquxH9MLGvc6a9thbxfde64xbfhtd7vJjLNfAuBmAJtE5NNFyO/FWMh/IyK3AdgL4IZJXBcRlUlo2FX1LQCuQ+6Vhe0OERULPy5L5AmGncgTDDuRJxh2Ik8w7ESeKO2WzUeGEXvzfWf5ty9fYjb/p+W/ddbeDFlu+cUD9rjowKg91XP2FPdHfRuMcW4AaEzYHxMO2/K5NmT7308y7k8mjsTsqZxZ50DLmAMj7umzAPB2boFZT+fcWzaPGDUg/PMJfaOzzPqJdf3O2mDGPf0VADoGG836wX57W+XUFDtab2VPc9aWznFvTQ4AdT3uxyxm/KnwyE7kCYadyBMMO5EnGHYiTzDsRJ5g2Ik8wbATeaKkWzY3SKMukfwnyvXf5N6y+dS/3262XTx9j1nfMGDP2/7IGHdNhyx5nIi5lw0GgCmJUbNeGzLeXB13z0mPTbyA0GdyIePs9XG7b2Fz7Ruq3PO6k3F7znfM2NZ4MuLG7/6n/vmRrjsZ8ntn1P6buGjaLmdt1Z6LzbbTlrm32V6nbRjQPm7ZTOQzhp3IEww7kScYdiJPMOxEnmDYiTzBsBN5ovTj7PGr3RfI2WuYRzF0/RKzvuTe9XY96R4XPbO622ybgD1eXBsynlwfs8fCU8ZjGPbf/K3hZrOeDbmG1z45y6ynjfHm7qMNZtuE8fmBybD2IRjOhGzZPGzPd4/H7Nyk3rDn2s/c6v7sRM0a+2/RwnF2ImLYiXzBsBN5gmEn8gTDTuQJhp3IEww7kSdCx9lFpBnAUwCaACiAVlV9RETuA/C3AHqDi96rqmus64o6n71SyQX2mvTDc+rMes0he2704Ml2+4Zd7nXpYyP2mvO5P28z6/TVYo2zT2aTiAyAu1R1g4gkAbwnIq8EtYdV9V8L1VEiKp7J7M/eBaArOD0oItsAzCt2x4iosL7Ua3YRmQ/gPADrgrPuFJEPRGSViMxwtFkpIu0i0p6G/XSViIpn0mEXkakAfgfgJ6o6AOCXAE4DsAhjR/4HJ2qnqq2q2qKqLQnY+6kRUfFMKuwiksBY0H+lqs8CgKp2q2pWVXMAHgOwuHjdJKKoQsMuIgLgCQDbVPWhcefPHXex6wBsLnz3iKhQJvNu/CUAbgawSUQ2BufdC2CFiCzC2HBcB4Dbi9LDrwBdv8ms25MlwzW8k3/baIsx0/FkMu/GvwVMuLi4OaZORJWFn6Aj8gTDTuQJhp3IEww7kScYdiJPMOxEnmDYiTzBsBN5gmEn8gTDTuQJhp3IEww7kScYdiJPMOxEnijpls0i0gtg77izZgE4WLIOfDmV2rdK7RfAvuWrkH07WVVnT1Qoadi/cOMi7araUrYOGCq1b5XaL4B9y1ep+san8USeYNiJPFHusLeW+fYtldq3Su0XwL7lqyR9K+trdiIqnXIf2YmoRBh2Ik+UJewislREtovIThG5pxx9cBGRDhHZJCIbRaS9zH1ZJSI9IrJ53HmNIvKKiOwIvk+4x16Z+nafiHQG991GEVlWpr41i8jrIrJVRLaIyI+D88t63xn9Ksn9VvLX7CISB/AhgKsA7AewHsAKVd1a0o44iEgHgBZVLfsHMETkMgBHADylqmcH5z0AoE9V7w/+Uc5Q1bsrpG/3AThS7m28g92K5o7fZhzAtQBuRRnvO6NfN6AE91s5juyLAexU1d2qOgrg1wCWl6EfFU9V1wLoO+bs5QBWB6dXY+yPpeQcfasIqtqlqhuC04MAPt1mvKz3ndGvkihH2OcB2Dfu5/2orP3eFcDLIvKeiKwsd2cm0KSqXcHpAwCaytmZCYRu411Kx2wzXjH3XT7bn0fFN+i+6FJVPR/ANQDuCJ6uViQdew1WSWOnk9rGu1Qm2Gb8M+W87/Ld/jyqcoS9E0DzuJ9PCs6rCKraGXzvAfAcKm8r6u5Pd9ANvveUuT+fqaRtvCfaZhwVcN+Vc/vzcoR9PYAFInKKiFQDuBHAC2XoxxeISH3wxglEpB7A1ai8rahfAHBLcPoWAM+XsS+fUynbeLu2GUeZ77uyb3+uqiX/ArAMY+/I7wLws3L0wdGvUwH8OfjaUu6+AXgGY0/r0hh7b+M2ADMBtAHYAeBVAI0V1Lf/AbAJwAcYC9bcMvXtUow9Rf8AwMbga1m57zujXyW53/hxWSJP8A06Ik8w7ESeYNiJPMGwE3mCYSfyBMNO5AmGncgT/w8K8iUImXY9pQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "idx = 0\n",
    "plt.imshow(x_train[idx])\n",
    "print('Sample instance from the MNIST data set.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1) x_test shape: (10000, 28, 28, 1)\n",
      "y_train shape: (60000, 10) y_test shape: (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "x_train = x_train.astype('float32') / 255\n",
    "x_test = x_test.astype('float32') / 255\n",
    "x_train = np.reshape(x_train, x_train.shape + (1,))\n",
    "x_test = np.reshape(x_test, x_test.shape + (1,))\n",
    "print('x_train shape:', x_train.shape, 'x_test shape:', x_test.shape)\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)\n",
    "print('y_train shape:', y_train.shape, 'y_test shape:', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional neural network\n",
    "Here we define and train a 2 layer convolutional neural network on the fashion MNIST data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model():\n",
    "    x_in = Input(shape=(28, 28, 1), name='input')\n",
    "    x = Conv2D(filters=64, kernel_size=2, padding='same', name='conv_1')(x_in)\n",
    "    x = Activation('relu', name='relu_1')(x)\n",
    "    x = MaxPooling2D(pool_size=2, name='maxp_1')(x)\n",
    "    x = Dropout(0.3, name='drop_1')(x)\n",
    "    \n",
    "    x = Conv2D(filters=64, kernel_size=2, padding='same', name='conv_2')(x)\n",
    "    x = Activation('relu', name='relu_2')(x)\n",
    "    x = MaxPooling2D(pool_size=2, name='maxp_2')(x)\n",
    "    x = Dropout(0.3, name='drop_2')(x)\n",
    "    \n",
    "    x = Flatten(name='flat')(x)\n",
    "    x = Dense(256, name='dense_1')(x)\n",
    "    x = Activation('relu', name='relu_3')(x)\n",
    "    x = Dropout(0.5, name='drop_3')(x)\n",
    "    x_out = Dense(10, name='dense_2')(x)\n",
    "    x_out = Activation('softmax', name='softmax')(x_out)\n",
    "    \n",
    "    cnn = Model(inputs=x_in, outputs=x_out)\n",
    "    cnn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    \n",
    "    return cnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input (InputLayer)           [(None, 28, 28, 1)]       0         \n",
      "_________________________________________________________________\n",
      "conv_1 (Conv2D)              (None, 28, 28, 64)        320       \n",
      "_________________________________________________________________\n",
      "relu_1 (Activation)          (None, 28, 28, 64)        0         \n",
      "_________________________________________________________________\n",
      "maxp_1 (MaxPooling2D)        (None, 14, 14, 64)        0         \n",
      "_________________________________________________________________\n",
      "drop_1 (Dropout)             (None, 14, 14, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv_2 (Conv2D)              (None, 14, 14, 64)        16448     \n",
      "_________________________________________________________________\n",
      "relu_2 (Activation)          (None, 14, 14, 64)        0         \n",
      "_________________________________________________________________\n",
      "maxp_2 (MaxPooling2D)        (None, 7, 7, 64)          0         \n",
      "_________________________________________________________________\n",
      "drop_2 (Dropout)             (None, 7, 7, 64)          0         \n",
      "_________________________________________________________________\n",
      "flat (Flatten)               (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 256)               803072    \n",
      "_________________________________________________________________\n",
      "relu_3 (Activation)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "drop_3 (Dropout)             (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                2570      \n",
      "_________________________________________________________________\n",
      "softmax (Activation)         (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 822,410\n",
      "Trainable params: 822,410\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "cnn = model()\n",
    "cnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 40s 674us/sample - loss: 0.5552 - acc: 0.7955\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 43s 717us/sample - loss: 0.3865 - acc: 0.8596\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 51s 852us/sample - loss: 0.3421 - acc: 0.8765\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 47s 782us/sample - loss: 0.3123 - acc: 0.8851\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 48s 802us/sample - loss: 0.2938 - acc: 0.8936\n"
     ]
    }
   ],
   "source": [
    "cnn.fit(x_train, y_train, batch_size=64, epochs=5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linearity of each Layer \n",
    "Here we calculate the linearity of the model considering each layer as the output in turn. The values are averaged over 100 random instances sampled from the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "inp = cnn.input\n",
    "outs = {l.name: l.output for l in cnn.layers}\n",
    "predict_fns = {name: K.function([inp], [out]) for name, out in outs.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate linearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Infering feature ranges.\n",
    "features_range = _infer_feature_range(x_test)\n",
    "\n",
    "# Selecting random instances from training set.\n",
    "rnd = np.random.randint(len(x_test) - 101, size=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "lins_layers = {}\n",
    "for name, l in predict_fns.items():\n",
    "    if name != 'input':\n",
    "        def predict_fn(x):\n",
    "            layer = l([x])\n",
    "            return layer[0]\n",
    "        if name == 'softmax':\n",
    "            lins_layers[name] = linearity_measure(predict_fn, x_test[rnd], feature_range=features_range,\n",
    "                                                  agg='global', model_type='classifier', nb_samples=20)\n",
    "        else:\n",
    "            lins_layers[name] = linearity_measure(predict_fn, x_test[rnd], feature_range=features_range, \n",
    "                                                  agg='global', model_type='regressor', nb_samples=20)\n",
    "lins_layers_mean = {k: v.mean() for k, v in lins_layers.items()}\n",
    "S = pd.Series(data=lins_layers_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linearity measure calculated taking as output each layer of a convolutional neural network.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA44AAAKXCAYAAADNWRRzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdeZhtV1kn/u8bkhACzZjbEIKBCEJwAuQySWMSJrtBaRtkaGQIEvPYCgI/HrWVIQSUqAyiIEMQmUEIogLN0BAI0AySgICMAiEEAobEQCCEIMP7++Ocq5Wi7rp17j11z6mqz+d56tl11l5733fn3jqp79lr7VXdHQAAANidAxZdAAAAAMtNcAQAAGBIcAQAAGBIcAQAAGBIcAQAAGBIcAQAAGDowEUXsCwOO+ywvsENbrDoMgAAABbigx/84IXdvWOtfYLj1A1ucIOcddZZiy4DAABgIarqC7vbZ6gqAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQwcuugCA/eXkk09edAnrctJJJy26BACAy3HHEQAAgCHBEQAAgKGlC45VdWhVnV1VXVXPmuG4u1XVe6vqW1V1UVWdVlVHbWStAAAA28HSBcckT0yyY5YDquqeSd6Q5EpJfjvJU5L8XJL3VNV1514hAADANrJUwbGqfibJI5Os+8kQVXVQkmcm+WKSO3T3s7v7lCQ/n+TaSZ6wAaUCAABsG0sTHKvqCkmen+TNSV47w6HHJLlukr/s7kt2NXb3h5OckeS+03AJAADAXlia4JjkUUmOTvKwGY+71XT7vjX2vT/JVZPceB/qAgAA2NaWIjhOH2JzcpIndvc5Mx6+aw7jeWvs29V2xF6WBgAAsO0tRXBM8twkZyd5+l4ce+h0+5019l22qs/lVNWJVXVWVZ11wQUX7MUfDQAAsPUtPDhW1QOS3CXJ/+ru7+7FKS6dbq+4xr5DVvW5nO4+tbt3dvfOHTtmepArAADAtrHQ4FhVV8zkLuMbk/xLVd2oqm6U5PrTLlebtl19cJovT7drDUfd1bbWMFYAAADWYdF3HK+UyZqNd0/ymRVfZ0z3P2D6+oTBOc6cbm+3xr7bJvlGkn+eQ60AAADb0oEL/vO/leTea7TvSPLsTJbmeEGSjyZJVR2e5GpJzu3uXcNP35nkK0lOqKo/3bUkR1XdLMmxSV64l0NgAQAAyIKD4zTQvWZ1e1XdYPrt57p75f5Tkjw4yXGZ3pXs7u9W1SOSvCrJu6vq+ZkswfGoJBckOWmDygcAANgWFj1UdS66+7Qk98jkyapPTfK7Sd6d5PbdbX4jAADAPlj0UNU1TddyrDXaj09y/G6OeUOSN2xkXQAAANvRlrjjCAAAwMYRHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABhaeHCsqptU1cur6pNVdXFVXVpVn6qqp1fV4es8xxlV1bv52rnR1wAAALCVHbjoApJcL8nhSf42yZeSfC/JTyU5Mcn9qurm3f3VdZznwiSPWqP97HkVCgAAsB0tPDh29+lJTl/dXlXvSvLqJMcn+ZN1nOpb3f2y+VYHAADAwoeqDnxhur3Geg+oqgOq6qpVVRtUEwAAwLazNMGxqg6pqsOq6npVddckz5vueuM6T3FEkkuSXJzkkqp6bVUdvRG1AgAAbCcLH6q6wglJnrni9TlJHtDd717HsZ9P8p4kH03y/SS3SfKwJHeqqv/S3f8051oBAAC2jWUKjn+X5FNJrpLkFknukeSw9RzY3Q9Z1fSaqnpdkjOSPD3JXdY6rqpOzOQhPDnyyCP3qmgAAICtbmmCY3d/KZOnqibJ31XV3yQ5s6oO7e5T9uJ8754+YOe4qrpSd397jT6nJjk1SXbu3Nn7UD4AAMCWtTRzHFfr7o8m+cckv7EPpzknyRUywwN2AAAAuLylDY5TV0pyzX04/scyWRfyovmUAwAAsP0sPDhW1XV2035ckp9M8v4VbYdX1dFVdeiKtqtV1RXWOP7uSW6f5K3dfdn8KwcAANgelmGO43Oq6vAkb89k7cZDktwyyf2SfDPJo1f0PSXJg5Mcl8mDbzL9/ulV9fokZ2dyh/HWSR6Q5MIkj9z4SwAAANi6liE4vjLJg5I8MMmOJJ1JgHxekqd097l7OP7TSc5K8gtJrp3koEwesvPcJE/u7vM2qG4AAIBtYeHBsbtfneTV6+x7fJLjV7V9Msl95l4YAAAASZZgjiMAAADLTXAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABgaOHBsapuUlUvr6pPVtXFVXVpVX2qqp5eVYfPcJ67VdV7q+pbVXVRVZ1WVUdtZO0AAADbwYGLLiDJ9ZIcnuRvk3wpyfeS/FSSE5Pcr6pu3t1fHZ2gqu6Z5DVJPpLkt5NcLckjk7ynqnZ295c3sH4AAIAtbeHBsbtPT3L66vaqeleSVyc5Psmf7O74qjooyTOTfDHJHbr7kmn7m5J8MMkTMgmhAAAA7IWFD1Ud+MJ0e4099DsmyXWT/OWu0Jgk3f3hJGckue80XAIAALAXliY4VtUhVXVYVV2vqu6a5HnTXW/cw6G3mm7ft8a+9ye5apIbz6lMAACAbWdpgmOSE5JckMmQ07ckuXqSB3T3u/dw3HWn2/PW2Ler7Yi5VAgAALANLXyO4wp/l+RTSa6S5BZJ7pHksHUcd+h0+5019l22qs/lVNWJmc5/PPLII2epFQAAYNtYmuDY3V/K5KmqSfJ3VfU3Sc6sqkO7+5TBoZdOt1dcY98hq/qs/jNPTXJqkuzcubNnrxoAAGDrW6ahqpfT3R9N8o9JfmMPXXcttbHWcNRdbWsNYwUAAGAdljY4Tl0pyTX30OfM6fZ2a+y7bZJvJPnneRYFAACwnSw8OFbVdXbTflySn8zkyai72g6vqqOrauWcxXcm+UqSE6rqKiv63izJsUlO6+7vbkTtAAAA28EyzHF8TlUdnuTtmazdeEiSWya5X5JvJnn0ir6nJHlwkuMyWaMx3f3dqnpEklcleXdVPT+TJTgelclTWk/aP5cBAACwNS1DcHxlkgcleWCSHUk6kwD5vCRP6e5z93SC7j6tqr6d5LFJnprJE1ZPT/K73W1+IwAAwD5YeHDs7lcnefU6+x6f5Pjd7HtDkjfMrTAAAACSLMEcRwAAAJab4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMCQ4AgAAMDQwoNjVd24qp5YVe+vqguq6ptV9eGqekxVXXmd5zijqno3Xzs3+hoAAAC2sgNnPaCqdiS5V5KbJrlyd5+wov2oJP/U3d+e4ZS/muQ3k7wuycuTfDfJcUn+IMl9quq26zzfhUketUb72TPUAgAAwCozBceqemiSP09ySJJK0klOmO6+dpL3JTkxyQtmOO1rkpzS3RevaHtuVX0myWOSPDTJs9Zxnm9198tm+HMBAABYh3UPVa2quyQ5Nck/J/kfSZ6zcn93fyzJx5P80iwFdPdZq0LjLq+abn9yhhoPqKqrVlXNUgMAAAC7N8scx99N8pUkx3T365J8dY0+H03y4/MoLMn1ptvz19n/iCSXJLk4ySVV9dqqOnpOtQAAAGxbswxV3Znkr7v7G4M+X0pynX0rKamqKyR5XJLvJXnFOg75fJL3ZBJcv5/kNkkeluROVfVfuvuf9rUmAACA7WqW4Hhwkm/toc/VMwlu++oZSW6X5Pe7+9N76tzdD1nV9Jqqel2SM5I8Pcld1jquqk7MZE5mjjzyyH2pFwAAYMuaZajqOUluuYc+t0myx6A3UlVPyuRu4andfcrenqe7353kXUmOq6or7abPqd29s7t37tixY2//KAAAgC1tluD490nuUFX3XmtnVT0kyU8n+Zu9LaaqnpDksUlemOTX9/Y8K5yT5ApJrjGHcwEAAGxLswxV/ZMk90vyyqr65SRXS5KqeliSOyS5Z5LPJHnm3hQyDY0nJXlxkhO6u/fmPKv8WCbzJC+aw7kAAAC2pXUHx+7+WlUdk+QlSVbedfzz6fbdSe7f3XuaB/lDqurxmYTGlyb51e7+wW76HZ5JYD23uy+dtl0tySXd/f1Vfe+e5PZJ3tTdl81aEwAAABOz3HFMd5+b5Niq+ulMHl5zrUyWv3h/d39wbwqoqt9McnKSc5O8Lcn9Vy3DeH53v3X6/SlJHpzkuEwefJPp90+vqtcnOTuTO4y3TvKAJBcmeeTe1AUAAMDEuoNjVf1ckm9094e7+6OZLH0xD7eabo/MZJjqau9M8tY12nf5dJKzkvxCkmsnOSiTZUGem+TJ3X3enOoEAADYlma54/iOJM9L8hvzLKC7j09y/N727e5PJrnPPGsCAADgP8zyVNULk3x7owoBAABgOc0SHM9I8rMbVAcAAABLapbg+NgkN6mqJ1XVQRtVEAAAAMtlljmOv5fkY0l+P8lDq+ojSf4lyer1Fru7Hzqn+gAAAFiwWYLj8Su+v870ay2dRHAEAADYImYJjkdtWBUAAAAsrXUHx+7+wkYWAgAAwHKa5eE4AAAAbEPrvuNYVUeut293n7t35QAAALBsZpnjeE5++Amqa+kZzwsAAMASmyXgvSRrB8erJ7l5kusnOSOJuZAAAABbyCwPxzl+d/uq6oAkj0vy60kevO9lAQAAsCzm8nCc7v5Bd5+cyXDWP5rHOQEAAFgO836q6nuT3HXO5wQAAGCB5h0cr5nkynM+JwAAAAs0t+BYVXdOct8kH5vXOQEAAFi8WdZxfPvgHD+SZNc6j0/c16IAAABYHrMsx3Hsbto7ydeSvCXJU7t7dwETAACATWiW5TjmPR8SAACATUAYBAAAYGifg2NVHVRVt6iqm8yjIAAAAJbLuoNjVd2nql5dVddc0XbDJB9PclaST1TVa6tqlnmTAAAALLlZ7jj+apKju/uiFW1PS3KjJO9I8tEk/z3JQ+ZXHgAAAIs2S3D88SRn7npRVVdNcrckr+7uOye5dZJPRXAEAADYUmYJjjuSfGXF69tl8lTWv06S7v5ukrcmueHcqgMAAGDhZgmO30xytRWvj8lkDcf/t6LtsiT/aQ51AQAAsCRmeZDNZ5L8t6q6YiaB8T5JPtrdF67oc/0kX51jfQAAACzYLHccT03yo5kEyE8mOSrJC1f1uWUmT1kFAABgi1h3cOzuFyf5oySHZjJk9VlJnrlrf1X9bP7jCasAAABsETOtudjdv5/k93ez+6wk10jyrX0tCgAAgOUxU3Ac6e5/S/Jv8zofAAAAy2GWOY4AAABsQzMFx6o6vKr+oqo+W1Xfrqrvr/H1vY0qFgAAgP1v3UNVq+qIJB9Icu1Mnpx6xSRfSPKdTJ62emCSDye5eP5lAgAAsCiz3HF8fJLrJPmv3X2zadsLu/voTILjW5JcKck951siAAAAizRLcPz5JG/u7ret3tHdX0py70yC48lzqg0AAIAlMEtwvE4mQ1R3+X4mQTFJ0t2XJHlrkv8+n9IAAABYBrMEx28kOXjF668lOWJVn4uT7NjXogAAAFgeswTHLyT5kRWvP5LkjlV1aJJU1QFJ7prkS/MrDwAAgEWbJTienuS4qjpo+vrFSa6b5L1V9ZQk70nyE0leNd8SAQAAWKR1L8eR5AWZDE89LMlXuvtlVXXLJA9P8tPTPn+d5A/nWyIAAACLtO7g2N2fSfLHq9oeVVVPzmQ5jnO6+/w51wcAAMCCzXLHcU3dfUGSC+ZQCwAAAEtor4JjVR2d5KZJrtLdL51vSQAAACyTWR6Ok6q6eVWdlcl6jq9J8qIV+46pqkur6hfnWyIAAACLtO7gWFU3TnJGkpsk+bMkb1rV5V1JLkryy/MqDgAAgMWb5Y7jSUkOTnKb7v7/kpy5cmd3d5L3JbnV/MoDAABg0WYJjndK8tru/sSgzxczWdsRAACALWKW4HiNJF/aQ5/K5K4kAAAAW8QswfH8JDfaQ5+fyOSu47pV1Y2r6olV9f6quqCqvllVH66qx1TVlWc4z92q6r1V9a2quqiqTquqo2apBQAAgB82S3B8e5JfrKqbrLWzqm6VyXDWt8xYw68meVSSzyV5YpLfTvLpJH+Q5L1VdaU9naCq7pnkDUmuND3+KUl+Lsl7qsrQWQAAgH0wyzqOpyS5d5J3VdUTMp3LWFU/kUlIOynJN5M8dcYaXpPklO6+eEXbc6vqM0kek+ShSZ61u4Or6qAkz8zkTucduvuSafubknwwyROSnDhjTQAAAEyt+45jd386yb0ymcP4rCQnZDKn8aNJ/mLafs/uPneWArr7rFWhcZdXTbc/uYdTHJNJiP3LXaFxet4PZ7J8yH2n4RIAAIC9MMsdx3T3m6fzBh+c5LZJrpXk4iTvT/LC7r5ojrVdb7o9fw/9di3/8b419r0/yR2T3DjJx+dUFwAAwLYyU3BMku7+epI/m35tiKq6QpLHJfleklfsofuuOYznrbFvV9sRERwBAAD2yiwPx9mfnpHkdkkePx0iO3LodPudNfZdtqrP5VTViVV1VlWddcEFF+xdpQAAAFvczHccq+qwJDfNZCjpmnMHu/sle1tQVT0pycOSnNrdp6zjkEun2yuuse+QVX0up7tPTXJqkuzcubNnLBUAAGBbWHdwrKpDkjwtk+UzDt5dtySdZK+C4/RprY9N8sIkv77Ow7483R6R5JOr9h0x3a41jBUAAIB1mOWO41OS/K9MwtmrMglj35tXIdPQeFKSFyc5obvXewfwzOn2dknetmrfbZN8I8k/z6NGAACA7WiW4HifTJbeuFV3f3eeRVTV4zMJjS9N8qvd/YPd9Ds8ydWSnNvdu4afvjPJV5KcUFV/umIdx5slOTaTp73OtV4AAIDtZJbgeOUkb92A0PibSU5Ocm4mdwzvX1Uru5zf3W+dfn9KJkuBHJfJGo3p7u9W1SMyuQv67qp6fpKrJnlUkgsyCaQAAADspVmC48eTHL4BNexah/HITIaprvbOJG9do/3fdfdpVfXtTOZHPjWTJ6yenuR3u9v8RgAAgH0wS3B8apIXVdWNu3tucwa7+/gkx+9r3+5+Q5I3zKksAAAAptYdHKd39Q7PZDjos5N8KMnFu+n7rjnVBwAAwILNuo7jNTKZ6/j4PfS7wt6VAwAAwLKZZR3H38vkQTP/msmDaL6cOS7HAQAAwHKa5Y7jiUnOTnLL7l5ziCoAAABbzwEz9L1OktcJjQAAANvLLMHx7CRX36hCAAAAWE6zBMfnJPnFqrrORhUDAADA8plljuPrkxyb5L1V9cQkH8zul+M4d99LAwAAYBnMEhw/n6STVJIXDPr1jOcFAABgic0S8F6SSSgEAABgG1l3cOzu4zewDgAAAJbULA/HAQAAYBsSHAEAABgSHAEAABgSHAEAABgSHAEAABgSHAEAABiaa3CsqkOq6qrzPCcAAACLNe87js9JctGczwkAAMACbcRQ1dqAcwIAALAg5jgCAAAwJDgCAAAwJDgCAAAwJDgCAAAwJDgCAAAwdOBoZ1V9f38VAgAAwHIaBsfs3dIavTeFAAAAsJyGwbG7DWUFAADY5gRDAAAAhgRHAAAAhgRHAAAAhgRHAAAAhgRHAAAAhva0HAcAABvo5JNPXnQJe3TSSSctugRgwdxxBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYOjARRcAAACwKVQtuoI9696Q07rjCAAAwNDCg2NV/V5VnVZVZ1dVV9U5e3GOc6bHrvV12AaUDQAAsG0sw1DVJye5KMmHklx9H87zqSR/uEb7N/fhnAAAANveMgTHG3b32UlSVR9LcpW9PM/53f2y+ZUFAABAsgRDVXeFxnmoqgOr6qrzOh8AAABLEBzn6DZJLk1ycVV9vapeXFXXXXRRAAAAm90yDFWdh48n+cskn0xyUJJjk5yQ5E5Vdevu/vICawMAANjUtkRw7O67r2r666p6V5KXJzk5ya+tdVxVnZjkxCQ58sgjN7RGAACAzWorDVW9nO5+RZJzkqwOlSv7nNrdO7t7544dO/ZbbQAAAJvJlg2OU+cksY4jAADAPtjqwfFGSc5fdBEAAACb2aYKjlV1ZFUdXVUHrWi75m76/maS6yV5/f6qDwAAYCta+MNxquqBSa4/fbkjycFV9djp6y9090tXdH9JkmOSHJXJMNQkeVBVPTTJm6dtB2byVNVfSvK5JCdtYPkAAABb3sKDY5KHZhIGV3rSdPvOJC/N2JlJ7pjkvpkEz0ry+SR/nOSPuvvr8ysVAABg+1l4cOzuY/elb3e/J8k95lgSAAAAK2yqOY4AAADsf4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ4IjAAAAQ0sRHKvq96rqtKo6u6q6qs7Zy/M8qKr+saq+XVXnV9VfVtWOOZcLAACwrSxFcEzy5CR3TPK5JF/bmxNU1aOSvDjJxUkekeR5Se6X5IyquvKc6gQAANh2Dlx0AVM37O6zk6SqPpbkKrMcXFWHJfmDJGcmuVN3f3/afmaS12USJJ8814oBAAC2iaW447grNO6DX0pyaJJn7gqN0/O+PsnZSR6wj+cHAADYtpYiOM7Brabb962x7/1Jjq6qme5iAgAAMLFVguN1p9vz1th3XpJa0QcAAIAZbJXgeOh0+5019l22qs+/q6oTq+qsqjrrggsu2LDiAAAANrOtEhwvnW6vuMa+Q1b1+XfdfWp37+zunTt2WLUDAABgLVslOH55uj1ijX1HJOkVfQAAAJjBVgmOZ063t1tj322TfLq7L9mP9QAAAGwZmy44VtWRVXV0VR20ovnvk3w7ycOq6gor+v5ikh9N8vL9XCYAAMCWceCiC0iSqnpgkutPX+5IcnBVPXb6+gvd/dIV3V+S5JgkRyU5J0m6+4KqelySpyZ5W1W9MpMhqo9O8qkkz9jwiwAAANiiliI4JnloJmFwpSdNt+9M8tLsQXc/rar+Ncmjkvx5km8keXWS/22YKgAAwN5biuDY3cfOo293vyjJi/a5IAAAYF1OPvnkRZewRyeddNKiS9j0Nt0cRwAAAPYvwREAAIAhwREAAIChpZjjCAAAW4l5f2w1giPAFrQZfmFJ/NIC281meG/yvgRrM1QVAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAIcERAACAoQMXXQAA7IuTTz550SWsy0knnbTP53Cty2Ue1wmwWbjjCAAAwJDgCAAAwJDgCAAAwJDgCAAAwJDgCAAAwJDgCAAAwJDgCAAAwJB1HGE3rCEGAAATgiMgJAMAMGSoKgAAAEOCIwAAAEOCIwAAAEOCIwAAAEOCIwAAAEOCIwAAAEOCIwAAAENLERyr6oCqelRVfaqqLquqL1bV06rqyus8vnfzdZsuXx8AACAASURBVMlG1w4AALDVHbjoAqb+NMlvJfnbJE9LctPp61tU1Z27+wfrOMe7k5y6qu27c60SAABgG1p4cKyqn0jy8CSv7e57rWj/fJI/T3K/JK9Yx6nO7u6XbUyVAAAA29cyDFX9n0kqyTNWtT8/yaVJHrDeE1XVwVV1lTnWBgAAsO0tQ3C8VZIfJPnAysbuvizJh6f71+OXMwma36yqr1bVM6vqanOtFAAAYBta+FDVJNdNcmF3f2eNfecl+dmqOri7/21wjg8kOS3JZ5NcNcndkjwsyTFV9bPd7SE5AAAAe2kZguOhSdYKjUly2Yo+uw2O3X2bVU0vqaqPJvnDJI+Ybn9IVZ2Y5MQkOfLII2coGQAAYPtYhqGqlya54m72HbKiz6yekknYvPvuOnT3qd29s7t37tixYy/+CAAAgK1vGYLjl5McVlVrhccjMhnGOhqmuqbu/u6uc+9jfQAAANvaMgTHMzOp49YrG6vqkCQ3T3LW3px0evz1kpy/rwUCAABsZ8sQHF+VpJM8clX7r2Uyt/Hluxqq6oZVdfTKTlV1rd2c90mZzOF8/fxKBQAA2H4W/nCc7v6nqvqLJA+rqtcmeWOSmyb5rSTvTPKKFd1PT3L9TNZ93OWxVXXbJO9Icm6Sq2TyVNXjkvxDkmdu+EUAAABsYQsPjlOPTHJOJk84vXuSCzMJfI/v7h/s4dgzkvx4kgcnuVaS7yf5TJLHJHn6dD1IAAAA9tJSBMfu/n6Sp02/Rv1usEbb3yf5+42pDAAAgGWY4wgAAMASExwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYEhwBAAAYOnDRBbC5nHzyyYsuYY9OOumkRZcAAABbijuOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADAmOAAAADC1FcKyqA6rqUVX1qaq6rKq+WFVPq6or74/jAQAA2L2lCI5J/jTJ05N8IsnDk5yW5LeSvL6q1lPjvh4PAADAbhy46AKq6icyCXuv7e57rWj/fJI/T3K/JK/YqOMBAAAYW4a7cf8zSSV5xqr25ye5NMkDNvh4AAAABpYhON4qyQ+SfGBlY3dfluTD0/0beTwAAAADyxAcr5vkwu7+zhr7zktyWFUdvIHHAwAAMFDdvdgCqj6X5KDuPnKNfS9J8sAk1+jur8/7+Ko6McmJ05c3SfLpvb6Q+TosyYWLLmI/ca1bk2vdmlzr1uRatybXujW51q1pma71+t29Y60dC384TibzEP/zbvYdsqLP3I/v7lOTnLqnAve3qjqru3cuuo79wbVuTa51a3KtW5Nr3Zpc69bkWremzXKtyzBU9cuZDCe94hr7jshkGOq/beDxAAAADCxDcDwzkzpuvbKxqg5JcvMkZ23w8QAAAAwsQ3B8VZJO8shV7b+W5NAkL9/VUFU3rKqj9/b4TWTphs9uINe6NbnWrcm1bk2udWtyrVuTa92aNsW1LvzhOElSVc9M8rAkf5vkjUlumuS3krwnyR27+wfTfudkMmGz9uZ4AAAAZrcswfEKmdwxPDHJDTJ5qtCrkjy+uy9Z0e+crB0c13U8AAAAs1uK4AgAAMDyWoY5jgAAc1NVR1bVlQb7r1RVP7T+MwC7JzgCAFvN55P8j8H+e0z7ACydqrraomtYi+C45KrqAVX19kXXAXurqn6kqn5u0XXsi6q6YlX9RlU9u6pOqqof202/O/t5ZZlV1X+uqoMH+3ds9p/XqdrD/gMyeSL7trAV3oe3m5o4eFXb1arqIVX16Kq6/aJq2wjb6L0pVXV6VV1nsP/2ST68H0taN8Fx+V0/yTGLLmJ/qarHVNX3Fl3H/lBVv1BVf7XoOvaDByV5x6KL2FtVdWiS9yd5ZpJfT3JSko9V1e+s0f3a2UY/r1tFVd2tqn6nqh5cVVffTZ/bbuaf16q6f1V9JclXklxcVS+sqmut0fWu2cQ/r6uMguFNk3x9fxWyBDb1+/B6VNVRVXXL6Trem1pVPSaTf5+XVtX/raprVdVRST6Z5AVJnpLkXVX1gkXWOQ/b9L3pZ5N8pKrutrJx+mHB4zK5zqXMaEtZFNvenj4p3ipuluTBiy6CPXpkJn9XT07y00nunuSDSU6pqmcvsrCNUlVHV9Wrq+oTVfWOqlrz32lV/UpVfX9/1zcvVXVgVb0pyeuT/FGSFyY5u6rus0b3G2aT/rxW1a2TvDTJgZksW/WRTK7lQ1X144usbZ6mwf/tK+76P3bX61VfH07y+9k6v4RuG1V1bFW9oareU1W/X1UHVNWhVfXGJJ9N8oEk5+/uPWszqKpfSvKkTIZSvyXJnZI8O8mfJTkjyX9Lcs9M/v0eX1X3W0yl+267vDet4TZJLkry+qp6elUdVFVHJHl7kpOT/J8kN19kgbtz4KIL2I6q6uwZui/lGOdZzPgAgjU/7We5VNWDZuh+iw0rZP+4d5JXdffjpq8/VlVvyeR/4r9ZVQd1968trrz5qqrrJXlfJu89F2USmH6uqu6d5H5bbImjX0vy80lenskSTj+S5LeTvLKqjuruP15kcXP0mCTnJdnZ3V9NJndZM/mF7R1VdZfu/ugiC5yTqyc5avp9J9mR5NBVfTrJJUn+KpP/LpvWNnsfTlXdPJMgVUkuS3LbTP5+r5bJ+9SfJLlSkvskeUFVfaq7/2FB5e6L30ryD0lu390/qKrHJ3lskv/b3fff1amqXpfk40kekuSvF1Lpvtsu702X090fraqfSfKsTD6cvlOS6ya5cpKHdffSfigtOC7GDZJ8LcmX19F39f/0NqNzsk3mksw4v+36G1bIxntRJn+n6707vJn//m+Y5DkrG7r7B0keXlVfT/KYqjqgux+6kOrm73FJDk5yl+4+vaqumskvLY9O8raq+vnuvnihFc7PQ5K8ubsfuKuhql6SSZB8clUd3N1PWlh183PzJM/d9YtZknT3G6vqtklOT/L2qrpzdy/lnJr16u4/y+QDnVTVD5I8srtfsdiqNtSLsn3eh5Pkf2cynPE2mazX/Yokv5Hkn5P8THd/K0mq6o+SfCzJI5Lcf+1TLbWjkzx1+v+ZJHl1kickOW1lp2mofGWSh+3f8uZqW7w3raW7v11Vv57kxklun8nP58OXOTQmguOifD7JZ7v75/fUsaoem8lt683se5kMITl9HX13Jrn1xpazoY7N5Hq/s46+B21sKRvqW5lM3H7GOvreK8l9N7acDXVZdvN31d2Pmw7VfHxVHZDJMJPN7phM/kd+epJ09zeS/E5VfSDJyzIJj3fp7q0wP+xGmfzy/e+6+1tVdc9Mhq0+oaqu0N1PWEBt83RYJr9wX053f6aqjslkyNvbququ+72yDdLd22EqznZ6H04mvxu8sLvPT5KqekomI0Jetis0Jkl3/8v0A6B7LabMfXbNTEZ77HLhdHveGn3Py+Yembbt3pt2qaobZnKn+Gcy+bDyvyR5xnR+5x9091J+0CM4LsYHkxy3zr5L+Q9nRp9Icml3P3xPHacTwjdzcDwvyYe7+xf31HGTfyjwkSRX7+6/2VPHqjp6P9SzkT6XyZCov1hrZ3c/oaqS5PGZfHCw2R2Zyaf1l9Pdr6mqS5P8TZLTq+rO+72yjfFDczSnn+QfP933uOmHAp/e34XN0ZfzH0M4L6e7P19Vx2Yyd+ptSV6y/8piH22n9+FkMpTv3BWvvzjdfnaNvp/O5GFlm9FFmQSqXb6f5F8z+RBztWsm+eb+KGqDbMv3pqr6lUzmrf4gkykgp9Vk+Y2/yuT3wuOq6le6+4dC9aJth0/kltE/JrlWVd1gHX2/kORdG1rNxvtQkptNf/na6j6U5Jbr7LuZPxT4UJKja7DA9iqb+YFHb01yj6r6T7vrML0j9YRs7uHHu1yYy//S8u+6+42ZrI3345mMIFjryXebyeezm5/X6ae9D03y4kzm4Tx6P9Y1b2dl8lCnNXX3OZl86HFxkj1+wLdZVNUNq+pZVXVmVX22qs5e9fW5Rde4j7bT+3AyCUgr34d3PYH939boe/Bu2jeDT2fyHpsk6e6vdfeO7v5/a/S9Sf4jQG9G2/K9KZM5nJ9IcovuPi1Juvvi7r5XJsOvb5PJB0NLZzv8Ir90uvuU7j5g+gOxp74v6+713p1cVm/K5Eln111H33cleeLGlrOhPpTkOlW1nmu9OJf/9HQzeWmSP0hylXX23cz/hl+aySeDa67duEt3PzGT/7Ft9k9FP5nkjrvb2d1vziQ8Hp3Jk0g3s3ck+aXazdphK8Lji7KkT7hbp/+T5BZVdYfddVjxC9oX9lNNG6qqfiqT9+MTMgkRP5rJ0M5DMnnOwPezed9/d9lO78PJ5N/mj654/bUkP5XJckmrHZU1hkBuEq/POp5vUVVXTvLLSd694RVtnG333jT1lCR3WCsHdPdzMwmO5+/votajlnQILbsxvWt3vST/0t2b9dO0LWv6Rn5Ykq/4+2Ez/rxW1f/f3r3H2zbX+x9/vYm2Q4hEJ5VEJd10k5NrbBxdcFRO2rGRRE5UihxKl5OSLuKRUmRz7KSE9OuBart14UR0QxRSEiJyy22/f398x7Kntddae61trjnmGOP9fDzWY5rjO/bymY/5/Yw1vt/xvewLHAE81/b1E5y3JXAmsLTtJQcVXz9JWg/4IPBF2z9bxLlHAC9vakeepCcCD9uecPsUScsBK9tu9E2apNMpc4Y2pAzzuxXYwvY8SXtQttfZxPaVNYYZUyDpy5RFcCaczqIyd+A64CLbU1l5tlFU9hheG/iz7dvrjmdxde3aNFmSZtgea3hyrfLEsXlWoQyv2rDuQAZB0gxJO0tqxFwF2/fa/mNTGgl1kLS8pONbMudmUZqYr3OA1zPG3L9ets+lfK7dBhHUdLB9ue23LqrRWJ27f1MbjQC2H1jUjVl13j29N2YNztcNgWNt/44F0wIEYPurlJEwTX9ivtga+r3uw+Tmka9MWTBoMosGNZbt+2z/srfR2MTvtYPXpkkZxkYjpOHYVE2fpzAVK1BWN1y37kCmm6RVJT0iadxhgi2xDGWD38kM522DRuWr7Tttn2N7kcP4bF9me87I+6Z19Dweydeh9yTKwlawYK7bsj3lP6FZHTr91rjv1fbDtu+bxHl/s32k7V+MHOvQtalx3+vj0OjPKukJkt4k6TOSvlY1gnt/jqs7xrFkVdVogkbdeD9OXfmsXfmcXTPS0TOTIZ2f0WddqcdN/Jy3AKsB2L5b0r2U/dJGPBlo5BDrPmri97q4unRt6tL32sjPKmklyhz7F1I+Q+9+rO45NnT7Q+eJY0RE9FMj/5BH61xB2Rd4xAXAvpI2rpb434chXbUwpk2uTTEsPkFZYO4dwHModXMrYB3gG8DPGdJVy9NwjIiIiLaZCzylZ6uKQyhPnc6jbCWzInBQTbFFRLe9DjjR9teBf1THHrH9O9uzgPuBw2qLbgJpOEZERESr2P6m7Y1t31+9v5wyV/69wHuAF4+zL15ExHRbjfJUERbsRzqjp/wM4I0DjWiSMscxIiIiWs/2n4Av1h1HRHTeHSxYrOtu4CHgGT3lD1HmYQ+dPHGMiIiIiIgYjGuAFwDYng9cDsyW9MRqf86dKXuRDp08cYwm8KJPiYiIrpJ0/GL8M9seulULI6L1zgX2l7SP7QeAzwGnUJ5EmrLVyDtrjG9caTjWTNKLbf9qCv/kDmAzyopxXZGV0NrlQcoKh3+vO5CpSr5OSjp62qUp+Tp7Mf7NUC53PyBN+V77qQvXpi59r03+rJ8Ejqgajdg+VdLDwCzgEeDbtr9ZZ4Djkd2FPBpekuZTbirnAHNt31ZzSFETSTOAHYGzbTdynylJzwPWrN5eZ/t3dcbTb8nXiVWba98MbGF7Xt3xTKfk63CR9HXgK7Yvrt5vAlxl+9Z6IxssScsCGwCrAj9sat3stzZcm9qUrxNJHR5uaTjWTNJhwE6USbEPAedQbkrPsv1gnbFNB0kfnsRptv3xaQ9mQCS9Ctiengs+cIbtS+qLqr8kvRY4irIvUa+rgffY/tHgo+q/ruVrFyVfm5mvkh4B3m57bs/7Wba/UW9kgyNpL8oS/stTnq7NtD1P0lOBG4H/sv3VOmOMqWtjvo4ndXj4peE4BCQJeC1lMuz2lJWW7qSMdz6xZTcs8ycoNmVYqm0vOaCQpo2kJYFjKUOoRg+3NXAi8A7bjww4tL6q/qidDTwAnAxcWRWtC7yVssT01k3t5R2tY/namY6e5Guz81XSrcCnbH+uej+f0nCcW29kgyFpB+BbwJnAWcDX6Hm6JukMYCnbr6svyv7pyrWprfk6lg7W4WUpHdFrAyszxt+dYZyDnYbjkKlWU9qBclO6GaUiXWN7nVoD6xNJzxrj8BOA51D211oB2MX2tQMNbBpI+gjwEcp+PIfz2Av+Byl79HzM9kfribA/JF0MPB14te2bRpWtDlwM/Mn2BnXEN506kK9d6uhJvjY4X6ubyo0oDfy/A4cC3wEmmpPc+IbFiOp7vdf25pJWBm7jsTfdBwN72B7rb3DjdOXa1NZ8HUuX6rCkfwO+C6w0wWlDWX/TcBxiknYCvgQ8aRgrT79VT3IuBC6yfVDd8Txekv4IXG17q3HKfwA8t+kXQUn3AZ+0/Ylxyg8BPmT7XwYb2WC1MV871tGTfKW5+SppDcqw8Q2pGg0semG1obwxWxyS7gUOsH30ODfduwNH216mzjj7pSvXprbm61i6VIclXUaZ8rI75Z73zppDmrTs4zhkJK0l6WOSrgNOAv4F+F7NYQ2ESy/GtylPb9rgqZQepfGcUZ3TdHdRNrAdzz8oQzlbp+35avuPY/z8wfa5wDaU1d92rTnMfkm+Fo3MV9s32N6Esoz9GpRG437Asyf4WXPMX9ZMjzDxPd2/AvcOKJZp16FrUyvzdRxdqsMvAD5j+6wmNRohDcehIGlFSe+S9FPgd8DBlIvF+4Gn29621gAHa2nKWO82uAZYbYLyp1XnNN23gLdKWmh7H0lLUeZhfGvgUU2T5GvRwo6e5GsL8tX2g7ZvpDx9vGScBsajP3XH20e/BMZ7Wr4E8Gbg5wONqCYtuza1Ol9H6VIdvpmywF7jpOFYM0nfplSgL1F6QD8PvNT2era/4A4t9y/pFcC+wFV1x9InhwHvlvSS0QWS1gP2puzl03RfpgwRulDSmyW9qPp5C2Xo8ZLAlyU9s/en1ogXU/J1IW3q6Em+tihfbe/apoWqJuFo4N8lfZwF86aWqLZw+BZlru4X6wquBm25NnUiXytdqsNfA3aqFmVrlMxxrJmk+ynDo+YA5zR9xb5FqYb0jWUl4EnAw8C2ts8eXFTTo1r1bVvgxcC5lKWzAdYBZlJ610YPjWvcYg3VIgWPLkYwurh6XehC08S5RV3L14lUHT1nALfaflnd8TxeyddSXL22Il+7RtIngIOA+ZQHA/Mp36mAQ21/rMbwBqZN16au5WtX6nC1psdRwKuAY4AbKEN1H8P2hYONbNHScKyZpBVs31V3HIMi6XwWvsgZuIMyDOxY2zcMOKxpsYhV38bTuMUaJB3KGH+4FqWJq1N2MF+71NGTfJ1AE/O1iyS9DHgbZc8/AdcCJ9m+tNbA+qwr16Yu5msX6nC1IvtxwFvGO4Uh/fuShmPENBln1bdFatm8m2iwjnX0JF8jGqJL16ZoH0lfp8zBPQO4iLKF0EJszxlkXJORhuMQqMaj78nEm4BuPvDAImIhydeIaApJL6c8hbvI9j/rjidiqtpYhyXdAZxme4+6Y5mqhVZpisGS9O/A6ZSJ3PcAt9cbUfSLpBfYvnIR52xv+/RBxTRdqhXPdgG2Z8ES99dRNuA+0fbiDAMcOsnX9kq+ti9fu0TS/sAmtt/Qc2wusGP19jpJG9q+pZYAY7F1JV87VodFQ1eIzRPHmkm6AngKsF2bxm+P6JnYPRW23fhODUn3APvZ/toYZU8EjgT2GMYx7FMhaRng+8DGlO/65qroaZSL4wXANm3oKWx7vnZZ8rV9+dolki6lbD/y7ur9a4EfAt8Afk3ZNugrtt9fX5QxVV3K1y7VYUmnA3fY3r3uWKaq8TfnLfB84OAW34SeyGJM7G6JXwBfkTSTcsP5DwBJ6wKnUJaW/lKN8fXLwcAmwBHAYbb/DmW/Q+BDwAeA/wYOqS3C/ml1vna5o4fkaxvztUvWAE7oeb8dpZExy7YlPQV4I2W/2cbp8LWpS/m6Bi2uw6PsB8yT9D7gaNsP1h3QZOWJY80k/Qn4jO227E0TlWp4yaGUpaX/CMwCXgp8Frgf2N32GbUF2CeSfg9cavs/xyk/BXiF7bUGG1n/tT1fJZ3A4q3gt2v/oxms5Ouj5a3J1y6ptgrax/Zx1fvfAJfZ3qV6vxtwlO1lawxzsXX12tSlfG17He5VrQq8LGUE0yOUBvLo7Ths+zmDjm1Rmt4T0wYnATvQnk1No1LNO/iwpHmU7/nHVdFFlB60P9cWXH+tTukNHc8FlJ7DNmh1vtqeXXcMdUm+PqpN+dolNwEvgkdXCH4B8Lme8icDD9QQV190+NrUpXxtdR0e5UYaOhovDcf6nQBsJulMyhya6xl7E9AbBxzXtJK0MbAlsCrwWdtXS1oOeBnwK9t31hpgf90HPMSC1Td/D/ytvnD67k5got7Otapz2uAEOpivHZN8bU++dslZwN6SngCsT7nB/n895S+kbDIezdKlfO1MHba9ad0xLDbb+anxB5hPufEceR3zp+44+/h5lwS+Oeozv7Yqm0HZg+mguuPs4+c9gHLx+yOwOeVJ1XzKRO8X1B1fnz7jSZShfFuNUbYl5UZ8Tt1x9umzdipfez73xsAngK8Cz6+OLVcdX7Hu+Pr4OZOvLcrXLv1Qnsb8qKqv9wN79pQtQ2lcfLbuOKfhc7f62tSlfO1SHa7q5yoTlD8F2LjuOMf6yRzHmkk6lEk8rrb90emPZvpJOgj4GGVy89nAVcAWtudV5ccBz7O9YX1R9oekc4CZwHeBXb1gUvsbgeMpF8L32j62vigfv2pIyc8pexpeDvy2KloXWI/ytOZVbsFG6R3M1yWBucCbKE/gDMy0PU/SDOAvwBG2P1ljmH2RfG1fvnaRpOWB+20/1HNsGeC5wJ9s31FbcH3UlWtTF/O1C3VY0iPA223PHad8R2Cuh3AV7zQcY6AkXQ381PZuklYGbuOxDcf9gffbflqdcfZDNdH7A7aPHqPsXyl/9DYaxgvDVEl6JnAY8AZKby/A3ZShJwc5QzcbqWMdPcnX5Gs0RMeuTcnXlqlWCZ41QcNxJ8qT5KUGG9miZY5jDNoalFUKx3MnZbhCG2xg+4qxCmz/RdJmlKW2G6/6w/U2SQJWqQ7f5vRMNd3OlA2mj6w6eka7CthmwDFNl+Rr8rXxJK0NrE15QqXR5bZPHHhQ06Mz16au5WuH6vBE39+/MaRz69NwHALVMvC7ANsDa1aHrwO+Q7kwzq8rtmlwN7DSBOVrUZ5CNt54N6E95QY+PqBwpkW1oNF3gZNtH1d9pltrDmtadSxf16AjHT3J12gySasCcyjDrWGMG27KjWpbbrrXoOXXpq7la9vrsKR9gX17Dn1B0v+MceqTgeUpUySGThqONavGbX+fMlHWlL1coPSUvQ7YWdI2tv9ZU4j99mNglqTDRxdIejKwG2XYSTSA7XskvRI4ue5YBqGD+dqZjp4u6Fq+dszRlBvuY4B5wO31hjPtWn9t6mC+tr0O30lZeA1Kx8ftwC2jzjHwG+Bi4PMDi2wK0nCs38HAJpR9eg7rWZBhReBDwAeA/wYOqS3C/vofSuNxHmVrA4CXVEMTDqRsiPqpekLrP0mvoXyP61N6kUb3oNl20/PwCmCduoMYkK7la6c6epKv0WAzgS/b3qfuQAakK9emLuVrq+uw7TmUJ6pIuh440PZ3641q6paoO4BgR+BU2x8cuQkFsH2n7QOAU4G31hZdn9m+lLKB+vOBr1eHj6D0MC0DbG/7yprC66tqr8rzKDehl1Dy7TzKCmmi9CqdVFuA/fMRYI9qDljbdSpfKR09a1M6el5fHXuJpD2BX9Cijp7kazTcEsAv6w5igLpybepSvra2Dkt6pFrwZsT5LPy0sRGa3nPaBqtTGk7juQDYbkCxTKtqmN+bgd9RHtNvSWlACrgWOMf2fbUF2H//TRnK+ArK8INbgU9Wy4VvCXwb2LvG+PplFnAj8ENJvwSuoewt1cu2dx94ZP3XmXyF0tEjaQfgazy2o0eU+tyajh6Sr73akq9dchHwkrqDGJQOXZu6lK9trsPzKfuYj9gZ+AGlk7JR0nCs352UsfjjWas6pw0eoFzk32P7EspS0mfVG9K0ehXwOdu3SRqZi7EEgO1zJZ1EWWzjtXUF2Ceze/77pdXPaAba8IetM/nawY6e5OsCbcnXLnkfcJ6kebZPqzuY6dSxa9Psnv9ue762uQ7fCGzEglErI3uPNk4ajvX7AfBuST+wfU5vQdXLvRfwrVoi6zPb8yXdSFktqgueCNxU/fcD1euTesqvoPQmNprtLg1570y+0r2OnuRrNNkxwD3AqZL+Qlnp+ZFR59j25gOPrP86c23qWL62uQ6fBHxY0ptY0Lk83qqqI2z7OdMf2tSk4Vi/g4GtgO9Luhz4bXV8XWA9yj4uH64ptukwB3i7pCNtP7DIs5vtZsrQRmzfK+lO4IXA6VX5bYmI2QAAC0ZJREFU6sDDNcUWi6cz+drBjp7kazTZmpQnGCObwT+zxlimVQevTV3R5jr8UcqKqlsAqwHPYuxVVYeeWrp/aKNIeiZwGPAGYLnq8N2UHrSDqs1fW0HS5pR5CDOAL1GGlSw0pMT2hQMOre8knQI82fZWPe+3BPajDIH7LHCJ7VZsUtwVHcvXQ4C3AK9oe0dP8jWiObp0bYr2kTQfmGV7bt2xTFUajkNEkoBVqre3uYVfTpUsvUZ/RlEezy9Jw0maSZmf8A7b90takzL5+2nVKX8FtrT9m5pCXCyS5i3GP2vq8JJxdSRfu9TRk3xdoHX5Gu3S1mtT8rUbJG0CXGm7cXuNpuEYAyVpl8mcV+130zqSlqUMVXgY+LHtu2oOacok3cDCDf5lgadU/z0yfn/F6vVvwD2215z+6KKfutTRM5bkazSNpDUodXZV4GTbN0hamjI87q+2H6wxvL5p67Up+dqdOgyPdkCvRxmmC2Ve5+XD3BGdhmPNJL2bsmz0FuOUnwucZvsrg40sYnKqJzPnAd8BPm37r9Xx1YADKdtTbGb7+vqi7I+u5WvXO3raqEv52jWSPk1ZmXJJSuNjZrWdzPLAX4CDbX+hzhj7pSvXpq7la8fq8NaUp+XPGlV0A7D36AX4hkUajjWT9HPgUtt7jVN+NPBK2+sPNrLoh2rD13dTNipeeYxTbLvRi1RJOhO4z/aYG99Xc8Vm2G78/obJ13ZLvrYrX7uk2vj+GOCLwPeAc4EtbM+ryucCq2ZIY7N0KV+7VIclvYbSIXAvZR/S3oX2ZlOeMm9m+6e1BDiBRv8BbIm1WbB57Vh+C+w0oFiijyQdTFlJ6xbgp8Df641o2mwKHDBB+fnApwYSyfRLvrZU8vVR59OefO2SvYHTbe8naaxOj18B+ww4pnj8NqU7+dqlOvxhyrz59W3f3Fsg6TPAJdU5W9cQ24TScKzfUpTJ3eOZsYjyGF57Uy7qW9t+qOZYppOBdSYoX3dQgQxA8rW9kq9Fm/K1S55LeVoznttYME8umqNL+dqlOrw+cMToRiOA7ZslfRV4/+DDWrQubSw6rK4BZk5QviXwhwHFEv21PHBqy29CoQwn2UvSztVEb6BM+q7moexZndMGydf2Sr62L1+75J+U4W3jeRYLFlaJ5uhSvnapDi9N2cZrPP+ozhk6aTjW7xvAlpI+Xq0aBYCkpSR9lHIj2rh9XgKAy4Fn1B3EALwPuIkyhPMmSRdIuqA6djxlQvv7aoyvn5Kv7ZV8bV++dsn/AduPVSBpBvB24CcDjSj6oUv52qU6fBXwn5IWGvlZHduxOmfoZHGcmklaitJbtAlwB3B1VfR8YCXKPmIz27T8cFdU+/ScRvn+Lq87nukkaQXKPIxteeyy0mcCh9tuRS9h8rW9kq/ty9cukbQFcA6l4+p44EfALOB2ytzdlwMb2/5ZbUHGYulKvnapDkt6B3As5Z7hcODKqmhd4APAhsA7bR9XT4TjS8NxCFQ3o++lLKqxdnX4GuBk4MgODJ1qLUnbAqcCF1OWWH5k1Cm2vfug44rFl3xtr+RrNJmkdwJHUoa4iQX7AT4I7GX7hJpCi5iULtXhauuR/ccp/oztAwcZz2Sl4RgxTSStT+k9W36C0xq3QXFEGyVfow2q/f3eTBkFIeBaytzdm2oNLGKSulSHJT2X8iT52dWh64Dv2r6mvqgmloZjxDSRdDFlWMnuwEVtGU4S0UbJ14iIqEs1mumNlGkvZ9n+a80hjSkNx4hpIuk+4FDbh9cdS0RMLPkaERGDIOlwYDPbr6zeCzgP2IjylPV24NW2h26V9uzjGDF9bqWMy4+I4Zd8jcaQNG8x/pltb973YCIWQ8fr8NbAD3vevwHYmLJQzhXAUcCBwB6DD21iaThGTJ/jgVmSjrb9cN3BRMSEkq/RJGuyYOGQEcuyYIP0kaHWK1avfwPuGUBcEZPV5Tr8DMrczRFvAK4fWRBH0rrA2+oIbFHScIyYPj8GXg9cLOlLwPUsvEojti8cdGARsZDkazSG7TV630takzLU7Ujg0yPzo6qFRg4EtgPa8KQmWqLjdXhpoLeDcjMe+wTyOuBpA41okjLHMWKaSJo/6tDoZBNZpTFiKCRfo8kknQncZ/ut45SfAsywvd1gI4uYnC7VYUlXAz+1vVv1dPHXwCzbc6vyA4D9ba9SZ5xjyRPHiOmza90BRMSkJV+jyTalbBI/nvOBTw0kkojFsyndqcOnAIdIeiqwLvAP4Ps95esBQ7cwDqThGDFtbM+pO4aImJzkazScgXUmKF93UIFELKYu1eHDKPMctwPuAnYe2QJK0gqUbTk+X19440vDMSIiIqLZzgX2knQZcJKreUjVMv87A3sCZ9QYX8SidKYO236Asmfw7mMU302Z33jfQIOapMxxjIiIiGgwSasDFwHPBG5hwYqNawOrAn8CNrT953oijJhY6nAzpOEYERER0XDVELcDgG0pWx1AWZ3xTODwkaFwEcMqdXj4peEYERERERERE1qi7gAiIiIiIiJiuKXhGBERERERERNKwzEiIiIiIiImlIZjRETEBCRZUhYEiIiITkvDMSIiIiIiIiaUhmNERERERERMKA3HiIiIPpG0naT/lXSNpHurn8skvUfSEqPO/UY1DHaTcX7XDlX50aOOryTpMElXSbpf0l2SfiRpyzF+x+zqd8yWtLWk86vzM/Q2IiKmJA3HiIiI/vkU8DLgEuAo4ERgOeBIYM6oc4+pXt85zu/as3r98sgBSc8CLgMOBG6ryr4JrAOcLWmPcX7Xm4DvAXf3/JuIiIhJk51Ox4iIiPGMPJ2zrUmc+xzbfxh1bAng68DOwKttX9JT9htgLeDptm/vOb4m8HvgZ7Zf03P8fGBjYCfbp/QcXxE4H3gesIbtW6rjs6v/t4FtbJ89lc8eERExIk8cIyIi+mR0o7E6Np/yxBFgq1HFxwBPBGaPOr4HIOArIwckvQTYBDitt9FY/T/uBD4CzAB2GCO0M9NojIiIx+MJdQcQERHRFpJWBj4AbAOsCSw76pSnj3p/ImV46zuBz1a/YylKQ/LvwKk9525Qva4g6dAx/verVK/rjFH2f5P6ABEREeNIwzEiIqIPquGiPweeTWmonQjcATwMrAjsS3m6+Cjbd0v6X+BdkjazfR7wRmA14Au2/9lz+srV68zqZzzLjXHsr1P/RBEREQuk4RgREdEf76A0Gj9q+9DeAkkbUBqOYzkGeBdlMZzzWLAozrGjzruret3X9henGFsWNIiIiMclcxwjIiL6Y63q9bQxysbccgPA9q+AnwDbS1of2AK40PZVo069uHrd6PEGGhERMVVpOEZERPTHDdXrpr0HJa0HfGgR//YYYGlKo1P0bMExwvalwEXAf0jabaxfIulFkp46pagjIiImIdtxRERETGBkOw4W3oex196UeYy/rl7PBK4F1gZeD3wH2BGYY3v2GP+PpYE/Uxa4+Ruwuu0HxjhvdWBe9Xt/Sdkv8k5gdeDFwAuBDWxfXJ0/m7Idx662T5j8p46IiHiszHGMiIiYnF0mKNvP9l8kbURZJXVDytYbV1MalT+kNBzHZPtBSScD+wEnjNVorM77s6SXA/9F2XbjbcCSlMVvrgSOojReIyIi+ipPHCMiIoaApPOBjYHn2b625nAiIiIeI3McIyIiaibpVZQFdM5JozEiIoZRhqpGRETURNJewNOBXYH5wEfqjSgiImJsGaoaERFRE0k3UBa2uQ441PbceiOKiIgYWxqOERERERERMaHMcYyIiIiIiIgJpeEYERERERERE0rDMSIiIiIiIiaUhmNERERERERMKA3HiIiIiIiImFAajhERERERETGh/w+0cRRgTWyG1AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "colors = ['gray' for l in S[:-1]]\n",
    "colors.append('r')\n",
    "ax = S.plot(kind='bar', linewidth=3, figsize=(15,10), color=colors, width=0.7, fontsize=18)\n",
    "ax.set_ylabel('L measure', fontsize=20)\n",
    "ax.set_xlabel('Layer', fontsize=20)\n",
    "print('Linearity measure calculated taking as output each layer of a convolutional neural network.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linearity measure in the locality of a given instance calculated taking as output each layer of a convolutional neural network trained on the fashion MNIST data set.\n",
    "* The linearity measure of the first convolutional layer conv_1 is 0, as expected since convolutions are linear operations.\n",
    "* The relu activation introduces non-linearity, which is increased by maxpooling. Dropout layers and flatten layers do no change the output at inference time so the linearity doesn't change.\n",
    "* The second convolutional layer conv_2 and the dense layers change the linearity even though they are linear operations. \n",
    "* The softmax layer in red is obtained by inverting the softmax function. \n",
    "* For more details see arxiv reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linearity and categories\n",
    "Here we calculate the linearity averaged over all instances belonging to the same class, for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_groups = []\n",
    "for i in range(10):\n",
    "    y = y_test.argmax(axis=1)\n",
    "    idxs_i = np.where(y == i)[0]\n",
    "    class_groups.append(x_test[idxs_i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating linearity for instances belonging to class 0\n",
      "Run time for class 0: 2.941605806350708\n",
      "Calculating linearity for instances belonging to class 1\n",
      "Run time for class 1: 3.3313376903533936\n",
      "Calculating linearity for instances belonging to class 2\n",
      "Run time for class 2: 3.178601026535034\n",
      "Calculating linearity for instances belonging to class 3\n",
      "Run time for class 3: 3.324582815170288\n",
      "Calculating linearity for instances belonging to class 4\n",
      "Run time for class 4: 3.085338830947876\n",
      "Calculating linearity for instances belonging to class 5\n",
      "Run time for class 5: 3.159513473510742\n",
      "Calculating linearity for instances belonging to class 6\n",
      "Run time for class 6: 3.4014275074005127\n",
      "Calculating linearity for instances belonging to class 7\n",
      "Run time for class 7: 3.3238165378570557\n",
      "Calculating linearity for instances belonging to class 8\n",
      "Run time for class 8: 2.9885218143463135\n",
      "Calculating linearity for instances belonging to class 9\n",
      "Run time for class 9: 3.4760279655456543\n",
      "Total run time: 32.22387504577637\n"
     ]
    }
   ],
   "source": [
    "def predict_fn(x):\n",
    "    return cnn.predict(x)\n",
    "lins_classes = []\n",
    "t_0 = time()\n",
    "for j in range(len(class_groups)):\n",
    "    print(f'Calculating linearity for instances belonging to class {j}')\n",
    "    class_group = class_groups[j]\n",
    "    class_group = np.random.permutation(class_group)[:2000]\n",
    "    t_i = time()\n",
    "    lin = linearity_measure(predict_fn, class_group, feature_range=features_range,\n",
    "                            agg='global', model_type='classifier', nb_samples=20)\n",
    "    t_i_1 = time() - t_i\n",
    "    print(f'Run time for class {j}: {t_i_1}')\n",
    "    lins_classes.append(lin)\n",
    "t_fin = time() - t_0\n",
    "print(f'Total run time: {t_fin}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(data=lins_classes).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linearity measure distribution means for each class in the fashion MNIST data set.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4EAAAJSCAYAAAB9d8D5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZTld1nn8c9DOoRNApoWQhaC7EQhSBM2kbiAgUGjLAqckUWYjCiKjmdGQCc9zRwVx4VhGzgZw+bCjhIVBBQcQCSkE0PIwhKQJTFCsyVEEEh85o97o0VZndRNbtWvq76v1zl1uu79fevWk3vSp/pdv626OwAAAIzhBlMPAAAAwOYRgQAAAAMRgQAAAAMRgQAAAAMRgQAAAAMRgQAAAAPZMfUAG+Gwww7rY445ZuoxAAAAJnHWWWd9rrt3rrVtW0bgMccck7179049BgAAwCSq6pP72+ZwUAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIGIQAAAgIFMGoFVdaOqen9VfaCqzq+qPWusOaSqXlNVF1XVGVV1zOZPCgAAsD1MvSfwa0m+v7vvkeS4JCdW1X1XrXlyki929x2SPDfJb27yjAAAANvGpBHYM1fMHx48/+hVy05K8or5569P8gNVVZs0IgAAwLYy9Z7AVNVBVXVOks8meXt3n7FqyRFJPp0k3X1lksuSfNvmTgkAALA9TB6B3X1Vdx+X5Mgkx1fVd16X16mqk6tqb1Xt3bdv33KHBAAA2CYmj8CrdfeXkrwzyYmrNl2S5KgkqaodSQ5N8vk1vv7U7t7V3bt27ty50eMCAABsSVNfHXRnVd1i/vmNkzw4yYdWLTs9yRPmnz8qyTu6e/V5gwAAAKzDjom//+FJXlFVB2UWpK/t7j+rqmcn2dvdpyc5LcnvV9VFSb6Q5DHTjQsAALC1TRqB3X1uknuu8fwpKz7/5ySP3sy5AAAAtqsD5pxAAAAANp4IBAAAGIgIBAAAGMjUF4YBroM9e/ZMPcKG271799QjAABsS/YEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADEQEAgAADGTSCKyqo6rqnVV1QVWdX1VPX2PNCVV1WVWdM/84ZYpZAQAAtoMdE3//K5P8UnefXVXfkuSsqnp7d1+wat27u/vhE8wHAACwrUy6J7C7L+3us+effznJhUmOmHImAACA7eyAOSewqo5Jcs8kZ6yx+X5V9YGqektVHbupgwEAAGwjUx8OmiSpqpsleUOSX+juy1dtPjvJbbv7iqp6WJI/SXLHNV7j5CQnJ8nRRx+9wRMDAABsTZPvCayqgzMLwD/s7jeu3t7dl3f3FfPP35zk4Ko6bI11p3b3ru7etXPnzg2fGwAAYCua+uqgleS0JBd29+/uZ82t5+tSVcdnNvPnN29KAACA7WPqw0EfkOQnk3ywqs6ZP/esJEcnSXe/JMmjkjy1qq5M8tUkj+nunmJYAACArW7SCOzu9ySpa1nzwiQv3JyJAAAAtrfJzwkEAABg84hAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgYhAAACAgeyYeoCtYs+ePVOPsOF279499QgAAMAGsycQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgIJNGYFUdVVXvrKoLqur8qnr6Gmuqqp5fVRdV1blV9d1TzAoAALAd7Jj4+1+Z5Je6++yq+pYkZ1XV27v7ghVrHprkjvOP+yR58fxPAAAAFjTpnsDuvrS7z55//uUkFyY5YtWyk5K8smfel+QWVXX4Jo8KAACwLRww5wRW1TFJ7pnkjFWbjkjy6RWPL86/D8VU1clVtbeq9u7bt2+jxgQAANjSDogIrKqbJXlDkl/o7suvy2t096ndvau7d+3cuXO5AwIAAGwTk0dgVR2cWQD+YXe/cY0llyQ5asXjI+fPAQAAsKCprw5aSU5LcmF3/+5+lp2e5PHzq4TeN8ll3X3ppg0JAACwjUx9ddAHJPnJJB+sqnPmzz0rydFJ0t0vSfLmJA9LclGSryR50gRzAgAAbAuTRmB3vydJXcuaTvKzmzMRAADA9jb5OYEAAABsHhEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwkB2LfkFV7UzyyCR3TXLT7n7Kiudvl+SD3f3VpU4JAADAUiwUgVX15CTPT3KjJJWkkzxlvvlWSf42yclJTlvijAAAACzJug8HraoHJzk1yUeS/FiSF6/c3t3nJTk/yY8uc0AAAACWZ5E9gb+c5NIkD+ruy6vqnmusOTfJ/ZYyGQAAAEu3yIVhdiX5s+6+/BrWXJzk1tdvJAAAADbKIhF4wyT/dC1rbpHkqus+DgAAABtpkQj8RJJ7Xcua+yT58HWeBgAAgA21SAS+KckDq+rRa22sqicluXuSNyxjMAAAAJZvkQvD/K8kj0nyqqp6VJJDk6SqnpbkgUkekeSjSV6w7CEBAABYjnVHYHd/saoelOSVSVbuDXz+/M93J3lcd1/beYMAAABMZKGbxXf3p5KcUFV3z+xWEN+W5LIk7+vuszZgPgAAAJZo3RFYVd+b5PLuPqe7z83snoAAAABsIYtcGOadSU7eqEEAAADYeItE4OeSfHWjBgEAAGDjLRKBf53k/hs0BwAAAJtgkQj81SR3rqr/WVUHb9RAAAAAbJxFrg76zCTnJXlWkidX1QeS/GOSXrWuu/vJS5oPAACAJVokAp+44vNbzz/W0klEIAAAwAFokQi83YZNAQAAwKZYdwR29yc3chAAAAA23iIXhgEAAGCLW/eewKo6er1ru/tT120cAAAANtIi5wR+Iv/+SqBr6QVfFwAAgE2ySKy9MmtH4C2SHJfktpndUN65gwAAAAeoRS4M88T9bauqGyT570l+OskTrv9YAAAAbISlXBimu/+lu/dkdsjoc5bxmgAAACzfsq8O+t4kD1nyawIAALAky47Ab01y0yW/JgAAAEuytAisqh9M8hNJzlvWawIAALBci9wn8B3X8BpHJbn6PoLPvr5DAQAAsDEWuUXECft5vpN8Mclbk/x2d+8vFgEAAJjYIreIWPb5gwAAANdoz549U4+wKXbv3r1p30vYAQAADOR6R2BVHVxV96yqOy9jIAAAADbOuiOwqn68ql5bVd+64rnbJzk/yd4kF1TVG6tqkfMMAQAA2ESL7An8qSR36e4vrHjud5LcIck7k5yb5KQkT1reeAAAACzTIhF4tyRnXv2gqm6e5GFJXtvdP5jk+CQfiggEAAA4YC0SgTuTXLri8f0yu7roq5Oku7+R5O1Jbr+06QAAAFiqRSLwy0kOXfH4QZndI/A9K5775yTfsoS5AAAA2ACLXMTlo0keWlWHZBZ/P57k3O7+3Io1t03y2SXOBwAAwBItsifw1CTfkVkMXpjkdkletmrNvTK7WigAAAAHoHVHYHe/Islzktwks8NCX5jkBVdvr6r759+uFAoAAMABaKF7+nX3s5I8az+b9ya5ZZJ/ur5DAQAAsDGWdmP37v56kq8v6/UAAABYvkXOCQQAAGCLWygCq+rwqnpRVV1UVV+tqqvW+Lhyo4YFAADg+ln34aBVdUSS9ye5VWZXAD0kySeTfC2zq4buSHJOksuWPyYAAADLsMiewFOS3DrJid19j/lzL+vuu2QWgW9NcuMkj1juiAAAACzLIhH4Q0n+orv/cvWG7r44yaMzi8A9S5oNAACAJVskAm+db74R/FWZRV+SpLuvSPL2JCctZzQAAACWbZEIvDzJDVc8/mKSI1atuSzJzus7FAAAABtjkQj8ZJKjVjz+QJLvr6qbJElV3SDJQ5JcvLzxAAAAWKZFIvCvknxfVR08f/yKJLdJ8t6q+q0kf5Pk2CSvWe6IAAAALMu6bxGR5LTMDgE9LMml3f0HVXWvJD+X5O7zNa9O8mvLHREAAIBlWXcEdvdHk/zmqud+sap+PbNbRHyiuz+z5PkAAABYokX2BK6pu/cl2beEWQAAANhgi5wT+K+q6i5V9WNV9ZPX55tX1Uur6rNVdd5+tp9QVZdV1Tnzj1Ouz/cDAAAY3UIRWFXHVdXezO4X+PokL1+x7UFV9ZWq+uEFXvLlSU68ljXv7u7j5h/PXmReAAAAvtm6I7Cq7pTkr5PcOcnzkrxl1ZJ3JflCkket9zW7++qvAQAAYBMssidwd2Y3i79Pd/+XJGeu3NjdneRvk9x7eeMlSe5XVR+oqrdU1bFLfm0AAIChLBKBP5Dkjd19wTWs+XRm9w5clrOT3La775HkBUn+ZH8Lq+rkqtpbVXv37XOdGgAAgLUsEoG3THLxtaypzPYWLkV3X97dV8w/f3OSg6vqsP2sPbW7d3X3rp07dy5rBAAAgG1lkVtEfCbJHa5lzbGZ7Q1ciqq6dZLPdHdX1fGZRevnl/X6AACwUfbs2TP1CJti9+7dU4/AghaJwHckeWxV3bm7P7x6Y1XdO7NDRl+03hesqlclOSHJYVV1cWbnHR6cJN39kswuMvPUqroyyVeTPGZ+7iEAAADXwSIR+BtJHp3kXVX1PzI/929+sZbvzSzgvpzkt9f7gt392GvZ/sIkL1xgRgAAAK7BuiOwuz9cVY9M8qr8W5hVknPnf34pySO6+1NLnxIAAIClWGRPYLr7L6rqdkmekOS+Sb4tyWVJ3pfkZd3tnn8AcB2McO6Q84YADgwLRWCSdPeXMrtZ/POWPw4AAAAbaZFbRAAAALDFLbwncH6fvrsmOTLzK3mu1t2vvJ5zAQAAsAHWHYFVdaMkv5Pkp7L/G8JXkk4iAgEAAA5Ai+wJ/K0kT01yYZLXJLkkyZUbMRQAAAAbY5EI/PHMbgdx7+7+xgbNAwAAwAZa5MIwN03ydgEIAACwdS0SgecnOXyjBgEAAGDjLRKBv53kx6rqThs1DAAAABtr3ecEdvfrqurwJO+uqv+T5Owkl+1n7buWNB8AAABLtOh9Am+Z2bmBp1zLuoOu2zgAAABspEXuE/jMJLuTfD6zW0T8Q9wiAgAAYEtZZE/gyUk+nuRe3b3mYaAAAAAc2Ba5MMytk5wuAAEAALauRSLw40lusVGDAAAAsPEWicAXJ/nhqrr1Rg0DAADAxlrknMA/TXJCkvdW1bOTnJX93yLiU9d/NAAAAJZtkQj8+ySdpJKcdg3resHXBQAAYJMsEmuvzCzwAAAA2KLWHYHd/cQNnAMAAIBNsMiFYQAAANjiRCAAAMBARCAAAMBARCAAAMBARCAAAMBARCAAAMBAlhqBVXWjqrr5Ml8TAACA5Vn2nsAXJ/nCkl8TAACAJdmIw0FrA14TAACAJXBOIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEB2XNPGqrpqswYBAABg411jBOa63e6hr8sgAAAAbLxrjMDudrgoAADANiLyAAAABiICAQAABiICAQAABiICAQAABiICAQAABiICAQAABiICAQAABnJtN4sHAGAwe/bsmXqETbF79+6pR4BJ2BMIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwEBEIAAAwkB1TD8BY9uzZM/UIG2737t1TjwAAAPtlTyAAAMBARCAAAMBARCAAAMBARCAAAMBARCAAAMBAJo3AqnppVX22qs7bz/aqqudX1UVVdW5VffdmzwgAALCdTL0n8OVJTryG7Q9Ncsf5x8lJXrwJMwEAAGxbk0Zgd78ryReuYclJSV7ZM+9LcouqOnxzpgMAANh+DvSbxR+R5NMrHl88f+7S1Qur6uTM9hbm6KOP3pThAEj27Nkz9Qgbbvfu3VOPAABLM/XhoEvT3ad2967u3rVz586pxwEAADggHegReEmSo1Y8PnL+HAAAANfBgR6Bpyd5/PwqofdNcll3/7tDQQEAAFifSc8JrKpXJTkhyWFVdXGS3UkOTpLufkmSNyd5WJKLknwlyZOmmRQAAGB7mDQCu/ux17K9k/zsJo0DAACw7R3oh4MCAACwRCIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgICIQAABgIDumHgAAYFn27Nkz9QibYvfu3VOPAGxh9gQCAAAMRAQCAAAMRAQCAAAMRAQCAAAMRAQCAAAMRAQCAAAMxC0igGGNcCl5l5EHAFazJxAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgIhAAAGAgk0dgVZ1YVR+uqouq6hlrbH9iVe2rqnPmH0+ZYk4AAIDtYMeU37yqDkryoiQPTnJxkjOr6vTuvmDV0td099M2fUAAAIBtZuo9gccnuai7P97dX0/y6iQnTTwTAADAtjV1BB6R5NMrHl88f261R1bVuVX1+qo6anNGAwAA2H6mjsD1+NMkx3T33ZO8Pckr1lpUVSdX1d6q2rtv375NHRAAAGCrmDoCL0mycs/ekfPn/lV3f767vzZ/+HtJ7rXWC3X3qd29q7t37dy5c0OGBQAA2OqmjsAzk9yxqm5XVTdM8pgkp69cUFWHr3j4I0ku3MT5AAAAtpVJrw7a3VdW1dOSvDXJQUle2t3nV9Wzk+zt7tOT/HxV/UiSK5N8IckTJxsYAABgi5s0ApOku9+c5M2rnjtlxefPTPLMzZ4LAABgO5r6cFAAAAA2kQgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYiAgEAAAYyOQRWFUnVtWHq+qiqnrGGtsPqarXzLefUVXHbP6UAAAA28OkEVhVByV5UZKHJrlbksdW1d1WLXtyki929x2SPDfJb27ulAAAANvH1HsCj09yUXd/vLu/nuTVSU5ateakJK+Yf/76JD9QVbWJMwIAAGwb1d3TffOqRyU5sbufMn/8k0nu091PW7HmvPmai+ePPzZf87lVr3VykpPnD++c5MOb8J+wkQ5L8rlrXcW18T4uj/dyObyPy+F9XA7v43J4H5fD+7gc3sfl2A7v4227e+daG3Zs9iQbpbtPTXLq1HMsS1Xt7e5dU8+x1Xkfl8d7uRzex+XwPi6H93E5vI/L4X1cDu/jcmz393Hqw0EvSXLUisdHzp9bc01V7UhyaJLPb8p0AAAA28zUEXhmkjtW1e2q6oZJHpPk9FVrTk/yhPnnj0ryjp7yGFYAAIAtbNLDQbv7yqp6WpK3JjkoyUu7+/yqenaSvd19epLTkvx+VV2U5AuZheIIts2hrRPzPi6P93I5vI/L4X1cDu/jcngfl8P7uBzex+XY1u/jpBeGAQAAYHNNfTgoAAAAm0gEAgAADEQEAgAADGTb3Cdwq6uquyQ5KckR86cuSXJ6d1843VSMav7/4xFJzujuK1Y8f2J3/8V0k20tVXV8ku7uM6vqbklOTPKh7n7zxKNtaVX1yu5+/NRzbHVV9T1Jjk9yXne/bep5toqquk+SC7v78qq6cZJnJPnuJBck+fXuvmzSAbeIqvr5JH/c3Z+eepatbMXV9f+hu/+yqh6X5P5JLkxyand/Y9IBt5Cq+o4kj8js1nRXJflIkj/q7ssnHWyDuDDMAaCqfjnJY5O8OsnF86ePzOwv9au7+zlTzbadVNWTuvtlU89xoJv/YP7ZzH6AHJfk6d39pvm2s7v7u6ecb6uoqt1JHprZL9venuQ+Sd6Z5MFJ3trdvzbheFtGVa2+bVAl+b4k70iS7v6RTR9qi6qq93f38fPP/1Nmf8//OMlDkvypnzXrU1XnJ7nH/Arnpyb5SpLXJ/mB+fOPmHTALaKqLkvyT0k+luRVSV7X3fumnWrrqao/zOznzE2SfCnJzZK8MbP/H6u7n3ANX87c/N8+D0/yriQPS/J3mb2fP5bkZ7r7r6ebbmOIwANAVX0kybGrf1sz/+3O+d19x2km216q6lPdffTUcxzoquqDSe7X3VdU1TGZ/ePm97v7eVX1d919z0kH3CLm7+NxSQ5J8o9Jjlyx5+CM7r77pANuEVV1dmZ7WH4vSWcWga/K/HZB3f3/pptua1n597eqzkzysO7eV1U3TfK+7v6uaSfcGqrqwu6+6/zzb/rFWFWd093HTTfd1lFVf5fkXkl+MMlPJPmRJGdl9vf7jd395QnH2zKq6tzuvntV7cjsKLLbdPdVVVVJPuBnzfpc/TN7/t7dJMmbu/uEqjo6yZu24799HA56YPiXJLdJ8slVzx8+38Y6VdW5+9uU5FabOcsWdoOrDwHt7k9U1QlJXl9Vt83sfWR9ruzuq5J8pao+dvXhJN391ary93r9diV5epJfSfJfu/ucqvqq+LtOblBVt8zsegB19V6X7v6nqrpy2tG2lPNWHFnygara1d17q+pOSRx6t37d3f+S5G1J3lZVB2d29MRjk/x2kp1TDreF3GC+0+Cmme0NPDSz+2ofkuTgKQfbgnZkdhjoIZntUU13f2r+/+a2IwIPDL+Q5K+q6qNJrj42/ugkd0jytMmm2ppuleSHknxx1fOV5L2bP86W9JmqOq67z0mS+R7Bhyd5aRJ7Ctbv61V1k+7+Sma/7U6SVNWh8cuddZv/I/G5VfW6+Z+fiZ9d19Whme1pqSRdVYd396VVdbP4Bc8inpLkeVX1q0k+l+Rvq+rTmf38fsqkk20t3/T/3PxoqNOTnD7fE8P6nJbkQ0kOyuyXZa+rqo8nuW9mpxmxPr+X5MyqOiPJA5P8ZpJU1c7MonrbcTjoAaKqbpDZCforLwxz5nxPAutUVacleWrIFdMAAAUASURBVFl3v2eNbX/U3Y+bYKwtpaqOzGwv1j+use0B3f03E4y15VTVId39tTWePyzJ4d39wQnG2vKq6j8keUB3P2vqWbaL+T+4b9Xdfz/1LFtJVd08ye0y+6XExd39mYlH2lKq6k7d/ZGp59gOquo2SdLd/1BVt8jsENtPdff7p51sa6mqY5PcNbOLZX1o6nk2mggEAAAYiPsEAgAADEQEAgAADEQEAsA6VNUxVdVV9fKpZwGA60MEAjC0qrpLVb2gqs6rqsuq6utV9Q9V9edV9eSqOmTqGQFgmVxmG4BhVdUpSXZn9kvRv03yiiRXZHa7mRMyu2z4UzO7XyEAbAsiEIAhVdWzkuzJ7P5uj+7uM9ZY8/Akv7TZswHARnI4KADDqapjkvyPJN9I8rC1AjBJuvvPkpx4La91p6p6TlXtrap9VfW1qvpkVZ06v+/m6vVVVU+oqvfO1/9zVX26qt5aVT+xau3dq+pVVfWJ+evuq6qzq+p/V9XB1/E/H4DB2RMIwIielOTgJK/u7vOuaWF3f+1aXusRSX46yTuTvDfJ15Mcm+QpSX64qnZ19yUr1v9akmcm+fskr01yWZLDk9w7yaOTvCaZBWCSM5J0ktPn62+e5A5JfibJr2YWsQCwEBEIwIi+Z/7nXy3htX4/yXNXx2JVPSTJWzKLtaeu2PSfk1yS5Du7+yurvuawFQ+fkORGSX60u9+0at0tk3zT1wLAeolAAEZ0+PzPi6/vC63ay7fy+bdV1flJfmiNzd9IctUaX/O5NdZ+dY11X1x0TgC4mnMCAeB6mJ/j9x+r6i/n5+xdOb+fYCf5riRHrPqSP0xyTJILquo3qurEqjp0jZd+TWah+CdV9cqqenxV3X5D/2MAGIIIBGBEl87/XB1o18XvZnZI6N2SvDXJ72R21dE9ST6Z5Iar1v/i/OOKJM/I7JDRz1XVm6rqDlcv6u73J3lgknckeVRmt6+4qKo+VFWPXcLcAAyqunvqGQBgU1XVniSnJHlVdz9unV9zTGYXZ3lFdz9x/ty3ZxaUFyS5f3d/edXXfDjJnbq79vOa357Z+YmPyeyiMB9Lcuwa5xcekuRemV2p9OeS3CLJg7v7L9czOwCsZE8gACN6WWbn5T2yqu52TQvnAbY/35HZz9K3rRGAR86371d3f7a739jdP57ZHr/bJ/nONdZ9rbvf292nJPn5+dMnXdNrA8D+iEAAhtPdn8jsPoE3TPLnVbVrrXVVdWJmh2vuzyfmf35PVR204utuluT/ZtUF2KrqkKp6wBrf5+Ak3zp/+JX5c/evqhuv8T1vtXIdACzK1UEBGFJ3/3pV7UiyO8mZVfXeJHszO1fvVkm+N8kd58/t7zX+sapendnhnOdU1duSHJrkwUn+Ock5SY5b8SU3TvKeqrooyVmZnTN4o/n6uyY5vbsvnK/9b0m+v6rendlhqFdkdv/Bhyb5YpJTr/ebAMCQnBMIwNCq6q6Z3Xz9+5IcnVmUfT6zgHt9kj/o7q+tdU7g/OtvkuRXkvxEkiOT7Mvs5u6nJHlDkgddfU7gfI/fL86/17FJvj3JlzM7F/DlSV7a3V+fr31IkscmuU9mF7DZkdktLd6a5He6+5Mb8HYAMAARCAAAMBDnBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAxEBAIAAAzk/wO5eloVqCFtfQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = df.mean().plot(kind='bar', linewidth=3, figsize=(15,10), color='gray', width=0.7, fontsize=10)\n",
    "ax.set_ylabel('L measure', fontsize=20)\n",
    "ax.set_xlabel('Class', fontsize=20)\n",
    "print(\"Linearity measure distribution means for each class in the fashion MNIST data set.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linearity measure distributions for each class in the fashion MNIST data set.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAJVCAYAAAAcMbuxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdf5Sc1X3n+fc3okkloBhoCaJ0Caqd4BhsJ4AVy1k8hNgHB4g3in+EYDs2YIh2cmAWNs5ZNPwTkz2TaHImxCbheCMbbBwzVjiBWbQE/yAYxsZZDC2MDUgh1mKIShEg1cT8dBtJ+e4f/cCWhZquUt/qqq56v87pU1X3qec+t6+x9NG997lPZCaSJEmavx/rdwMkSZKGhcFKkiSpEIOVJElSIQYrSZKkQgxWkiRJhRzS7wYALFu2LBuNRr+bIUmSNKfNmzfvzszlBzo2EMGq0WgwNTXV72ZIkiTNKSIen+2YU4GSJEmFGKwkSZIKMVhJkiQVMhBrrCRJ0mjZs2cPzWaT6enpfjdlVrVajXq9ztjYWMfnGKwkSdKCazabLF26lEajQUT0uzmvkJm0Wi2azSaTk5Mdn+dUoCRJWnDT09OMj48PZKgCiAjGx8e7HlEzWEmSpL4Y1FD1koNpn8FKkiSpENdYSZKkvmus+7ui9T22/tfn/M6XvvQlLr30Uvbt28dFF13EunXr5n1dR6wkSdLI2bdvHxdffDFf/OIX2bJlC1/4whfYsmXLvOs1WEmSpJFz77338nM/93O89rWv5dBDD+Xcc8/llltumXe9BitJkjRyduzYwcqVK1/+XK/X2bFjx7zrNVhJkiQVYrCSJEkjZ2Jigu3bt7/8udlsMjExMe965wxWEbEyIu6MiC0R8XBEXFqVHxURt0fEd6vXI6vyiIirI2JbRHwnIk6ZdyslSZIK+qVf+iW++93v8r3vfY8XX3yRjRs38hu/8RvzrreT7Rb2Ah/NzPsjYimwOSJuB84H7sjM9RGxDlgHXA6cBRxf/awGPlm9SpIkHVAn2yOUdMghh/CXf/mX/Nqv/Rr79u3jIx/5CG94wxvmX+9cX8jMncDO6v2zEbEVmADWAKdXX7seuIuZYLUG+FxmJnBPRBwRESuqeiRJkgbC2Wefzdlnn120zq7WWEVEAzgZ+CZwTFtYegI4pno/AWxvO61Zle1f19qImIqIqV27dnXZbEmSpMHTcbCKiMOBm4DLMvOZ9mPV6FR2c+HM3JCZqzJz1fLly7s5VZIkaSB1FKwiYoyZUHVDZt5cFT8ZESuq4yuAp6ryHcDKttPrVZkkSdLLZsZlBtfBtK+TuwIDuBbYmplXtR3aBJxXvT8PuKWt/MPV3YFvBZ52fZUkSWpXq9VotVoDG64yk1arRa1W6+q8Tu4KPBX4EPBgRDxQlV0BrAdujIgLgceBc6pjtwFnA9uAF4ALumqRJEkaevV6nWazySCvs67VatTr9a7O6eSuwLuBmOXwOw7w/QQu7qoVkiRppIyNjTE5OdnvZhTnzuuSJEmFGKwkSZIKMVhJkiQVYrCSJEkqxGAlSZJUiMFKkiSpEIOVJElSIZ1sECppQDTW/V2/m1DMY+t/vd9NkKTiDFaS+mIhQqLhTdJCcypQkiSpEEesCvNf4YNnoabP/N9FkmSwWoSGaZ2NJEnDxGAlFWLglSQZrCQNLafmJS00g5UkzYNr+CS1865ASZKkQkZmxMr1L5IWM6c1pcXBEStJkqRCehKsIuLMiHgkIrZFxLpeXEOSJGnQFJ8KjIglwDXAGUATuC8iNmXmltLXkiSV40J8af56scbqLcC2zHwUICI2AmsAg5UkyfViGmq9CFYTwPa2z01g9f5fioi1wNrq43MR8Uih6y8Ddheqa9TZl2XYj+XYl+UMdV/Gf16wSw11Py6wxdSXx812oG93BWbmBmBD6XojYiozV5WudxTZl2XYj+XYl+XYl2XYj+UMS1/2YvH6DmBl2+d6VSZJkjTUehGs7gOOj4jJiDgUOBfY1IPrSJIkDZTiU4GZuTciLgG+DCwBrsvMh0tf51UUn14cYfZlGfZjOfZlOfZlGfZjOUPRl5GZ/W6DJEnSUHDndUmSpEIMVpIkSYUYrCRJkgrp2z5W7ZYtW5aNRqPfzZAkSZrT5s2bd2fm8gMdG4hg1Wg0mJqa6nczJEmS5hQRj892zKlASZKkQgZixEoD6mOvWYBrPN37a0iStEAcsZIkSSrEEStJkrTg9uzZQ7PZZHp6ut9NmVWtVqNerzM2NtbxOQYrSZK04JrNJkuXLqXRaBAR/W7OK2QmrVaLZrPJ5ORkx+c5FShJkhbc9PQ04+PjAxmqACKC8fHxrkfUHLFajBZiUbkkST02qKHqJQfTPkesJEmSCnHESpIk9V/p2ZgOtvP5yEc+wq233srRRx/NQw89VOSyjlhJkqSRdP755/OlL32paJ0GK0mSNJJOO+00jjrqqKJ1GqwkSZIKMVhJkiQVYrCSJEkqZM5gFRErI+LOiNgSEQ9HxKVV+VERcXtEfLd6PbIqj4i4OiK2RcR3IuKUXv8SkiRJg6CT7Rb2Ah/NzPsjYimwOSJuB84H7sjM9RGxDlgHXA6cBRxf/awGPlm9SpIkHVgH2yOU9v73v5+77rqL3bt3U6/XufLKK7nwwgvnVeecwSozdwI7q/fPRsRWYAJYA5xefe164C5mgtUa4HOZmcA9EXFERKyo6pEkSRoIX/jCF4rX2dUaq4hoACcD3wSOaQtLTwDHVO8ngO1tpzWrsv3rWhsRUxExtWvXri6bLUmSNHg6DlYRcThwE3BZZj7TfqwancpuLpyZGzJzVWauWr58eTenSpIkDaSOglVEjDETqm7IzJur4icjYkV1fAXwVFW+A1jZdnq9KpMkSXrZzLjM4DqY9nVyV2AA1wJbM/OqtkObgPOq9+cBt7SVf7i6O/CtwNOur5IkSe1qtRqtVmtgw1Vm0mq1qNVqXZ3XyV2BpwIfAh6MiAeqsiuA9cCNEXEh8DhwTnXsNuBsYBvwAnBBVy2SJElDr16v02w2GeR11rVajXq93tU5ndwVeDcQsxx+xwG+n8DFXbVCo6v008wPeI2Fv4VXkvTqxsbGmJyc7HczinPndUmSpEIMVpIkSYUYrCRJkgrpZPG6urEQa4YkSdJAcsRKkiSpEIOVJElSIQYrSZKkQgxWkiRJhRisJEmSCjFYSZIkFWKwkiRJKsRgJUmSVIjBSpIkqRCDlSRJUiE+0kbDb6EeM/SxpxfmOpKkgeWIlSRJUiEGK0mSpEIMVpIkSYWMzhqrhVpnI0mSRlZPglVEnAl8AlgCfDoz1/fiOtJAWYjw7gJ5SRpoxacCI2IJcA1wFnAi8P6IOLH0dSRJkgZNL0as3gJsy8xHASJiI7AG2NKDa0mjxVExSRpovQhWE8D2ts9NYPX+X4qItcDa6uNzEfFIoesvA3YXqmvU2ZdlLK5+vDL63YJXs7j6crDZl2XYj+Uspr48brYDfVu8npkbgA2l642IqcxcVbreUWRflmE/lmNflmNflmE/ljMsfdmL7RZ2ACvbPterMkmSpKHWi2B1H3B8RExGxKHAucCmHlxHkiRpoBSfCszMvRFxCfBlZrZbuC4zHy59nVdRfHpxhNmXZdiP5diX5diXZdiP5QxFX0Zm9rsNkiRJQ8FH2kiSJBVisJIkSSrEYCVJklTIQDyEedmyZdloNPrdDEmSpDlt3rx5d2YuP9CxgQhWjUaDqampfjdDkiRpThHx+GzHnAqUJEkqxGAlSZJUiMFKkiSpkIFYYyVJkkbLnj17aDabTE9P97sps6rVatTrdcbGxjo+x2AlSZIWXLPZZOnSpTQaDSKi3815hcyk1WrRbDaZnJzs+DynAiVJ0oKbnp5mfHx8IEMVQEQwPj7e9YiawUqSJPXFoIaqlxxM+wxWkiRJhbjGSpIk9d2brn9T0foePO/BVz2+fft2PvzhD/Pkk08SEaxdu5ZLL7103tc1WEmSpJFzyCGH8Gd/9meccsopPPvss7z5zW/mjDPO4MQTT5xXvU4FSpKkkbNixQpOOeUUAJYuXcoJJ5zAjh075l2vwUqSJI20xx57jG9961usXr163nUZrCRJ0sh67rnneO9738vHP/5xfuqnfmre9c0ZrCJiZUTcGRFbIuLhiLi0Kj8qIm6PiO9Wr0dW5RERV0fEtoj4TkScMu9WSpIkFbZnzx7e+9738sEPfpD3vOc9RersZMRqL/DRzDwReCtwcUScCKwD7sjM44E7qs8AZwHHVz9rgU8WaakkSVIhmcmFF17ICSecwO///u8Xq3fOuwIzcyews3r/bERsBSaANcDp1deuB+4CLq/KP5eZCdwTEUdExIqqHkmSpFeYa3uE0r7xjW/w13/917zpTW/ipJNOAuCP//iPOfvss+dVb1fbLUREAzgZ+CZwTFtYegI4pno/AWxvO61Zlf1IsIqItcyMaHHsscd22WxJkqSD97a3vY2ZMaCyOl68HhGHAzcBl2XmM+3HqtGprlqXmRsyc1Vmrlq+fHk3p0qSJA2kjoJVRIwxE6puyMybq+InI2JFdXwF8FRVvgNY2XZ6vSqTJEkaap3cFRjAtcDWzLyq7dAm4Lzq/XnALW3lH67uDnwr8LTrqyRJ0v56MRVX0sG0r5M1VqcCHwIejIgHqrIrgPXAjRFxIfA4cE517DbgbGAb8AJwQdetkiRJQ61Wq9FqtRgfH2dmDGewZCatVotardbVeZ3cFXg3MNtv/I4DfD+Bi7tqhSRJGin1ep1ms8muXbv63ZRZ1Wo16vV6V+f4EGZJkrTgxsbGmJyc7HczivORNpIkSYUYrCRJkgoxWEmSJBVisJIkSSrEYCVJklSIwUqSJKkQg5UkSVIhBitJkqRCDFaSJEmFGKwkSZIK8ZE20iLypuvf1PNrPHjegz2/hiQNK4OVJAlYmOAOhncNN4OVZrVQf8gOC/+y6I7/fUkaRgYrST/CwKNh4LS5+sVgJRViIJE64/9XNMwMVouQfyhJUv+5Jk0HYrAqzNAjSdLoch8rSZKkQnoyYhURZwKfAJYAn87M9b24jiRJw86F+ItL8WAVEUuAa4AzgCZwX0Rsyswtpa/VDafoJEk6MMNbOb0YsXoLsC0zHwWIiI3AGqCvwUqSJPXPqCz270WwmgC2t31uAqv3/1JErAXWVh+fi4hHCl1/GbC7UF2jzr4sw34sx74sx74sw34sp0hfxvlRoClzOm62A327KzAzNwAbStcbEVOZuap0vaPIvizDfizHvizHvizDfixnWPqyF3cF7gBWtn2uV2WSJElDrRfB6j7g+IiYjIhDgXOBTT24jiRJ0kApPhWYmXsj4hLgy8xst3BdZj5c+jqvovj04gizL8uwH8uxL8uxL8uwH8sZir6MzOx3GyRJkoaCO69LkiQVYrCSJEkqxGAlSZJUSN/2sWq3bNmybDQa/W6GJEnSnDZv3rw7M5cf6NhABKtGo8HU1FS/myFJkjSniHh8tmNOBUqSJBUyECNWw2Tr60/o+TVO+MetPb+GJEnqniNWkiRJhThiJUmSFtyePXtoNptMT0/3uymzqtVq1Ot1xsbGOj7HYCVJkhZcs9lk6dKlNBoNIqLfzXmFzKTVatFsNpmcnOz4PKcCJUnSgpuenmZ8fHwgQxVARDA+Pt71iJrBSpIk9cWghqqXHEz7DFaSJEmFuMZKkiT1XentijrZmmh6eprTTjuNH/7wh+zdu5f3ve99XHnllfO6rsFKkiSNpB//8R/nq1/9Kocffjh79uzhbW97G2eddRZvfetbD7pOpwIlSdJIiggOP/xwYGb7hz179sx73ZfBSpIkjax9+/Zx0kkncfTRR3PGGWewevXqedVnsJIkSSNryZIlPPDAAzSbTe69914eeuihedVnsJIkSSPviCOO4Fd/9Vf50pe+NK965gxWEbEyIu6MiC0R8XBEXFqVHxURt0fEd6vXI6vyiIirI2JbRHwnIk6ZVwslSZJ6YNeuXXz/+98H4Ac/+AG33347r3/96+dVZyd3Be4FPpqZ90fEUmBzRNwOnA/ckZnrI2IdsA64HDgLOL76WQ18snqVJEk6oE62Ryht586dnHfeeezbt49/+7d/45xzzuFd73rXvOqcM1hl5k5gZ/X+2YjYCkwAa4DTq69dD9zFTLBaA3wuMxO4JyKOiIgVVT2SJEkD4Rd+4Rf41re+VbTOrtZYRUQDOBn4JnBMW1h6Ajimej8BbG87rVmV7V/X2oiYioipXbt2ddlsSZKkwdPxBqERcThwE3BZZj7Tvs9DZmZEZDcXzswNwAaAVatWdXXuwSi9o6skSdL+OhqxiogxZkLVDZl5c1X8ZESsqI6vAJ6qyncAK9tOr1dlkiRJL5tZNTS4DqZ9ndwVGMC1wNbMvKrt0CbgvOr9ecAtbeUfru4OfCvwtOurJElSu1qtRqvVGthwlZm0Wi1qtVpX53UyFXgq8CHgwYh4oCq7AlgP3BgRFwKPA+dUx24Dzga2AS8AF3TVIkmSNPTq9TrNZpNBXmddq9Wo1+tdndPJXYF3A7M9OOcdB/h+Ahd31Qp1ZaHWi/Xj1ldJ0mgYGxtjcnKy380ozp3XJUmSCjFYSZIkFWKwkiRJKsRgJUmSVIjBSpIkqRCDlSRJUiEdP9JG6oWF2DrCbSMkSQvFEStJkqRCHLHSrHxwtSRJ3XHESpIkqRCDlSRJUiEGK0mSpEIMVpIkSYW4eF1Db6EW4butgyTJEStJkqRCDFaSJEmFOBUoFeIu8pIkR6wkSZIK6UmwiogzI+KRiNgWEet6cQ1JkqRBU3wqMCKWANcAZwBN4L6I2JSZW0pfSxo1TjdK0mDrxRqrtwDbMvNRgIjYCKwBDFbSIuD2FJJ08HoRrCaA7W2fm8Dq/b8UEWuBtdXH5yLikULXXwbsLlTXqLMvy7AfDyTiYM6yL8uxL8uwH8tZTH153GwH+nZXYGZuADaUrjcipjJzVel6R5F9WYb9WI59WY59WYb9WM6w9GUvFq/vAFa2fa5XZZIkSUOtF8HqPuD4iJiMiEOBc4FNPbiOJEnSQCk+FZiZeyPiEuDLwBLgusx8uPR1XkXx6cURZl+WYT+WY1+WY1+WYT+WMxR9GZnZ7zZIkiQNBXdelyRJKsRgJUmSVIjBSpIkqZC+7WPVbtmyZdloNPrdDEmSpDlt3rx5d2YuP9CxgQhWjUaDqampfjdDkiRpThHx+GzHnAqUJEkqZCBGrNSda/79VxfkOhf/n29fkOtIkjQsHLGSJEkqxBErSZK04Pbs2UOz2WR6errfTZlVrVajXq8zNjbW8TkGK0mStOCazSZLly6l0WgQEf1uzitkJq1Wi2azyeTkZMfnORUoSZIW3PT0NOPj4wMZqgAigvHx8a5H1AxWkiSpLwY1VL3kYNpnsJIkSSrENVaSJKnvSm8l1OmWQfv27WPVqlVMTExw6623zvu6BivNaiH2y3KvLElSP33iE5/ghBNO4JlnnilSn1OBkiRpJDWbTf7u7/6Oiy66qFidBitJkjSSLrvsMv70T/+UH/uxcnHIYCVJkkbOrbfeytFHH82b3/zmovUarCRJ0sj5xje+waZNm2g0Gpx77rl89atf5Xd+53fmXe+cwSoiVkbEnRGxJSIejohLq/KjIuL2iPhu9XpkVR4RcXVEbIuI70TEKfNupSRJUkF/8id/QrPZ5LHHHmPjxo28/e1v5/Of//y86+3krsC9wEcz8/6IWApsjojbgfOBOzJzfUSsA9YBlwNnAcdXP6uBT1avkiRJBzQsd4nPOWKVmTsz8/7q/bPAVmACWANcX33teuA3q/drgM/ljHuAIyJiRfGWS5IkFXD66acX2cMKulxjFREN4GTgm8AxmbmzOvQEcEz1fgLY3nZasyrbv661ETEVEVO7du3qstmSJEmDp+NgFRGHAzcBl2Xmj+yilZkJZDcXzswNmbkqM1ctX768m1MlSZIGUkfBKiLGmAlVN2TmzVXxky9N8VWvT1XlO4CVbafXqzJJkqSXzYzLDK6Dad+ci9dj5tHO1wJbM/OqtkObgPOA9dXrLW3ll0TERmYWrT/dNmU49BbiMTCSJC12tVqNVqvF+Pg4M1FjsGQmrVaLWq3W1Xmd3BV4KvAh4MGIeKAqu4KZQHVjRFwIPA6cUx27DTgb2Aa8AFzQVYskSdLQq9frNJtNBnmdda1Wo16vd3XOnMEqM+8GZouS7zjA9xO4uKtWSJKkkTI2Nsbk5GS/m1GcO69LkiQVYrCSJEkqxGAlSZJUiMFKkiSpEIOVJElSIQYrSZKkQjrZx0rqmYXYUHVYnpguSRp8jlhJkiQVYrCSJEkqxGAlSZJUiMFKkiSpEIOVJElSIQYrSZKkQgxWkiRJhbiPlYbeQuyVBe6XJUkaoWC1UH+5SpKk0eVUoCRJUiEGK0mSpEJ6Eqwi4syIeCQitkXEul5cQ5IkadAUX2MVEUuAa4AzgCZwX0Rsyswtpa8lDRIfKC1J6sWI1VuAbZn5aGa+CGwE1vTgOpIkSQOlF3cFTgDb2z43gdX7fyki1gJrq4/PRcQjha6/DNhdqK5RZ1+WUawfL/mrErUsav43WY59WYb9WM5i6svjZjvQt+0WMnMDsKF0vRExlZmrStc7iuzLMuzHcuzLcuzLMuzHcoalL3sxFbgDWNn2uV6VSZIkDbVeBKv7gOMjYjIiDgXOBTb14DqSJEkDpfhUYGbujYhLgC8DS4DrMvPh0td5FcWnF0eYfVmG/ViOfVmOfVmG/VjOUPRlZGa/2yBJkjQU3HldkiSpEIOVJElSIQYrSZKkQgxWkiRJhfRtg9B2y5Yty0aj0e9mSJIkzWnz5s27M3P5gY4NRLBqNBpMTU31uxmSJElziojHZzvmVKAkSVIhBitJkqRCDFaSJEmFDMQaK0mSNFr27NlDs9lkenq6302ZVa1Wo16vMzY21vE5BitJkrTgms0mS5cupdFoEBH9bs4rZCatVotms8nk5GTH5zkVKEmSFtz09DTj4+MDGaoAIoLx8fGuR9QMVpIkqS8GNVS95GDaZ7CSJEkqxDVWkiSp7/7st99VtL6P/s2tHX2v0WiwdOlSlixZwiGHHDLvDcsNVpIkaaTdeeedLFu2rEhdTgVKkiQVYrCSJEkjKyJ45zvfyZvf/GY2bNgw7/qcCpQkSSPr7rvvZmJigqeeeoozzjiD17/+9Zx22mkHXd+cI1YRsTIi7oyILRHxcERcWpUfFRG3R8R3q9cjq/KIiKsjYltEfCciTjno1kmSJPXQxMQEAEcffTTvfve7uffee+dVXydTgXuBj2bmicBbgYsj4kRgHXBHZh4P3FF9BjgLOL76WQt8cl4tlCRJ6oHnn3+eZ5999uX3X/nKV3jjG984rzrnnArMzJ3Azur9sxGxFZgA1gCnV1+7HrgLuLwq/1xmJnBPRBwRESuqeiRJkl6h0+0RSnryySd597vfDcDevXv5wAc+wJlnnjmvOrtaYxURDeBk4JvAMW1h6QngmOr9BLC97bRmVfYjwSoi1jIzosWxxx7bZbMlSZLm57WvfS3f/va3i9bZcbCKiMOBm4DLMvOZ9m3eMzMjIru5cGZuADYArFq1qqtzNTxKbwh3IP34V5AkaTR1tN1CRIwxE6puyMybq+InI2JFdXwF8FRVvgNY2XZ6vSqTJEkaanOOWMXM0NS1wNbMvKrt0CbgPGB99XpLW/klEbERWA087foq9dNCjIqBI2OS1K3MHOgHMc8sF+9OJ1OBpwIfAh6MiAeqsiuYCVQ3RsSFwOPAOdWx24CzgW3AC8AFXbdKkiQNtVqtRqvVYnx8fCDDVWbSarWo1WpdndfJXYF3A7P9xu84wPcTuLirVkiSpJFSr9dpNpvs2rWr302ZVa1Wo16vd3WOO68vQgs1tSVJUq+MjY0xOTnZ72YU57MCJUmSCjFYSZIkFWKwkiRJKsRgJUmSVIjBSpIkqRDvCpQK8fE8kiRHrCRJkgoxWEmSJBVisJIkSSrEYCVJklSIi9cL83EzkiSNLkesJEmSCnHESlpE3NJBkgabI1aSJEmFGKwkSZIKMVhJkiQVYrCSJEkqxMXrkn7EQm0Z4iJ5ScOoJyNWEXFmRDwSEdsiYl0vriFJkjRoio9YRcQS4BrgDKAJ3BcRmzJzS+lrdcONOyVJUq/1YirwLcC2zHwUICI2AmuAvgYrSYNlWP6x45SmpHa9CFYTwPa2z01g9f5fioi1wNrq43MR8Uih6y8Ddheqa9TZl2XYj+UMXF/+wY3R7yYcrIHry0XKfixnMfXlcbMd6Nvi9czcAGwoXW9ETGXmqtL1jiL7sgz7sRz7shz7sgz7sZxh6cteLF7fAaxs+1yvyiRJkoZaL4LVfcDxETEZEYcC5wKbenAdSZKkgVJ8KjAz90bEJcCXgSXAdZn5cOnrvIri04sjzL4sw34sx74sx74sw34sZyj6MjKz322QJEkaCj7SRpIkqRCDlSRJUiEGK0mSpEIG4iHMy5Yty0aj0e9mSJIkzWnz5s27M3P5gY4NRLBqNBpMTU31uxmSJElziojHZzs2EMFqmDTXfb3n16iv/3c9v4YkSeqea6wkSZIKMVhJkiQV4lSgJElacHv27KHZbDI9Pd3vpsyqVqtRr9cZGxvr+ByDlSRJWnDNZpOlS5fSaDSIiH435xUyk1arRbPZZHJysuPznAqUJEkLbnp6mvHx8YEMVQARwfj4eNcjagYrSZLUF4Maql5yMO0zWEmSJBXiGitJktR3pfeB7GTPx+9///tcdNFFPPTQQ0QE1113Hb/8y788r+sarCRJ0ki69NJLOfPMM/nbv/1bXnzxRV544YV512mwkiRJI+fpp5/ma1/7Gp/97GcBOPTQQzn00EPnXa9rrCRJ0sj53ve+x/Lly7ngggs4+eSTueiii3j++efnXa/BSpIkjZy9e/dy//3383u/93t861vf4rDDDmP9+vXzrtdgJUmSRk69Xqder7N69WoA3ve+93H//ffPu945g1VErIyIOyNiS0Q8HBGXVuVHRcTtEfHd6vXIqjwi4uqI2BYR34mIU+bdSkmSpIJ++qd/mpUrV/LII48AcMcdd3DiiSfOu95OFq/vBT6amfdHxCzhnnsAABroSURBVFJgc0TcDpwP3JGZ6yNiHbAOuBw4Czi++lkNfLJ6VSGlb0mdTSe3qkqSVEI//s75i7/4Cz74wQ/y4osv8trXvpbPfOYz865zzmCVmTuBndX7ZyNiKzABrAFOr752PXAXM8FqDfC5zEzgnog4IiJWVPVIkiQNhJNOOompqamidXa1xioiGsDJwDeBY9rC0hPAMdX7CWB722nNqmz/utZGxFRETO3atavLZkuSJA2ejoNVRBwO3ARclpnPtB+rRqeymwtn5obMXJWZq5YvX97NqZIkSQOpo2AVEWPMhKobMvPmqvjJiFhRHV8BPFWV7wBWtp1er8okSZJeNjMuM7gOpn2d3BUYwLXA1sy8qu3QJuC86v15wC1t5R+u7g58K/C066skSVK7Wq1Gq9Ua2HCVmbRaLWq1WlfndXJX4KnAh4AHI+KBquwKYD1wY0RcCDwOnFMduw04G9gGvABc0FWLJEnS0KvX6zSbTQZ5nXWtVqNer3d1Tid3Bd4NxCyH33GA7ydwcVetWAALtUWBJEma29jYGJOTk/1uRnHuvC5JklSIwUqSJKkQg5UkSVIhBitJkqRCDFaSJEmFdLLdgkbUQtxJ6YOeJUnDxBErSZKkQgxWkiRJhRisJEmSCjFYSZIkFWKwkiRJKsRgJUmSVIjbLaiv3NJBkjRMHLGSJEkqxGAlSZJUiMFKkiSpENdYaegtxDoucC2XJMlgJRXjQnxJUk+CVUScCXwCWAJ8OjPX9+I60qgxvEnSYCserCJiCXANcAbQBO6LiE2ZuaX0tSQtXoZEScOoFyNWbwG2ZeajABGxEVgDGKykRWCh1qQthGEKb8P0u0jDrBfBagLY3va5Caze/0sRsRZYW318LiIeKXT9ZcDuQnWNOvuyDPuxnMHry//c7wYctFf25eL9Xfpp8P6bXLwWU18eN9uBvi1ez8wNwIbS9UbEVGauKl3vKLIvy7Afy7Evy7Evy7AfyxmWvuzFPlY7gJVtn+tVmSRJ0lDrRbC6Dzg+IiYj4lDgXGBTD64jSZI0UIpPBWbm3oi4BPgyM9stXJeZD5e+zqsoPr04wuzLMuzHcuzLcuzLMuzHcoaiLyMz+90GSZKkoeCzAiVJkgoxWEmSJBVisJIkSSpkIB7CvGzZsmw0Gv1uhiRJ0pw2b968OzOXH+jYQASrRqPB1NRUv5shSZI0p4h4fLZjTgVKkiQVYrCSJEkqxGAlSZJUyECssZIkSaNlz549NJtNpqen+92UWdVqNer1OmNjYx2fY7CSJEkLrtlssnTpUhqNBhHR7+a8QmbSarVoNptMTk52fJ5TgZIkacFNT08zPj4+kKEKICIYHx/vekTNYCVJkvpiUEPVSw6mfQYrSZKkQlxjJUmS+u5jH/vYgtf3yCOP8Nu//dsvf3700Uf5oz/6Iy677LKDvq7BSpIkjaSf//mf54EHHgBg3759TExM8O53v3tedToVKEmSRt4dd9zBz/7sz3LcccfNqx6DlSRJGnkbN27k/e9//7zrMVhJkqSR9uKLL7Jp0yZ+67d+a951GawkSdJI++IXv8gpp5zCMcccM++65ly8HhErgc8BxwAJbMjMT0TEUcDfAA3gMeCczPzXmNn04RPA2cALwPmZef+8W6qXlb5zot/XkSSpn77whS8UmQaEzu4K3At8NDPvj4ilwOaIuB04H7gjM9dHxDpgHXA5cBZwfPWzGvhk9SpJknRA/frH/PPPP8/tt9/OX/3VXxWpb86pwMzc+dKIU2Y+C2wFJoA1wPXV164HfrN6vwb4XM64BzgiIlYUaa0kSVJBhx12GK1Wi9e85jVF6utqjVVENICTgW8Cx2TmzurQE8xMFcJM6NredlqzKtu/rrURMRURU7t27eqy2ZIkSYOn4w1CI+Jw4Cbgssx8pv35OZmZEZHdXDgzNwAbAFatWtXVuVoYCzEs6zouSdIw6WjEKiLGmAlVN2TmzVXxky9N8VWvT1XlO4CVbafXqzJJkqSXZQ72uMrBtK+TuwIDuBbYmplXtR3aBJwHrK9eb2krvyQiNjKzaP3ptinDoecITHccFZOk0VSr1Wi1WoyPj9M+CzYoMpNWq0WtVuvqvE6mAk8FPgQ8GBEPVGVXMBOoboyIC4HHgXOqY7cxs9XCNma2W7igqxZJkqShV6/XaTabDPI661qtRr1e7+qcOYNVZt4NzBYl33GA7ydwcVetkCRJI2VsbIzJycl+N6M4d16XJEkqxGAlSZJUiMFKkiSpEIOVJElSIQYrSZKkQgxWkiRJhXT8SBtpsVqoDULdiFSS5IiVJElSIQYrSZKkQgxWkiRJhRisJEmSCjFYSZIkFWKwkiRJKsTtFqRCFmK7Bbd0kKTB5oiVJElSIQYrSZKkQgxWkiRJhbjGSlpEXMclSYOtJyNWEXFmRDwSEdsiYl0vriFJkjRoIjPLVhixBPgn4AygCdwHvD8zt8x2zqpVq3JqaqpoO/bnv8KlweL/JyUtVhGxOTNXHehYL0as3gJsy8xHM/NFYCOwpgfXkSRJGii9WGM1AWxv+9wEVu//pYhYC6ytPj4XEY8Uuv4yYHehukadfVmG/XgAV1555cGcZl+WY1+WYT+Ws5j68rjZDvRt8XpmbgA2lK43IqZmG55Td+zLMuzHcuzLcuzLMuzHcoalL3sxFbgDWNn2uV6VSZIkDbVeBKv7gOMjYjIiDgXOBTb14DqSJEkDpfhUYGbujYhLgC8DS4DrMvPh0td5FcWnF0eYfVmG/ViOfVmOfVmG/VjOUPRl8e0WJEmSRpWPtJEkSSrEYCVJklSIwUqSJKmQgXgI87Jly7LRaPS7GZIkSXPavHnz7sxcfqBjAxGsGo0GvX5WoCRJUgkR8fhsx5wKlCRJKsRgJUmSVIjBSpIkqZCBWGMlSZJGy549e2g2m0xPT/e7KbOq1WrU63XGxsY6PsdgJUmSFlyz2WTp0qU0Gg0iot/NeYXMpNVq0Ww2mZyc7Pg8pwIlSdKCm56eZnx8fCBDFUBEMD4+3vWImsFKkiT1xaCGqpccTPsMVpIkSYW4xkqSJPXdHV/92aL1vePt/++c3/nzP/9zPv3pTxMRvOlNb+Izn/kMtVptXtd1xEqSJI2cHTt2cPXVVzM1NcVDDz3Evn372Lhx47zrNVhJkqSRtHfvXn7wgx+wd+9eXnjhBX7mZ35m3nUarCRJ0siZmJjgD/7gDzj22GNZsWIFr3nNa3jnO98573oNVpIkaeT867/+K7fccgvf+973+Jd/+Reef/55Pv/5z8+7XoOVJEkaOX//93/P5OQky5cvZ2xsjPe85z38wz/8w7zrnTNYRcTKiLgzIrZExMMRcWlVflRE3B4R361ej6zKIyKujohtEfGdiDhl3q2UJEkq6Nhjj+Wee+7hhRdeIDO54447OOGEE+ZdbyfbLewFPpqZ90fEUmBzRNwOnA/ckZnrI2IdsA64HDgLOL76WQ18snqVJEk6oE62Ryhp9erVvO997+OUU07hkEMO4eSTT2bt2rXzrnfOYJWZO4Gd1ftnI2IrMAGsAU6vvnY9cBczwWoN8LnMTOCeiDgiIlZU9UiSJA2EK6+8kiuvvLJonV2tsYqIBnAy8E3gmLaw9ARwTPV+AtjedlqzKtu/rrURMRURU7t27eqy2ZIkSYOn453XI+Jw4Cbgssx8pv35OZmZEZHdXDgzNwAbAFatWtXVuaOu9O60s1noYVlJkha7jkasImKMmVB1Q2beXBU/GRErquMrgKeq8h3AyrbT61WZJEnSy2ZWDQ2ug2lfJ3cFBnAtsDUzr2o7tAk4r3p/HnBLW/mHq7sD3wo87foqSZLUrlar0Wq1BjZcZSatVqvrZwd2MhV4KvAh4MGIeKAquwJYD9wYERcCjwPnVMduA84GtgEvABd01SJJkjT06vU6zWaTQV5nXavVqNfrXZ3TyV2BdwMxy+F3HOD7CVzcVSskSdJIGRsbY3Jyst/NKM6d1yVJkgoxWEmSJBXS8XYLGj0Lsa2DWzpIkoaJwaqwhdpjalgY3iRJw8SpQEmSpEIMVpIkSYUYrCRJkgoxWEmSJBXi4nUNPR9aLUlaKI5YSZIkFWKwkiRJKsSpQKkQ9+SSJDliJUmSVIgjVtIi4qiYJA02R6wkSZIKccRK0o9wewpJOniOWEmSJBXiiJWkvnC9mKRhNDLBaqGmNyRJ0ujqyVRgRJwZEY9ExLaIWNeLa0iSJA2a4iNWEbEEuAY4A2gC90XEpszcUvpakvRqhmmkeiGmNb1xQZq/XkwFvgXYlpmPAkTERmANYLCSpIM0TCFxWH4XA6IOpBfBagLY3va5Caze/0sRsRZYW318LiIeKXT9ZcDuQnWNOvuyDPuxHPuyHPty3gLsx5IWU18eN9uBvi1ez8wNwIbS9UbEVGauKl3vKLIvy7Afy7Evy7Evy7AfyxmWvuzF4vUdwMq2z/WqTJIkaaj1IljdBxwfEZMRcShwLrCpB9eRJEkaKMWnAjNzb0RcAnwZWAJcl5kPl77Oqyg+vTjC7Msy7Mdy7Mty7Msy7MdyhqIvIzP73QZJkqSh4LMCJUmSCjFYSZIkFWKwkiRJKmQgHsK8bNmybDQa/W6GJEnSnDZv3rw7M5cf6NhABKtGo8HU1FS/myFJkjSniHh8tmNOBUqSJBVisJIkSSrEYCVJklTIQKyxkiRJo2XPnj00m02mp6f73ZRZ1Wo16vU6Y2NjHZ9jsJIkSQuu2WyydOlSGo0GEdHv5rxCZtJqtWg2m0xOTnZ8nlOBkiRpwU1PTzM+Pj6QoQogIhgfH+96RM1gJUmS+mJQQ9VLDqZ9BitJkqRCXGMlSZL67qfvfKBofU/86klzfucTn/gEn/rUp8hMfvd3f5fLLrts3td1xEqSJI2chx56iE996lPce++9fPvb3+bWW29l27Zt867XYCVJkkbO1q1bWb16NT/5kz/JIYccwq/8yq9w8803z7teg5UkSRo5b3zjG/n6179Oq9XihRde4LbbbmP79u3zrtc1VpIkaeSccMIJXH755bzzne/ksMMO46STTmLJkiXzrnfOEauIWBkRd0bEloh4OCIurcqPiojbI+K71euRVXlExNURsS0ivhMRp8y7lZIkSYVdeOGFbN68ma997WsceeSRvO51r5t3nZ1MBe4FPpqZJwJvBS6OiBOBdcAdmXk8cEf1GeAs4PjqZy3wyXm3UpIkqbCnnnoKgH/+53/m5ptv5gMf+MC865xzKjAzdwI7q/fPRsRWYAJYA5xefe164C7g8qr8c5mZwD0RcURErKjqkSRJeoVOtkco7b3vfS+tVouxsTGuueYajjjiiHnX2dUaq4hoACcD3wSOaQtLTwDHVO8ngPbVX82q7EeCVUSsZWZEi2OPPbbLZkuSJM3P17/+9eJ1dnxXYEQcDtwEXJaZz7Qfq0anspsLZ+aGzFyVmauWL1/ezamSJEkDqaNgFRFjzISqGzLzpU0enoyIFdXxFcBTVfkOYGXb6fWqTJIkaajNORUYM08gvBbYmplXtR3aBJwHrK9eb2krvyQiNgKrgaddX6V+Kv2YhNn0Y32AJC1mmTnQD2KemZDrTidrrE4FPgQ8GBEv/Q11BTOB6saIuBB4HDinOnYbcDawDXgBuKDrVkmSpKFWq9VotVqMj48PZLjKTFqtFrVaravzOrkr8G5gtt/4HQf4fgIXd9UKSZI0Uur1Os1mk127dvW7KbOq1WrU6/WuznHndfXVQk3TSZIGy9jYGJOTk/1uRnEGK6mQhQiJruOSpMFmsNKsHE2SJKk7He9jJUmSpFdnsJIkSSrEYCVJklSIwUqSJKkQF68vQi4qlyRpMDliJUmSVIgjVtIi4l5ZkjTYHLGSJEkqxGAlSZJUiMFKkiSpEIOVJElSIS5el/QjFmo7DxfJSxpGBqvC3GNKkqTR5VSgJElSIQYrSZKkQgxWkiRJhfRkjVVEnAl8AlgCfDoz1/fiOpIWL3eRlzSMigeriFgCXAOcATSB+yJiU2ZuKX0tSXo1hjdJC60XI1ZvAbZl5qMAEbERWAP0NVh5t56kXnB7CkntehGsJoDtbZ+bwOr9vxQRa4G11cfnIuKRQtdfBuwuVNeosy/LsB/LGdm+jPJVjmxfFmY/lrOY+vK42Q70bR+rzNwAbChdb0RMZeaq0vWOIvuyDPuxHPuyHPuyDPuxnGHpy17cFbgDWNn2uV6VSZIkDbVeBKv7gOMjYjIiDgXOBTb14DqSJEkDpfhUYGbujYhLgC8zs93CdZn5cOnrvIri04sjzL4sw34sx74sx74sw34sZyj6MjKz322QJEkaCu68LkmSVIjBSpIkqZChCVYRcWZEPBIR2yJiXb/bs1hFxHUR8VREPNTvtix2EbEyIu6MiC0R8XBEXNrvNi1WEVGLiHsj4ttVX17Z7zYtZhGxJCK+FRG39rsti1lEPBYRD0bEAxEx1e/2LGYRcURE/G1E/GNEbI2IX+53mw7WUKyxqh6j80+0PUYHeL+P0eleRJwGPAd8LjPf2O/2LGYRsQJYkZn3R8RSYDPwm/532b2ICOCwzHwuIsaAu4FLM/OePjdtUYqI3wdWAT+Vme/qd3sWq4h4DFiVmYtlU8uBFRHXA1/PzE9XOwr8ZGZ+v9/tOhjDMmL18mN0MvNF4KXH6KhLmfk14H/0ux3DIDN3Zub91ftnga3MPJlAXcoZz1Ufx6qfxf+vwj6IiDrw68Cn+90WCSAiXgOcBlwLkJkvLtZQBcMTrA70GB3/AtPAiIgGcDLwzf62ZPGqpq8eAJ4Cbs9M+/LgfBz434F/63dDhkACX4mIzdVj2nRwJoFdwGeqKepPR8Rh/W7UwRqWYCUNrIg4HLgJuCwzn+l3exarzNyXmScx8zSHt0SEU9Vdioh3AU9l5uZ+t2VIvC0zTwHOAi6ullKoe4cApwCfzMyTgeeBRbtWeliClY/R0UCq1gPdBNyQmTf3uz3DoJoiuBM4s99tWYROBX6jWhu0EXh7RHy+v01avDJzR/X6FPDfmFmWou41gWbbKPTfMhO0FqVhCVY+RkcDp1pwfS2wNTOv6nd7FrOIWB4RR1Tvf4KZG1X+sb+tWnwy8z9mZj0zG8z8OfnVzPydPjdrUYqIw6qbUqimrd4JeDf1QcjMJ4DtEfHzVdE7gEV7k0/xR9r0wwA8RmdoRMQXgNOBZRHRBP4wM6/tb6sWrVOBDwEPVmuDAK7IzNv62KbFagVwfXUH8I8BN2amWwWon44B/tvMv584BPivmfml/jZpUfsPwA3V4MijwAV9bs9BG4rtFiRJkgbBsEwFSpIk9Z3BSpIkqRCDlSRJUiEGK0mSpEIMVpIkSYUYrCRJkgoxWEn6ERGREeE+LJJ0EAxWkiRJhRisJEmSCjFYSSouIj5bTSlORsQlEbElIqYj4rGIuKJ6jiIR8VsRcW9EPB8RT0XEX1bPAjxQna+v6t0eES9GxJMR8V/bni/W/t3XRcT6iJiKiF0R8cOIeDwiNkRE/QDfj4g4LyL+ofr+dHWdL0fEb+/33YyIu+b4vRttZY2q7LNVu/6m+l3/LSJOb/veURHxJxGxNSJ+EBFPR8QdEfHOznpd0iAYimcFShpY/4WZZ0/+38BXgN8A/hNwaET8D2A98H8BX2fmwcoXM/O8z99rryQizgRuBsaqurYBdeA9wK9HxK9m5v1tp7wH+PfAncA/AC8CbwAuAv7niFiVmTvavv+fgP8IfA+4EXiamecT/hLwW8DfzL8r+Fngm8A/ATcAPwE8U/1+xwF3AQ1m+uJLwGHAu4AvRcT/kpmfKtAGST3mswIl/YiXFq5nZsyjjs8C5wGPA6e+FGIi4ghmQtFPAC8Ap2Xm1urYjwPfYiaArMzMp6ryI5l5KOu+6vtb2q7zRuAe4J8y85S28glgd2b+cL92vRP4IrAhM3+vrbwF/AB4XWa+sN85yzJz9379898z8/RX+b0nM/OxqqzBTGAD+JPMvOIA590FnAZ8IDM3tpUfwUzg+nmgkZlP7n+upMHiVKCkXvo/2keGMvP7wCbgJ4FPvhSqqmM/ZGZk6FDghLY6PgwcAfxhe6iqznkI+BRwckSc2Fa+Y/9QVZV/BXgY+LUDtHUPM+Ft/3N2H+C7B+NJ4Mr9CyPiF4FfAW5qD1XVtb8P/CFQA95bqB2SesipQEm9NHWAsn+pXjcf4NhLIax9HdQvV6+/GBEfO8A5r6teTwC2wMyaKeCDwPnALwJHMjPF+JIX96vjBuA/AFsi4kbgvwP/T2Y+fYDrHaxvHyjs8f//fq+Z5fdbXr2ecIBjkgaMwUpSLx0omOzt4NhYW9l49fq7c1zr8Lb3VwGXATuBLzMT2H5QHTsfOG6/c/83ZqYbLwDWVT97I+I24KOZuW2Oa3fiiVnKX/r9zqh+ZnP4qxyTNCAMVpIG3UsB7Bcz8ztzfTkijgb+V+Ah4H/KzGf3O/7+/c/JzH3Ax4GPV+e/DTiXmYXrb4iIN7SNNiWz/9l5xKs0bbYFrS/9fpdm5tWvcr6kRcA1VpIG3T3V67/r8PuvZebPtq8cIFTVq+OzysynMvPmzDwH+P/auZ8Xm+IwjuPvj0kopewpeyal1PhRLGQ7ochGsbOQhbIRf4Q1tiwkljRWFpLFlLKiKEmEKSm7Y/Ecde6dM4zbmeku3q/N7Xa+53vv2X16zvN9nlLN9Ls7S74DO8bvSzID7F3lf+z63+eTNMUMVpKm3R1gCbiRZP/4xSQbuvOggHft56E27PxZt5VqdB+pNiXZlORgz74bge3t1+5JwRfAzp75UtdY/orxn5qmeUmNWDiR5HzfmiR72kqapCnnq0BJvdrRASu5OD6WYK00TfM1ySngAfA8yQJ1sq+hKkdzVJ/S5nb9pyR3qVd5i0keA9uo/qVfwCKjlaUtwLMkb6iG+vftXseohvFH3dOL1Gyu48DDJPeAb8ABYBc1GuHIBI95lqqO3UpyiZp3tUQ18c9SFbM54PMEe0taRwYrSSs595drlxmt4qyppmkWkswCV6hQc5g62feRCiT3x265QDWjn6aGjn6hxjxc71n7E7gKHKUC0jzwA3hLDSq93fNf5tu9zrT3P2l/a9k4hVU+34ck+6iTiSepE40zVMP7a+Am8GqSvSWtLweESpIkDcQeK0mSpIEYrCRJkgZisJIkSRqIwUqSJGkgBitJkqSBGKwkSZIGYrCSJEkaiMFKkiRpIAYrSZKkgfwGJgAoxH47G1gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x720 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax2 = df.plot(kind='hist', subplots=True, bins=20, figsize=(10,10), sharey=True)\n",
    "for a in ax2:\n",
    "    a.set_xlabel('L measure', fontsize=20)\n",
    "    a.set_ylabel('', rotation=True, fontsize=10)\n",
    "#ax2.set_ylabel('F', fontsize=10)\n",
    "print('Linearity measure distributions for each class in the fashion MNIST data set.')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}