18_kernel_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kernel Classifier\n",
    "\n",
    "The purpose of this tutorial is to make a dataset linearly separable.\n",
    "The tutorial is divided into two parts:\n",
    "\n",
    "1.  Feature transformation\n",
    "\n",
    "2.  Train a Kernel classifier with Tensorflow\n",
    "\n",
    "In the first part, you will understand the idea behind a kernel\n",
    "classifier while in the second part, you will see how to train a kernel\n",
    "classifier with Tensorflow. You will use the adult dataset. The\n",
    "objective of this dataset is to classify the revenue below and above\n",
    "50k, knowing the behavior of each household.\n",
    "\n",
    "## Why do you need Kernel? \n",
    "\n",
    "The aim of every classifier is to predict the classes correctly. For\n",
    "that, the dataset should be separable. Look at the plot below; it is\n",
    "fairly simple to see that all points above the black line belong to the\n",
    "first class and the other points to the second class. However, it is\n",
    "extremely rare to have a dataset that simple. In most case, the data are\n",
    "not separable. It gives naive classifiers like a logistic regression a\n",
    "hard time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "x_lin = np.array([1,2,3,4,5,6,7,8,9,10])\n",
    "y_lin = np.array([2,2,3,2,2,9,6,8,8,9])\n",
    "label_lin = np.array([0,0,0,0,0,1,1,1,1,1])\n",
    "\n",
    "fig = plt.figure()\n",
    "ax=fig.add_subplot(111)\n",
    "plt.scatter(x_lin, y_lin, c=label_lin, s=60)\n",
    "plt.plot([-2.5, 10], [12.5, -2.5], 'k-', lw=2)\n",
    "ax.set_xlim([-5,15])\n",
    "ax.set_ylim([-5,15])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/18_kernel_income_v4_files/image001.png)\n",
    "\n",
    "In the figure below, we plot a dataset which is not linearly separable.\n",
    "If we draw a straight line, most of the points will be not be classified\n",
    "in the correct class.\n",
    "\n",
    "One way to tackle this problem is to take the dataset and transform the\n",
    "data in another feature map. It means, you will use a function to\n",
    "transform the data in another plan, which should be linearable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHkRJREFUeJzt3XmUXGWd//H3t6u7estOmrDFhADGH+GQgE0UREwEA0QEXIYJKrJpAGF+MKPHX5SR8cw4KHgEBoMkgMgiBHRYhbBERhZFIJ0YSJAliywJkHQSsvVW3V3f3x91k+l0V3V3uqrv7er7eZ1Tp6vu81TdT25XvnX7qefea+6OiIjER0nUAUREJFwq/CIiMaPCLyISMyr8IiIxo8IvIhIzKvwiIjGjwi8iEjMq/CIiMaPCLyISM6VRB8hm9OjRPn78+KhjiIgUjSVLlmx095re9B2QhX/8+PHU1dVFHUNEpGiY2du97auhHhGRmFHhFxGJGRV+EZGYUeEXEYmZAfnl7p7ytlX4jhuh5ZnMgvLPYEMuwkoPjjaYiMgAVPR7/J56Cd/0ZWh+FHxb5tb8KL7xy3jqpajjiYgMOEVd+N3b8S2XgjcB6Q4taaAJ33Ip7ukczxYRiaeiLvykXgJvzt3uzZk+IiKyS3EX/vSGXvRZ3/85RESKSHEX/sQBQHcXi3dIjA0rjYhIUSjuwl92JNjI3O02EsqOCC+PiEgR6LHwm9mtZrbBzFZ0WHavmS0Lbm+Z2bIcz33LzJYH/Qp+8h0zw0beADYESHZoSYINwUb+EjMr9GpFRIpab+bx3wbMBe7YucDd/3HnfTP7ObC1m+dPd/eNfQ3YEys7FEY/hjf+BpqfyCysOBGr+jqWGNNfqxURKVo9Fn53f9bMxmdrs8zu9BnAZwsba89YYgw29Dsw9DtRxhARKQr5jvF/Gljv7itztDvwpJktMbPZ3b2Qmc02szozq6uvr88zloiI5JJv4T8TWNBN+7HufiRwMnCxmR2Xq6O73+Tute5eW1PTq2sJiIhIH/S58JtZKfAl4N5cfdx9XfBzA/AAMLWv6xMRkcLIZ4//BOB1d1+brdHMqs1s6M77wAxgRba+IiISnt5M51wA/AWYaGZrzez8oGkWnYZ5zGw/M1sYPBwD/MnMXgZeAh5198cLF11ERPqiN7N6zsyx/Jwsy94DZgb31wCT88wnIiIFVtxH7oqIyB5T4RcRiRkVfhGRmFHhFxGJGRV+EZGYUeEXEYkZFX4RkZhR4RcRiRkVfhGRmFHhFxGJGRV+EZGYUeEXEYkZFX4RkZhR4RcRiRkVfhGRmFHhFxGJGRV+EZGYUeEXEYkZFf5+0t7ezuYPPqRxe1PUUUREdtPjNXdlz7S3t3PvVQ/x39f8npbGFtLtaQ49ZiLfvu5cDpo8Pup4IiI97/Gb2a1mtsHMVnRY9iMzW2dmy4LbzBzPPcnM3jCzVWY2p5DBB6qfnvUL7r7yPrZv3kGquZW21nZeeeZvXHbsD1m17O9RxxMR6dVQz23ASVmWX+vuU4Lbws6NZpYAbgBOBg4FzjSzQ/MJO9CtXLqGvzy8mJbGVJe25oZmfnnpryNIJSKyux4Lv7s/C2zuw2tPBVa5+xp3TwH3AKf14XWKxqI7n6G1uTVn+2svrmT7hztCTCQi0lU+X+5eYmavBENBI7O07w+82+Hx2mBZVmY228zqzKyuvr4+j1jR2bZpO+m052xPlJbQ3NASYiIRka76WvhvBA4CpgDvAz/PN4i73+Tute5eW1NTk+/LRWLS0ROpqC7P2V5aVsrIMcNDTCQi0lWfCr+7r3f3dndPAzeTGdbpbB0wtsPjA4Jlg9bxXz+OkkT2TVpeleT0/3sypWWaSCUi0epT4TezfTs8/CKwIku3xcAhZnagmSWBWcDDfVlfsagaWslPHrucqmGVu/b8rcSoqC7nyBMO56wf/kPECUVEejGP38wWANOA0Wa2Fvg3YJqZTQEceAu4IOi7H3CLu8909zYzuwR4AkgAt7r7q/3yrxhADj16Ine/M49FdzzDq39+nSEjq5lx9nQ+NvVgzCzqeCIimHvuLyOjUltb63V1dVHHEBEpGma2xN1re9NXp2wQEYkZFX4RkZhR4RcRiRkVfhGRmFHhFxGJGRV+EZGYUeEXEYkZFX4RkZhR4RcRiRkVfhGRmFHhFxGJGRV+EZGYUeEXEYmZQXFVEG9bg++YDy1PZxaUT8OGXICVTog0l4jIQFT0hd9Tdfjm84EU0J5Z2Pww3vw4jPoVluzVWUpFRGKjqId63NP4lkuBJnYVfQjuN+FbLiVzdUgREdmpqAs/qcXgjbnbvTHTR0REdinuwp/+oDB9RERipLgLf2J/Mpf9zcWDPiIislNxF/6yj4ONyN1uIzJ9RERklx4Lv5ndamYbzGxFh2U/M7PXzewVM3vALHv1NbO3zGy5mS0zs4JfPd3MsJE3gFUDZR1aysCGYCNvwMwKvVoRkaLWmz3+24CTOi1bBBzm7ocDbwLf7+b50919Sm+v/r6nrGwSNvoxqDoLEmMzt6qzsNELsbJJ/bFKEZGi1uM8fnd/1szGd1r2ZIeHLwBfKWysPWOJfbBhc4A5UcYQESkKhRjjPw94LEebA0+a2RIzm93di5jZbDOrM7O6+vr6AsQSEZFs8ir8ZnY50AbclaPLse5+JHAycLGZHZfrtdz9JnevdffampqafGKJiEg3+lz4zewc4BTga+6edU6lu68Lfm4AHgCm9nV9IiJSGH0q/GZ2EvA94FT37IfOmlm1mQ3deR+YAazI1ldERMLTm+mcC4C/ABPNbK2ZnQ/MBYYCi4KpmvOCvvuZ2cLgqWOAP5nZy8BLwKPu/ni//CtERKTXejOr58wsi3+Vo+97wMzg/hpgcl7pRESk4Ir7yF0REdljKvwiIjGjwi8iEjMq/CIiMaPCLyISMyr8IiIxo8IvIhIzPc7jFwFwd1574U1+P38RG9/dxITDx/GFb5/IAYfsG3U0EdlDKvzSI3fnmm/dyB/veZ5UUwp3Z8WfXuORmxZx0bXncMrsz0UdUUT2gIZ6pEeL7niGp+99npbGFnaej6+ttZ1UU4p5/3wbf1/xTsQJRWRPqPBLj+756QM0N7RkbWtNtXH/tY+EnEhE8qHCLz16b836nG3p9jRvLlkTYhoRyZcKv/SockhFt+3Da4aFlERECkGFX3p04jnTKU1mnwdQMaSCL1x0YsiJRCQfKvzSo6/+4EuM2HsYpWWJ3ZaXVyaZWHsQx5xWG1EyEekLFX7p0bC9hnLjkquZcc50kpVJShIlDNtrCLPmnM5PHr+cRCLR84uIyIBhOS6XG6na2lqvq6uLOoZk4e60trRSVl6GmUUdR0QCZrbE3Xv157cO4JI9YmYkK5JRxxCRPGioR6RA3J1tm7fTsK0x6igi3dIev0gB/M/dz/HrH97DxnWbcXcmHD6OC372DSZPmxR1NJEuerXHb2a3mtkGM1vRYdkoM1tkZiuDnyNzPPfsoM9KMzu7UMFFBop7r36Qa2bP44O/b6At1UZ7azsrl6zh8s9fyYuPLok6nkgXvR3quQ04qdOyOcBT7n4I8FTweDdmNgr4N+ATwFTg33J9QIgUo22bt3PHj35LS2OqS1tLU4prZs8jnU5HkEwkt14Vfnd/FtjcafFpwO3B/duB07M89URgkbtvdvcPgUV0/QARKVrPP7iYkkTu/0ZN25t5s251iIlEepbPl7tj3P394P4HwJgsffYH3u3weG2wrAszm21mdWZWV19fn0cskfA0bG2krbU9Z3tJooSGrfqyVwaWgszq8czBAHkdEODuN7l7rbvX1tTUFCKWSL+bMHkcZTlOZwGQam5l3KSxISYS6Vk+hX+9me0LEPzckKXPOqDju/6AYJnIoDBl+mEMrxmW9WC2smQpH//c4Yzeb1QEyURyy6fwPwzsnKVzNvBQlj5PADPMbGTwpe6MYJnIoGBm/OTxf2XE3sN2O4tp5ZAK9v/ovnzv9ksiTCeSXa/m8ZvZAmAaMNrM1pKZqfNT4Ldmdj7wNnBG0LcWuNDdv+num83sP4DFwUv9u7t3/pJYpKgdcMi+3LnmBp6+93lefHQpZeWlTPvHTzF15hE6j5EMSDpXj4jIILAn5+rRKRtERGJGp2zowN2hdSmkloCVQfnxWOlHoo4lIlJQKvwBb9+Ef3gutL8NngISsP0avOJEbPhPMdOmEpHBQUM9ZPb0/cNzoW0VeBPQDqSAFmh+Et/+s4gTiogUjgo/ZIZ32t8B2rI0NkPjAjzdEHYqEZF+ocIPkKoDb8ndbqXQ9kZ4eURE+pEKP2S+yKW7+dYe9BERKX4q/ADlxwPdXT82CaWHhpVGRKRfqfADVjoOKmYAFVlaK2Do9zDTEZgiMjhojmLAhl+Fl+wFjfdkxvRxIAlDv0dJ1ZejjiciUjAq/AGzUmzYD/Ahl0Lb62CZ4R3t6YvIYKPC34mVVEPy41HHEBHpNxrjFxGJGRV+EZGYUeEXEYkZFX4RkZhR4RcRiRkVfhGRmFHhFxGJGRV+EZGY6XPhN7OJZrasw22bmV3Wqc80M9vaoc8V+UcWEZF89PnIXXd/A5gCYJnzGqwDHsjS9Tl3P6Wv6xERkcIq1FDP8cBqd3+7QK8nIiL9pFCFfxawIEfb0Wb2spk9ZmaTcr2Amc02szozq6uvry9QLBER6Szvwm9mSeBU4HdZmpcC49x9MvAL4MFcr+PuN7l7rbvX1tTU5BtLRERyKMQe/8nAUndf37nB3be5+47g/kKgzMxGF2CdIiLSR4Uo/GeSY5jHzPYxMwvuTw3Wt6kA6xQRkT7K63z8ZlYNfA64oMOyCwHcfR7wFeAiM2sDmoBZ7u75rFNERPKTV+F39wZgr07L5nW4PxeYm886RESksHTkrohIzKjwi4jEjAq/iEjMqPCLiMSMCr+ISMyo8IuIxIwKv4hIzOQ1j1+y8/RmvPFeSNVByXCs8suQPBqz4v6cfaNuNY/Mf5L6dzcx4fBxfOGiGex74JioY0kWG9dt4pH5i3j9pVWM2mcEM791ApOOmUhwIL3EnA3EA2lra2u9rq4u6hh94i0v4FsuBE8DzZmFVgVlk7GRN5M5p11xcXeuv/gWFt3xNK3NraTTTmmylJJECf8093xOOvezUUeUDp67/0WuOut60mmntaUVM6O8KsknPn8kP7j7MkpKinsHRLIzsyXuXtubvnoHFJCntwVFv5FdRR8yj1N/xbdfF1m2fDx113P84c5naGlMkU5ndhTaUm2kmlLMveRXvPXquxEnlJ02vLuRq866npamFK0trUDmg7u5oYUXHlnKA9cvjDihDAQq/AXkTQ9Czr+gmqFpAe6pUDMVwt1X3k9zQ0vWttZUG/dd90jIiSSX39/4BOn2dNa2lsYWfnv1QyEnkoFIhb+QWpeTORddDt4O6eI7Oel7qz/I2ZZuT/Pm4tUhppHuvP7iKlpTbTnbt2zYSqq5+HY+pLBU+AupZBSQ6KZDO9iQsNIUTOWQim7bh9cMCymJ9GT43t3/LkpKE5QmNacj7lT4C8gqvwSU5WqFZC1WMjTMSAUx4+xpOYtFRXUFX7hwRsiJJJfPf+sEKqqzf1CXJEr4zBnH6MtdUeEvJCubCJVfACo7t4BVYcN+GEWsvJ35/S8yfPRQEqW7/zWTrExy8BHjOea0oyJKJp1NmX4YUz57GOVVu88eKykxqkdUcd5/nhlRMhlIVPgLzIb9GIZ+F0r2JjPsk4Dy6dhe92GlB0cdr09G1AznxiVXc/zXPk2yooxEaYLqEVX8w7+cwlWLrujygSDRMTN+dN93+fq/fmXXh3VpspTPnHEM85Zczd5jdeVT0Tz+fuPu4E1gZZjlGv4pPu3t7aSaUlRUV+hgoAEuM42zmWRFUh/OMbAn8/j1LU8/McsM7ww2iUSCyiGdh7JkIDIz/a4kKw31iADNjS007ehmKm6I3J2GrQ2adin9Rnv8EmtL//AKN/+/3/D35W8DsP9H9+P8K7/KMaeG/4W1u/P7G5/g7ivvZ0v9NnCYPH0SF/zsG0w4fFzoeWTwynuM38zeArYD7UBb5zEmywwE/xcwE2gEznH3pd295mAY45eB79n//gtXnz2Xlqbd96zLq5J8+7pzmfnNE0LN84tLbuHJ256muXH3o6Qrqiu49tl/5+AjDgw1jxSXKM7VM93dp+RY6cnAIcFtNnBjgdYp0mdtrW1cd8H8LkUfoKUxxS8vu42mhuYsz+wf776xjsd//ccuRR+guaGZ//r2zaFlkcEvjDH+04A7POMFYISZ7RvCekVyWv7ca7TnOKcNQCJRwuLH/hpanqfueo721vac7auX/Z0P128JLY8MboUo/A48aWZLzGx2lvb9gY6nb1wbLNuNmc02szozq6uvry9ALJHcdmxp7LY93Z7usU8hbd24jfa23IW/tKyUhq3h5ZHBrRCF/1h3P5LMkM7FZnZcX17E3W9y91p3r62pqSlALJHcDjxsLG3d7GFjMOHwj4SWZ2LtwVR0c06kdNqpGbtXaHlkcMu78Lv7uuDnBuABYGqnLuuAsR0eHxAsE4nMAR/dj49+fELWA5tKSowx42qYeFR4R1pPm/UpEons/x2TlUlOPm865ZXloeWRwS2vwm9m1WY2dOd9YAawolO3h4FvWMYnga3u/n4+6xUphCt+9x32/sjo3c4+WjGkgpH7jODHj3w/1COTK6rKuXLhD6gaWrnbeXYqhlTwsaMO5ltXnxVaFhn88prOaWYTyOzlQ+aYgLvd/T/N7EIAd58XTOecC5xEZjrnue7e7VxNTeeUsLSmWnn+wcU889vnaU+nOfb0T/CZM44mWRHNJTK3f7iDJ379R5Y9/SrVw6o48ZxpTPnsYTqjpvRoT6Zz6lw9IiKDgK65KyIiOanwi4jEjAq/iEjMqPCLiMSMCr+ISMyo8IuIxIwKv4hIzKjwi4jEjAq/iEjMqPCLiMSMCr+ISMyo8IuIxIwKv4hIzJRGHUCKg3sKb3wAmu6E9CZIHIhVfxPKp4d63nqRfG3bvJ2H5j7OojueprkxxaSjJzLr+19kYu1BUUcLjQq/9Mg9hW8+C9peB2/KLExvwre8CpVfhmE/VPGXorDxvc1cfNQctn+4g9bmVgD+/OBLLH78r1w2fzYnfP0zEScMh4Z6pEfecBu0vva/RX+XJmi6D1oXRxFLZI9dd8F8tmzYuqvoA7g7LU0prp09n60bt0WYLjwq/NKzxjuB5hyNzXjD7WGmEemTbZu3s/QPy0m3p7N3MOOpu54LN1REVPilZ+lN3TQ6tL8TWhSRvtr8/hbKkrlHt1NNKd5bvT7ERNFR4ZeelezVTaNB4iOhRRHpq1H7jKA11ZazvbwyyX4HjQkxUXRU+KVnVV8HKnI0VmDVZ4eZRqRPhu01lCOOP4ySRPay5+4c/7VPh5wqGn0u/GY21sz+aGZ/M7NXzezSLH2mmdlWM1sW3K7IL65EwarPhbKPAZWdWiqh8otQdlQUsUT22D/Pv4ARew8nWV62a5mZUV6V5LL5FzB89LAI04Unn+mcbcB33H2pmQ0FlpjZInf/W6d+z7n7KXmsRyJmloRRv8Eb7w/m8W+GxPhgHv9nNZVTisbo/ffi5uU/56G5j/Pk7U/T0tjCpGM+xqw5pzPxqIOjjhcac/fCvJDZQ8Bcd1/UYdk04Lt7Wvhra2u9rq6uILlEROLAzJa4e21v+hZkjN/MxgNHAC9maT7azF42s8fMbFI3rzHbzOrMrK6+vr4QsUREJIu8C7+ZDQHuAy5z985HPywFxrn7ZOAXwIO5Xsfdb3L3WnevrampyTeWSK+5p/DUEjxVh3c5SE1k8MnrlA1mVkam6N/l7vd3bu/4QeDuC83sl2Y22t035rNekUJwd7zhFmj4JbDze4p2vOo8bMg/YaZJbzI49bnwW+YbvV8Br7n7NTn67AOsd3c3s6lk/sLo7mggkdB4w42wYz7QaS+/4Vbcd2DDLo8kl0h/y2eP/1PAWcByM1sWLPsB8BEAd58HfAW4yMzayPzvmuWF+jZZJA+eboAd88h+KoomaFyAD7kIKxkVdjSRftfnwu/uf+J//z7O1WcuMLev6xDpN62LwUoh525IKbQ8B5WnhZlKJBQaxJR48taeOvSij0hxUuGXeCqbAp7qpkMakr2aEi1SdFT4JZYsUQMVJwLlWVqTkDwKKx0fciqRcOgKXBJbNvxK3Bug5c9kBvsdSEDZZGzE9RGnE+k/KvwSW2bl2Mh5eNvqzBe5tEPyGKzs/0QdTaRfqfBL7FnpQVAanwtti2iMX0QkZrTHLyI5eds7eNO90LoSEmOxqllY2SFRxxpU3JuheSHe/BSQwCpPhvITyJwRp3+o8ItIVumGu2D7T4F2MpffSOBNv8Wrz6Vk6L9EnG5w8LZ38c2zwBvAGzPLUs9CybUwagGW6O6yp32noR4R6cJbl8P2q4AWMkUfMh8ALdB4O978P9GFGyTcHf/wW5DetKvoZxoaoX0tvuWyflu3Cr+IdOE7bgFyHODmTZkT3El+WpdB+gMgnaWxDVqX4W3v9MuqVfhFpKu2FWQvSDvbV4UWZdBqex28m21sZdC2sl9WrcIvIl1ZDxcdt+pwcgxmNhQs0U0Hh5L+ufi7Cr+IdGFVXwUqc7QmofKMMOMMTuXTwdu76ZCEsiP7ZdUq/CLSVeWpUDoeSHZqKIWS0Vj12RGEGlyspBqGzgEqsrRWYMN/jHX7F0HfqfCLSBdm5diou6Hqq2BVQBlQDpWnY6Pvx0qGRx1xUCip/io24ueQmEBmdn0CSidhI+djFZ/rt/XaQLwgVm1trdfV1UUdQ0QA93bw7WDV/XpQUdx5egdQgpVU9en5ZrbE3Xt1LnEdwCUi3TJLgI2IOsagZyVDQluXhnpERGJGhV9EJGZU+EVEYkaFX0QkZgbkrB4zqwfejjpHL4wGNkYdYg8VW+ZiywvKHIZiywv9n3mcu9f0puOALPzFwszqejt9aqAotszFlheUOQzFlhcGVmYN9YiIxIwKv4hIzKjw5+emqAP0QbFlLra8oMxhKLa8MIAya4xfRCRmtMcvIhIzKvw9MLOxZvZHM/ubmb1qZpdm6TPNzLaa2bLgdkUUWTvkecvMlgdZupztzjKuN7NVZvaKmfXPSb97ycwmdth2y8xsm5ld1qlP5NvYzG41sw1mtqLDslFmtsjMVgY/R+Z47tlBn5VmFto5jXNk/pmZvR787h8wy34inp7eRyHm/ZGZrevwu5+Z47knmdkbwft6Thh5u8l8b4e8b5nZshzPDX0bA8EFf3XLeQP2BY4M7g8F3gQO7dRnGvBI1Fk75HkLGN1N+0zgMcCATwIvRp25Q7YE8AGZOckDahsDxwFHAis6LLsamBPcnwNcleV5o4A1wc+Rwf2REWaeAZQG96/Klrk376MQ8/4I+G4v3jergQlkLiLwcuf/p2Fm7tT+c+CKgbKN3V17/D1x9/fdfWlwfzvwGrB/tKnydhpwh2e8AIwws32jDhU4Hljt7gPuAD53fxbY3GnxacDtwf3bgdOzPPVEYJG7b3b3D4FFwEn9FrSDbJnd/Ul3bwsevgAcEEaW3sixjXtjKrDK3de4ewq4h8zvpt91l9nMDDgDWBBGlt5S4d8DZjYeOAJ4MUvz0Wb2spk9ZmaTQg3WlQNPmtkSM5udpX1/4N0Oj9cycD7MZpH7P8lA2sY7jXH394P7HwBjsvQZyNv7PDJ//WXT0/soTJcEQ1O35hhOG6jb+NPAenfPddX0SLaxCn8vmdkQ4D7gMnff1ql5KZmhicnAL4AHw87XybHufiRwMnCxmR0XcZ5eMbMkcCrwuyzNA20bd+GZv92LZpqcmV0OtAF35egyUN5HNwIHAVOA98kMnRSLM+l+bz+SbazC3wuWuezQfcBd7n5/53Z33+buO4L7C4EyMxsdcsyOedYFPzcAD5D5M7ijdcDYDo8PCJZF7WRgqbuv79ww0LZxB+t3DpMFPzdk6TPgtreZnQOcAnwt+MDqohfvo1C4+3p3b3f3NHBzjhwDcRuXAl8C7s3VJ6ptrMLfg2CM7lfAa+5+TY4++wT9MLOpZLbrpvBS7pal2syG7rxP5ou8FZ26PQx8I5jd80lga4fhiijl3DsaSNu4k4eBnbN0zgYeytLnCWCGmY0MhilmBMsiYWYnAd8DTnX3xhx9evM+CkWn75++mCPHYuAQMzsw+MtxFpnfTZROAF5397XZGiPdxmF/m1xsN+BYMn++vwIsC24zgQuBC4M+lwCvkplJ8AJwTIR5JwQ5Xg4yXR4s75jXgBvIzIJYDtQOgO1cTaaQD++wbEBtYzIfSu8DrWTGkM8H9gKeAlYCfwBGBX1rgVs6PPc8YFVwOzfizKvIjIfvfD/PC/ruByzs7n0UUd47g/fpK2SK+b6d8waPZ5KZdbc6rLy5MgfLb9v5/u3QN/Jt7O46cldEJG401CMiEjMq/CIiMaPCLyISMyr8IiIxo8IvIhIzKvwiIjGjwi8iEjMq/CIiMfP/AdxnfMU1N27oAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.array([1,1,2,3,3,6,6,6,9,9,10,11,12,13,16,18])\n",
    "y = np.array([18,13,9,6,15,11,6,3,5,2,10,5,6,1,3,1])\n",
    "label = np.array([1,1,1,1,0,0,0,1,0,1,0,0,0,1,0,1])\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.scatter(x, y, c=label, s=60)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/18_kernel_income_v4_files/image002.png)\n",
    "\n",
    "The data from the figure above is in a two-dimension plan which is not\n",
    "separable. You can try to transform these data in a three-dimension, it\n",
    "means, you create a figure with 3 axes.\n",
    "\n",
    "In our example, we will apply a polynomial mapping to bring our data to\n",
    "a 3D dimension. The formula to transform the data is as follow.\n",
    "\n",
    "\n",
    "\n",
    "$$\\phi(x,y) = (x^{2},\\sqrt{(}2)\\text{xy},y^{2})\\quad\\quad$$\n",
    "\n",
    "You define a function in Python to create the new feature maps\n",
    "\n",
    "You can use numpy to code the above formula:\n",
    "\n",
    "| Formula | Equivalent Numpy Code |\n",
    "|---------|-----------------------|\n",
    "| $$x$$     | x[:,0]**              |\n",
    "| $$y$$    | x[:,1]                |\n",
    "| $$x^2$$  | x[:,0]**2             |\n",
    "| $$\\sqrt(2)$$ | np.sqrt(2)*           |\n",
    "| $$xy$$    | x[:,0]*x[:,1]         |\n",
    "| $$y^2$$ | x[:,1]**2             |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "### illustration purpose\n",
    "def mapping(x, y):\n",
    "    x = np.c_[(x, y)]\n",
    "    if len(x) > 2:\n",
    "        x_1 = x[:,0]**2\n",
    "        x_2 = np.sqrt(2)*x[:,0]*x[:,1]\n",
    "        x_3 = x[:,1]**2  \n",
    "    else:    \n",
    "        x_1 = x[0]**2\n",
    "        x_2 = np.sqrt(2)*x[0]*x[1]\n",
    "        x_3 = x[1]**2\n",
    "        \n",
    "    trans_x = np.array([x_1, x_2, x_3])\n",
    "    return trans_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new mapping should be with 3 dimensions with 16 points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 16)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_1  = mapping(x, y)\n",
    "x_1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a new plot with 3 axis, `x`, `y` and `z` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXmYXGWZ9/89Sy1dXUvvS9JJOunOvkKAENCMgCg6bCMYMiogZGR++Kq8IGMYgRF1VLz84Tbh944LIMhARFTy6iDIAFFJSIKQTshGJyS9b7Xvy9l+fxTPSe11TnVVpbp5PtdVV9Jnr1PnfJ/7uZ/7vh9GURRQKBQK5ezDnu0LoFAoFEoSKsgUCoVSI1BBplAolBqBCjKFQqHUCFSQKRQKpUaggkyhUCg1AhVkCoVCqRGoIFMoFEqNQAWZQqFQagRe5/Y0rY9CoVD0w2jZiFrIFAqFUiNQQaZQKJQagQoyhUKh1AhUkCkUCqVGoIJMoVAoNQIVZAqFQqkRqCBTKBRKjUAFmUKhUGoEKsgUCoVSI1BBplAolBqBCjKFQqHUCFSQKRQKpUaggkyhUCg1AhVkCoVCqRGoIFMoFEqNQAWZQqFQagQqyBQKhVIjUEGmUCiUGoEKMoVCodQIeufUo1QIRck9XSHDaJqKi0KhzAKoINcI4XAYDMOAZbM7LflEWVEUBINBOByOtO0KiTjDMHnX611ebB2FQtEHFeQaQJIkxONxGI1G8PyZnySf1Zy6/tixY9iwYUPa9sX204ssyxgYGMCiRYtyrtcq5LTBoFAKQwW5BojH4zlFtJiYsCwLWZbzCl85cTqd6O3tTVumpcHI9Xeh/fbt26c2MOUi1/1wu91oaWnJuR1tMChnCyrIZxlZlpFIJHK6KopRrZeZYZiSGoxCx8uHoigl3Yt8x8pHf38/mpubc25fzh7GwMAAOjs7YTKZ8m5Tjh4GuWaWZWmDMYOhgnyWSSQS6v/L7WooFzP1pSx23aUIlF78fj86OjpyNjLl7GEMDg6C53nMnTt3Glebm9T7IYoiAoEAmpqadLukOI4rW2M7W6GCfBZRFAXxeFx1PdSqIFNKR5blvCJUzh6Goijgeb6sgpfreYzH4xgeHkZTU5NulxQV4+JQQT6LJBKJsnbRKbWHoihV6WEUEv5SyXXdsiyD5/mS3BwztadVTagSnCUURUF/fz9CoRCA/H5aysxmJgtyLiRJKvk8VJCLQwX5LCGKIhKJBCRJmvaxqJDXLrNRkDmOq/h53q9QQT4LEN8xz/OQZRlA6dYDx3HqMSi1BxVkih6oIJ8FZFmGKIrgOC7NQi7F0mVZtixWNqUyVGuMYCYIMnVZFIcK8lkgHo8DQNkEmVrItQu1kCl6oIJcZVITQVIFeToui2pZyNRXrZ9q3bNaF2RqHWuDhr1VGRLqxjAMOI5LSwwJh8MQRVHNtmJZNuuTmbpbLQuZRIHQF0s/s81CNpvNFT/P+xUqyFVEURScOnUKXV1dANL9v8FgEMePH0dTU5OaJCLLctYn0+KKRqPw+/1pVkumgOcT9lzL862TJAk+n09NPsi3DxXss0OtW8gUbVBBriKJRALj4+OYN28egHR3w+joKBYtWqSKtVaOHTuG9vZ2NDU1AUCWkBcS9lzLJUnKuU4QBIyOjgJAweMVo1gjEYvFcPz4cc0NiJ6GZTY3FrUuyLP53pcTKshVQlEUxGIxMAyjvjwkZC2RSCAcDqOnp0f3i5XpsiCukHJbMcFgEEuXLoXRaCz5GIqi5G0IiKh7PB60trbmbTwEQSi5wUklFAphz549actyibvenkTmclEU4fP5NO0zHWpdkCnaoIJcJURRhCzLqlWc6goYHR3VbRkTquVDLsd5iP+7kHDwPJ9Vha0S7NmzBxdddJH6d6HGItfyfL2IzHWiKGJ0dLRg46HlvhZrLMLhMPr7+3U3HlobFgIV5MpCBbkKEOs41SoGki4LURQxNTWF8847r6RoiWpFWcz21G4tjUUpuN1urFy5clrHyNdYpC7zeDxoa2vLK/z5XFHFeiyZ4xaRSATBYDDtPqUKdz5xb2lpUV11lPxQQa4CkiSpiSCpAspxHKLRKFpbW0sW1plkIVNKo5Z6Fm+88QbWrl2ruq4yG4t8Al9XV1fxa5sNUEGuAvF4XH2pMmOPY7FYWg1bvVYotZAp1STTZaG1Z0HdHNqgiSEVRhRFteYxkC6gwWAQDMMUnE2iGNW0kKkgU0odPKRRFtqgglxhnE4nTp48qT6QqYI8MjIyraiFzONVEhIdUg2o8Nc2VFwrBxXkCpJrFJ1EVkSjUdWvnIpeMZptPmTqGqG8n6GCXEESiUSWBUv+HhkZyRp1LsXyoEJZu1Trfs2E34Va1dqgglwhSM1jg8GQJciCIMDr9WZNQ09GrPVQLZcFFX79zLZKb5TKQ3/FCiEIgjrxZKYg+3w+dHZ2qi8rEaBSxGi2CeVssqRmmyDTsMfKQwW5AhDrmIQDpQoZy7IIBALo7OxU/57Ogz7bfMjAzOiCa2E2CjINX6ssNA65AkiSlHcyyFAoBJPJBJ5P3nricuA4DqOjo5icnMz50OdLe5VlGX6/H8eOHSspbbbYdkRQqmkhU0HWx0yoYzGbej6VhApyBRgfH0ddXV3O+GK32w2LxaL+TQRZlmVMTEzgvPPOS1sPZFdwS/3EYjHEYjG0t7fn3SZf2qyegjwklvrUqVNp11Yu4SfLRVGE3++HwWAoun+tQwWZohcqyGWGRFAsWLAgS5CDwSA4jkt7eYiV63Q60dTUlPPFKlTBzWw2g+M4tfxmpRgYGIDBYMjKKtRaFyHXdpIkZVVvIyVKSdxzvv2LWdHFGod8ZT7LUZCHiHA1XQlUkGcHVJDLTOZs0gRFUTA8PIzOzk643W51ObGQR0dHsXjxYt3nI3HNlSaXD7kSpT6rVebT7XZnlflM7VGQQVm9PYrUhoL0YFLLfJarxGfqJxKJQBRFBAKBgseZrrVOXRaVhwpyGSEWXmZkBbHIIpEIFi1ahKmpKXUdx3EIBoMwGAyoq6t730dZANMf1KuVYjyRSATvvPMOzjnnHADaqrYVW5fZo5BlGdFoFJFIBIODgwWPU4jUe5bvE4/HEY1GceLECV2NB8dx0yoP8H6CCnIZIfPj5UoGGR0dxdy5c3OGwU1NTaG7u7ukc1bL8mBZFoIgVPw8s3lQr5IlPp1OJ5YtW1byMbS4mzweD1iWhcPhyFpXqEfB83xWzD0lN1SQywQJdSMWQaaF7HK5sHDhQjAMk7aO1EpuaGio6ThPGoesn5k0qKfF/URmvGlra9N9bIo2an+oeoaQSCQgy3JWiU0gmSTS0NCQNaAHJMPgHA5HVpJIrVEt1whQu/dALzNJkLVAB/UqDxXkMpBqHQPpLgtFURCJRHJ22WRZRigUgtVqTTtWLULjkPWjKAqNsgC1kPVABbkMhMNhTE1NqS9Fqp/Y5/PBaDTmfGFcLleaGNcy1bSQZwukx1SN89SyIFO0QwV5miiKgnA4DKfTqS5LDUUbHh6Gw+HIGZo2MjKClpaWtBlEpnMdlYRayPqplssiX1ZoJc5DBbmyUEGeJkRMUwWXWMik5rHFYskS5GAwCJ7nUVdXlzXIp5dqWK+zMbyu0lAfchLqstAOFeRpkq/msSiKas3jXFEXw8PD6OrqKkv5zGqI5WwSympRLUFWFKUqliu1kCsPFeRpQBJBitU8zhRdhmEQCoXQ2NhYFjGtRk1kaiHrh7osKHqhccjTIB6P57ROOI5DNBpFV1dX3jC4tra2rHWKouDIkSMQRVFTkZ7U1NnBwUGYTCZd++XKsMonILNJKKsFjbJIQl0W2qGCXCIk1C1fqcx4PK7WPE4VXVmWEY/H1WJAqeu8Xi/MZjNWrVpVsLhOrgwqg8Gg1tDIVbRHa8GffEiShEQigX379mmusVBKXQZJkhCPx2E2m9P2mYnMNh9ytc7zfoYKcokkEom8L5zH4wHHcVk1j4FkmmtqXn9qRMbIyAi6u7vThEsLU1NTaG5uhsPhmO7Xyks4HEZ/fz9Wr15dtP5CvoI9Wvbx+/1qTDc5TzHLvBThTyQSGB4e1rVP5jmKie1sC3sTRVF9pimVgd7dElAUBYODg1kppESgR0ZG0qqVpQryyMgI7Ha7+jfHcZBlGZFIBIqiwGKx1GSBIZZl1SmpKsnhw4cxb948zY2L3mI9pHEg+5KehN4GRkuxHlEUoSgKfD5fycKvZR9ynkpb5KUKP3VZaIcKcgmIoojR0dE0QSaiGIlEslKkiSCHQiGwLAuTyaSKAhE6UnyoFKoxqFerccillAAlDer8+fNLuURNxye/aTweR1dXl+bGopDLKd++gUAAwWBQk/BNR/zj8TgmJyd170vHHrRDBVknpBgQiTUmQkBEcXh4GPPmzcPJkyfVfYgVPDIygq6uLoRCoazYY4/Hg56eHoiiqPuaqmUhVyvKYqaTWtWN53mYzeaKnq+vrw+LFy9GfX19we309ALy9Tyi0aju/datW4eGhoaK3oPZAhVknUiSBFEU1Vhj4prgeR6RSASRSER9+EgXkmWTpSsTiQSWLl2qPtQEURTR2dmpWtW16LKYSfWQtRy/WvHBtTTYNt0JBUZGRrBo0SLd+xkMhpLO936EDpnqhJQgzFXXeGJiAnPnzs0KZ+M4DvF4HB0dHTlD3RKJBObMmQOgNAuRxiHXJrNtUK9UZkOvp1rU7q9YgxDrmGVz1zz2eDyqXzlX7HGuMDgy7U7qIOD73UKeLcymsDf621cHKsg6iMfjAJBl5QJJy9nhcGT5lIFkqFtqGFxqqNvo6CiMRmPaA6/34X8/D+rVMrNJkGvdCp8t0DusEVlOpkmTh5LneXUATlEUhEKhtIGLzFC3zDA4crxwOAyj0TgtQa2WhVwNquXbnU3nqYZY0glOqwMVZI2EQiGIoqg+XKmC6/P51LRlAlkfDocBIOe68fFxzJkzJ8sfrZdqWMjVhFrI+qn0eWgdi+pABVkDiqLg9OnT8Pv96rJUERweHkZTU1NayBpZT0LdSCo0WSeKIiYnJ9He3j5tQa3WgFs1oC6L2oTWsagOVJA1IAhClmgSlwWpeWy1WrOiLhKJBPx+P5qbm7OiLsjEpjzPZwlqLQ7qVQvqSqhNqIVcHWgcchEyE0EIqRbwvHnz0gbqgKRgu93urFA3g8EAjuMQiUSwfPnytGMByWmdBgcHs1KUi6XOhkIh9Pf3607LnU6thkpBLeTagwpydaCCXIS3334bf/jDH7B582ZEo1F1Oal5HAwG0dvbC7/fnxUG5/f7sXTpUnV7sp6U7SSZVanrRkdHsXbt2rRaDsUyrEKhEARBQGNj47TTcFM/mYRCIezZswcASqrJoCXdNpFIIBaLIRKJ1FQjUQpUkKnLQi9UkIugKAqOHz+eZSHzPI9QKJQz2QMAotEoTCaTmqWUun58fDwte4m4HILBIAwGQ1pEBlA8w4rneTidTrS2tpbte+diz549uOiii9RaDXqK8aR+BEHIu7/P50MoFILb7dZd0Eer+APJ3+f06dNlaVDyQQWZohcqyEWw2+0IBoM5E0EikYiaYZcp2B6PJ21GabK/JEnwer1pLgmybmxsTE0e0UO1oyxSazWUm3fffRf19fXo6OjQtZ+eBkIQBDidTphMprRlpTQuhYjFYnC5XGkTB5SrZ5F6PFmW1YSlSvUiqCBXByrIRbDZbAiFQmlxxwDg9/vB83xazWOyPhKJqMsIRDSnpqbQ2toKl8uVtk4QBPj9fvT09NT0oF6lrb5Sj62nToMgCBgZGVEb00px5MgRtLe3w26363YXaZ1kQJIkxGIxvPnmm5p6EaU2CKSE6Pj4uO59KdqhglwEm82GQCCQZYWOjY3ltHKBZCJIe3u7GoOcun5sbAyrVq1KE2SWZeH1etHe3l5S2Fe1LGRybZXuhs+mQb1cLqhyIooi3nrrLVxwwQWFN5QDYBL/DUhDkJk5ELiPQlbsmnsC5PkqVu0t8zNnzhwsXry4Yt9/tkEFuQgmkwmCIKRZyMFgMMtFQaxUURTh8/mwcuVKBAIBdT3HcQiFQjCZTOqMIanV4Px+P5YsWaIu10O1LORqxAhX4xyzKVNPywSnTPxFcJH7kv9HDApMMOBHkCz3QbFcq+k8sVgMNpsN7e3tuq6Pujn0QfsTGslMBJk3b17aevLiTUxMoKOjI6dgezwetQh9aqIImZvPaDSW9AJXa+Comq6R2UA1BLlorLP4DrjIfWAQA4MYAIBBHAzi4CLfAiMe1HSeUme2ni2DmtWCCrJGWJZVS2WGw+GcBbcVRVEH5nK5ERKJhLofqWcBJAcAU4uL12qXfbZYyNWiFgSZiz0KIJFnbRxs9Geaz0Pn06s8VJA1kCoSZKol8qKliockSbDb7WryR6og+3w+1NfXZ9XCEAQBsVhsRnTtqmEhU0HWRzFBZsS3wCD3b8ZAASP1aTpPqRYyRR/0DmvAYrGok5BOTU2pfrRMgUokEmkuCYKiKPD7/airq1OXEUGemJhAa2vrjBAh0kugaKMWBBlMsemjTEXWJ6GJIdWBCrIGSOibKIpobW3NWfM4EomAYZg00SW43W5YrdY0MSNhchMTE2hvb8+aY68WfbWpfu9KnmM2DepV2qosJsiy8WooeURXgQGy8SpN56FxyNWBCrIG7HY7/H5/2qwfQHpN5NHRUZjN5pzhZ6Ojo+jo6MiqheH3+2G1WrNmoS6V2eLfnS1WeC1YyLLpBoBxQEG6mCpgAcYK2XyjpvNQQa4OVJA1YLPZ4PP51MpsBGIhi6IIr9eLurq6LEGORCKQZRk2my0r6sLlcmHu3LlZrg9BEBAKhRCJRBCLxZBIJCCKYsHA/2qIZbV8yLMFWa78nHpFXRasHaJ9BxT+YigwQkF98l/+Aoi2HQDbrOk8VJCrAx021YDdblcjK3JVfJucnERbWxui0WjaeoZh1EHAXFEX8XgcNpsNAFShk2UZb7/9NhwOR96p2HMRDofx+uuvg+O4sqbmpn4kSUI8HldnTqlEJlY13CKzKQ65qCADANsKyfYwIHsBeQpgWzQLsa7z5GA2NbDVgAqyBliWxauvvopPfepTaenTPM9DEASMjY1h7dq1GBwcTFvPsizcbjd6enoAIE2QQ6EQHA5H1gPrcrnQ1NSEFStW6LJI/va3v2HVqlUwGAwl1WTQkqrr9/vViBC9BX9IQ5ErdTe1EQkEAlAURV2Wr4EhqdJ6iv1Um5oRZALbmPyUSC3d29kKFWQNvPHGG1i1alWWlctxHAKBAKxWK4xGY9Z6SZLQ3NysvjBEvMgcfJnJJUAyJXvhwoW6r5GIpNZ6DqVw7NgxtLe3o6mpSdP2pVSEC4fDEAQBoijm3F+SpLw1IYqlj6cWRAqHwzhw4EDFehPEvXO2B/UoMwsqyEVQFAWiKKK7uzutgBCQFMHJycmcheaBZLppS0tL1jE9Hg8sFkuWdRmNRqEoCurq6moyfVrvOfQU/CEoioJ4PI7u7u4SrrAwRLij0SiOHTuGpUuXam4oSqkGFwqFsHfv3rzXU47Kbz6fDwzDwO12a9q+2lYutar1QQW5CAzD4Etf+hKOHTuWlQ5NBvSIHzhVkAOBAHiezylGo6OjaG5uThN3IGkdz5kzp6SHuBoFhmZ60gYRJVKlz2KxVPR8e/bswcaNG3P+nuWoKU0aFwBwOp2a9itEoQaCNGKFxD7TLcXzPNra2ipyb2crVJA1YLfbEQqFskTP7/erYgykl+AcGRmBzWbLEt1YLAZBEGC1WuHxeNTliqLA5XJh4cKFJQlrLVrIpTDTRT+TfI1ruWpKMwwDk8k07VKihRoIUjCLxMvn2o5knKYuo4KsHyrIGiBF6nmeRyyWLNAiSRJCoVDaLB1kvSAICIfDaG5uzhJXYgVnirssy2hqalKjGWrRZVGtWhYU7ZTLh1yogRAEASaTSfPYQeoxKfqgowEaIJl6qSI6OTmJpqamNBEk68fHx9HZ2Zmz4pvT6URbW1uWIAuCoIo7sVb0UA2XRTVEH6h8YshsmlqJDORWEhqDXD1mtCDHYjFccMEFWLt2LVauXImvfe1rAIDTp09jw4YN6O3txQ033IBEIlntKh6P44YbbkBvby82bNiAgYEBTedJtZDJ6D/Jvssc5CPp0B0dHTmtYIfDoQ50pRb9BqDWSS5FLGaThTybXBaVphpRFrSORfWY0YJsMpnwyiuv4ODBg+jr68MLL7yAvXv3Ytu2bbjzzjtx8uRJNDY24pFHHgEAPPLII2hsbMTJkydx5513Ytu2bZrOk2khk8ptmanSHMchGo3C4XCoA3qp6+PxuGoFZ056arFYpiWos8mHTNFONaqwUQu5esxoQWYYRp1IVBAECIIAhmHwyiuv4PrrrwcA3HzzzXjuuecAADt37sTNN98MALj++uvx8ssva7LGyDROxEIeGRlBV1dXzjC4cDisVnwrZAUTX7Esy3C5XKivr88qMKSHarksaC2L2qKWLWSKfma0IAPJh2XdunVoa2vD5Zdfjp6eHjQ0NKjFtLu6ujA6OgogGW5GkjF4nofD4YDb7S56DoPBoD6UiUQCiUQCdrs9SwRJvQnSSKSuHx0dhdVqVf8miRwej0e93ukIcrWs19lgIc82H3KtCvJsucfVZMYLMsdx6Ovrw8jICPbv34/jx49X9FzRaFS1gDNfhImJCRgMhrTtiRXsdrvTrGDysI6OjmLOnDnTFtTZFIdMLWTt0GzA2cWsCXtraGjAJZdcgtdffx0+nw+iKILneYyMjKgCOnfuXAwPD6OrqwuiKMLv96O5WXuRFZK1lyu2UpIkeDyeNEuCiKTT6URLS4vqpiCQdF+r1Qq3262ui8ViGB4eVqvLFarpQD7BYBCxWAxer7dg4D4JbyqF2TJwOJuohliSd4lSeWb0XXY6nTAYDGhoaEA0GsVLL72Ebdu24ZJLLsGzzz6LLVu24PHHH8c111wDALj66qvx+OOPY+PGjXj22Wdx6aWXahYnhmEwOTmZVYKTMDU1hdbW1jQXCBHk0dFRLF++HE6nMyvUbf78+WnbAkmr2W63o7m5OWcNh1zpvJFIBJFIBOPj43mztLTENxcS/Xg8DkEQ1IEkIvBaGw0tqbzVclnMFmrZQqYuC/3MaEEeHx/HzTffrIrV5s2bceWVV2LFihXYsmUL7rvvPpxzzjnYunUrAGDr1q248cYb0dvbi6amJuzYsUPzuaxWK0ZGRtJcEgQyuemqVauyBDmRSIDnedTV1aWJbqa1TaxnSZLg9/vR09OTNvFpMTweT1pdjVIolsrr8XgQCATQ2NiY1VCIoohEIqE7BTgTURQhiiLcbnfeim56P5nHiMViainRs1nnoRxQC3l2MaPv8po1a3DgwIGs5YsWLcL+/fuzlpvNZvz6178u6VwtLS2YmJjAnDlz0gaFWJaF3++H0WhUIygIxOe8bNky9W8SE03cG+Rl4jgOgiDA6XSiublZtziUw51QrBgQEd3U7MRyQxqWZcuWQVGUnD2DYp/MUqKZx0gkEggGg3j77bdLrvNQrKFgGAaJRAIjIyNFXUi5jqf196+WhUyjLKrDjBbkanL48GF87GMfU4WPPKDET93V1QXgTCQCCRETRVH1U2e6JUgcc2rM8vj4OBYvXlyTYW/ViLJIPVc5aj3kIhQK4d1338XatWs1bZ+rzkOxhkKSJDAMU7DONDkGOXbqMQuR2jiEQiEcOnRIk8iX2rugURbVgwqyBnw+H4LBIHp6erIeUIZh1NlEgDPCyLLJNGme59UHk6wjvlij0ZgmyPF4HEBylmu94jpbBtzO9iSnQkLEO/tPIh6JY8HKeWiZ21RS4yBJEkZGRrBgwYJyXTaA7MZh//79aWVECzUUWiYhyPWJRqMYHx/HyZMns66nUIW4JUuWwG63l/X7z3aoIGugoaEBW7ZsQSgUQktLC0RRhNFoBJDMvkt1MRDRNRgMGBsbyxkGR2pdeDyetMlNU4vW12IccrVE/2yx61d78MS/PQNRkMAwgChKOPey1bj9R5+FxZY9m3ghKhXrnNk4sCxb8TKiR44cwdy5c1Wjg1CshKjZbK7odc1GaHChRhobGxEMBrMG5mKxWNqDStYHg0EYjcY0q4pk9k1NTaG9vT3tWCybrDlbqn+WFheaHq///m/4+VeehBAXwXFJC8/A83jzpUP47mf+Q/c1zabkk3wui9S6x0ajEWazGRaLBVarFXa7nQ4ElgAVZI2Q9OnUdGmPx5NlBaSGupH4Z/IycxyHWCyWVmCICFwwGITBYCh58IS6LEpHURQ89e+/BRSAN5y5/wzLwGgy4NTBQbyz/13dx5ztglyM2fL9qwkVZI04HA6EQqG0FOfR0VE4HI40y5TnecTjcTU8LNVy5TgOkUhELSZOQt2AMzHVqegR2NRjVQoyUDnbcI144J3wgTdmW3QMw0CIi3jrfw7pOqaiKBWPfqgWtJZF9ZjxT8zw8DAuueQSrFixAitXrsSPfvQjAMADDzyAuXPnYt26dVi3bh2ef/55dZ/vfOc76O3txdKlS/Hiiy9qOo/NZlNdFqIoIhqNQhRFWCyWrAJDbrcb7e3tahgZEUoy4JI55VMikYAgCGl+wVJ8yJWmWrUsqj2op8gKUMSakyR931uW5VljIVJBrh4z3snD8zweeughnHvuuQgGg1i/fj0uv/xyAMCdd96Ju+++O237o0ePYseOHThy5AjGxsbw4Q9/GP39/UUfOLvdjnA4rFrIY2NjmDt3LhRFUWOLgaQwejwenHfeeQDSfbsk049A1pH6yePj4+q6WnyZZ6vLomVeE2wN9Qh6wzBkWMmKosBg4rHukpW6jlkNl0W17hN1WVSPGW8hd3Z24txzzwWQtGKXL1+uVnfLxc6dO7FlyxaYTCYsXLgQvb29OZNIMrHb7aoPWRAEuFwutLa25pz5w2AwqFEYZL2iKJiamsqKuhBFEZOTk+jo6Kh5d0C1BvWqDcuy2LztajURhaAoChKxBOb0dmDlxUt1HbNaglwt0aPiWh1mvCCnMjAwgAMHDmDDhg0AgO3bt2PNmjW49dZb4fV6AaSX4ATSy3MWgrgseJ5HIBBAa2trWuA8we/3p8VekvVerxeJW5NAAAAgAElEQVR2uz3twWZZFuFwGPX19WnxykCywJDH44HP50MgEEAoFEIkElHn7Ctl3r3pMlsGDnMJ2Ye2XIwt91wDRXlvNnFBhChIWHJeD+791f/W7RKqhlhWI0uvVKiAl8aMd1kQQqEQrrvuOvzwhz+E3W7H7bffjvvvvx8Mw+D+++/Hl7/8ZTz66KMlHz915ulAIIAlS5YAgFq0Hkhax6R2BSE19nj+/Pl455130tb5/X4sXrw47VyKouDo0aNoaWnJG+dJ/NGZ92DPnj3q31prQaRO315oeyCZPh2JRHKmCs/kl5BhGFz1+Y/ishs34dCuo4hH4li4ZgHmL59b0vGqJcjUtzu7mBWCLAgCrrvuOnz605/GJz7xCQBAe3u7uv5zn/scrrzySgBnSnASUstzFoJYyKQgDQl3y5yKiVRoI5AMvFgspg7mpb6siUQiK+De4/HAbrdj6dKlul64PXv2YOPGjaqVmSslt1AWF/l/oVTfWCyG/v7+rGMWsmoLZXNlfshs3kNDQ5qLBeVqHErFYqvDhVetL3l/QjWiLGrZQqaUxowXZEVRsHXrVixfvhx33XWXupxkwwHA7373O6xatQpAsgTnpz71Kdx1110YGxvDiRMncMEFFxQ9DxFkp9OZFnuc6iOemJhAb28vpqam0tZ7PB50dHSoViQR5EAgALPZnD7iryjqtevtuqceOzWjq5wB+nv27MG6det07aO1IJCiKIhGo+q1a6kDketTCOJ2icfjeOutt3SLvdbtqxFlUY359Ep1H83k3tLZZMYL8u7du/HLX/4Sq1evVoXi29/+Np5++mn09fWBYRh0d3fjJz/5CQBg5cqV2Lx5M1asWAGe5/Hwww9rskI5joPJZEIkEkmr6kZcFj6fDzabDSaTKc2nzLLJanCpFd/Ii+TxeNLSXlk2WXM4Go2q1rQeiBDUmtWU6vIoRiQSgcfjSfPzlwvSY/B6vRgbG8PixYs1i31mw0AakHwNA5nqy+/357yWfL0Gre4j8qwIglBwUoKivQZFBhAGwACozwr/oyFv1WXGC/IHPvCBnK34xz/+8bz73Hvvvbj33nt1n6u9vR0WiyUtzC01M2/+/Pk5Z5o2Go2qlZpaYCgzEYTjOExOTqrullIrvs3klNVKDuoRkSPFnOrq9NWn0EOh+tTFakBoLRAUiUQQj8cLTkqQv9egoM44AVvdAHg2OTmwpNgRERZDlFvUxoH0Wk6cOKGrh2A2m3PWDqcUZua+uVVGlmX89a9/xQ9+8AOcOnVKXU6SJWKxGOx2uxoBQfD5fGmF5olojo2NoaOjI8u9MTk5qVr6pIRjLusm1zKWZSGKYlZd5pnEbOnqFhrUK1dpUa/XC47jsGLFCv07i8fAiENQmJVQFCMURYYshQBlDHFlEWSmFZIkIRqNIhQKweFw5Ow1xOPxnA1IU1NTSb289ztUkDXy+uuvg2GYrAE4IDmoSLrYqbUuSNQFmYWarCfdzO7u7rRkEFIlzmg0IpFIqPWU85FpSZLzpb7oxcRci9iTv4n1mtrgFBLQUgXnbJbfnEnnKNk9pUTBSIcApg0Mw73npWDBcQ2AYoSBOQ7FsBBgGHW2m1zzSBaCujlKgwqyRi6++GKsWLECoVAobXmuqZgIExMTarlOAhnka2lpAc/zad3JeDyOzs5OdUAo3/x9uSCWear45yJV7EjXOd82meuIEAuCkPeY+dDaMJDueCKRKNgwaDk+4Wz41Gs6ykJ2AQoANodoMhYw8hQUJQgwdupDrjJUkHXgcDgQDAbTlpGpmDIfWhJ1sWTJEoyMjKjLOY6Dy+XCqlWr0mpWEBGyWq0QRTFteictEN9zJUWANCRafINaEkhShZ/8PxQKwWw2ZzUW5P+FfMxaGwbSzY7FYuqy1PW59sn1d6FtSeOVeh/K/duUPoCrvPfJt1YBkLxumjZdXagg64Akh6RO0zQ6OqpGVqQOpvn9ftTX16vTNBFIzK7FYsHUkAtv7jyMQ8+eAGdm0bzEDmFJ0vrU+xJMTEyooX2VYmpqSvMMGKWKj9frRVtbW9kGJnM1DKFQCHa7Xb3HuRqGzGMQgdHqTonFYlAURZ0FJpNSXEmZy0lPRRRFXb0GlnGAQR5JVgSAMQBMctyDWsjVZUYIciQSqfisCFog9SwMBgNEUYQsy+ogWqYgj42NYc6cOWk+ZeCMGPS9fBjP/fgFBAIBtM9phdvpgfyaAle/D5/61+t0vQQkdreSMzTIsoxQKFTxgRqv14ve3t6yHS9Xw+Dz+bBs2bKK9ibC4TDa2trUmial9hgy16c2DLFYLOv5ynfMdIzgpFawyhgUtiVlYxmsMgWJXQcFMoC42qCkRhYB+RsPEj8+OjqKRYsWFfvKlAxqJmD1vvvuw6FDh3I+QF/96lcLPnT5SnB6PB5cfvnlWLx4MS6//HK1noWiKPjSl76E3t5erFmzBm+99Zama0ytZ0EiJYjoZtYiJiPTJPuMnDccDkMISdj5Hy/A1mSFraUepnoTLA11aJ7TiGO7T+Lgq0c1XQ+BVIurJG63u6TZsPUQjUZhNBorapERIdMS8hb0hND3ymH8/uEX8Yf/8xKOvHYckUBU03kCgUBaTROtSSYkLC/Xx2AwpP0/Go3C4XDAYDDk/RiNxrQP2Z8xbgDDd4KHExxc4OACj36wCIDDCBgkx0pisRiMRqPaSGSG7EmSBFEU1Y8gCBgcHMQXv/jF0n6g9zk1I8iPPvoorrjiCmzfvh1DQ0Np6/70pz9l+W5TISU4jx49ir179+Lhhx/G0aNH8eCDD+Kyyy7DiRMncNlll+HBBx8EAPzxj3/EiRMncOLECfz0pz/F7bffrukaU+tZCIIAp9OZs+KbKIpobW3NCm3yeDyor6/HsddOQJEBoznpi41GozDX1YFhGFjsZuz53d803zdSRU7vKLheqnEOl8uFlpaW4htOg0AgoMnKd4168OKjr6L/zVPgTQawPIvDu9/Bi4/tQsCV/1kEzmQmVjoePBQKFR3EzURtALg6wLgJsvGjAL8cvPg/MAi/hUH6LYzCQ7BEN8Mk7UAslkxSKtRIZH5GR0fLPrnr+4WacVn09PRg+/bt+NrXvoaXX34ZW7duxcc//nFwHIfm5maEQiE0Njbm3Lezs1NNk04twblz507s2rULAHDzzTfjQx/6EL773e9i586duOmmm8AwDC688EL4fL60VOt8pFrIHo8HTU1NqkVDLHhFUSAIApqbm7P2HxsbQ3NzM1zDb8JoOTMwFo1G0dzcjFgsBomRMHxiBAcPHgLPF0/jjcfjYBgGPp+voOVFkiJKKQIkyzICgUBp8a46cLlcWLpUX5lLvXi9XjQ1NRXcRpZlvL7zDZgsRljsZ1xlprlN8LuCeOOFPlz2mQ/m3T8YDFbctUOs1Gn3JhgH+Pi/gFFOQ4EZyYw9QFEk8IlfoJ7zwmL5Z12HHBwcRHd39/Su631KzQgymSjxueeew2OPPYYHH3wQu3btwrZt21BfX49IJKLpOKklOCcnJ1WR7ejowOTkJID8JTiLCbLD4cDk5CQ4jsPU1JQ6iJZqIQcCgZzhaolEAvF4HDabDZbGOjhPedO+O8uyMJlMYBQGFmM9ent7itZ+kCQJbrcbdXV18Pv9mmpGFPJl5kvnFQQBiqKgv7+/oNDrqQOR2TCQ7MVKjxV4vV41jT0f7lEvwv4oWudlN6r2Ziucw24E3CHYm3Nbp36/Hw6HoyzXm49IJFKWTENG6gMrD0FRzOlp0wwHRZExt+F5sLy2HiRheHgYq1evnva1vR+pGUE2m82qW+KWW27BtddeizvuuAPbtm3Drl27VP9vITJLcKZSjvKQNpsN7777rupHI+KROrAyNjYGq9Wa5VMmfl6O49Bz/gIMvDECSUoKZH19vTpgE/JE8OGbNqVl9+VDURSMjo5i5cqVZRmgIhEgmbUa3n33XXR1dcFqteYUeDLAqaeIUGbDQHyQe/fuVRuGUoS+WCMQjUbVBjRftlw8kjsyAjjzHMWjcQD5BbnSFmIp7opcsOKbUJQEwBizVzIG8FwIwAQUdGk+5uDgIK666qppX9v7kZoRZIfDoVrBoiiisbERTzzxBJ577jmcOnVKHa3OR74SnMQVMT4+rvpASy3BabfbEQwGEQwG0ywgMsgniiKCwSAaGxtzCvI555yDRCKBxrl2XHTt+XjtN/sRSURhtzsQDcUQ8oXRtXgONl59nqZ75vV61YHDckCOw3GcGmusKAoikQhWr15dlvNMnJ7Cnp1/w8SpKbTNb8aFV5+HriWdeOedd9Dc3Izm5uaCdR4KfURRLFgHIh6PQxRFvP322+qyXIPIQVcIw8NDCAp+sAwLhj1TPY8Bg4AzhOGxIXgj7pzC7/P5EA6HVfHX0qjoNRbKFvHCcMRLkYWiSGAYQNY51DQ8PIyFCxdO/9reh9SMIH/xi19UrQoyGCJJEq699lp87GMfKxj/ma8E59VXX43HH38c99xzDx5//HFcc8016vLt27djy5Yt2LdvHxwOR1F3BXBmXr1IJJLmh+Q4DolEQrWCiTshFYvFAoPBoIrAR275EOrb6vDnZ/Yg5A3B1mjDFVsvxYaPnwOTRVstivHxcXUG60pBZkAphxi/+tRr+N2P/ghFVsAZOBx7vR9/fmYvPvZPl6BpjRW9vb1lq/OQi4GBARiNxqL3TFEUiOMMYpE46hssaZEF3kk/Fq9dhIWLu3OKviAIqiWup9dQiFwCTtwiWiq9FfoYcC4sylMAZIBJv+eKkoCkNIJhir8bqffO4/HkHEOhFKdmBPniiy/OWsZxHGRZLlosJ18JznvuuQebN2/GI488ggULFuCZZ54BkKwE9/zzz6O3txcWiwWPPfaYpmu02WxoaGiA3W5P63ITl4XH48GaNWswOTmZFqaXSCQwf/58AFDD4BiGQf1cEz7xrx9Da2srHA6HrjhiSZKqMtBWruiK028P4bc/fB4WuwW84cxAlCRK+O+fvIyP3PEBcBdWNgFBi/8YSLolLrr2fLzy1G54xn2od1gARUHIF0VDkx0fvHZjclkOXC4X2traymYh5ust9PX1Yf78+QV7E8SVVLhUqIQl7V2wmU9BVgwg5jIDEWBkHB/+MPwn9mmaZKCvrw/79+9HNBrFj3/8Y5jNZqxYsQKbNm0qy714P1AzgpxKZi0Fkkqcr1uXrwQnALz88stZyxiGwcMPP6z7uux2O44dO4Zbb701bZCR47hk6JrZrMbREquHdKOJT5isi0aT8awmk0n9fnogIWKVjAtWFAVut7ssiRqvPr0bDJg0MQYAjuegQMaxl9/FNTdN+zR5IRX5tA6EOVrt+Ng/XYqBI8MY7R8DGBbLL1qK+cvmwGjO7z4r94AecXekPh+iKMJgMOQsdFUSyv8BF/t/YRT/AgUcGChQUIdB1z+gs/sTWN7QUNSNJEkSFi9eDK/Xi76+PjQ3N6suIop2alKQM2sFkO5rNWfZzcXw8DAOHDighr8ReJ5HMBhU59kjccpA0sKsq6tTLWpSv2J8fBwdHR2Ix+MlhS+Nj4+XNaMtF8FgEFartSzug+FjozBZcgsZa2DgGfFN+xyFCAaDWQO9xTDXm7Dsgl4su0D7ffb7/ZrGI6ZDOBwuy4CeClMHse5+iLIbrHwCCsxQuJUY6z+E1XOTPYFcDUMmTU1NiEQi6O/vx4033li+63sfUTOCLIoiJiYmEAgE4PP54PP54PV64ff7MTg4iA0bNqiDdWeLI0eOwGw258zMSyQSapx05jx7Npsta3p5l8uF9evXY2RkBJOTk4jFYpr9fsTCTo0ayBdONh2m466IhmI4vu8Eju89CVlWkIgkkIgJMNZli7KQEGGfr08s9eL1evPGsZcLUruikinsQAXjnNlmyOwZ328ikdBdZH5oaIjGIE+DmhHk//qv/8Itt9yC3t5eKIoCq9UKm82mvkSV9pVqYevWrdi+fXtaIgiQfNmNRqMqhESQQ6GQmr6aKsiSJKGxsREcx6GtrU3t4moNKQsEAmBZFu++++6044wLjf6PjY2hu7sbY2NjeRNOci0PuEN45sGdCHrDqHfUAQygMIBzxA1jnSGtyy8kEpAFBX+3+cJy/1xpaPUfT4dq1VwJhUJpk/hWAlmWS4r+GB4ezjlLCkUbNSPI9fX1uO666/DrX/867zZn22VByLSQnU5nWposEWwScheJRNK2FwQBnZ2dakEii8Wiyy3wxhtvYO3atbotscw440I+wUgkAqPRCJZlNQ4OnfEl/uXx/fCNB2BtssAfSsb0WufUwXiax9DxUTTOdcBoNkASJESCMXT0tMC20Ixjx45p7iVoiU0mvQW9/uNSyaxfUSmCwSB6enoqeo5oNFrS/RocHCw4fRqlMDUjyA0NDRgbGwOQfBhS42BTLbuzDfGlEYENBoMwmUxqbV3gTNRFIBBAT0+PWn8XgCpuxI2h93tFo1GwLFtStzhXnHE+Tp06hQULFmgKB0zFPe7FXyNvYvGqnqzGs/O6DhzZ/U5SIMNx1DssWH5pL7b870/AZDEWFHkSUqY3NhlI9kgSiQT27t2bdi9K+RRqCMhAazwen1aMcSHIhAiVnq+uVGufuiymR00JMhGMSlsy0yHTOiYV31Ln2SNRFy0tZyaLJC6OyclJGI1GtYat3gI0WmpulAOXy4VzzjlH934hTxhMHhHiDBw6eztw9ec/gkVrFyAWj6G/vx8t7ZWNWc2MPy42yWhmVmFqjyCRSOTd3u12q2MhqcvzRQBpCSXLbAzIszcxMaF5n1LiuksRZBKV09raqms/yhlqRpB7enqwdetWjI+PIxqNYtGiRYjH44jFYprCiG699Vb84Q9/QFtbGw4fPgwAeOCBB/Czn/1MfUC+/e1vq92p73znO3jkkUfAcRx+/OMf46Mf/aim6yTZeuRF9fv9WLJkSZYgx2IxVTh5nlct6ImJCdTV1UEURdUloBVS2e2887Rl8pVKNBpVK3npxWKvU0Uop2UoK6iz1YHjOXjGPRWv7gZk+48rkXwiSRKCwaCuRkxLKFk+V1IsFtO1b6HEqlxCHgwGYbFYEAgENLmHRkZGMDExAUVR1MHvxsbGqvy+s4maEWS73Y5oNIqPfvSjmD9/Pu666y6sWrUKP/3pT3HfffcV3f+zn/0svvCFL+Cmm9KDWe+8807cfffdacuOHj2KHTt24MiRIxgbG8OHP/xh9Pf3awo9Sw15m5ycRFtbW5bwkGI8qbUuJElCOBxWp6AXRVG324G8JJUu60i+Vym0dDWhbX4LAq4A7C3pkQAhXwS2Zivm9CYHpKpR3a1a/uNSIh+0hJJlEo/H0dDQULb617l6C5Ik4ciRI5g3b56anFVswPnQoUN4+eWXEY1G8dBDDyEWi2Hjxo244447ynKd7xdqRpBjsRi+9a1v4dixYzh06BC2bduG1157DTt27NAkyJs2bcLAwICmc+3cuRNbtmyByWTCwoUL0dvbi/3792Pjxo1F9yUWMsMwGB8fT5s2iViF4+PjadYleaiJu8HlcpU0NU613BVOpxNr164taV+GYfCxf7oUOx58Dq4RD2zNVjAMg6A7CJbjcO0Xr1BD96pR3a2U+ONSCAQCFa/wBiQjLMrpEsjXW5BlWdeEBD09PVi8eDGeffZZ/Od//mfObWKxGDZt2qQmjFx//fX4+te/jtOnT2PLli1wu91Yv349fvnLX8JoNCIej+Omm27Cm2++iebmZvzqV7+a9f7psz9K9h4Wi0Wd8uXiiy9GIBDAxMTEtOu9bt++HWvWrMGtt96qVozLV35TC6RIPZB0RZC0biIyJMY41YoliSJut1v1K5MiRaFQCJFIBLFYDIlEQrU6MruY1aoRQOorFyvmVIjWec24+eubcf4V66BICkRBwrrLVuGmb3wSc3qTll0l44JH+sfxfx9+ET//ypPY8eBzCIyEC4YDloNqlNwEkkkhlW7EZFkGz8XAKD5AKVxnI5WhoSG1REAuTCYTXnnlFRw8eBB9fX144YUXsHfvXmzbtg133nknTp48icbGRjzyyCMAgEceeQSNjY04efIk7rzzTmzbtm3a363WqRkLmRSL/+QnP4lPfOITiMVi+NGPfoQrr7wSgiCU5M+8/fbbcf/994NhGNx///348pe/jEcffXRa12mz2RAIBFBfX5+WkUUqvpEXMxAIqOvIIF9TUxNYlkVTUxN8Ph9GR0eLRggQSHnK/fv3ax7I0Vq2MnW78fFxtLS0QJblaSWZ2Fts+OAnL8QHP5k7vpg0TuXm9Z1vYNeOPeCNPOpsZkyMTWLyhAtT/R5c+fmPgOMqY4OUPXsuB+Q3qWS0ESOPQAn/HkvnHAUf3wUwFkjcJij8hvR6yTkYGhpSs1VzHpth1HskCIJaiOmVV17BU089BSA5kcQDDzyA22+/HTt37sQDDzwAALj++uvxhS98oWZCXytFzQgycGZGjT179mDLli1YsGABtm7dWvLxUoPnP/e5z+HKK68EUHr5TeCMhWwymdJ8hsRPPD4+jgULFiAQCKgPDxHkZcuWqUkh7e3tul6sw4cPo6urCw6HI+9gULE44cwkk1zb+Xy+5IzYU1MFrcrpNgpOpxNNTU1qtbJ82+tpFCZOT2HXr15HY2cDeAMPKIDJakRHRzsO//U4Fq6djzWbyp9glEgkkvPUVVgowuGwpjrZpcLII+BiP0dCUCAzHVDYZkCJgRN/DxkByIaPFNx/aGio6OC4JElYv349Tp48if/1v/4Xenp60NDQoPYoU3urqT1ZnufhcDgq1pDXCjUlyE8++aT6f1K+8J133kE4HEYoFMKGDRuKVn5LJdXn+rvf/U7191599dX41Kc+hbvuugtjY2M4ceIELrjgAk3HJPHDqfUpgDNWcCwWg81mA8uykGVZFWpZlmG1WiEIgu7YYzKC73A4KlqeMpFI4ODBgzj//PMLblcobKxYwokoimnTThXbXk+j8Obv30YoFAQXeC8hREnuGwwEwRiBV371VzT3OHQ3IsUahWq5K0htkUrBCi9AYcyICwwMhveeL8YMBXPACrshc+cDbH4309DQUNEqdxzHoa+vDz6fD//wD/+A48ePl/MrzHhqSpC///3v49ChQ7BYLOA4DoODg2hpaUFDQwN8Ph+WLFmSd3T5H//xH7Fr1y64XC50dXXh61//Onbt2oW+vj4wDIPu7m785Cc/AQCsXLkSmzdvxooVK8DzPB5++GHNvmqHwwFBEGC329PSp3meh8vlUq+PuDA4jlMz+Yi46I2SIJOpVtoCI+cpxnQbBeJrTPXj6yVXo9D32+NoamtCndX83gzfIZjNZvAGHnU2BgFnUHNPQU+jEI/HwfM8wuFwSQklWrZnGAahUKhyYwhKAIw0BIXthCC4YLWmRIwwHMDIYKR3obC5Qy7J2InW6JyGhgZccskleP311+Hz+SCKInieT+utkp5sV1cXRFGE3++f9XWWa0qQY7FkosBVV12FRYsWwWg0or+/H5dffjl6enoKhhU9/fTTWcsKuTvuvfde3Hvvvbqv0ev14oknnsBjjz2WliBCMrXIwCQJbTMYDJicnITBYEjLOtTD+Ph4Qd9cuZiamqp4GBqQDHebbr2DXI1CW1crnINuGE3JAUl/wI8mhwM8zyMSiKJjfvu0GoFMSKNw4MABLFmyRJ2AQGtPQYubKTMG2eVyZT0/pYp86odn/XBwAmQmgXg8AasVavISwzBgFQZQ8k9tRQahCz3bTqdTLRsajUbx0ksvYdu2bbjkkkvw7LPPYsuWLVkTSTz++OPYuHEjnn32WVx66aWz2n8M1JggHzp0CN/73vfUYvU33HADNm/ejJaWFs0uhUrz5z//GZ2dnVkFhkiiBxl8JK6KYDCIuro61QWjN2okkUhAEISK+g6B5PUnEomKj+CTNOZKxAWvvWQl+l45DEmSwb0X9cLzPBRFQdAdwqY8A4ylQsQhkUhUfJZpAHj99ddx4YUXpolSvjhiLSKf3iiI4OtliJILgpCA3++Dopw5vol3YsjlRCSxN+u6nnnmGfz5z3+Gx+PBFVdcAbPZjI985CP4/Oc/n7bd+Pg4br75ZvW6Nm/ejCuvvBIrVqzAli1bcN999+Gcc85RDamtW7fixhtvRG9vL5qamrBjx47K3uAaoKYEub6+Hq+++iq6uroQjUYxMjKiZgoBZypQnU02bdqEAwcOpKVDA8n40NR4VyLIZObrd999V12uh8nJyYpX9gKS1ku1suYqFe42p7cdF117Pvb8bj9YIweOYRHyhRH2RbDk/EVY9UF91d5i4Tj8zgBMFhMa2nLHMlerwlsikUirKEgo6D5SFDDyO2DF1wElDoVfk/QDM7kjlljhGrCJ/0Y8waE91TUouwBmNRrmXZ01zRMAXHDBBdi9ezeefvppfO9730MsFssa6xkeHsYdd9yhjh/cdtttuOOOO9Kyaa1WK2655RZ13x/84Afqu/bNb35T7X3OZmpKkP/1X/8VX/nKV/Dqq69i3bp1OHToEP7+7/8eF110EYDC3aFqccUVV+C1115T06MBqANVqbG7JPaY+L6JIOv9DhMTE1izZk35vkAepqamKl5BDEi6KypV64BhGPzdDRsxf8Vc/OmpV+GfDKKxvQEfvnETll+4GByvrTFMxBL4nyf+ir+92AdFViBLMuYtm4O//+fL0dmT3jhWa0BP9yzTSgJ87Ltgpb8lY4kZFhBfgMI0Qaj7d4DNnldQ5i9EPDqBevMrYOQxJOumKlDYDkjGT+UUYwBquGRvb2/e35bneTz00EM499xzEQwGsX79elx++eUAyp9NO5OpKUHu7e3Fb3/7W7z99tvo6+vDbbfdpvo0ayX+kIS9kUE7IClmDocjzafMcRy8Xi9aWlqgKAo4jsPRo0fT/HjFQsMEQVDrKkcikaLRAaXeH1LwvtJuEQBqA1UpGIbBojULcIG4BitWrCipROmO7zyHE6eCQAoAACAASURBVG+eRmNHA3gDl5zh5bQTP9/2FG576DNoX3BGdAKBQFnmHCyG3tRsLvE4WHE/wDgA9sxzwShuGKJfg2D5SbbAMix88Q8iFJsPBy8BSEBh50Jhu/OKMWFoaKigBdvZ2alGPNlsNixfvrxgMtZ0smlnMjUlyADQ19eH3/72txgaGsLu3btx1VVXYdOmTVXx0WmB1LLInBWkq6srLRmE53lMTk5i9erVUBQFK1euVK3jYr49SZIgCAJcLhfMZjM8Ho+mqIB8kFH6fMJPZisZGBgoeVBISyhfJBKByWSqeE9HlpNV2UopUTpweBgn3zqNlq6mtKnEHC02eCf9+POOPdi87Rp1+0AgUPGptICkhax5QFKJghNeABhrdjIHYwcjT4GRDkLhswshRSIRmOrmQDbom818aGgIl156qaZtBwYGcODAAWzYsAG7d+/G9u3b8cQTT+C8887DQw89hMbGRoyOjuLCC8/4/PVk085kakqQjx49in/7t3/D+eefj87OTrz22mvo6OjA6OgobrvttprwIdfX1yMSiaiDeqRgUF1dnZqaDZyZ/cNisUAQhLQBPy2QeffWrl07rWJCWmKGT506hfb2dpjN5rT1xeoPZw4eFasolkgkwLIsDh48OC3hL7SeYZhpTXF0dPc7YPncE+o6Wmw4srsfkiiB4zn1+1e62BOgLykk6W5QACbfdYlgpVOQ8ghyKfVSilnIhFAohOuuuw4//OEPYbfbK5JNO5OpKUE+cOAAGhoacP/99+Opp56Cw+HANddcg/vvvx+33XZbwRceyF2C0+Px4IYbbsDAwAC6u7vxzDPPoLGxEYqi4I477sDzzz8Pi8WCX/ziFzj33HOLXmPmNE2kHnJmneRAIACr1aparnp9XyQ9e7ove7GYYWJNzp8/v2IuIdIg9PX1obe3F0ajUZPFnxoFUCwkjGwHQC0Qn1qQXuv0VeOjk4jH4wiHwgADsAwDEHeQkhxc83q8MNWZEA6HUVdXpzY0WpJIpnP/ND9DjBlQZCRFOde1sFDY3FEupQ5SOp3OohXoBEHAddddh09/+tPq/JiVyKadydSUIDscDrW0pcViwV/+8he0tLSohcWLPei5SnA++OCDuOyyy3DPPffgwQcfxIMPPojvfve7+OMf/4gTJ07gxIkT2LdvH26//Xbs27dP87XyPA9BEBCNRtHT0wNBENSoC0VREAgE0NraOq3Y43KVWCyEx+NBU1NTRf3z5NiCIFRlAOytt97K8h8TC77Y9FO953Tj5P4BMAwDBQokYvkrQCQQhb3DiknnZDKU7r1n9fDhw+nHkmSMHZvE6bdGIAkS5q7oQPc5c2GymEqy+kl2p9PpLLgdWQdmDhR2DhhlAkCGVa1IAFjIXO4w0lImNiX3ttAzpCgKtm7diuXLl+Ouu+5Sl1cim3YmU1OC3NHRob5E3d3dGBgYwNtvv41bbrkFAIpayLlKcO7cuRO7du0CkCxc8qEPfQjf/e53sXPnTtx0001qUSOfz6ervCXHcYjH42hra1NfBmKh+Xw+1NfX67ds3kOWZXi93qolg1SjpGc1Zn0Gkvcu18zPpEEs9ltc9DE7/vZ/30Y8Goe9+YzbQ0yIiPriuPb/+TiWr1wMIBk3v2jRorToh2gohp9/5UkMHhkBy7FgWBauE+/g1F9G8M/fvwkt85o0W/2SJKmx7BzHwe/353U7pX4URYHN/EEsm/sLQIlAks0AGLBMAiwnYNL/d5gIDINlR7PcPYIg4NSpU5rdRk6nUx3UDgQCMJvNOcPzdu/ejV/+8pdYvXo11q1bByA5YcTTTz9d9mzamUxNCfK5556Ln/3sZwCAdevW4Te/+U1aRhfHcbqjLUgcMJAU/MnJSQD5S3BqESeDwaAmbBDrPXOQr7W1FT6fT12nB4/Hg8bGxor7yxVFgd/vr8oswZUMd0tluhONmiwm3PzNzfivb/wGrlEPFDlpBPA8h7//5w9j+YWL1W1z+XV3/viPGDg8gsYOR9pzGvSE8OhXn8Y9//VF3W6okydPor29XWc0x4VgpHXg44+CEQ8DiALgILHnoM16NZrmLob8Xo+B9BzC4bDqf890HeVrQJ566ins27cPY2NjapXGa665Bl/5ylfSruYDH/hAToOq0ISo+bJpv/nNb+Kxxx7DhRdeiJ6eHixZsgTLli1DT08PmpqadNyj2qOmBJlhGExOTuL06dPweDyIRqPYv38/nE4nXC4XjEYjvvGNb0zr+OXomtvtdng8HgBQ/W3kuKIoIhQKobu7Gy6Xq2R3RaG6suXC6/WqBYsqTaXD3Qher3faL2Xb/BZ86Sefw6mDg3ANu2G2mrB4/SJYG86IL+nap967kC+MN//nbTja7Fn31NZkhWfch3cPDGDxen0JDqFQqCT/qcIth8R/GJziBRQWCmsHo8RhlrZD5j8IyZQeWywIAhobG3UJ/7e+9S385je/wcDAAO6///6c2wwPD+Omm27C5ORkWlJIqeM7n/3sZ9Vp0/r7+7Fnzx4MDAwgGAzC5XLhV7/6FT75yU/qvl+1QE0JssfjwerVq1XXhcVigd1uR0NDA2w2W0ki1d7erroixsfH1YdtOoMGNpsNExMTWZYOmfOOuDFI6q4eSORGNWa5mJqaqkoWYDgchtlsrkqEjNfrxYoV0y+xyXEsFp+7EIvPzV29LFdCiGfMC5Zh8tZcVmQFzmG3bkGORqMlhfAx0gA44XdQmA6Afe85ZKxQFBms+BfI3Aoo/BmhK3VAb3BwEAsWLMi7Pl9SyC9+8YuSxnfmzZuHG264Iee5IpFITeQrlEpNCXJzczMikUhZj0kKlNxzzz1ZhUu2b9+OLVu2YN++fXA4HJp9qXa7HYFAAHV1dVkulImJCaxYsUJNrX7jjTdyHiOfX47EBJ88ebJsYWC5UBSlan7qatWwnU78sV5yCbLZaoYs5R/gYlgGZqu+ayNV0EoRGVb8MwA+O/yNYaHACk74H4gZglxqyNumTZvyrs+XFFLq+A65v3v27MGf/vQnjIyM4Pvf/75agIm4EWciNSXI0yVXCc577rkHmzdvxiOPPIIFCxbgmWeeAZD0XT3//PPo7e2FxWLBY489pvk8DMPgr3/9Kz7+8Y+nWcGKooBlWZjNZgiCgPXr1+ccsc5XEEaWZRw7dgzz5s1LCw0j6zLLRhYLGysUG0zEq6+vT7fI62kgGIYpS3U3LUzXf6wHv9+flajROq8ZHYva4Rpxw9qY7lsW4gI4nsOyDfqSSHSnTKfAyMOAEgArTQDgoDDtUNhGAAwACxhlPG376VjIWutMpCaFlDq+wzAMJEnCN77xDVxyySV46aWXYLfbMTU1hU9/+tN4+eWXz3q+QqnMKkHOVYITAF5++eWsZQzD4OGHHy7pPG+88QY+85nPqMkhRJBFUcT8+fOLxh7niw0mhVeqEfXQ398Pu92OpqYmXSKvJVkkc1kkEsHBgwez7kE5GoHU5VNTU7BarWlxwZV4MRVFQSKRyCqgwzAMPnn3lfj/7vgFfFMB2JutYFgGYX8EiUgC1//LVbDY9FW5K1WQGXkIrLgHjOKDAgMABSzGocgNkPk1AGJQmPSol1KnStMaqZOZFJJ2vTrHd4aGhhAOh7Ft2zY899xz6jECgcCMFWNglglyNSDZd21tbWmRFYqiQBRFNDQ0QJZLm/usWpXdyISpPT09FQ8lcjqdOV0jqT0ELankmckiuda53W5YrVZ1Vu9i2YOFRL5YjRGO4+B2u7PWtS1qwe3/8Vm8/MRfceS1d6AoChYs78JHbv1QWoSGVoLBoP4GWhHBR/8dCoxIvuI8AAYKAEbxghVPQGEdkAzXqruQZ1ava4Tc42LPer6kkFLHdxKJBJYtW4af/exnqovqrbfeqloPqVJQQdYJy7L47Gc/q6ZMk2QQj8eT5mYoJcNuYmJCjdGsJKFQSJ2VpdK43e6co/ZE8DiOK8kqy0SWZezbtw/r16/XtH2+dPJiDYQoivD5fOA4rmCNkTWfWIwVVy1MplkbOPjhxt697rRr0NIDINFFxPLT0nsw4CAMsgdAAwAfAB+SMstCgRGMMgSJvR4yv0G9lkgkUlKNarfbXTSxKF9SyHTGdxYtWoSrrroKjz76KBwOB+6++24cOHAA99xzj+7vUEtQQS4Bh8OBwcHBrNjj+vp6tW6FXrELh8MwGAxpJTwrBYkEqTTVHDgMBAK6sgBTGwS9hMNhzJ8/f1qJLpnjCLkaAUmSMDExgfr6+qweQqEi9M31+zHHEQDPTYJhEoDCg2EkMIwMBlEIsgl9JxcgJh5Is/olScLhw4c1u4gOHjyIkydPor6+HgcOHIDZbEZbW1vWNEv5kkKmM75jMBhw9dVXo6mpCS+99BJ6e3tx9913VyW7tZJQQS4Bm82GUCikRlKQFGoy314pvsuJiYmqPUwul6tgmFK5IFZXtcLdqpEJCCTFf7qNjJZ5CaPRKGw2m+7nghWmwEd/D0AGg8xBOgUGNoo1q+ZA5s9TRXxoaAg8z6OlpSVvD0GSpDSX0RtvvIGDBw/C6XTi4YcfRiwWw5VXXoktW7aknTFfUghQ+vjO1NQUnnzySUSjUcydOxednZ0zPikEAGau9/ssYrfbEQwG1ZrIREyJxazX6lIUBU6nsypWazgchslkqkqFMpfLVbUp26slyKRHVA13T6kDejJ3HhiEwSDXNYoA6sDJR8CyLHieh9FohCAIaGhogNVqhcPhQGNjI5qbm9Ha2or29nZ0dnaiq6sL8+fPR3d3NxYtWoSvfvWr2LhxI26//Xb8/Oc/x5NPPpklxrfeeiva2trUGhUA8MADD2Du3LlYt24d1q1bh+eff15d953vfAe9vb1YunQpXnzxxayrJ/f/3nvvxZtvvglBEOB0OnHXXXfhX/7lXxAOh3Xfr1pi1lvI3d3dsNls4Dju/2/v3IOjLO89/nk3m4SAmEQREpMDCSwJIVwEm6Yd0EFpMNpCwYmA2qnKRUGtaE+gdqi32gIitCRTHE+Ll+lIycwRj0FsA0K5eGmKFgQVkSTkSgghyWYTctmwu8/5Y3lfsslusteXAM9nZmey7+572+x+n9/ze34XjEYjX3zxhccMIW9Ri9R3b9M0adIkamtrsdls/dbc6InFYuG6667T5Ueul7sCnP7FYCRp9Iee8ceBlPb0Fb9D3pQYhBKLIsyAwPkzF8AFwIDDkIgiml128TfkraqqSuvo4w53Bb/A/y4h6t/FxcV89dVX2vbVq1czadIkGhsbdWm0ECquekEG2Ldvn4ul5qkCnLeoReqNRiNms5nIyEgiIiIYPHgw1dXV1NXVeRRld+Fe58+fJyoqihMnTngV7uWNf8/TIsu5c+eYMqV3HdxgozYz1UMk9Y4/1qNiHTjF36/BU1FwhKVjsFeh0ACiAwChxCIMiSiiHUeYawaiP1XewBmDnJzsPpsR3Bf88kR/XUKEEGzZsoWYmBhSUlL4wx/+QFZWFrGxsYSFhTF06NArOikErhFB7omnDCFvUTP1wsLCaGlpITk5GYfDQWxsLCNGjOjTL9hzIcdms3H06FHGjBmjvebOh9dX/K+nil89cTgcdHZ2askgvoi9twOB+lDLeuqBnv5ji8WiW7NNfyMfABzhszE4/geHMsGZB4ICGJzNTpUw7MZLmXWqm82fbMC6ujq/6mz40yWkpaWFY8eOERkZSWxsLAUFBZw6dQqr1cqBAwfIzs7WxRUXSq7sq/cCRVGYNWsWiqLw2GOP8eijj3rMEPIWdVHPYDDQ0dGhJVd4E3vcM9xL7fash9VVVVUFOOM8e4p4Xws5akspX8LDOjo6MBqNWhEmFW8z/HyZJTQ0NJCamqrFB4eiSLyKL507AsFutweU2OIwTsfuKMVwYR8IAyjhKKoYRywHwyXLu6Ojwy/hVwd+X11t/nYJiY6O5ne/+x0Wi4W2tjYMBgOVlZVYLBbS09N1mY2FmqtekD/55BMSEhKor68nKyuLceNcW8H78+ONioqio6OD1tZWrUdc94w9Xzhz5owuEQ/g9B9PnDjR73AvbxFCUFxcTGZmpoug9JUy3pfl31cyiNoEtrS01OU9nghkNmC321EUhebm5n7dRYEOCAELv6Jgj3gYh3E6BtsnKMKM3ZCEw3gbGFzLoPpriZvNZmJiYny+10C6hERHRxMdHY3ZbObLL78kLi6O9PR0hg0b1itz8krkqhdk9R86fPhw5s2bx6FDhzxmCHmL+oNraGggPDxcs459FTmbzUZ7e7su/k+r1Qqgy5dWXSDqad15E+rlK83NzRiNRq8WD72J/XW3Xa0H3NraisHgTNHu7xie8HYgOH/+PA6Hg5qaGp9mDi5WtaIgwsZiD+s7QzBUVd484W+XEIfD2VPzgw8+oLi4mE8//ZSuri7CwsKoq6tj1apVLF261OfrGUhc1YLc1taGw+Fg6NChtLW1sXv3bp5//nmPGUK+oPbLUyMt/JleqhEPepQLPHfunC4F4sEZ7tYzOSBU+OI/DnRAKC0tJS4uzu8oFV+yAy0Wi9Z1pqury2t3kZow4omeIq6ugwwZMkRzw3njQvr222+1VOXTp08zaNAghg4d2iuxyV3Br/379/vVJUS9r3fffZfs7Gxqa2uZNm0aSUlJvPXWW15naQ5krmpBPnv2LPPmzQOc1ugDDzxAdnY2GRkZbjOEfEHNylO/JP64AOrq6nq5UEJFfX29LhXXQL9wN3CmrKenp+tyLncV3nzBl+zA6upqkpKSgpq56W5AsNvtfPPNN1qjXneC391lpG7fsmULVVVVdHV1sXz5cjo7O1m2bJlWp0LFXcGvxYsXe7xGT11C4FITiMrKSu644w7+/ve/ExcXx49+9CPy8vI4f/58AJ/OwOCqFuTRo0f3qjIGzrrL7jKEvEUIwb///W+SkpI0n5evVq7VasVut/s1VfQVtQGrvyv2vqAuAOqxwKJGn+hxLk8V3kKFmoIfTDwNCEIIhg0b5tN3+I033iA3N5f58+czY8YMt+8Jdhd4dWYze/ZsDAYDmZmZHDhwAIfDQUVFBYmJiT58GgMTmannByUlJQwePBij0aitMg/kVGk93RV6hrvpGX+sV3QFOAdrPWqawKVoDn/cZv35kB9++GGKiopctqk5ACUlJcycOZN169YBuHQJ+fOf/8zy5cs9HvfRRx8lOjqaRYsW0dXVRX5+PmvWrNEtHDGUXNUWcqhISUkhMzOTlpYWwsPD+c9//qO95m241unTp0lISKCmpsbrmGD1ua8/nvr6esaO9b30oz80NDToUkIULjWD1QM9xT+QovS+4m/IGzgX5/qySkPVBb57pmReXh4Ar776KrNnz/brPgYSUpD9JDo6GovFwqRJk7RprLqK76lyl/p3e3s74eHhREZGatNuq9XqccGm53NvswDVqWlzczPV1dVeJ4L0Vye4r7ZQzc3NpKamhuZD74HZbNYtM8tiseg2o9FTkP0NeVP90b6GegbSBb67D1ztGhIZGclnn33G1q1bWblypc/3MdCQguwn3bP1VLxdxS8tLSUpKSnoP3B3q+7nzp0jLCyMESNG9Bnj6008cH8hXUIIrFYrR48e9Tvet78BQh0M9PQfQ3AqvHlLa2urbrHp/oa8NTc3B9yx3Nd47Z4+cHUwqK+vD2ixdSAhBdlPuocJ+YJa2a2v/H9/cRd6Z7FYSE5ODnlBHCEEFRUVGAwG4uLivI7zVUO6+ptVqM9VbDYbNpuN4uJi4NLswBfXj7ezAnVGEsz46b7Q01/d3t5OTEyMz/tVVlbq2gX+pZde4qWXXiItLY2bb76ZMWPGMHbsWGbMmMGuXbtISkry+VoGItecIBcVFbFixQrsdjtLlizxu8NAdXU1RUVFFBYWMmjQIO0RGRnp8ndUVJTLtrKyMk6ePOmyT8/9g/XDVxNP9Jj+KoqihaDpEYlw6tQpBg8erM0yvInT9ZTw4c2gYbVaXVrR98TfIlDu1gfsdjtWq7XXe0IRr97R0eF3Uog/Iuhvl5AXXniBxYsXU1lZybfffsuJEyc4ePAgW7du5dixY7z++us+X8tA5JoSZLvdzhNPPMFHH31EYmIiGRkZzJkzx6+Y2VdeeQWLxQJAZ2eny6Ojo8Pl7+bmZm3bzp07MZlMvP32273eqz66uwXUH6EQQvM7R0VF9RJ/d4NBZWUl586do7a21uX1noNE92P6m1Ld3t5OW1ubbi4Es9nsYkH5k5jjLRUVFURERHj0V/uT/ae6XHpu6+zsxG63c/LkyV77esJX0e/+6Ojo0Fxv/b23O1VVVf0KcrC7wCcmJpKYmMi0adP6+Y9duVxTgnzo0CFMJpMWHrNw4UIKCwv9EuT4+Hi/ukN3dnaybt06v4rY22w2j8Lvbtv27dv5wQ9+QE1NTb/vVR9qj0AVRVG00L6+BL2pqYmmpiY+//xz7bW+Bg93g0h4eLhXVmBnZyfl5eW6ZWa1tLT0GVIVzP6AdXV1REdHe+3S6i8dvK/u4er6gdls7ndWobptLBYLq1atwmq1EhkZyeuvv86gQYNYt25dL6EMRRd49X7VRyjS8S8n15Qgu1vJ7WsaGgpeffVVv/ZTFIXw8HDCw8O99gdbLBaefPLJoHxZbTYbVqvVo6C/8sorzJgxA5PJ5PK6xWLxejDo6upye9+Ai3ALIWhsbGT//v29xL/78/5mAz23exoMDhw4wMSJEwP+DL3h/PnzPvcG9FeQzp8/T3t7u88hkXfddRcLFixg06ZNJCUlYbVa+x2IgtUoIpSV/AYC15QgX2s89dRTQTuW0WjEaDR6XGxqamri7rvvDkoH6Z44HA6XwWDjxo3ExcVx55139ukqamxsdCv87gYEq9XqsYa02WzmH//4h0dRdyf+fQ0I7rargrp9+3aWLVsW9M/QHf5GWADU1tYycuRIwsLCvD6Gv40iurq6aGxsJD4+XqvdrFrHVxvXlCB7U95P4h9z5swJ2bENBgNRUVFavOySJUuIi4vzKzrAV95//32Ki4v59a9/7bWrSJ0ZnD171uvZgeojrqysZMeOHZqrKCIiol9B728twZPbaMeOHYwaNYoxY8b49JmosfaBFoP3tlHEl19+yX333cfWrVuZPn06cGnmdOjQIZqamsjOzg7oWgYKio/933xrFjfAsNlspKSksHfvXhISEsjIyOBvf/ubbsVpJFcebW1tWK1WXdLBW1pamD17NgcOHAC8Xzfovr3nwNDXYFBaWsoNN9yghRP2tDgNBoOLoHd37RQXF1NaWur1vSUnJxMbG+vSKCImJobm5mbtXmNjY7XnPdm7dy+/+c1vmDt3Lk8//TR79uzh448/5uOPP2b69Ok+dfy5THhlzl9TFrLRaORPf/oTd911F3a7nUWLFkkxlvTJkCFDdIsJvv7669mzZ4/23J91A1948MEHyc/P91gq1dO6QWtrKwsWLPDpXIE2ipg5cybR0dGsWrWKd955h/T0dMaPH09hYaFunc314JqykCUSyeXnxRdf5LrrruMvf/kL+/fv15JEZsyYwXfffdfr/Varlffee4/8/HzS0tKoqKjAZDKxYcMG3eqLBAGvLOSrI1ZEIpEMWNra2mhtbdX+3r17NxMmTNCSRIA+G0XMnTuXDz74gA0bNpCXl8c777xDZGQkv/zlL7WynlcL0kKWSCQh5dSpU70aRaxevZrGxkbmz59PVVWVliTizlff1NTksqir8txzz3HgwAHeffddv7u46IhXFrIUZIlEcsVy8OBBpk2bFtKmvUFCuiwkEsnAoqioiNTUVEwmk1acPhBuv/32K0GMvUZayBKJRBfsdjspKSkutWS2bdumW//Fy4y0kCUSycChey2ZiIgIrZaM5BJSkCUSiS546goiuYQU5CCyaNEihg8fzoQJE3q9tnHjRhRFoaGhAXBmJj311FOYTCYmTZrE4cOH9b5ciUQywJCCHETcddkFZzH73bt3u3RY8KXLrjvcif/KlSsZN24ckyZNYt68eS5pqGvXrsVkMpGamsquXbv8uDuJJDBkLZn+kYIcRG6//Xa3cZTPPPMM69evd0kN9dRl11vciX9WVhZff/01x44dIyUlhbVr1wJw/PhxCgoK+OabbygqKuLxxx93aYfUH+7Ev6mpiaysLMaOHUtWVhZmsxmQlr/EMxkZGZSUlFBeXk5XVxcFBQUhLUp1JSIFOcQUFhaSkJDA5MmTXbYH6k9zJ/6zZs3SKnCphenVa1i4cCGRkZEkJydjMpk4dOiQ1+dyJ/5q6cSSkhJmzpyphTAFavn35I9//CPp6elMmDCB+++/XytOn5mZiclkYsGCBW7rKPtDc3MzOTk5jBs3jrS0NP71r395HHgkvtO9lkxaWhrz58+XtWR6IAU5hLS3t7NmzRp++9vf6n7uN998k7vvvhsIjfgXFhby0EMPAc7Sie+//762PRDLvzunT58mPz+fL774gq+//hq73U5BQQG/+tWveOaZZygtLSU2NpY33njDr+P3ZMWKFWRnZ3PixAmOHj1KWlqax4EnWNjtdqZMmcJPfvITgJANNgOFe+65h5MnT1JWVsbq1asv9+UMOKQgh5CysjLKy8uZPHkySUlJ1NTUMHXqVOrq6kLqT/v973+P0WjkwQcfDMrx3HH27FmthVVcXBxnz54Fgr+SbrPZ6Ojo0Bq2xsfH889//pOcnBzAdTAIBIvFwsGDB1m8eDEAERERxMTEeBx4gkVeXh5paWna81AMNtXV1dxxxx2MHz+e9PR08vLyAM9uJ8nlQwpyCJk4cSL19fVUVFRQUVFBYmIihw8fJi4ujjlz5vDXv/4VIQTFxcW9uuz6y9tvv83OnTvZunWr5rMO9WJKqNrqJCQkkJuby8iRI4mPjyc6Oppbb72VmJgYzTUTrNCp8vJybrrpJh555BGmTJnCkiVLaGtr8zjwBIOamho+/PBDlixZAjj976EYbIxGIxs3buT48eMUFxezbFVL4wAABdNJREFUefNmjh8/HnLrX+I7UpCDyP33388Pf/hDvvvuOxITE/u0bu655x5Gjx6NyWRi6dKlvPbaawGfv6ioiPXr17Njxw6Xtjpz5syhoKAAq9VKeXk5JSUlfP/73w/oXCNGjNBcEWfOnNGKuwRT/M1mM4WFhZSXl1NbW0tbW5vbKJZgYLPZOHz4MMuXL+fIkSMMGTKkl0AFe+B5+umnWb9+vda+qbGxMSSDTXx8PFOnTgVg6NChpKWlcfr06ZBb/xLfkYIcRLZt28aZM2e4cOECNTU12vRXpaKiQiumrXbZLSsr46uvvuJ73/ueT+dyJ/5PPvkkra2tZGVlccstt2i92dLT05k/fz7jx48nOzubzZs3B5z/76l0YjAt/z179pCcnMxNN91EeHg49957L59++inNzc1ad+xgWftqi/nMzEwAcnJyOHz4sMeBJ1B27tzJ8OHDdeucrVJRUcGRI0fIzMwMqfUv8ZOebbX7eUiuQRYuXCji4uKE0WgUCQkJYsuWLaKhoUHceeedwmQyiZkzZ4rGxkYhhBAOh0M8/vjjYvTo0WLChAni888/9/u8xcXFYvz48aKtrU04HA7x85//XOTn54ucnByxbds2IYQQjz32mNi8eXNQ7nP69OnixIkTQgghXnjhBZGbmytyc3PF2rVrhRBCrF27VqxcuTIo53r22WdFQkKCGDVqlBgxYoSIiooSDzzwgLjxxhvFhQsXhBBCfPbZZ2LWrFlBOZ8QQrS2toqpU6eK7du3CyGEiI6Odnk9JiYmaOeS9MIrjZWCLBnQPP/88yI1NVWkp6eLn/3sZ6Kzs1OUlZWJjIwMMWbMGJGTkyM6OzuDcq4jR46IW2+9VUycOFH89Kc/FU1NTR4HnmCyb98+8eMf/1gIIUI22HR1dYlZs2aJjRs3attSUlJEbW2tEEKI2tpakZKSEpRzSdzilcbKam8SyWVm//79bNiwgZ07d3Lq1CkWLlxIU1MTU6ZM0bpjBIIQgoceeogbbriBTZs2adtXrlzJjTfeyLPPPsu6detoampi/fr1gd6OxD2yQL3kykEIEZJIDYmzwehtt93GxIkTtQXENWvWkJmZ6VXHDklQkIIsGTgsWrSIKVOmsGzZMsLDwykvL+fll18mNzdXq4f73HPPMXv27F4RIA6HQ/tbjXQwm83U19czcuTIXq19JJIBSEgEWSLxC0VRhgE7gV8IIT5XFGUnsBfYLITouvieL4EXhRD9xl8pinI38AiwQgjRZyqgoihhQgj7xXPmCSE+CvR+JJJQYLzcFyC5+lEURRFCNCiK8iFwl6IomcBZoEAI0aUoilEIYQM6gOYe+8YCOcBQoBXYLoRoAuzAOSBdUZThQJ0Q4uzFfQxAJGATQlzg0sxuBF5aKhLJ5UAKsiTkiIvTMCHEy4qi7AYmAzO7WbaqT2IQYO2x+5CLrzuAW4BwRVFexymyqcCTwI1AvaIoTwgh6oDZwDLgZkVRTgK/AOpwxt23h+YuJZLAkYkhEr3ZB2wFyty8ZgQ6u28QQtQAx4Eq4ASwFIgFbMAY4AkhxG1ANDBDUZRROMX4F0KIyUAJsODi4Qz0FnyJZMAgLWSJ3liBBi5ZxXDJpWDE6YpAUZRwIcQFRVHmAvfhdFe0AcOACzgt5z0XtwH8H5CEU6z/C3hPUZQunEK9q9vxXQRfIhlISEGW6I0RpwBrFfKF68pyycVtFy4+Xwq8LYT4X0VREoC5F/dVgJFcckGMwmlFdwK7hBD/rR5QURT1ex6BFGTJAEYKskRvuvuEexIHFCqKUg904XRv7AXmKoqSCIwG4i++NhSYAmQpinIGyAIWAWeAlYqivAV8C1wHXA+oFY+ky0IyYJGCLNGbtUCYEMKdIN8LJAKDLz6agUKcohqP0/Vw4aIr4wywCVgIJAOvCSGOACiKsgJ4E6dF7ACewinIMTjdHRLJgETGIUuuGRRFiQZahPzSSwYoUpAlEolkgCDD3iQSiWSA8P82PmsCMb1spQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.scatter(x_1[0], x_1[1], x_1[2], c=label, s=60)\n",
    "ax.view_init(30, 185)\n",
    "ax.set_xlabel('X Label')\n",
    "ax.set_ylabel('Y Label')\n",
    "ax.set_zlabel('Z Label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/18_kernel_income_v4_files/image010.png)\n",
    "\n",
    "We see an improvement but if we change the orientation of the plot, it\n",
    "is clear that the dataset is now separable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VNX9+P/XncyShQQIEIgJBJIJaxLWENwAgYBQDSCWRRQQi4pr3dBvEdFWBa1aP/0BRREVbCVFK4Yqq6h1hUAgLsHKCEG2ECD7OpOZOb8/0lyJBAhwA5Pk/Xw88niQc2/OvO9l5p2Tc899X00phRBCiEvPdKkDEEIIUU0SshBC+AhJyEII4SMkIQshhI+QhCyEED5CErIQQvgISchCCOEjJCELIYSPkIQshBA+wnyO+8ttfUIIce60+uwkI2QhhPARkpCFEMJHSEIWQggfIQlZCCF8hCRkIYTwEZKQhRDCR0hCFkIIHyEJWQghfIQkZCGE8BGSkIUQwkdIQhZCCB8hCVkIIXyEJGQhhPARkpCFEMJHSEIWQggfIQlZCCF8hCRkIYTwEZKQhRDCR0hCFkIIHyEJWQghfIQkZCGE8BGSkIUQwkdIQhZCCB8hCVkIIXyEJGQhhPARkpCFEMJHSEIWQggfIQlZCCF8hCRkIYTwEZKQhRDCR0hCFkIIHyEJWQghfIQkZCGE8BGSkIUQwkdIQhZCCB8hCVkIIXyEJGQhhPARkpCFEMJHSEIWQggfIQnZQF6vF6XUpQ5DCNFISUI2iFKKkSNH4nA48Hq9lzocIUQjZL7UATQl+fn5HDx4EI/HQ0xMDBaLBU3TLnVYQohGQkbIBklLS8PPzw+lFEeOHKG8vJw9e/bIaFkIUW9NLiHPnDmTsLAw4uLi9Lb8/HySk5OJjY0lOTmZgoICoHqa4b777sNut5OQkMDOnTv1n1mxYgWxsbHExsayYsUKvT0jI4P4+Hjsdjv33XefPmc8btw4OnbsSEVFBVVVVWiaxqFDh/jvf/+L0+mUuWUhxFk1uYQ8Y8YMNmzYUKtt4cKFDB8+HIfDwfDhw1m4cCEA69evx+Fw4HA4ePXVV5k9ezZQncCfeuoptm3bRnp6Ok899ZSexGfPns2yZcv0nzv5tfbv309gYCBer5fMzEy8Xi9Hjx5l7969lJWV4fF4LtJZEEI0Rk0uIQ8ePJjQ0NBabWlpaUyfPh2A6dOn8/777+vt06ZNQ9M0Bg0aRGFhITk5OWzcuJHk5GRCQ0Np3bo1ycnJbNiwgZycHIqLixk0aBCapjFt2jS9rxqapuHv709sbCyVlZU4nU5ycnJwu938+OOPMloWQpxWs7iol5ubS3h4OAAdOnQgNzcXgMOHD9OxY0d9v8jISA4fPnzG9sjIyFPa6xIcHExgYCBVVVWUl5dTWFjI0aNHAYiKisJkqv5deODAATp27Gjoxb+ysjLKy8tp166dYX1C44oV4NChQ7X+vy5UQ57XyMhI/T1hhIsVq8lkwmazGfoazVmzSMgn0zTtoq180DQNq9WK2WzmyJEjVFRUcPjw4VoJOScnh06dOhn6YaysrKSoqIj27dsb1ic0rlihOiF36tTJsP6cTieFhYWGx3r06FHDz2tDxtqxY0c9VrlobaxmkZDbt29PTk4O4eHh5OTkEBYWBkBERAQHDx7U9zt06BARERFERETw6aef1mofOnQoERERHDp06JT9z8ZkMhEfH8/XX39NZWUlWVlZ+hu6vLyc77//3tBfEjVTJU6n07A+oXHFClBRUcF3331nWH81sbpcLsP6hIY9rw0Ra1ZWlh5rly5dCAgIMPQ1mrNmkZBTUlJYsWIFjz32GCtWrGDs2LF6+6JFi5g8eTLbtm2jZcuWhIeHM2rUKP7whz/oF/I2bdrEggULCA0NJSQkhK1bt5KUlMTKlSu599576x2H2WzGZDLRqVMn/Pz8ACgqKiIqKsrQD2NBQQHFxcVERUUZ1ic0rlgBCgsLDe23sLDQ8D4B/fiNPK8NGevJo3mr1Wpo/81dk0vIU6ZM4dNPP+XEiRNERkby1FNP8dhjjzFx4kSWL19OVFQUq1evBmDMmDGsW7cOu91OYGAgb7zxBgChoaHMmzePxMREAJ544gn9QuGSJUuYMWMGFRUVjB49mtGjR59TfJqmERwcjNlcfer9/PwIDg42/M9Vp9NJSEiIYX1C44oVquM1sl+Xy0VFRUWjOK8NHWvNgEKmLIzV5BLyqlWr6mzfsmXLKW2aprF48eI69585cyYzZ848pX3AgAF8//33FxakEELUocktexNCiMZKErIQQvgISchCCOEjJCELIYSPkIQshBA+QhKyQfbt2ydLgIQQF0QSskFKSkqIioqioqJCqroJIc5Lk03IP/74I3369NG/QkJCePnll3nyySeJiIjQ29etW6f/zIIFC7Db7XTr1o2NGzfq7Rs2bKBbt27Y7Xa9dOev9e7dm4MHD2K1WnG5XOzcuRO32y2V3YQQ9dbkbgyp0a1bNzIzMwHweDxEREQwfvx43njjDR544AEefvjhWvvv3r2b1NRUsrKyOHLkCCNGjGDPnj0A3H333WzevJnIyEgSExNJSUmhZ8+edb6un58fAQEBdO/enYyMDJxOJ1arVaYzhBBn1WQT8sm2bNlCTEzMGe/rT0tLY/LkydhsNrp06YLdbic9PR0Au91OdHQ0AJMnTyYtLe20CblGYGAg/v7+eL1eXC4XO3bsoKqqSr9lWgghfq1ZZIfU1FSmTJmif79o0SJWrlzJgAEDePHFF2ndujWHDx9m0KBB+j4n1zr+dW3kbdu21fu1TSYT/v7+9OnTh23btukVyGoKyZSXl/Ptt99e6CHWUlPlq7y83NB+G1OsUB1vzV9JRnC5XDidTioqKgzrE6prFxt9Xhs61pr3r91ul2pvBmryCdnlcrF27VoWLFgAVD+Cad68eWiaxrx583jooYd4/fXXGzwOi8WCzWbDYrEQHR2tj5RLS0uJiYkxtLBMfn4+RUVFdOnSxbA+oXHFCtUXWmNjYw3rr6CggMLCQsNjLSsrM/y8NnSsNcWFpNqbsZp8Ql6/fj39+vXTC3WfXLB71qxZXHfddcDpayMDp20/H5qmERQUVKvaW1BQkKEfxvLycioqKggKCjKsT2hcscIv8RqloqICi8XSKM5rQ8cq1d4aRpNdZVFj1apVtaYrcnJy9H+vWbNGfzp1SkoKqampOJ1OsrOzcTgcDBw4kMTERBwOB9nZ2bhcLlJTU0lJSbnoxyGEaPqa9Ai5rKyMzZs388orr+htc+bMITMzE03T6Ny5s76tV69eTJw4kZ49e2I2m1m8eLE+Cli0aBGjRo3C4/Ewc+ZMevXqdUmORwjRtDXphBwUFEReXl6ttrfeeuu0+8+dO5e5c+ee0j5mzBjGjBljeHxCCHGyJj9lIYQQjYUkZCGE8BGSkIUQwkdIQhZCCB8hCVkIIXyEJGSDvP/++9hsNqqqqqTCmxDivDT5hNy5c2fi4+Pp06cPAwYMAKpv101OTiY2Npbk5GQKCgoAUEpx3333YbfbSUhIYOfOnXo/K1asIDY2ltjYWFasWHHK64wdO1YvJlRRUcHOnTtxuVxyJ5MQot6afEIG+OSTT8jMzGTHjh0ALFy4kOHDh+NwOBg+fLhe43j9+vU4HA4cDgevvvoqs2fPBqoT+FNPPcW2bdtIT0/nqaee0pN4DU3TKCoqwmazERgYqN88UllZSVlZGdnZ2Xg8Hhk9CyFOq0nfGHI6aWlpfPrppwBMnz6doUOH8txzz5GWlsa0adPQNI1BgwZRWFhITk4On376KcnJyYSGhgKQnJzMhg0bat2S/Ws2mw2r1YrVakUpRWBgoD5izszM1OsWlJWVsWvXLr16lhGcTidVVVWUlpYa1ic0rlihuk7GyX/lXCiXy4XL5aKsrMywPqG6aJPR57UhY6250xUgNjZWqr0ZqMknZE3TGDlyJJqmcccdd3D77beTm5tLeHg4AB06dCA3NxeAw4cPn1Jq8/Dhw6dtP5cY2rdvz4EDB/B6vXTv3l2/LXvnzp306NHD0A9jQ1VQa0yxAmRkZNCjRw/D+isoKKCgoECvjW2UXbt2GX5eGzLW7t276wMKi8ViaP/NXZNPyF988QUREREcO3aM5ORkunfvXmu7pmmGfhDORtM0/P399WpvNfWSjaz0ZbVaMZvNho9cGlOsUB2vkf2WlZU1mvPa0LFKtbeG0eTnkGtKZYaFhTF+/HjS09Np3769XvUtJyeHsLAwfd+6Sm2eqTSnEEIYpUkn5LKyMkpKSvR/b9q0ibi4OFJSUvSVEitWrGDs2LFAdQnOlStXopRi69attGzZkvDwcEaNGsWmTZv0PwM3bdrEqFGjLtlxCSGapiY9ZZGbm8v48eMBcLvd3HTTTVx77bUkJiYyceJEli9fTlRUFKtXrwaqq7qtW7cOu91OYGAgb7zxBgChoaHMmzePxMREAJ544gn9Ap8QQhilSSfk6Ohovvnmm1Pa27Rpw5YtW05p1zSNxYsX19nXzJkzmTlzpuExCiFEjSY9ZSGEEI2JJGQhhPARkpCFEMJHSEIWQggfIQlZCCF8hCRkgxw8eBCbzSbFg4QQ583nl70FBwfrtzbXJDtN01BKoWkaxcXFlzI83bFjxwgJCaGiogKlFDt37qSyshKTyYTJZNLbhRDidLRzTBKNJqMcPHiQadOmkZubi6Zp3H777dx///08+eSTLFu2jHbt2gHw7LPPMmbMGAAWLFjA8uXL8fPz469//at+N96GDRu4//778Xg8/O53v+Oxxx6r8zX79evHCy+8AECfPn3IyMjA6/Xi9XoJCQkhPz8fgFatWum/ZIqKimjZsqWhx15VVUVVVRWBgYGG9tuYYgUoLCykVatWhvUn5/XUWLt3707btm0NfY0mql4Fc3x+hHyyL774AofDwa233sqJEycoKSk5bZUws9nMiy++SL9+/SgpKaF///4kJycD8MADD/Dwww/X2n/37t2kpqaSlZXFkSNHGDFiBHv27AHg7rvvZvPmzURGRpKYmEhKSgo9e/Y8Y6wWiwU/Pz+9CEtCQgLbt2/H6/USFxenFxfavn078fHxhhaWycvLo7CwkJiYGMP6hMYVK0B6ejq9e/c2rL+8vDwKCgqw2+2G9QmwY8eOBjmvDRVrXFyc/r42MmbRiBLyU089xY4dO/jxxx+59dZbcblc3HzzzXz55Zd17h8eHq6X2AwODqZHjx5nLJmZlpbG5MmTsdlsdOnSBbvdTnp6OgB2u10vYzh58mTS0tLOmpBPR9M0vcIZVL+hrVaroW9ss9mMn58fVqvVsD6hccUKv8RrlJpfso3hvDZ0rFLtrWE0ml9va9asYe3atQQFBQFw2WWX6YWDzmb//v3s2rWLpKQkABYtWkRCQgIzZ87Un/zRULWQhRCivhpNQrZarbVqF9f3SQilpaVMmDCBl19+mZCQEGbPns3evXvJzMwkPDychx56qCHDFkKIems0CXnixInccccdFBYWsmzZMkaMGMGsWbPO+DNVVVVMmDCBqVOncsMNNwDQvn17/Pz8MJlMzJo1S5+WkFrIQohLrdHMIT/88MNs3ryZkJAQ9uzZwx//+Ef9Il1dlFLcdttt9OjRgwcffFBvz8nJ0eeW16xZQ1xcHFBdC/mmm27iwQcf5MiRIzgcDgYOHIhSCofDQXZ2NhEREaSmpvL222837MEKIZqlRpOQAeLj46moqEDTNOLj48+475dffslbb71FfHw8ffr0AaqXuK1atUp/SGPnzp155ZVXAOjVqxcTJ06kZ8+emM1mFi9erF+4WLRoEaNGjcLj8TBz5kz9idJCCGGkRpOQX3vtNf74xz8ybNgwlFLce++9PPHEE6etUXzVVVfVeSNGzZrjusydO5e5c+fW+TNn+jkhhDBCo0nIf/7zn9m1axdt2rQBqtdZXnHFFVI0XtRJKYWnyqPf0Xk2xXkl5B0poH2XdlhtFqqcbqwBVvz86r7MopQi73A+Xq+iTUToaferi9frZe+u/ZQVlxPZ9bJa2wqPFZG7/zjWACudekaeU78NwV3lJuvLH/lp1378A63Q2g2JlzSkJq3RJOQ2bdoQHBysfx8cHKwnZ+F7tKrtmCpeRfP8AJqGMsXgtaagTPH4ld2JxjEAwv2gbWgQWsV8lC0RTBd211d5SQX/eukDNq/4D6VFZbzWajWjfzeM8b//DbaAU9fk7tmxlz/PWELOvlz9PtRW7VuSdF0/WrUNIWFIT3pd1Q2z5ZePyjefZrHm5XXkHy1E0zSCWgUyZtZwrhw/8KzJf+dH37H0929SVlSO5mfCU+UmMr4D0cvsbHr9U775TxYmkwmlFIEhgUx48DfEX93jgs7J+Tp24AQv3/4qBUcL8Xg8ADidTo7syGPms1PwM/tdkriaMp9PyC+99BJQfXNGUlISY8eORdM00tLSSEhIuMTRibqYKv6GqXI54Ky+X1QBnl34VfwAlAMn3UeqgcVUBpVz8Ko78FqTwdztvF63stzJvOsWcujHHDQ/E2arGWeFizV/3UDmJ7v507/nYLFZ9P33ffMzc4b/CbfLXaufwtwiPv7750x46Hq2frCTYwdOMPyWqwH44XMHH736Jf5B/oS0ra6z4ix3kbrgfcqLKxg5Y+hp49v91R5emLEYj8dbncy8Cs1kYu+OAzxw5RN06NKO1h1a6TeIVJY5WTHvn8z68y10SzT+TsYz8Xi8/N+dyyjILaRF6yC9vbiomO3rdxHWqS1j77n2osbUHPj8sreSkhJKSkqIiYlh3Lhx+ghk7Nixp71tWlxC7v9iqnwdrSYZ/4+Gl1OScc22/zWYnG+huXeBt+i8XvqjlZ9x2HEUP4uf/qe+n9kPP7OJn7MO8tk7W2vt///dvRx3lbuurqhyuvn2P7tp37kt2d/+TM7eXDxuDx+/+RWBIYEEtPDX34u2QCvBbYJZt2wLZUXlp43v7398F3eVB7PFrP9szdr60oJStP8VoqrhH2TDGmDlw1c+uuiFqX74eg/5OQW0aBVUq10zaQSGBLDlrc9wVbouakzNgc+PkOfPn3+pQxDnwOR8G3DWue3sM7nloMrB8zOYzv2vn01vflrnnLGmaXg8Hja8/gnDb64e6VaUVrLv25/PWC7rp537uHJcIrYgG3sy9tHysha4yl20bH1qISCzxQ/l9fLj9r30G3HqCqDKskp+2pWN2XrqR87r8eL1KvJzCgjtULsYUlDLQI78dJSywvJaI9WGtj/rIG6Xp85tZquZ8ooKThzOp0OXsIsWU3Pg8wm5xvHjx3n++efJysqisrJSb//4448vYVTi1zTPfrTzKAqoUZ0bNZygSs6rrGBpYdlp60FopupRaA1XhQvlPfOreNzVdRrMVjOVZU6CnP6/DOfroBRUOavq3Ob1eM86v3zaeDQu+gjZP9CGdpq/n5VSeL0K60nTP8IYjSYhT506lUmTJvHBBx+wdOlSVqxYoZfQvFD1La95Nv369dNv6d6+fXut27tP/n7btm21bgHfunXrqZ1dALfbjdvt5sSJE4b2W59Yu4VbCWsJpnoVG/xFTb7J/vkoReUhlFaeYwdAcPtACo8X6aNQpRRVVdUJ0u10E9Q+gK+++gqoTpAWfzPOstP/2e0fYiM7O5viY6VorTrh5/LgcrrIz8vH9KvVD0opyksrOF6eq7/Gr7cHhwVScKQYP0vti2Emkwk0D36BJvLy8mptc5a5CGzpz7c/fFOv1SI1at4Dv+6vvqpaVOB0OikuKkY76T/T4/GQd6yA1h1C+PHn/9LR25HY2Njzeg1xqkaTkPPy8rjtttv4v//7P4YMGcKQIUNITLzw9Tcej+e8ymvWZefOndx0000AJCYmsn37dn1bzfdKKZKSkvRqb19//TVJSUmGVvo6fvw4BQUFdO3a1bA+oX6xau5gtJJbqWvaomaMV2da0QBa0LlzD5R1NJjOvT5w4PwQnrtlEWgaJpNGVVUVFosFr9eLX6AfM5+YSs8rfjknjjsO8a+/fHDaaYurxyfRtl0bWvpXknLzdVS6K/hvcjZ7vsimZWjLWg9OKDpeTN8hCVz329Gnjc/8rD8v37EMQE/oSincLjf+ATZatWxJ69BfamW7KqugsoRb/t8k4q88t5UWJ06cIC8vj27dzu8CKUDhD+V8supL/Fv4Y/W3VC/1y80nwN+f2c/fStcBMVLtzWA+f1GvhsVS/edReHg4H374Ibt27dILvl+I9PR0vbym1WrVy2uK86PM8XhtE1HYarejwf/aTs5/Sv0yOvbabkGZ+55XMgboMyyOlHtGgVJUOavwVHn0KYSJc1JqJWOAaX+cSO+hver8DRE7oAu2ABuVZU5GzRyqX9waduuVxA/pSfGJEvKPFpJ/tIDiEyV0juvIrc9MPmN8Sdf155Ynf4tmqn7ijcddPUfbOqIlz26ci71fFwqOFlZ/5RbiLHfy20dSLtmyt0mPjeW3D1+Pn9lEaVE5pQVlhHZsxf2v3k7XARd31Udz0WhGyI8//jhFRUW8+OKL3HvvvRQXF/Pyyy9fcL91ldfctm3bBffbnHkDHkGZEzFVvILm3QeAMkXhtf0GZbocv/IHQR3Q96/yhmBqMR9l7Q+mC1tbPuX/jefy6wew6c1P2b3zR+ISezDq1qF07H5qQSg/PxNPf/gY33yaxTvPryX/aCEduoRx1Q1JtGgdROsOLenUIxL/wF9+uVhsZm7/880cduTww1YHXo8Xe98uRPeOqteUwphZwxk66XIyNn1LWXEFUT0jKVQniOkTRWy/6RzNPsbR/cew2qzE9O1c59rpi8VkMpE8fQjDpl5Ffk4hFpuZ/+77gdh+srqpoTSahHzdddcB0LJlSz755BMAQxKyaACahrJeg8d6TZ2bPdYP9X/r0yttjZte6RzXkdtfuIWvvvqKK6644iyhavS5Jo4+18TVu39N04jsetkpd9nVV2BIIFffOEj/fuvWX+Z5O3QJ87mVC35mP9p1/N8vyn2XNpamrtFMWdSl5qaRCyHlNYUQvqJRJ2QjlgIlJibq5TVdLhepqamkpKQYEJ0QQpybRjNlUZdzWQZ0OmazWcprCiF8gs8n5ODg4DoTr1KKiooKQ15DymsKIXyBzyfk+j7IVAghGrtGPYcshBBNiSRkIYTwEZKQhRDCR2jnuHTs4pacAjZu3MioUaPq3PbOO+/w29/+9iJHdHr9+vXjhRdeAKqX06Wnp/+vMpaXTp06sW/fPrxeL0FBv5RRLCsrq/W9EdxuNx6PB5vNdvadz0FjihWgtLSUFi1aGNZfTcEef39/w/qEhjuvFyPWXr16ERbmWzey+Kh6LQnz+Yt6Y8aMYfDgwfz9738/5YaNBQsW+ExCzs3NpU2bNjidTrxeLzt27KC8vBxN0/RiPGZzdWHyxMTEJltc6Fw1VKwAX331FZdffrlh/RlRsKcuW7duZeDAgYae14aMNTExUX8iuxQXMpbPT1kkJCRw0003MWjQIN59991a2y52jdgz2bdvH06nE5PJhNVqpW/fvgQFBREYGIi/vz8RERGYzWZMJpP+lAj5kq+m8CWM4/MJWdM0Zs2axZYtW3juuee49dZbKS8v17fV1yOPPEL37t1JSEhg/PjxFBYWArB//34CAgLo06cPffr04c4779R/JiMjg/j4eOx2O/fdd5/+CyA/P5/k5GRiY2NJTk6moKCAyy+/nNLSUiwWC35+fvoIQggh6svnE3KNrl278vXXX9O+fXv69u17zhXZkpOT+f777/n222/p2rUrCxYs0LfFxMSQmZlJZmYmS5cu1dtnz57NsmXLcDgcOBwONmzYAMDChQsZPnw4DoeD4cOHs3DhQmMOUgjRrPl8Qj55WsJsNrNw4UJeeeUVpkyZgsPhqHc/I0eO1OdtBw0axKFDh864f05ODsXFxQwaNAhN05g2bRrvv/8+AGlpaUyfPh2A6dOn6+1CCHEhfD4h1/WQ06FDh5KRkcHcuXPPq8/XX3+d0aN/ebJDdnY2ffv2ZciQIXz++edAdZ3kyMhIfZ/IyEgOHz4MVF/ACw8PB6BDhw7k5uaeVxxCCHEyn19lMW7cuDrbW7dufcqz70aMGMHRo0dP2feZZ55h7Nix+r/NZjNTp04Fqp9AcuDAAdq0aUNGRgbjxo0jKyur3vHJhQ0hhFF8PiGfi48++uiM2998800++OADtmzZoidRm82mr4Ht378/MTEx7Nmzh4iIiFrTGifXSW7fvj05OTmEh4eTk5Mj6zCFEIbw+SkLo2zYsIHnn3+etWvXEhgYqLcfP34cj6f62Wb79u3D4XAQHR1NeHg4ISEhbN26FaUUK1eu1EfZKSkprFixAoAVK1bo7UIIcSGa1Aj5TO655x6cTifJyclA9YW9pUuX8tlnn/HEE09gsVgwmUwsXbqU0NBQAJYsWcKMGTOoqKhg9OjR+rzzY489xsSJE1m+fDlRUVGsXr36kh2XEKLpaDYJ+aeffqqzfcKECUyYMKHObQMGDOD7778/pb1NmzZs2bLF0PiEEKLZTFkIIYSvk4QshBA+otlMWVxqLpeLqqoqqqqq2LZtm77Ko7S0lK+//trQpXM1lb5OnDhhWJ/QuGKF6ni/+uorw/qriTUvL8+wPqG6glpDndeGiHXr1q16rL169SIgIMDQ12jOJCE3IK/Xq38wvvvuO7xeLzabjaSkJKn29j8NXe3tiiuuMKw/qfYm1d4amkxZGMzj8eB0Otm+fTuVlZUA+Pv7079/f2w2m6EfOiFE0yLZwSAfffQRISEhuFwuTCYTffv2JTAwEKvVKklYCFEvMmVhkEGDBlFYWKjPp9VMSQghRH01m6Hbk08+SUREhF73eN26dfq2BQsWYLfb6datGxs3btTbN2zYQLdu3bDb7bVKbGZnZ5OUlITdbmfSpEm4XC5atGghI2EhxAVpVhnkgQce0OsejxkzBoDdu3eTmppKVlYWGzZs4K677sLj8eDxeLj77rtZv349u3fvZtWqVezevRuARx99lAceeICffvqJ1q1bs3z58kt5WEKIJqJZJeS6pKWlMXnyZGw2G126dMFut5Oenk56ejp2u53o6GisViuTJ08mLS2LLJO3AAAZYElEQVQNpRQff/wxN954IyD1kIUQxmlWCXnRokUkJCQwc+ZMCgoKgOq6xx07dtT3qal7fLr2vLw8WrVqpc8Rn1wnWQghLkSTSsgjRowgLi7ulK+0tDRmz57N3r17yczMJDw8nIceeuhShyuEELU0qaUAZ6uHXGPWrFlcd911AERERHDw4EF928l1j+tqb9OmDYWFhbjdbsxmc639hRDiQjSpEfKZ5OTk6P9es2YNcXFxQHVt49TUVJxOJ9nZ2TgcDgYOHEhiYiIOh4Ps7GxcLhepqamkpKSgaRrXXHMN7777LiD1kIUQxmlSI+QzmTNnDpmZmWiaRufOnXnllVeA6nvxJ06cSM+ePTGbzSxevFi/LXTRokWMGjUKj8fDzJkz6dWrFwDPPfcckydP5vHHH6dv377cdtttl+y4hBBNR7NJyG+99dZpt82dO7fOB6aOGTNGXx53sujoaNLT0w2NTwghmk1C9gUejweXy1Wr2ltN9SwjNVQFtcYUKzSuam8NdV6l2lvjIgm5gSml8Hg8ZGZm4nK5sFgsUu3tJFLtTaq9iV80m4t6F5tSiqqqKsrLy6mqqiI2NpaAgAD9jSyEEL8mI2SD1STiHTt24PF4CAgIwGQyERQUdKlDE0L4OBkhG6SkpITo6GjKy8tRStGnTx/8/f2l4JAQot5khGyQ9PR0Dhw4QGBgIJqmYbFYLnVIQohGptkM3yZNmqSX3uzcuTN9+vQBYP/+/QQEBOjb7rzzTv1nMjIyiI+Px263c99996GUAiA/P5/k5GRiY2NJTk6moKCA4cOH43a7DX0umhCieWk2Cfmf//ynXnpzwoQJ3HDDDfq2mJgYfdvSpUv19tmzZ7Ns2TIcDgcOh4MNGzYAsHDhQoYPH47D4WD48OG1aiULIcT5ajYJuYZSitWrVzNlypQz7peTk0NxcTGDBg1C0zSmTZuml9lMS0tj+vTpgJTfFEIYp9kl5M8//5z27dsTGxurt2VnZ9O3b1+GDBnC559/DlSX5YyMjNT3ObnMZm5uLuHh4QB06NCB3Nzci3gEQoimqkld1BsxYgRHjx49pf2ZZ57RCwCtWrWq1ug4PDycAwcO0KZNGzIyMhg3bhxZWVn1fk1N02TeWAhhiCaVkM9WftPtdvPee++RkZGht9lsNmw2GwD9+/cnJiaGPXv2EBERwaFDh/T9Ti6z2b59e3JycggPDycnJ4ewsLAGOBohRHPTrKYsPvroI7p3715rKuL48eN4PB4A9u3bh8PhIDo6mvDwcEJCQti6dStKKVauXKmPslNSUlixYgUg5TeFEMZpUiPks0lNTT3lYt5nn33GE088gcViwWQysXTpUkJDQwFYsmQJM2bMoKKigtGjRzN69GgAHnvsMSZOnMjy5cuJiopi9erVF/1YhBBNj1aztraezmnn5qZfv3688MILACQmJrJ9+3Z9W2JiItu2bcPpdBIQEFCr2pvRt1XXVPry9/c3tN/GFCtUV3tr0aKFYf3Jea2OtebmJ6iu9iZTdvVSrwtNzWqEfKkopTh8+DAVFRVYrVap9nYSqfYm1d7ELyQhNzCPx4PT6aS8vFwfGbvdbrxeL263m6qqKlwul6EfxoqKCr32slFqyogaHWtVVZXhsdZQShnab0PFqpSioqLC0Nvt3W43TqezQWItKSkhMDCQI0eO8M477/Doo49KqQCDSEJuIDWJ2OPxYLPZiImJoaCgQN+el5fH3r17iYyM1JOdEfLz8zlw4AC9evUyrE+Ao0ePEhQUZGisUH0eWrdubWifNYyONSgoiH379hkea4cOHdizZw/du3c3rM/g4GAOHjzIvn376NSpk2FLM7t06cL333/PZZddRmhoKOXl5Vx11VUsXryYAQMGGPIazZnMIRuoZg7Z7Xbj5+dHVVUVZrMZTdPo27cv3377LSUlJfqfeUavYVZKoZRq9v3W8Hq9hlfb83q9jeY81PQLxr7XTu73pZdeYseOHbjdbpRS+Pv707VrV9q2bauXGhBAPeeQJSEbaPDgwTz66KMAJCUl8c033wDg5+eH1+vF4/FQWVmJxWLBYrEY+uFzuVy43e5aFwyNoJSivLwcf39/w4vrN+QFPWiYi2U10xZGx1xznmvqZxvZr8vlwuv14u/vb+h7o2ZaxGq1Mm/ePE6cOEFJSQlt2rShRYsWkpRrq9eJb1brkBvSxx9/DIDFYqlVB9nPz0+fvqhZYWG1Wg0drVRWVurF8I0eaTqdTiwWS4M86aSqqqrRzT2azWY8Hg/nOJA5K03T8Pf3p6KiwtC+NU3DZrPh5+dneN9ms5nAwEA8Hg/z58/n1VdfJSoqioKCAg4cOMCxY8e49tprDXu95kASskH69+9PUVGRnrhMJhNms1l/jJOmaQ0y+qmsrNQ/zEYn46qqKpRSDZI0lVINMqXQ0DRN03/JGs3Pzw+LxYLT6TS8b6vVisVioby83NCVETXvPYvFQkVFBc888wxRUVEEBwdz4MAB9u7dy4ABAyQx11Pj+jScg3feeYdevXphMpnYsWNHrW0LFizAbrfTrVs3Nm7cqLdv2LCBbt26Ybfba5XUzM7OJikpCbvdzqRJk/Qr106nk0mTJmG32xk5ciRVVVVA9Qdr27Zt7NmzB6fTib+/v6GjYqj+c/znn3/Gz88Pm81meDLOzc2lsLCwQRI9VNeh9ng8jbIOiNlsxuFwNEjfFosFj8fDvn37GqRvm81GUVGRXijLKDWjZbfbzX333Ue7du0IDQ2lpKSE7Oxsjh49Kkm5HprsHPIPP/yAyWTijjvu4IUXXtCvAO/evZspU6aQnp7OkSNHGDFiBHv27AGga9eubN68mcjISBITE1m1ahU9e/Zk4sSJ3HDDDUyePJk777yT3r17M3v2bJYsWcK3337L0qVLSU1NZfr06XTp0gWlFKWlpbRr1w6n09kgazWLi4u57LLLKC0tNbxvqH4kVVhYGBUVFQ3Sf2lpKa1atcLtdjdI/1A9j79t2zbD+635/23RokWD/EKxWq2cOHGCkJAQw/sGaNmyJQcOHKBly5YN0n+7du3Ys2cPrVu35tChQ/pF7p49ewI017nl5n1jSI8ePepsT0tLY/LkydhsNrp06YLdbic9PR0Au91OdHQ0AJMnTyYtLY0ePXrw8ccf8/bbbwPV9Y+ffPJJZs+eTVpaGk8++SQAN954I/fccw8//PBDoxz1CSEuvSY7ZXE6hw8fpmPHjvr3NXWOT9eel5dHq1at9DvrTq6LfPLPmM1mWrZsSV5e3kU8GiFEU9KoR8j1qX8shBCNRaNOyGerf1yXiIgIDh48qH9/cp3jutrbtGlDYWEhbrcbs9lca/+aviIjI3G73RQVFdGmTZsLPCohxK+5XC6sVuulDqPBNbspi5SUFFJTU3E6nWRnZ+NwOBg4cCCJiYk4HA6ys7NxuVykpqaSkpKCpmlcc801vPvuu0Dt+scn10V+9913GTZsmMwfC/E/lZWVDBw4kN69e9OrVy/mz58P1G/VUlJSEvv379f7mj59OmvWrLkUh3Fx1dwGWc+vRuO9995TERERymq1qrCwMDVy5Eh929NPP62io6NV165d1bp16/T2Dz/8UMXGxqro6Gj19NNP6+179+5ViYmJKiYmRt14442qsrJSKaVURUWFuvHGG1VMTIxKTExUe/fuvXgHKISP83q9qqSkRB04cEANHjxY+fv7q86dO6s+ffqoVatWqfnz56vAwEAVERGhevfurWbPnq3uuOMOpZRSkyZNUkFBQapr165qw4YN6p577lGffPLJpT2gC1OvHNtkl70JIXxDTk4OX3/9NdOnT6d169YcPHiQOXPmEBAQwOHDh3nvvfcIDQ0lPz+ft99+m44dOzJ58mQcDgft2rUjNzeXgQMH8sADD3Ddddfx5Zdf8tFHH/HMM89c6kM7F3LrtBDi0vJ4PIwePZpbbrmFCRMmcNVVVwHVRYmOHz/Od999p5enLS4u5qabbuL555+nX79+aJrGLbfcgqZpfPXVV6xbt46hQ4fyj3/8o8HWaF9qMkIWQjS4wsJCxo8fz9ChQ/nTn/4EVC8VdTqd+Pn50a9fP06cOEFOTg42m02/pT4sLIyAgAAyMzOB6gvpa9eupV+/fnrfjeQWfBkhCyF8Q6tWrbjyyitZtmwZNpsNpRSpqakAeu2L/fv3U1lZidlspqioiMLCQqqqqvjhhx+A6nrUDz/8MLNnz9bvfq1Jxkop1q9ff8mOzyiSkIUQDeL48eMUFhYC1bf6L168mOHDh2M2m2ndujV9+vQBoHv37vz4448opTCZTNx+++0EBAQAEBISoteIMZvNfPrppxQWFpKbmwugr2qaOHEizz///MU+ROPV9+qfamSrLC6V+fPnq8suu0z17t1b9e7dW3344Yf6tmeffVbFxMToV47Px8MPP6y6deum4uPj1bhx41RBQYFSSqns7Gzl7++vv27N1erzsX79etW1a1cVExOjFixYcN791Dhw4IAaOnSo6tGjh+rZs6d6+eWXlVJnPlfnKioqSsXFxanevXur/v37K6WUysvLUyNGjFB2u12NGDFC5efnn1ff//3vf/UYe/furYKDg9Vf/vKXC4r/1ltvVe3atVO9evXS204Xr9frVffee6+KiYlR8fHxKiMj47z6N/q9U9drnHxOunXrpqKjo1V8fLxq2bKlioyMVAEBAapVq1bKarWqyMhIpWma2r59u3rhhRcUoCwWiwoKClJ+fn5K0zRlMpmUzWZTAQEBKigoSLVr104NGzZMbd++XVVVVSmllNq8ebMKCwtTJSUlSimlPB5PveK/yOqVYyUhG2z+/Pnqz3/+8yntWVlZKiEhQVVWVqp9+/ap6Oho5Xa7z7n/jRs36m/EOXPmqDlz5iilqj9UJ38wzpfb7VbR0dFq7969yul0qoSEBJWVlXVBfR45ckRPIsXFxSo2NlZlZWWd9lydj6ioKHX8+PFabY888oj+C2XBggX6uboQbrdbtW/fXu3fv/+C4v/Pf/6jMjIyav2fnS7eDz/8UF177bXK6/Wqr7/+Wg0cOPC8+jf6vVPXa9R1Tj7//HMFKJPJpNq0aaNiYmKU2WxWY8eOVX5+fqpDhw7q+uuvV4Bq06aN8vf3V4MGDVJUX7NS3bt3VxaLRUVGRiqr1aqGDRumtm3bpvcfFRWlli9frpRS+vH5oHrlWJmyuEjOVNToXIwcOVKvqzFo0CAOHTpkaJzp6el6kSWr1aoXWboQ4eHh+kWY4OBgevToYXj5x7qkpaUxffp0oPrGgvfff/+C+9yyZQsxMTFERUVdUD+DBw8mNDS0Vtvp4k1LS2PatGlomsagQYMoLCwkJyfnnPs3+r1T12vU5aqrruL222/H6/Vy2WWXYbVa8Xq99OrVC7vdjt1u54cffiAgIIDbbruNUaNGUVhYqN+Zl5uby7hx4/TKgyffLXvPPfcQExPDzJkzAep8kEJjejK2JOQGsGjRIhISEpg5c6b+YNPTFS+6EK+//jqjR4/Wv8/OzqZv374MGTKEzz///Lz6bIg4T7Z//3527dpFUlISUPe5Oh+apjFy5Ej69+/Pq6++ClR/kMPDw4HqB4nWzDteiNTUVKZMmaJ/b1T8cPp4G8t7p0Zd58RisfDWW2/Rq1cvCgoKUEqxZMkSSkpK6Ny5M/7+/gQFBfHvf/+bv/3tb0RERGC327HZbFgsFj788ENuvPFGrFYrISEhREREsH37dt5++21WrlwJoNfXrqys5J133uG9994Dqh8W0RAPFGgIkpDPw4gRI4iLizvlKy0tjdmzZ7N3714yMzMJDw/noYceMrT/Gs888wxms5mpU6cC1aPQAwcOsGvXLl566SVuuukmiouLDTtmI5SWljJhwgRefvllQkJCDDlXNb744gt27tzJ+vXrWbx4MZ999lmt7UY85NPlcrF27Vp++9vfAhga/6811INfoWHfO2c7J6tWrSInJ4fOnTsDcOzYMT755BNmz57N9u3bCQ8P5+qrr2bHjh1MmTKFDh068Pvf/54ZM2awcuVKvF4vf/vb3wAYO3Ysc+fOJSIiQl8+B9XJPyEhgddee42pU6fWepLPjz/+eF7HdbE06uJCl0p9ixrNmjWL6667DjhzUaNz7f/NN9/kgw8+YMuWLfqH1mazYbPZgOrHScXExLBnz55zfjT7ucR5LqqqqpgwYQJTp07lhhtuAKB9+/b69pPP1fmoiTEsLIzx48eTnp5O+/btycnJITw8nJycHMLCwi7oGNavX0+/fv30uI2Mv6a/uuI18v+kId87NcdQ40zv/9jYWJ588kkuv/xyAPbt26fXiLHZbDzyyCM8+uijLFu2jH//+98cO3YMPz8/Nm7cSP/+/fF4PCQlJbFy5Uq6deumv866dev4+eefcbvdfPjhh/z+979nx44dDB8+nKysLF5++WWGDRvGpEmTfHLtsu9F1MidPLe3Zs0a4uLigNMXNTpXGzZs4Pnnn2ft2rUEBgbq7cePH9f/LNu3bx8Oh0Mvtn8uTldk6UIopbjtttvo0aMHDz74oN5+unN1rsrKyigpKdH/vWnTJuLi4moVfzq5KNT5WrVqVa3pCqPir3G6eFNSUli5ciVKKbZu3UrLli31qY1z0dDvHTj/9390dDTffPMN33zzDVlZWcydOxez2czf/vY3Tpw4gVKKxx57jCuuuEJ/ruGaNWt45plnuOuuu3j88cfZvXs3d911FxaLhYyMDAYMGEBOTg55eXm43W7WrVtHWFgYV199NSaTqUGfVnPe6nv1T8kqi3q5+eabVVxcnIqPj1fXX3+9OnLkiL7tdEWNzkVMTIyKjIw8ZYnSu+++q3r27Kl69+6t+vbtq9auXXvex3C6Ikvnq+Yqe3x8fK0lYmc6V+di7969KiEhQSUkJKiePXvqMZ84cUINGzZM2e12NXz4cJWXl3fex1BaWqpCQ0NVYWGh3nYh8U+ePFl16NBBmc1mFRERoV577bXTxuv1etVdd92loqOjVVxcnNq+fft59W/0e6eu12jo939dvF6vevfdd9XatWvVzJkz9fa//OUvavDgwerEiRNq69atavTo0WrixIlqyZIlhr32OZBlb6Jx8nq96sorr6z1oV29erUaNWpUrf3qWup2Om+88Ya6++67zymOc+lfXHrHjh1TcXFxatq0aerRRx9VgwcPVm+88YYqLCxU999/vxo1apT6+OOP1bXXXqvmzZtXa72y1+tt6PBk2ZtonDRNY+nSpTz44INUVlZSWlrKH/7wBxYvXnypQxM+rF27dnz33XeMGTMGh8NBhw4dmDFjBqmpqVgsFv70pz9xzTXX8Pjjj7Nt27ZaKy80TftllHoJyUU94ZPi4uK4/vrree655ygrK2PatGnExMSc9efS09O5//77qaysJCAggDfeeINu3boB1U+EGTp0KIcPH+bmm2/WC6b//e9/569//Ssul4ukpCSWLFlS53pW0ThMmjSJAQMG4O/vz9atW3nrrbcYNGgQiYmJALz33ntcffXVWCwWMjMz+fLLL0lKStIvYno8nkv2/y8JWfis+fPn069fP6xWKzt27KjXz3Tv3p3PP/8cs9nMRx99xB/+8Af+9a9/AdXJ+vvvvycwMJDExER+85vfEBQUxD//+U++/PJLLBYLd911F//4xz+YNm1aQx6aaGA1v7x/+uknoqOjadmyJQCvvfYaAN26dePZZ59l165dpKSkMGvWLKZMmcKcOXP0ZPzxxx8zbNiwixq3JGThs4KCgpg0aRItWrTQl2WdTVFREdOnT8fhcKBpml6YBiA5OVl/5uENN9zAF198gdlsJiMjQx89VVRUXPDyOOE7hgwZQmxsLDNmzGDIkCEcP36cRYsW8d133+FwONi+fTu/+93v+Oyzz7jvvvs4ceIEbdu25dlnn+Wvf/1rnQ9RbkiSkIVPM5lM57RedN68eVxzzTWsWbOG/fv3M3ToUH3br2+0qJk3nD59OgsWLDAqZOFjLrvsMjZt2sTOnTtp27YtnTp1IiMjg+7duzN37lweeugh7rrrLuLj42nbti1HjhzhxRdfZOPGjRc9VknIokkpKirSb5p48803a23bvHkz+fn5BAQE8P777/P6668TGBjI2LFjeeCBBwgLCyM/P5+SkpILrlUhfM/JRe1NJhNOpxO73U5aWhrvvfeeXir0lltuYerUqed1Y8yFkoQsGrWEhAR9BD1x4kTmzJnD9OnTefrpp/nNb35Ta9+BAwcyYcIEDh06xM0336x/4J5++mlGjhyJ1+vFYrGwePFiSchN3ODBg5kxYwZlZWU8/PDD+t2jb775Jnv37mXLli2XJC55hJMQolkqKCjg3nvvJTw8nIULF1JeXk6XLl1YvXp1Q1zMq1dhEknIQohmrbi4mJCQEK644gq6du16ylSXQeSZekIIcTbBwcF4vV6uuOKKhkrG9SYjZCGEaHgyQhZCiMZEErIQQvgISchCCOEjJCELIYSPkIQshBA+QhKyEEL4CEnIQgjhIyQhCyGEj5CELIQQPkISshBC+AhJyEII4SMkIQshhI+QhCyEED5CErIQQvgISchCCOEjzvWZevWq6SmEEOLcyQhZCCF8hCRkIYTwEZKQhRDCR0hCFkIIHyEJWQghfIQkZCGE8BGSkIUQwkdIQhZCCB8hCVkIIXyEJGQhhPAR/z9BxYD7iwpOFQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.scatter(x_1[0], x_1[1], x_1[1], c=label, s=60)\n",
    "ax.view_init(0, -180)\n",
    "ax.set_ylim([150,-50])\n",
    "ax.set_zlim([-10000,10000])\n",
    "ax.set_xlabel('X Label')\n",
    "ax.set_ylabel('Y Label')\n",
    "ax.set_zlabel('Z Label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/18_kernel_income_v4_files/image011.png)\n",
    "\n",
    "To manipulate a large dataset and you may have to create more than 2\n",
    "dimensions, you will face a big problem using the above method. In fact,\n",
    "you need to transform all data points, which is clearly not sustainable.\n",
    "It will take you ages, and your computer may run out of memory.\n",
    "\n",
    "The most common way to overcome this issue is to use a **kernel**.\n",
    "\n",
    "## What is a Kernel?\n",
    "\n",
    "The idea is to use a higher-dimension feature space to make the data\n",
    "almost linearly separable as shown in the figure above.\n",
    "\n",
    "There are plenty of higher dimensional spaces to make the data points\n",
    "separable. For instance, we have shown that the polynomial mapping is a\n",
    "great start.\n",
    "\n",
    "We have also demonstrated that with lots of data, these transformation\n",
    "is not efficient. Instead, you can use a kernel function to modify the\n",
    "data without changing to a new feature plan.\n",
    "\n",
    "The magic of the kernel is to find a function that avoids all the\n",
    "trouble implied by the high-dimensional computation. The result of a\n",
    "kernel is a scalar, or said differently we are back to one-dimensional\n",
    "space\n",
    "\n",
    "After you found this function, you can plug it to the standard linear\n",
    "classifier.\n",
    "\n",
    "Let's see an example to understand the concept of Kernel. You have two\n",
    "vectors, `x1` and `x2`. The objective is to create a higher dimension by\n",
    "using a polynomial mapping. The output is equal to the dot product of\n",
    "the new feature map. From the method above, you need to:\n",
    "\n",
    "1.  Transform `x1` and `x2` into a new dimension\n",
    "\n",
    "2.  Compute the dot product: common to all kernels\n",
    "\n",
    "3.  Transform `x1` and `x2` into a new dimension\n",
    "\n",
    "You can use the function created above to compute the higher dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  9.         100.        ]\n",
      " [ 25.45584412 141.42135624]\n",
      " [ 36.         100.        ]]\n"
     ]
    }
   ],
   "source": [
    "## Kernel\n",
    "x1 = np.array([3,6])\n",
    "x2 = np.array([10,10])\n",
    "\n",
    "x_1 = mapping(x1, x2)\n",
    "print(x_1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the dot product\n",
    "\n",
    "You can use the object `dot` from numpy to compute the dot product\n",
    "between the first and second vector stored in `x_1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8100.0\n"
     ]
    }
   ],
   "source": [
    "print(np.dot(x_1[:,0], x_1[:,1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output is 8100. You see the problem, you need to store in memory a\n",
    "new feature map to compute the dot product. If you have a dataset with\n",
    "millions of records, it is computationally ineffective.\n",
    "\n",
    "Instead, you can use the **polynomial kernel** to compute the dot\n",
    "product without transforming the vector. This function computes the dot\n",
    "product of `x1` and `x2` as if these two vectors have been transformed\n",
    "into the higher dimension. Said differently, a kernel function computes\n",
    "the results of the dot product from another feature space.\n",
    "\n",
    "You can write the polynomial kernel function in Python as follow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def polynomial_kernel(x, y, p=2):\n",
    "    return (np.dot(x, y)) ** p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is the power of the dot product of two vectors. Below, you return the\n",
    "second degree of the polynomial kernel. The output is equal to the other\n",
    "method. This is the magic of the kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8100"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_kernel(x1, x2, p=2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Type of Kernel\n",
    "\n",
    "There are lots of different kernels available. The simplest is the\n",
    "linear kernel. This function works pretty well for text classification.\n",
    "The other kernel is:\n",
    "\n",
    "-   Polynomial kernel\n",
    "\n",
    "-   Gaussian Kernel\n",
    "\n",
    "[]{#header-n134 .anchor}\n",
    "\n",
    "In the example with TensorFlow, we will use the Random Fourier.\n",
    "TensorFlow has a build in estimator to compute the new feature space.\n",
    "This function is an approximation of the Gaussian kernel function.\n",
    "\n",
    "$$e\\frac{- \\parallel x - y\\overset{2}{\\parallel}}{2\\sigma^{2}}$$\n",
    "\n",
    "This function computes the similarity between the data points in a much\n",
    "higher dimensional space.\n",
    "\n",
    "# Train Kernel classifier with TensorFlow\n",
    "\n",
    "The objective of the algorithm is to classify the household earning more\n",
    "or less than 50k.\n",
    "\n",
    "You will evaluate a logistic regression to have a benchmark model. After\n",
    "that, you will train a Kernel classifier to see if you can get better\n",
    "results.\n",
    "\n",
    "You use the following variables from the `adult` dataset:\n",
    "\n",
    "-   age\n",
    "\n",
    "-   workclass\n",
    "\n",
    "-   fnlwgt\n",
    "\n",
    "-   education\n",
    "\n",
    "-   education\\_num\n",
    "\n",
    "-   marital\n",
    "\n",
    "-   occupation\n",
    "\n",
    "-   relationship\n",
    "\n",
    "-   race\n",
    "\n",
    "-   sex\n",
    "\n",
    "-   capital\\_gain\n",
    "\n",
    "-   capital\\_loss\n",
    "\n",
    "-   hours\\_week\n",
    "\n",
    "-   native\\_country\n",
    "\n",
    "-   label\n",
    "\n",
    "You will proceed as follow before you train and evaluate the model:\n",
    "\n",
    "Step 1: Import the libraries\n",
    "\n",
    "Step 2: Import the data\n",
    "\n",
    "Step 3: Prepare the data\n",
    "\n",
    "Step 4: Construct the `input_fn`\n",
    "\n",
    "Step 5: Construct the logistic model: Baseline model\n",
    "\n",
    "Step 6: Evaluate the model\n",
    "\n",
    "Step 7: Construct the Kernel classifier\n",
    "\n",
    "Step 8: Evaluate the Kernel classifier\n",
    "\n",
    "**Step 1:** Import the libraries\n",
    "\n",
    "To import and train the model, you need to import `tensorflow`, `pandas`\n",
    "and `numpy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "#import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2:** Import the data\n",
    "\n",
    "You download the data from the following\n",
    "[website](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/)\n",
    "and you import it as a panda dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Define path data\n",
    "COLUMNS = ['age','workclass', 'fnlwgt', 'education', 'education_num', 'marital', 'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss', 'hours_week', 'native_country', 'label']\n",
    "PATH = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\"\n",
    "PATH_test = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test\"\n",
    "## Import \n",
    "\n",
    "df_train = pd.read_csv(PATH, skipinitialspace=True, names = COLUMNS, index_col=False)\n",
    "df_test = pd.read_csv(PATH_test,skiprows = 1, skipinitialspace=True, names = COLUMNS, index_col=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the train and test set are defined, you can change the column\n",
    "label from string to integer. tensorflow does not accept string value\n",
    "for the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32561, 15)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label = {'<=50K': 0,'>50K': 1}\n",
    "df_train.label = [label[item] for item in df_train.label]\n",
    "label_t = {'<=50K.': 0,'>50K.': 1}\n",
    "df_test.label = [label_t[item] for item in df_test.label]\n",
    "\n",
    "df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3:** Prepare the data\n",
    "\n",
    "The dataset contains both continuous and categorical features. A good\n",
    "practice is to standardize the values of the continuous variables. You\n",
    "can use the function `StandardScaler` from sci-kit learn. You create a\n",
    "user-defined function as well to make it easier to convert the train and\n",
    "test set. Note that, you concatenate the continuous and categorical\n",
    "variables to a common dataset and the array should be of the type:\n",
    "`float32`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "COLUMNS_INT = ['age','fnlwgt','education_num','capital_gain', 'capital_loss', 'hours_week']\n",
    "CATE_FEATURES = ['workclass', 'education', 'marital', 'occupation', 'relationship', 'race', 'sex', 'native_country']\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn import preprocessing\n",
    "\n",
    "def prep_data_str(df):\n",
    "    \n",
    "    scaler = StandardScaler()\n",
    "    le = preprocessing.LabelEncoder()   \n",
    "    df_toscale = df[COLUMNS_INT]\n",
    "    df_scaled = scaler.fit_transform(df_toscale.astype(np.float64))\n",
    "    X_1 = df[CATE_FEATURES].apply(le.fit_transform)\n",
    "    y = df['label'].astype(np.int32)\n",
    "    X_conc = np.c_[df_scaled, X_1].astype(np.float32)\n",
    "    return X_conc, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The transformer function is ready, you can\n",
    "convert the dataset and create the `input_fn` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32561, 14)\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = prep_data_str(df_train)\n",
    "X_test, y_test = prep_data_str(df_test)\n",
    "print(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next step, you will train a logistic regression. It will give you a baseline accuracy. The objective is to beat the baseline with a different algorithm, namely a Kernel classifier.\n",
    "\n",
    "**Step 4**: Construct the logistic model: Baseline model\n",
    "\n",
    "You construct the feature column with the object `real_valued_column`.\n",
    "It will make sure all variables are dense numeric data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_column = tf.contrib.layers.real_valued_column('features', dimension=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimator is defined using TensorFlow `Estimator`, you instruct the\n",
    "feature columns and where to save the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'kernel_log', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1c1d4d8e48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "estimator = tf.estimator.LinearClassifier(feature_columns=[feat_column],\n",
    "                                          n_classes=2,\n",
    "                                          model_dir = \"kernel_log\"\n",
    "                                         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will train the logisitc regression using mini-batches of size 200."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model\n",
    "train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"features\": X_train},\n",
    "    y=y_train,\n",
    "    batch_size=200,\n",
    "    num_epochs=None,\n",
    "    shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can train the model with 1.000 iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into kernel_log/model.ckpt.\n",
      "INFO:tensorflow:loss = 138.62949, step = 1\n",
      "INFO:tensorflow:global_step/sec: 261.643\n",
      "INFO:tensorflow:loss = 74.75966, step = 101 (0.385 sec)\n",
      "INFO:tensorflow:global_step/sec: 308.446\n",
      "INFO:tensorflow:loss = 71.47853, step = 201 (0.326 sec)\n",
      "INFO:tensorflow:global_step/sec: 333.308\n",
      "INFO:tensorflow:loss = 83.11995, step = 301 (0.299 sec)\n",
      "INFO:tensorflow:global_step/sec: 353.282\n",
      "INFO:tensorflow:loss = 85.21626, step = 401 (0.284 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.193\n",
      "INFO:tensorflow:loss = 79.21164, step = 501 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.697\n",
      "INFO:tensorflow:loss = 85.23293, step = 601 (0.302 sec)\n",
      "INFO:tensorflow:global_step/sec: 293.68\n",
      "INFO:tensorflow:loss = 74.68625, step = 701 (0.342 sec)\n",
      "INFO:tensorflow:global_step/sec: 288.926\n",
      "INFO:tensorflow:loss = 70.72325, step = 801 (0.347 sec)\n",
      "INFO:tensorflow:global_step/sec: 322.193\n",
      "INFO:tensorflow:loss = 95.31248, step = 901 (0.307 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into kernel_log/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 73.30052.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.linear.LinearClassifier at 0x1c1cf67cc0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator.train(input_fn=train_input_fn, steps=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 6:** Evaluate the model\n",
    "\n",
    "You define the numpy estimator to evaluate the model. You use the entire\n",
    "dataset for evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-12:36:21\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from kernel_log/model.ckpt-1000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-12:36:22\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8247651, accuracy_baseline = 0.76377374, auc = 0.8493674, auc_precision_recall = 0.6747448, average_loss = 0.3867324, global_step = 1000, label/mean = 0.23622628, loss = 6296.39, precision = 0.6719778, prediction/mean = 0.2614143, recall = 0.50442016\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: kernel_log/model.ckpt-1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.8247651,\n",
       " 'accuracy_baseline': 0.76377374,\n",
       " 'auc': 0.8493674,\n",
       " 'auc_precision_recall': 0.6747448,\n",
       " 'average_loss': 0.3867324,\n",
       " 'label/mean': 0.23622628,\n",
       " 'loss': 6296.39,\n",
       " 'precision': 0.6719778,\n",
       " 'prediction/mean': 0.2614143,\n",
       " 'recall': 0.50442016,\n",
       " 'global_step': 1000}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluation\n",
    "test_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"features\": X_test},\n",
    "    y=y_test,\n",
    "    batch_size=16281,\n",
    "    num_epochs=1,\n",
    "    shuffle=False)\n",
    "\n",
    "estimator.evaluate(input_fn=test_input_fn, steps=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have an accuracy of 82 percents. In the next section, you will try\n",
    "to beat the logistic classifier with a Kernel classifier\n",
    "\n",
    "**Step 7:** Construct the Kernel classifier\n",
    "\n",
    "The kernel estimator is not so different from the traditional linear\n",
    "classifier, at least in term of construction. The idea behind is to use\n",
    "the power of explicit kernel with the linear classifier.\n",
    "\n",
    "You need two pre-defined estimators available in TensorFlow to train the\n",
    "Kernel Classifier:\n",
    "\n",
    "-   `RandomFourierFeatureMapper`\n",
    "\n",
    "-   `KernelLinearClassifier`\n",
    "\n",
    "You learned in the first section that you need to transform the low\n",
    "dimension into a high dimension using a kernel function. More precisely,\n",
    "you will use the Random Fourier, which is an approximation of the\n",
    "Gaussian function. Luckily, Tensorflow has the function in its library:\n",
    "`RandomFourierFeatureMapper`. The model can be trained using the\n",
    "estimator `KernelLinearClassifier`.\n",
    "\n",
    "To build the model, you will follow these steps:\n",
    "\n",
    "A.  Set the high dimension Kernel function\n",
    "\n",
    "B.  Set the L2 hyperparameter\n",
    "\n",
    "C.  Build the model\n",
    "\n",
    "D.  Train the model\n",
    "\n",
    "E.  Evaluate the model\n",
    "\n",
    "**Step A:** Set the high dimension Kernel function\n",
    "\n",
    "The current dataset contains 14 features that you will transform to a\n",
    "new high dimension of the 5.000-dimensional vector. You use the random\n",
    "Fourier features to achieve the transformation. If you recall the\n",
    "Gaussian Kernel formula, you note that there is the standard deviation\n",
    "$\\sigma$ parameter to define. This parameter controls for the similarity\n",
    "measure employs during the classification.\n",
    "\n",
    "You can tune all the parameters in `RandomFourierFeatureMapper` with:\n",
    "\n",
    "-   `input_dim = 14`\n",
    "\n",
    "-   `output_dim= 5000`\n",
    "\n",
    "-   `stddev=4`\n",
    "\n",
    "<!-- -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Prep Kernel\n",
    "kernel_mapper = tf.contrib.kernel_methods.RandomFourierFeatureMapper(\n",
    "input_dim=14, output_dim=5000, stddev=4, name='rffm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to construct the kernel mapper by using the feature columns\n",
    "created before: `feat_column`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Map Kernel\n",
    "kernel_mappers = {feat_column: [kernel_mapper]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step B:** Set the L2 hyperparameter\n",
    "\n",
    "To prevent overfitting, you penalize the loss function with the L2\n",
    "regularizer. You set the L2 hyperparameter to 0.1 and the learning rate\n",
    "to 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.FtrlOptimizer(\n",
    "learning_rate=5, l2_regularization_strength=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step C:** Build the model\n",
    "\n",
    "The next step is similar to the linear classification. You use the\n",
    "build-in estimator `KernelLinearClassifier`. Note that you add the\n",
    "kernel mapper defined previously and change the model directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/site-packages/tensorflow/contrib/kernel_methods/python/kernel_estimators.py:305: multi_class_head (from tensorflow.contrib.learn.python.learn.estimators.head) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.contrib.estimator.*_head.\n",
      "WARNING:tensorflow:From /Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1179: BaseEstimator.__init__ (from tensorflow.contrib.learn.python.learn.estimators.estimator) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please replace uses of any Estimator from tf.contrib.learn with an Estimator from tf.estimator.*\n",
      "WARNING:tensorflow:From /Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:427: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.\n",
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1c1e1c3f98>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'kernel_train'}\n"
     ]
    }
   ],
   "source": [
    "### Prep estimator\n",
    "estimator_kernel = tf.contrib.kernel_methods.KernelLinearClassifier(\n",
    "n_classes=2,\n",
    "optimizer=optimizer,\n",
    "kernel_mappers=kernel_mappers, \n",
    "model_dir=\"kernel_train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step D:** Train the model\n",
    "\n",
    "Now that the Kernel classifier is built, you are ready to train it. You\n",
    "choose to iterate 2000 times the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:800: calling expand_dims (from tensorflow.python.ops.array_ops) with dim is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use the `axis` argument instead\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:From /Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:678: ModelFnOps.__new__ (from tensorflow.contrib.learn.python.learn.estimators.model_fn) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "When switching to tf.estimator.Estimator, use tf.estimator.EstimatorSpec. You can use the `estimator_spec` method to create an equivalent one.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into kernel_train/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.6931474, step = 1\n",
      "INFO:tensorflow:global_step/sec: 62.9676\n",
      "INFO:tensorflow:loss = 0.42536858, step = 101 (1.590 sec)\n",
      "INFO:tensorflow:global_step/sec: 76.5478\n",
      "INFO:tensorflow:loss = 0.3803177, step = 201 (1.307 sec)\n",
      "INFO:tensorflow:global_step/sec: 70.7709\n",
      "INFO:tensorflow:loss = 0.33762196, step = 301 (1.413 sec)\n",
      "INFO:tensorflow:global_step/sec: 87.447\n",
      "INFO:tensorflow:loss = 0.40866998, step = 401 (1.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 65.3455\n",
      "INFO:tensorflow:loss = 0.4405681, step = 501 (1.531 sec)\n",
      "INFO:tensorflow:global_step/sec: 85.9181\n",
      "INFO:tensorflow:loss = 0.35242862, step = 601 (1.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 69.1656\n",
      "INFO:tensorflow:loss = 0.3657984, step = 701 (1.446 sec)\n",
      "INFO:tensorflow:global_step/sec: 64.3009\n",
      "INFO:tensorflow:loss = 0.3184782, step = 801 (1.555 sec)\n",
      "INFO:tensorflow:global_step/sec: 67.7812\n",
      "INFO:tensorflow:loss = 0.38625008, step = 901 (1.476 sec)\n",
      "INFO:tensorflow:global_step/sec: 68.6125\n",
      "INFO:tensorflow:loss = 0.42177472, step = 1001 (1.457 sec)\n",
      "INFO:tensorflow:global_step/sec: 79.0373\n",
      "INFO:tensorflow:loss = 0.3676224, step = 1101 (1.265 sec)\n",
      "INFO:tensorflow:global_step/sec: 75.4601\n",
      "INFO:tensorflow:loss = 0.35513246, step = 1201 (1.325 sec)\n",
      "INFO:tensorflow:global_step/sec: 72.8742\n",
      "INFO:tensorflow:loss = 0.3081485, step = 1301 (1.372 sec)\n",
      "INFO:tensorflow:global_step/sec: 72.7422\n",
      "INFO:tensorflow:loss = 0.30261478, step = 1401 (1.376 sec)\n",
      "INFO:tensorflow:global_step/sec: 79.2251\n",
      "INFO:tensorflow:loss = 0.27042457, step = 1501 (1.260 sec)\n",
      "INFO:tensorflow:global_step/sec: 89.6457\n",
      "INFO:tensorflow:loss = 0.3486829, step = 1601 (1.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 86.4946\n",
      "INFO:tensorflow:loss = 0.38481957, step = 1701 (1.157 sec)\n",
      "INFO:tensorflow:global_step/sec: 95.7964\n",
      "INFO:tensorflow:loss = 0.29908475, step = 1801 (1.043 sec)\n",
      "INFO:tensorflow:global_step/sec: 93.8122\n",
      "INFO:tensorflow:loss = 0.2940178, step = 1901 (1.066 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into kernel_train/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.34932885.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "KernelLinearClassifier(params={'head': <tensorflow.contrib.learn.python.learn.estimators.head._BinaryLogisticHead object at 0x10f1fe7b8>, 'feature_columns': {_RealValuedColumn(column_name='features_MAPPED', dimension=5000, default_value=None, dtype=tf.float32, normalizer=None)}, 'optimizer': <tensorflow.python.training.ftrl.FtrlOptimizer object at 0x1c1d4ef208>, 'kernel_mappers': {_RealValuedColumn(column_name='features', dimension=14, default_value=None, dtype=tf.float32, normalizer=None): [<tensorflow.contrib.kernel_methods.python.mappers.random_fourier_features.RandomFourierFeatureMapper object at 0x1c1d4ef2e8>]}})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### estimate \n",
    "estimator_kernel.fit(input_fn=train_input_fn, steps=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step E:** Evaluate the model\n",
    "\n",
    "Last but not least, you evaluate the performance of your model. You\n",
    "should be able to beat the logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-12:36:52\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from kernel_train/model.ckpt-2000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-12:36:54\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.8389534, accuracy/baseline_label_mean = 0.23622628, accuracy/threshold_0.500000_mean = 0.8389534, auc = 0.8904966, auc_precision_recall = 0.72741604, global_step = 2000, labels/actual_label_mean = 0.23622628, labels/prediction_mean = 0.22733423, loss = 0.34312406, precision/positive_threshold_0.500000_mean = 0.74717283, recall/positive_threshold_0.500000_mean = 0.48101923\n"
     ]
    }
   ],
   "source": [
    "# Evaluate and report metrics.\n",
    "eval_metrics = estimator_kernel.evaluate(input_fn=test_input_fn, steps=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final accuracy is 84%, it is a 2% improvement compared to the logistic regression. There is a tradeoff between accuracy improvement and computational cost. You need to think if 2% improvement worth the time consumed by the different classifier and if it has a compelling impact on your business. \n",
    "\n",
    "## Summary\n",
    "\n",
    "A kernel is a great tool to transform non-linear data to (almost)\n",
    "linear. The shortcoming of this method is it computationally\n",
    "time-consuming and costly.\n",
    "\n",
    "Below, you can find the most important code to train a kernel classifer\n",
    "\n",
    "*Set the high dimension Kernel function*\n",
    "\n",
    "-   `input_dim = 14`\n",
    "\n",
    "-   `output_dim= 5000`\n",
    "\n",
    "-   `stddev=4`\n",
    "\n",
    "Prep Kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "kernel_mapper = tf.contrib.kernel_methods.RandomFourierFeatureMapper(\n",
    "input_dim=14, output_dim=5000, stddev=4, name='rffm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the L2 hyperparameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.FtrlOptimizer(\n",
    "learning_rate=5, l2_regularization_strength=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1c1df20710>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'kernel_train'}\n"
     ]
    }
   ],
   "source": [
    "estimator_kernel = tf.contrib.kernel_methods.KernelLinearClassifier(\n",
    "n_classes=2,\n",
    "optimizer=optimizer,\n",
    "kernel_mappers=kernel_mappers,\n",
    "model_dir=\"kernel_train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from kernel_train/model.ckpt-2000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into kernel_train/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.29066592, step = 2001\n",
      "INFO:tensorflow:global_step/sec: 81.0098\n",
      "INFO:tensorflow:loss = 0.3658589, step = 2101 (1.236 sec)\n",
      "INFO:tensorflow:global_step/sec: 78.2854\n",
      "INFO:tensorflow:loss = 0.34852684, step = 2201 (1.277 sec)\n",
      "INFO:tensorflow:global_step/sec: 67.8807\n",
      "INFO:tensorflow:loss = 0.34097832, step = 2301 (1.474 sec)\n",
      "INFO:tensorflow:global_step/sec: 67.7932\n",
      "INFO:tensorflow:loss = 0.4329201, step = 2401 (1.474 sec)\n",
      "INFO:tensorflow:global_step/sec: 77.801\n",
      "INFO:tensorflow:loss = 0.32452166, step = 2501 (1.289 sec)\n",
      "INFO:tensorflow:global_step/sec: 78.6713\n",
      "INFO:tensorflow:loss = 0.3111766, step = 2601 (1.268 sec)\n",
      "INFO:tensorflow:global_step/sec: 72.3864\n",
      "INFO:tensorflow:loss = 0.3285103, step = 2701 (1.381 sec)\n",
      "INFO:tensorflow:global_step/sec: 69.1575\n",
      "INFO:tensorflow:loss = 0.36192822, step = 2801 (1.445 sec)\n",
      "INFO:tensorflow:global_step/sec: 78.6935\n",
      "INFO:tensorflow:loss = 0.3834426, step = 2901 (1.271 sec)\n",
      "INFO:tensorflow:global_step/sec: 88.0434\n",
      "INFO:tensorflow:loss = 0.30431166, step = 3001 (1.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 90.1264\n",
      "INFO:tensorflow:loss = 0.33217943, step = 3101 (1.110 sec)\n",
      "INFO:tensorflow:global_step/sec: 83.3535\n",
      "INFO:tensorflow:loss = 0.32786164, step = 3201 (1.199 sec)\n",
      "INFO:tensorflow:global_step/sec: 88.6038\n",
      "INFO:tensorflow:loss = 0.3120848, step = 3301 (1.129 sec)\n",
      "INFO:tensorflow:global_step/sec: 79.5437\n",
      "INFO:tensorflow:loss = 0.30408162, step = 3401 (1.257 sec)\n",
      "INFO:tensorflow:global_step/sec: 79.0231\n",
      "INFO:tensorflow:loss = 0.3368484, step = 3501 (1.266 sec)\n",
      "INFO:tensorflow:global_step/sec: 69.3101\n",
      "INFO:tensorflow:loss = 0.3951297, step = 3601 (1.443 sec)\n",
      "INFO:tensorflow:global_step/sec: 68.4297\n",
      "INFO:tensorflow:loss = 0.2609773, step = 3701 (1.461 sec)\n",
      "INFO:tensorflow:global_step/sec: 82.7538\n",
      "INFO:tensorflow:loss = 0.3478087, step = 3801 (1.208 sec)\n",
      "INFO:tensorflow:global_step/sec: 82.9163\n",
      "INFO:tensorflow:loss = 0.32420224, step = 3901 (1.206 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 4000 into kernel_train/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.29162148.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "KernelLinearClassifier(params={'head': <tensorflow.contrib.learn.python.learn.estimators.head._BinaryLogisticHead object at 0x1c1df20b38>, 'feature_columns': {_RealValuedColumn(column_name='features_MAPPED', dimension=5000, default_value=None, dtype=tf.float32, normalizer=None)}, 'optimizer': <tensorflow.python.training.ftrl.FtrlOptimizer object at 0x10f214278>, 'kernel_mappers': {_RealValuedColumn(column_name='features', dimension=14, default_value=None, dtype=tf.float32, normalizer=None): [<tensorflow.contrib.kernel_methods.python.mappers.random_fourier_features.RandomFourierFeatureMapper object at 0x1c1d4ef2e8>]}})"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator_kernel.fit(input_fn=train_input_fn, steps=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-12:37:23\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from kernel_train/model.ckpt-4000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-12:37:25\n",
      "INFO:tensorflow:Saving dict for global step 4000: accuracy = 0.84270006, accuracy/baseline_label_mean = 0.23622628, accuracy/threshold_0.500000_mean = 0.84270006, auc = 0.8950697, auc_precision_recall = 0.73921394, global_step = 4000, labels/actual_label_mean = 0.23622628, labels/prediction_mean = 0.22281823, loss = 0.3347098, precision/positive_threshold_0.500000_mean = 0.7528532, recall/positive_threshold_0.500000_mean = 0.4973999\n"
     ]
    }
   ],
   "source": [
    "eval_metrics = estimator_kernel.evaluate(input_fn=test_input_fn,\n",
    "steps=1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}