docs/advanced_examples/ConvolutionalNeuralNetwork.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b760a0f6",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network\n",
    "\n",
    "This example shows how to design a simple quantization aware training CNN and how to measure \n",
    "the maximum accumulator bit width. We also show the impact of quantization on accuracy. \n",
    "\n",
    "To analyze the potential FHE-compatible quantization configurations, we use the simulation mode of\n",
    " the FHE circuit of the CNN. The maximum accumulator \n",
    "bit width that is currently allowed in Concrete ML is **16** bits, which limits the bit-widths \n",
    "that are usable for weights and activations.\n",
    "\n",
    "We find an FHE compatible configuration for 3 bits weights and activations and run the CNN in FHE.\n",
    "The accuracy in this highly quantized configuration is 92%. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "253288cf",
   "metadata": {},
   "source": [
    "### Import required modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6200ab62",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.utils\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "from tqdm import tqdm\n",
    "\n",
    "from concrete.ml.torch.compile import compile_torch_model\n",
    "\n",
    "# And some helpers for visualization.\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed920ba",
   "metadata": {},
   "source": [
    "### Load the data-set and visualize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b9f955f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAH5CAYAAABzvRxpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAykUlEQVR4nO3df3BU93nv8WdXQitkJOEELLGWAMtEVtANwoFqQwp1HMtD5JsMJrVLiJo4CrFD4psbW+HOreyMpc60KDO+deg4ICc3JWqbNEDT2sy9l6ixVZuGAKEFp0lwbBCWrVVkiR+1tRI2+rG7948Eyaoh4ln2nLP7nPdrZmdAfL/7fI8+u+dhtavvCSSTyaQAAAATgl4vAAAApA+NHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAIbluFkskEtLf3y+FhYUSCATcLI3LSCaTMjw8LOFwWILB9Pw/j5wzCxn7Q7pzJuPMc6UZu9rY+/v7pby83M2SuELRaFTKysrScl/knJnI2B/SlTMZZ66ZMna1sRcWFoqIyGq5Q3JllmN1zjXWquf89y/9g2r8nz9/h7rGkocG1XMmBk+r56juX8blgOybzCYd3Mo5FaVdc1Tjbyw4o67x/75xi3rO3O8dUc+5Un7L+M2PrVSN3/61b6hr/MVAnXrOwG0j6jka6c7ZrYxfbdWfr4998q9U438wMlddY3f9CvWcTDlfu9rYL/44J1dmSW7AuQdKTl6+ek7BnBzV+GCBvkZuME89Rxz8PomIyG83FE7nj9rcyjkVeXN0GeQX6NefyuPP0e+TzzLOnaX7/s8p1P/YOm9E/1x2/PuU5pzdyjiYr3++FCkzKxDd+V0ku8/XKb0Rs337dlm8eLHk5+dLJBKRI0ece7UBb5CxfWTsD+TsP+rGvnv3bmlqapKWlhY5duyY1NTUyNq1a+X0aWd/BAH3kLF9ZOwP5OxP6sb+2GOPyb333iuNjY2ydOlSeeKJJ6SgoEB27tzpxPrgATK2j4z9gZz9SdXYx8bG5OjRo1JXN/XBkWAwKHV1dXLo0KF3jB8dHZVYLDbthsymzViEnLMNGfsD52v/UjX2s2fPSjwel5KSkmlfLykpkYGBgXeMb2trk+Li4skbvzqR+bQZi5BztiFjf+B87V+O7jzX3NwsQ0NDk7doNOpkOXiEnO0jY/vI2A7Vr7vNmzdPcnJyZHBw+u9jDw4OSmlp6TvGh0IhCYVCV7dCuEqbsQg5Zxsy9gfO1/6lesWel5cnK1askK6ursmvJRIJ6erqklWrVqV9cXAfGdtHxv5Azv6l3qCmqalJ7rnnHlm5cqXU1tbKtm3b5Pz589LY2OjE+uABMraPjP2BnP1J3dg3bNggZ86ckUceeUQGBgZk+fLl0tnZ+Y4PaCB7kbF9ZOwP5OxPgWQymXSrWCwWk+LiYvmQrHN0i8LGl15Vz/lE4euq8R/42V3qGoeX/0A9Z0XrF1Tj533r0r+udDkTyXF5TvbK0NCQFBUVqeZejls5p2Ls6UWq8c9W71XXuKHzc+o5lZ/9N/WcK5XNGSduuVk95+nvf0c1/sT4eXWNF8b0jbH9PUvUczTSnXOqGZ9o1+393vbhv9cuTf5s50bV+F9+eYe6xu9/+fPqOXP+/qfqORpXmjHXYwcAwBAaOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMER9ERgvTHx4hWr8Jwp/pq5R/5FPqMYX//xFdY0/OnCbes5/3BxXjZ+nrpC9UtlH/JuV31DOuEZdo+gXeeo5uLSX79RfH3zr2ZtU4/+q61Z1jVMbnlDPaVfPyE5V7THV+L/9U93e8iIiX93/fdX4XcPXqms4ve+7k3jFDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGEJjBwDAEBo7AACG0NgBADCExg4AgCE0dgAADKGxAwBgCI0dAABDsuIiMBferVvmV0+/T10jkcJFXbT+9Rc3Ol4jm/W2flA1fm/jo+oalbP0F3XRuv5H59RzdJf68Y+bvvayes7uXt3Fln74gP5xdOvxT6rn5Mmr6jnZSH0uXValrvGJwtdV4//oZf0FuHJL9e1xYmBQPccJvGIHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGEJjBwDAEBo7AACG0NgBADAkO/aKv1b3/4/vHVqlrlEpR9RztHKLx9RzJobyHFhJZlrYelA1/oH29eoa+57/kXqO1vi8AvUcv/wPO6fkOtX4l/6kQl1j021d6jlas//4LfUcrgdwaalcp+O/vn+tavzNnf3qGtKpn/L8R8Kq8U7tLe+X8wkAAL5AYwcAwBAaOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMCQrLgKT/3pCNf733ndKXWNIOT63tERdY8PSo+o5e364Wj0H3jr9/tnqOaX7HVhIBvpV20LV+J6PPOHQSqbUPrRFPefawUMOrARXSnvxFO3FWUREzu0sVM8ZbHmXanzlF7gIDAAAmAGNHQAAQ1SNvbW1VQKBwLRbVVWVU2uDB8jYH8jZPjL2L/V77NXV1fLMM89M3UFuVrxNDwUy9gdyto+M/Umdcm5urpSWll7R2NHRURkdHZ38eywW05aDBzQZi5BztuK5bB8Z+5P6PfaTJ09KOByWiooKaWhokN7e3suObWtrk+Li4slbeXn5VS0W7tBkLELO2Yrnsn1k7E+qxh6JRKSjo0M6Ozulvb1denp6ZM2aNTI8PHzJ8c3NzTI0NDR5i0ajaVk0nKPNWIScsxHPZfvI2L9UP4qvr6+f/POyZcskEonIokWLZM+ePbJp06Z3jA+FQhIKha5+lXCNNmMRcs5GPJftI2P/uqpfd5s7d65UVlZKd3d3utaDDEPG/kDO9pGxf1xVYx8ZGZFTp07JggUL0rUeZBgy9gdyto+M/UPV2Lds2SL79++XV155RQ4ePCjr16+XnJwc2bhxo1Prg8vI2B/I2T4y9i/Ve+x9fX2yceNGOXfunMyfP19Wr14thw8flvnz5zu1PriMjP2BnO0jY/9SNfZdu3Y5tY7fqegl3SVaWsr+r7rGp+9rUo2fdecZdY1U3NDs7sUmvMoY7vIq5yV/HVeN37ryJnWNh+a9pBp/ZGu7usatDevUc85/T3chkms7ru65ny3P5RPtteo54X8OqMZfuFb/rvPfLH1MPefON76gnuME9ooHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGEJjBwDAEBo7AACG0NgBADBEtVe8VxI/f1E1fkP7V9Q1vvqV76vGbzt1m7rGvy7PUc/B5cUHT6vn3Hpct8f3s9V71TUmVuuubSAiIl/XT8lGwf3Pq8bvXzZbXePZWxpV4ye++h/6Gik8Lm74g8+pxl/boS6RlWa9oT8vfunPnN8H/86D+n3fKz75s/QvJAW8YgcAwBAaOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMMTVveKTyaSIiEzIuEjSuTrx0QvqOW+OxHU1zo+qa0wkx9VznDYhv1nTxWzSwa2cUzGhzC02nFDXiL+ZWY8Nv2WcmNA9/7WPCZHUHheJt5TrUj4m0p2zWxknLjh/vk5F4k39upw+x19pxoFkOp/tM+jr65Py8nK3ykEhGo1KWVlZWu6LnDMTGftDunIm48w1U8auNvZEIiH9/f1SWFgogUBg2r/FYjEpLy+XaDQqRUVFbi0pI3h57MlkUoaHhyUcDkswmJ53Zi6XMxmTsWVeH3u6c+Z8/U5eH/eVZuzqj+KDweCM/5MsKiry1QPl7bw69uLi4rTe30w5kzEZW+blsaczZ87Xl5fpGfPhOQAADKGxAwBgSMY09lAoJC0tLRIKhbxeiuv8cux+Oc5L8cux++U4L8VPx+6nY327bDluVz88BwAAnJUxr9gBAMDVo7EDAGAIjR0AAENo7AAAGJIRjX379u2yePFiyc/Pl0gkIkeOHPF6SY5rbW2VQCAw7VZVVeX1shxFzvZzJmMytijbMva8se/evVuampqkpaVFjh07JjU1NbJ27Vo5ffq010tzXHV1tbz22muTtwMHDni9JMeQs/2cyZiMLcuqjJMeq62tTd5///2Tf4/H48lwOJxsa2vzcFXOa2lpSdbU1Hi9DNeQs31kbB8ZZwdPX7GPjY3J0aNHpa6ubvJrwWBQ6urq5NChQx6uzB0nT56UcDgsFRUV0tDQIL29vV4vyRHkbD9nMiZj67IpY08b+9mzZyUej0tJScm0r5eUlMjAwIBHq3JHJBKRjo4O6ezslPb2dunp6ZE1a9bI8PCw10tLO3K2nzMZk7Fl2Zaxq1d3w5T6+vrJPy9btkwikYgsWrRI9uzZI5s2bfJwZUgncraPjO3Ltow9fcU+b948ycnJkcHBwWlfHxwclNLSUo9W5Y25c+dKZWWldHd3e72UtCPnKVZzJuMpZGxfpmfsaWPPy8uTFStWSFdX1+TXEomEdHV1yapVqzxcmftGRkbk1KlTsmDBAq+XknbkPMVqzmQ8hYzty/SMXf1RfCKRkP7+fiksLJRAICAiIps3b5bNmzdLdXW1rFixQnbs2CEjIyNy1113SSwWc3N5rnr44Yelvr5eysvLZWBgQLZu3SqBQEA++tGPunrcyWRShoeHJRwOSzCYnv/nkfOUTMiZjJ2VCRmLpD9nMp6SbRm7enW3vr4+KS8vd6scFKLRqJSVlaXlvsg5M5GxP6QrZzLOXDNl7Oor9sLCQhERWS13SK7McrP0jHKum68a/9b/zlfXyFsXVc9x2oSMywHZN5lNOriVc2nXHPWco6/pTlTX//Gv1DUyTTZnnArt4+LGgjPqGj9ZNVs9x2npztmtjKPNEfWc8eKEavynbvkXdY0t79a/f949fl41/k9u/Zhq/ERiTPaf/ZsZM06psW/fvl0effRRGRgYkJqaGnn88celtrZ2xnkXf5yTK7MkN5BZJ4OcYJ5qfO41IXWNTDtmERH57c9rLmZzUaoZv/2+nM45b44uMxGRnAJdbhmZmVYWZ5wK7eMiv0C//kw7ZhFJe85uZZyTr3+RFM/XNfb8Ofr1FxXq386YM66bk6vsOxf954z/M/XK/byloF+QsX1k7A/k7E/qxv7YY4/JvffeK42NjbJ06VJ54oknpKCgQHbu3OnE+uABMraPjP2BnP1J1di1WwqOjo5KLBabdkNmS2XbSHLOLmTsD5yv/UvV2LVbCra1tUlxcfHkjU9YZr5Uto0k5+xCxv7A+dq/HN2gprm5WYaGhiZv0WjmfSocV4+c7SNj+8jYDtWn4rVbCoZCIQmF9J8eh3dS2TaSnLMLGfsD52v/Ur1iZ0tB+8jYPjL2B3L2L/XvsTc1Nck999wjK1eulNraWtm2bZucP39eGhsbnVgfPEDG9pGxP5CzP6kb+4YNG+TMmTPyyCOPyMDAgCxfvlw6Ozvf8QENZC8yto+M/YGc/cnVveJjsZgUFxfLh2Rdxu3c1Nv6QdX4MeWWhiIiSx48rJ7jtInkuDwne2VoaEiKiorScp9u5fyFk/otH++8ZsSBlUz31Hn9Vrft71niwEp+I5szfv0z+h8ZH9narhp/4+7N6hp+eC67lbH23JuKef8eV8/J/eKlf0Pkd1lc+B+q8f0fGFaNv9KMPb1sKwAASC8aOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMER9EZhskFNynXrOpz7eNfOgt9n9ndvUNXKqb1LP0Yoff8nxGpnihbeuV8+58xrd9+fE+Hl1jYd/3qCes6jkjGp8fPC0ukY2urPpnx2vUfHUqOM1cHkLWw86XqP76x9Qz9lU8qJ6zoHbFyln6PaKv1K8YgcAwBAaOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAISYvAtPzhSXqOduKn1SN3//12eoav9q5Uj0nOKSLaMmD6hJZ6+nBKvWch+bpLgJTOesadY3EL4rVc+KDx9Vz/GDp7F+r52w9q7vYUnD/8+oauLw310dU4/v/IODQSqb88ON/4XgNEZHdn9RdHKz0685czIlX7AAAGEJjBwDAEBo7AACG0NgBADCExg4AgCE0dgAADKGxAwBgCI0dAABDaOwAABhCYwcAwBAaOwAAhmTFXvGvf2aVavyv7tuhrlF96D7V+DLR7+3d85Fvq+fUPPpF9Ry/yLv9VfWcNes/rxp/tiZHXSOVx997RZfzwtaD6hrZaGneoHrO3nM3q8b3tr5PXeOGvz+nnhM/rrtOQbYqPPGGavzCL15Q1/hm5d+p52hteqBJPaf0ycx4XvKKHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGEJjBwDAEBo7AACGZMVFYEJDCdX4E+Pn1TWOr/qeavzWn9+krpGK6/+uWzU+7tA6rCh48qeq8fMk4tBKpruwcMyVOtnmB0PvV8/5zsIfq8Zv/fhpdY2H7tNf0OX2jY2q8cH9z6trZALtxW7ybtfXqOy/RjW+9qEvqGtc++Qh9ZxMwSt2AAAMobEDAGCIqrG3trZKIBCYdquqqnJqbfAAGfsDOdtHxv6lfo+9urpannnmmak7yM2Kt+mhQMb+QM72kbE/qVPOzc2V0tJSJ9aCDEHG/kDO9pGxP6nfYz958qSEw2GpqKiQhoYG6e3tvezY0dFRicVi027IfJqMRcg5W/Fcto+M/UnV2CORiHR0dEhnZ6e0t7dLT0+PrFmzRoaHhy85vq2tTYqLiydv5eXlaVk0nKPNWIScsxHPZfvI2L9Ujb2+vl7uvvtuWbZsmaxdu1b27dsnb7zxhuzZs+eS45ubm2VoaGjyFo1G07JoOEebsQg5ZyOey/aRsX9d1Scp5s6dK5WVldLdfelNVEKhkIRCoaspAY/NlLEIOVvAc9k+MvaPq/o99pGRETl16pQsWLAgXetBhiFjfyBn+8jYP1SNfcuWLbJ//3555ZVX5ODBg7J+/XrJycmRjRs3OrU+uIyM/YGc7SNj/1L9KL6vr082btwo586dk/nz58vq1avl8OHDMn/+fKfWB5eRsT+Qs31k7F+qxr5r1y6n1vE7aS/c8aUnf19dI3HLzarx2//mG+oa1YfuU88pGzyunnM1vMo4Fa9/ZpV6jvaCQkv+5wvqGqko+z85rtS5KFty/tt/vE09R3uBlqcH9bux3VV8TD3n5Tt1718v2a8uMU22ZHxi50r9nPGfqMbP++EpdY1svqAWe8UDAGAIjR0AAENo7AAAGEJjBwDAEBo7AACG0NgBADCExg4AgCE0dgAADKGxAwBgCI0dAABDaOwAABhyVddjt2TW2TdV4ytnXaOu8a7vzlHPweWd+YNx9Zyej3zbgZVMV32oQT2nTHk9BL+4of3S1w7/nXMWfk41/p9u+0t1jc+f+KR6TsVTo+o5fnDvyh+r5/xxyxbV+GsHD6lrZDNesQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGOLqXvHJZFJERCZkXCTpZuWZJeO6fZxjwwl1jYnxC/o5Sf1+6Kr7l9/c/8Vs0sGtnBNv6b+fqeSmFX9Tvye4kzlnc8bJxJh6jvZxMZLKc/l8Cvu+T+jWFVQ+JtKds1sZXxjRP/bjY7rvpdPnUbdcacaBZDqf7TPo6+uT8vJyt8pBIRqNSllZWVrui5wzExn7Q7pyJuPMNVPGrjb2RCIh/f39UlhYKIFAYNq/xWIxKS8vl2g0KkVFRW4tKSN4eezJZFKGh4clHA5LMJied2YulzMZk7FlXh97unPmfP1OXh/3lWbs6o/ig8HgjP+TLCoq8tUD5e28Ovbi4uK03t9MOZMxGVvm5bGnM2fO15eX6Rnz4TkAAAyhsQMAYEjGNPZQKCQtLS0SCoW8Xorr/HLsfjnOS/HLsfvlOC/FT8fup2N9u2w5blc/PAcAAJyVMa/YAQDA1aOxAwBgCI0dAABDaOwAABhCYwcAwJCMaOzbt2+XxYsXS35+vkQiETly5IjXS3Jca2urBAKBabeqqiqvl+UocrafMxmTsUXZlrHnjX337t3S1NQkLS0tcuzYMampqZG1a9fK6dOnvV6a46qrq+W1116bvB04cMDrJTmGnO3nTMZkbFlWZZz0WG1tbfL++++f/Hs8Hk+Gw+FkW1ubh6tyXktLS7KmpsbrZbiGnO0jY/vIODt4+op9bGxMjh49KnV1dZNfCwaDUldXJ4cOHfJwZe44efKkhMNhqaiokIaGBunt7fV6SY4gZ/s5kzEZW5dNGXva2M+ePSvxeFxKSkqmfb2kpEQGBgY8WpU7IpGIdHR0SGdnp7S3t0tPT4+sWbNGhoeHvV5a2pGz/ZzJmIwty7aMXb1sK6bU19dP/nnZsmUSiURk0aJFsmfPHtm0aZOHK0M6kbN9ZGxftmXs6Sv2efPmSU5OjgwODk77+uDgoJSWlnq0Km/MnTtXKisrpbu72+ulpB05T7GaMxlPIWP7Mj1jV1+xJxIJ6e/vl8LCQgkEAiIisnz5cvnhD38oH/7whyfHPP3003LfffdJLBZzc3meGhkZke7ubrn77rtdPe5kMinDw8MSDoclGEzP//PI+fK8yJmM3WXluUzGl5fpGbt6dbe+vj4pLy93qxwUotGolJWVpeW+yDkzkbE/pCtnMs5cM2Xs6iv2wsJCERFZLXdIrsxyrE73jpvVc9pW/YNq/MP/tEFd4z2PvaKeEz99Rj1HY0LG5YDsm8wmHdzKeWyv/qSzcM7rqvEDt42oa2SabM4457r56jkvteqa2pO3tKtrfLn7bvWcvHVR9RyNdOfsVsZu2PT8y+o5L15YoJ5zeN1C1Xjt+f1KM3a1sV/8cU6uzJLcgHMPlODsfPWcgsIcXY18fY3cYJ56TsDB75OIiCQv1gmk7S7dyjlxTUg9J2+OLgMn1++aLM44J4XnjPb5P6dQ/2Pr3BQee44/ltKcs1sZu0F7fhcRyc/VH7P2HK8+v19hxim9EePHLQX9hoztI2N/IGf/UTd2P28p6BdkbB8Z+wM5+5O6sT/22GNy7733SmNjoyxdulSeeOIJKSgokJ07dzqxPniAjO0jY38gZ39SNXbtloKjo6MSi8Wm3ZDZUtk2kpyzCxn7A+dr/1I1du2Wgm1tbVJcXDx541cnMl8q20aSc3YhY3/gfO1fju4819zcLENDQ5O3aNTZX/eAN8jZPjK2j4ztUP26m3ZLwVAoJKGQ/tdC4J1Uto0k5+xCxv7A+dq/VK/Y8/LyZMWKFdLV1TX5tUQiIV1dXbJq1aq0Lw7uI2P7yNgfyNm/1BvUNDU1yT333CMrV66U2tpa2bZtm5w/f14aGxudWB88QMb2kbE/kLM/qRv7hg0b5MyZM/LII4/IwMCALF++XDo7O9/xAQ1kLzK2j4z9gZz9ydWLwMRiMSkuLpYPyTpHtygMH9bvlbzu3c87sJLp9p7T72Hf/4FhB1YyZSI5Ls/JXhkaGpKioqK03GeqOedU36Sqs+/p3dqluWLrWd1xiIjsXzbbgZX8RiZlrDX29CL1nGer96rG37h7s7rGX3z0u+o5LTs+rRpf+vWDqvHpztmtjFPx+md0byUc2aq/HkAq7rhddw2R+PGXVOOvNGNPr8cOAADSi8YOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMUV8EJhs894J+r+4jxQtV48v+8Li6xuOvdqrnbFrfpBpf8ORP1TUyxfi8AsdrNPauUY0/8mvd40JE5M+X6fYqFxHZL0vUc7KR9noAz1brrwdQfahBNX7Jg4fVNR4s/oR6jrxvTDX8nVdMx0UPPfy3jtfQnitE9Hu/O4VX7AAAGEJjBwDAEBo7AACG0NgBADCExg4AgCE0dgAADKGxAwBgCI0dAABDaOwAABhCYwcAwBAaOwAAhtDYAQAwxORFYJb8dVw95+nvf081vvGw/gIBL4yVqOcUnnhDNV5/5Jlj1ou/drzG4LrZqvG1e3vVNZbmDarniE8uAiNnX3e8xLu+O8fxGsEhk6fOtMgpuU41/tUd89U17rzmZ+o5fsIrdgAADKGxAwBgCI0dAABDaOwAABhCYwcAwBAaOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ0xueHzhXXmO1/jOwh+r59xx+wb1nPjxl9RzslV88LRq/NazN6lr7Hv+R6rxN3R+Tl2jeUGnek5Ote5YsvVxMfzBG7xeAhw2XnW9anzt9d3qGk+d110P4M5rRtQ1nntBf36plH9Tz3ECr9gBADCExg4AgCE0dgAADKGxAwBgCI0dAABDaOwAABhCYwcAwBAaOwAAhtDYAQAwhMYOAIAhNHYAAAyhsQMAYEhWXAQmccvNqvE/3v5NdY0bd29Wjc9fOKyu0fB9/QUCDmxcrhqfrRcHScX+ZbPVc569pVE1vnK/PrO1O7+snrN42xnV+Lzb1SUyQuHBHsdrjBbrXq8UllynrrHwv7ymnpP7Z+9Sz8lGwf3Pq8b3f0BfY+tnPqUaf+fWdnWNf7rtL9VzviS/r57jBF6xAwBgCI0dAABDVI29tbVVAoHAtFtVVZVTa4MHyNgfyNk+MvYv9Xvs1dXV8swzz0zdQW5WvE0PBTL2B3K2j4z9SZ1ybm6ulJaWXtHY0dFRGR0dnfx7LBbTloMHNBmLkHO24rlsHxn7k/o99pMnT0o4HJaKigppaGiQ3t7ey45ta2uT4uLiyVt5eflVLRbu0GQsQs7ZiueyfWTsT6rGHolEpKOjQzo7O6W9vV16enpkzZo1Mjx86V/9am5ulqGhoclbNBpNy6LhHG3GIuScjXgu20fG/qX6UXx9ff3kn5ctWyaRSEQWLVoke/bskU2bNr1jfCgUklAodPWrhGu0GYuQczbiuWwfGfvXVf2629y5c6WyslK6u7vTtR5kGDL2B3K2j4z946oa+8jIiJw6dUoWLFiQrvUgw5CxP5CzfWTsH6rGvmXLFtm/f7+88sorcvDgQVm/fr3k5OTIxo0bnVofXEbG/kDO9pGxf6neY+/r65ONGzfKuXPnZP78+bJ69Wo5fPiwzJ8/36n1wWVk7A/kbB8Z+5eqse/atcupdfxOs178tWr8ifHz6ho3fe1l1fjxquvVNR76vv4CLTd+7lbV+CUPqktM41XGbtFeoOLEzpXqGqlcPGLTA02q8XnyqrrG23mVc3zwtGp8Y+8adY3qzb9UjT/ysYXqGpLCr3iXKR97V8vyczk0lHC8xgtjJY7XcAp7xQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGKLaK94r2v2lP3/ik+oazz6/VzU+lf3obz2uX5d2D/u4ukL2SmUf9w8t1e3Xf0vBj9U17v/0f1PPKdj/U/UcPxhcN1s959UduoucNLzn39Q1nv3SB9VzkD6FB3tU47eevUld46F5+mt7fKvkOtV4bW+7UrxiBwDAEBo7AACG0NgBADCExg4AgCE0dgAADKGxAwBgCI0dAABDaOwAABhCYwcAwBAaOwAAhtDYAQAwxNW94pPJpIiITMi4SNK5OhPnR9VzYsMJ1fiRcd14kdTWNZEYU42PJ8d19y+/GX8xm3RwK+fEWxfUc8ZGdN/PCwnd91NEZGJCv66gMjeNbM44qXz8i4jE39Q9zy6MZH/GIunP2a2MU6F9XKSScSyUwjk+Q87XgWQ6n+0z6Ovrk/LycrfKQSEajUpZWVla7oucMxMZ+0O6cibjzDVTxq429kQiIf39/VJYWCiBQGDav8ViMSkvL5doNCpFRUVuLSkjeHnsyWRShoeHJRwOSzCYnndmLpczGZOxZV4fe7pz5nz9Tl4f95Vm7OqP4oPB4Iz/kywqKvLVA+XtvDr24uLitN7fTDmTMRlb5uWxpzNnzteXl+kZ8+E5AAAMobEDAGBIxjT2UCgkLS0tEgqFvF6K6/xy7H45zkvxy7H75TgvxU/H7qdjfbtsOW5XPzwHAACclTGv2AEAwNWjsQMAYAiNHQAAQ2jsAAAYkhGNffv27bJ48WLJz8+XSCQiR44c8XpJjmttbZVAIDDtVlVV5fWyHEXO9nMmYzK2KNsy9ryx7969W5qamqSlpUWOHTsmNTU1snbtWjl9+rTXS3NcdXW1vPbaa5O3AwcOeL0kx5Cz/ZzJmIwty6qMkx6rra1N3n///ZN/j8fjyXA4nGxra/NwVc5raWlJ1tTUeL0M15CzfWRsHxlnB09fsY+NjcnRo0elrq5u8mvBYFDq6urk0KFDHq7MHSdPnpRwOCwVFRXS0NAgvb29Xi/JEeRsP2cyJmPrsiljTxv72bNnJR6PS0lJybSvl5SUyMDAgEerckckEpGOjg7p7OyU9vZ26enpkTVr1sjw8LDXS0s7crafMxmTsWXZlrGrV3fDlPr6+sk/L1u2TCKRiCxatEj27NkjmzZt8nBlSCdyto+M7cu2jD19xT5v3jzJycmRwcHBaV8fHByU0tJSj1bljblz50plZaV0d3d7vZS0I+cpVnMm4ylkbF+mZ+xpY8/Ly5MVK1ZIV1fX5NcSiYR0dXXJqlWrPFyZ+0ZGRuTUqVOyYMECr5eSduQ8xWrOZDyFjO3L9Ixd/VF8IpGQ/v5+KSwslEAgICIimzdvls2bN0t1dbWsWLFCduzYISMjI3LXXXdJLBZzc3muevjhh6W+vl7Ky8tlYGBAtm7dKoFAQD760Y+6etzJZFKGh4clHA5LMJie/+eR85RMyJmMnZUJGYukP2cynpJtGbt6dbe+vj4pLy93qxwUotGolJWVpeW+yDkzkbE/pCtnMs5cM2Xs6iv2wsJCERFZLXdIrsxyrM4bDbXqOVWffUE1/szG2eoa8dNn1HOcNiHjckD2TWaTDqnmnPPe96jqvPSlOarxIiJP3tKuGv/i2HXqGs2H/lA9p+Lv4qrxwQM/v+KxmZSxG17+2u+pxv/Dxx5X1/iTWz+mnuP08z/dOaeasfb8qz33iojcWKD7Xm55tzvvhd+17uOq8fFfnVSNv9KMU2rs27dvl0cffVQGBgakpqZGHn/8camtnTnMiz/OyZVZkhtw7mSQk5evnpM3J081PjeoGy8iEnDwmFP225/XXMzmolQzfvt9aXPOyQld8VgRkeBsfc5zCnU/oiwYy1HXSGVdubnKxq55LGVQxm4I5uu+/9rHhEiGPv/TnHPKz2Pl+Vd77hURyS/QfS+LUsg4FbnKc5j6MXGZjP8z9dH6eUtBvyBj+8jYH8jZn9SN/bHHHpN7771XGhsbZenSpfLEE09IQUGB7Ny504n1wQNkbB8Z+wM5+5OqsWu3FBwdHZVYLDbthsyWyraR5JxdyNgfOF/7l6qxa7cUbGtrk+Li4skbn7DMfKlsG0nO2YWM/YHztX85+omC5uZmGRoamrxFo1Eny8Ej5GwfGdtHxnaoPhWv3VIwFApJKKT7lCC8lcq2keScXcjYHzhf+5fqFTtbCtpHxvaRsT+Qs3+pf4+9qalJ7rnnHlm5cqXU1tbKtm3b5Pz589LY2OjE+uABMraPjP2BnP1J3dg3bNggZ86ckUceeUQGBgZk+fLl0tnZ+Y4PaCB7kbF9ZOwP5OxPru4VH4vFpLi4WD4k6xzdrerxV3+invPCmO6B3vy9T6trLGw9qJ7jtInkuDwne2VoaEiKiorScp+p5vzm+oiqTv2fPqdcmchfdd2qGp8onlDX6PnIt9Vzbty9WTV+yYOHr3hsJmWslVOi39L3vgO659kLb12vrnHg9kXqOVrxQd0mMunOOdWMx57WfW+erd6rXZqcGD+vGl//j19R1wj/i741Fjz5U/UcjSvN2NPLtgIAgPSisQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAEPUF4HJBj8Yer96jnbv5w/s/YW6Rn+reoqvaPdZ3v/kbHWNOQ/q/i/7p1/cpa6h3cdaRKTiqVH1HD94dcd89ZyleYMzD3qb7Z+7W11j+5FvqOd8/sQnVePzbleXyAi9v1ygGv/U4jnqGn/5yjrV+Ju+9rK6hnav/kzCK3YAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGJIVF4HJqb5JNf57J/WHVTZ4XDV+3bu71TXaZYl6DtLrXXf82vEaD9zxWfWc4PHnHVhJ5ult/aBq/K9W7VDXeO+3/odq/A0v6p/LlbOuUc/RXhxlibyqrpGN7rxmRD+neq9q/FMH9BeaaX9P9p6vecUOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGBIVuwVHz/+kmr8okd0e8uLiMSV41PZ3/hbJdep58QHT6vn4PJmP5CvGr9036C6xlvbLqjn5N2unpKVLiwcc7zGpz7epRq/tMH56weIiLz73wOu1PHaTV97WTW+pveLDq1kyr//D/01B9odWIdbeMUOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAEOy4iIwWtqLxoiIDDz4QdX4E+M/Udfggi7e0z42Hrjjs+oa39y3Uz1n0/om1fiCJ3+qrpEJ3tvcqxpfXdygrvHkym+pxlfOukZd46nzc9Rzru04pJ6TjbTnudKv68+Lr39mlXqOVuKWm9Vzgvufd2AlerxiBwDAEBo7AACGqBp7a2urBAKBabeqqiqn1gYPkLE/kLN9ZOxf6vfYq6ur5Zlnnpm6g1yTb9P7Ghn7AznbR8b+pE45NzdXSktLnVgLMgQZ+wM520fG/qR+j/3kyZMSDoeloqJCGhoapLf38p9yHR0dlVgsNu2GzKfJWIScsxXPZfvI2J9UjT0SiUhHR4d0dnZKe3u79PT0yJo1a2R4ePiS49va2qS4uHjyVl5enpZFwznajEXIORvxXLaPjP1L1djr6+vl7rvvlmXLlsnatWtl37598sYbb8iePXsuOb65uVmGhoYmb9FoNC2LhnO0GYuQczbiuWwfGfvXVX2SYu7cuVJZWSnd3d2X/PdQKCShUOhqSsBjM2UsQs4W8Fy2j4z946p+j31kZEROnTolCxYsSNd6kGHI2B/I2T4y9g9VY9+yZYvs379fXnnlFTl48KCsX79ecnJyZOPGjU6tDy4jY38gZ/vI2L9UP4rv6+uTjRs3yrlz52T+/PmyevVqOXz4sMyfP9+p9cFlZOwP5GwfGfuXqrHv2rXLqXWk1YmdK9Vzej6yQ1djXF0ipXUFh3Qfg7jp26+rxifjoyK/mvq7lxnnlFynGn+2/kZ1jdFrA6rxGxq71DVSuahIbHGOanyBusJ0XuWsvUBI2R/qLxDyQMl61fh9z/9IXePhn69TzymT4+o5V8OrjN14Hn/3T/+XavzWs+9X18iUC7qkgr3iAQAwhMYOAIAhNHYAAAyhsQMAYAiNHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMuarrsWeqJX8dV8+5tVy393PvL/WXPrz3tmfVc06+qdt3uftflqrGT4xfmLZXvKfmXasaXr35lw4t5Orcely/j3jp1w86sBJ/enWH7iInJ8bPq2u867tz1HP8YviDN6jGa/d9F9Ffj+HAxuXqGiIvpTAnM/CKHQAAQ2jsAAAYQmMHAMAQGjsAAIbQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGEJjBwDAEFf3ik8mkyIiMiHjIknn6iQmLqjnTJwf1dW4oK9xYWRcPWfszTHV+Ilx3brivx1/MZt0SDXnZFyXwdiI7nuTigsJfWbax5KISDCpr3OlJuQ3950JGbsh/qbu+z8ynFDX0D7PREQmHMxYJP05p5qx9nuTyvc/Nks3Z0J5bhERiTucVyquNONAMp3P9hn09fVJeXm5W+WgEI1GpaysLC33Rc6ZiYz9IV05k3HmmiljVxt7IpGQ/v5+KSwslEAgMO3fYrGYlJeXSzQalaKiIreWlBG8PPZkMinDw8MSDoclGEzPOzOXy5mMydgyr4893Tlzvn4nr4/7SjN29UfxwWBwxv9JFhUV+eqB8nZeHXtxcXFa72+mnMmYjC3z8tjTmTPn68vL9Iz58BwAAIbQ2AEAMCRjGnsoFJKWlhYJhUJeL8V1fjl2vxznpfjl2P1ynJfip2P307G+XbYct6sfngMAAM7KmFfsAADg6tHYAQAwhMYOAIAhNHYAAAyhsQMAYEhGNPbt27fL4sWLJT8/XyKRiBw5csTrJTmutbVVAoHAtFtVVZXXy3IUOdvPmYzJ2KJsy9jzxr57925pamqSlpYWOXbsmNTU1MjatWvl9OnTXi/NcdXV1fLaa69N3g4cOOD1khxDzvZzJmMytiyrMk56rLa2Nnn//fdP/j0ejyfD4XCyra3Nw1U5r6WlJVlTU+P1MlxDzvaRsX1knB08fcU+NjYmR48elbq6usmvBYNBqaurk0OHDnm4MnecPHlSwuGwVFRUSENDg/T29nq9JEeQs/2cyZiMrcumjD1t7GfPnpV4PC4lJSXTvl5SUiIDAwMercodkUhEOjo6pLOzU9rb26Wnp0fWrFkjw8PDXi8t7cjZfs5kTMaWZVvGrl62FVPq6+sn/7xs2TKJRCKyaNEi2bNnj2zatMnDlSGdyNk+MrYv2zL29BX7vHnzJCcnRwYHB6d9fXBwUEpLSz1alTfmzp0rlZWV0t3d7fVS0o6cp1jNmYynkLF9mZ6xp409Ly9PVqxYIV1dXZNfSyQS0tXVJatWrfJwZe4bGRmRU6dOyYIFC7xeStqR8xSrOZPxFDK2L+Mz9vrTe7t27UqGQqFkR0dH8oUXXkjed999yblz5yYHBga8XpqjvvKVrySfe+65ZE9PT/InP/lJsq6uLjlv3rzk6dOnvV6aI8jZfs5kTMZWZVvGnjf2ZDKZfPzxx5MLFy5M5uXlJWtra5OHDx/2ekmO27BhQ3LBggXJvLy85PXXX5/csGFDsru72+tlOYqc7edMxmRsUbZlzPXYAQAwxPOd5wAAQPrQ2AEAMITGDgCAITR2AAAMobEDAGAIjR0AAENo7AAAGEJjBwDAEBo7AACG0NgBADCExg4AgCH/H/DzqsqCo/I4AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 600x600 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = load_digits(return_X_y=True)\n",
    "\n",
    "# The sklearn Digits data-set, though it contains digit images, keeps these images in vectors\n",
    "# so we need to reshape them to 2D first. The images are 8x8 px in size and monochrome\n",
    "X = np.expand_dims(X.reshape((-1, 8, 8)), 1)\n",
    "\n",
    "nplot = 4\n",
    "fig, ax = plt.subplots(nplot, nplot, figsize=(6, 6))\n",
    "for i in range(0, nplot):\n",
    "    for j in range(0, nplot):\n",
    "        ax[i, j].imshow(X[i * nplot + j, ::].squeeze())\n",
    "plt.show()\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.25, shuffle=True, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c5e392d",
   "metadata": {},
   "source": [
    "### Define the neural network\n",
    "\n",
    "Since the accumulator bit width in FHE is small, we prune the convolutional\n",
    "filters to limit the number of connections per neuron.\n",
    "\n",
    "Neural network **pruning** is the process by which the synapses of individual neurons in a layer\n",
    "are forced to have a weight equal to zero. This basically eliminates them from the computation \n",
    "and thus they do not increase the accumulator bit width. It has been shown that neural networks can \n",
    "maintain their accuracy with a degree of pruning that can \n",
    "even exceed 70% for some over-parametrized networks such as VGG16 or large ResNets.\n",
    "\n",
    "See: https://arxiv.org/pdf/2003.03033.pdf, Figure 8 in Section 7.2, for an evaluation on the \n",
    "simple pruning method used in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f43e2387",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TinyCNN(nn.Module):\n",
    "    \"\"\"A very small CNN to classify the sklearn digits data-set.\"\"\"\n",
    "\n",
    "    def __init__(self, n_classes) -> None:\n",
    "        \"\"\"Construct the CNN with a configurable number of classes.\"\"\"\n",
    "        super().__init__()\n",
    "\n",
    "        # This network has a total complexity of 1216 MAC\n",
    "        self.conv1 = nn.Conv2d(1, 8, 3, stride=1, padding=0)\n",
    "        self.conv2 = nn.Conv2d(8, 16, 3, stride=2, padding=0)\n",
    "        self.conv3 = nn.Conv2d(16, 32, 2, stride=1, padding=0)\n",
    "        self.fc1 = nn.Linear(32, n_classes)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"Run inference on the tiny CNN, apply the decision layer on the reshaped conv output.\"\"\"\n",
    "        x = self.conv1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.conv2(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.conv3(x)\n",
    "        x = torch.relu(x)\n",
    "        x = x.flatten(1)\n",
    "        x = self.fc1(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1449d54",
   "metadata": {},
   "source": [
    "### Train the CNN"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "518771f3",
   "metadata": {},
   "source": [
    "Note that the training code for quantization aware training is the same as it would be for\n",
    "floating point training. Indeed, the Brevitas layers used in the CNN class will handle\n",
    "quantization during training. \n",
    "\n",
    "We train the network for varying weights and activations bit-width, to find an FHE compatible \n",
    "configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3035684",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   0%|          | 0/150 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   1%|          | 1/150 [00:00<00:18,  8.23it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   1%|▏         | 2/150 [00:00<00:17,  8.59it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   2%|▏         | 3/150 [00:00<00:16,  8.73it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   3%|▎         | 4/150 [00:00<00:16,  8.81it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   3%|▎         | 5/150 [00:00<00:16,  8.72it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   4%|▍         | 6/150 [00:00<00:16,  8.78it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   5%|▍         | 7/150 [00:00<00:16,  8.82it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   5%|▌         | 8/150 [00:00<00:16,  8.85it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   6%|▌         | 9/150 [00:01<00:15,  8.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   7%|▋         | 10/150 [00:01<00:15,  8.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   7%|▋         | 11/150 [00:01<00:15,  8.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   8%|▊         | 12/150 [00:01<00:15,  8.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   9%|▊         | 13/150 [00:01<00:15,  8.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:   9%|▉         | 14/150 [00:01<00:15,  8.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  10%|█         | 15/150 [00:01<00:15,  8.87it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  11%|█         | 16/150 [00:01<00:15,  8.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  11%|█▏        | 17/150 [00:01<00:14,  8.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  12%|█▏        | 18/150 [00:02<00:14,  8.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  13%|█▎        | 19/150 [00:02<00:14,  8.88it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  13%|█▎        | 20/150 [00:02<00:14,  8.83it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  14%|█▍        | 21/150 [00:02<00:14,  8.82it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  15%|█▍        | 22/150 [00:02<00:14,  8.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  15%|█▌        | 23/150 [00:02<00:14,  8.89it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  16%|█▌        | 24/150 [00:02<00:14,  8.87it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  17%|█▋        | 25/150 [00:02<00:14,  8.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  17%|█▋        | 26/150 [00:02<00:13,  8.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  18%|█▊        | 27/150 [00:03<00:13,  8.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  19%|█▊        | 28/150 [00:03<00:13,  8.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  19%|█▉        | 29/150 [00:03<00:13,  8.87it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  20%|██        | 30/150 [00:03<00:13,  8.92it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  21%|██        | 31/150 [00:03<00:13,  8.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  21%|██▏       | 32/150 [00:03<00:13,  8.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  22%|██▏       | 33/150 [00:03<00:13,  9.00it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  23%|██▎       | 34/150 [00:03<00:12,  8.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  23%|██▎       | 35/150 [00:03<00:12,  8.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  24%|██▍       | 36/150 [00:04<00:12,  8.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  25%|██▍       | 37/150 [00:04<00:12,  8.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  25%|██▌       | 38/150 [00:04<00:12,  8.99it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  26%|██▌       | 39/150 [00:04<00:12,  9.00it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  27%|██▋       | 40/150 [00:04<00:12,  8.99it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  27%|██▋       | 41/150 [00:04<00:12,  8.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  28%|██▊       | 42/150 [00:04<00:12,  8.96it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  29%|██▊       | 43/150 [00:04<00:11,  8.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  29%|██▉       | 44/150 [00:04<00:11,  8.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  30%|███       | 45/150 [00:05<00:11,  8.93it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  31%|███       | 46/150 [00:05<00:11,  8.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  31%|███▏      | 47/150 [00:05<00:11,  8.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  32%|███▏      | 48/150 [00:05<00:11,  8.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  33%|███▎      | 49/150 [00:05<00:11,  8.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  33%|███▎      | 50/150 [00:05<00:11,  8.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  34%|███▍      | 51/150 [00:05<00:11,  8.98it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  35%|███▍      | 52/150 [00:05<00:10,  9.01it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  35%|███▌      | 53/150 [00:05<00:10,  8.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  36%|███▌      | 54/150 [00:06<00:10,  8.80it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  37%|███▋      | 55/150 [00:06<00:10,  8.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  37%|███▋      | 56/150 [00:06<00:10,  8.89it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  38%|███▊      | 57/150 [00:06<00:10,  8.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  39%|███▊      | 58/150 [00:06<00:10,  8.96it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  39%|███▉      | 59/150 [00:06<00:10,  8.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  40%|████      | 60/150 [00:06<00:10,  8.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  41%|████      | 61/150 [00:06<00:10,  8.86it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  41%|████▏     | 62/150 [00:06<00:09,  8.90it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  42%|████▏     | 63/150 [00:07<00:09,  8.92it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  43%|████▎     | 64/150 [00:07<00:09,  9.04it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  43%|████▎     | 65/150 [00:07<00:09,  9.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  44%|████▍     | 66/150 [00:07<00:09,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  45%|████▍     | 67/150 [00:07<00:09,  9.18it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  45%|████▌     | 68/150 [00:07<00:08,  9.16it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  46%|████▌     | 69/150 [00:07<00:08,  9.18it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  47%|████▋     | 70/150 [00:07<00:08,  9.27it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  47%|████▋     | 71/150 [00:07<00:08,  9.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  48%|████▊     | 72/150 [00:08<00:08,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  49%|████▊     | 73/150 [00:08<00:08,  9.18it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  49%|████▉     | 74/150 [00:08<00:08,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  50%|█████     | 75/150 [00:08<00:08,  9.24it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  51%|█████     | 76/150 [00:08<00:07,  9.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  51%|█████▏    | 77/150 [00:08<00:07,  9.13it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  52%|█████▏    | 78/150 [00:08<00:07,  9.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  53%|█████▎    | 79/150 [00:08<00:07,  9.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  53%|█████▎    | 80/150 [00:08<00:07,  9.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  54%|█████▍    | 81/150 [00:09<00:07,  9.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  55%|█████▍    | 82/150 [00:09<00:07,  9.02it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  55%|█████▌    | 83/150 [00:09<00:07,  9.04it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  56%|█████▌    | 84/150 [00:09<00:07,  9.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  57%|█████▋    | 85/150 [00:09<00:07,  8.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  57%|█████▋    | 86/150 [00:09<00:07,  8.95it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  58%|█████▊    | 87/150 [00:09<00:06,  9.07it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  59%|█████▊    | 88/150 [00:09<00:06,  9.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  59%|█████▉    | 89/150 [00:09<00:06,  9.19it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  60%|██████    | 90/150 [00:10<00:06,  9.24it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  61%|██████    | 91/150 [00:10<00:06,  9.24it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  61%|██████▏   | 92/150 [00:10<00:06,  9.11it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  62%|██████▏   | 93/150 [00:10<00:06,  9.09it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  63%|██████▎   | 94/150 [00:10<00:06,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  63%|██████▎   | 95/150 [00:10<00:05,  9.19it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  64%|██████▍   | 96/150 [00:10<00:05,  9.21it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  65%|██████▍   | 97/150 [00:10<00:05,  9.27it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  65%|██████▌   | 98/150 [00:10<00:05,  9.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  66%|██████▌   | 99/150 [00:10<00:05,  9.26it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  67%|██████▋   | 100/150 [00:11<00:05,  9.26it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  67%|██████▋   | 101/150 [00:11<00:05,  9.30it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  68%|██████▊   | 102/150 [00:11<00:05,  9.37it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  69%|██████▊   | 103/150 [00:11<00:05,  9.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  69%|██████▉   | 104/150 [00:11<00:04,  9.40it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  70%|███████   | 105/150 [00:11<00:04,  9.46it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  71%|███████   | 106/150 [00:11<00:04,  9.45it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  71%|███████▏  | 107/150 [00:11<00:04,  9.30it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  72%|███████▏  | 108/150 [00:11<00:04,  9.24it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  73%|███████▎  | 109/150 [00:12<00:04,  9.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  73%|███████▎  | 110/150 [00:12<00:04,  9.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  74%|███████▍  | 111/150 [00:12<00:04,  9.40it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  75%|███████▍  | 112/150 [00:12<00:04,  9.38it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  75%|███████▌  | 113/150 [00:12<00:03,  9.38it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  76%|███████▌  | 114/150 [00:12<00:03,  9.35it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  77%|███████▋  | 115/150 [00:12<00:03,  9.37it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  77%|███████▋  | 116/150 [00:12<00:03,  9.38it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  78%|███████▊  | 117/150 [00:12<00:03,  9.32it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  79%|███████▊  | 118/150 [00:13<00:03,  9.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  79%|███████▉  | 119/150 [00:13<00:03,  9.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  80%|████████  | 120/150 [00:13<00:03,  9.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  81%|████████  | 121/150 [00:13<00:03,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  81%|████████▏ | 122/150 [00:13<00:03,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  82%|████████▏ | 123/150 [00:13<00:02,  9.20it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  83%|████████▎ | 124/150 [00:13<00:02,  9.20it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  83%|████████▎ | 125/150 [00:13<00:02,  9.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  84%|████████▍ | 126/150 [00:13<00:02,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  85%|████████▍ | 127/150 [00:14<00:02,  9.09it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  85%|████████▌ | 128/150 [00:14<00:02,  9.13it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  86%|████████▌ | 129/150 [00:14<00:02,  8.93it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  87%|████████▋ | 130/150 [00:14<00:02,  9.00it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  87%|████████▋ | 131/150 [00:14<00:02,  9.07it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  88%|████████▊ | 132/150 [00:14<00:01,  9.04it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  89%|████████▊ | 133/150 [00:14<00:01,  9.04it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  89%|████████▉ | 134/150 [00:14<00:01,  8.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  90%|█████████ | 135/150 [00:14<00:01,  9.06it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  91%|█████████ | 136/150 [00:15<00:01,  9.12it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  91%|█████████▏| 137/150 [00:15<00:01,  9.08it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  92%|█████████▏| 138/150 [00:15<00:01,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  93%|█████████▎| 139/150 [00:15<00:01,  9.19it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  93%|█████████▎| 140/150 [00:15<00:01,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  94%|█████████▍| 141/150 [00:15<00:00,  9.16it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  95%|█████████▍| 142/150 [00:15<00:00,  9.20it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  95%|█████████▌| 143/150 [00:15<00:00,  9.16it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  96%|█████████▌| 144/150 [00:15<00:00,  9.20it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  97%|█████████▋| 145/150 [00:15<00:00,  9.18it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  97%|█████████▋| 146/150 [00:16<00:00,  9.15it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  98%|█████████▊| 147/150 [00:16<00:00,  9.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  99%|█████████▊| 148/150 [00:16<00:00,  9.10it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training:  99%|█████████▉| 149/150 [00:16<00:00,  9.08it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training: 100%|██████████| 150/150 [00:16<00:00,  9.12it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Training: 100%|██████████| 150/150 [00:16<00:00,  9.07it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArMAAAGJCAYAAACZ7rtNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABRXUlEQVR4nO3deVhUZf8G8PvMCgMMi8imiKhkikoqLmiW/VJJTdNMW6xc3t4yNTVatdzabDMtLX1bXq3MNEvprdxwz9z3jUwTwQ1QEIZ1GGbO7w9kcmSbA7MwzP25Li6ZM+fMfOc7UDfPPOc5giiKIoiIiIiIXJDM2QUQEREREdUWwywRERERuSyGWSIiIiJyWQyzREREROSyGGaJiIiIyGUxzBIRERGRy2KYJSIiIiKXxTBLRERERC6LYZaIiIiIXBbDLBHVaPTo0WjevHmtjp01axYEQbBtQS6gefPmGD16tLPLsIq93qOG+t6fP38egiBg6dKltTpeEATMmjXLpjURuTOGWSIXJgiCVV/btm1zdqkub/ny5Zg/f76zyyAr8f0ich+CKIqis4sgotpZtmyZxe1vvvkGSUlJ+Pbbby229+3bF8HBwbV+HoPBAJPJBLVaLfnY0tJSlJaWwsPDo9bPXx/cf//9OHHiBM6fP2/V/s2bN0fv3r1rPXrnSLNmzcLs2bNh6/8dOPO9l/p+SSGKIvR6PZRKJeRyueTji4uLoVAooFAobF4bkTvibxKRC3v88cctbu/ZswdJSUkVtt+qsLAQGo3G6udRKpW1qg8A/6fthgoKCuDl5eUy731xcTFUKhVkMus+rBQEoU4B3dX/sCOqbzjNgKiB6927N9q1a4eDBw/irrvugkajwbRp0wAAP//8MwYOHIiwsDCo1Wq0bNkSb775JoxGo8Vj3DpntnzO4IcffojPP/8cLVu2hFqtRpcuXbB//36LYyubNykIAiZOnIjExES0a9cOarUa0dHRWL9+fYX6t23bhtjYWHh4eKBly5b4z3/+Y/VczDNnzmDYsGEICQmBh4cHmjZtikceeQS5ubkW+y1btgydO3eGp6cnAgIC8Mgjj+DChQsWPfztt9+QmppqnrpRmznE586dw/DhwxEQEACNRoPu3bvjt99+q7DfggULEB0dDY1GA39/f8TGxmL58uXm+/Py8jBlyhQ0b94carUaQUFB6Nu3Lw4dOlRjDTt37kSXLl0s+nmr6uaE3jrfs/y9OHXqFB577DH4+/vjzjvvtLjv1uPt/d5X935t27YNgiBgxYoVeP3119GkSRNoNBrodDpkZ2fjxRdfRPv27eHt7Q2tVov+/fvj6NGjNfZn9OjR8Pb2xqVLlzBkyBB4e3ujcePGePHFFyv8PlXVw7Nnz2L06NHw8/ODr68vxowZg8LCQotji4qKMGnSJAQGBsLHxweDBw/GpUuXOA+X3Fr9/5OZiOosKysL/fv3xyOPPILHH3/cPOVg6dKl8Pb2RkJCAry9vbFlyxbMmDEDOp0OH3zwQY2Pu3z5cuTl5eGZZ56BIAh4//338eCDD+LcuXM1jubu3LkTq1evxvjx4+Hj44NPPvkEw4YNQ1paGho1agQAOHz4MO677z6EhoZi9uzZMBqNeOONN9C4ceMaayspKUF8fDz0ej2ee+45hISE4NKlS/j111+Rk5MDX19fAMDbb7+N6dOnY8SIEXjqqadw9epVLFiwAHfddRcOHz4MPz8/vPbaa8jNzcXFixcxb948AIC3t3eNNdwsIyMDPXr0QGFhISZNmoRGjRrh66+/xuDBg/Hjjz9i6NChAIAvvvgCkyZNwkMPPYTJkyejuLgYx44dw969e/HYY48BAMaNG4cff/wREydORNu2bZGVlYWdO3ciOTkZnTp1qrKG48ePo1+/fmjcuDFmzZqF0tJSzJw5s05TUMoNHz4cUVFReOedd2qcrmDv996a9+vNN9+ESqXCiy++CL1eD5VKhVOnTiExMRHDhw9HZGQkMjIy8J///Ad33303Tp06hbCwsGqf12g0Ij4+Ht26dcOHH36ITZs2Ye7cuWjZsiWeffbZGuseMWIEIiMjMWfOHBw6dAhffvklgoKC8N5775n3GT16NH744Qc88cQT6N69O7Zv346BAwfW+NhEDZpIRA3GhAkTxFt/re+++24RgLh48eIK+xcWFlbY9swzz4gajUYsLi42bxs1apQYERFhvp2SkiICEBs1aiRmZ2ebt//8888iAPGXX34xb5s5c2aFmgCIKpVKPHv2rHnb0aNHRQDiggULzNsGDRokajQa8dKlS+ZtZ86cERUKRYXHvNXhw4dFAOKqVauq3Of8+fOiXC4X3377bYvtx48fFxUKhcX2gQMHWvSgJhEREeKoUaPMt6dMmSICEH///Xfztry8PDEyMlJs3ry5aDQaRVEUxQceeECMjo6u9rF9fX3FCRMmWF1LuSFDhogeHh5iamqqedupU6dEuVxu0c/y93fJkiUVHgOAOHPmTPPt8vf30UcfrbCvs957Uaz6/dq6dasIQGzRokWFn//i4mLz+1AuJSVFVKvV4htvvGGx7db+jBo1SgRgsZ8oimLHjh3Fzp07V+hBZT0cO3asxX5Dhw4VGzVqZL598OBBEYA4ZcoUi/1Gjx5d4TGJ3AmnGRC5AbVajTFjxlTY7unpaf4+Ly8P165dQ69evVBYWIg///yzxsd9+OGH4e/vb77dq1cvAGUfp9ekT58+aNmypfl2hw4doNVqzccajUZs2rQJQ4YMsRgRa9WqFfr371/j45ePvG7YsKHCR7XlVq9eDZPJhBEjRuDatWvmr5CQEERFRWHr1q01Po+11q5di65du5o/ggfKRguffvppnD9/HqdOnQIA+Pn54eLFixWma9zMz88Pe/fuxeXLl61+fqPRiA0bNmDIkCFo1qyZeXubNm0QHx9fi1dkady4cVbva+/33hqjRo2y+PkHyn5PyufNGo1GZGVlwdvbG61bt7ZqCgdQsQ+9evWy6vehqmOzsrKg0+kAwDwVY/z48Rb7Pffcc1Y9PlFDxTBL5AaaNGkClUpVYfvJkycxdOhQ+Pr6QqvVonHjxuaTx26dV1qZm0MRAHOwvX79uuRjy48vPzYzMxNFRUVo1apVhf0q23aryMhIJCQk4Msvv0RgYCDi4+Px6aefWryuM2fOQBRFREVFoXHjxhZfycnJyMzMrPF5rJWamorWrVtX2N6mTRvz/QDwyiuvwNvbG127dkVUVBQmTJiAP/74w+KY999/HydOnEB4eDi6du2KWbNm1RiYrl69iqKiIkRFRVW4r7K6pIqMjLR6X3u/99aorF6TyYR58+YhKioKarUagYGBaNy4MY4dO2bV74OHh0eFaRA3v66a1PT7lJqaCplMVqF2W/WEyFUxzBK5gVtHoAAgJycHd999N44ePYo33ngDv/zyC5KSkszz80wmU42PW9WyRKIVSzzV5VhrzZ07F8eOHcO0adPMJ85ER0fj4sWLAMpeoyAIWL9+PZKSkip8VXZylL21adMGp0+fxooVK3DnnXfip59+wp133omZM2ea9xkxYgTOnTuHBQsWICwsDB988AGio6Oxbt06m9RQ1QlWt57IdLPKfsaq4oj3viaV1fvOO+8gISEBd911F5YtW4YNGzYgKSkJ0dHRdfp9sFZ96AuRK+IJYERuatu2bcjKysLq1atx1113mbenpKQ4sap/BAUFwcPDA2fPnq1wX2XbqtK+fXu0b98er7/+Onbt2oWePXti8eLFeOutt9CyZUuIoojIyEjcdttt1T5OXa9kFRERgdOnT1fYXj6dIyIiwrzNy8sLDz/8MB5++GGUlJTgwQcfxNtvv42pU6eal3UKDQ3F+PHjMX78eGRmZqJTp054++23q/wYvnHjxvD09MSZM2cq3HdrXeUjgjk5ORbby0eP7c0W731t3q8ff/wR99xzD7766iuL7Tk5OQgMDJT8eLYWEREBk8mElJQUixF2Kb8PRA0RR2aJ3FT5KNDNoz4lJSX47LPPnFWSBblcjj59+iAxMdFibujZs2etGoHU6XQoLS212Na+fXvIZDLo9XoAwIMPPgi5XF7pBQNEUURWVpb5tpeXl1UfNVdlwIAB2LdvH3bv3m3eVlBQgM8//xzNmzdH27ZtAcDiOQFApVKhbdu2EEURBoMBRqOxQh1BQUEICwszv67KyOVyxMfHIzExEWlpaebtycnJ2LBhg8W+Wq0WgYGB2LFjh8V2R/1s1PW9B2r3fsnl8go/B6tWrcKlS5ckPY69lM9tvvV9WLBggTPKIao3ODJL5KZ69OgBf39/jBo1CpMmTYIgCPj222/r1Ueas2bNwsaNG9GzZ088++yzMBqNWLhwIdq1a4cjR45Ue+yWLVswceJEDB8+HLfddhtKS0vx7bffQi6XY9iwYQCAli1b4q233sLUqVNx/vx5DBkyBD4+PkhJScGaNWvw9NNP48UXXwQAdO7cGStXrkRCQgK6dOkCb29vDBo0yOrX8uqrr+L7779H//79MWnSJAQEBODrr79GSkoKfvrpJ/OJR/369UNISAh69uyJ4OBgJCcnY+HChRg4cCB8fHyQk5ODpk2b4qGHHkJMTAy8vb2xadMm7N+/H3Pnzq22htmzZ2P9+vXo1asXxo8fj9LSUvOatseOHbPY96mnnsK7776Lp556CrGxsdixYwf++usvq19vXdXlvQdq937df//9eOONNzBmzBj06NEDx48fx3fffYcWLVrY6FXVTefOnTFs2DDMnz8fWVlZ5qW5yt+Xun56QOSqGGaJ3FSjRo3w66+/4oUXXsDrr78Of39/PP7447j33nttcna7LXTu3Bnr1q3Diy++iOnTpyM8PBxvvPEGkpOTa1xtISYmBvHx8fjll19w6dIlaDQaxMTEYN26dejevbt5v1dffRW33XYb5s2bh9mzZwMAwsPD0a9fPwwePNi83/jx43HkyBEsWbIE8+bNQ0REhKQwGxwcjF27duGVV17BggULUFxcjA4dOuCXX36xWCf0mWeewXfffYePPvoI+fn5aNq0KSZNmoTXX38dAKDRaDB+/Hhs3LjRvBpDq1at8Nlnn9W4lmmHDh2wYcMGJCQkYMaMGWjatClmz56NK1euVAizM2bMwNWrV/Hjjz/ihx9+QP/+/bFu3ToEBQVZ/Zrroi7vPVC792vatGkoKCjA8uXLsXLlSnTq1Am//fYbXn31VVu9rDr75ptvEBISgu+//x5r1qxBnz59sHLlSrRu3ZpXFiO3JYj1aRiGiMgKQ4YMwcmTJyud/0kNG9/7io4cOYKOHTti2bJlGDlypLPLIXI4zpklonqtqKjI4vaZM2ewdu1a9O7d2zkFkcPwva/o1p4AwPz58yGTySxO5CRyJ5xmQET1WosWLTB69Gi0aNECqampWLRoEVQqFV5++WVnl0Z2xve+ovfffx8HDx7EPffcA4VCgXXr1mHdunV4+umnER4e7uzyiJyC0wyIqF4bM2YMtm7divT0dKjVasTFxeGdd95Bp06dnF0a2Rnf+4qSkpIwe/ZsnDp1Cvn5+WjWrBmeeOIJvPbaa1AoOD5F7olhloiIiIhcFufMEhEREZHLYpglIiIiIpfldhNsTCYTLl++DB8fHy4wTURERFQPiaKIvLw8hIWFmS8qUxW3C7OXL1/mGZ9ERERELuDChQto2rRptfu4XZj18fEBUNYcrVZr9+czGAzYuHEj+vXrB6VSaffnc3XslzTsl/XYK2nYL2nYL+uxV9K4a790Oh3Cw8PNua06bhdmy6cWaLVah4VZjUYDrVbrVj+EtcV+ScN+WY+9kob9kob9sh57JY2798uaKaE8AYyIiIiIXBbDLBERERG5LIZZIiIiInJZDLNERERE5LIYZomIiIjIZTHMEhEREZHLYpglIiIiIpfFMEtERERELothloiIiIhcFsOsnZ1Oz8ORLAFnMvKdXQoRERFRg8Mwa2ff7k3Dkr/k2HAqw9mlEBERETU4DLN2pvUou46yrrjUyZUQERERNTwMs3bm61kWZnOLDE6uhIiIiKjhYZi1M62nAgCgY5glIiIisjmGWTvzvTHNIIdhloiIiMjmGGbtTHtjmoGuiHNmiYiIiGyNYdbO/MrnzBZzZJaIiIjI1hhm7YxzZomIiIjsh2HWzspXMygymFBSanJyNUREREQNC8OsnfmoFRAgAuDyXERERES2xjBrZzKZAA952fcMs0RERES2xTDrAJqyabMMs0REREQ2xjDrADfOAeNJYEREREQ2xjDrABpF2ZzZnKISJ1dCRERE1LAwzDqApnzObCFHZomIiIhsiWHWAf6ZM8urgBERERHZEsOsA3jyBDAiIiIiu2CYdYDyObMMs0RERES2xTDrAFyai4iIiMg+nBpm58yZgy5dusDHxwdBQUEYMmQITp8+XeNxq1atwu233w4PDw+0b98ea9eudUC1tceluYiIiIjsw6lhdvv27ZgwYQL27NmDpKQkGAwG9OvXDwUFBVUes2vXLjz66KP417/+hcOHD2PIkCEYMmQITpw44cDKpdHwCmBEREREdqFw5pOvX7/e4vbSpUsRFBSEgwcP4q677qr0mI8//hj33XcfXnrpJQDAm2++iaSkJCxcuBCLFy+2e821wTmzRERERPbh1DB7q9zcXABAQEBAlfvs3r0bCQkJFtvi4+ORmJhY6f56vR56vd58W6fTAQAMBgMMBvuHS4PBcNNqBiUOeU5XVt4f9sk67Jf12Ctp2C9p2C/rsVfSuGu/pLxeQRRF0Y61WM1kMmHw4MHIycnBzp07q9xPpVLh66+/xqOPPmre9tlnn2H27NnIyMiosP+sWbMwe/bsCtuXL18OjUZjm+JrUFgKTN1flmjndiuFgqfdEREREVWpsLAQjz32GHJzc6HVaqvdt96MzE6YMAEnTpyoNsjWxtSpUy1GcnU6HcLDw9GvX78am2MLBoMBGzYmQQAgAoi7+1409lHb/XldlcFgQFJSEvr27QulUunscuo99st67JU07Jc07Jf12Ctp3LVf5Z+kW6NehNmJEyfi119/xY4dO9C0adNq9w0JCakwApuRkYGQkJBK91er1VCrK4ZHpVLpsB8KmQBoPRXILSpFYanoVj+MteXI96chYL+sx15Jw35Jw35Zj72Sxt36JeW1OvUDb1EUMXHiRKxZswZbtmxBZGRkjcfExcVh8+bNFtuSkpIQFxdnrzJtQutR9qbwJDAiIiIi23FqmJ0wYQKWLVuG5cuXw8fHB+np6UhPT0dRUZF5nyeffBJTp0413548eTLWr1+PuXPn4s8//8SsWbNw4MABTJw40RkvwWq+ngyzRERERLbm1DC7aNEi5Obmonfv3ggNDTV/rVy50rxPWloarly5Yr7do0cPLF++HJ9//jliYmLw448/IjExEe3atXPGS7Ca9saSBgyzRERERLbj1Dmz1iyksG3btgrbhg8fjuHDh9uhIvvxLZ9mUMgwS0RERGQrXCTKQbTmaQalTq6EiIiIqOFgmHUQX04zICIiIrI5hlkHKV/NIKeoxMmVEBERETUcDLMO4qcpC7M6jswSERER2QzDrINoPTjNgIiIiMjWGGYdhOvMEhEREdkew6yDMMwSERER2R7DrIPwoglEREREtscw6yDlF00oNpigLzU6uRoiIiKihoFh1kG81QoIQtn3HJ0lIiIisg2GWQeRyQTzWrNcnouIiIjINhhmHah8rdmcQoZZIiIiIltgmHUgrmhAREREZFsMsw7EMEtERERkWwyzDqRlmCUiIiKyKYZZB+LILBEREZFtMcw6EMMsERERkW0xzDoQwywRERGRbTHMOlB5mOU6s0RERES2wTDrQByZJSIiIrIthlkH8vPkRROIiIiIbIlh1oG4NBcRERGRbTHMOhCnGRARERHZFsOsA/lqysKsvtSEYoPRydUQERERuT6GWQfyVikgE8q+54oGRERERHXHMOtAMpnAebNERERENsQw62CcN0tERERkOwyzDqb1uHHhhGKGWSIiIqK6Yph1MC+1HACQr+cJYERERER1xTDrYN5qBQCgUF/q5EqIiIiIXB/DrIN53Qiz+QyzRERERHXGMOtgGlVZmC3gNAMiIiKiOmOYdTDvG3NmC0s4MktERERUVwyzDsZpBkRERES2wzDrYF7maQYMs0RERER1xTDrYOUjswUlnDNLREREVFcMsw5Wvs4sR2aJiIiI6o5h1sE4zYCIiIjIdhhmHYzTDIiIiIhsh2HWwTjNgIiIiMh2GGYdzDwyyzBLREREVGcMsw7mfdM0A1EUnVwNERERkWtjmHUwjapsmoHRJEJfanJyNURERESujWHWwcpXMwA41YCIiIiorhhmHUwmE8yjswV6rmhAREREVBcMs06guTE6m8+RWSIiIqI6YZh1Au8by3MVljDMEhEREdWF5DBbVFSEwsJC8+3U1FTMnz8fGzdutGlhDVn58lwcmSUiIiKqG8lh9oEHHsA333wDAMjJyUG3bt0wd+5cPPDAA1i0aJHNC2yI/rmkLefMEhEREdWF5DB76NAh9OrVCwDw448/Ijg4GKmpqfjmm2/wySef2LzAhsh8FTBOMyAiIiKqE8lhtrCwED4+PgCAjRs34sEHH4RMJkP37t2Rmppq8wIbIg2vAkZERERkE5LDbKtWrZCYmIgLFy5gw4YN6NevHwAgMzMTWq3W5gU2RN4qhlkiIiIiW5AcZmfMmIEXX3wRzZs3R7du3RAXFwegbJS2Y8eONi+wIfK66ZK2RERERFR7ipp3sfTQQw/hzjvvxJUrVxATE2Pefu+992Lo0KE2La6hMs+Z5cgsERERUZ3Uap3ZkJAQdOzYETKZDDqdDomJifDx8cHtt98u6XF27NiBQYMGISwsDIIgIDExsdr9t23bBkEQKnylp6fX5mU4DZfmIiIiIrINyWF2xIgRWLhwIYCyNWdjY2MxYsQIdOjQAT/99JOkxyooKEBMTAw+/fRTScedPn0aV65cMX8FBQVJOt7ZysNsIZfmIiIiIqoTydMMduzYgddeew0AsGbNGoiiiJycHHz99dd46623MGzYMKsfq3///ujfv7/UEhAUFAQ/Pz+r9tXr9dDr9ebbOp0OAGAwGGAwGCQ/t1Tlz3Hzc3mUzTJAXrFjanAllfWLqsZ+WY+9kob9kob9sh57JY279kvK6xVEURSlPLinpyf++usvhIeH48knn0RYWBjeffddpKWloW3btsjPz5dcMAAIgoA1a9ZgyJAhVe6zbds23HPPPYiIiIBer0e7du0wa9Ys9OzZs8pjZs2ahdmzZ1fYvnz5cmg0mlrVWlfHsgV8dVqO5t4inm/P0VkiIiKimxUWFuKxxx5Dbm5ujatlSR6ZDQ8Px+7duxEQEID169djxYoVAIDr16/Dw8OjdhVbKTQ0FIsXL0ZsbCz0ej2+/PJL9O7dG3v37kWnTp0qPWbq1KlISEgw39bpdAgPD0e/fv0cspSYwWBAUlIS+vbtC6VSCQDw+zsLX50+CJXGBwMG9LB7Da6ksn5R1dgv67FX0rBf0rBf1mOvpHHXfpV/km4NyWF2ypQpGDlyJLy9vREREYHevXsDKJt+0L59e6kPJ0nr1q3RunVr8+0ePXrg77//xrx58/Dtt99WeoxarYZara6wXalUOvSH4ubn02rK6ikoMbrVD6YUjn5/XB37ZT32Shr2Sxr2y3rslTTu1i8pr1VymB0/fjy6du2KCxcuoG/fvpDJys4ha9GiBd566y2pD1dnXbt2xc6dOx3+vHXhXX4CGC9nS0RERFQnksMsAMTGxiI2NhaiKEIURQiCgIEDB9q6NqscOXIEoaGhTnnu2vrncracL0tERERUF7VaZ/abb75B+/bt4enpCU9PT3To0KHKj/mrk5+fjyNHjuDIkSMAgJSUFBw5cgRpaWkAyua7Pvnkk+b958+fj59//hlnz57FiRMnMGXKFGzZsgUTJkyozctwmvLL2ZYYTSgpNTm5GiIiIiLXJXlk9qOPPsL06dMxceJE8yoCO3fuxLhx43Dt2jU8//zzVj/WgQMHcM8995hvl5+oNWrUKCxduhRXrlwxB1sAKCkpwQsvvIBLly5Bo9GgQ4cO2LRpk8VjuALNjSuAAWVTDVQKlROrISIiInJdksPsggULsGjRIosR08GDByM6OhqzZs2SFGZ79+6N6lYGW7p0qcXtl19+GS+//LLUkusdpVwGlUKGklIT8vWl8NMwzBIRERHVhuRpBleuXEGPHhWXk+rRoweuXLlik6LcgTfnzRIRERHVmeQw26pVK/zwww8Vtq9cuRJRUVE2KcodeN2YalDAFQ2IiIiIak3yNIPZs2fj4Ycfxo4dO8xzZv/44w9s3ry50pBLlfNSlY/MMswSERER1Zbkkdlhw4Zh7969CAwMRGJiIhITExEYGIh9+/Zh6NCh9qixQfJSM8wSERER1VWt1pnt3Lkzli1bZrEtMzMT77zzDqZNm2aTwho6L86ZJSIiIqqzWq0zW5krV65g+vTptnq4Bs9LxTmzRERERHVlszBL0pSPzOZzmgERERFRrTHMOkn5yGwhpxkQERER1RrDrJNwZJaIiIio7qw+Aaz8UrNVuXr1ap2LcSdczYCIiIio7qwOs4cPH65xn7vuuqtOxbgT8zSDEk4zICIiIqotq8Ps1q1b7VmH2+E0AyIiIqK645xZJ/G+EWYLuTQXERERUa0xzDqJxjwyy2kGRERERLXFMOsk3uobF03gNAMiIiKiWmOYdRIvTjMgIiIiqjOGWSfxUvEEMCIiIqK6khxmmzdvjjfeeANpaWn2qMdtlI/MFhtMKDWanFwNERERkWuSHGanTJmC1atXo0WLFujbty9WrFgBvV5vj9oaNM2NdWYBoNDAk8CIiIiIaqNWYfbIkSPYt28f2rRpg+eeew6hoaGYOHEiDh06ZI8aGyS1QgaFTADAk8CIiIiIaqvWc2Y7deqETz75BJcvX8bMmTPx5ZdfokuXLrjjjjvw3//+F6Io2rLOBkcQBF7SloiIiKiOrL4C2K0MBgPWrFmDJUuWICkpCd27d8e//vUvXLx4EdOmTcOmTZuwfPlyW9ba4Hip5MgtMqCAa80SERER1YrkMHvo0CEsWbIE33//PWQyGZ588knMmzcPt99+u3mfoUOHokuXLjYttCHiyCwRERFR3UgOs126dEHfvn2xaNEiDBkyBEqlssI+kZGReOSRR2xSYEPmpebyXERERER1ITnMnjt3DhEREdXu4+XlhSVLltS6KHfhdeMqYIUlnGZAREREVBuSw2x5kD1w4ACSk5MBAG3atEFsbKxtK3MDvHACERERUd1IDrMXL17Eo48+ij/++AN+fn4AgJycHPTo0QMrVqxA06ZNbV1jg+XNObNEREREdSJ5aa6nnnoKBoMBycnJyM7ORnZ2NpKTk2EymfDUU0/Zo8YGS3NjmkEBpxkQERER1Yrkkdnt27dj165daN26tXlb69atsWDBAvTq1cumxTV0XM2AiIiIqG4kj8yGh4fDYDBU2G40GhEWFmaTotxF+ZzZwhKGWSIiIqLakBxmP/jgAzz33HM4cOCAeduBAwcwefJkfPjhhzYtrqH7Z2kuTjMgIiIiqg3J0wxGjx6NwsJCdOvWDQpF2eGlpaVQKBQYO3Ysxo4da943OzvbdpU2QN7lc2Y5zYCIiIioViSH2fnz59uhDPekUXHOLBEREVFdSA6zo0aNskcdbsm8NBfnzBIRERHViuQwC5Sd7JWYmGi+aEJ0dDQGDx4MuVxu0+Iaun9WM+CcWSIiIqLakBxmz549iwEDBuDSpUvm5bnmzJmD8PBw/Pbbb2jZsqXNi2yoNCrOmSUiIiKqC8mrGUyaNAktW7bEhQsXcOjQIRw6dAhpaWmIjIzEpEmT7FFjg8UrgBERERHVTa0umrBnzx4EBASYtzVq1AjvvvsuevbsadPiGjrzNIMSI0wmETKZ4OSKiIiIiFyL5JFZtVqNvLy8Ctvz8/OhUqlsUpS78FL/M8e40MB5s0RERERSSQ6z999/P55++mns3bsXoihCFEXs2bMH48aNw+DBg+1RY4PlqZRDfmM0Nq+44lXViIiIiKh6ksPsJ598gpYtWyIuLg4eHh7w8PBAz5490apVK3z88cf2qLHBEgQBvp5KAEBuEcMsERERkVSS5syKogidTocVK1bg0qVL5qW52rRpg1atWtmlwIbO11OJ7IIS5BYyzBIRERFJJTnMtmrVCidPnkRUVBQDrA1ob4zM6oq5ogERERGRVJKmGchkMkRFRSErK8te9bgdrUfZ3xOcZkBEREQkneQ5s++++y5eeuklnDhxwh71uB3OmSUiIiKqPcnrzD755JMoLCxETEwMVCoVPD09Le7Pzs62WXHugGGWiIiIqPYkh9l58+ZBELi4v62Uh1kdwywRERGRZJLD7OjRo+1QhvviyCwRERFR7UmeMyuXy5GZmVlhe1ZWFuRyeSVHUHUYZomIiIhqT3KYFUWx0u16vZ6Xs60FTjMgIiIiqj2rpxl88sknAMquWvXll1/C29vbfJ/RaMSOHTtw++23277CBo4js0RERES1Z3WYnTdvHoCykdnFixdbTClQqVRo3rw5Fi9ebPsKGzgtwywRERFRrVkdZlNSUgAA99xzD1avXg1/f3+7FeVOODJLREREVHuS58xu3brVZkF2x44dGDRoEMLCwiAIAhITE2s8Ztu2bejUqRPUajVatWqFpUuX2qQWZykfmdWXmlBsMDq5GiIiIiLXInlpLqPRiKVLl2Lz5s3IzMyEyWSyuH/Lli1WP1ZBQQFiYmIwduxYPPjggzXun5KSgoEDB2LcuHH47rvvsHnzZjz11FMIDQ1FfHy81JdSL/ioFRAEQBTLTgLzUHJFCCIiIiJrSQ6zkydPxtKlSzFw4EC0a9euThdQ6N+/P/r372/1/osXL0ZkZCTmzp0LAGjTpg127tyJefPmuWyYlckEaD2UyC0yILfIgCCth7NLIiIiInIZksPsihUr8MMPP2DAgAH2qKdau3fvRp8+fSy2xcfHY8qUKVUeo9frodfrzbd1Oh0AwGAwwGCw/zzV8ueo7rm0HgrkFhmQlVeE5gHuHWat6Rf9g/2yHnslDfslDftlPfZKGnftl5TXKznMqlQqtGrVSuphNpGeno7g4GCLbcHBwdDpdCgqKoKnp2eFY+bMmYPZs2dX2L5x40ZoNBq71XqrpKSkqu8skQMQsGXnHmScrHwdX3dTbb+oAvbLeuyVNOyXNOyX9dgradytX4WFhVbvKznMvvDCC/j444+xcOHCOk0xcJSpU6ciISHBfFun0yE8PBz9+vWDVqu1+/MbDAYkJSWhb9++UCqVle6zMvMALvydjduiYzDgjjC711SfWdMv+gf7ZT32Shr2Sxr2y3rslTTu2q/yT9KtITnM7ty5E1u3bsW6desQHR1dobGrV6+W+pBWCwkJQUZGhsW2jIwMaLXaSkdlAUCtVkOtVlfYrlQqHfpDUd3z+WnKrpyWX2Jyqx/U6jj6/XF17Jf12Ctp2C9p2C/rsVfSuFu/pLxWyWHWz88PQ4cOlXqYTcTFxWHt2rUW25KSkhAXF+eUemzln7VmS51cCREREZFrkRxmlyxZYrMnz8/Px9mzZ823U1JScOTIEQQEBKBZs2aYOnUqLl26hG+++QYAMG7cOCxcuBAvv/wyxo4diy1btuCHH37Ab7/9ZrOanIFXASMiIiKqHasvmpCZmVnt/aWlpdi3b5+kJz9w4AA6duyIjh07AgASEhLQsWNHzJgxAwBw5coVpKWlmfePjIzEb7/9hqSkJMTExGDu3Ln48ssvXXZZrnK8ChgRERFR7Vg9MhsaGoorV64gKCgIANC+fXusXbsW4eHhAICsrCzExcXBaLT+Kla9e/eGKFZ99n5lV/fq3bs3Dh8+bPVzuAKGWSIiIqLasXpk9tbQef78+QprgFUXTKlq5WFWV8wwS0RERCSF1WHWGq6wVFd9ZA6zHJklIiIiksSmYZZqh9MMiIiIiGrH6jmzgiAgLy8PHh4eEEURgiAgPz/fvKitlMVtyZLWg2GWiIiIqDasDrOiKOK2226zuF2+CkH5bU4zqJ3ykdnCEiMMRhOUcg6YExEREVnD6jC7detWe9bh1srXmQXKRmcDvStesYyIiIiIKrI6zN599932rMOtyWUCfNQK5OlLGWaJiIiIJODn2fUErwJGREREJB3DbD3B5bmIiIiIpGOYrSe4PBcRERGRdAyz9QRHZomIiIikq3OY1el0SExMRHJysi3qcVtaz7Jz8TgyS0RERGQ9yWF2xIgRWLhwIQCgqKgIsbGxGDFiBDp06ICffvrJ5gW6C04zICIiIpJOcpjdsWMHevXqBQBYs2YNRFFETk4OPvnkE7z11ls2L9BdMMwSERERSSc5zObm5iIgIAAAsH79egwbNgwajQYDBw7EmTNnbF6gu2CYJSIiIpJOcpgNDw/H7t27UVBQgPXr16Nfv34AgOvXr8PDw8PmBboLrfkEsFInV0JERETkOqy+Ali5KVOmYOTIkfD29kZERAR69+4NoGz6Qfv27W1dn9vgyCwRERGRdJLD7Pjx49G1a1dcuHABffv2hUxWNrjbokULzpmtA4ZZIiIiIukkh1kAiI2NRWxsLADAaDTi+PHj6NGjB/z9/W1anDvRcp1ZIiIiIskkz5mdMmUKvvrqKwBlQfbuu+9Gp06dEB4ejm3bttm6PrdRPjKbpy+F0SQ6uRoiIiIi1yA5zP7444+IiYkBAPzyyy9ISUnBn3/+ieeffx6vvfaazQt0F+VhFuDoLBEREZG1JIfZa9euISQkBACwdu1aDB8+HLfddhvGjh2L48eP27xAd6GUy6BRyQFw3iwRERGRtSSH2eDgYJw6dQpGoxHr169H3759AQCFhYWQy+U2L9Cd8CQwIiIiImkknwA2ZswYjBgxAqGhoRAEAX369AEA7N27F7fffrvNC3Qnvp5KXMkthq6YYZaIiIjIGpLD7KxZs9CuXTtcuHABw4cPh1qtBgDI5XK8+uqrNi/QnWg5MktEREQkSa2W5nrooYcqbBs1alSdi3F3nGZAREREJI3kObMAsH37dgwaNAitWrVCq1atMHjwYPz++++2rs3taD0YZomIiIikkBxmly1bhj59+kCj0WDSpEmYNGkSPD09ce+992L58uX2qNFtcGSWiIiISBrJ0wzefvttvP/++3j++efN2yZNmoSPPvoIb775Jh577DGbFuhOfHkVMCIiIiJJJI/Mnjt3DoMGDaqwffDgwUhJSbFJUe7K17PsbwuOzBIRERFZR3KYDQ8Px+bNmyts37RpE8LDw21SlLvy1ZSPzJY6uRIiIiIi1yB5msELL7yASZMm4ciRI+jRowcA4I8//sDSpUvx8ccf27xAd8I5s0RERETSSA6zzz77LEJCQjB37lz88MMPAIA2bdpg5cqVeOCBB2xeoDsJ8Cpbs/dqnt7JlRARERG5BklhtrS0FO+88w7Gjh2LnTt32qsmtxURoAEApOuKUVRihKeKlwcmIiIiqo6kObMKhQLvv/8+Sks5p9Me/L1U5qkG57MKnFwNERERUf0n+QSwe++9F9u3b7dHLQSgeaAXAOD8NYZZIiIioppInjPbv39/vPrqqzh+/Dg6d+4MLy8vi/sHDx5ss+LcUWQjDY5eyEEKR2aJiIiIaiQ5zI4fPx4A8NFHH1W4TxAEGI3GulflxjgyS0RERGQ9yWHWZDLZow66IdIcZgudXAkRERFR/Sd5zizZV/NGZWGW0wyIiIiIamZ1mN2yZQvatm0LnU5X4b7c3FxER0djx44dNi3OHZVPM7iap0e+nqtGEBEREVXH6jA7f/58/Pvf/4ZWq61wn6+vL5555hnMmzfPpsW5I19PJQK8VAA4b5aIiIioJlaH2aNHj+K+++6r8v5+/frh4MGDNinK3TVvVHbxBK41S0RERFQ9q8NsRkYGlEpllfcrFApcvXrVJkW5u8hAbwAcmSUiIiKqidVhtkmTJjhx4kSV9x87dgyhoaE2KcrdRQaWjcymcEUDIiIiompZHWYHDBiA6dOno7i4uMJ9RUVFmDlzJu6//36bFueuyk8CS7mW7+RKiIiIiOo3q9eZff3117F69WrcdtttmDhxIlq3bg0A+PPPP/Hpp5/CaDTitddes1uh7qR8ea7zWRyZJSIiIqqO1WE2ODgYu3btwrPPPoupU6dCFEUAZVf9io+Px6efforg4GC7FepOykdmswtKkFtkgK9n1XOViYiIiNyZpCuARUREYO3atbh+/TrOnj0LURQRFRUFf39/e9XnlrzVCjT2UeNqnh7nrxUgJtzP2SURERER1UuSL2cLAP7+/ujSpYuta6GbRDbyKguzWQyzRERERFXh5WzrqebmFQ24PBcRERFRVRhm66nyebNca5aIiIioagyz9VTkjRUNUriiAREREVGV6kWY/fTTT9G8eXN4eHigW7du2LdvX5X7Ll26FIIgWHx5eHg4sFrH4MgsERERUc2cHmZXrlyJhIQEzJw5E4cOHUJMTAzi4+ORmZlZ5TFarRZXrlwxf6WmpjqwYscoX2s2t8iA6wUlTq6GiIiIqH5yepj96KOP8O9//xtjxoxB27ZtsXjxYmg0Gvz3v/+t8hhBEBASEmL+aojr23qq5AjRlo04p2RxdJaIiIioMrVamstWSkpKcPDgQUydOtW8TSaToU+fPti9e3eVx+Xn5yMiIgImkwmdOnXCO++8g+jo6Er31ev10Ov15ts6nQ4AYDAYYDAYbPRKqlb+HLV5ruaNPJGuK8bfGTq0D/W2dWn1Ul365Y7YL+uxV9KwX9KwX9Zjr6Rx135Jeb2CWH4pLye4fPkymjRpgl27diEuLs68/eWXX8b27duxd+/eCsfs3r0bZ86cQYcOHZCbm4sPP/wQO3bswMmTJ9G0adMK+8+aNQuzZ8+usH358uXQaDS2fUE2tvJvGXZlyhDfxIQBzUzOLoeIiIjIIQoLC/HYY48hNzcXWq222n2dOjJbG3FxcRbBt0ePHmjTpg3+85//4M0336yw/9SpU5GQkGC+rdPpEB4ejn79+tXYHFswGAxISkpC3759oVRKuyxt+h/nsWv9XzD6BGPAgI52qrB+qUu/3BH7ZT32Shr2Sxr2y3rslTTu2q/yT9Kt4dQwGxgYCLlcjoyMDIvtGRkZCAkJseoxlEolOnbsiLNnz1Z6v1qthlqtrvQ4R/5Q1Ob5urQIBPAXDqTlQC5XQCYT7FNcPeTo98fVsV/WY6+kYb+kYb+sx15J4279kvJanXoCmEqlQufOnbF582bzNpPJhM2bN1uMvlbHaDTi+PHjCA0NtVeZTtO+iS80KjlyCg04nZHn7HKIiIiI6h2nr2aQkJCAL774Al9//TWSk5Px7LPPoqCgAGPGjAEAPPnkkxYniL3xxhvYuHEjzp07h0OHDuHxxx9HamoqnnrqKWe9BLtRymXoHOEPANh7LsvJ1RARERHVP06fM/vwww/j6tWrmDFjBtLT03HHHXdg/fr15uW20tLSIJP9k7mvX7+Of//730hPT4e/vz86d+6MXbt2oW3bts56CXbVvUUj/H7mGvacy8bonpHOLoeIiIioXnF6mAWAiRMnYuLEiZXet23bNovb8+bNw7x58xxQVf3QvUUAAGDf+WyIoghBcJ95s0REREQ1cfo0A6pe+yZ+8FDKkF1QgjOZ+c4uh4iIiKheYZit51QKGWIjykZn93DeLBEREZEFhlkX0C2yLMzuPZft5EqIiIiI6heGWRfQvWUjAMDelCw48YJtRERERPUOw6wL6NDUF2qFDNfyS/D3Vc6bJSIiIirHMOsC1Aq5eb3Z3ZxqQERERGTGMOsiukXemGrAk8CIiIiIzBhmXUT5erN7U7I5b5aIiIjoBoZZFxET7geVQoareXqcu1bg7HKIiIiI6gWGWRfhoZSjUzM/AMD201edWwwRERFRPcEw60L6twsFAKzcf4FTDYiIiIjAMOtShnZqAk+lHKcz8nAw9bqzyyEiIiJyOoZZF6L1UGJwTBgA4Lu9aU6uhoiIiMj5GGZdzGPdmgEAfjt+BdkFJU6uhoiIiMi5GGZdTIemvmjXRIuSUhN+OnjR2eUQERERORXDrIsRBAEju0UAAJbvS4PJxBPBiIiIyH0xzLqgwTFh8FYrkHKtALt5RTAiIiJyYwyzLshLrcDQjk0AAN/tTXVyNURERETOwzDrospPBNt4MgOXcoqcXA0RERGRczDMuqg2oVrEtWiEUpOIWf876exyiIiIiJyCYdaFzX4gGgqZgKRTGdh4Mt3Z5RARERE5HMOsC7st2Af/vqsFAGDW/06iQF/q5IqIiIiIHIth1sVN+r8ohAd44nJuMeZv+svZ5RARERE5FMOsi/NUyfHGA+0AAP/94zxOXdY5uSIiIiIix2GYbQDuaR2Ege1DYTSJeOWnYyg2GJ1dEhEREZFDMMw2EDMGtYWvpxLHL+Vi4vJDKDWanF0SERERkd0xzDYQwVoPfPFkLNQKGTYlZ2LamuMQRV7qloiIiBo2htkGpGtkABY82hEyAfjhwEV8sOG0s0siIiIisiuG2QamX3QI3hnaHgDw2ba/sWjb3xyhJSIiogaLYbYBeqRrM7wU3xoA8N76PzHj55OcQ0tEREQNEsNsAzW+d0u8NqANBAH4dk8q/vX1AeQVG5xdFhEREZFNMcw2UIIg4N93tcDixzvDQynD9r+uYvji3UjLKnR2aUREREQ2wzDbwMVHh+CHZ+LQ2EeNP9PzcN/HO/DtnlTOoyUiIqIGgWHWDXRo6ofECT3RNTIAhSVGTE88gSe+2odLOUXOLo2IiIioThhm3UQTP0+s+Hd3zLi/LTyUMuw8ew3x83bgix3nUFLKk8OIiIjINTHMuhGZTMDYOyOxdlIvdGrmh3x9Kd5em4z75u/A1j8znV0eERERkWQMs26oRWNvrBrXA+8P64BAbxXOXSvAmKX78cRXe7HzzDXOpyUiIiKXoXB2AeQccpmAEV3C0b99CBZsOYslf6Tg9zPX8PuZa2gd7IOxdzbHA3c0gYdS7uxSiYiIiKrEkVk35+OhxLQBbbAp4W6MiouARiXH6Yw8vPLTcXR9exNeTzyOIxdyOFpLRERE9RJHZgkAENHIC7MfaIeEfq2xcn8avt6Viks5RVi2Jw3L9qShVZA3BseEYUD7ELQK8nF2uUREREQAGGbpFr6eSjx9V0s8dWcL7D6XhR8PXsS6E1dwNjMfHyX9hY+S/kJUkDf6twtB79uD0KGJLxRyDvATERGRczDMUqVkMgE9WwWiZ6tAvPFANNafSMe6E+n4/cxVnMnMx5ktZ/HJlrPw8VCgZ8tA9LotEL1aNUazRhpnl05ERERuhGGWauTjocTw2HAMjw1HbpEBW/7MwMaTGfjj7DXoikux/mQ61p9MBwBENNLgzlaBuKd1EHrdFgi1gieQERERkf0wzJIkvp5KDO3YFEM7NoXRJOLYxRzsvLEKwqG060jNKkRqVhq+25sGH7UCfaODMahDGLq3aARPFYMtERER2RbDLNWaXCagYzN/dGzmj+fujUK+vhR7/s7C9r+uYuOpdGTo9Fh96BJWH7oEmQBEBnqhTagWbcO06BjujzvC/RhwiYiIqE4YZslmvNUK9GkbjD5tgzF7cDQOpF7Hb8cuY/3JsmD799UC/H21AL8euwIAUMgERIdp0TkiALHN/REb4Q9/T4ZbIiIish7DLNmFTCaga2QAukYGYNbgaFzN0+PUFR1OXdHhxKVcHEy9jgydHkcv5uLoxVz8948UAEC4vydC5DLk7r+Abi0aIyrIGzKZ4ORXQ0RERPUVwyzZnSAICNJ6IEjrgd6tgwAAoiji4vUiHEy9jgOp2Thw/jpOZ+ThwvUiXIAM+/+XDCAZWg8FOkWUjdp2jgjg1AQiIiKywDBLTiEIAsIDNAgP0GBIxyYAAF2xAQfOXcPKzQegUwfi6MVc6IpLse30VWw7fRVA2dSEFo29EBXsg6ggb0QF+eC2YG9ENPKCSsH1bomIiNwNwyzVG1oPJXpFBSLvjAkDBsRCkMmRfCWvbOQ29ToOnr+OdF0x/srIx18Z+RbHKmQCmgd6oVVjb0Q00piDcrMADZr4eTLoEhERNVAMs1RvKeQytG/qi/ZNfTGmZyREUcSV3GKczsjD2Yx8nMnMw18Z+TibmY98fSnOZpZ9fyuZAIT6eiI8wBNhfp4I9fVAiK8nQrUeCPH1QKivBwK8VBAEzs0lIiJyNQyz5DIEQUCYX1kgvefG3FugbP5t+Yjtuav5uJBdhLTsQlzILkRadiGKDEZcyinCpZyiKh9bpZAh5KZwG+LrgWAfDzTyVsFfo0KA1z9fHkrO2SUiIqovGGbJ5QmCgFBfT4T6euLu2xpb3CeKIq7ll+DC9bJwezmnGOm5RbiSW4x0XTEu5xTjWr4eJaUmpN0IvzXRqOTw16gqDboBXhW3+XkquSIDERGRnTDMUoMmCAIa+6jR2EeNTs38K92npNSEDF1ZuL2S+0/YzczT43pBCbJvfF0vLIHBKKKwxIjCkupHem8mEwC/8oCrUcHfS4kALzUCbvpX66GEt1oBbw+FxfdKOef6EhERVadehNlPP/0UH3zwAdLT0xETE4MFCxaga9euVe6/atUqTJ8+HefPn0dUVBTee+89DBgwwIEVU0OiUsjMJ4xVRxRF5OlLcb2gBFkFJRZBN7uwBNn5ZYE3+6btuuJSmESYb0vloZTBx0MJnxvh1sdDAQ+FHAq5AIVMBpkgIitdhuMb/oKfRgUfDyU8lDKoFDKoFXKo5DKolTKo5GXbyrbLoJLLoVLIoJQL5u0quYzzhomIyOU4PcyuXLkSCQkJWLx4Mbp164b58+cjPj4ep0+fRlBQUIX9d+3ahUcffRRz5szB/fffj+XLl2PIkCE4dOgQ2rVr54RXQO5CEARoPcpGUSMaeVl1jMFowvXCElwvMCCrQI/rBQZkF+iRXWDA9cJ/QnFesQF5xaXI05civ7gURQYjAKDYYEKxQY+refpqnkWGXZnn6/4CgbJwK5dBqbgpAN/4VymXQS4ToJAJZf/KBchlsn9u3/hXbnH7lvvlN/4VBAhC2XaZUNZbmVD2vUwQIJPd9P2N+yvbVy4TbtyG5fGCAJnsn33lggCTqRRncgXsO58NlVJZYX/hxvcAIAg3viDc+Lfsdpmbtwnm+8q++2c/QbC8/8aR5mNh7d8NIiACMIkiRLHyf01i2R9bIgCVXIYQXw+Lud2lRhMu5ZTNJff1VKJFY294qy3/8280ibiQXYjDF67jSFoODqVdx7XrcuzQn0CnG2s8twryhlrBOeNEVL8IoiiKziygW7du6NKlCxYuXAgAMJlMCA8Px3PPPYdXX321wv4PP/wwCgoK8Ouvv5q3de/eHXfccQcWL15c4/PpdDr4+voiNzcXWq3Wdi+kCgaDAWvXrsWAAQOgVCrt/nyujv0qYzCaUKAvLQu4xaXI15eaA6++1AiDUYTRJKK4xIAjJ5IR2qwFCkpMyNMboDeYoC81oaTUBL3RBL3BiBLjjdulJhhufF9SakKpyam//mRH/holQnw9oTcYkZZdWOG9DtF6IDzAE3nFpbiWr0dWQQms+b9BgJcKwVoPBGvV8FIpoJQLUN70B0/5iL9cJjPn9ZvD/M23y7bdss8tnw5Y7nvLHwxVPP6tx938uLceY7mt8se/eeeq/gYxGo04ceIE2rdvB7m8+sAvWPGXjDUfkljz95B1j1PDTjarpWwvo7EUx44eQ4eYDpDLLf+ossVrsslrtvJxrGHNJ17V7WE0GnHk8GF07NgR8mr+mHTUz9VdtzWGl9r+Y6FS8ppTR2ZLSkpw8OBBTJ061bxNJpOhT58+2L17d6XH7N69GwkJCRbb4uPjkZiYWOn+er0eev0/o1o6nQ5AWWgyGAx1fAU1K38ORzxXQ8B+/cNLKcBLqUSIT9Wh3mAwICT3FPr2aVGr8G8yiWXh1mhCiVEsC7lGEww3/i0pNcFgFMu2GU0wGkUYxbIgXWqq/N+y78v3BYwm0y33lf1784iiqXyE0XTT9zfdZzSJFvuJImAUxQqjk6ZKHrf8WKMoIi8vHxovr0qfw2QqG9ksH+EURUCEaBHwbt528z64sU/F48tuWx5vuc0a5aPP5n+BG6PJ5SPK/4wqFxuMKDKYcL3QgOuF//weqRUyNPX3RG6RAdfyS5B+Y574zZRyAdFhWtzR1Bftw3yQfOIY5I1b4PiVPBy/pENecal5ykzyFUkvwU3IsSol2dlFuAg5vvv7pLOLcCFyLD1zzNlFAAA2PX8nImqYlmcLUnKAU8PstWvXYDQaERwcbLE9ODgYf/75Z6XHpKenV7p/enp6pfvPmTMHs2fPrrB948aN0Gjs/2aUS0pKcthzNQTslzTO6Jf8xpeqNgc79bw2nTOf3O5EESgyAjl6IKdEgEIGNPYQ4asCZELZvO3CUiCzCMjWC/BUAFqlCB8l4KUE5EIWgCzgMtA+AIDxb7QNAkY0LjsupwTILRGQWwIYTIBRBEpFoNQEGEUBRlPZbeMteV2s5EZlkb582615v7p9b75TrLjJojdSH1Os7L46qOtnoTapwZp9bPBEdX0IR9XgOu9r9WOmjurXH9u34aS67s9Vk8LCmlcXKuf0ObP2NnXqVIuRXJ1Oh/DwcPTr189h0wySkpLQt29ft/7Y3FrslzTsl/XYK2nYL2nYL+uxV9K4a7/KP0m3hlPDbGBgIORyOTIyMiy2Z2RkICQkpNJjQkJCJO2vVquhVlf8E0KpVDr0h8LRz+fq2C9p2C/rsVfSsF/SsF/WY6+kcbd+SXmtTv2wT6VSoXPnzti8ebN5m8lkwubNmxEXF1fpMXFxcRb7A2UfsVa1PxERERE1XE6fZpCQkIBRo0YhNjYWXbt2xfz581FQUIAxY8YAAJ588kk0adIEc+bMAQBMnjwZd999N+bOnYuBAwdixYoVOHDgAD7//HNnvgwiIiIicgKnh9mHH34YV69exYwZM5Ceno477rgD69evN5/klZaWBpnsnwHkHj16YPny5Xj99dcxbdo0REVFITExkWvMEhEREbkhp4dZAJg4cSImTpxY6X3btm2rsG348OEYPny4nasiIiIiovqOF34nIiIiIpfFMEtERERELothloiIiIhcFsMsEREREbkshlkiIiIiclkMs0RERETksurF0lyOJIoiAGnX/K0Lg8GAwsJC6HQ6t7oMXW2xX9KwX9Zjr6Rhv6Rhv6zHXknjrv0qz2nlua06bhdm8/LyAADh4eFOroSIiIiIqpOXlwdfX99q9xFEayJvA2IymXD58mX4+PhAEAS7P59Op0N4eDguXLgArVZr9+dzdeyXNOyX9dgradgvadgv67FX0rhrv0RRRF5eHsLCwiyuBFsZtxuZlclkaNq0qcOfV6vVutUPYV2xX9KwX9Zjr6Rhv6Rhv6zHXknjjv2qaUS2HE8AIyIiIiKXxTBLRERERC6LYdbO1Go1Zs6cCbVa7exSXAL7JQ37ZT32Shr2Sxr2y3rslTTsV83c7gQwIiIiImo4ODJLRERERC6LYZaIiIiIXBbDLBERERG5LIZZIiIiInJZDLN29umnn6J58+bw8PBAt27dsG/fPmeX5HRz5sxBly5d4OPjg6CgIAwZMgSnT5+22Ke4uBgTJkxAo0aN4O3tjWHDhiEjI8NJFdcv7777LgRBwJQpU8zb2C9Lly5dwuOPP45GjRrB09MT7du3x4EDB8z3i6KIGTNmIDQ0FJ6enujTpw/OnDnjxIqdw2g0Yvr06YiMjISnpydatmyJN9980+Ja6O7cqx07dmDQoEEICwuDIAhITEy0uN+a3mRnZ2PkyJHQarXw8/PDv/71L+Tn5zvwVThOdf0yGAx45ZVX0L59e3h5eSEsLAxPPvkkLl++bPEY7tKvmn62bjZu3DgIgoD58+dbbHeXXlmDYdaOVq5ciYSEBMycOROHDh1CTEwM4uPjkZmZ6ezSnGr79u2YMGEC9uzZg6SkJBgMBvTr1w8FBQXmfZ5//nn88ssvWLVqFbZv347Lly/jwQcfdGLV9cP+/fvxn//8Bx06dLDYzn794/r16+jZsyeUSiXWrVuHU6dOYe7cufD39zfv8/777+OTTz7B4sWLsXfvXnh5eSE+Ph7FxcVOrNzx3nvvPSxatAgLFy5EcnIy3nvvPbz//vtYsGCBeR937lVBQQFiYmLw6aefVnq/Nb0ZOXIkTp48iaSkJPz666/YsWMHnn76aUe9BIeqrl+FhYU4dOgQpk+fjkOHDmH16tU4ffo0Bg8ebLGfu/Srpp+tcmvWrMGePXsQFhZW4T536ZVVRLKbrl27ihMmTDDfNhqNYlhYmDhnzhwnVlX/ZGZmigDE7du3i6Ioijk5OaJSqRRXrVpl3ic5OVkEIO7evdtZZTpdXl6eGBUVJSYlJYl33323OHnyZFEU2a9bvfLKK+Kdd95Z5f0mk0kMCQkRP/jgA/O2nJwcUa1Wi99//70jSqw3Bg4cKI4dO9Zi24MPPiiOHDlSFEX26mYAxDVr1phvW9ObU6dOiQDE/fv3m/dZt26dKAiCeOnSJYfV7gy39qsy+/btEwGIqampoii6b7+q6tXFixfFJk2aiCdOnBAjIiLEefPmme9z115VhSOzdlJSUoKDBw+iT58+5m0ymQx9+vTB7t27nVhZ/ZObmwsACAgIAAAcPHgQBoPBone33347mjVr5ta9mzBhAgYOHGjRF4D9utX//vc/xMbGYvjw4QgKCkLHjh3xxRdfmO9PSUlBenq6Rb98fX3RrVs3t+tXjx49sHnzZvz1118AgKNHj2Lnzp3o378/APaqOtb0Zvfu3fDz80NsbKx5nz59+kAmk2Hv3r0Or7m+yc3NhSAI8PPzA8B+3cxkMuGJJ57ASy+9hOjo6Ar3s1eWFM4uoKG6du0ajEYjgoODLbYHBwfjzz//dFJV9Y/JZMKUKVPQs2dPtGvXDgCQnp4OlUpl/g9cueDgYKSnpzuhSudbsWIFDh06hP3791e4j/2ydO7cOSxatAgJCQmYNm0a9u/fj0mTJkGlUmHUqFHmnlT2u+lu/Xr11Veh0+lw++23Qy6Xw2g04u2338bIkSMBgL2qhjW9SU9PR1BQkMX9CoUCAQEBbt+/4uJivPLKK3j00Ueh1WoBsF83e++996BQKDBp0qRK72evLDHMklNNmDABJ06cwM6dO51dSr114cIFTJ48GUlJSfDw8HB2OfWeyWRCbGws3nnnHQBAx44dceLECSxevBijRo1ycnX1yw8//IDvvvsOy5cvR3R0NI4cOYIpU6YgLCyMvSK7MRgMGDFiBERRxKJFi5xdTr1z8OBBfPzxxzh06BAEQXB2OS6B0wzsJDAwEHK5vMIZ5RkZGQgJCXFSVfXLxIkT8euvv2Lr1q1o2rSpeXtISAhKSkqQk5Njsb+79u7gwYPIzMxEp06doFAooFAosH37dnzyySdQKBQIDg5mv24SGhqKtm3bWmxr06YN0tLSAMDcE/5uAi+99BJeffVVPPLII2jfvj2eeOIJPP/885gzZw4A9qo61vQmJCSkwgm/paWlyM7Odtv+lQfZ1NRUJCUlmUdlAfar3O+//47MzEw0a9bM/N/81NRUvPDCC2jevDkA9upWDLN2olKp0LlzZ2zevNm8zWQyYfPmzYiLi3NiZc4niiImTpyINWvWYMuWLYiMjLS4v3PnzlAqlRa9O336NNLS0tyyd/feey+OHz+OI0eOmL9iY2MxcuRI8/fs1z969uxZYam3v/76CxEREQCAyMhIhISEWPRLp9Nh7969btevwsJCyGSW/xuQy+UwmUwA2KvqWNObuLg45OTk4ODBg+Z9tmzZApPJhG7dujm8ZmcrD7JnzpzBpk2b0KhRI4v72a8yTzzxBI4dO2bx3/ywsDC89NJL2LBhAwD2qgJnn4HWkK1YsUJUq9Xi0qVLxVOnTolPP/206OfnJ6anpzu7NKd69tlnRV9fX3Hbtm3ilStXzF+FhYXmfcaNGyc2a9ZM3LJli3jgwAExLi5OjIuLc2LV9cvNqxmIIvt1s3379okKhUJ8++23xTNnzojfffedqNFoxGXLlpn3effdd0U/Pz/x559/Fo8dOyY+8MADYmRkpFhUVOTEyh1v1KhRYpMmTcRff/1VTElJEVevXi0GBgaKL7/8snkfd+5VXl6eePjwYfHw4cMiAPGjjz4SDx8+bD773pre3HfffWLHjh3FvXv3ijt37hSjoqLERx991Fkvya6q61dJSYk4ePBgsWnTpuKRI0cs/tuv1+vNj+Eu/arpZ+tWt65mIIru0ytrMMza2YIFC8RmzZqJKpVK7Nq1q7hnzx5nl+R0ACr9WrJkiXmfoqIicfz48aK/v7+o0WjEoUOHileuXHFe0fXMrWGW/bL0yy+/iO3atRPVarV4++23i59//rnF/SaTSZw+fboYHBwsqtVq8d577xVPnz7tpGqdR6fTiZMnTxabNWsmenh4iC1atBBfe+01i3Dhzr3aunVrpf+tGjVqlCiK1vUmKytLfPTRR0Vvb29Rq9WKY8aMEfPy8pzwauyvun6lpKRU+d/+rVu3mh/DXfpV08/WrSoLs+7SK2sIonjTpV6IiIiIiFwI58wSERERkctimCUiIiIil8UwS0REREQui2GWiIiIiFwWwywRERERuSyGWSIiIiJyWQyzREREROSyGGaJiIiIyGUxzBIRuSlBEJCYmOjsMoiI6oRhlojICUaPHg1BECp83Xfffc4ujYjIpSicXQARkbu67777sGTJEottarXaSdUQEbkmjswSETmJWq1GSEiIxZe/vz+AsikAixYtQv/+/eHp6YkWLVrgxx9/tDj++PHj+L//+z94enqiUaNGePrpp5Gfn2+xz3//+19ER0dDrVYjNDQUEydOtLj/2rVrGDp0KDQaDaKiovC///3Pvi+aiMjGGGaJiOqp6dOnY9iwYTh69ChGjhyJRx55BMnJyQCAgoICxMfHw9/fH/v378eqVauwadMmi7C6aNEiTJgwAU8//TSOHz+O//3vf2jVqpXFc8yePRsjRozAsWPHMGDAAIwcORLZ2dkOfZ1ERHUhiKIoOrsIIiJ3M3r0aCxbtgweHh4W26dNm4Zp06ZBEASMGzcOixYtMt/XvXt3dOrUCZ999hm++OILvPLKK7hw4QK8vLwAAGvXrsWgQYNw+fJlBAcHo0mTJhgzZgzeeuutSmsQBAGvv/463nzzTQBlAdnb2xvr1q3j3F0ichmcM0tE5CT33HOPRVgFgICAAPP3cXFxFvfFxcXhyJEjAIDk5GTExMSYgywA9OzZEyaTCadPn4YgCLh8+TLuvffeamvo0KGD+XsvLy9otVpkZmbW9iURETkcwywRkZN4eXlV+NjfVjw9Pa3aT6lUWtwWBAEmk8keJRER2QXnzBIR1VN79uypcLtNmzYAgDZt2uDo0aMoKCgw3//HH39AJpOhdevW8PHxQfPmzbF582aH1kxE5GgcmSUichK9Xo/09HSLbQqFAoGBgQCAVatWITY2FnfeeSe+++477Nu3D1999RUAYOTIkZg5cyZGjRqFWbNm4erVq3juuefwxBNPIDg4GAAwa9YsjBs3DkFBQejfvz/y8vLwxx9/4LnnnnPsCyUisiOGWSIiJ1m/fj1CQ0MttrVu3Rp//vkngLKVBlasWIHx48cjNDQU33//Pdq2bQsA0Gg02LBhAyZPnowuXbpAo9Fg2LBh+Oijj8yPNWrUKBQXF2PevHl48cUXERgYiIceeshxL5CIyAG4mgERUT0kCALWrFmDIUOGOLsUIqJ6jXNmiYiIiMhlMcwSERERkcvinFkionqIM8CIiKzDkVkiIiIiclkMs0RERETkshhmiYiIiMhlMcwSERERkctimCUiIiIil8UwS0REREQui2GWiIiIiFwWwywRERERuaz/Bx3/wDwrwqVoAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "\n",
    "def train_one_epoch(net, optimizer, train_loader):\n",
    "    # Cross Entropy loss for classification when not using a softmax layer in the network\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    net.train()\n",
    "    avg_loss = 0\n",
    "    for data, target in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        output = net(data)\n",
    "        loss_net = loss(output, target.long())\n",
    "        loss_net.backward()\n",
    "        optimizer.step()\n",
    "        avg_loss += loss_net.item()\n",
    "\n",
    "    return avg_loss / len(train_loader)\n",
    "\n",
    "\n",
    "# Create the tiny CNN with 10 output classes\n",
    "N_EPOCHS = 150\n",
    "\n",
    "# Create a train data loader\n",
    "train_dataset = TensorDataset(torch.Tensor(x_train), torch.Tensor(y_train))\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=64)\n",
    "\n",
    "# Create a test data loader to supply batches for network evaluation (test)\n",
    "test_dataset = TensorDataset(torch.Tensor(x_test), torch.Tensor(y_test))\n",
    "test_dataloader = DataLoader(test_dataset)\n",
    "\n",
    "# Train the network with Adam, output the test set accuracy every epoch\n",
    "net = TinyCNN(10)\n",
    "losses_bits = []\n",
    "optimizer = torch.optim.Adam(net.parameters())\n",
    "for _ in tqdm(range(N_EPOCHS), desc=\"Training\"):\n",
    "    losses_bits.append(train_one_epoch(net, optimizer, train_dataloader))\n",
    "\n",
    "fig = plt.figure(figsize=(8, 4))\n",
    "plt.plot(losses_bits)\n",
    "plt.ylabel(\"Cross Entropy Loss\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.title(\"Training set loss during training\")\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "303fdc93",
   "metadata": {},
   "source": [
    "### Test the torch network in fp32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "822f1736",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy for fp32 weights and activations: 98.22%\n"
     ]
    }
   ],
   "source": [
    "def test_torch(net, test_loader):\n",
    "    \"\"\"Test the network: measure accuracy on the test set.\"\"\"\n",
    "\n",
    "    # Freeze normalization layers\n",
    "    net.eval()\n",
    "\n",
    "    all_y_pred = np.zeros((len(test_loader)), dtype=np.int64)\n",
    "    all_targets = np.zeros((len(test_loader)), dtype=np.int64)\n",
    "\n",
    "    # Iterate over the batches\n",
    "    idx = 0\n",
    "    for data, target in test_loader:\n",
    "        # Accumulate the ground truth labels\n",
    "        endidx = idx + target.shape[0]\n",
    "        all_targets[idx:endidx] = target.numpy()\n",
    "\n",
    "        # Run forward and get the predicted class id\n",
    "        output = net(data).argmax(1).detach().numpy()\n",
    "        all_y_pred[idx:endidx] = output\n",
    "\n",
    "        idx += target.shape[0]\n",
    "\n",
    "    # Print out the accuracy as a percentage\n",
    "    n_correct = np.sum(all_targets == all_y_pred)\n",
    "    print(\n",
    "        f\"Test accuracy for fp32 weights and activations: \"\n",
    "        f\"{n_correct / len(test_loader) * 100:.2f}%\"\n",
    "    )\n",
    "\n",
    "\n",
    "test_torch(net, test_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a14bc3f",
   "metadata": {},
   "source": [
    "### Define the Concrete ML testing function\n",
    "\n",
    "We introduce the `test_with_concrete` function which allows us to test a Concrete ML model in one of two modes:\n",
    "- in FHE\n",
    "- in the clear, using simulated FHE execution\n",
    "\n",
    "Note that it is trivial to toggle between between the two modes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "978a6c4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_with_concrete(quantized_module, test_loader, use_sim):\n",
    "    \"\"\"Test a neural network that is quantized and compiled with Concrete ML.\"\"\"\n",
    "\n",
    "    # Casting the inputs into int64 is recommended\n",
    "    all_y_pred = np.zeros((len(test_loader)), dtype=np.int64)\n",
    "    all_targets = np.zeros((len(test_loader)), dtype=np.int64)\n",
    "\n",
    "    # Iterate over the test batches and accumulate predictions and ground truth labels in a vector\n",
    "    idx = 0\n",
    "    for data, target in tqdm(test_loader):\n",
    "        data = data.numpy()\n",
    "        target = target.numpy()\n",
    "\n",
    "        fhe_mode = \"simulate\" if use_sim else \"execute\"\n",
    "\n",
    "        # Quantize the inputs and cast to appropriate data type\n",
    "        y_pred = quantized_module.forward(data, fhe=fhe_mode)\n",
    "\n",
    "        endidx = idx + target.shape[0]\n",
    "\n",
    "        # Accumulate the ground truth labels\n",
    "        all_targets[idx:endidx] = target\n",
    "\n",
    "        # Get the predicted class id and accumulate the predictions\n",
    "        y_pred = np.argmax(y_pred, axis=1)\n",
    "        all_y_pred[idx:endidx] = y_pred\n",
    "\n",
    "        # Update the index\n",
    "        idx += target.shape[0]\n",
    "\n",
    "    # Compute and report results\n",
    "    n_correct = np.sum(all_targets == all_y_pred)\n",
    "\n",
    "    return n_correct / len(test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cccf0b5",
   "metadata": {},
   "source": [
    "### Test the network using Simulation\n",
    "\n",
    "Note that this is not a test in FHE. The simulated FHE mode gives \n",
    "insight about the impact of FHE execution on the accuracy.\n",
    "\n",
    "The torch neural network is converted to FHE by Concrete ML using a dedicated function, `compile_torch_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "33250d57",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/450 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  6%|▌         | 25/450 [00:00<00:01, 248.54it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|█         | 50/450 [00:00<00:01, 249.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 17%|█▋        | 75/450 [00:00<00:01, 249.43it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 22%|██▏       | 101/450 [00:00<00:01, 250.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 28%|██▊       | 127/450 [00:00<00:01, 250.25it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 34%|███▍      | 153/450 [00:00<00:01, 250.00it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|███▉      | 179/450 [00:00<00:01, 249.73it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 45%|████▌     | 204/450 [00:00<00:00, 249.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 51%|█████     | 230/450 [00:00<00:00, 249.73it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 57%|█████▋    | 255/450 [00:01<00:00, 249.10it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 62%|██████▏   | 280/450 [00:01<00:00, 249.21it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████▊   | 305/450 [00:01<00:00, 248.50it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 73%|███████▎  | 330/450 [00:01<00:00, 247.97it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|███████▉  | 355/450 [00:01<00:00, 248.17it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|████████▍ | 380/450 [00:01<00:00, 248.22it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 406/450 [00:01<00:00, 249.31it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 96%|█████████▌| 431/450 [00:01<00:00, 248.91it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 450/450 [00:01<00:00, 249.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Simulated FHE execution for 6 bit network accuracy: 0.98%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "n_bits = 6\n",
    "\n",
    "q_module = compile_torch_model(net, x_train, rounding_threshold_bits=6, p_error=0.1)\n",
    "\n",
    "start_time = time.time()\n",
    "accs = test_with_concrete(\n",
    "    q_module,\n",
    "    test_dataloader,\n",
    "    use_sim=True,\n",
    ")\n",
    "sim_time = time.time() - start_time\n",
    "\n",
    "print(f\"Simulated FHE execution for {n_bits} bit network accuracy: {accs:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2875e825",
   "metadata": {},
   "source": [
    "### Generate Keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6e8b6471",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keygen time: 1.01s\n"
     ]
    }
   ],
   "source": [
    "# Generate keys first\n",
    "t = time.time()\n",
    "q_module.fhe_circuit.keygen()\n",
    "print(f\"Keygen time: {time.time()-t:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4dd257f",
   "metadata": {},
   "source": [
    "### 3. Execute in FHE on encrypted data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5a82392b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/100 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  1%|          | 1/100 [00:02<04:26,  2.69s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  2%|▏         | 2/100 [00:05<04:04,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 3/100 [00:13<08:37,  5.33s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  4%|▍         | 4/100 [00:16<06:45,  4.22s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|▌         | 5/100 [00:18<05:39,  3.58s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  6%|▌         | 6/100 [00:22<05:58,  3.81s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  7%|▋         | 7/100 [00:30<07:47,  5.03s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  8%|▊         | 8/100 [00:33<06:30,  4.24s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  9%|▉         | 9/100 [00:35<05:39,  3.73s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 10%|█         | 10/100 [00:38<05:01,  3.35s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|█         | 11/100 [00:45<06:37,  4.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 12%|█▏        | 12/100 [00:47<05:40,  3.87s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 13%|█▎        | 13/100 [00:50<05:00,  3.45s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 14%|█▍        | 14/100 [00:52<04:33,  3.18s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 15%|█▌        | 15/100 [00:55<04:12,  2.97s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|█▌        | 16/100 [00:57<03:57,  2.83s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 17%|█▋        | 17/100 [01:00<03:45,  2.72s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 18%|█▊        | 18/100 [01:02<03:37,  2.65s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 19%|█▉        | 19/100 [01:05<03:30,  2.60s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|██        | 20/100 [01:07<03:31,  2.65s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|██        | 21/100 [01:10<03:25,  2.60s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 22%|██▏       | 22/100 [01:12<03:20,  2.56s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 23%|██▎       | 23/100 [01:15<03:15,  2.54s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 24%|██▍       | 24/100 [01:17<03:11,  2.52s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 25%|██▌       | 25/100 [01:20<03:08,  2.51s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|██▌       | 26/100 [01:22<03:05,  2.51s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 27%|██▋       | 27/100 [01:25<03:02,  2.50s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 28%|██▊       | 28/100 [01:27<02:59,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 29%|██▉       | 29/100 [01:30<02:56,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 30%|███       | 30/100 [01:32<02:54,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 31%|███       | 31/100 [01:35<02:51,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|███▏      | 32/100 [01:37<02:48,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 33%|███▎      | 33/100 [01:41<03:08,  2.81s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 34%|███▍      | 34/100 [01:44<03:06,  2.83s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 35%|███▌      | 35/100 [01:47<03:12,  2.96s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 36%|███▌      | 36/100 [01:49<03:01,  2.83s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|███▋      | 37/100 [01:52<02:51,  2.73s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 38%|███▊      | 38/100 [01:54<02:44,  2.65s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 39%|███▉      | 39/100 [01:57<02:38,  2.60s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|████      | 40/100 [01:59<02:34,  2.58s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 41%|████      | 41/100 [02:02<02:30,  2.55s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|████▏     | 42/100 [02:04<02:26,  2.52s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 43%|████▎     | 43/100 [02:07<02:22,  2.51s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 44%|████▍     | 44/100 [02:09<02:21,  2.52s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 45%|████▌     | 45/100 [02:12<02:17,  2.50s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 46%|████▌     | 46/100 [02:14<02:14,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|████▋     | 47/100 [02:17<02:11,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 48%|████▊     | 48/100 [02:19<02:08,  2.47s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 49%|████▉     | 49/100 [02:22<02:05,  2.47s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 50/100 [02:24<02:03,  2.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 51%|█████     | 51/100 [02:27<02:00,  2.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 52%|█████▏    | 52/100 [02:29<01:57,  2.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|█████▎    | 53/100 [02:31<01:55,  2.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 54%|█████▍    | 54/100 [02:34<01:52,  2.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 55%|█████▌    | 55/100 [02:36<01:50,  2.46s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 56%|█████▌    | 56/100 [02:40<02:03,  2.82s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 57%|█████▋    | 57/100 [02:44<02:21,  3.29s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|█████▊    | 58/100 [02:48<02:20,  3.34s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 59%|█████▉    | 59/100 [02:50<02:06,  3.10s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|██████    | 60/100 [02:53<01:56,  2.91s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 61%|██████    | 61/100 [02:55<01:48,  2.78s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 62%|██████▏   | 62/100 [02:58<01:41,  2.68s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|██████▎   | 63/100 [03:00<01:36,  2.62s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 64%|██████▍   | 64/100 [03:03<01:35,  2.65s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 65%|██████▌   | 65/100 [03:05<01:31,  2.60s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 66%|██████▌   | 66/100 [03:08<01:27,  2.56s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 67%|██████▋   | 67/100 [03:10<01:23,  2.54s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████▊   | 68/100 [03:13<01:20,  2.52s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 69%|██████▉   | 69/100 [03:15<01:17,  2.51s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 70%|███████   | 70/100 [03:18<01:14,  2.50s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 71%|███████   | 71/100 [03:20<01:12,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 72%|███████▏  | 72/100 [03:23<01:09,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 73%|███████▎  | 73/100 [03:25<01:07,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|███████▍  | 74/100 [03:28<01:04,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 75%|███████▌  | 75/100 [03:30<01:01,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 76%|███████▌  | 76/100 [03:33<00:59,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 77%|███████▋  | 77/100 [03:35<00:57,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 78%|███████▊  | 78/100 [03:38<00:54,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|███████▉  | 79/100 [03:40<00:52,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████  | 80/100 [03:43<00:49,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 81%|████████  | 81/100 [03:45<00:47,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 82%|████████▏ | 82/100 [03:48<00:44,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 83%|████████▎ | 83/100 [03:50<00:42,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|████████▍ | 84/100 [03:53<00:39,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 85%|████████▌ | 85/100 [03:55<00:37,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 86%|████████▌ | 86/100 [03:58<00:34,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 87%|████████▋ | 87/100 [04:00<00:32,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 88%|████████▊ | 88/100 [04:02<00:29,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|████████▉ | 89/100 [04:05<00:27,  2.48s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 90%|█████████ | 90/100 [04:07<00:24,  2.49s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 91%|█████████ | 91/100 [04:11<00:23,  2.67s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 92%|█████████▏| 92/100 [04:13<00:20,  2.62s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 93%|█████████▎| 93/100 [04:16<00:18,  2.59s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 94%|█████████▍| 94/100 [04:20<00:18,  3.06s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|█████████▌| 95/100 [04:23<00:15,  3.09s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 96%|█████████▌| 96/100 [04:25<00:11,  2.93s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 97%|█████████▋| 97/100 [04:28<00:08,  2.80s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 98%|█████████▊| 98/100 [04:30<00:05,  2.70s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 99%|█████████▉| 99/100 [04:33<00:02,  2.64s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 100/100 [04:35<00:00,  2.59s/it]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "100%|██████████| 100/100 [04:35<00:00,  2.76s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time per inference in FHE: 2.76 with 100.00% accuracy\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Run inference in FHE on a single encrypted example\n",
    "mini_test_dataset = TensorDataset(torch.Tensor(x_test[:100, :]), torch.Tensor(y_test[:100]))\n",
    "mini_test_dataloader = DataLoader(mini_test_dataset)\n",
    "\n",
    "t = time.time()\n",
    "accuracy_test = test_with_concrete(\n",
    "    q_module,\n",
    "    mini_test_dataloader,\n",
    "    use_sim=False,\n",
    ")\n",
    "elapsed_time = time.time() - t\n",
    "time_per_inference = elapsed_time / len(mini_test_dataset)\n",
    "accuracy_percentage = 100 * accuracy_test\n",
    "\n",
    "print(\n",
    "    f\"Time per inference in FHE: {time_per_inference:.2f} \"\n",
    "    f\"with {accuracy_percentage:.2f}% accuracy\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "550f48bf",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "In this example, a simple CNN model is trained with torch and reach 98% accuracy in clear. The model is then converted to FHE and evaluated over 100 samples in FHE.\n",
    "\n",
    "The model in FHE achieves **the same accuracy** as the original torch model with a FHE execution time of **2.9 seconds** per image."
   ]
  }
 ],
 "metadata": {
  "execution": {
   "timeout": 10800
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}