Homework2/SVM_StefanoIvancich_1227846.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM for classification, without and with kernels\n",
    "\n",
    "In this notebook we are going to explore the use of Support Vector Machines (SVMs) for image classification. We are going to use the famous MNIST dataset, that is a dataset of handwritten digits. We get the data from mldata.org, that is a public repository for machine learning data.\n",
    "\n",
    "The dataset consists of 70,000 images of handwritten digits (i.e., 0, 1, ... 9). Each image is 28 pixels by 28 pixels and we can think of it as a vector of 28x28 = 784 numbers. Each number is an integer between 0 and 255. For each image we have the corresponding label (i.e., 0, 1, ..., 9)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load the required packages\n",
    "\n",
    "%matplotlib inline  \n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sklearn\n",
    "from sklearn.datasets import fetch_openml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fix your ID (\"numero di matricola\") and the seed for random generator\n",
    "ID = 1227846\n",
    "np.random.seed(ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load the dataset. 'data' contains the input, 'target' contains the label. We normalize the data by dividing each value by 255 so that each value is in [0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load the MNIST dataset and let's normalize the features so that each value is in [0,1]\n",
    "mnist = fetch_openml('mnist_784', version=1)\n",
    "# rescale the data\n",
    "X, y = mnist.data / 255., mnist.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now split into training and test. We keep 500 samples in the training set. Make sure that each label is present at least 10 times\n",
    "in training. If it is not, then keep adding permutations to the initial data until this \n",
    "happens.\n",
    "\n",
    "**IMPORTANT**: if you cannot run the SVM with 500 samples or 1000 samples (see below), try with a smaller number of samples (e.g. 200 here and 400 below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels and frequencies in training dataset: \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(array(['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'], dtype=object),\n",
       " array([47, 60, 50, 38, 43, 48, 59, 59, 51, 45], dtype=int64))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#random permute the data and split into training and test taking the first 500\n",
    "#data samples as training and the rests as test\n",
    "permutation = np.random.permutation(X.shape[0])\n",
    "\n",
    "X = X[permutation]\n",
    "y = y[permutation]\n",
    "\n",
    "m_training = 500\n",
    "\n",
    "X_train, X_test = X[:m_training], X[m_training:]\n",
    "y_train, y_test = y[:m_training], y[m_training:]\n",
    "\n",
    "print(\"Labels and frequencies in training dataset: \")\n",
    "np.unique(y_train, return_counts = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now provide a function to print an image in a dataset, the corresponding true label, and the index of the image in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for plotting a digit and printing the corresponding labe\n",
    "def plot_digit(X_matrix, labels, index):\n",
    "    print(\"INPUT:\")\n",
    "    plt.imshow(\n",
    "        X_matrix[index].reshape(28,28),\n",
    "        cmap          = plt.cm.gray_r,\n",
    "        interpolation = \"nearest\"\n",
    "    )\n",
    "    plt.show()\n",
    "    print(\"LABEL: %s\" % labels[index])\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, let's print the 100-th image in X_train and the 40,000-th image in X_test and their true labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANpElEQVR4nO3db6hc9Z3H8c9HrQoxD6K5mpjKplt8sLJglEEElxKRFRXBf3RtHhQXo5Go+Ic+ULJIhRgI67b+YynEVZoVVym0okGxMVJQUapjyOaPcdesZts0iblGoTEQmuh3H9yT5Rrv/OZm5sycSb7vFwwzc75z7vky937umTm/OfNzRAjA8e+EphsAMByEHUiCsANJEHYgCcIOJHHSMDc2e/bsmD9//jA3CaSyfft2ffbZZ56q1lfYbV8h6TFJJ0r6t4hYWXr8/Pnz1W63+9kkgIJWq9Wx1vPLeNsnSvpXSVdKOk/SItvn9frzAAxWP+/ZL5K0LSI+joi/SHpe0jX1tAWgbv2EfZ6kP066v6Na9g22l9hu226Pj4/3sTkA/egn7FMdBPjWZ28jYlVEtCKiNTY21sfmAPSjn7DvkHTOpPvflbSzv3YADEo/YX9P0rm2v2f7ZEk/kvRSPW0BqFvPQ28Rccj2nZJ+q4mht6cjYkttnQGoVV/j7BHxiqRXauoFwADxcVkgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5IY6ldJYzDefPPNjrV33nmnuO5DDz1UrM+ZM6dYv+WWW4r1mTNndqwtXbq0uC7qxZ4dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnP0Y8PbbbxfrDz/8cMfamjVr+tr2vn37ivX77ruvWJ81a1bH2imnnFJcd8GCBcX6hRdeWKzjm9izA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLOPgP379xfry5cvL9ZfffXVOtup1RdffNGxtnjx4uK6DzzwQLHOOPvR6SvstrdL2ifpK0mHIqJVR1MA6lfHnv3SiPishp8DYIB4zw4k0W/YQ9Ja2+/bXjLVA2wvsd223R4fH+9zcwB61W/YL4mICyVdKekO2z848gERsSoiWhHRGhsb63NzAHrVV9gjYmd1vUfSC5IuqqMpAPXrOey2Z9ieefi2pMslba6rMQD16udo/FmSXrB9+Of8R0SM7oDvCNu9e3exPsrj6Dh29Bz2iPhY0vk19gJggBh6A5Ig7EAShB1IgrADSRB2IAlOcR0BzzzzTNMtIAH27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPsI2Djxo1Nt9DRSSf19ydy6NChntd9+eWXi/Vly5YV66eeemrP2z4esWcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ0fRypUri/VuX3O9bt26nre9fv36Yv3gwYPFOuPs38SeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJx9BFx99dXF+rZt24r1TZs2daw9//zzxXWvu+66Yr3b+exMJ33s6Lpnt/207T22N09adrrt12x/VF3PGmybAPo1nZfxv5R0xRHL7pf0ekScK+n16j6AEdY17BHxhqTPj1h8jaTV1e3Vkq6tuS8ANev1AN1ZEbFLkqrrMzs90PYS223b7fHx8R43B6BfAz8aHxGrIqIVEa2xsbFBbw5AB72G/VPbcyWput5TX0sABqHXsL8k6abq9k2SXqynHQCD0nWc3fZzkhZKmm17h6SfSlop6Ve2F0v6g6QfDrLJ493NN99crF9//fXF+v79+zvWzj777OK6tov1Rx99tFh/6623ivV+3HPPPcX6jBkzBrbt41HXsEfEog6ly2ruBcAA8XFZIAnCDiRB2IEkCDuQBGEHkuAU12PAzp07i/UdO3Z0rG3ZsqW47ooVK4r1d999t1g/cOBAsd6PmTNnFusnnMC+6mjwbAFJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzD8HWrVuL9bvuuqtY//DDD4v10jj7KOt2eu2cOXOG1EkO7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2Wuwe/fuYv2xxx4r1tetW1dnO8eMk08+uVi//fbbh9RJDuzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlrcOmllxbr3c5Hz+rgwYPF+t13312sL1++vFgvfe98t3Ppj0dd9+y2n7a9x/bmScsetP0n2xuqy1WDbRNAv6bzMv6Xkq6YYvkjEbGgurxSb1sA6tY17BHxhqTPh9ALgAHq5wDdnbY3Vi/zZ3V6kO0lttu22+Pj431sDkA/eg37LyR9X9ICSbsk/azTAyNiVUS0IqI1NjbW4+YA9KunsEfEpxHxVUR8LelJSRfV2xaAuvUUdttzJ929TtLmTo8FMBq6jrPbfk7SQkmzbe+Q9FNJC20vkBSStku6bYA9joQNGzZ0rO3du3eInRw/vv7662L98ccf76t+7733dqzNmtXxMJMkafbs2cX60qVLi/VR1DXsEbFoisVPDaAXAAPEx2WBJAg7kARhB5Ig7EAShB1IglNcK4888kixvmzZso61AwcO1N0OatDtd1pywgnl/eAHH3xQrD/xxBM9b3tQ2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs1e6fa0xY+m5dDv99qmnyid+3nDDDcX6woULj7alvrFnB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGdP7uKLLy7WN28uTwnw5Zdf1tnOMeO0004r1psYR++GPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4+3HgzDPP7Fh78skni+tefvnlxfqzzz5brK9YsaJY/+STT4p1DE/XPbvtc2z/zvZW21ts310tP932a7Y/qq7LE14DaNR0XsYfkvSTiPgbSRdLusP2eZLul/R6RJwr6fXqPoAR1TXsEbErItZXt/dJ2ippnqRrJK2uHrZa0rWDahJA/47qAJ3t+ZIukPR7SWdFxC5p4h+CpCnfONpeYrttuz0+Pt5ftwB6Nu2w2z5N0q8l3RMRf57uehGxKiJaEdEaGxvrpUcANZhW2G1/RxNBfzYiflMt/tT23Ko+V9KewbQIoA5dh95sW9JTkrZGxM8nlV6SdJOkldX1iwPpMIFu0wPPmzevWF+zZk3H2vnnn99TT4ctXry4WL/22vKhmkWLFnWsbdiwoaeepmvv3r0da92+KrqbG2+8sa/1mzCdcfZLJP1Y0ibbh387yzQR8l/ZXizpD5J+OJgWAdSha9gj4i1J7lC+rN52AAwKH5cFkiDsQBKEHUiCsANJEHYgCU5xHYKlS5cW6xdccEGxfuutt9bZTq3OOOOMYn3t2rVD6uTbVq1a1bHW7dTcyy4rDzTddtttPfXUJPbsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5CEI2JoG2u1WtFut4e2PSCbVquldrs95Vmq7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgia5ht32O7d/Z3mp7i+27q+UP2v6T7Q3V5arBtwugV9OZJOKQpJ9ExHrbMyW9b/u1qvZIRPzL4NoDUJfpzM++S9Ku6vY+21slzRt0YwDqdVTv2W3Pl3SBpN9Xi+60vdH207ZndVhnie227fb4+HhfzQLo3bTDbvs0Sb+WdE9E/FnSLyR9X9ICTez5fzbVehGxKiJaEdEaGxuroWUAvZhW2G1/RxNBfzYifiNJEfFpRHwVEV9LelLSRYNrE0C/pnM03pKekrQ1In4+afncSQ+7TtLm+tsDUJfpHI2/RNKPJW2yvaFatkzSItsLJIWk7ZKOvTlsgUSmczT+LUlTfQ/1K/W3A2BQ+AQdkARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCUfE8DZmj0v630mLZkv6bGgNHJ1R7W1U+5LorVd19vZXETHl978NNezf2rjdjohWYw0UjGpvo9qXRG+9GlZvvIwHkiDsQBJNh31Vw9svGdXeRrUvid56NZTeGn3PDmB4mt6zAxgSwg4k0UjYbV9h+79sb7N9fxM9dGJ7u+1N1TTU7YZ7edr2HtubJy073fZrtj+qrqecY6+h3kZiGu/CNOONPndNT38+9Pfstk+U9N+S/l7SDknvSVoUER8MtZEObG+X1IqIxj+AYfsHkr6U9O8R8bfVsn+W9HlErKz+Uc6KiPtGpLcHJX3Z9DTe1WxFcydPMy7pWkn/qAafu0Jf/6AhPG9N7NkvkrQtIj6OiL9Iel7SNQ30MfIi4g1Jnx+x+BpJq6vbqzXxxzJ0HXobCRGxKyLWV7f3STo8zXijz12hr6FoIuzzJP1x0v0dGq353kPSWtvv217SdDNTOCsidkkTfzySzmy4nyN1ncZ7mI6YZnxknrtepj/vVxNhn2oqqVEa/7skIi6UdKWkO6qXq5ieaU3jPSxTTDM+Enqd/rxfTYR9h6RzJt3/rqSdDfQxpYjYWV3vkfSCRm8q6k8Pz6BbXe9puJ//N0rTeE81zbhG4LlrcvrzJsL+nqRzbX/P9smSfiTppQb6+BbbM6oDJ7I9Q9LlGr2pqF+SdFN1+yZJLzbYyzeMyjTenaYZV8PPXePTn0fE0C+SrtLEEfn/kfRPTfTQoa+/lvSf1WVL071Jek4TL+sOauIV0WJJZ0h6XdJH1fXpI9TbM5I2SdqoiWDNbai3v9PEW8ONkjZUl6uafu4KfQ3leePjskASfIIOSIKwA0kQdiAJwg4kQdiBJAg7kARhB5L4P20NGmIlmDmFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LABEL: 2\n",
      "INPUT:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAOKElEQVR4nO3df6xU9ZnH8c8DS2MCxIBcFCzZixXNGhRKRtyAaTBoRYjB/lEFk4oJCdVgAkmjS7p/YPxDzbLSbHSDUuXHmi6NSTUSQ7olpEaLCXE0rOKSXX9d6i03cAnGgiay0Gf/uIfNLdzznWHOmTkDz/uVTGbmPHPmPEz43DNzvmfma+4uAJe+UVU3AKAzCDsQBGEHgiDsQBCEHQjibzq5sUmTJnlvb28nNwmE0tfXp2PHjtlItUJhN7NFkv5F0mhJL7r706nH9/b2ql6vF9kkgIRarZZba/ltvJmNlvSvku6SdIOk5WZ2Q6vPB6C9inxmnyvpE3f/zN1PSfq1pKXltAWgbEXCfrWkL4bd78+W/RUzW2VmdTOrDw4OFtgcgCKKhH2kgwDnnXvr7pvdvebutZ6engKbA1BEkbD3S5o27P53JR0u1g6AdikS9nclzTCz6Wb2HUnLJO0spy0AZWt56M3dT5vZI5L+Q0NDb1vc/aPSOgNQqkLj7O6+S9KuknoB0EacLgsEQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEB2dsvlS1d/fn6zv3bu3UL2RrVu35tZOnjxZ6Lnnzp2brH/11VfJ+qxZs3JrR44cSa570003Jev33ntvsj5v3rzc2qhR8fZz8f7FQFCEHQiCsANBEHYgCMIOBEHYgSAIOxCEuXvHNlar1bxer3dse2V67LHHcmvbtm1Lrnvs2LGSu0EzFi5cmFtbv359ct1bb7217HY6olarqV6v20i1QifVmFmfpBOSzkg67e61Is8HoH3KOIPuNndn1wV0OT6zA0EUDbtL+p2ZvWdmq0Z6gJmtMrO6mdUHBwcLbg5Aq4qGfb67z5F0l6TVZvaDcx/g7pvdvebutZ6enoKbA9CqQmF398PZ9VFJr0lKf0UKQGVaDruZjTWz8WdvS/qhpANlNQagXEWOxl8p6TUzO/s8/+7uvy2lqy60YcOG3Fr2GqDL7NmzJ7c2bty45LoX6zh7Ssthd/fPJOX/MgGArsLQGxAEYQeCIOxAEIQdCIKwA0HwU9JNuu2223Jrb775ZucaKdn06dOT9bFjx3aok/N9/fXXyfrnn3/e8nM3+qr14cOHk/WpU6e2vO2qsGcHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAYZ2/SCy+8kFvbtGlTct1Dhw4V2va1116brC9atKjl554zZ06yfvnll7f83EV9+umnyXpqSmZJSv0M2qlTp5LrnjlzJlm/GLFnB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgGGdv0owZM3JrGzdu7GAnl4433ngjWV+5cmWyXmQ6sZkzZybr06ZNa/m5uxV7diAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgnF2tNXrr7+eW7vvvvuS6zb6znkRkydPbttzd6uGe3Yz22JmR83swLBlE81st5l9nF1PaG+bAIpq5m38Nknn/hTKOkl73H2GpD3ZfQBdrGHY3f0tScfPWbxU0vbs9nZJ95TcF4CStXqA7kp3H5Ck7Dr3A5CZrTKzupnVi5zLDKCYth+Nd/fN7l5z91pPT0+7NwcgR6thP2JmUyQpuz5aXksA2qHVsO+UtCK7vUJS/vgKgK7QcJzdzHZIWiBpkpn1S1ov6WlJr5jZSkl/lPTjdjaJ7pUaR5ekBx54ILfWznF0SZo/f35u7bnnnmvrtrtRw7C7+/Kc0sKSewHQRpwuCwRB2IEgCDsQBGEHgiDsQBB8xfUSd/r06WT9xIkTyfpDDz2UrO/cuTNZ//bbb5P1Im655ZZk/cknn8ytXXHFFWW30/XYswNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIyzXwIOHTqUW1u9enVy3V27dpXdTtPGjBmTrC9ZsiRZf/HFF5P1iRMnXnBPlzL27EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBOPsF4Hnn38+WX/qqadya1988UXZ7VyQ6dOn59YWL16cXPfZZ58tu53Q2LMDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCMs3fAmTNnkvUFCxYk6/v27UvWG/02fBFmlqzPnDkzWX/iiSdya0uXLm2pJ7Sm4Z7dzLaY2VEzOzBs2eNm9icz259d0mdHAKhcM2/jt0laNMLyX7j77OxS3c+dAGhKw7C7+1uSjnegFwBtVOQA3SNm9kH2Nn9C3oPMbJWZ1c2sPjg4WGBzAIpoNeybJH1P0mxJA5KeyXugu29295q713p6elrcHICiWgq7ux9x9zPu/hdJv5Q0t9y2AJStpbCb2ZRhd38k6UDeYwF0h4bj7Ga2Q9ICSZPMrF/SekkLzGy2JJfUJ+mnbeyx6x0/nj5++eijjybre/fuLbOdUq1duzZZf+aZ3E9w6DINw+7uy0dY/FIbegHQRpwuCwRB2IEgCDsQBGEHgiDsQBB8xbUEjaYO3rp1a4c6Od/111+frO/YsSNZb/QVVlw82LMDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCMszcp9ZNajaZULuqyyy5L1pcsWZJbe/nllws9d1HffPNNbq3RV4MHBgaS9VdffTVZP3nyZG6tynMfJOnhhx/OrW3YsKEt22TPDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM7epC+//DK31tfX19Zt33zzzcl6asz2nXfeKbTtLVu2JOupcXRJ6u/vz63V6/WWeroUVDE7Ent2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcfaLwNtvv52s33777R3qBGfNmzcvWV+3bl2yfscdd5TZTlMa7tnNbJqZ/d7MDprZR2a2Jls+0cx2m9nH2fWE9rcLoFXNvI0/Leln7v53kv5e0mozu0HSOkl73H2GpD3ZfQBdqmHY3X3A3d/Pbp+QdFDS1ZKWStqePWy7pHva1SSA4i7oAJ2Z9Ur6vqR9kq509wFp6A+CpMk566wys7qZ1VO/4wagvZoOu5mNk/QbSWvd/c/Nrufum9295u61Kk7+BzCkqbCb2RgNBf1X7n72Jz2PmNmUrD5F0tH2tAigDA2H3szMJL0k6aC7bxxW2ilphaSns+vX29Jhl5gxY0ZurdEwyu7du8tuB5LGjx+frPf29ubWXnnlleS611xzTbI+evToZH3UqO47haWZcfb5kn4i6UMz258t+7mGQv6Kma2U9EdJP25PiwDK0DDs7v4HSZZTXlhuOwDapfveawBoC8IOBEHYgSAIOxAEYQeC4CuuTRo63WBkd999d3LdyOPsU6dOza3deOONyXWnTJmSrK9ZsyZZnzVrVrIeDXt2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcfYSPPjgg8n6pEmTkvX777+/xG4uzHXXXZes33nnncl6o+/y12q13NpVV12VXBflYs8OBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0Ewzl6CcePGJevLli0rVAfKwJ4dCIKwA0EQdiAIwg4EQdiBIAg7EARhB4JoGHYzm2Zmvzezg2b2kZmtyZY/bmZ/MrP92WVx+9sF0KpmTqo5Leln7v6+mY2X9J6ZnZ314Bfu/s/taw9AWZqZn31A0kB2+4SZHZR0dbsbA1CuC/rMbma9kr4vaV+26BEz+8DMtpjZhJx1VplZ3czqg4ODhZoF0Lqmw25m4yT9RtJad/+zpE2Svidptob2/M+MtJ67b3b3mrvXenp6SmgZQCuaCruZjdFQ0H/l7q9Kkrsfcfcz7v4XSb+UNLd9bQIoqpmj8SbpJUkH3X3jsOXDp9j8kaQD5bcHoCzNHI2fL+knkj40s/3Zsp9LWm5msyW5pD5JP21LhwBK0czR+D9IGmly8l3ltwOgXTiDDgiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EIS5e+c2ZjYo6dCwRZMkHetYAxemW3vr1r4kemtVmb39rbuP+PtvHQ37eRs3q7t7rbIGErq1t27tS6K3VnWqN97GA0EQdiCIqsO+ueLtp3Rrb93al0RvrepIb5V+ZgfQOVXv2QF0CGEHgqgk7Ga2yMz+28w+MbN1VfSQx8z6zOzDbBrqesW9bDGzo2Z2YNiyiWa228w+zq5HnGOvot66YhrvxDTjlb52VU9/3vHP7GY2WtL/SLpDUr+kdyUtd/f/6mgjOcysT1LN3Ss/AcPMfiDppKR/c/eZ2bJ/knTc3Z/O/lBOcPd/6JLeHpd0suppvLPZiqYMn2Zc0j2SHlSFr12ir3vVgdetij37XEmfuPtn7n5K0q8lLa2gj67n7m9JOn7O4qWStme3t2voP0vH5fTWFdx9wN3fz26fkHR2mvFKX7tEXx1RRdivlvTFsPv96q753l3S78zsPTNbVXUzI7jS3Qekof88kiZX3M+5Gk7j3UnnTDPeNa9dK9OfF1VF2EeaSqqbxv/mu/scSXdJWp29XUVzmprGu1NGmGa8K7Q6/XlRVYS9X9K0Yfe/K+lwBX2MyN0PZ9dHJb2m7puK+sjZGXSz66MV9/P/umka75GmGVcXvHZVTn9eRdjflTTDzKab2XckLZO0s4I+zmNmY7MDJzKzsZJ+qO6binqnpBXZ7RWSXq+wl7/SLdN4500zropfu8qnP3f3jl8kLdbQEflPJf1jFT3k9HWNpP/MLh9V3ZukHRp6W/e/GnpHtFLSFZL2SPo4u57YRb29LOlDSR9oKFhTKurtVg19NPxA0v7ssrjq1y7RV0deN06XBYLgDDogCMIOBEHYgSAIOxAEYQeCIOxAEIQdCOL/AOxKLofRI6/xAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LABEL: 2\n"
     ]
    }
   ],
   "source": [
    "#let's try the plotting function\n",
    "plot_digit(X_train,y_train,100)\n",
    "plot_digit(X_test,y_test,40000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TO DO 1\n",
    "Run SVM with cross validation to pick a kernel and values of parameters. Use a 5-fold cross-validation to pick the best kernel and choice of parameters. We provide some potential choice for parameters, but change the grid if needed (e.g., it takes too long). For the SVM for classification use SVC from sklearn.svm; for the grid search we suggest you use GridSearchCV from sklearn.model_selection, but you can implement your own cross-validation for model selection if you prefer.\n",
    "\n",
    "Print the best parameters used as well as the score obtained by the best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESULTS FOR LINEAR KERNEL\n",
      "\n",
      "Best parameters set found:\n",
      "{'C': 1}\n",
      "Score with best parameters:\n",
      "0.8699999999999999\n",
      "\n",
      "All scores on the grid:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.121520</td>\n",
       "      <td>0.004837</td>\n",
       "      <td>0.025631</td>\n",
       "      <td>0.005720</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1}</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.024495</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.116609</td>\n",
       "      <td>0.009005</td>\n",
       "      <td>0.030734</td>\n",
       "      <td>0.000720</td>\n",
       "      <td>10</td>\n",
       "      <td>{'C': 10}</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.024495</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.113780</td>\n",
       "      <td>0.008695</td>\n",
       "      <td>0.028915</td>\n",
       "      <td>0.007483</td>\n",
       "      <td>100</td>\n",
       "      <td>{'C': 100}</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.024495</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "0       0.121520      0.004837         0.025631        0.005720       1   \n",
       "1       0.116609      0.009005         0.030734        0.000720      10   \n",
       "2       0.113780      0.008695         0.028915        0.007483     100   \n",
       "\n",
       "       params  split0_test_score  split1_test_score  split2_test_score  \\\n",
       "0    {'C': 1}               0.87                0.9               0.89   \n",
       "1   {'C': 10}               0.87                0.9               0.89   \n",
       "2  {'C': 100}               0.87                0.9               0.89   \n",
       "\n",
       "   split3_test_score  split4_test_score  mean_test_score  std_test_score  \\\n",
       "0               0.86               0.83             0.87        0.024495   \n",
       "1               0.86               0.83             0.87        0.024495   \n",
       "2               0.86               0.83             0.87        0.024495   \n",
       "\n",
       "   rank_test_score  \n",
       "0                1  \n",
       "1                1  \n",
       "2                1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "RESULTS FOR POLY DEGREE=2 KERNEL\n",
      "\n",
      "Best parameters set found:\n",
      "{'C': 10, 'gamma': 0.01}\n",
      "Score with best parameters:\n",
      "0.868\n",
      "\n",
      "All scores on the grid:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>param_gamma</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.170597</td>\n",
       "      <td>0.009201</td>\n",
       "      <td>0.028908</td>\n",
       "      <td>0.001269</td>\n",
       "      <td>10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 10, 'gamma': 0.01}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.029933</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.176617</td>\n",
       "      <td>0.016893</td>\n",
       "      <td>0.029519</td>\n",
       "      <td>0.002239</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 1, 'gamma': 0.1}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.164910</td>\n",
       "      <td>0.004440</td>\n",
       "      <td>0.027272</td>\n",
       "      <td>0.005684</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1, 'gamma': 1.0}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.156463</td>\n",
       "      <td>0.005498</td>\n",
       "      <td>0.022631</td>\n",
       "      <td>0.005924</td>\n",
       "      <td>10</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 10, 'gamma': 0.1}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.152231</td>\n",
       "      <td>0.006148</td>\n",
       "      <td>0.023470</td>\n",
       "      <td>0.006569</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 10, 'gamma': 1.0}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.142222</td>\n",
       "      <td>0.004779</td>\n",
       "      <td>0.030050</td>\n",
       "      <td>0.004028</td>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 100, 'gamma': 0.01}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.143717</td>\n",
       "      <td>0.006248</td>\n",
       "      <td>0.021869</td>\n",
       "      <td>0.007653</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 100, 'gamma': 0.1}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.144404</td>\n",
       "      <td>0.004761</td>\n",
       "      <td>0.026170</td>\n",
       "      <td>0.010548</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 100, 'gamma': 1.0}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.024980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.187920</td>\n",
       "      <td>0.011729</td>\n",
       "      <td>0.037104</td>\n",
       "      <td>0.003863</td>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 1, 'gamma': 0.01}</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.85</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.820</td>\n",
       "      <td>0.034641</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "3       0.170597      0.009201         0.028908        0.001269      10   \n",
       "1       0.176617      0.016893         0.029519        0.002239       1   \n",
       "2       0.164910      0.004440         0.027272        0.005684       1   \n",
       "4       0.156463      0.005498         0.022631        0.005924      10   \n",
       "5       0.152231      0.006148         0.023470        0.006569      10   \n",
       "6       0.142222      0.004779         0.030050        0.004028     100   \n",
       "7       0.143717      0.006248         0.021869        0.007653     100   \n",
       "8       0.144404      0.004761         0.026170        0.010548     100   \n",
       "0       0.187920      0.011729         0.037104        0.003863       1   \n",
       "\n",
       "  param_gamma                     params  split0_test_score  \\\n",
       "3        0.01   {'C': 10, 'gamma': 0.01}               0.84   \n",
       "1         0.1     {'C': 1, 'gamma': 0.1}               0.84   \n",
       "2           1     {'C': 1, 'gamma': 1.0}               0.84   \n",
       "4         0.1    {'C': 10, 'gamma': 0.1}               0.84   \n",
       "5           1    {'C': 10, 'gamma': 1.0}               0.84   \n",
       "6        0.01  {'C': 100, 'gamma': 0.01}               0.84   \n",
       "7         0.1   {'C': 100, 'gamma': 0.1}               0.84   \n",
       "8           1   {'C': 100, 'gamma': 1.0}               0.84   \n",
       "0        0.01    {'C': 1, 'gamma': 0.01}               0.79   \n",
       "\n",
       "   split1_test_score  split2_test_score  split3_test_score  split4_test_score  \\\n",
       "3               0.91               0.89               0.83               0.87   \n",
       "1               0.89               0.89               0.83               0.87   \n",
       "2               0.89               0.89               0.83               0.87   \n",
       "4               0.89               0.89               0.83               0.87   \n",
       "5               0.89               0.89               0.83               0.87   \n",
       "6               0.89               0.89               0.83               0.87   \n",
       "7               0.89               0.89               0.83               0.87   \n",
       "8               0.89               0.89               0.83               0.87   \n",
       "0               0.87               0.85               0.81               0.78   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  \n",
       "3            0.868        0.029933                1  \n",
       "1            0.864        0.024980                2  \n",
       "2            0.864        0.024980                2  \n",
       "4            0.864        0.024980                2  \n",
       "5            0.864        0.024980                2  \n",
       "6            0.864        0.024980                2  \n",
       "7            0.864        0.024980                2  \n",
       "8            0.864        0.024980                2  \n",
       "0            0.820        0.034641                9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "RESULTS FOR rbf KERNEL\n",
      "\n",
      "Best parameters set found:\n",
      "{'C': 10, 'gamma': 0.01}\n",
      "Score with best parameters:\n",
      "0.898\n",
      "\n",
      "All scores on the grid:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>param_gamma</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.221965</td>\n",
       "      <td>0.006803</td>\n",
       "      <td>0.031804</td>\n",
       "      <td>0.005202</td>\n",
       "      <td>10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 10, 'gamma': 0.01}</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.92</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.898</td>\n",
       "      <td>0.027857</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.197406</td>\n",
       "      <td>0.005615</td>\n",
       "      <td>0.033033</td>\n",
       "      <td>0.010303</td>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 100, 'gamma': 0.01}</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.92</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.898</td>\n",
       "      <td>0.027857</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.173476</td>\n",
       "      <td>0.008588</td>\n",
       "      <td>0.025023</td>\n",
       "      <td>0.007676</td>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 1, 'gamma': 0.01}</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.85</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.864</td>\n",
       "      <td>0.025768</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.282868</td>\n",
       "      <td>0.008318</td>\n",
       "      <td>0.033063</td>\n",
       "      <td>0.007282</td>\n",
       "      <td>10</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 10, 'gamma': 0.1}</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.49</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.452</td>\n",
       "      <td>0.049558</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.270238</td>\n",
       "      <td>0.004467</td>\n",
       "      <td>0.039076</td>\n",
       "      <td>0.007547</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 100, 'gamma': 0.1}</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.49</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.452</td>\n",
       "      <td>0.049558</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.284600</td>\n",
       "      <td>0.013232</td>\n",
       "      <td>0.040817</td>\n",
       "      <td>0.003437</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 1, 'gamma': 0.1}</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.40</td>\n",
       "      <td>0.41</td>\n",
       "      <td>0.36</td>\n",
       "      <td>0.396</td>\n",
       "      <td>0.020591</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.283568</td>\n",
       "      <td>0.018938</td>\n",
       "      <td>0.034709</td>\n",
       "      <td>0.004293</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 10, 'gamma': 1.0}</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.011662</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.275807</td>\n",
       "      <td>0.005615</td>\n",
       "      <td>0.030941</td>\n",
       "      <td>0.004726</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 100, 'gamma': 1.0}</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.011662</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.272896</td>\n",
       "      <td>0.014728</td>\n",
       "      <td>0.033383</td>\n",
       "      <td>0.004266</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1, 'gamma': 1.0}</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.012000</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "3       0.221965      0.006803         0.031804        0.005202      10   \n",
       "6       0.197406      0.005615         0.033033        0.010303     100   \n",
       "0       0.173476      0.008588         0.025023        0.007676       1   \n",
       "4       0.282868      0.008318         0.033063        0.007282      10   \n",
       "7       0.270238      0.004467         0.039076        0.007547     100   \n",
       "1       0.284600      0.013232         0.040817        0.003437       1   \n",
       "5       0.283568      0.018938         0.034709        0.004293      10   \n",
       "8       0.275807      0.005615         0.030941        0.004726     100   \n",
       "2       0.272896      0.014728         0.033383        0.004266       1   \n",
       "\n",
       "  param_gamma                     params  split0_test_score  \\\n",
       "3        0.01   {'C': 10, 'gamma': 0.01}               0.91   \n",
       "6        0.01  {'C': 100, 'gamma': 0.01}               0.91   \n",
       "0        0.01    {'C': 1, 'gamma': 0.01}               0.84   \n",
       "4         0.1    {'C': 10, 'gamma': 0.1}               0.42   \n",
       "7         0.1   {'C': 100, 'gamma': 0.1}               0.42   \n",
       "1         0.1     {'C': 1, 'gamma': 0.1}               0.39   \n",
       "5           1    {'C': 10, 'gamma': 1.0}               0.13   \n",
       "8           1   {'C': 100, 'gamma': 1.0}               0.13   \n",
       "2           1     {'C': 1, 'gamma': 1.0}               0.12   \n",
       "\n",
       "   split1_test_score  split2_test_score  split3_test_score  split4_test_score  \\\n",
       "3               0.92               0.93               0.87               0.86   \n",
       "6               0.92               0.93               0.87               0.86   \n",
       "0               0.90               0.89               0.85               0.84   \n",
       "4               0.52               0.45               0.49               0.38   \n",
       "7               0.52               0.45               0.49               0.38   \n",
       "1               0.42               0.40               0.41               0.36   \n",
       "5               0.12               0.12               0.12               0.15   \n",
       "8               0.12               0.12               0.12               0.15   \n",
       "2               0.12               0.12               0.12               0.15   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  \n",
       "3            0.898        0.027857                1  \n",
       "6            0.898        0.027857                1  \n",
       "0            0.864        0.025768                3  \n",
       "4            0.452        0.049558                4  \n",
       "7            0.452        0.049558                4  \n",
       "1            0.396        0.020591                6  \n",
       "5            0.128        0.011662                7  \n",
       "8            0.128        0.011662                7  \n",
       "2            0.126        0.012000                9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import SVC\n",
    "from sklearn.svm import SVC\n",
    "#import for Cross-Validation\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from IPython.display import display  #used to display\n",
    "import pandas as pd                  #used to display the grid search\n",
    "\n",
    "# parameters for linear SVM\n",
    "parameters = {'C': [1, 10, 100]}\n",
    "\n",
    "#run linear SVM\n",
    "linear_SVM = SVC(kernel='linear')\n",
    "\n",
    "#find best model uusing 5-fold CV \n",
    "#and train it using all the training data\n",
    "linear_grid_search=GridSearchCV(linear_SVM,parameters,cv=5)\n",
    "linear_grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('RESULTS FOR LINEAR KERNEL\\n')\n",
    "\n",
    "print(\"Best parameters set found:\")\n",
    "print(linear_grid_search.best_params_)\n",
    "\n",
    "print(\"Score with best parameters:\")\n",
    "print(linear_grid_search.best_score_)\n",
    "\n",
    "print(\"\\nAll scores on the grid:\")\n",
    "display(pd.DataFrame(linear_grid_search.cv_results_).sort_values(by='rank_test_score'))\n",
    "\n",
    "# parameters for poly with degree 2 kernel\n",
    "parameters = {'C': [1, 10, 100],'gamma':[0.01,0.1,1.]}\n",
    "\n",
    "#run SVM with poly of degree 2 kernel\n",
    "poly2_SVM = SVC(kernel='poly',degree=2)\n",
    "\n",
    "poly2_grid_search=GridSearchCV(poly2_SVM,parameters,cv=5)\n",
    "poly2_grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('\\nRESULTS FOR POLY DEGREE=2 KERNEL\\n')\n",
    "\n",
    "print(\"Best parameters set found:\")\n",
    "print(poly2_grid_search.best_params_)\n",
    "\n",
    "print(\"Score with best parameters:\")\n",
    "print(poly2_grid_search.best_score_)\n",
    "\n",
    "print(\"\\nAll scores on the grid:\")\n",
    "display(pd.DataFrame(poly2_grid_search.cv_results_).sort_values(by='rank_test_score'))\n",
    "\n",
    "# parameters for rbf SVM\n",
    "parameters = {'C': [1, 10, 100],'gamma':[0.01,0.1,1.]}\n",
    "\n",
    "#run SVM with rbf kernel\n",
    "rbf_SVM = SVC(kernel='rbf')\n",
    "\n",
    "rbf_grid_search=GridSearchCV(rbf_SVM,parameters,cv=5)\n",
    "rbf_grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('\\nRESULTS FOR rbf KERNEL\\n')\n",
    "\n",
    "print(\"Best parameters set found:\")\n",
    "print(rbf_grid_search.best_params_)\n",
    "\n",
    "print(\"Score with best parameters:\")\n",
    "print(rbf_grid_search.best_score_)\n",
    "\n",
    "print(\"\\nAll scores on the grid:\")\n",
    "display(pd.DataFrame(rbf_grid_search.cv_results_).sort_values(by='rank_test_score'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TO DO 2\n",
    "For the \"best\" SVM kernel and choice of parameters from above, train the model on the entire training set and measure the training error. Also make predictions on the test set and measure the test error. Print the training and the test error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best SVM training error: 0.000000\n",
      "Best SVM test error: 0.103353\n"
     ]
    }
   ],
   "source": [
    "#get training and test error for the best SVM model from CV\n",
    "best_SVM = rbf_grid_search.best_estimator_\n",
    "\n",
    "# fit the model on the entire training set\n",
    "# ADD CODE\n",
    "best_SVM.fit(X_train, y_train)\n",
    "\n",
    "#get the training and test error\n",
    "training_error = 1. - best_SVM.score(X_train,y_train)\n",
    "test_error = 1. - best_SVM.score(X_test,y_test)\n",
    "\n",
    "print (\"Best SVM training error: %f\" % training_error)\n",
    "print (\"Best SVM test error: %f\" % test_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use logistic regression for comparison\n",
    "\n",
    "## TO DO 3\n",
    "\n",
    "Just for comparison let's also use logistic regression, first with the default values of the parameter for regularization and then with cross-validation to fix the value of the parameter. For cross validation, use 5-fold cross validation and the default values of the regularization parameters for the function linear_model.LogisticRegressionCV(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best logistic regression training error: 0.000000\n",
      "Best logistic regression test error: 0.155928\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best logistic regression training error: 0.048000\n",
      "Best logistic regression test error: 0.152676\n"
     ]
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "lr = linear_model.LogisticRegression(solver='liblinear', random_state=ID, max_iter=10000)\n",
    "# fit the model on the training data\n",
    "lr.fit(X_train, y_train)\n",
    "\n",
    "#compute training and test error for model above\n",
    "training_error = 1. - lr.score(X_train,y_train)\n",
    "test_error = 1. - lr.score(X_test,y_test)\n",
    "\n",
    "print (\"Best logistic regression training error: %f\" % training_error)\n",
    "print (\"Best logistic regression test error: %f\" % test_error)\n",
    "\n",
    "#logistic regression with 5-fold CV: you can use use linear_model.LogisticRegressionCV\n",
    "# use 5-fold CV to find the best choice of the parameter, than train\n",
    "# the model on the entire training set\n",
    "lr_cv = linear_model.LogisticRegressionCV(cv=5)\n",
    "lr_cv.fit(X_train, y_train)\n",
    "training_error_cv = 1. - lr_cv.score(X_train,y_train)\n",
    "test_error_cv = 1. - lr_cv.score(X_test,y_test)\n",
    "\n",
    "print (\"Best logistic regression training error: %f\" % training_error_cv)\n",
    "print (\"Best logistic regression test error: %f\" % test_error_cv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TO DO 4 \n",
    "Compare and comment the results from SVM and logistic regression.\n",
    "\n",
    "\n",
    "**ANSWER:** SVM has 0 traingn error while Logistic Regression has 4.8%, that means they both fit very well the training set, probably there is some overfit. And we have 10.3% test error on SVM and 15.2% on Logistic, this means that SMV has generalized better the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TO DO 5\n",
    "Write the code that finds and plots a digit that is missclassified by logistic regression (optimized for the regularization parameter) and correctly classified by the \"best\" SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANS0lEQVR4nO3db6xU9Z3H8c9HKU8oJigXF+11QcMDzSZL8QY3wRDXukV9IPaBWkwKS4jwQBKaNHHVTayJT8zGtvaBIaErlmKXphEMorhbQ5qQPqlckRWQ+GfJ3ZZKYIgmgD5A9LsP7nFzi3fOXOacmTP6fb+Sm5k53znnfDO5n3vmnt+Z+TkiBODr75KmGwDQH4QdSIKwA0kQdiAJwg4kMa2fO5s9e3bMmzevn7sEUhkbG9OpU6c8Wa1S2G3fLunnki6V9O8R8WTZ8+fNm6fR0dEquwRQYmRkpG2t67fxti+V9IykOyTdIGmF7Ru63R6A3qryP/tiSe9HxNGIOCfpN5KW19MWgLpVCfvVkv484fGxYtlfsb3W9qjt0VarVWF3AKqoEvbJTgJ86drbiNgUESMRMTI0NFRhdwCqqBL2Y5KGJzz+lqQPqrUDoFeqhH2fpAW259ueLun7kl6qpy0Adet66C0iztteL+m/ND70tjkiDtfWGYBaVRpnj4jdknbX1AuAHuJyWSAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASlaZstj0m6YykzySdj4iROpoCUL9KYS/8Y0ScqmE7AHqIt/FAElXDHpJ+Z/sN22sne4LttbZHbY+2Wq2KuwPQraphXxIRiyTdIelB20svfEJEbIqIkYgYGRoaqrg7AN2qFPaI+KC4PSnpRUmL62gKQP26DrvtGbZnfnFf0nclHaqrMQD1qnI2/kpJL9r+Yjv/ERH/WUtXAGrXddgj4qikv6+xFwA9xNAbkARhB5Ig7EAShB1IgrADSdTxQRg07Pnnn29bW7lyZaVtR0RpvRh67crGjRtL6+vWret62/gyjuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7DU4evRoaX337t2l9bvvvru0PmPGjNL6M88807ZWZRx8Kqpsf8OGDaX1Tz/9tLS+fv36rvedEUd2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfYa3H///aX1ffv2ldZ37dpVWn/66adL66+//nrb2vTp00vXXbFiRWn9tttuK613Gmcv+0z6xx9/XLruQw89VFpfuHBhaf3mm28urWfDkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcvQbDw8Ol9U7j7KdPny6tdxpnL3PjjTeW1jdv3tz1tqdi2rT2v2IPPPBA6bpnzpwprZ87d66rnrLqeGS3vdn2SduHJiy73PZrtt8rbmf1tk0AVU3lbfwvJd1+wbKHJe2JiAWS9hSPAQywjmGPiL2SPrxg8XJJW4r7WySVf68SgMZ1e4Luyog4LknF7Zx2T7S91vao7dFWq9Xl7gBU1fOz8RGxKSJGImJkaGio17sD0Ea3YT9he64kFbcn62sJQC90G/aXJK0q7q+StLOedgD0SsdxdtvbJN0iabbtY5J+LOlJSb+1vUbSnyTd08smB91NN91UWt+xY0dp/e233y6tv/vuu6X1OXPanjLpuO9eu+ee9r8aW7duLV33lVdeKa13mt/91ltvLa1n0zHsEdHu2w2+U3MvAHqIy2WBJAg7kARhB5Ig7EAShB1Igo+41qDqx0TPnj1baf3HHnusba1sWK5pq1evLq13Gnp75513SusfffRR29qsWfk+qMmRHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJz9a+Daa69tuoVGHD58uLS+d+/etrXly5fX3c7A48gOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzl6De++9t7T+xBNPVNr+9ddfX1pfsmRJpe1/VV133XWl9UWLFvWpk68GjuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7DVYuXJlaX379u2lddul9Z07d5bW58+fX1ofVLt27aq0/hVXXFFaHx4errT9r5uOR3bbm22ftH1owrLHbf/F9oHi587etgmgqqm8jf+lpNsnWf6ziFhY/Oyuty0AdesY9ojYK+nDPvQCoIeqnKBbb/ut4m1+24mzbK+1PWp7tNVqVdgdgCq6DftGSddJWijpuKSftHtiRGyKiJGIGBkaGupydwCq6irsEXEiIj6LiM8l/ULS4nrbAlC3rsJue+6Eh9+TdKjdcwEMho7j7La3SbpF0mzbxyT9WNItthdKCkljktb1sMeB1+l72w8ePNinTgbP/v3729ZefvnlSttetmxZpfWz6Rj2iFgxyeJne9ALgB7iclkgCcIOJEHYgSQIO5AEYQeS4COu6Km77rqrbe3UqVOVtr106dJK62fDkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHT11ySXtjyedvkL7kUceKa0zzn5xOLIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs6OS5557rrR+4sSJtrVrrrmmdN01a9aU1qdN49f3YnBkB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGKhEJfv27Sutnz9/vm1t0aJFpevOnz+/q54wuY5HdtvDtn9v+4jtw7Y3FMsvt/2a7feK21m9bxdAt6byNv68pB9FxPWS/kHSg7ZvkPSwpD0RsUDSnuIxgAHVMewRcTwi9hf3z0g6IulqScslbSmetkXS3b1qEkB1F3WCzvY8Sd+W9EdJV0bEcWn8D4KkOW3WWWt71PZoq9Wq1i2Ark057La/KWm7pB9GxOmprhcRmyJiJCJGhoaGuukRQA2mFHbb39B40H8dETuKxSdszy3qcyWd7E2LAOrQcejN49/3+6ykIxHx0wmllyStkvRkcbuzJx2iUWNjY6X1rVu3dr3tBQsWdL0uLt5UxtmXSPqBpIO2DxTLHtV4yH9re42kP0m6pzctAqhDx7BHxB8ktfs2/+/U2w6AXuFyWSAJwg4kQdiBJAg7kARhB5LgI64otW3bttL6J5980vW2V69e3fW6uHgc2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZ0VP33Xdf29pVV13Vx07AkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHaVeeOGFSus/9dRTbWszZ86stG1cHI7sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5DEVOZnH5b0K0l/I+lzSZsi4ue2H5f0gKRW8dRHI2J3rxpFM958883Sut1ugl8MmqlcVHNe0o8iYr/tmZLesP1aUftZRLS/agLAwJjK/OzHJR0v7p+xfUTS1b1uDEC9Lup/dtvzJH1b0h+LRettv2V7s+1ZbdZZa3vU9mir1ZrsKQD6YMpht/1NSdsl/TAiTkvaKOk6SQs1fuT/yWTrRcSmiBiJiJGhoaEaWgbQjSmF3fY3NB70X0fEDkmKiBMR8VlEfC7pF5IW965NAFV1DLvHT7c+K+lIRPx0wvK5E572PUmH6m8PQF2mcjZ+iaQfSDpo+0Cx7FFJK2wvlBSSxiSt60mHaNSyZctK62fPni2tX3bZZXW2gwqmcjb+D5ImG0xlTB34CuEKOiAJwg4kQdiBJAg7kARhB5Ig7EASfJU0Sr366qtNt4CacGQHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQcEf3bmd2S9L8TFs2WdKpvDVycQe1tUPuS6K1bdfb2txEx6fe/9TXsX9q5PRoRI401UGJQexvUviR661a/euNtPJAEYQeSaDrsmxref5lB7W1Q+5LorVt96a3R/9kB9E/TR3YAfULYgSQaCbvt222/Y/t92w830UM7tsdsH7R9wPZow71stn3S9qEJyy63/Zrt94rbSefYa6i3x23/pXjtDti+s6Hehm3/3vYR24dtbyiWN/ralfTVl9et7/+z275U0ruS/knSMUn7JK2IiLf72kgbtsckjURE4xdg2F4q6aykX0XE3xXL/k3ShxHxZPGHclZE/MuA9Pa4pLNNT+NdzFY0d+I045LulvTPavC1K+nrXvXhdWviyL5Y0vsRcTQizkn6jaTlDfQx8CJir6QPL1i8XNKW4v4Wjf+y9F2b3gZCRByPiP3F/TOSvphmvNHXrqSvvmgi7FdL+vOEx8c0WPO9h6Tf2X7D9tqmm5nElRFxXBr/5ZE0p+F+LtRxGu9+umCa8YF57bqZ/ryqJsI+2VRSgzT+tyQiFkm6Q9KDxdtVTM2UpvHul0mmGR8I3U5/XlUTYT8maXjC429J+qCBPiYVER8UtyclvajBm4r6xBcz6Ba3Jxvu5/8N0jTek00zrgF47Zqc/ryJsO+TtMD2fNvTJX1f0ksN9PEltmcUJ05ke4ak72rwpqJ+SdKq4v4qSTsb7OWvDMo03u2mGVfDr13j059HRN9/JN2p8TPy/yPpX5vooU1f10r67+LncNO9Sdqm8bd1n2r8HdEaSVdI2iPpveL28gHqbaukg5Le0niw5jbU280a/9fwLUkHip87m37tSvrqy+vG5bJAElxBByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ/B+CteoPEk5iXwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LABEL: 7\n",
      "SVM prediction: 7\n",
      "LOG prediction: 9\n"
     ]
    }
   ],
   "source": [
    "# ADD CODE\n",
    "svm_predictions=best_SVM.predict(X_test)\n",
    "log_predictions=lr_cv.predict(X_test)\n",
    "\n",
    "index=0\n",
    "for i in range(len(y_test)):\n",
    "    if log_predictions[i]!=y_test[i] and svm_predictions[i]==y_test[i]:\n",
    "        index=i\n",
    "        break\n",
    "plot_digit(X_test,y_test,index)\n",
    "print(\"SVM prediction:\",svm_predictions[index])\n",
    "print(\"LOG prediction:\",log_predictions[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More data\n",
    "Now let's do the same but using 1000 data points for training. \n",
    "\n",
    "## TO DO 6\n",
    "Repeat the entire analysis above using 1000 samples. Of course you can copy the code from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels and frequencies in training dataset: \n",
      "RESULTS FOR LINEAR KERNEL\n",
      "\n",
      "Best parameters set found:\n",
      "{'C': 1}\n",
      "Score with best parameters:\n",
      "0.8699999999999999\n",
      "\n",
      "All scores on the grid:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.279468</td>\n",
       "      <td>0.011858</td>\n",
       "      <td>0.075992</td>\n",
       "      <td>0.004226</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1}</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.855</td>\n",
       "      <td>0.865</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.011402</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.269016</td>\n",
       "      <td>0.007686</td>\n",
       "      <td>0.087147</td>\n",
       "      <td>0.008079</td>\n",
       "      <td>10</td>\n",
       "      <td>{'C': 10}</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.855</td>\n",
       "      <td>0.865</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.011402</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.272143</td>\n",
       "      <td>0.006260</td>\n",
       "      <td>0.081322</td>\n",
       "      <td>0.006206</td>\n",
       "      <td>100</td>\n",
       "      <td>{'C': 100}</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.855</td>\n",
       "      <td>0.865</td>\n",
       "      <td>0.87</td>\n",
       "      <td>0.011402</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "0       0.279468      0.011858         0.075992        0.004226       1   \n",
       "1       0.269016      0.007686         0.087147        0.008079      10   \n",
       "2       0.272143      0.006260         0.081322        0.006206     100   \n",
       "\n",
       "       params  split0_test_score  split1_test_score  split2_test_score  \\\n",
       "0    {'C': 1}               0.87               0.87               0.89   \n",
       "1   {'C': 10}               0.87               0.87               0.89   \n",
       "2  {'C': 100}               0.87               0.87               0.89   \n",
       "\n",
       "   split3_test_score  split4_test_score  mean_test_score  std_test_score  \\\n",
       "0              0.855              0.865             0.87        0.011402   \n",
       "1              0.855              0.865             0.87        0.011402   \n",
       "2              0.855              0.865             0.87        0.011402   \n",
       "\n",
       "   rank_test_score  \n",
       "0                1  \n",
       "1                1  \n",
       "2                1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "RESULTS FOR POLY DEGREE=2 KERNEL\n",
      "\n",
      "Best parameters set found:\n",
      "{'C': 1, 'gamma': 0.1}\n",
      "Score with best parameters:\n",
      "0.9\n",
      "\n",
      "All scores on the grid:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>param_gamma</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.351280</td>\n",
       "      <td>0.007116</td>\n",
       "      <td>0.081332</td>\n",
       "      <td>0.006450</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 1, 'gamma': 0.1}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.351862</td>\n",
       "      <td>0.005876</td>\n",
       "      <td>0.079881</td>\n",
       "      <td>0.003547</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1, 'gamma': 1.0}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.363039</td>\n",
       "      <td>0.020577</td>\n",
       "      <td>0.082872</td>\n",
       "      <td>0.006291</td>\n",
       "      <td>10</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 10, 'gamma': 0.1}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.350465</td>\n",
       "      <td>0.005673</td>\n",
       "      <td>0.086052</td>\n",
       "      <td>0.006742</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 10, 'gamma': 1.0}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.351528</td>\n",
       "      <td>0.006590</td>\n",
       "      <td>0.081409</td>\n",
       "      <td>0.005647</td>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 100, 'gamma': 0.01}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.348080</td>\n",
       "      <td>0.009011</td>\n",
       "      <td>0.087757</td>\n",
       "      <td>0.007890</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 100, 'gamma': 0.1}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.388469</td>\n",
       "      <td>0.022411</td>\n",
       "      <td>0.093547</td>\n",
       "      <td>0.009154</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 100, 'gamma': 1.0}</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.023238</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.346393</td>\n",
       "      <td>0.006501</td>\n",
       "      <td>0.085826</td>\n",
       "      <td>0.006988</td>\n",
       "      <td>10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 10, 'gamma': 0.01}</td>\n",
       "      <td>0.880</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.899</td>\n",
       "      <td>0.023958</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.435929</td>\n",
       "      <td>0.011269</td>\n",
       "      <td>0.096975</td>\n",
       "      <td>0.006192</td>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 1, 'gamma': 0.01}</td>\n",
       "      <td>0.850</td>\n",
       "      <td>0.885</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.85</td>\n",
       "      <td>0.880</td>\n",
       "      <td>0.876</td>\n",
       "      <td>0.024372</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "1       0.351280      0.007116         0.081332        0.006450       1   \n",
       "2       0.351862      0.005876         0.079881        0.003547       1   \n",
       "4       0.363039      0.020577         0.082872        0.006291      10   \n",
       "5       0.350465      0.005673         0.086052        0.006742      10   \n",
       "6       0.351528      0.006590         0.081409        0.005647     100   \n",
       "7       0.348080      0.009011         0.087757        0.007890     100   \n",
       "8       0.388469      0.022411         0.093547        0.009154     100   \n",
       "3       0.346393      0.006501         0.085826        0.006988      10   \n",
       "0       0.435929      0.011269         0.096975        0.006192       1   \n",
       "\n",
       "  param_gamma                     params  split0_test_score  \\\n",
       "1         0.1     {'C': 1, 'gamma': 0.1}              0.885   \n",
       "2           1     {'C': 1, 'gamma': 1.0}              0.885   \n",
       "4         0.1    {'C': 10, 'gamma': 0.1}              0.885   \n",
       "5           1    {'C': 10, 'gamma': 1.0}              0.885   \n",
       "6        0.01  {'C': 100, 'gamma': 0.01}              0.885   \n",
       "7         0.1   {'C': 100, 'gamma': 0.1}              0.885   \n",
       "8           1   {'C': 100, 'gamma': 1.0}              0.885   \n",
       "3        0.01   {'C': 10, 'gamma': 0.01}              0.880   \n",
       "0        0.01    {'C': 1, 'gamma': 0.01}              0.850   \n",
       "\n",
       "   split1_test_score  split2_test_score  split3_test_score  split4_test_score  \\\n",
       "1              0.895              0.945               0.88              0.895   \n",
       "2              0.895              0.945               0.88              0.895   \n",
       "4              0.895              0.945               0.88              0.895   \n",
       "5              0.895              0.945               0.88              0.895   \n",
       "6              0.895              0.945               0.88              0.895   \n",
       "7              0.895              0.945               0.88              0.895   \n",
       "8              0.895              0.945               0.88              0.895   \n",
       "3              0.895              0.945               0.88              0.895   \n",
       "0              0.885              0.915               0.85              0.880   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  \n",
       "1            0.900        0.023238                1  \n",
       "2            0.900        0.023238                1  \n",
       "4            0.900        0.023238                1  \n",
       "5            0.900        0.023238                1  \n",
       "6            0.900        0.023238                1  \n",
       "7            0.900        0.023238                1  \n",
       "8            0.900        0.023238                1  \n",
       "3            0.899        0.023958                8  \n",
       "0            0.876        0.024372                9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "RESULTS FOR rbf KERNEL\n",
      "\n",
      "Best parameters set found:\n",
      "{'C': 10, 'gamma': 0.01}\n",
      "Score with best parameters:\n",
      "0.914\n",
      "\n",
      "All scores on the grid:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>param_gamma</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.458981</td>\n",
       "      <td>0.008838</td>\n",
       "      <td>0.103134</td>\n",
       "      <td>0.007612</td>\n",
       "      <td>10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 10, 'gamma': 0.01}</td>\n",
       "      <td>0.905</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.935</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.012000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.466453</td>\n",
       "      <td>0.011679</td>\n",
       "      <td>0.096861</td>\n",
       "      <td>0.006242</td>\n",
       "      <td>100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 100, 'gamma': 0.01}</td>\n",
       "      <td>0.905</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.935</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.914</td>\n",
       "      <td>0.012000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.491770</td>\n",
       "      <td>0.016646</td>\n",
       "      <td>0.120212</td>\n",
       "      <td>0.008132</td>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 1, 'gamma': 0.01}</td>\n",
       "      <td>0.880</td>\n",
       "      <td>0.890</td>\n",
       "      <td>0.915</td>\n",
       "      <td>0.880</td>\n",
       "      <td>0.895</td>\n",
       "      <td>0.892</td>\n",
       "      <td>0.012884</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.018707</td>\n",
       "      <td>0.006737</td>\n",
       "      <td>0.135694</td>\n",
       "      <td>0.005828</td>\n",
       "      <td>10</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 10, 'gamma': 0.1}</td>\n",
       "      <td>0.690</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.670</td>\n",
       "      <td>0.670</td>\n",
       "      <td>0.620</td>\n",
       "      <td>0.657</td>\n",
       "      <td>0.025612</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.021313</td>\n",
       "      <td>0.013507</td>\n",
       "      <td>0.138399</td>\n",
       "      <td>0.013193</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 100, 'gamma': 0.1}</td>\n",
       "      <td>0.690</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.670</td>\n",
       "      <td>0.670</td>\n",
       "      <td>0.620</td>\n",
       "      <td>0.657</td>\n",
       "      <td>0.025612</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.009969</td>\n",
       "      <td>0.003731</td>\n",
       "      <td>0.134340</td>\n",
       "      <td>0.007659</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 1, 'gamma': 0.1}</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.595</td>\n",
       "      <td>0.655</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.600</td>\n",
       "      <td>0.626</td>\n",
       "      <td>0.024166</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.029450</td>\n",
       "      <td>0.009645</td>\n",
       "      <td>0.135890</td>\n",
       "      <td>0.006388</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 10, 'gamma': 1.0}</td>\n",
       "      <td>0.145</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.145</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.008944</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.055874</td>\n",
       "      <td>0.018263</td>\n",
       "      <td>0.142716</td>\n",
       "      <td>0.011738</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 100, 'gamma': 1.0}</td>\n",
       "      <td>0.145</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.145</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.008944</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.017686</td>\n",
       "      <td>0.008318</td>\n",
       "      <td>0.134136</td>\n",
       "      <td>0.007490</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1, 'gamma': 1.0}</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.145</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.130</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.008602</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "3       0.458981      0.008838         0.103134        0.007612      10   \n",
       "6       0.466453      0.011679         0.096861        0.006242     100   \n",
       "0       0.491770      0.016646         0.120212        0.008132       1   \n",
       "4       1.018707      0.006737         0.135694        0.005828      10   \n",
       "7       1.021313      0.013507         0.138399        0.013193     100   \n",
       "1       1.009969      0.003731         0.134340        0.007659       1   \n",
       "5       1.029450      0.009645         0.135890        0.006388      10   \n",
       "8       1.055874      0.018263         0.142716        0.011738     100   \n",
       "2       1.017686      0.008318         0.134136        0.007490       1   \n",
       "\n",
       "  param_gamma                     params  split0_test_score  \\\n",
       "3        0.01   {'C': 10, 'gamma': 0.01}              0.905   \n",
       "6        0.01  {'C': 100, 'gamma': 0.01}              0.905   \n",
       "0        0.01    {'C': 1, 'gamma': 0.01}              0.880   \n",
       "4         0.1    {'C': 10, 'gamma': 0.1}              0.690   \n",
       "7         0.1   {'C': 100, 'gamma': 0.1}              0.690   \n",
       "1         0.1     {'C': 1, 'gamma': 0.1}              0.645   \n",
       "5           1    {'C': 10, 'gamma': 1.0}              0.145   \n",
       "8           1   {'C': 100, 'gamma': 1.0}              0.145   \n",
       "2           1     {'C': 1, 'gamma': 1.0}              0.140   \n",
       "\n",
       "   split1_test_score  split2_test_score  split3_test_score  split4_test_score  \\\n",
       "3              0.915              0.935              0.900              0.915   \n",
       "6              0.915              0.935              0.900              0.915   \n",
       "0              0.890              0.915              0.880              0.895   \n",
       "4              0.635              0.670              0.670              0.620   \n",
       "7              0.635              0.670              0.670              0.620   \n",
       "1              0.595              0.655              0.635              0.600   \n",
       "5              0.150              0.145              0.125              0.135   \n",
       "8              0.150              0.145              0.125              0.135   \n",
       "2              0.145              0.135              0.120              0.130   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  \n",
       "3            0.914        0.012000                1  \n",
       "6            0.914        0.012000                1  \n",
       "0            0.892        0.012884                3  \n",
       "4            0.657        0.025612                4  \n",
       "7            0.657        0.025612                4  \n",
       "1            0.626        0.024166                6  \n",
       "5            0.140        0.008944                7  \n",
       "8            0.140        0.008944                7  \n",
       "2            0.134        0.008602                9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "permutation = np.random.permutation(X.shape[0])\n",
    "\n",
    "X = X[permutation]\n",
    "y = y[permutation]\n",
    "\n",
    "m_training = 1000\n",
    "\n",
    "X_train, X_test = X[:m_training], X[m_training:]\n",
    "y_train, y_test = y[:m_training], y[m_training:]\n",
    "\n",
    "print(\"Labels and frequencies in training dataset: \")\n",
    "np.unique(y_train, return_counts = True)\n",
    "\n",
    "\n",
    "# parameters for linear SVM\n",
    "parameters = {'C': [1, 10, 100]}\n",
    "\n",
    "#run linear SVM\n",
    "linear_SVM = SVC(kernel='linear')\n",
    "\n",
    "#find best model uusing 5-fold CV \n",
    "#and train it using all the training data\n",
    "linear_grid_search=GridSearchCV(linear_SVM,parameters,cv=5)\n",
    "linear_grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('RESULTS FOR LINEAR KERNEL\\n')\n",
    "\n",
    "print(\"Best parameters set found:\")\n",
    "print(linear_grid_search.best_params_)\n",
    "\n",
    "print(\"Score with best parameters:\")\n",
    "print(linear_grid_search.best_score_)\n",
    "\n",
    "print(\"\\nAll scores on the grid:\")\n",
    "display(pd.DataFrame(linear_grid_search.cv_results_).sort_values(by='rank_test_score'))\n",
    "\n",
    "# parameters for poly with degree 2 kernel\n",
    "parameters = {'C': [1, 10, 100],'gamma':[0.01,0.1,1.]}\n",
    "\n",
    "#run SVM with poly of degree 2 kernel\n",
    "poly2_SVM = SVC(kernel='poly',degree=2)\n",
    "\n",
    "poly2_grid_search=GridSearchCV(poly2_SVM,parameters,cv=5)\n",
    "poly2_grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('\\nRESULTS FOR POLY DEGREE=2 KERNEL\\n')\n",
    "\n",
    "print(\"Best parameters set found:\")\n",
    "print(poly2_grid_search.best_params_)\n",
    "\n",
    "print(\"Score with best parameters:\")\n",
    "print(poly2_grid_search.best_score_)\n",
    "\n",
    "print(\"\\nAll scores on the grid:\")\n",
    "display(pd.DataFrame(poly2_grid_search.cv_results_).sort_values(by='rank_test_score'))\n",
    "\n",
    "# parameters for rbf SVM\n",
    "parameters = {'C': [1, 10, 100],'gamma':[0.01,0.1,1.]}\n",
    "\n",
    "#run SVM with rbf kernel\n",
    "rbf_SVM = SVC(kernel='rbf')\n",
    "\n",
    "rbf_grid_search=GridSearchCV(rbf_SVM,parameters,cv=5)\n",
    "rbf_grid_search.fit(X_train, y_train)\n",
    "\n",
    "print ('\\nRESULTS FOR rbf KERNEL\\n')\n",
    "\n",
    "print(\"Best parameters set found:\")\n",
    "print(rbf_grid_search.best_params_)\n",
    "\n",
    "print(\"Score with best parameters:\")\n",
    "print(rbf_grid_search.best_score_)\n",
    "\n",
    "print(\"\\nAll scores on the grid:\")\n",
    "display(pd.DataFrame(rbf_grid_search.cv_results_).sort_values(by='rank_test_score'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best SVM training error: 0.000000\n",
      "Best SVM test error: 0.080304\n",
      "Best logistic regression training error: 0.004000\n",
      "Best logistic regression test error: 0.137884\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "C:\\Users\\stefano\\.conda\\envs\\gputest\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best logistic regression training error: 0.000000\n",
      "Best logistic regression test error: 0.134725\n"
     ]
    }
   ],
   "source": [
    "#get training and test error for the best SVM model from CV\n",
    "best_SVM = rbf_grid_search.best_estimator_\n",
    "\n",
    "# fit the model on the entire training set\n",
    "# ADD CODE\n",
    "best_SVM.fit(X_train, y_train)\n",
    "\n",
    "#get the training and test error\n",
    "training_error = 1. - best_SVM.score(X_train,y_train)\n",
    "test_error = 1. - best_SVM.score(X_test,y_test)\n",
    "\n",
    "print (\"Best SVM training error: %f\" % training_error)\n",
    "print (\"Best SVM test error: %f\" % test_error)\n",
    "\n",
    "lr = linear_model.LogisticRegression(solver='liblinear', random_state=ID, max_iter=10000)\n",
    "# fit the model on the training data\n",
    "lr.fit(X_train, y_train)\n",
    "\n",
    "#compute training and test error for model above\n",
    "training_error = 1. - lr.score(X_train,y_train)\n",
    "test_error = 1. - lr.score(X_test,y_test)\n",
    "\n",
    "print (\"Best logistic regression training error: %f\" % training_error)\n",
    "print (\"Best logistic regression test error: %f\" % test_error)\n",
    "\n",
    "#logistic regression with 5-fold CV: you can use use linear_model.LogisticRegressionCV\n",
    "# use 5-fold CV to find the best choice of the parameter, than train\n",
    "# the model on the entire training set\n",
    "lr_cv = linear_model.LogisticRegressionCV(cv=5)\n",
    "lr_cv.fit(X_train, y_train)\n",
    "training_error_cv = 1. - lr_cv.score(X_train,y_train)\n",
    "test_error_cv = 1. - lr_cv.score(X_test,y_test)\n",
    "\n",
    "print (\"Best logistic regression training error: %f\" % training_error_cv)\n",
    "print (\"Best logistic regression test error: %f\" % test_error_cv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TO DO 7\n",
    "Compare and comment on the differences with the results above.\n",
    "\n",
    "\n",
    "**ANSWER**: As we can see both SVM and Logistic improved their test error trained with more data. But now we notice that for both the training error is almost 0. This means that the models are \"overfitting\" less. And SVM with just 8.03% Test error is still superior to the logistic that has 13.47% test error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gputest",
   "language": "python",
   "name": "gputest"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}