006-support-vector-machine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines\n",
    "Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.\n",
    "\n",
    "**The advantages of support vector machines are:**\n",
    "\n",
    "- Effective in high dimensional spaces.  \n",
    "- Still effective in cases where number of dimensions is greater than the number of samples.  \n",
    "- Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.  \n",
    "\n",
    "*Versatile*: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.\n",
    "\n",
    "**The disadvantages of support vector machines include:**\n",
    "\n",
    "- If the number of features is much greater than the number of samples, avoid over-fitting in choosing Kernel functions and regularization term is crucial.\n",
    "- SVMs do not directly provide probability estimates, these are calculated using an expensive five-fold cross-validation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Size of the feature vector', (978, 785))\n",
      "[[ 6.  1.  1. ...,  1.  1.  1.]\n",
      " [ 6.  1.  1. ...,  1.  1.  1.]\n",
      " [ 6.  1.  1. ...,  1.  1.  1.]\n",
      " ..., \n",
      " [ 6.  1.  1. ...,  1.  1.  1.]\n",
      " [ 6.  1.  1. ...,  1.  1.  1.]\n",
      " [ 6.  1.  1. ...,  1.  1.  1.]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# global variable\n",
    "global img_number\n",
    "img_number = 550\n",
    "\n",
    "output_filename = 'digit_fv.train'\n",
    "digit_data = np.loadtxt(output_filename, dtype='float64')\n",
    "\n",
    "print(\"Size of the feature vector\", digit_data.shape)\n",
    "print digit_data[0:10,0:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "from skimage import io\n",
    "\n",
    "# show image\n",
    "def vec2img(vec):\n",
    "    \n",
    "    img_row = int(math.sqrt(vec.shape[0]))\n",
    "    img_col = img_row\n",
    "    img = vec.reshape((img_row, img_col))\n",
    "    \n",
    "    io.imshow(img)\n",
    "    io.show()\n",
    "    \n",
    "# check predict\n",
    "def chkpredict(actual_class, predict_class):\n",
    "    if(actual_class==predict_class):\n",
    "        print(\"(Correct prediction)\")\n",
    "    else:\n",
    "        print(\"(Incorrect prediction)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARoAAAEYCAYAAACDezmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAEf5JREFUeJzt3X2MVfWdx/HPlwdFJGkpLKCg+IBibGKIOiQbG3u1tQ/r1gdqWR/SSFdoA+yuSZt2rTFhMNrUbWpXd4OJlBqKrY9JF0iqIJGb0W59SFcWERyNAmp5mhC0ixqgzHf/mCsdp3fme5w73zv3zrxfyYR7z/3ce34cZj6ce+5vzjF3FwBkGjHYAwAw9FE0ANJRNADSUTQA0lE0ANJRNADSjarlyWb2FUn/rq7CWuHud1XJ8Pk5MEy4u1Vb3u89GjMbIek/JX1Z0mclXWdm5/Sy8mNfS5Ys+dj9ZvhizIyZMcdffanlrdNsSa+7+053PyLpYUlX1vB6AIaoWopmqqS3u91/p7IMAD6m7geDS6VSvVdZM8ZcH4y5PgZjzLUcDP6jpFO73Z9WWfZXWltbj93mH6Y+GHN9DOcxl8tllcvlQlmLDuL0+kSzkZLaJX1B0m5JL0i6zt239ch5f9cBoHmYmbyXT536vUfj7kfN7J8krddfPt7eFjwNwDDU7z2awitgj6ZPR44cCTM7duwIM+vWrQszGzZsCDMvvfRSmHnnnXfCTGdnZ5g58cQTw8wpp5wSZlpaWsLM1VdfPSCvM23atDAzXPW1R8PMYADpKBoA6SgaAOkoGgDpKBoA6SgaAOkoGgDpKBoA6Ziw10+HDx8OM2+++WaYue+++8LMY489FmZ2794dZsaNGxdmpk+fHmbOOuusMHPCCSeEmQMHDoSZIpMV29vbw0wR559/fpiZO3dumPnmN78ZZiZPnhxmRoxorv0AJuwBGFQUDYB0FA2AdBQNgHQUDYB0FA2AdBQNgHQUDYB0TNirYv/+/WHmhRdeCDM/+MEPwsz27dvDTJEJcosXLw4zl112WZgpcka7ek4kO3ToUJjp6OgIM8uWLQsza9euDTNvvfVWmPnqV78aZubPnx9mLrnkkjAzcuTIMFMvTNgDMKgoGgDpKBoA6SgaAOkoGgDpKBoA6SgaAOkoGgDp+n3t7Wb14YcfhpkiE7d+8pOfhJldu3aFmUWLFoWZefPmhZlzzz03zDSj448/PswUuUzt7bffHmaKTJBbsWJFmHnyySfDzJgxY8LMqFHxj2epVAozjYA9GgDpKBoA6SgaAOkoGgDpKBoA6SgaAOkoGgDpKBoA6YbdhL1nn302zGzYsCHMFDkL3/e///0wc8UVV4SZoToZr56KTH77/Oc/H2amTp0aZt59990w89xzz4WZIopcxnfs2LFhpsj2qUVNr25mOyS9J6lT0hF3nz0QgwIwtNRaY52SSu4eX60dwLBV6zEaG4DXADDE1VoSLmmdmb1oZgsGYkAAhp5a3zpd5O67zexvJD1lZtvcPT7aCmBYqalo3H135c8OM/uNpNmS/qpoWltbj90ulUpN86vtAHpXLpdVLpcLZftdNGY2VtIIdz9oZidK+pKkpdWy3YsGwNDQc6dh6dKqP/6SatujmSzpN2bmldf5lbuvr+H1AAxR/S4ad98uadYAjiVaX5g5cuRImPntb38bZl555ZUwU2QSXZHJeDNnzgwzqI/jjjsuzBS5ZPDChQvDzH333Rdmtm3bFmaef/75MFNkUt+ECRPCTC34aBpAOooGQDqKBkA6igZAOooGQDqKBkA6igZAOooGQLqmOcNekcl4Bw7Ep8V5/fXXw8xAnY1t0qRJYWb06NFhBo3jhBNOCDMXXHBBmDn11FPDzNtvvx1m2trawsz06dPDDBP2ADQ9igZAOooGQDqKBkA6igZAOooGQDqKBkA6igZAuqaZsHf48OEw09HREWY++OCDMPPpT386zJx99tlhZsyYMWEGzWXEiPj/5iLfP+PHjw8zZhZmikxAff/998NMkTNYFhlPb9ijAZCOogGQjqIBkI6iAZCOogGQjqIBkI6iAZCOogGQrmkm7I0dOzbMnHnmmWHm/vvvDzNFJmVNnDgxzIwbNy7MoLnUMmkt43WKTGTt7OwckHXVgj0aAOkoGgDpKBoA6SgaAOkoGgDpKBoA6SgaAOkoGgDpmmbCXpFJdEUuVzpjxoyBGA6GqaNHj4aZIpdm3r9//4Csa+bMmWGmyMTRgZpA2Jvwp9fMVpjZXjPb3G3ZeDNbb2btZrbOzD6VOkoATa3IW6cHJH25x7JbJG1w95mSnpb0w4EeGIChIywad39WUs99wSslrazcXinpqgEeF4AhpL8Hgye5+15Jcvc9kiYN3JAADDUD9alTfK0GAMNWfz912mtmk919r5lNkbSvr3Bra+ux26VSSaVSqZ+rBdAoyuWyyuVyoWzRorHK10fWSJon6S5JN0pa3deTuxcNgKGh507D0qVLe80W+Xj715L+W9LZZvaWmX1L0o8lXWZm7ZK+ULkPAFWFezTufn0vD31xgMcCNLw//elPYWbt2rVhZuvWrWGmyKV1L7/88jAzadLgf1bDryAASEfRAEhH0QBIR9EASEfRAEhH0QBIR9EASEfRAEjXNGfYA7J1dHSEmd///vdh5mc/+1mYGTNmTJj52te+FmZaWlrCzOjRo8NMNvZoAKSjaACko2gApKNoAKSjaACko2gApKNoAKSjaACkY8IehoWdO3eGmVWrVoWZZcuWhZnOzs4wM3/+/DBz4403hpnjjz8+zDQC9mgApKNoAKSjaACko2gApKNoAKSjaACko2gApKNoAKSjaACkY2YwGtquXbvCTFtbW5gpcnrN9vb2MHP66aeHmTvvvDPMzJ49O8xMnDgxzDQL9mgApKNoAKSjaACko2gApKNoAKSjaACko2gApKNoAKQzd+87YLZC0t9L2uvu51WWLZG0QNK+SuxWd3+yl+d7tA40lyKnqnzvvffCzKOPPhpmHnnkkTCzcePGMDNjxowwc91114WZb3/722FmypQpYWbUqKE3V9bM5O5W7bEiezQPSPpyleV3u/v5la+qJQMAUoGicfdnJR2o8lDV5gKAnmo5RrPYzDaZ2c/N7FMDNiIAQ05/i2aZpDPdfZakPZLuHrghARhq+nVEyt07ut1dLmltX/nW1tZjt0ulkkqlUn9WC6CBlMtllcvlQtmiRWPqdkzGzKa4+57K3TmStvT15O5FA2Bo6LnTsHTp0l6zYdGY2a8llSRNMLO3JC2RdImZzZLUKWmHpO/UMmAAQ1tYNO5+fZXFDySMBcAQFU7Yq3kFTNhrGEX+Hd54440w89RTT4WZe++9N8wUuR72ySefHGauueaaMHPDDTeEmXPOOSfMjB49OswMV7VO2AOAmlA0ANJRNADSUTQA0lE0ANJRNADSUTQA0lE0ANIxYa8JHDp0KMwcOFDtlEEft2zZsjCzZs2aMLNt27Ywc9ppp4WZefPmhZk5c+aEmZkzZ4YZ5GPCHoBBRdEASEfRAEhH0QBIR9EASEfRAEhH0QBIR9EASMeEvURF/t5btvR5XndJ0urVq8PMT3/60zDzwQcfhJkLL7wwzBQ5W93Xv/71MDNx4sQwM3LkyDCDxsCEPQCDiqIBkI6iAZCOogGQjqIBkI6iAZCOogGQjqIBkC689vZw1NnZGWY6OjrCzIMPPhhmHn744TDz2muvhZkzzjgjzHzve98LM5deemmYKXKZWqA79mgApKNoAKSjaACko2gApKNoAKSjaACko2gApKNoAKQLz7BnZtMk/VLSZEmdkpa7+71mNl7SI5KmS9ohaa67v1fl+Q11hr3Dhw+HmfXr14eZ5cuXh5mNGzeGmfPOOy/MfOMb3wgzCxYsCDNjxowJMyNG8H8P+qfWM+z9WdJ33f2zkv5W0mIzO0fSLZI2uPtMSU9L+uFADRjA0BIWjbvvcfdNldsHJW2TNE3SlZJWVmIrJV2VNUgAze0T7Seb2WmSZkl6TtJkd98rdZWRpEkDPTgAQ0PhX6o0s3GSHpd0s7sfNLOeB156PRDT2tp67HapVFKpVPpkowTQcMrlssrlcqFsoaIxs1HqKplV7v7RtT/2mtlkd99rZlMk7evt+d2LBsDQ0HOnYenSpb1mi751+oWkre5+T7dlayTNq9y+UVJ88SEAw1K4R2NmF0m6QdLLZvaSut4i3SrpLkmPmtk/StopaW7mQAE0r7Bo3P13knq7XOAXB3Y4AIaiIXWGvSJnxlu5cmWYeeihh8LM5s2bw8xNN90UZq699tow09LSEmaYaIdGxncngHQUDYB0FA2AdBQNgHQUDYB0FA2AdBQNgHQUDYB0TTNh78CBA2Fmy5YtYabIhL2DBw+GmYULF4aZRYsWhZkpU6aEGbOqJy0DmgZ7NADSUTQA0lE0ANJRNADSUTQA0lE0ANJRNADSUTQA0jXEhL0iZ8Zrb28PMz/60Y/CTEdHR5i5/PLLw8zNN98cZsaPHx9mmIyH4YA9GgDpKBoA6SgaAOkoGgDpKBoA6SgaAOkoGgDpKBoA6eoyYe/o0aN9Pl7kjHbbt28PM88880yYKTIZ7+KLLw4zEyZMCDNMxgO6sEcDIB1FAyAdRQMgHUUDIB1FAyAdRQMgHUUDIB1FAyBdOGHPzKZJ+qWkyZI6Jd3v7v9hZkskLZC0rxK91d2frPYaR44c6XMdRS53u2/fvjBz6NChMPPqq6+GmSKXzX3iiSfCDJCtyKTQSy+9NMy0tLSEmZNOOqnQmKopMjP4z5K+6+6bzGycpD+Y2VOVx+5297v7vXYAw0JYNO6+R9Keyu2DZrZN0tTKw8yxBxD6RMdozOw0SbMkPV9ZtNjMNpnZz83sUwM8NgBDROFfqqy8bXpc0s2VPZtlkm53dzezOyTdLemmas+94447jt2++OKLC/3SIoDG1tbWpra2tkLZQkVjZqPUVTKr3H21JLl79+uWLJe0trfn33bbbYUGA6B59NxpuPPOO3vNFn3r9AtJW939no8WmNmUbo/PkbTlkw0TwHBR5OPtiyTdIOllM3tJkku6VdL1ZjZLXR9575D0ncRxAmhiRT51+p2kkVUeqjpnBgB6qssZ9ty9z8ejCX1Sscl4RS6tu3nz5gHJAM1ixIj4CMnJJ58cZj7zmc/0fwz9fiYAFETRAEhH0QBIR9EASEfRAEhH0QBIR9EASEfRAEhn0WS6mldg5tnrADD4zEzuXvUcVezRAEhH0QBIR9EASEfRAEhX96Ipl8v1XmXNGHN9MOb6GIwxUzQFMOb6YMz1MSyKBsDwQ9EASFeXCXupKwDQMHqbsJdeNADAWycA6SgaAOnqWjRm9hUze9XMXjOzf63nuvvLzHaY2f+a2Utm9sJgj6caM1thZnvNbHO3ZePNbL2ZtZvZuka7NnovY15iZu+Y2f9Uvr4ymGPszsymmdnTZvaKmb1sZv9SWd6w27nKmP+5srzu27lux2jMbISk1yR9QdIuSS9KutbdX63LAPrJzN6UdIG7HxjssfTGzD4n6aCkX7r7eZVld0na7+7/Vin18e5+y2COs7texrxE0v+5+92DOrgqKldmneLumyrXof+DpCslfUsNup37GPM/qM7buZ57NLMlve7uO939iKSH1fWXbnSmBn+L6e7PSupZhFdKWlm5vVLSVXUdVKCXMUtd27vhuPsed99UuX1Q0jZJ09TA27mXMU+tPFzX7VzPH6Cpkt7udv8d/eUv3chc0joze9HMFgz2YD6BSe6+V+r6hpM0aZDHU9RiM9tkZj9vpLch3ZnZaZJmSXpO0uRm2M7dxvx8ZVFdt3ND/0/dIC5y9wsl/Z26/nE+N9gD6qdmmMewTNKZ7j5L0h5JjfgWapykxyXdXNlL6LldG247Vxlz3bdzPYvmj5JO7XZ/WmVZQ3P33ZU/OyT9Rl1vAZvBXjObLB17r75vkMcTcveObqdjXC6pZTDH05OZjVLXD+wqd19dWdzQ27namAdjO9ezaF6UNMPMppvZcZKulbSmjuv/xMxsbOV/A5nZiZK+JGnL4I6qV6aPv+9eI2le5faNklb3fEID+NiYKz+oH5mjxtvWv5C01d3v6bas0bfzX415MLZzXWcGVz5Gu0ddBbfC3X9ct5X3g5mdrq69GJc0StKvGnHMZvZrSSVJEyTtlbRE0n9JekzSKZJ2Sprr7u8O1hh76mXMl6jrOEKnpB2SvvPR8Y/BZmYXSWqT9LK6vh9c0q2SXpD0qBpwO/cx5utV5+3MryAASMfBYADpKBoA6SgaAOkoGgDpKBoA6SgaAOkoGgDpKBoA6f4fp8/fwEbFw1cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1c052ee910>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show image\n",
    "vec2img(digit_data[img_number,1:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "X = digit_data[:,1:]\n",
    "y = digit_data[:,0]\n",
    "\n",
    "# create SVM model\n",
    "clf_svm = svm.SVC()\n",
    "clf_svm.fit(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 1.000000\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score: %f\" % clf_svm.score(X, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Actual', 2.0)\n",
      "('Predict', array([ 2.]))\n",
      "(Correct prediction)\n"
     ]
    }
   ],
   "source": [
    "# Prediction\n",
    "\n",
    "print(\"Actual\", y[img_number])\n",
    "predict_class = clf_svm.predict([X[img_number,:]])\n",
    "print(\"Predict\", predict_class)\n",
    "chkpredict(y[img_number], predict_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RBF SVM parameters\n",
    "This example illustrates the effect of the parameters ```gamma``` and ```C``` of the Radial Basis Function (RBF) kernel SVM.  \n",
    "\n",
    "Intuitively, the ```gamma``` parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close’. The ```gamma``` parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.\n",
    "\n",
    "The ```C``` parameter trades off misclassification of training examples against simplicity of the decision surface. A low ```C``` makes the decision surface smooth, while a high ```C``` aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors.  \n",
    "\n",
    "useful link: http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'C': 10000000000.0, 'gamma': 1.0000000000000001e-09} with a score of 1.00\n",
      "Model with rank: 1\n",
      "Mean validation score: 1.000 (std: 0.000)\n",
      "Parameters: {'C': 10000000000.0, 'gamma': 1.0000000000000001e-09}\n",
      "\n",
      "Model with rank: 2\n",
      "Mean validation score: 0.996 (std: 0.004)\n",
      "Parameters: {'C': 100.0, 'gamma': 1000.0}\n",
      "\n",
      "Model with rank: 2\n",
      "Mean validation score: 0.996 (std: 0.004)\n",
      "Parameters: {'C': 10000000000.0, 'gamma': 1000.0}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import Normalize\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "# Utility function to report best scores\n",
    "def report(results, n_top=3):\n",
    "    for i in range(1, n_top + 1):\n",
    "        candidates = np.flatnonzero(results['rank_test_score'] == i)\n",
    "        for candidate in candidates:\n",
    "            print(\"Model with rank: {0}\".format(i))\n",
    "            print(\"Mean validation score: {0:.3f} (std: {1:.3f})\".format(\n",
    "                  results['mean_test_score'][candidate],\n",
    "                  results['std_test_score'][candidate]))\n",
    "            print(\"Parameters: {0}\".format(results['params'][candidate]))\n",
    "            print(\"\")\n",
    "            \n",
    "            \n",
    "# It is usually a good idea to scale the data for SVM training.\n",
    "scaler = StandardScaler()\n",
    "X = scaler.fit_transform(X)\n",
    "\n",
    "\n",
    "# #############################################################################\n",
    "# Train classifiers\n",
    "#\n",
    "# For an initial search, a logarithmic grid with basis\n",
    "# 10 is often helpful. Using a basis of 2, a finer\n",
    "# tuning can be achieved but at a much higher cost.\n",
    "\n",
    "C_range = np.logspace(2, 10, 2)\n",
    "gamma_range = np.logspace(-9, 3, 2)\n",
    "param_grid = dict(gamma=gamma_range, C=C_range)\n",
    "\n",
    "cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)\n",
    "grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv)\n",
    "grid.fit(X, y)\n",
    "\n",
    "print(\"The best parameters are %s with a score of %0.2f\"\n",
    "      % (grid.best_params_, grid.best_score_))\n",
    "report(grid.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  1.00000000e-02   1.00000000e+10]\n",
      "[  1.00000000e-09   1.00000000e+03]\n"
     ]
    }
   ],
   "source": [
    "C_range = np.logspace(-2, 10, 2)\n",
    "gamma_range = np.logspace(-9, 3, 2)\n",
    "\n",
    "print C_range\n",
    "print gamma_range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## linear kernel "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best parameters are {'kernel': 'linear', 'C': 100.0} with a score of 1.00\n",
      "Model with rank: 1\n",
      "Mean validation score: 1.000 (std: 0.000)\n",
      "Parameters: {'kernel': 'linear', 'C': 100.0}\n",
      "\n",
      "Model with rank: 1\n",
      "Mean validation score: 1.000 (std: 0.000)\n",
      "Parameters: {'kernel': 'linear', 'C': 10000.0}\n",
      "\n",
      "Model with rank: 1\n",
      "Mean validation score: 1.000 (std: 0.000)\n",
      "Parameters: {'kernel': 'linear', 'C': 1000000.0}\n",
      "\n",
      "Model with rank: 1\n",
      "Mean validation score: 1.000 (std: 0.000)\n",
      "Parameters: {'kernel': 'linear', 'C': 100000000.0}\n",
      "\n",
      "Model with rank: 1\n",
      "Mean validation score: 1.000 (std: 0.000)\n",
      "Parameters: {'kernel': 'linear', 'C': 10000000000.0}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "C_range = np.logspace(2, 10, 5)\n",
    "param_grid = dict(C=C_range, kernel=['linear'])\n",
    "\n",
    "cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)\n",
    "grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv)\n",
    "grid.fit(X, y)\n",
    "\n",
    "print(\"The best parameters are %s with a score of %0.2f\"\n",
    "      % (grid.best_params_, grid.best_score_))\n",
    "report(grid.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}