kNN-Basic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *k*-NN Basics in scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First use `NearestNeighbors` to identify neighbours.  \n",
    "\n",
    "#### Two objects:   \n",
    "- **KNeighborsClassifier** is the classifier object (fit, predict, etc)  \n",
    "- **NearestNeighbors** an object for returning NNs (not a classifier)\n",
    "\n",
    "Athlete Selection Data  \n",
    "First load dataset into a data frame.  \n",
    "`AthleteSelection.csv` files needs to be in the same directory as the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import preprocessing\n",
    "import sys\n",
    "\n",
    "from sklearn.neighbors import NearestNeighbors\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "athlete = pd.read_csv('AthleteSelection.csv',index_col = 'Athlete')\n",
    "athlete.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = athlete.index\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Store features and labels in numpy arrays X and y\n",
    "`X` is a numpy array containing the training features.  \n",
    "`y` contains the class labels.   \n",
    "`q` is a query example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = athlete.pop('Selected').values\n",
    "X = athlete.values\n",
    "q = [5.0,7.5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as mpatches\n",
    "%matplotlib inline\n",
    "\n",
    "color= ['red' if l == 'No' else 'green' for l in y]\n",
    "x1 = X[:,0]\n",
    "x2 = X[:,1]\n",
    "plt.figure(figsize=(7,6))\n",
    "plt.scatter(x1,x2, color=color)\n",
    "plt.scatter(q[0],q[1],color='black')\n",
    "plt.annotate('q',(q[0]+0.05,q[1]))\n",
    "plt.title(\"Athlete Selection\")\n",
    "plt.xlabel(\"Speed\")\n",
    "plt.ylabel(\"Agility\")\n",
    "plt.grid()\n",
    "red_patch = mpatches.Patch(color='red', label='Not Selected')\n",
    "blue_patch = mpatches.Patch(color='green', label='Selected')\n",
    "plt.legend(handles=[red_patch, blue_patch],loc=4)\n",
    "for i, txt in enumerate(names):\n",
    "    plt.annotate(txt, (x1[i]+0.05, x2[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Normalization\n",
    "Features may be measured on very different scales.  \n",
    "(Not really an issue here.)  \n",
    "Rescale the data so that all features have the same scale, two options:\n",
    "- N(0,1) rescale with zero mean and unit variance\n",
    "- MinMax scaling - typically in the range (0,1)\n",
    "\n",
    "### N(0,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = preprocessing.StandardScaler().fit(X)\n",
    "X_scaled = scaler.transform(X)\n",
    "q_scaled = scaler.transform([q])\n",
    "q_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as mpatches\n",
    "color= ['red' if l == 'No' else 'green' for l in y]\n",
    "x1 = X_scaled[:,0]\n",
    "x2 = X_scaled[:,1]\n",
    "plt.figure(figsize=(7,6))\n",
    "plt.scatter(x1,x2, color=color)\n",
    "plt.scatter(q_scaled[0,0],q_scaled[0,1],color='black')\n",
    "plt.annotate('q',(q_scaled[0,0]+0.05,q_scaled[0,1]))\n",
    "plt.title(\"Athlete Selection (Normalized)\")\n",
    "plt.xlabel(\"Speed N(0,1)\")\n",
    "plt.ylabel(\"Agility N(0,1)\")\n",
    "plt.grid()\n",
    "red_patch = mpatches.Patch(color='red', label='Not Selected')\n",
    "blue_patch = mpatches.Patch(color='green', label='Selected')\n",
    "plt.legend(handles=[red_patch, blue_patch],loc=4)\n",
    "for i, txt in enumerate(names):\n",
    "    plt.annotate(txt, (x1[i]+0.05, x2[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding Neighbours\n",
    "Find the first two NNs for `q`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "athlete_neigh = NearestNeighbors(n_neighbors=2)\n",
    "athlete_neigh.fit(X_scaled) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distances and the indexes of the two NNs for `q`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "athlete_neigh.kneighbors(q_scaled, 2, return_distance=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three NNs for `q`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find three nearest neighbours for q\n",
    "q3n = athlete_neigh.kneighbors(q_scaled, 3)[1][0]\n",
    "# q3n contains the 'index' of the nearest neighbours\n",
    "for n in q3n:\n",
    "    print(names[n], end = ' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *k*-NN Classifier\n",
    "Use `KNeighboursClassifier` to build a *k*-NN classifier.\n",
    "Two methods:\n",
    "- `fit` sets up the classifier with the training data, takes two arguments, the features and the labels. \n",
    "- `predict` produces the output for the test set (just one test example in this case).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kNN = KNeighborsClassifier(n_neighbors = 3)\n",
    "kNN = kNN.fit(X_scaled,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kNN.predict(q_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Breast Cancer example\n",
    "The `scikit-learn` distribution contains a number of example datasets.  \n",
    "`load_breast_cancer()` loads the dataset as a dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "data = load_breast_cancer()\n",
    "X = data[\"data\"]\n",
    "y = data[\"target\"]\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"feature_names\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the relative size of the features to see if normalisation is required.  \n",
    "It is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f_min = sys.maxsize\n",
    "f_max = 0\n",
    "for i in range(X.shape[1]):\n",
    "    if X[i].mean() < f_min:\n",
    "        f_min = X[i].mean()\n",
    "    if X[i].mean() > f_max:\n",
    "        f_max = X[i].mean()\n",
    "print(\"Smallest Mean: {0:.2f}\".format(f_min)) \n",
    "print(\"Largest Mean: {0:.2f}\".format(f_max)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the data and use `train_test_split` to carve off 1/3 of the data to use as a test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B_scaler = preprocessing.StandardScaler().fit(X)\n",
    "X_scaled = B_scaler.transform(X)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=1/3)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`accuracy_score` calculates the accuracy of the predicted labels `y_dash`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kNN = KNeighborsClassifier(n_neighbors = 3)\n",
    "kNN = kNN.fit(X_train,y_train)\n",
    "y_dash = kNN.predict(X_test)\n",
    "accuracy_score(y_test, y_dash)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the first 35 predictions with the actuals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(y_test[:35])\n",
    "print(y_dash[:35])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}