02-non-linearly-separable-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ziamAoLF_-lW"
   },
   "source": [
    "# Non-Linearly Separable Data\n",
    "## CSC2034\n",
    "### Cameron Trotter (c.trotter2@ncl.ac.uk)\n",
    "\n",
    "### Google Colab Setup\n",
    "\n",
    "All of the notebooks you will be running in these lab sessions are designed to be ran using [Google Colab](https://colab.research.google.com/). For setup instructions, see this repo's README. \n",
    "\n",
    "In order to make things work on colab, we need to clone this repo and then (in another cell because colab dictates this...) move into the repo directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_DGUbFoA_-lZ",
    "outputId": "dd52f0df-215f-45fc-ace9-a825091f2f6b"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/Trotts/csc2034-ds-demos.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mixFhdjp_-la"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir('csc2034-ds-demos')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fH99yOFN_-lb"
   },
   "source": [
    "### Building Synthetic Data\n",
    "\n",
    "As well as being able to more simple data, `sklearn` can also produce more complicated non-linarly separable data. This data cannot be split using a single line, e.g. data which is *circular*. Like the first notebook, let's build a synthetic dataset to work with. \n",
    "\n",
    "Task: Using sklearn's `make_circles` method, build a non-linearly separable dataset. The [docs](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_circles.html) may help you. I have provided the hyperparameters you will need. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "u36MyGnw_-lc"
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_circles\n",
    "\n",
    "n_samples = 1000\n",
    "noise = 0.1\n",
    "factor = 0.5\n",
    "random_state = 1\n",
    "\n",
    "data, labels = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jhPWx6dR_-ld"
   },
   "source": [
    "Now we have a dataset, let's visualise it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "dePen-Pl_-ld",
    "outputId": "9b1f6f53-e0c9-433a-f189-ad90b1e72346"
   },
   "outputs": [],
   "source": [
    "from helpers import show_scatterplot\n",
    "\n",
    "show_scatterplot(data, labels, title = 'Non-linearly separable data',\n",
    "                 xlabel = 'x1 feature', ylabel = 'x2 feature')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EhVjyxKu_-le"
   },
   "source": [
    "### Can We Use The Models We Made Before?\n",
    "\n",
    "As you might be able to tell from the scatterplot above, it will not be possible with the data distribution we have created to generate a line of best fit. To prove this, we can try. \n",
    "\n",
    "#### Logistic Regression\n",
    "\n",
    "Task: Create a Logistic Regression model, based on the code from `01-linearly-separable-data`, fit it to the dataset, generate a line of best fit, and predict on the test set. Remeber to split and scale your data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Rs0TA-tk_-lf"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "data_train, data_test, labels_train,  labels_test = #...\n",
    "\n",
    "#...\n",
    "\n",
    "penalty = 'l2'\n",
    "C = 1\n",
    "solver = 'lbfgs'\n",
    "multi_class = 'ovr'\n",
    "\n",
    "logistic_regression = #...\n",
    "\n",
    "#..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zXccDLQQ_-lh"
   },
   "source": [
    "Let's plot the line of best fit, and output some evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 445
    },
    "id": "OIbApoI0_-li",
    "outputId": "0871d70c-61eb-4b9a-83da-7c61e59e5279"
   },
   "outputs": [],
   "source": [
    "from helpers import plot_line_of_best_fit\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "plot_line_of_best_fit(classifier = logistic_regression,\n",
    "                      data = data_test_scaled,\n",
    "                      labels = labels_test,\n",
    "                      logistic = True,\n",
    "                      title = \"Logistic regression line of best fit and test data\")\n",
    "\n",
    "\n",
    "test_acc = accuracy_score(labels_test, logistic_regression_label_predictions)\n",
    "\n",
    "\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pq8WZ9Q-_-lj"
   },
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8vDsbB2e_-lk",
    "outputId": "9cb0b975-2228-4fcd-9cbc-219f332bd02e"
   },
   "outputs": [],
   "source": [
    "print(f\"Test acc check: {test_acc == 0.45}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w61J4VLO_-lk"
   },
   "source": [
    "The line of best fit generated by the model is not capable of capturing the non-linearity of data, and this is reflected in the low test accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OKBqMo5__-ll"
   },
   "source": [
    "#### Decision Trees\n",
    "\n",
    "Some models can be used for more complex datasets. One of these is decision trees. \n",
    "\n",
    "Task: Create a Decision Tree model, based on the code from `01-linearly-separable-data`, fit it to the dataset, and predict on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "apKRmxpo_-ll",
    "outputId": "4d585192-7ada-4599-b471-07e73c9f645b"
   },
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "\n",
    "decision_tree = tree.DecisionTreeClassifier()\n",
    "decision_tree.fit(data_train_scaled, labels_train)\n",
    "\n",
    "decision_tree_predictions = decision_tree.predict(data_test_scaled)\n",
    "\n",
    "test_acc = accuracy_score(labels_test, decision_tree_predictions)\n",
    "\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rZ5oPusp_-lm"
   },
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BgGJ7hJ6_-ln",
    "outputId": "8b8c075a-336a-4380-a31a-28b1c3f63022"
   },
   "outputs": [],
   "source": [
    "print(f\"Test acc check: {test_acc == 0.975}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x0Ryxme-_-ln"
   },
   "source": [
    "Let's visualise the state space produced by the decision tree..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 390
    },
    "id": "zkKGHi5D_-lo",
    "outputId": "e828c222-9cc2-4bbb-eb91-eaeb2beb713a"
   },
   "outputs": [],
   "source": [
    "from helpers import plot_contour_fit\n",
    "\n",
    "plot_contour_fit(decision_tree, data_train_scaled, labels_train, data_test_scaled, labels_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0SphvtIN_-lo"
   },
   "source": [
    "The state space is capable of capturing the non-lineararity of the data. We can also visualise the tree using `graphviz`, but this may not be very informative for our generated data. \n",
    "\n",
    "#### Non-Linear SVM\n",
    "\n",
    "SVMs are another model that can be either linear or non-linear. This depends on the kernel hyperparameter. Like before, lets see how a linear SVM performs on the data.\n",
    "\n",
    "Task: Create a Linear SVM model, based on the code from `01-linearly-separable-data`, fit it to the dataset, and predict on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "u8U-9Men_-lp",
    "outputId": "71f18952-4415-48cb-ae63-f2b3da41bde2"
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "#...\n",
    "\n",
    "test_acc = accuracy_score(labels_test, svm_predictions)\n",
    "\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F57GHulE_-lp"
   },
   "source": [
    "Not much better than the Logistic Regression. What about if we change the kernel so that we can work with non-linear data? For this, we can utilise sklearn's `SVC` model implementation, the docs for which can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).\n",
    "\n",
    "Task: Build a non-linear SVM using the following hyperparameters, fit it to the data, and use it to predict on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "id": "Z-pBG5RE_-lq",
    "outputId": "ff2f8976-97ed-48ed-bc7b-e17629bee28c"
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "kernel = 'poly'\n",
    "degree = 3\n",
    "C = 5\n",
    "coef0 = 1\n",
    "\n",
    "non_linear_svm = SVC(kernel = kernel,\n",
    "          degree = degree,\n",
    "          C = C,\n",
    "          coef0 = coef0)\n",
    "\n",
    "non_linear_svm.fit(data_train_scaled, labels_train)\n",
    "\n",
    "non_linear_svm_predictions = non_linear_svm.predict(data_test_scaled)\n",
    "\n",
    "# Delete Above\n",
    "test_acc = accuracy_score(labels_test, non_linear_svm_predictions)\n",
    "\n",
    "print(f\"Test acc: {test_acc * 100}%\")\n",
    "plot_contour_fit(non_linear_svm, data_train_scaled, labels_train,\n",
    "                 data_test_scaled, labels_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CPOxAqhK_-lq"
   },
   "source": [
    "There are multiple different non-linear kernels we can use. How does utilising another effect the SVM?\n",
    "\n",
    "Task: Implement a non-linear SVM with the `rbf` kernel, and produce a test accuracy and plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "id": "hr42GW5k_-lr",
    "outputId": "f0e3d123-1fb4-442a-c68c-e3df5437937d"
   },
   "outputs": [],
   "source": [
    "#...\n",
    "\n",
    "print(f\"Test acc: {test_acc * 100}%\")\n",
    "\n",
    "#..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jQZaCgXB_-lr"
   },
   "source": [
    "Your state space should now be completely different, however the accuracy is very similar. A little bit of this is due to the ease of the data we created. Try out other hyperparameters and see if you can get better!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m4_9NEcz_-lr"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}