01-logistic-regression-svm-decision-trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y6G4W7WRtyef"
   },
   "source": [
    "# Logistic Regression, Linear SVM, and Decision Trees\n",
    "## CSC2034\n",
    "### Cameron Trotter (c.trotter2@ncl.ac.uk)\n",
    "\n",
    "\n",
    "Welcome to the CSC2034 practicals. Throughout these sessions, you will learn how to use Python and the [sklearn package](https://scikit-learn.org/stable/) to perform basic data science, using models to infer relationships from data. I'd encourage you to make use of the demonstrators here, as well as explore things at your own pace. There are potentially multiple ways of performing operations in these notebooks, and Paolo will go through the theoretics in his lectures. Remeber, Google and Stack Overflow are your friend - just try to understand the code you find rather than blindly copying it in!\n",
    "\n",
    "Data science is an extremely broad topic, so let's begin with one of the simplest forms of model you might encounter, [logistic regression](https://www.w3schools.com/python/python_ml_logistic_regression.asp). This type of model is great for binary classification of data (that which contains only two possible classes). After this, we will take a look at a [Linear Support Vector Machine (SVM)](https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47) and [Decision Trees](https://www.w3schools.com/python/python_ml_decision_tree.asp).\n",
    "\n",
    "### Google Colab Setup\n",
    "\n",
    "All of the notebooks you will be running in these lab sessions are designed to be ran using [Google Colab](https://colab.research.google.com/). For setup instructions, see this repo's README. \n",
    "\n",
    "In order to make things work on colab, we need to clone this repo and then (in another cell because colab dictates this...) move into the repo directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4h2CVpAjtyeh",
    "outputId": "3b338a1e-e9b9-49aa-e3b4-da25f3240459"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/Trotts/csc2034-ds-demos.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KpzuqvcYtyei"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir('csc2034-ds-demos')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pNNXnowDtyej"
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "\n",
    "n_samples = 1000\n",
    "n_classes = 2\n",
    "n_features = 2\n",
    "n_clusters_per_class = 2\n",
    "n_redundant = 0\n",
    "n_informative = 2\n",
    "random_state = 5\n",
    "flip_y = 0.1\n",
    "\n",
    "data, labels = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EqB_0Ggstyek"
   },
   "source": [
    "Once created, `data` should be a numpy array of shape (n_samples, n_features), with each feature being a float. `labels` should be a numpy array of shape (1000,) representing the interger class labels of each datapoint, either 0 or 1.\n",
    "\n",
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BflqrAUktyek",
    "outputId": "37d906b1-49df-41b6-91e0-8b8fa091c9fc"
   },
   "outputs": [],
   "source": [
    "print(f\"data is of shape (n_samples, n_features): {data.shape == (n_samples, n_features)}\")\n",
    "print(f\"data is float: {data.dtype == 'float64'}\")\n",
    "print(f\"labels is of shape (n_samples,): {labels.shape == (n_samples,)}\")\n",
    "print(f\"labels is int: {labels.dtype == 'int64'}\")\n",
    "print(f\"Random state set correctly: {data[0].tolist() == [1.031573753611461, 0.9858931626901614]} \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uCeHV3Ontyel"
   },
   "source": [
    "### Visualisation\n",
    "\n",
    "Being able to visually examine the data you have created is extremely important, so let's take a look at what we have. [Seaborn](https://seaborn.pydata.org/) is a powerful visualisation package built on top of [matplotlib](https://matplotlib.org/stable/users/index.html); let's use it. We're going to make lots of scatterplots here, so I've made a function to reduce code duplication, which can be found in `helpers.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "WclusWZZtyem",
    "outputId": "8a123c2c-90e8-497b-976f-a6f83f2782a6"
   },
   "outputs": [],
   "source": [
    "from helpers import show_scatterplot\n",
    "\n",
    "show_scatterplot(data, labels, title = \"Scatterplot of synthetic data\",\n",
    "                xlabel = 'x1 feature', ylabel = 'x2 feature')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y-qOAHVgtyen"
   },
   "source": [
    "From the plot we can see we have created a simple dataset, where each datapoint contains two features, `x1` and `x2`, and a class label (commonly denoted `y` in data science package documentation), either `0` or `1`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SncfDTyZtyeo"
   },
   "source": [
    "### Spliting Into Sets\n",
    "\n",
    "Now that we have some data, we need to put it in a form usable to train a model on. For this, we need to split the data into a train and test set. In practice we might also split these further to create a validation set, but here we will keep it simple.\n",
    "\n",
    "As the names imply, we will utilise the train set to train our model and the test set to test its performance. It is _very_ important that we do not mix the sets, as using training data at test time can skew your metrics. Thankfully, sklearn has the ability to split the data for us through the `train_test_split` method. \n",
    "\n",
    "For your next task, use `train_test_split` to create two sets where the train set contains 80% of the entire dataset. The [docs](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) may help you. \n",
    "\n",
    "NOTE: Make sure you set the `random_state` of the method to the same value as previously to ensure reproducability. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oEn4uPS2tyep"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, labels_train,  labels_test = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KDonZgSktyep"
   },
   "source": [
    "Once complete your train set should contain 80% of the whole dataset, with your test set containing the rest.\n",
    "\n",
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BSn8nK8Styeq",
    "outputId": "cf662a55-b7cc-4001-e943-3bfec612029e"
   },
   "outputs": [],
   "source": [
    "print(f\"train set is 80% of dataset: {len(data_train) == n_samples * 0.8}\")\n",
    "print(f\"test set is 20% of dataset: {len(data_test) == n_samples * 0.2}\")\n",
    "print(f\"Random state set correctly: {data_train[0].tolist() == [-1.2833740134353988, -1.4154849762712642]} \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nAh6v-qmtyer"
   },
   "source": [
    "### Data Scaling\n",
    "\n",
    "In the vast majority of cases, before we can use the splits we have created we will need to scale them. This processes forces our data to have a mean of 0 and a variance of 1, and is required to allow for optimal performance of many common machine learning models. If you want to see the negative effect not scaling your data can have, scikit-learn has a [section on the effects of not standardizing your data](\n",
    "https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html#sphx-glr-auto-examples-preprocessing-plot-scaling-importance-py).\n",
    "\n",
    "Sklearn provides the `StandardScaler` functionality which allows us to very quickly scale our splits to meet the required conditions. Unlike the other functionality provided by sklearn we have used so far, `StandardScaler` needs to be fitted to the training data, that is it needs to learn some parameters in order to operate. The [docs](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) for `StandardScaler` provides more information regarding this. \n",
    "\n",
    "For your next task, scale the train and test splits you have created so that the data has a mean of 0 and a variance of 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "u1uvvCjityer"
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "#...\n",
    "\n",
    "data_train_scaled = #...\n",
    "data_test_scaled = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W9ozt12Ztyes"
   },
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rv_WtIv5tyes",
    "outputId": "e5861c13-c35b-4b9b-b3df-e4c53ff38c76"
   },
   "outputs": [],
   "source": [
    "print(f\"train set has been correctly scaled: {data_train_scaled[0].tolist() == [-0.9534808523834634, -1.1352230069380507]}\")\n",
    "print(f\"test set has been correctly scaled: {data_test_scaled[0].tolist() == [1.2480977883641757, -0.8652814153775584]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JZ8nH1hFtyet"
   },
   "source": [
    "Now let's visualise our scaled splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 791
    },
    "id": "ym3SSjeTtyet",
    "outputId": "a6a8b314-2b48-4f74-9ba0-399540aae58b"
   },
   "outputs": [],
   "source": [
    "show_scatterplot(data_train_scaled, labels_train, title = \"Scatterplot of synthetic, scaled, train data\",\n",
    "                xlabel = 'x1 feature', ylabel = 'x2 feature')\n",
    "\n",
    "\n",
    "show_scatterplot(data_test_scaled, labels_test, title = \"Scatterplot of synthetic, scaled, test data\",\n",
    "                xlabel = 'x1 feature', ylabel = 'x2 feature')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xDchajcltyeu"
   },
   "source": [
    "### Model Building\n",
    "\n",
    "Now that we have our synthetic data in a useable form, let's now look at how we train statistical models. There are a whole host of models provided by sklearn, but lets start with some of the most basic. Don't worry, we won't be going into the statistics behind how these models work - that will come if you decide to undertake optional modules in machine learning in later stages of your degree. \n",
    "\n",
    "#### Logistic Regression\n",
    "\n",
    "One of the most basic forms of machine learning is logistic regression. This method is very similar to linear regression, which you may have come across before if you have taken any stats modules in the past, except we output a binary variable rather than something continuous.\n",
    "\n",
    "Logistic regression works well for the data we have, as we are aiming to classify a data point into one of two classes. Sklearn provides functionality for logistic regression through the `LogisticRegression` class. Like `StandardScaler`, we need to fit the logistic regression model to the data.\n",
    "\n",
    "Task: Build a logistic regression model and train it on the data. All machine learning models take as input variables called hyperparameters. These are parameters which must be user defined rather than learned, and allow learning to take place. I have defined some hyperparameters for you. Once again, the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) will help you in building the model and passing the hyperparameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "duMlfUb2tyeu",
    "outputId": "721ec210-f220-4741-d982-b098d855ad80"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "penalty = 'l2'\n",
    "C = 1\n",
    "solver = 'lbfgs'\n",
    "multi_class = 'ovr'\n",
    "\n",
    "logistic_regression = #...\n",
    "\n",
    "#...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TRGP9Bvvtyev"
   },
   "source": [
    "Now we have created a model and trained it, let's see what it looks like! The code for this is a bit complex, so I have provided it in `helpers.py`. \n",
    "\n",
    "First a bit of theory to help us understand what is going on. As I mentioned previously, logistic and linear regression are very similar. Linear regression can be used to output a continuous value (think house prices) by finding a straight line based on the training data it is given. Predictions from the model are sampled from this line. \n",
    "\n",
    "Logistic regression modifies linear regression by, essentially, passing the line of best fit through the [sigmoid function](https://deepai.org/machine-learning-glossary-and-terms/sigmoid-function). This constrains the outputs of the model, scaling the continuous output to one that is within the range [0,1]. As such, this can now be used as a _probability_ for some binary output, shifting the regression problem into a binary classification one. This will work well for our problem here, as our output is binary (the classes for each point can either be 0 or 1). \n",
    "\n",
    "To show the difference between linear and logistic regression lines of best fit, let's first plot the _linear_ line of best fit, before it is transformed by the sigmoid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 390
    },
    "id": "gKky7J8Ptyev",
    "outputId": "fcaecfa8-8619-4075-c1b9-2fc5f8b1d692"
   },
   "outputs": [],
   "source": [
    "from helpers import plot_line_of_best_fit\n",
    "\n",
    "plot_line_of_best_fit(classifier = logistic_regression,\n",
    "                      data = data_train_scaled,\n",
    "                      labels = labels_train,\n",
    "                      logistic = False,\n",
    "                      title = \"Linear regression line of best fit and training data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iXWog6hetyew"
   },
   "source": [
    "The red line in the plot above is linear, and does not fit the data well at all. Let's see how it looks on the test set..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 390
    },
    "id": "q5lux12Ttyew",
    "outputId": "bf275f62-f62b-4fc6-c7a4-581cd21e2c79"
   },
   "outputs": [],
   "source": [
    "plot_line_of_best_fit(classifier = logistic_regression,\n",
    "                      data = data_test_scaled,\n",
    "                      labels = labels_test,\n",
    "                      logistic = False,\n",
    "                      title = \"Linear regression line of best fit and test data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HQQ578nU8Rih"
   },
   "source": [
    "Again, not great. What happens if we take this linear line and pass it through the sigmoid function? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 836
    },
    "id": "OPbpQhpJ8Q2Q",
    "outputId": "d78deb44-209a-4d72-9b30-984fc2512b32"
   },
   "outputs": [],
   "source": [
    "plot_line_of_best_fit(classifier = logistic_regression,\n",
    "                      data = data_train_scaled,\n",
    "                      labels = labels_train,\n",
    "                      logistic = True,\n",
    "                      title = \"Logistic regression line of best fit and training data\")\n",
    "\n",
    "\n",
    "plot_line_of_best_fit(classifier = logistic_regression,\n",
    "                      data = data_test_scaled,\n",
    "                      labels = labels_test,\n",
    "                      logistic = True,\n",
    "                      title = \"Logistic regression line of best fit and test data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_q13x4NG8dSI"
   },
   "source": [
    "Our line now fits the training data better! We can use this line now as a decision boundary. No model is perfect however, so some data points don't fit this rule - these are the misclassifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FfkypeH5tyex"
   },
   "source": [
    "Noe we have a line of best and a trained model, we can use this to predict values for the test set data points. By doing this, we will be able to see how good our line of best fit is at correctly labelling our data. The model will produce predictions for each data point, and we can then compare these against the ground truth labels in the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "s_NaVecHtyex"
   },
   "outputs": [],
   "source": [
    "logistic_regression_label_predictions = logistic_regression.predict(data_test_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IaIEpn_Rtyex"
   },
   "source": [
    "We can also aquire some numerical metrics to show how the model is performing. One of the main ones is accuracy, which tells us how many data points in the set the model correctly labels. We take each data point in the set, predict a class for it using the model, and compare this to the ground truth label. \n",
    "\n",
    "Sklearn provides the ability to calculate these metrics using `accuracy_score`. The docs for the method are available [here](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html?highlight=accuracy_score#sklearn.metrics.accuracy_score).\n",
    "\n",
    "Task: Produce accuracy scores for the model you have created against test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hDADuvoItyex"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "test_acc = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LpEKxB2Styex"
   },
   "source": [
    "Let's check the accuracy score we achieve. We can display this as a percentage by multiplying by 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zeTFMSRJtyex",
    "outputId": "b2601eca-ff5e-45aa-99fa-b3a5e2fdf6d2"
   },
   "outputs": [],
   "source": [
    "print(f\"Logistic regression test accuracy: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z_S0PDMktyez"
   },
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "s5uU9ctKtyez",
    "outputId": "8d386489-6653-42ac-9309-014ee2c9c6b6"
   },
   "outputs": [],
   "source": [
    "print(f\"Test accuracy check: {test_acc == 0.835}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "reSlyJ6rtyez"
   },
   "source": [
    "There are lots of other metrics that sklearn can provide for us, such as precision, recall, and f-1 score. These can be produced using `classification_report`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IaG7m3Wctye0",
    "outputId": "1c3adb72-3b68-438c-8cbb-13e6b0340388"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(labels_test, logistic_regression_label_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6MQUn1Kotye0"
   },
   "source": [
    "We can also produce a confusion matrix, which can provide us insight into which classes the model is mislabelling. These are especially useful when you have a multiclass model to understand how your model's accuracy can be improved. Sklearn's `conusion_matrix` method can product a textual confusion matrix for you, which can be visually enhanced using seaborn. I have made a function in `helpers` to do this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "YS-8U11Xtye1",
    "outputId": "dc5b9a7f-6667-40cb-f6ec-ea5135c9e3fd"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "from helpers import plot_confusion_matrix\n",
    "import numpy as np\n",
    "\n",
    "conf_matrix = confusion_matrix(labels_test, logistic_regression_label_predictions)\n",
    "# Normalise the data for better visualisation\n",
    "conf_matrix = conf_matrix.astype('float') / conf_matrix.sum(axis=1)[:, np.newaxis]\n",
    "\n",
    "labels = np.unique(labels_test)\n",
    "\n",
    "plot_confusion_matrix(conf_matrix, labels, \"Confusion Matrix\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fxKELk6Atye1"
   },
   "source": [
    "This tells us that 85% of data points labelled as class 0 were correctly classified (True Negative) and 15% were incorrectly classified (False Positive). Further, 82% of data points labelled as class 1 were correctly classified (True Positive) and 18% were incorrectly classified (False Negative)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9tN3o_Oxtye2"
   },
   "source": [
    "We can also plot an ROC graph, which shows us the performance of the model at different thresholds. Again, these can be somewhat arduous to create, so code is provided for you in `helpers`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 405
    },
    "id": "dJ0L1l7otye2",
    "outputId": "c35edc7b-fc31-4d26-c441-14ccd0163bb3"
   },
   "outputs": [],
   "source": [
    "from helpers import plot_ROC\n",
    "\n",
    "plot_ROC(logistic_regression, data_test_scaled, labels_test, logistic_regression_label_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9JkiYhlEtye2"
   },
   "source": [
    "#### Support Vector Machine (SVM)\n",
    "\n",
    "There are lots of different models which a data scientist can utilise, each with their own strengths and weaknesses depening on the data. Thankfully sklearn contains multiple model implementations - one of which is an SVM. \n",
    "\n",
    "SVMs can be utilised for both linearly and non-linearly separable data, although sklearn dictates the use of two different functions depening. The documentation for a Linear SVM can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html).\n",
    "\n",
    "Task: Using the following hyperparameters, implement a Linear SVM on the generated data.\n",
    "\n",
    "* C = 1\n",
    "* Loss = hinge\n",
    "\n",
    "Once you have created a model and trained it on the correct split, create predictions from the model and evaluate their performance using the methods utilised previously. As our data is synthetic, and quite simple, expect very similar model performance to your linear line of best fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "qcCUebC5tye3",
    "outputId": "353d4cdc-ea3d-4ddf-9dbb-426c7f83768f"
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "#..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N4-1FLnFtye3"
   },
   "source": [
    "#### Decision Trees\n",
    "\n",
    "Decision Trees are another type of data science model. It utilises a tree structure to model outcomes based on provided values. \n",
    "\n",
    "Task: Create a decision tree using sklearn, fit it to your data, and generate evaluation metrics. You do not need to worry about hyperparameters here, nor do you need to use `plot_linear_fit` - instead use `plot_contour_fit`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "zRRa_v9utye3",
    "outputId": "84f03a6f-e264-4e75-ef30-ce56b4580ffc"
   },
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "from helpers import plot_contour_fit\n",
    "\n",
    "#..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T4Zxjb3otye3"
   },
   "source": [
    "We can visualise the decision tree using the `graphviz` package. Using this package is slightly easier than seaborn for this type of model. As our data is numeric, this will likely make little sense, but hopefully you will be able to see it's use when we have categorical data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "CGxKck2etye4",
    "outputId": "44467274-bd40-465b-e654-95f4c6f8631c",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import graphviz\n",
    "\n",
    "dot = tree.export_graphviz(decision_tree, out_file=None) \n",
    "graph = graphviz.Source(dot) \n",
    "graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8PzX_TY4tye4"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}