00- Tutorials/Machine Learning for Beginners Tutorial and Assessment/Machine Learning for Beginners.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ad53740",
   "metadata": {},
   "source": [
    "## Machine Learning for Neuroscience, <br>Department of Brain Sciences, Faculty of Medicine, <br> Imperial College London\n",
    "### Contributors: Francesca Palermo, Nan Fletcher-Lloyd, Alex Capstick, Yu Chen\n",
    "**Winter 2022**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9246f7fe",
   "metadata": {},
   "source": [
    "# Machine Learning for Beginners"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b161ecd",
   "metadata": {},
   "source": [
    "This tutorial will provide an introduction to machine learning using the machine learning library for Python scikit-learn (https://scikit-learn.org/stable)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca16fa6e",
   "metadata": {},
   "source": [
    "scikit-learn features various machine learning algorithms and can also be used for dimensionality reduction (reducing the number of random variables to consider), model selection, and pre-processing of data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb7b588",
   "metadata": {},
   "source": [
    "## Import scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa77cbf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn as sk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c97eaa",
   "metadata": {},
   "source": [
    "## Types of Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9546e5c",
   "metadata": {},
   "source": [
    "There are many types of machine learning. \n",
    "\n",
    "There are three main types of machine learning. These are supervised learning (classification and regression), unsupervised learning (clustering), and reinforcement learning (decision making).\n",
    "\n",
    "Other types include semi-supervised and self-supervised learning (not discussed in this module).\n",
    "\n",
    "Deep learning and deep reinforcement learning are further types of machine learning that we will not discuss in this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cebdae8",
   "metadata": {},
   "source": [
    "In this tutorial, we will focus on supervised and unsupervised learning examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c97eaa",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6e7882",
   "metadata": {},
   "source": [
    "First, we must remember to import our other dependencies."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c97eaa",
   "metadata": {},
   "source": [
    "### Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa77cbf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b8253d",
   "metadata": {},
   "source": [
    "### Supervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9618181b",
   "metadata": {},
   "source": [
    "Supervised learning uses labelled datasets to train algorithms to identify the category to which an obejct belongs or predict outcomes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331de788",
   "metadata": {},
   "source": [
    "#### Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b325901",
   "metadata": {},
   "source": [
    "Classification involves identifying a category to which an object belongs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed69c4c6",
   "metadata": {},
   "source": [
    "For this example, you are going to use the sci-kit learn dataset breast cancer.\n",
    "\n",
    "More on this dataset can be found here: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10f0b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.load_breast_cancer(as_frame=True) # this loads the dataset as a dictionary\n",
    "features = data.data # this derives features as a dataframe (30 features by 569 instances)\n",
    "labels = data.target # this derives labels as a dataframe (569 instances)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae0f399",
   "metadata": {},
   "source": [
    "Let's just do a quick check to make sure everything has loaded as it should."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3255149",
   "metadata": {},
   "outputs": [],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dec9a6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99497d36",
   "metadata": {},
   "source": [
    "*N.B. The labels are binary, with 0 indicating a negative diagnosis and 1 indicating a positive diagnosis of breast cancer, respectively.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4044c5ab",
   "metadata": {},
   "source": [
    "Here, we are going to focus on two features (mean area and mean smoothness), so the first thing we'll do is extract this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "079a9e7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "area_and_smoothness = features[['mean area', 'mean smoothness']]\n",
    "area_and_smoothness"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d5aa97",
   "metadata": {},
   "source": [
    "For this example, we are going to use logistic regression which, despite its name, is a linear classification model.\n",
    "\n",
    "Learn more about logistic regression here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "420abd6e",
   "metadata": {},
   "source": [
    "To use the Logistic Regression classifier, we must first import it using the following line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ad0e8e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "788e6f67",
   "metadata": {},
   "source": [
    "But before we starting training our model, we must first scale our feature data.\n",
    "\n",
    "Feature scaling is one of the most critical pre-processing steps in machine learning, with the most common techniques being standardization and normalization.\n",
    "\n",
    "Machine learning algorithms that calculate distance or assume normality are sensitive to relative scales of features, meaning that if the data is not scaled, features with a higher value range start dominating the model's decision-making process. Feature scaling is therefore needed to bring features with different ranges into comparable ranges.\n",
    "\n",
    "Feature scaling also allows for much faster model convergence.\n",
    "\n",
    "Learn more about scaling (when you need to and why, and how to do so) using the links below:\n",
    "\n",
    "https://towardsdatascience.com/all-about-feature-scaling-bcc0ad75cb35 \n",
    "\n",
    "https://towardsdatascience.com/feature-scaling-effectively-choose-input-variables-based-on-distributions-3032207c921f\n",
    "\n",
    "*N.B. scaling will feature in assessed labs later on during this module, so it would definitely be valuable to read through these two articles.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a938886d",
   "metadata": {},
   "source": [
    "In this scenario, we are going to use the sci-kit learn StandardScaler to standardize our data.\n",
    "\n",
    "Learn more about this scaler here: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html\n",
    "\n",
    "Another commonly used scaler is the MinMax scaler. The MinMax scaler normalizes data. Learn more here: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba8c9f1",
   "metadata": {},
   "source": [
    "First, let's import out scaler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "579021c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb590e45",
   "metadata": {},
   "source": [
    "Next, let's implement our standardization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89b04426",
   "metadata": {},
   "outputs": [],
   "source": [
    "standardized_data = pd.DataFrame(scaler.fit_transform(area_and_smoothness),\n",
    "                   columns=['mean area','mean smoothness'])\n",
    "standardized_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "710d673c",
   "metadata": {},
   "source": [
    "Now we've completed this step, we can move onto training our model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c6475a7",
   "metadata": {},
   "source": [
    "As you learned in today's lecture, whenever we are building a machine learning model that uses a supervised learning algorithm, it is important we have some way of evaluating the performance of that model.  However, learning the parameters of the prediction function and testing it on the same data would mean a model would just be repeating the labels of the samples it has just seen, deriving a perfect score, but failing to predict anything useful on as-of-yet unseen data. This situation is known as overfitting, and to avoid it the most common practice is to hold out part of the available data as a test set.\n",
    "\n",
    "The most simple way to do this is by using a technique called the train-test split on our data. \n",
    "\n",
    "You can read more about how and when to use the train-test split at the following links: \n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n",
    "\n",
    "https://machinelearningmastery.com/train-test-split-for-evaluating-machine-learning-algorithms/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50c0f06d",
   "metadata": {},
   "source": [
    "So let's import out train-test split helper from sklearn and apply it to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "116f7c3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f2dd6a4",
   "metadata": {},
   "source": [
    "In this case, we want to have a train_size of 80% and a test size of 20%. \n",
    "\n",
    "We also want to shuffle our data before splitting as, without this, we risk creating batch data not representative of the overall dataset.\n",
    "\n",
    "Finally, we use the random_state parameter to control the shuffling applied to the data before the split to ensure a reproducible output across multiple calls of the function. To apply this, pass any integer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "570d19fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(standardized_data, labels, test_size = 0.20, shuffle=True, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6fe23e",
   "metadata": {},
   "source": [
    "Check that the output of this function is what you might expect (number of instances in the training and test sets)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3217a047",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3217a047",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3217a047",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3217a047",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af0e79f6",
   "metadata": {},
   "source": [
    "In this case, we then train the model with the following line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d9b073b",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression(random_state=42).fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c05b472",
   "metadata": {},
   "source": [
    "Next, we want to return the mean accuracy on the given test data and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfa9860d",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = model.score(x_test, y_test) \n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61bf76c",
   "metadata": {},
   "source": [
    "Now, accuracy is not the only metric that can be used to evaluate model performance. \n",
    "\n",
    "Read more about the different metrics that can be used to evaluate model performance including how to calculate them and when to use them: https://towardsdatascience.com/performance-metrics-for-classification-machine-learning-problems-97e7e774a007"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99a271d",
   "metadata": {},
   "source": [
    "For this, we must first use our model to predict the labels of the test feature data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aebb4e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_labels = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f3dff2",
   "metadata": {},
   "source": [
    "We now want to use these predicted labels to derive a confusion matrix. A confusion matrix displays the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).\n",
    "\n",
    "TPs are test results that correctly indicate the presence of a condition or characteristic.\n",
    "\n",
    "TNs are test results that correctly indicate the absence of a condition or characteristic.\n",
    "\n",
    "FPs are test results that incorrectly indicate the presence of a condition or characteristic.\n",
    "\n",
    "FNs are test results that incorrectly indicate the absence of a condition or characteristic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb0c3909",
   "metadata": {},
   "source": [
    "Now let's import the confusion matrix helper function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c2f677c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8679f3",
   "metadata": {},
   "source": [
    "We can then print the confusion matrix using the following line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d3bfc05",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, pred_labels)\n",
    "print(cm)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b30af09e",
   "metadata": {},
   "source": [
    "From this confusion matrix, we can then derive further metrics, including precision, recall, and the F1-score.\n",
    "\n",
    "Precision is the ratio of correctly classified positive instances to the total predicted positive classifications. \n",
    "\n",
    "Recall or sensitivity is the ratio of correctly classified positive instances to the total positive instances.\n",
    "\n",
    "Precision helps us understand how useful results are; however, recall helps us understand how complete the results are.\n",
    "\n",
    "The F1-score balances the two previous scores, being the harmonic mean of precision and recall.\n",
    "\n",
    "*N.B. accuracy is the ratio of correctly classified instances to the total instances.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ab9b319",
   "metadata": {},
   "source": [
    "First, let's import our helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4da8e22a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import mean\n",
    "from numpy import std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7164ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78fd3d16",
   "metadata": {},
   "source": [
    "Now, let's print all these scores.\n",
    "\n",
    "For precision, recall, and the F1-score, you should report the average and standard deviation of these scores. This is because the precision, recall, and F1-score is provided for each class in the dataset.\n",
    "\n",
    "However, particularly when working with multiclass data, it is important to understand the performance of the model for each class. For this, we print a classification report, which tells you the precision, recall, and F1-score for each class and support (weighted by number of instances in each class in comparison to total number of instances for each class, respectively)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65f22a5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5d325a",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1 = f1_score(y_test, pred_labels)\n",
    "recall = recall_score(y_test, pred_labels)\n",
    "precision = precision_score(y_test, pred_labels)\n",
    "\n",
    "f1_avg = mean(f1_score(y_test, pred_labels, average=None))\n",
    "recall_avg = mean(recall_score(y_test, pred_labels, average=None))\n",
    "precision_avg = mean(precision_score(y_test, pred_labels, average=None))\n",
    "\n",
    "f1_sd = std(f1_score(y_test, pred_labels, average=None))\n",
    "recall_sd = std(recall_score(y_test, pred_labels, average=None))\n",
    "precision_sd = std(precision_score(y_test, pred_labels, average=None))\n",
    "\n",
    "print('\\nf1:\\t\\t',f1)\n",
    "print('recall\\t\\t',recall)\n",
    "print('precision\\t',precision)\n",
    "\n",
    "print('\\nf1_avg:\\t\\t',f1_avg)\n",
    "print('recall_avg\\t',recall_avg)\n",
    "print('precision_avg\\t',precision_avg)\n",
    "\n",
    "print('\\nf1_sd:\\t\\t',f1_sd)\n",
    "print('recall_sd\\t',recall_sd)\n",
    "print('precision_sd\\t',precision_sd)\n",
    "\n",
    "print('\\n',classification_report(y_test, pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e54d2496",
   "metadata": {},
   "source": [
    "Another metric that can be used to evaluate model performance is the area under the receiver operating characteristic curve (ROC-AUC), which plots the true positive rate (recall) against the false positive rate. The AUC ranges in value from 0 to 1. The closer the score to 1, the greater the number of correct predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b48a87ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6e14cdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(roc_auc_score(y_test, pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2411bed2",
   "metadata": {},
   "source": [
    "Now, even using a train-test split, there is still a risk of overfitting on the test set when tuning the parameters of a model until the estimator performs optimally. In this scenario, knowledge can leak into the model until th evaluation metrics no longer report on generalization performance. \n",
    "\n",
    "One way of solving this is hold yet another aprt of the available dataset out as a so-called validation set, on which an initial evaluation is done. However, the problem with partitioning the available data into three sets is that we drastically reduce the number of samples which can be used for learning the model (a particular issue for small datasets and imbalanced datasets). It also means that the results can depend on any random choice for the pair of train and validation sets.\n",
    "\n",
    "A solution to this and another way of evaluating model performance is to use cross-validation, which removes the need for a validation set.\n",
    "\n",
    "In the basic approach, otherwise known as k-fold cross-validation, the data is split into k folds. The model is trained using k - 1 of the folds as training data and validation on the remaining part of the data. This is repeated for each fold. The performance measures reported by k-fold cross-validation is then the average of the values computed in the loop.\n",
    "\n",
    "Learn more about cross-validation here: https://towardsdatascience.com/5-reasons-why-you-should-use-cross-validation-in-your-data-science-project-8163311a1e79 \n",
    "\n",
    "*N.B the article above includes more information on when it is particularly useful to use cross-validation and the different strategies for doing so.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0a96a4",
   "metadata": {},
   "source": [
    "For this, we must first import the cross_val_predict helper function. This allows us to fit a model to our data and generate cross-validated estimates for each input data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944adf50",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18e26670",
   "metadata": {},
   "source": [
    "We then reset the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cd5831d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression(random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55042d85",
   "metadata": {},
   "source": [
    "And finally, we generate cross-validated estimates for each input data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c09af385",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_pred_labels = cross_val_predict(model, standardized_data, labels, cv=10) # Here we use a 10-fold cross-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3df144",
   "metadata": {},
   "source": [
    "Now all we need to do is output our results as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5d325a",
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = accuracy_score(labels, cv_pred_labels)\n",
    "\n",
    "cm = confusion_matrix(labels, cv_pred_labels)\n",
    "\n",
    "f1 = f1_score(labels, cv_pred_labels)\n",
    "recall = recall_score(labels, cv_pred_labels)\n",
    "precision = precision_score(labels, cv_pred_labels)\n",
    "\n",
    "f1_avg = mean(f1_score(labels, cv_pred_labels, average=None))\n",
    "recall_avg = mean(recall_score(labels, cv_pred_labels, average=None))\n",
    "precision_avg = mean(precision_score(labels, cv_pred_labels, average=None))\n",
    "\n",
    "f1_sd = std(f1_score(labels, cv_pred_labels, average=None))\n",
    "recall_sd = std(recall_score(labels, cv_pred_labels, average=None))\n",
    "precision_sd = std(precision_score(labels, cv_pred_labels, average=None))\n",
    "\n",
    "print('accuracy:\\t', accuracy)\n",
    "\n",
    "print(cm)\n",
    "\n",
    "print('\\nf1:\\t\\t',f1)\n",
    "print('recall\\t\\t',recall)\n",
    "print('precision\\t',precision)\n",
    "\n",
    "print('\\nf1_avg:\\t\\t',f1_avg)\n",
    "print('recall_avg\\t',recall_avg)\n",
    "print('precision_avg\\t',precision_avg)\n",
    "\n",
    "print('\\nf1_sd:\\t\\t',f1_sd)\n",
    "print('recall_sd\\t',recall_sd)\n",
    "print('precision_sd\\t',precision_sd)\n",
    "\n",
    "print('\\n',classification_report(labels, cv_pred_labels))\n",
    "\n",
    "print(roc_auc_score(labels, cv_pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "664cb551",
   "metadata": {},
   "source": [
    "And that's it! You have now learnt all about classification."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331de788",
   "metadata": {},
   "source": [
    "#### Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b325901",
   "metadata": {},
   "source": [
    "Regression involves predicting a continuous-valued attribute associated with an object."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed69c4c6",
   "metadata": {},
   "source": [
    "For this example, you are going to use the sci-kit learn dataset diabetes.\n",
    "\n",
    "More on this dataset can be found here: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html#sklearn.datasets.load_diabetes "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10f0b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "data = datasets.load_diabetes(as_frame=True) # this loads the dataset as a dictionary\n",
    "features = data.data # this derives features as a dataframe (10 features by 442 instances)\n",
    "labels = data.target # this derives a continuous-valued attribute as a dataframe (442 instances)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae0f399",
   "metadata": {},
   "source": [
    "Let's just do a quick check to make sure everything has loaded as it should."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3255149",
   "metadata": {},
   "outputs": [],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddadf8df",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ca51fc5",
   "metadata": {},
   "source": [
    "You might think the features dataframe looks a bit odd. This is because each of the 10 feature variables have already been mean centred and scaled by the product of the standard deviation and number of samples (i.e. the sum of squares of each column totals 1) to bring them to a common scale."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a1d58ef",
   "metadata": {},
   "source": [
    "First, let's check the linear correlation between variables in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb25bfcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([features, labels], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d54713e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = df.corr()\n",
    "\n",
    "# visualize the correlation matrix\n",
    "\n",
    "plt.subplots(figsize=(7,7))\n",
    "sns.heatmap(corr, cmap= 'RdPu', annot=True) # plots correlation matrix as a heatmap with values\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "16f0a8c5",
   "metadata": {},
   "source": [
    "This is the point at which multicolinear variables (indepdendent variables that are significantly correlated with one another) can be removed for feature selection, but we are just going to focus on demonstrating the methodology behing regression models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "078c98d9",
   "metadata": {},
   "source": [
    "For now, we are just going to use a simple train-test split to evaluate model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39bdf523",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size = 0.20, shuffle=True, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00c8c45c",
   "metadata": {},
   "source": [
    "For this example, we are going to use linear regression.\n",
    "\n",
    "Learn more about linear regression here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c28cd89c",
   "metadata": {},
   "source": [
    "Let's import it now!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "549cb40b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "model = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f880c6",
   "metadata": {},
   "source": [
    "Now we want to fit the model using all the feature variables and the dependent variable and use this model to predict values of the dependent variable based on the test feature variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1af7b40",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5550a4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_y = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27846615",
   "metadata": {},
   "source": [
    "Finally, let's evaluate the model.\n",
    "\n",
    "To speed things up, we can import metrics from sklearn using the code in the next cell.\n",
    "\n",
    "Learn more about the different metrics (what they measure and when to use them) here: https://scikit-learn.org/stable/modules/model_evaluation.html "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2779dd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics as mt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80631692",
   "metadata": {},
   "source": [
    "We are going to look at three measures:\n",
    "- the coefficient of determination or R² score (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score)\n",
    "- the mean absolute error or MAE (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error)\n",
    "- the mean squared error or MSE (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f28de59",
   "metadata": {},
   "source": [
    "The R² score represents the proportion of the variance that has been explained by the independent variables of the model. It provides an indication of goodness of fit, with the best possible score being 1.0.\n",
    "\n",
    "The MAE computes a risk metric corresponding to the expected value of the absolute error loss.\n",
    "\n",
    "The MSE computes a risk metric corresponding to the expected value of the squared error or loss. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea471697",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The R² score of the model is\", np.round(mt.r2_score(y_test,pred_y),2))\n",
    "print(\"The Mean Absolute Error of model is\", np.round(mt.mean_absolute_error(y_test,pred_y),2))\n",
    "print(\"The Mean Squared Error of the model is\" , np.round(mt.mean_squared_error(y_test,pred_y),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf3268b8",
   "metadata": {},
   "source": [
    "Finally, we want to determine the coefficients and interecept of the regression equation as calculated by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "651dc4e6",
   "metadata": {},
   "source": [
    "Regression coefficients tell you two things:\n",
    "\n",
    "1) Whether there was a positive or negative correlation \n",
    "between the independent variable and the dependent variable.\n",
    "\n",
    "*If there is a positive correlation, as the value of the independent variable increases, the dependent variable also tends to increase. If there is a negative correlation, as the value of the independent variable increases, the dependent variable tends to decrease.*\n",
    "\n",
    "2) The size of the effect of the independent variable on the dependent variable.\n",
    "\n",
    "*The coefficient value signifies how much the mean of the dependent variable changes given a unit shift in the independent variable while all other variables are held constant.*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "03f76e21",
   "metadata": {},
   "source": [
    "The intercept represents the mean value of the dependent variable when all of the independent variables in the model are equal to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c050816c",
   "metadata": {},
   "outputs": [],
   "source": [
    "coef = pd.Series(model.coef_, index=x_train.columns) # derives the coefficient for each variable\n",
    "coef"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28c4a99e",
   "metadata": {},
   "outputs": [],
   "source": [
    "intercept = model.intercept_ # derives the model intercept\n",
    "intercept"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7db4488",
   "metadata": {},
   "source": [
    "The regression equation as calculated by the model would be as follows:\n",
    "\n",
    "Diabetes Progression = intercept + coef(1) x age + coef(2) x sex + coef(3) x bmi + coef(4) x bp + coef(5) x s1 + coef(6) x s2 + coef(7) x s3 + coef(8) x s4 + coef(9) x s5 + coef(10) x s6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b8253d",
   "metadata": {},
   "source": [
    "### Unsupervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9618181b",
   "metadata": {},
   "source": [
    "Unsupervised learning uses machine learning algorithms to analzye and cluster unlabelled datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331de788",
   "metadata": {},
   "source": [
    "#### Clustering"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9618181b",
   "metadata": {},
   "source": [
    "Clustering is the automatic grouping of similar objects into sets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed69c4c6",
   "metadata": {},
   "source": [
    "For this example, you are going to use the sci-kit learn Iris flower dataset containing a total of 150 samples across 3 classes (species of Iris) for which four features (sepal length, sepal width, petal length, and petal width) were measured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10f0b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.load_iris(as_frame=True) # this loads the dataset as a dictionary\n",
    "features = data.data # this derives features as a dataframe (4 features by 150 instances)\n",
    "labels = data.target # this derives labels as a dataframe (150 instances)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15f0479d",
   "metadata": {},
   "source": [
    "*N.B. We added two lines of code above to rename the columns to make our visualisations neater and easier to code later on.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7cc06bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "features"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "256e925f",
   "metadata": {},
   "source": [
    "*N.B. We will also standardize the feature data. The feature variables are all measured in cm, meaning they share a common scale. In this case, standardizing the data may not have much effect, but it is still good practice.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89b04426",
   "metadata": {},
   "outputs": [],
   "source": [
    "standardized_data = pd.DataFrame(scaler.fit_transform(features), columns = ['Sepal Length (cm)', 'Sepal Width (cm)', 'Petal Length (cm)', 'Petal Width (cm)'])\n",
    "standardized_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28947590",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c89f4d4",
   "metadata": {},
   "source": [
    "Now, before we look at clustering this dataset, we want to understand the relationship between each variable and every other variable within the dataset. To do this, we use the function sns.pairplot()."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b735327f",
   "metadata": {},
   "source": [
    "First, we need to concatenate the data (features and labels) into one dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "333d1a1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.concat([standardized_data, labels], axis=1)\n",
    "iris.rename(columns = {'target':'Class'}, inplace=True)\n",
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4219a37",
   "metadata": {},
   "source": [
    "Next, we plot the pairplot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70bcab9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,12))\n",
    "g = sns.pairplot(data=iris, hue=\"Class\", size=3)\n",
    "sns.move_legend(g, \"lower center\", bbox_to_anchor=(0.5, -0.035), ncol=3, title='Class', frameon=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d5aa97",
   "metadata": {},
   "source": [
    "For this example, we are going to use K-Means Clustering.\n",
    "\n",
    "This is a distance-based algorithm.\n",
    "\n",
    "In K-Means Clustering, k (any random whole number) data points can be assigned as a centroid. All other points are then assigned to the cluster of the closest centroid based on distance. At this point, the centroids of the clusters are recomputed.\n",
    "\n",
    "This exercise is performed in a loop to find updated cluster centers and allocate each observation.\n",
    "\n",
    "Learn more about K-Means Clustering and other types of clustering methods here: https://scikit-learn.org/stable/modules/clustering.html#k-means\n",
    "\n",
    "Learn more about how to use K-Means Clustering function here: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f99f6ae",
   "metadata": {},
   "source": [
    "Now, let's import KMeans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc1d4c83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b9e6f2e",
   "metadata": {},
   "source": [
    "Next, we want to find the optimum number of clusters for k-means classification. The KMeans algorithm clusters data by minimizing what is known as the inertia (the within-cluster sum-of-squares). \n",
    "\n",
    "To find the optimum number of clusters, we must first calculate the within-cluster sum-of-squares. We do this using only the features. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c90196ea",
   "metadata": {},
   "source": [
    "For this to work, our feature data must be in array form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac182aa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_array = standardized_data.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96cef20a",
   "metadata": {},
   "outputs": [],
   "source": [
    "wcss = []\n",
    "\n",
    "for k in range(1, 11): # sets k in the range of 1 - 11\n",
    "    kmeans = KMeans(n_clusters = k, init = 'k-means++', max_iter = 300, n_init = 10, random_state = 42) \n",
    "    kmeans.fit(feat_array)\n",
    "    wcss.append(kmeans.inertia_)\n",
    "\n",
    "# max_iter is the number of iterations of the k-means algorithm in a single run (number of times the centroid is recomputed)\n",
    "# the n_init is the number of times the k-means algorithm is run with different centroid seeds"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a750fe96",
   "metadata": {},
   "source": [
    "We then use something known as the elbow method to determine the optimal number of clusters for k-means clustering. \n",
    "\n",
    "Here, we know that there should be three clusters, but otherwise we pick the 'elbow' of the curve as the optimum number of clusters, i.e. where adding another cluster does not allow for much better modelling of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d31b0654",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(7,5))\n",
    "plt.plot(range(1, 11), wcss, color='indigo')\n",
    "plt.title('The Elbow Method')\n",
    "plt.xlabel('Number of Clusters')\n",
    "plt.ylabel('WCSS') #within cluster sum of squares\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bb07f1d",
   "metadata": {},
   "source": [
    "Now we've found the optimum number of clusters, we can implement K-Means clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71ac54fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters = 3, init = 'k-means++', max_iter = 300, n_init = 10, random_state = 42)\n",
    "y_kmeans = kmeans.fit_predict(feat_array) # predict the cluster index of each sample"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b80b48f0",
   "metadata": {},
   "source": [
    "We can then visualise the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad0a5b60",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,axes = plt.subplots(figsize=(7,5))\n",
    "\n",
    "# visualising the clusters \n",
    "plt.scatter(feat_array[y_kmeans == 0, 0], feat_array[y_kmeans == 0, 1], s = 100, c = 'indigo', label = 'Cluster 0')\n",
    "plt.scatter(feat_array[y_kmeans == 1, 0], feat_array[y_kmeans == 1, 1], s = 100, c = 'purple', label = 'Cluster 1')\n",
    "plt.scatter(feat_array[y_kmeans == 2, 0], feat_array[y_kmeans == 2, 1], s = 100, c = 'pink', label = 'Cluster 2')\n",
    "\n",
    "# plotting the centroids of the clusters\n",
    "plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:,1], s = 100, c = 'red', label = 'Centroids')\n",
    "\n",
    "# plots legend outside of the grid, bottom centre of the figure\n",
    "plt.legend(ncol = 4, bbox_to_anchor=(0.9, -0.1))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5a30277d",
   "metadata": {},
   "source": [
    "From this point, there are two avenues we can explore.\n",
    "\n",
    "Improving our model by running a dimensionality algorithm such as Principal Component Analysis (PCA) prior to applying the clustering algorithm. Such an approach reduces the number of dimensions by deriving a select number of feature variables, meaning we can visualize the dataset in fewer dimensions, leading to an interpretation to determine if different groups are well-separated.\n",
    "\n",
    "However, as the newly created feature variables aren't directly comparable to the original feature variables, while running a PCA can help us, it doesn't help to communicate our findings to others.\n",
    "\n",
    "It is very important for us to be able to describe our results, so the remainder of this tutorial will focus on explaining our algorithm. \n",
    "\n",
    "But if you'd like to learn more about PCA, this article provides a good overview (with examples): https://towardsdatascience.com/pca-using-python-scikit-learn-e653f8989e60 "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7c326fda",
   "metadata": {},
   "source": [
    "So, we want to understand how our samples cluster, and the easiest way to do this is to use the mean of their characteristics."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7c326fda",
   "metadata": {},
   "source": [
    "For these next visualisations, we need to install and import plotly, an open-source plotting library in python.\n",
    "\n",
    "Learn more here: https://plotly.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "190b4ecf",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user plotly -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4eae06e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly as plotly\n",
    "import plotly.express as px"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "79a312ee",
   "metadata": {},
   "source": [
    "Now, we want to visualise the mean of each feature variable by cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e79f05f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans.fit(feat_array) # fit the model \n",
    "clusters = standardized_data.copy()\n",
    "clusters['Cluster'] = kmeans.labels_ # assigns cluster labels to features\n",
    "polar = clusters.groupby(\"Cluster\").mean().reset_index() # finds the mean of each feature variable by cluster\n",
    "polar = pd.melt(polar, id_vars=[\"Cluster\"])\n",
    "fig = px.line_polar(polar, r=\"value\", theta=\"variable\", color=\"Cluster\", line_close=True, height=400, width=700, color_discrete_sequence=['darkviolet','deeppink','orchid'], template=\"plotly_dark\")\n",
    "fig.update_layout(legend=dict(\n",
    "    yanchor=\"top\",\n",
    "    y=1.05,\n",
    "    xanchor=\"right\",\n",
    "    x=1.05\n",
    "))\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4cb77401",
   "metadata": {},
   "source": [
    "It's also useful to be able to visualise how many samples are in each cluster. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8c3491d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pie = clusters.groupby('Cluster').size().reset_index()\n",
    "pie.columns=['Cluster','Value']\n",
    "fig = px.pie(pie, values='Value', names='Cluster', color='Cluster', category_orders={\"Cluster\": [\"0\", \"1\", \"2\"]}, height=400, width=700, color_discrete_sequence=['darkviolet','deeppink','orchid'], template=\"plotly_dark\")\n",
    "fig.update_layout(legend=dict(\n",
    "    yanchor=\"top\",\n",
    "    y=1.05,\n",
    "    xanchor=\"right\",\n",
    "    x=1.05\n",
    "))\n",
    "fig.show()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f2bed662",
   "metadata": {},
   "source": [
    "From these plots, we can understand the characteristics of each cluster of irises. \n",
    "\n",
    "For example, if you look at Cluster 1, you can understand that the petal width, petal length, and sepal length of these irises were much smaller than in the other two clusters but that they're sepal width was a lot larger.\n",
    "\n",
    "Clusters 0 and 2 have a similar shape, with Cluster 0 having consistently larger measurements than Cluster 2."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4426bc04",
   "metadata": {},
   "source": [
    "Finally, you can apply names to your clusters based on these characteristics.\n",
    "\n",
    "- Cluster 0 is \"the common irises\"/\"the large irises\"\n",
    "- Cluster 1 is \"the wide sepal irises\"\n",
    "- Cluster 2 is \"the small irises\"/\"the common irises\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6fc9bfe",
   "metadata": {},
   "source": [
    "And that's it! You've learnt the basics of clustering."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cd194e07",
   "metadata": {},
   "source": [
    "Now you've finished this tutorial, follow the instructions and complete the assessment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.8 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8 (default, Apr 13 2021, 12:59:45) \n[Clang 10.0.0 ]"
  },
  "vscode": {
   "interpreter": {
    "hash": "730e964555fb2542a911b5002c3c4a5ea6b8ea7e74d00811d465953d33b870ef"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}