Week6/IntroductionToMachineLearning/MachineLearningExample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Example\n",
    "\n",
    "Python macro for classifying events (in 2D) using Machine Learning (based on SciKitLearn). The 2D case is chosen, so that the data and solutions can be visually inspected and evaluated. Several methods are illustrated.\n",
    "\n",
    "Note: This exercise includes two additional packages to those originally required (ipywidgets and scikitlearn).\n",
    "\n",
    "***\n",
    "\n",
    "### Authors: \n",
    "- Christian Michelsen (Niels Bohr Institute)\n",
    "- Troels C. Petersen (Niels Bohr Institute)\n",
    "\n",
    "### Date:    \n",
    "- 30-12-2022 (latest update)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from ipywidgets import interactive     # To make plots interactive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And set the parameters of the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = np.random\n",
    "r.seed(42)\n",
    "\n",
    "export_tree = True\n",
    "plot_fisher_discriminant = False\n",
    "\n",
    "test_point = np.array([0, 0.5]).reshape(1, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions:\n",
    "\n",
    "First we define `get_corr_pars` which generate (linearly) correlated numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "def get_corr_pars(mu1, mu2, sig1, sig2, rho12) : \n",
    "\n",
    "    # Transform to correlated random numbers (see Barlow page 42-44):\n",
    "    # This is nothing more than a matrix multiplication written out!!!\n",
    "    # Note that the absolute value is taken before the square root to avoid sqrt(x) with x<0.\n",
    "    theta = 0.5 * np.arctan( 2.0 * rho12 * sig1 * sig2 / ( sig1**2 - sig2**2 ) )\n",
    "    sigu = np.sqrt( np.abs( ((sig1*np.cos(theta))**2 - (sig2*np.sin(theta))**2 ) / ( np.cos(theta)**2 - np.sin(theta)**2) ) )\n",
    "    sigv = np.sqrt( np.abs( ((sig2*np.cos(theta))**2 - (sig1*np.sin(theta))**2 ) / ( np.cos(theta)**2 - np.sin(theta)**2) ) )\n",
    "\n",
    "    u = r.normal(0.0, sigu)\n",
    "    v = r.normal(0.0, sigv)\n",
    "\n",
    "    # Transform into (possibly) correlated random Gaussian numbers x1 and x2:\n",
    "    x1 = mu1 + np.cos(theta)*u - np.sin(theta)*v\n",
    "    x2 = mu2 + np.sin(theta)*u + np.cos(theta)*v\n",
    "\n",
    "    return x1, x2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the function `plot_decision_regions` which plots decision boundaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "def plot_decision_regions(X, y, classifier, resolution=0.02, title=None, fig=None, ax=None):\n",
    "    \n",
    "    # define colors\n",
    "    colors = ('red', 'blue')\n",
    "    cmap = ListedColormap(colors)\n",
    "    \n",
    "    # define signal and background\n",
    "    sig = X[y == 1]\n",
    "    bkg = X[y == 0]\n",
    "    \n",
    "    # compute the decision surface\n",
    "    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution), np.arange(x2_min, x2_max, resolution))\n",
    "    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n",
    "    Z = Z.reshape(xx1.shape)\n",
    "    \n",
    "    # set up the figure\n",
    "    if fig is None and ax is None:\n",
    "        fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n",
    "    \n",
    "    # plot the decision surface and plot individual points on ax[0]\n",
    "    ax[0].contourf(xx1, xx2, Z, alpha=0.2, cmap=cmap)\n",
    "    ax[0].scatter(sig[:, 0], sig[:, 1], s=4, c='blue',  label='sig', alpha=0.3)\n",
    "    ax[0].scatter(bkg[:, 0], bkg[:, 1], s=4, c='red', label='bkg', alpha=0.3)\n",
    "\n",
    "    ax[0].set(xlim=(xx1.min(), xx1.max()), ylim=(xx2.min(), xx2.max()), xlabel='Parameter A', ylabel='Parameter B')\n",
    "    \n",
    "    \n",
    "    # predict and plot the prediction of the test point on ax[0]\n",
    "    z_test = classifier.predict(test_point)[0]\n",
    "    if z_test == 0:\n",
    "        color = 'red'\n",
    "    else:\n",
    "        color = 'blue'\n",
    "    ax[0].scatter(test_point[0,0], test_point[0,1], c='w', s=200, marker='o')\n",
    "    ax[0].scatter(test_point[0,0], test_point[0,1], c=color, s=150, marker='*')\n",
    "   \n",
    "    # set the legend on ax[0]\n",
    "    ax[0].legend()\n",
    "                    \n",
    "    \n",
    "    # set up ax[1]:\n",
    "        \n",
    "    # compute y prediction probabilities:\n",
    "    y_predicted_proba = classifier.predict_proba(X)[:, 1]\n",
    "    \n",
    "    # Compute ROC curve and ROC area\n",
    "    FPR, TPR, _ = roc_curve(y, y_predicted_proba)\n",
    "    roc_auc = auc(FPR, TPR)\n",
    "    \n",
    "    lw = 2\n",
    "\n",
    "    # plot the ROC curve\n",
    "    ax[1].plot(FPR, TPR, color='darkorange', lw=lw, label='ROC curve (area = %0.3f)' % roc_auc)\n",
    "    ax[1].plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "    ax[1].set(xlim=[-0.01, 1.0], ylim=[-0.01, 1.05], xlabel='False Positive Rate', ylabel='True Positive Rate')\n",
    "    ax[1].legend(loc=\"lower right\")\n",
    "  \n",
    "    if title:\n",
    "        ax[0].set(title=title)\n",
    "        ax[1].set(title=title)\n",
    "        \n",
    "    return fig, ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we define `animate_ML_estimator_generator` which takes an estimator, fits it given the specified keywords and plots the decision regions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def animate_ML_estimator_generator(clf, title, X, y, **kwargs): \n",
    "    estimator = clf(**kwargs)\n",
    "    estimator = estimator.fit(X, y)\n",
    "    plot_decision_regions(X, y, classifier=estimator, title=title.capitalize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate and plot data:\n",
    "\n",
    "Set number of data points to generate and parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 10000\n",
    "\n",
    "mu_sig = [0.0, 1.0]\n",
    "sigma_sig = [1.5, 2.0]\n",
    "rho_sig = 0.87\n",
    "\n",
    "mu_bkg = [-2.0, 4.0]\n",
    "sigma_bkg = [1.5, 2.0] \n",
    "rho_bkg = 0.82"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate correlated parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The desired covariance matrix.\n",
    "V_sig = np.array([[sigma_sig[0]**2, sigma_sig[0]*sigma_sig[1]*rho_sig],\n",
    "                [sigma_sig[0]*sigma_sig[1]*rho_sig, sigma_sig[1]**2]])\n",
    "V_bkg = np.array([[sigma_bkg[0]**2, sigma_bkg[0]*sigma_bkg[1]*rho_bkg],\n",
    "                [sigma_bkg[0]*sigma_bkg[1]*rho_bkg, sigma_bkg[1]**2]])\n",
    "\n",
    "# Generate the random samples.\n",
    "sig = np.random.multivariate_normal(mu_sig, V_sig, size=N)\n",
    "bkg = np.random.multivariate_normal(mu_bkg, V_bkg, size=N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "Plot the generated random numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "Nbins = 100\n",
    "\n",
    "fig, ax = plt.subplots(1, 4, figsize=(18, 5))\n",
    "\n",
    "ax[0].hist(sig[:, 0], Nbins, histtype='step', label='sig', color='blue')\n",
    "ax[0].hist(bkg[:, 0], Nbins, histtype='step', label='bkg', color='red')\n",
    "ax[0].set(xlabel='Variable A', ylabel='Counts', title='Histogram of Variable A')\n",
    "ax[0].legend()\n",
    "\n",
    "ax[1].hist(sig[:, 1], Nbins, histtype='step', label='sig', color='blue')\n",
    "ax[1].hist(bkg[:, 1], Nbins, histtype='step', label='bkg', color='red')\n",
    "ax[1].set(xlabel='Variable B', ylabel='Counts', title='Histogram of Variable B')\n",
    "ax[1].legend()\n",
    "\n",
    "ax[2].scatter(sig[:, 0], sig[:, 1], s=4, c='blue',  label='sig', alpha=0.5)\n",
    "ax[2].scatter(bkg[:, 0], bkg[:, 1], s=4, c='red', label='bkg', alpha=0.5)\n",
    "ax[2].set(xlabel='Variable A', ylabel='Variable B', title='Scatterplot of Variable A and B')\n",
    "ax[2].legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fishers Discriminant: \n",
    "How to perform a __[Fishers Linear Discimant Analysis](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html)__ in Scikit Learn.  \n",
    "\n",
    "\n",
    "\n",
    "First load the proper package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the two numpy arrays one big one called `X`. `y` denotes the class (1 for signal, 0 for background)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.vstack((sig, bkg))\n",
    "\n",
    "y = np.zeros(2*N) \n",
    "y[:N] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now do the Linear Discriminant Analysis (LDA):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise the LDA method\n",
    "sklearn_lda = LDA()\n",
    "\n",
    "# fit the data\n",
    "sklearn_lda.fit(X, y)\n",
    "\n",
    "# transform the data\n",
    "X_lda_sklearn = sklearn_lda.transform(X) \n",
    "\n",
    "print(f\"LDA coefficients: {sklearn_lda.scalings_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the tranformed variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sig_lda = X_lda_sklearn[y == 1]\n",
    "bkg_lda = X_lda_sklearn[y == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and plot it on the figure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax[3].hist(sig_lda, Nbins, histtype='step', label='sig', color='red')\n",
    "ax[3].hist(bkg_lda, Nbins, histtype='step', label='bkg', color='blue')\n",
    "ax[3].set(xlabel='Fisher Discriminant', ylabel='Counts', title='Fishers discriminant')\n",
    "ax[3].legend()\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What if the dataset was completely different?\n",
    "\n",
    "Now we want to eksamine the dataset given in `DataSet_ML.txt`. First we load it, extract the relevant data and plot it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data:\n",
    "data = np.loadtxt('DataSet_ML.txt')\n",
    "N = len(data)\n",
    "Nbins = 50\n",
    "\n",
    "# Divide data into input variables (X) and \"target\" variable (y)\n",
    "X = data[:, :2]\n",
    "y = data[:, 2]\n",
    "\n",
    "# As signal and background:\n",
    "sig = X[y == 1]\n",
    "bkg = X[y == 0]\n",
    "\n",
    "fig2, ax2 = plt.subplots(1, 3, figsize=(20, 8))\n",
    "ax2[0].hist(sig[:, 0], Nbins, histtype='step', label='sig', color='blue')\n",
    "ax2[0].hist(bkg[:, 0], Nbins, histtype='step', label='bkg', color='red')\n",
    "ax2[0].set(xlabel='Variable A', ylabel='Counts', title='Histogram of Variable A')\n",
    "ax2[0].legend()\n",
    "\n",
    "ax2[1].hist(sig[:, 1], Nbins, histtype='step', label='sig', color='blue')\n",
    "ax2[1].hist(bkg[:, 1], Nbins, histtype='step', label='bkg', color='red')\n",
    "ax2[1].set(xlabel='Variable B', ylabel='Counts', title='Histogram of Variable B')\n",
    "ax2[1].legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TASK:\n",
    " 1. Think about how you think the above data looks in 2D before you continue.\n",
    " 2. Draw on a piece of paper, what you think it looks like in 2D before you continue.\n",
    " \n",
    "When you have talked with your collaborators you should uncomment the below five lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ax2[2].scatter(sig[:, 0], sig[:, 1], s=4, c='blue',  label='sig', alpha=0.5)\n",
    "# ax2[2].scatter(bkg[:, 0], bkg[:, 1], s=4, c='red', label='bkg', alpha=0.5)\n",
    "# ax2[2].set(xlabel='Variable A', ylabel='Variable B', title='Scatterplot of Variable A and B')\n",
    "# ax2[2].legend()\n",
    "# fig2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactive Machine Learning Part\n",
    "\n",
    "In this part we will further investigate different standard ML models.  \n",
    "\n",
    "## Linear Fisher Discriminant:\n",
    "\n",
    "First we look at __[Linear Fisher Discriminant](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html)__. This is the same as we saw above so we will go through this example quite quickly (also because in 2D there are no interesting hyperparameters to change)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LDA\n",
    "clf_fisher = LDA()                      # Initialise the LDA method\n",
    "clf_fisher.fit(X, y)                    # Fit the data\n",
    "X_fisher = clf_fisher.transform(X)      # Transform the data\n",
    "print(\"LDA coefficients\", clf_fisher.scalings_)\n",
    "\n",
    "# Extract the tranformed variables\n",
    "sig_fisher = X_fisher[y == 1]\n",
    "bkg_fisher = X_fisher[y == 0]\n",
    "\n",
    "if plot_fisher_discriminant :        # You gotta switch this on, if you want to see it :-)\n",
    "\n",
    "    # Plot decision region of fisher:\n",
    "    fig_fisher, ax_fisher = plot_decision_regions(X, y, classifier=clf_fisher, title='Fisher')\n",
    "\n",
    "    # Plot fisher discriminant values:\n",
    "    fig_fisher2, ax_fisher2 = plt.subplots(figsize=(13.3, 6))\n",
    "    ax_fisher2.hist(sig_fisher, Nbins, histtype='step', label='sig', color='blue')\n",
    "    ax_fisher2.hist(bkg_fisher, Nbins, histtype='step', label='bkg', color='red')\n",
    "    ax_fisher2.set(xlabel='Fisher Discriminant', ylabel='Counts', title='Fishers discriminant')\n",
    "    ax_fisher2.legend()\n",
    "    fig_fisher2.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Trees\n",
    "\n",
    "We load __[Decision Trees](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)__ from Scikit-learn (sklearn). At first try to increase the `max_depth` slider and see how that affects the plots. Does that make sense? For a given `max_depth`, e.g. `max_depth = 30`, change the `min_samples_leaf` and see how it simplifies (via _regularization_) the model. Think about when you'd want a simpler model. Finally, given a set of values for `max_depth` and `min_samples_leaf` switch between `criterion` being `gini` and `entropy`. Does this change much? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "def animate_ML_estimator_DT(criterion, min_samples_leaf=1, max_depth=1):\n",
    "    animate_ML_estimator_generator(DecisionTreeClassifier, 'Decision Tree', X, y, \n",
    "                                   max_depth=max_depth, \n",
    "                                   criterion=criterion,\n",
    "                                   #splitter=splitter,\n",
    "                                   #min_samples_split=min_samples_split, \n",
    "                                   min_samples_leaf=min_samples_leaf)\n",
    "\n",
    "kwargs_DT = { 'max_depth': (1, 50), \n",
    "              'criterion': [\"gini\", \"entropy\"], \n",
    "              #'splitter': [\"best\", \"random\"], \n",
    "              #'min_samples_split': (2, 50),\n",
    "              'min_samples_leaf': (1, 50),\n",
    "            }    \n",
    "\n",
    "\n",
    "interactive_plot = interactive(animate_ML_estimator_DT, **kwargs_DT)\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boosted Decision Trees (BDTs)\n",
    "\n",
    "For __[BDTs](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier)__ try to slowly increase `n_estimators` and see how it affects the model. Does it make sense? Try also to increase the learning rate and see what that changes:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "def animate_ML_estimator_BDT(learning_rate=1., n_estimators=1):\n",
    "    animate_ML_estimator_generator(AdaBoostClassifier, 'Boosted Decision Trees', X, y, \n",
    "                                   learning_rate=learning_rate, \n",
    "                                   n_estimators=n_estimators,\n",
    "                                   )\n",
    "\n",
    "kwargs_BDT = {'learning_rate': (0.01, 2, 0.01), \n",
    "              'n_estimators': (1, 100),\n",
    "            }    \n",
    "\n",
    "interactive_plot = interactive(animate_ML_estimator_BDT, **kwargs_BDT)\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## k-Nearest Neighbours\n",
    "\n",
    "For __[kNNs](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)__ we only have the parameter n_neighbors to change:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "def animate_ML_estimator_KNN(n_neighbors=1):\n",
    "    animate_ML_estimator_generator(KNeighborsClassifier, 'KNN', X, y, n_neighbors=n_neighbors)\n",
    "\n",
    "kwargs_KNN = {'n_neighbors': (1, 50), \n",
    "            }    \n",
    "\n",
    "interactive_plot = interactive(animate_ML_estimator_KNN, **kwargs_KNN)\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machine (SVM):\n",
    "\n",
    "__[SVms](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)__ are quite different ML models than the rest and we won't be going through it in much detail. However, see if you can make sense of the difference between the `RBF` kernel compared to the `poly` one. Also, how much does `C` matter for the different kernels?\n",
    "Notice that `degree` is only relevant for the `poly` kernel.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "def animate_ML_estimator_SVM(kernel='rbf', C=1, degree=2, ):\n",
    "    animate_ML_estimator_generator(SVC, 'SVM', X, y, \n",
    "                                   #gamma=gamma,\n",
    "                                   gamma='scale',\n",
    "                                   kernel=kernel, \n",
    "                                   C=C,\n",
    "                                   \n",
    "                                   probability=True,\n",
    "                                   degree=degree,\n",
    "                                   )\n",
    "\n",
    "kwargs_SVM = {'C': (0.1, 10),\n",
    "              'kernel': ['poly', 'rbf'],\n",
    "              'degree': (1, 10),\n",
    "              #'gamma': (0.1, 10),\n",
    "            }    \n",
    "\n",
    "interactive_plot = interactive(animate_ML_estimator_SVM, **kwargs_SVM)\n",
    "interactive_plot\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    " Machine Learning (ML) is a fascinating subject, which is very much in vouge these days.\n",
    " There are two classic usages for ML:\n",
    " \n",
    "  - Classification (determine which category a case belongs to, i.e. ill or healthy)\n",
    "  - Regression (determine a value, i.e. what was the energy of this electron)\n",
    "\n",
    " Python has packages - \"scikit-learn\" (among others) - that allows anybody to  easily apply ML to smaller scale problems (i.e. below 1 GB). The following questions/exercise is meant to illustrate a classification problem, and to wet your appetite for more...\n",
    "\n",
    "\n",
    "# Questions:\n",
    "\n",
    "1. Consider the data, and make sure that you by eye can see, how an algorithm should decide between the two categories. Also see, if you can guess how well the Fisher performs. What is the error rate of type1 and type2 roughly?\n",
    "\n",
    "2. Now consider the various ML algorithms. Can you (again by eye) tell, if it is doing \"just well\" or \"very well\"? I.e. can you rank the methods by eye? What you have tried this, compare to the ROC curves and see how well you did.\n",
    "\n",
    "3. What we are currently plotting in the ROCcurve is the ROC-curve for the training data. Try to split up the data into a training and a test set: Let 2/3 be for training, and then apply the result of this training on the last 1/3 of the data, which is thus \"unseen\" by the algorithm, corresponding to \"new data\". Then add the ROC-curve of the test set to the `plot_decision_regions` function.\n",
    "\n",
    "4. Try to put all the ROC curves into one final plot, which shows how well the different methods perform.\n",
    "\n",
    "\n",
    "### Advanced questions:\n",
    "\n",
    "5. Try to find some other data, and apply the above ML methods to it. Can you get it to run there, and does it perform better than other methods you can implement yourself?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning points:\n",
    "\n",
    "The exercise is mostly meant for illustration, and for giving the opportunity to play around.\n",
    "\n",
    "From this exercise you should:\n",
    "1. Have an idea of how Machine Learning (ML) manages to work in many dimensions in a non-linear way.\n",
    "2. Know about how Tree-based and Neural Network-based algorithms work.\n",
    "3. That you don't code things up from scratch, but use packages for ML."
   ]
  }
 ],
 "metadata": {
  "executable": "/usr/bin/env python",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "main_language": "python"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}