4-analysis/4-3-machine-learning-with-sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### GESIS Fall Seminar in Computational Social Science 2022\n",
    "### Introduction to Computational Social Science with Python\n",
    "# Day 4-3: Machine Learning with sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "* Machine learning (a very brief intro)\n",
    "* Scikit-learn\n",
    "* Training data vs test data\n",
    "* Random forests\n",
    "* Feature importance\n",
    "* Hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Machine learning (a very brief intro)\n",
    "* The use of algorithms to \"learn\" patterns in large datasets and make predictions, without being explicitly programmed.\n",
    "* Contrast it with traditional programming, where we manually code structures like if/else statements.\n",
    "* Supervised vs unsupervised learning:\n",
    "    * Supervised - supply computer with labelled data, computer learns by comparing output against labels.\n",
    "    * Unsupervised - no labels provided, computer must identify emergent patterns itself.\n",
    "* Can be used for regression, classification, clustering tasks, many many more applications.\n",
    "* It's not magic, it's just statistics!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Supervised Learning](figs/supervised.png \"Supervised_Learning\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scikit-learn\n",
    "* Popular machine learning (ML) library for Python.\n",
    "* Vast functionality for processing data, building unsupervised and supervised ML models, scoring models.\n",
    "* Integrates well with pandas, NumPy, matplotlib.\n",
    "* More cutting-edge ML research is done using libraries such as TensorFlow, PyTorch, and Keras, but sklearn still forms the basis of much ML work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sklearn\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training data vs test data\n",
    "* We must randomly split our full dataset into data the ML model trains on, and the test data we evaluate the model on.\n",
    "* The model will learn the patterns in the training data that are associated with a particular label (supervised learning).\n",
    "* Our data consists of observations with **features** (independent / explanatory variables, \"X\") and a **target** variable (dependent variable, label, \"y\").\n",
    "* We can use sklearn's `train_test_split` to get our training features, testing features, training targets, and testing targets.\n",
    "* NEVER TRAIN YOUR MODEL ON YOUR TEST DATA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Load our dataset about passengers aboard the Titanic.\n",
    "# Based on data available here: https://www.kaggle.com/competitions/titanic/overview\n",
    "# We want to predict whether someone died or survived based on features like age, sex, etc\n",
    "\n",
    "data = pd.read_csv('data/titanic.csv')\n",
    "\n",
    "# Some info about the columns:\n",
    "# 'PassengerId': Unique passenger ID\n",
    "# 'Name': Passenger name\n",
    "# 'Pclass': Ticket class (1st=1, 2nd=2, 3rd=3)\n",
    "# 'Sex': Male=1, Female=0\n",
    "# 'Age': Age in years\n",
    "# 'SibSp': Number of siblings/spouses also aboard the Titanic\n",
    "# 'ParCh': Number of parents/children also aboard the Titanic\n",
    "# 'Fare': Ticket price\n",
    "# 'Embarked_C': Embarked in Cherbourg, Yes=1, No=0\n",
    "# 'Embarked_Q': Embarked in Queenstown, Yes=1, No=0\n",
    "# 'Embarked_S': Embarked in Southampton, Yes=1, No=0\n",
    "# 'Survived': Yes=1, No=0 (The target variable)\n",
    "\n",
    "# preview the data\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split dataset into features and target\n",
    "X = data[['Sex', 'Age', 'Embarked_C', 'Embarked_Q', 'Embarked_S']]  # Let's build a model only based on passenger embarkation, sex, and age\n",
    "y = data['Survived']        \n",
    "\n",
    "# Split dataset into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.70) # 70% training and 30% test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Random forests\n",
    "* Supervised model, can be used for classifcation and regression.\n",
    "* An \"ensemble\" method - it takes the average or aggregated result of many \"decision trees\".\n",
    "* Advantages:\n",
    "    - Versatile \"out of the box\" ML method.\n",
    "    - Handles non-linear and non-scaled data well.\n",
    "    - Tends to resist overfitting.\n",
    "    - Feature importance metrics enable better interpretability than black-box models.\n",
    "* Disadvantages:\n",
    "    - Less interpretable than a single decision tree.\n",
    "    - More sophisticated, better performing models often available (e.g. gradient boosted trees)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Analogy\n",
    "* You want to know if your expensive new shoes are fashionable, so you ask a group of friends. Each friend is a \"decision tree\" that has their own rules about what they think is fashionable, based on their knowledge of shoe fashion and features like colour, material, style. Together, your friends form a \"random forest\" of decision trees, and you take the majority opinion from them on whether the shoes are fashionable or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "An example of a decision tree:\n",
    "\n",
    "![Decision_Tree](figs/Decision_Tree.jpg \"Decision_Tree\")\n",
    "\n",
    "A full forest of decision trees is automatically generated by sampling random subsets of our training data, and creating a tree for each subset that best classifies its respective data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "#Create a Random Forest Classifier with 5 trees\n",
    "clf = RandomForestClassifier(n_estimators=5)\n",
    "\n",
    "#Train the model using the training set (.fit(), .predict(), .score() are the main methods to know)\n",
    "clf.fit(X_train, y_train)\n",
    "training_score = clf.score(X_train, y_train)\n",
    "\n",
    "# prediction on test set\n",
    "y_pred = clf.predict(X_test)\n",
    "testing_score = clf.score(X_test, y_test)\n",
    "\n",
    "print(training_score, testing_score) # print accuracy (proportion correctly predicted)\n",
    "pd.DataFrame({'Test':y_test, 'Predicted':y_pred})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature importance\n",
    "* Machine learning models frequently receive criticism for their uninterpretability / unexplainability.\n",
    "    - What features contributed to the machine's decision? What would need to change in order to be classifed as x rather than y?\n",
    "    - No neat mathematical formula like a simple linear regression model.\n",
    "* With random forests, it is possible to get a measure of what features contribute most to the classification.\n",
    "* Sometimes the removal of unimportant features can improve model accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "feature_imp = pd.Series(clf.feature_importances_, index=X_train.columns).sort_values(ascending=False)\n",
    "feature_imp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Hyper-parameter tuning\n",
    "* Not every machine learning algorithm has a 'one size fits all' solution to every problem.\n",
    "* Machine learning methods can vary in success depending on methodological choices of train/test split, relevant features, algorithm hyper-parameter values.\n",
    "* Good to make informed choices and experiment with differently specified classifiers/regressors, but beware of overfitting!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Let's fit 2 models with a different number of trees\n",
    "\n",
    "for n_trees in [5, 100]:\n",
    "    clf = RandomForestClassifier(n_estimators=n_trees)\n",
    "    clf.fit(X_train, y_train)\n",
    "    train_score = clf.score(X_train, y_train)\n",
    "    test_score = clf.score(X_test, y_test)\n",
    "    print('Model with %d trees: training accuracy = %f. test accuracy = %f.'\n",
    "          %(n_trees, train_score, test_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Overfitting\n",
    "* If we tune hyperparameters incorrectly (such as making the trees too deep) we are in danger of overfitting out model to the training data.\n",
    "* This means that we get very high training accuracy, but lower testing accuracy.\n",
    "* Our model has learnt peculiarities of our training data that do not generalise to the testing data.\n",
    "* We can help prevent this with *cross-validation*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Demonstration of overfitting\n",
    "fit = []\n",
    "for run in range(50): # average over 50 runs\n",
    "    for n in range(1,20):\n",
    "        clf = RandomForestClassifier(n_estimators=10, max_depth=n)\n",
    "        clf.fit(X_train, y_train)\n",
    "        train_score = clf.score(X_train, y_train)\n",
    "        test_score = clf.score(X_test, y_test)\n",
    "        fit.append([n, train_score, test_score])\n",
    "        \n",
    "fitdf = pd.DataFrame(fit, columns=['Depth', 'Train Score', 'Test Score'])\n",
    "fitdf.groupby('Depth').mean().plot()\n",
    "plt.show()\n",
    "\n",
    "# More depth has improved the training score, but worsened the test score!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### K-fold cross validation\n",
    "Cross validation helps us design a model that does not overfit the training data, and can predict labels on test data accurately.\n",
    "1. Take the **training** data, divide it into K equal parts.\n",
    "2. Train a model on K-1 parts of the training data, test it on the remaining part, repeat for each \"train\"/\"test\" combination.\n",
    "3. Take the average of these \"test\" scores as the model training accuracy.\n",
    "\n",
    "Optional:\n",
    "\n",
    "4. Repeat the K-fold cross validation process for models with different hyper-parameters.\n",
    "5. Select the model with the best (averaged) training accuracy, refit on full training data.\n",
    "6. Test this model on the test data for the final test accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![crossval](figs/grid_search_cross_validation.png \"crossval\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Model selection, we look for the model with the highest cross-validated training score\n",
    "for n in [\"sqrt\", \"log2\", None]:\n",
    "    clf = RandomForestClassifier(n_estimators=10, max_features=n)    \n",
    "    scores = cross_val_score(clf, X_train, y_train, cv=5) # 5-fold cross validation\n",
    "    clf.fit(X_train, y_train)\n",
    "    print(n, np.mean(scores))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# We select the best performing hyper-parameters, train again on the *full* training data, and test on our final data\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=10, max_features=\"log2\") # best hyper-parameters from cross validation\n",
    "clf.fit(X_train, y_train) # fit model\n",
    "print(clf.score(X_train, y_train)) # print training/test scores\n",
    "print(clf.score(X_test, y_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Grid Search\n",
    "We have evaluated model score by changing one hyper-parameter, but you may want to change other hyper-parameters, or see how different combinations of hyper-parameters work. You can manually accomplish this with a for loop (like above), nested for loops, or the sklearn [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) function. This performs cross validation on models built with all combinations of hyper-parameters specified. Beware that too many parameter combinations will take a long time to fit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# We want to see performance with varying n_estimators and max_features\n",
    "param_grid = {'n_estimators':[2, 5, 10, 50, 100], 'max_features':[\"sqrt\", \"log2\", None]}\n",
    "\n",
    "clf = RandomForestClassifier() # Create base classfier\n",
    "grid_search = GridSearchCV(clf, param_grid=param_grid, cv=5) # specify the model, search 'grid', and folds\n",
    "grid_search.fit(X_train, y_train) # Fit all the models in the grid search and identify the best\n",
    "\n",
    "# Let's look at the results for the best models\n",
    "pd.DataFrame(grid_search.cv_results_).sort_values('mean_test_score', ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# The grid search automatically refits the model on the full training data after doing cross-validation\n",
    "# We can get our best performing model like so, and score it on the test data\n",
    "\n",
    "bestmodel = grid_search.best_estimator_\n",
    "print(bestmodel.score(X_test, y_test))\n",
    "\n",
    "# This should be better than our initial model accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 🏋️‍♀️ PRACTICE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q1: Attempt to create a random forest classifier that improves on the accuracy of the example above.\n",
    "# You may use different features, different hyper-parameters, but please keep the train/test split below (random state 42)\n",
    "# More about the hyperparameters here:\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html\n",
    "# Show us the (cross-validated) training accuracy, testing accuracy, and feature importances.\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}