05a - Deep Neural Networks (PyTorch).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning with PyTorch\n",
    "\n",
    "Classical machine learning relies on using statistics to determine relationships between features and labels, and can be very effective for creating predictive models. However, a massive growth in the availability of data coupled with advances in the computing technology required to process it has led to the emergence of new machine learning techniques that mimic the way the brain processes information in a structure called an artificial neural network.\n",
    "\n",
    "PyTorch is a framework for creating machine learning models, including deep neural networks (DNNs). In this example, we'll use PyTorch to create a simple neural network that classifies penguins into species based on the length and depth of their culmen (bill), their flipper length, and their body mass.\n",
    "\n",
    "> **Citation**: The penguins dataset used in the this exercise is a subset of data collected and made available by [Dr. Kristen\n",
    "Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php)\n",
    "and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a\n",
    "member of the [Long Term Ecological Research\n",
    "Network](https://lternet.edu/).\n",
    "\n",
    "## Explore the Dataset\n",
    "\n",
    "Before we start using PyTorch to create a model, let's load the data we need from the Palmer Islands penguins dataset, which contains observations of three different species of penguin.\n",
    "\n",
    "> **Note**: In reality, you can solve the penguin classification problem easily using classical machine learning techniques without the need for a deep learning model; but it's a useful, easy to understand dataset with which to demonstrate the principles of neural networks in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# load the training dataset (excluding rows with null values)\n",
    "penguins = pd.read_csv('data/penguins.csv').dropna()\n",
    "\n",
    "# Deep Learning models work best when features are on similar scales\n",
    "# In a real solution, we'd implement some custom normalization for each feature, but to keep things simple\n",
    "# we'll just rescale the FlipperLength and BodyMass so they're on a similar scale to the bill measurements\n",
    "penguins['FlipperLength'] = penguins['FlipperLength']/10\n",
    "penguins['BodyMass'] = penguins['BodyMass']/100\n",
    "\n",
    "# The dataset is too small to be useful for deep learning\n",
    "# So we'll oversample it to triple its size\n",
    "for i in range(1,3):\n",
    "    penguins = penguins.append(penguins)\n",
    "\n",
    "# Display a random sample of 10 observations\n",
    "sample = penguins.sample(10)\n",
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Species** column is the label our model will predict. Each label value represents a class of penguin species, encoded as 0, 1, or 2. The following code shows the actual species to which these class labels corrrespond."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguin_classes = ['Amelie', 'Gentoo', 'Chinstrap']\n",
    "print(sample.columns[0:5].values, 'SpeciesName')\n",
    "for index, row in penguins.sample(10).iterrows():\n",
    "    print('[',row[0], row[1], row[2],row[3], int(row[4]), ']',penguin_classes[int(row[-1])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As is common in a supervised learning problem, we'll split the dataset into a set of records with which to train the model, and a smaller set with which to validate the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "features = ['CulmenLength','CulmenDepth','FlipperLength','BodyMass']\n",
    "label = 'Species'\n",
    "   \n",
    "# Split data 70%-30% into training set and test set\n",
    "x_train, x_test, y_train, y_test = train_test_split(penguins[features].values,\n",
    "                                                    penguins[label].values,\n",
    "                                                    test_size=0.30,\n",
    "                                                    random_state=0)\n",
    "\n",
    "print ('Training Set: %d, Test Set: %d \\n' % (len(x_train), len(x_test)))\n",
    "print(\"Sample of features and labels:\")\n",
    "\n",
    "# Take a look at the first 25 training features and corresponding labels\n",
    "for n in range(0,24):\n",
    "    print(x_train[n], y_train[n], '(' + penguin_classes[y_train[n]] + ')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *features* are the measurements for each penguin observation, and the *label* is a numeric value that indicates the species of penguin that the observation represents (Amelie, Gentoo, or Chinstrap).\n",
    "\n",
    "## Install and import the PyTorch libraries\n",
    "\n",
    "Since we plan to use PyTorch to create our penguin classifier, we'll need to run the following two cells to install and import the PyTorch libraries we intend to use. The specific installation of of PyTorch depends on your operating system and whether your computer has graphics processing units (GPUs) that can be used for high-performance processing via *cuda*. You can find detailed instructions at https://pytorch.org/get-started/locally/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.utils.data as td\n",
    "\n",
    "# Set random seed for reproducability\n",
    "torch.manual_seed(0)\n",
    "\n",
    "print(\"Libraries imported - ready to use PyTorch\", torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the data for PyTorch\n",
    "\n",
    "PyTorch makes use of *data loaders* to load training and validation data in batches. We've already loaded the data into numpy arrays, but we need to wrap those in PyTorch datasets (in which the data is converted to PyTorch *tensor* objects) and create loaders to read batches from those datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a dataset and loader for the training data and labels\n",
    "train_x = torch.Tensor(x_train).float()\n",
    "train_y = torch.Tensor(y_train).long()\n",
    "train_ds = td.TensorDataset(train_x,train_y)\n",
    "train_loader = td.DataLoader(train_ds, batch_size=20,\n",
    "    shuffle=False, num_workers=1)\n",
    "\n",
    "# Create a dataset and loader for the test data and labels\n",
    "test_x = torch.Tensor(x_test).float()\n",
    "test_y = torch.Tensor(y_test).long()\n",
    "test_ds = td.TensorDataset(test_x,test_y)\n",
    "test_loader = td.DataLoader(test_ds, batch_size=20,\n",
    "    shuffle=False, num_workers=1)\n",
    "print('Ready to load data..')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a neural network\n",
    "\n",
    "Now we're ready to define our neural network. In this case, we'll create a network that consists of 3 fully-connected layers:\n",
    "* An input layer that receives an input value for each feature (in this case, the four penguin measurements) and applies a *ReLU* activation function.\n",
    "* A hidden layer that receives ten inputs and applies a *ReLU* activation function.\n",
    "* An output layer that uses a *Softmax* activation function to generate an output for each penguin species (which represent the classification probabilities for each of the three possible penguin species). Softmax functions produce a vector with probability values that sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Number of hidden layer nodes\n",
    "hl = 10\n",
    "\n",
    "# Define the neural network\n",
    "class PenguinNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(PenguinNet, self).__init__()\n",
    "        self.fc1 = nn.Linear(len(features), hl)\n",
    "        self.fc2 = nn.Linear(hl, hl)\n",
    "        self.fc3 = nn.Linear(hl, len(penguin_classes))\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = torch.relu(self.fc2(x))\n",
    "        x = torch.softmax(self.fc3(x),dim=1)\n",
    "        return x\n",
    "\n",
    "# Create a model instance from the network\n",
    "model = PenguinNet()\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "To train the model, we need to repeatedly feed the training values forward through the network, use a loss function to calculate the loss, use an optimizer to backpropagate the weight and bias value adjustments, and validate the model using the test data we withheld.\n",
    "\n",
    "To do this, we'll create a function to train and optimize the model, and function to test the model. Then we'll call these functions iteratively over 50 epochs, logging the loss and accuracy statistics for each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def train(model, data_loader, optimizer):\n",
    "    # Set the model to training mode\n",
    "    model.train()\n",
    "    train_loss = 0\n",
    "    \n",
    "    for batch, tensor in enumerate(data_loader):\n",
    "        data, target = tensor\n",
    "        #feedforward\n",
    "        optimizer.zero_grad()\n",
    "        out = model(data)\n",
    "        loss = loss_criteria(out, target)\n",
    "        train_loss += loss.item()\n",
    "\n",
    "        # backpropagate\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    #Return average loss\n",
    "    avg_loss = train_loss / (batch+1)\n",
    "    print('Training set: Average loss: {:.6f}'.format(avg_loss))\n",
    "    return avg_loss\n",
    "           \n",
    "            \n",
    "def test(model, data_loader):\n",
    "    # Switch the model to evaluation mode (so we don't backpropagate)\n",
    "    model.eval()\n",
    "    test_loss = 0\n",
    "    correct = 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        batch_count = 0\n",
    "        for batch, tensor in enumerate(data_loader):\n",
    "            batch_count += 1\n",
    "            data, target = tensor\n",
    "            # Get the predictions\n",
    "            out = model(data)\n",
    "\n",
    "            # calculate the loss\n",
    "            test_loss += loss_criteria(out, target).item()\n",
    "\n",
    "            # Calculate the accuracy\n",
    "            _, predicted = torch.max(out.data, 1)\n",
    "            correct += torch.sum(target==predicted).item()\n",
    "            \n",
    "    # Calculate the average loss and total accuracy for this epoch\n",
    "    avg_loss = test_loss/batch_count\n",
    "    print('Validation set: Average loss: {:.6f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
    "        avg_loss, correct, len(data_loader.dataset),\n",
    "        100. * correct / len(data_loader.dataset)))\n",
    "    \n",
    "    # return average loss for the epoch\n",
    "    return avg_loss\n",
    "\n",
    "# Specify the loss criteria (CrossEntropyLoss for multi-class classification)\n",
    "loss_criteria = nn.CrossEntropyLoss()\n",
    "\n",
    "# Use an \"Adam\" optimizer to adjust weights\n",
    "# (see https://pytorch.org/docs/stable/optim.html#algorithms for details of supported algorithms)\n",
    "learning_rate = 0.001\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "optimizer.zero_grad()\n",
    "\n",
    "# We'll track metrics for each epoch in these arrays\n",
    "epoch_nums = []\n",
    "training_loss = []\n",
    "validation_loss = []\n",
    "\n",
    "# Train over 50 epochs\n",
    "epochs = 50\n",
    "for epoch in range(1, epochs + 1):\n",
    "\n",
    "    # print the epoch number\n",
    "    print('Epoch: {}'.format(epoch))\n",
    "    \n",
    "    # Feed training data into the model to optimize the weights\n",
    "    train_loss = train(model, train_loader, optimizer)\n",
    "    \n",
    "    # Feed the test data into the model to check its performance\n",
    "    test_loss = test(model, test_loader)\n",
    "    \n",
    "    # Log the metrics for this epoch\n",
    "    epoch_nums.append(epoch)\n",
    "    training_loss.append(train_loss)\n",
    "    validation_loss.append(test_loss)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the training process is running, let's try to understand what's happening:\n",
    "\n",
    "1. In each *epoch*, the full set of training data is passed forward through the network. There are four features for each observation, and four corresponding nodes in the input layer - so the features for each observation are passed as a vector of four values to that layer. However, for efficiency, the feature vectors are grouped into batches; so actually a matrix of multiple feature vectors is fed in each time.\n",
    "2. The matrix of feature values is processed by a function that performs a weighted sum using initialized weights and bias values. The result of this function is then processed by the activation function for the input layer to constrain the values passed to the nodes in the next layer.\n",
    "3. The weighted sum and activation functions are repeated in each layer. Note that the functions operate on vectors and matrices rather than individual scalar values. In other words, the forward pass is essentially a series of nested linear algebra functions. This is the reason data scientists prefer to use computers with graphical processing units (GPUs), since these are optimized for matrix and vector calculations.\n",
    "4. In the final layer of the network, the output vectors contain a probability value for each possible class (in this case, classes 0, 1, and 2). This vector is processed by a *loss function* to determine how far the values calculated by the network are from the actual values - so for example, suppose the output for a Gentoo penguin (class 1) observation is \\[0.3, 0.4, 0.3\\]. The correct prediction should be \\[0.0, 1.0, 0.0\\], so the variance between the predicted and actual values (how far away the each predicted value is from what it should be) is \\[0.3, 0.6, 0.3\\]. This variance is aggregated for each batch and maintained as a running aggregate to calculate the overall level of error (*loss*) incurred by the training data for the epoch. \n",
    "5. At the end of each epoch, the validation data is passed through the network, and its loss and accuracy (proportion of correct predictions based on the highest probability value in the output vector) are also calculated. It's important to do this because it enables us to compare the performance of the model using data on which it was not trained, helping us determine if it will generalize well for new data or if it's *overfitted* to the training data.\n",
    "6. After all the data has been passed forward through the network, the output of the loss function for the *training* data (but <u>not</u> the *validation* data) is passed to the opimizer. The precise details of how the optimizer processes the loss vary depending on the specific optimization algorithm being used; but fundamentally you can think of the entire network, from the input layer to the loss function as being one big nested (*composite*) function. The optimizer applies some differential calculus to calculate *partial derivatives* for the function with respect to each weight and bias value that was used in the network. It's possible to do this efficiently for a nested function due to something called the *chain rule*, which enables you to determine the derivative of a composite function from the derivatives of its inner function and outer functions. You don't really need to worry about the details of the math here (the optimizer does it for you), but the end result is that the partial derivatives tell us about the slope (or *gradient*) of the loss function with respect to each weight and bias value - in other words, we can determine whether to increase or decrease the weight and bias values in order to decrease the loss.\n",
    "7. Having determined in which direction to adjust the weights and biases, the optimizer uses the *learning rate* to determine by how much to adjust them; and then works backwards through the network in a process called *backpropagation* to assign new values to the weights and biases in each layer.\n",
    "8. Now the next epoch repeats the whole training, validation, and backpropagation process starting with the revised weights and biases from the previous epoch - which hopefully will result in a lower level of loss.\n",
    "9. The process continues like this for 50 epochs.\n",
    "\n",
    "## Review training and validation loss\n",
    "\n",
    "After training is complete, we can examine the loss metrics we recorded while training and validating the model. We're really looking for two things:\n",
    "* The loss should reduce with each epoch, showing that the model is learning the right weights and biases to predict the correct labels.\n",
    "* The training loss and validation loss should follow a similar trend, showing that the model is not overfitting to the training data.\n",
    "\n",
    "Let's plot the loss metrics and see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.plot(epoch_nums, training_loss)\n",
    "plt.plot(epoch_nums, validation_loss)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('loss')\n",
    "plt.legend(['training', 'validation'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the learned weights and biases\n",
    "\n",
    "The trained model consists of the final weights and biases that were determined by the optimizer during training. Based on our network model we should expect the following values for each layer:\n",
    "* Layer 1: There are four input values going to ten output nodes, so there should be 10 x 4 weights and 10 bias values.\n",
    "* Layer 2: There are ten input values going to ten output nodes, so there should be 10 x 10 weights and 10 bias values.\n",
    "* Layer 3: There are ten input values going to three output nodes, so there should be 3 x 10 weights and 3 bias values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for param_tensor in model.state_dict():\n",
    "    print(param_tensor, \"\\n\", model.state_dict()[param_tensor].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance\n",
    "\n",
    "So, is the model any good? The raw accuracy reported from the validation data would seem to indicate that it predicts pretty well; but it's typically useful to dig a little deeper and compare the predictions for each possible class. A common way to visualize the performace of a classification model is to create a *confusion matrix* that shows a crosstab of correct and incorrect predictions for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Pytorch doesn't have a built-in confusion matrix metric, so we'll use SciKit-Learn\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import numpy as np\n",
    "\n",
    "# Set the model to evaluate mode\n",
    "model.eval()\n",
    "\n",
    "# Get predictions for the test data\n",
    "x = torch.Tensor(x_test).float()\n",
    "_, predicted = torch.max(model(x).data, 1)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm = confusion_matrix(y_test, predicted.numpy())\n",
    "plt.imshow(cm, interpolation=\"nearest\", cmap=plt.cm.Blues)\n",
    "plt.colorbar()\n",
    "tick_marks = np.arange(len(penguin_classes))\n",
    "plt.xticks(tick_marks, penguin_classes, rotation=45)\n",
    "plt.yticks(tick_marks, penguin_classes)\n",
    "plt.xlabel(\"Actual Species\")\n",
    "plt.ylabel(\"Predicted Species\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix should show a strong diagonal line indicating that there are more correct than incorrect predictions for each class.\n",
    "\n",
    "## Save the trained model\n",
    "Now that we have a model we believe is reasonably accurate, we can save its trained weights for use later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model weights\n",
    "model_file = 'models/penguin_classifier.pt'\n",
    "torch.save(model.state_dict(), model_file)\n",
    "del model\n",
    "print('model saved as', model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the trained model\n",
    "\n",
    "When we have a new penguin observation, we can use the model to predict the species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# New penguin features\n",
    "x_new = [[50.4,15.3,20,50]]\n",
    "print ('New sample: {}'.format(x_new))\n",
    "\n",
    "# Create a new model class and load weights\n",
    "model = PenguinNet()\n",
    "model.load_state_dict(torch.load(model_file))\n",
    "\n",
    "# Set model to evaluation mode\n",
    "model.eval()\n",
    "\n",
    "# Get a prediction for the new data sample\n",
    "x = torch.Tensor(x_new).float()\n",
    "_, predicted = torch.max(model(x).data, 1)\n",
    "\n",
    "print('Prediction:',penguin_classes[predicted.item()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "\n",
    "This notebook was designed to help you understand the basic concepts and principles involved in deep neural networks, using a simple PyTorch example. To learn more about PyTorch, take a look at the [tutorials on the PyTorch web site](https://pytorch.org/tutorials/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}