05a - Deep Neural Networks (TensorFlow).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning with TensorFlow\n",
    "\n",
    "Classical machine learning relies on using statistics to determine relationships between features and labels, and can be very effective for creating predictive models. However, a massive growth in the availability of data coupled with advances in the computing technology required to process it has led to the emergence of new machine learning techniques that mimic the way the brain processes information in a structure called an artificial neural network.\n",
    "\n",
    "TensorFlow is a framework for creating machine learning models, including deep neural networks (DNNs). In this example, we'll use Tensorflow to create a simple neural network that classifies penguins into species based on the length and depth of their culmen (bill), their flipper length, and their body mass.\n",
    "\n",
    "> **Citation**: The penguins dataset used in the this exercise is a subset of data collected and made available by [Dr. Kristen\n",
    "Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php)\n",
    "and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a\n",
    "member of the [Long Term Ecological Research\n",
    "Network](https://lternet.edu/).\n",
    "\n",
    "## Explore the dataset\n",
    "\n",
    "Before we start using TensorFlow to create a model, let's load the data we need from the Palmer Islands penguins dataset, which contains observations of three different species of penguin.\n",
    "\n",
    "> **Note**: In reality, you can solve the penguin classification problem easily using classical machine learning techniques without the need for a deep learning model; but it's a useful, easy to understand dataset with which to demonstrate the principles of neural networks in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# load the training dataset (excluding rows with null values)\n",
    "penguins = pd.read_csv('data/penguins.csv').dropna()\n",
    "\n",
    "# Deep Learning models work best when features are on similar scales\n",
    "# In a real solution, we'd implement some custom normalization for each feature, but to keep things simple\n",
    "# we'll just rescale the FlipperLength and BodyMass so they're on a similar scale to the bill measurements\n",
    "penguins['FlipperLength'] = penguins['FlipperLength']/10\n",
    "penguins['BodyMass'] = penguins['BodyMass']/100\n",
    "\n",
    "# The dataset is too small to be useful for deep learning\n",
    "# So we'll oversample it to triple its size\n",
    "for i in range(1,3):\n",
    "    penguins = penguins.append(penguins)\n",
    "\n",
    "# Display a random sample of 10 observations\n",
    "sample = penguins.sample(10)\n",
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Species** column is the label our model will predict. Each label value represents a class of penguin species, encoded as 0, 1, or 2. The following code shows the actual species to which these class labels corrrespond."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "penguin_classes = ['Amelie', 'Gentoo', 'Chinstrap']\n",
    "print(sample.columns[0:5].values, 'SpeciesName')\n",
    "for index, row in penguins.sample(10).iterrows():\n",
    "    print('[',row[0], row[1], row[2],row[3], int(row[4]), ']',penguin_classes[int(row[-1])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As is common in a supervised learning problem, we'll split the dataset into a set of records with which to train the model, and a smaller set with which to validate the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "features = ['CulmenLength','CulmenDepth','FlipperLength','BodyMass']\n",
    "label = 'Species'\n",
    "   \n",
    "# Split data 70%-30% into training set and test set\n",
    "x_train, x_test, y_train, y_test = train_test_split(penguins[features].values,\n",
    "                                                    penguins[label].values,\n",
    "                                                    test_size=0.30,\n",
    "                                                    random_state=0)\n",
    "\n",
    "print ('Training Set: %d, Test Set: %d \\n' % (len(x_train), len(x_test)))\n",
    "print(\"Sample of features and labels:\")\n",
    "\n",
    "# Take a look at the first 25 training features and corresponding labels\n",
    "for n in range(0,24):\n",
    "    print(x_train[n], y_train[n], '(' + penguin_classes[y_train[n]] + ')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *features* are the measurements for each penguin observation, and the *label* is a numeric value that indicates the species of penguin that the observation represents (Amelie, Gentoo, or Chinstrap).\n",
    "\n",
    "## Install and import TensorFlow libraries\n",
    "\n",
    "Since we plan to use TensorFlow to create our penguin classifier, we'll need to run the following two cells to install and import the libraries we intend to use.\n",
    "\n",
    "> **Note** *Keras* is an abstraction layer over the base TensorFlow API. In most common machine learning scenarios, you can use Keras to simplify your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras import utils\n",
    "from tensorflow.keras import optimizers\n",
    "\n",
    "# Set random seed for reproducability\n",
    "tensorflow.random.set_seed(0)\n",
    "\n",
    "print(\"Libraries imported.\")\n",
    "print('Keras version:',keras.__version__)\n",
    "print('TensorFlow version:',tensorflow.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the data for TensorFlow\n",
    "\n",
    "We've already loaded our data and split it into training and validation datasets. However, we need to do some further data preparation so that our data will work correctly with TensorFlow. Specifically, we need to set the data type of our features to 32-bit floating point numbers, and specify that the labels represent categorical classes rather than numeric values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set data types for float features\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "\n",
    "# Set data types for categorical labels\n",
    "y_train = utils.to_categorical(y_train)\n",
    "y_test = utils.to_categorical(y_test)\n",
    "print('Ready...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a neural network\n",
    "\n",
    "Now we're ready to define our neural network. In this case, we'll create a network that consists of 3 fully-connected layers:\n",
    "* An input layer that receives an input value for each feature (in this case, the four penguin measurements) and applies a *ReLU* activation function.\n",
    "* A hidden layer that receives ten inputs and applies a *ReLU* activation function.\n",
    "* An output layer that uses a *SoftMax* activation function to generate an output for each penguin species (which represent the classification probabilities for each of the three possible penguin species). Softmax functions produce a vector with probability values that sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a classifier network\n",
    "hl = 10 # Number of hidden layer nodes\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(hl, input_dim=len(features), activation='relu'))\n",
    "model.add(Dense(hl, input_dim=hl, activation='relu'))\n",
    "model.add(Dense(len(penguin_classes), input_dim=hl, activation='softmax'))\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "To train the model, we need to repeatedly feed the training values forward through the network, use a loss function to calculate the loss, use an optimizer to backpropagate the weight and bias value adjustments, and validate the model using the test data we withheld.\n",
    "\n",
    "To do this, we'll apply an Adam optimizer to a categorical cross-entropy loss function iteratively over 50 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#hyper-parameters for optimizer\n",
    "learning_rate = 0.001\n",
    "opt = optimizers.Adam(lr=learning_rate)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=opt,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Train the model over 50 epochs using 10-observation batches and using the test holdout dataset for validation\n",
    "num_epochs = 50\n",
    "history = model.fit(x_train, y_train, epochs=num_epochs, batch_size=10, validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the training process is running, let's try to understand what's happening:\n",
    "\n",
    "1. In each *epoch*, the full set of training data is passed forward through the network. There are four features for each observation, and four corresponding nodes in the input layer - so the features for each observation are passed as a vector of four values to that layer. However, for efficiency, the feature vectors are grouped into batches; so actually a matrix of multiple feature vectors is fed in each time.\n",
    "2. The matrix of feature values is processed by a function that performs a weighted sum using initialized weights and bias values. The result of this function is then processed by the activation function for the input layer to constrain the values passed to the nodes in the next layer.\n",
    "3. The weighted sum and activation functions are repeated in each layer. Note that the functions operate on vectors and matrices rather than individual scalar values. In other words, the forward pass is essentially a series of nested linear algebra functions. This is the reason data scientists prefer to use computers with graphical processing units (GPUs), since these are optimized for matrix and vector calculations.\n",
    "4. In the final layer of the network, the output vectors contain a probability value for each possible class (in this case, classes 0, 1, and 2). This vector is processed by a *loss function* to determine how far the values calculated by the network are from the actual values - so for example, suppose the output for a Gentoo penguin (class 1) observation is \\[0.3, 0.4, 0.3\\]. The correct prediction should be \\[0.0, 1.0, 0.0\\], so the variance between the predicted and actual values (how far away the each predicted value is from what it should be) is \\[0.3, 0.6, 0.3\\]. This variance is aggregated for each batch and maintained as a running aggregate to calculate the overall level of error (*loss*) incurred by the training data for the epoch. The accuracy (proportion of correct predictions based on the highest probability value in the output vector) for the training data is also calculated.\n",
    "5. At the end of each epoch, the validation data is passed through the network, and its loss and accuracy are also calculated. It's important to do this because it enables us to compare the performance of the model using data on which it was not trained, helping us determine if it will generalize well for new data or if it's *overfitted* to the training data.\n",
    "6. After all the data has been passed forward through the network, the output of the loss function for the *training* data (but <u>not</u> the *validation* data) is passed to the opimizer. The precise details of how the optimizer processes the loss vary depending on the specific optimization algorithm being used; but fundamentally you can think of the entire network, from the input layer to the loss function as being one big nested (*composite*) function. The optimizer applies some differential calculus to calculate *partial derivatives* for the function with respect to each weight and bias value that was used in the network. It's possible to do this efficiently for a nested function due to something called the *chain rule*, which enables you to determine the derivative of a composite function from the derivatives of its inner function and outer functions. You don't really need to worry about the details of the math here (the optimizer does it for you), but the end result is that the partial derivatives tell us about the slope (or *gradient*) of the loss function with respect to each weight and bias value - in other words, we can determine whether to increase or decrease the weight and bias values in order to decrease the loss.\n",
    "7. Having determined in which direction to adjust the weights and biases, the optimizer uses the *learning rate* to determine by how much to adjust them; and then works backwards through the network in a process called *backpropagation* to assign new values to the weights and biases in each layer.\n",
    "8. Now the next epoch repeats the whole training, validation, and backpropagation process starting with the revised weights and biases from the previous epoch - which hopefully will result in a lower level of loss.\n",
    "9. The process continues like this for 50 epochs.\n",
    "\n",
    "## Review training and validation loss\n",
    "\n",
    "After training is complete, we can examine the loss metrics we recorded while training and validating the model. We're really looking for two things:\n",
    "* The loss should reduce with each epoch, showing that the model is learning the right weights and biases to predict the correct labels.\n",
    "* The training loss and validation loss should follow a similar trend, showing that the model is not overfitting to the training data.\n",
    "\n",
    "Let's plot the loss metrics and see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "epoch_nums = range(1,num_epochs+1)\n",
    "training_loss = history.history[\"loss\"]\n",
    "validation_loss = history.history[\"val_loss\"]\n",
    "plt.plot(epoch_nums, training_loss)\n",
    "plt.plot(epoch_nums, validation_loss)\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('loss')\n",
    "plt.legend(['training', 'validation'], loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the learned weights and biases\n",
    "\n",
    "The trained model consists of the final weights and biases that were determined by the optimizer during training. Based on our network model we should expect the following values for each layer:\n",
    "* Layer 1: There are four input values going to ten output nodes, so there should be 4 x 10 weights and 10 bias values.\n",
    "* Layer 2: There are ten input values going to ten output nodes, so there should be 10 x 10 weights and 10 bias values.\n",
    "* Layer 3: There are ten input values going to three output nodes, so there should be 10 x 3 weights and 3 bias values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for layer in model.layers:\n",
    "    weights = layer.get_weights()[0]\n",
    "    biases = layer.get_weights()[1]\n",
    "    print('------------\\nWeights:\\n',weights,'\\nBiases:\\n', biases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance\n",
    "\n",
    "So, is the model any good? The raw accuracy reported from the validation data would seem to indicate that it predicts pretty well; but it's typically useful to dig a little deeper and compare the predictions for each possible class. A common way to visualize the performace of a classification model is to create a *confusion matrix* that shows a crosstab of correct and incorrect predictions for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tensorflow doesn't have a built-in confusion matrix metric, so we'll use SciKit-Learn\n",
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "class_probabilities = model.predict(x_test)\n",
    "predictions = np.argmax(class_probabilities, axis=1)\n",
    "true_labels = np.argmax(y_test, axis=1)\n",
    "\n",
    "# Plot the confusion matrix\n",
    "cm = confusion_matrix(true_labels, predictions)\n",
    "plt.imshow(cm, interpolation=\"nearest\", cmap=plt.cm.Blues)\n",
    "plt.colorbar()\n",
    "tick_marks = np.arange(len(penguin_classes))\n",
    "plt.xticks(tick_marks, penguin_classes, rotation=85)\n",
    "plt.yticks(tick_marks, penguin_classes)\n",
    "plt.xlabel(\"Actual Class\")\n",
    "plt.ylabel(\"Predicted Class\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix should show a strong diagonal line indicating that there are more correct than incorrect predictions for each class.\n",
    "\n",
    "## Save the trained model\n",
    "Now that we have a model we believe is reasonably accurate, we can save its trained weights for use later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the trained model\n",
    "modelFileName = 'models/penguin-classifier.h5'\n",
    "model.save(modelFileName)\n",
    "del model  # deletes the existing model variable\n",
    "print('model saved as', modelFileName)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the trained model\n",
    "\n",
    "When we have a new penguin observation, we can use the model to predict the species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load the saved model\n",
    "model = models.load_model(modelFileName)\n",
    "\n",
    "# CReate a new array of features\n",
    "x_new = np.array([[50.4,15.3,20,50]])\n",
    "print ('New sample: {}'.format(x_new))\n",
    "\n",
    "# Use the model to predict the class\n",
    "class_probabilities = model.predict(x_new)\n",
    "predictions = np.argmax(class_probabilities, axis=1)\n",
    "\n",
    "print(penguin_classes[predictions[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "\n",
    "This notebook was designed to help you understand the basic concepts and principles involved in deep neural networks, using a simple Tensorflow example. To learn more about Tensorflow, take a look at the <a href=\"https://www.tensorflow.org/\" target=\"_blank\">Tensorflow web site</a>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}