09. Neural Networks Advanced - R.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Exercise 9 - Advanced Neural Networks\n===\n\nThere are many factors that influence how well a neural network might perform. AI practitioners tend to play around with the structure of the hidden layers, the activation function, the optimisation function, and the number of epochs (training cycles).\n\nIn this exercise, we will look at how changing these parameters impacts the accuracy and performance of our network.\n\nLet's start by loading the libraries required for this session.\n\n**Run the code below**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this code box to load the packages we need\n# It might take a few minutes...\n\nsuppressMessages(install.packages(\"tidyverse\"))\nsuppressMessages(library(\"tidyverse\"))\n\nsuppressMessages(install.packages(\"keras\"))\nsuppressMessages(library(\"keras\"))\nsuppressMessages(install_keras())\n\noptions(repr.plot.width = 7, repr.plot.height = 5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 1\n---\n\nWe will use the same dog data set as in Exercise 8, building on what we learnt before and trying different parameters for a network to try and improve performance.\n\nLet's open up our data set and create training and test sets.\n\n### In the cell below replace:\n#### 1. `<featureColumns>` with `1:3`\n#### 2. `<labelColumn>` with `4`\n#### 3. `<featureColumns>` with `1:3`\n#### 4. `<labelColumn>` with `4`\n#### then __run the code__."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to set up our training and test datasets\n\n# Load the dog data\ndog_data <- read.csv(\"Data/dog_data.csv\")\n\n# Check structure\nstr(dog_data)\nhead(dog_data)\n\n# Take the first 160 observations, separate the features from the labels, and assign them to the training set\n###\n# REPLACE <featureColumns> WITH 1:3 AND <labelColumn> WITH 4\n###\ntrain_X <- as.matrix(dog_data[1:160, <featureColumns>])\nraw_train_Y <- as.matrix(dog_data[1:160, <labelColumn>])\n###\n\n# Take the last 40 observations, separate the features from the labels, and assign them to the test set\n###\n# REPLACE <featureColumns> WITH 1:3 AND <labelColumn> WITH 4\n###\ntest_X <- as.matrix(dog_data[161:200, <featureColumns>])\nraw_test_Y <- as.matrix(dog_data[161:200, <labelColumn>])\n###",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "And just like the last exercise, we will transform the raw labels into one-hot vectors\n\n### In the cell below replace:\n#### 1. `<one-hot-function>` with `to_categorical`\n#### 2. `<numberOfClasses>` with `3`\n#### 3. `<one-hot-function>` with `to_categorical`\n#### 4. `<numberOfClasses>` with `3`\n#### then __run the code__."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Set the testing and training labels as categories using one-hot vectors\n###\n# REPLACE <one-hot-function> WITH to_categorical AND <numberOfClasses> WITH 3\n###\ntrain_Y <- <one-hot-function>(raw_train_Y, num_classes = <numberOfClasses>)\ntest_Y <- <one-hot-function>(raw_test_Y, num_classes = <numberOfClasses>)\n###\n\nhead(train_Y)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Done!\n\nStep 2\n---\n\nThe code block below contains a custom function `train_network` to help us quickly change the training factors of our neural network. We will use this function throughout the remainder of this exercise.\n\nThe `train_network` function allows us to change:\n\n* the size and/or number of layers;\n* the activation function the layers use;\n* the optimizer of the model;\n* the number of training cycles for the model (`epochs`).\n\n**Run the code below**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to prepare functions for later\n\n# Define our custom function `train_network` with four arguments\ntrain_network <- function(structure, activation, optimizer, epochs){\n    suppressMessages(use_session_with_seed(1))\n    model = keras_model_sequential()\n    \n    model %>%\n    layer_dense(units = structure[2], activation = activation, input_shape = structure[1]) %>%\n    layer_dense(units = structure[3], activation = activation) %>% \n    layer_dense(units = structure[4], activation = \"softmax\")\n    \n    model %>% \n    compile(loss = \"categorical_crossentropy\", optimizer = optimizer, metrics = c(\"accuracy\"))\n    \n    history = model %>% \n    fit(x = train_X, y = train_Y, shuffle = T, epochs = epochs, batch_size = 5, \n        validation_split = 0.3)\n    \n    history_df <- as.data.frame(history)   \n    acc <<- history_df[nrow(history_df), 2]\n    print(\"Accuracy based on training set...\")\n    print(history_df[nrow(history_df), 2])\n    \n    perf <- model %>% evaluate(test_X, test_Y)\n    print(\"Accuracy based on test set...\")\n    print(perf$acc)\n    testacc <<- perf$acc\n    \n    plot(history)\n}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Let's recreate the neural network from Exercise 8 to use as our bench mark, but we will change it to have two hidden layers. You do not need to edit the code block below.\n\n### In the cell below replace:\n#### 1. `<activationFunction>` with `\"relu\"`\n#### 2. `<optimizer>` with `\"adagrad\"`\n#### 3. `<epochNumber>` with `30`\n#### then __run the code__."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this code to train the network\n\n# Create variables for each of the inputs to our custom function\nsample_structure <- c(3, 10, 10, 3)\n###\n# REPLACE <activationFunction> WITH \"relu\" (INCLUDING THE QUOTATION MARKS!)\n###\nsample_activation <- <activationFunction>\n###\n\n###\n# REPLACE <activationFunction> WITH \"adagrad\" (INCLUDING THE QUOTATION MARKS!)\n###\noptimizer <- <activationFunction>\n###\n\n###\n# REPLACE <epochNumber> WITH 30\n###\nsample_epochs <- <epochNumber>\n###\n\n# Run our custom function specifying our arguments in correct order: structure, activation, optimizer, epochs\ntrain_network(sample_structure, sample_activation, optimizer, sample_epochs)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 3\n---\n\nNow, let's start playing with the structure of our neural network, in particular the size of our hidden layers. We can easily do this by changing the input to the first argument of our `train_network` function, `structure`.\n\nHere we will test the size of our two hidden layers, testing values 1 through to 10. For simplicity, we will make the size of the two hidden layers the same, e.g. when we test a layer size of 5, the structure of our neural network will be `[3, 5, 5, 3]`, and when we test a layer size of 9, our neural network structure will be `[3, 9, 9, 3]`. Note that both the input and output layers of our network must remain as size 3, as our data have 3 input features.\n\n**In the code below:**  \n**1. Run the first box to alter the structure of the network**  \n**2. Run the second box to plot the results**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this code box to alter the structure of the network\n\n# Initialise empty lists to store results\ntrain_acc <- c()\ntest_acc <- c()\n\n# Change the input to our first argument of our `train_network` function\nfor(i in 1:10){\n    NN_structure <- c(3, i, i, 3)\n    print(\"TEST THE FOLLOWING NUMBER OF HIDDEN LAYERS...\")\n    print(i)\n    train_network(NN_structure, sample_activation, optimizer, sample_epochs)\n    train_acc[i] <- acc\n    test_acc[i] <- testacc\n}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to plot the results\n\n# Reshape the results for plotting\ntrain_results <- data.frame(dataType = rep(\"Training\", 10), acc = train_acc, nLayers = seq(1, 10, 1), stringsAsFactors = FALSE)\ntest_results <- data.frame(dataType = rep(\"Test\", 10), acc = test_acc, nLayers = seq(1, 10, 1), stringsAsFactors = FALSE)\n\nhiddenLayerDf <- train_results %>%  mutate(dataType = 'Training') %>%\n       bind_rows(test_results %>%\n           mutate(dataType = 'Test'))\n\nggplot(hiddenLayerDf,aes(y = acc,x = nLayers,color = dataType)) + \n  geom_line() +\n  labs(title = \"\", x = \"Size of hidden layers\", y = \"Accuracy\", colour = \"Data type\") +\nscale_x_discrete(limits = seq(1, 10, 1))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "So, experimenting with different sizes of hidden layers can dramatically improve your results.\n\nStep 4\n---\n\nNow we'll look at how different **activation functions** impact the performance of neural networks. To do this, we need to change the second argument to our custom function `train_network`, the `activation` argument.\n\nThere are many different activation functions to try, so let's store them all as a vector and try them all!\n\n#### Replace `<addActivation>` with `activation_functions[i]` and run the code."
    },
    {
      "metadata": {
        "scrolled": false,
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to run the network with different activation functions\n\n# Initialise empty lists to store results\ntrain_acc <- c()\ntest_acc <- c()\n\n# Create a vector listing all the activation functions we wish to test\nactivation_functions <- c(\"elu\", \"hard_sigmoid\", \"linear\", \"relu\", \"selu\", \"sigmoid\", \n                         \"softplus\", \"softsign\", \"tanh\")\n\n# # Uncomment the code below to play with the structure, optimizer, and epochs\n# sample_structure <- c(3, ?, ?, 3) # e.g. c(3, 4, 4, 3)\n# optimizer <- \"?\" # e.g. \"adagrad\"\n# sample_epochs <- ? # e.g. 20\n\n# Test all the different activation functions and save results\nfor(i in 1:length(activation_functions)){\n    print(\"Evaluating model with hidden layer activation function... \")\n    print(activation_functions[i])\n###\n# REPLACE <addActivation> WITH activation_functions[i]\n###    \n    train_network(sample_structure, <addActivation>, optimizer, sample_epochs)\n###    \n    train_acc[i] <- acc\n    test_acc[i] <- testacc\n}\n\nprint(\"Finished!\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Now run the code below to plow the results."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to plot the result\n\n# Reshape the results for plotting\ntrain_results <- data.frame(dataType = \"Train\", actFuncName = activation_functions, funcAcc = train_acc,\n                            stringsAsFactors = FALSE)\ntest_results <- data.frame(dataType = \"Test\", actFuncName = activation_functions, funcAcc = test_acc,\n                           stringsAsFactors = FALSE)\n\nresults <- bind_rows(train_results, test_results) %>%\nmutate(dataType = as.factor(dataType))\n\n# Create line plot: activation function vs. accuracy coloured by data type\nresults %>%\nggplot(aes(actFuncName, funcAcc, group = dataType, colour = dataType)) +\ngeom_line() +\nlabs(title = \"\", x = \"Activation function\", y = \"Function accuracy\", colour = \"Data type\") +\ntheme(plot.title = element_text(hjust = 0.5))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "There's quite a lot of variance there. It's always good to quickly test different activation functions first.\n\nStep 5\n---\n\nThe __optimisation function__ is the next major parameter of the network architecture. It changes how the network is trained, so it can have a __very large impact on training time and end performance__.\n\n#### Replace `<optimizerFunction>` with `optimizer_functions[i]` and run the cell."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to try different optimization functions\n\n# Initialise empty lists to store results\ntrain_acc <- c()\ntest_acc <- c()\n\n# Create a vector listing all the optimization functions we wish to test\noptimizer_functions = c(\"adadelta\", \"adagrad\", \"adam\", \"adamax\",\n                        \"nadam\", \"rmsprop\", \"sgd\")\nNN_structure <- c(3, 9, 9, 3)\nsample_activation <- \"relu\"\n\n# Uncomment the code below to play with the structure, activation, and epochs\n# NN_structure <- c(3, ?, ?, 3) # e.g. c(3, 4, 4, 3)\n# sample_activation <- ? # e.g. \"tanh\"\n# sample_epochs <- ? # e.g. 20\n\n# Test all the different optimization functions and save results\nfor(i in 1:length(optimizer_functions)){\n    print(\"Evaluating model with hidden layer optimization function... \")\n    print(optimizer_functions[i])\n###\n# REPLACE <optimizerFunction> WITH optimizer_functions[i]\n###    \n    train_network(NN_structure, sample_activation, <optimizerFunction>, sample_epochs)\n###    \n    train_acc[i] <- acc\n    test_acc[i] <- testacc\n}\n\ntrain_acc\ntest_acc",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Now run the code below to plot the results."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this box to plot the results\n\n# Reshape the results to create plot\ntrain_results <- data.frame(dataType = \"Train\", optFuncName = optimizer_functions, funcAcc = train_acc,\n                            stringsAsFactors = FALSE)\ntest_results <- data.frame(dataType = \"Test\", optFuncName = optimizer_functions, funcAcc = test_acc,\n                           stringsAsFactors = FALSE)\n\nresults <- bind_rows(train_results, test_results) %>%\nmutate(dataType = as.factor(dataType))\n\n# Create line plot: optimzation function vs. accuracy coloured by data type\nresults %>%\nggplot(aes(optFuncName, funcAcc, group = dataType, colour = dataType)) +\ngeom_line() +\nlabs(title = \"Performance of training and test sets using different optimizer functions\",\n     x = \"Optimization function\", y = \"Function accuracy\", colour = \"Data type\") +\ntheme(plot.title = element_text(hjust = 0.5))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 6\n---\n\nNow let's test the number of training cycles for the model, i.e. `epochs`, the final argument in our custom function.\n\n**In the code below, change the epochs below to any positive whole number and press Run. Try this with several different numbers.**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "###\n# CHANGE 15 TO ANY POSITIVE INTEGER\n###\nepochs <- 15\n\ntrain_network(sample_structure, sample_activation, optimizer, epochs)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "You will notice a trend: the higher the number of epoch/training cycles, the greater the accuracy of the model.\n\nStep 7\n---\n\nLet's try to combine what we've seen above and try to create a neural network that performs better than what we made in Exercise 7, where we used the structure `[3, 4, 2, 3]`, the activation function `relu`, and the optimiser `sgd` (stochastic gradient descent).\n\n**Follow the instructions in the code below**"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "###\n# Run this box to train once more with a good selection of options\n# Then change the configurations as you like and run again to see how the network performs\n###\n\nsample_structure <- c(3, 9, 9, 3)\nsample_activation <- \"selu\"\noptimizer <- \"adam\"\nsample_epochs <- 10\n\ntrain_network(sample_structure, sample_activation, optimizer, sample_epochs)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "How does it look? Were we able to beat the other network? Try out a number of different configurations to see how they perform!\n\nConclusion\n---\n\nWe've compared how different neural network architecture parameters influence accuracy performance, and we've tried to combine them in such a way that we maximise this performance."
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "r",
      "display_name": "R",
      "language": "R"
    },
    "language_info": {
      "mimetype": "text/x-r-source",
      "name": "R",
      "pygments_lexer": "r",
      "version": "3.4.1",
      "file_extension": ".r",
      "codemirror_mode": "r"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}