code/day17 ReLU activation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab - ReLU activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.gridspec import GridSpec\n",
    "plt.style.use('./deeplearning.mplstyle')\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, LeakyReLU\n",
    "from tensorflow.keras.activations import linear, relu, sigmoid\n",
    "%matplotlib widget\n",
    "from matplotlib.widgets import Slider\n",
    "from lab_utils_common import dlc\n",
    "from autils import plt_act_trio\n",
    "from lab_utils_relu import *\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=UserWarning)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"2\"></a>\n",
    "## 2 - ReLU Activation\n",
    "This week, a new activation was introduced, the Rectified Linear Unit (ReLU). \n",
    "$$ a = max(0,z) \\quad\\quad\\text {# ReLU function} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1eb08c3481f9446884693cbbc367096e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt_act_trio()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"right\" src=\"./images/C2_W2_ReLu.png\"     style=\" width:380px; padding: 10px 20px; \" >\n",
    "The example from the lecture on the right shows an application of the ReLU. In this example, the derived \"awareness\" feature is not binary but has a continuous range of values. The sigmoid is best for on/off or binary situations. The ReLU provides a continuous linear relationship. Additionally it has an 'off' range where the output is zero.     \n",
    "The \"off\" feature makes the ReLU a Non-Linear activation. Why is this needed? Let's examine this below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why Non-Linear Activations?  \n",
    "<img align=\"left\" src=\"./images/C2_W2_ReLU_Graph.png\"     style=\" width:250px; padding: 10px 20px; \" > The function shown is composed of linear pieces (piecewise linear). The slope is consistent during the linear portion and then changes abruptly at transition points. At transition points, a new linear function is added which, when added to the existing function, will produce the new slope. The new function is added at transition point but does not contribute to the output prior to that point. The non-linear activation function is responsible for disabling the input prior to and sometimes after the transition points. The following exercise provides a more tangible example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exercise will use the network below in a regression problem where you must model a piecewise linear target :\n",
    "<img align=\"center\" src=\"./images/C2_W2_ReLU_Network.png\"     style=\" width:650px; padding: 10px 20px; \">  \n",
    "The network has 3 units in the first layer. Each is required to form the target. Unit 0 is pre-programmed and fixed to map the first segment. You will modify weights and biases in unit 1 and 2 to model the 2nd and 3rd segment. The output unit is also fixed and simply sums the outputs of the first layer.  \n",
    "\n",
    "Using the sliders below, modify weights and bias to match the target. \n",
    "Hints: Start with `w1` and `b1` and leave `w2` and `b2` zero until you match the 2nd segment. Clicking rather than sliding is quicker.  If you have trouble, don't worry, the text below will describe this in more detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc513b4bc8e2419a86663a31a55a1fbf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = plt_relu_ex()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "The goal of this exercise is to appreciate how the ReLU's non-linear behavior provides the needed ability to turn functions off until they are needed. Let's see how this worked in this example.\n",
    "<img align=\"right\" src=\"./images/C2_W2_ReLU_Plot.png\"     style=\" width:600px; padding: 10px 20px; \"> \n",
    "The plots on the right contain the output of the units in the first layer.   \n",
    "Starting at the top, unit 0 is responsible for the first segment marked with a 1. Both the linear function $z$ and the function following the ReLU $a$ are shown. You can see that the ReLU cuts off the function after the interval [0,1]. This is important as it prevents Unit 0 from interfering with the following segment. \n",
    "\n",
    "Unit 1 is responsible for the 2nd segment. Here the ReLU kept this unit quiet until after x is 1. Since the first unit is not contributing, the slope for unit 1, $w^{[1]}_1$, is just the slope of the target line. The bias must be adjusted to keep the output negative until x has reached 1. Note how the contribution of Unit 1 extends to the 3rd segment as well.\n",
    "\n",
    "Unit 2 is responsible for the 3rd segment. The ReLU again zeros the output until x reaches the right value.The slope of the unit, $w^{[1]}_2$, must be set so that the sum of unit 1 and 2 have the desired slope. The bias is again adjusted to keep the output negative until x has reached 2. \n",
    "\n",
    "The \"off\" or disable feature  of the ReLU activation enables models to stitch together linear segments to model complex non-linear functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "You are now more familiar with the ReLU and the importance of its non-linear behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}