BuildingDeepNeuralNetwork.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "coursera": {
      "course_slug": "neural-networks-deep-learning",
      "graded_item_id": "c4HO0",
      "launcher_item_id": "lSYZM"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    },
    "colab": {
      "name": "Building_your_Deep_Neural_Network_Step_by_Step.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjvXu8322KX4",
        "colab_type": "text"
      },
      "source": [
        "**Notation**:\n",
        "- Superscript $[l]$ denotes a quantity associated with the $l^{th}$ layer. \n",
        "    - Example: $a^{[L]}$ is the $L^{th}$ layer activation. $W^{[L]}$ and $b^{[L]}$ are the $L^{th}$ layer parameters.\n",
        "- Superscript $(i)$ denotes a quantity associated with the $i^{th}$ example. \n",
        "    - Example: $x^{(i)}$ is the $i^{th}$ training example.\n",
        "- Lowerscript $i$ denotes the $i^{th}$ entry of a vector.\n",
        "    - Example: $a^{[l]}_i$ denotes the $i^{th}$ entry of the $l^{th}$ layer's activations)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ExNM1Sqs2KX8",
        "colab_type": "code",
        "colab": {},
        "outputId": "1e83e5d3-3e48-43d3-dea2-896507f6fa35"
      },
      "source": [
        "import numpy as np\n",
        "import h5py\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "%matplotlib inline\n",
        "plt.rcParams['figure.figsize'] = (5.0, 4.0) # set default size of plots\n",
        "plt.rcParams['image.interpolation'] = 'nearest'\n",
        "plt.rcParams['image.cmap'] = 'gray'\n",
        "\n",
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "np.random.seed(1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The autoreload extension is already loaded. To reload it, use:\n",
            "  %reload_ext autoreload\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KEDcJDD05kJ9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sigmoid(Z):\n",
        "    \"\"\"\n",
        "    Implements the sigmoid activation in numpy\n",
        "    \n",
        "    Arguments:\n",
        "    Z -- numpy array of any shape\n",
        "    \n",
        "    Returns:\n",
        "    A -- output of sigmoid(z), same shape as Z\n",
        "    cache -- returns Z as well, useful during backpropagation\n",
        "    \"\"\"\n",
        "    \n",
        "    A = 1/(1+np.exp(-Z))\n",
        "    cache = Z\n",
        "    \n",
        "    return A, cache\n",
        "\n",
        "def relu(Z):\n",
        "    \"\"\"\n",
        "    Implement the RELU function.\n",
        "\n",
        "    Arguments:\n",
        "    Z -- Output of the linear layer, of any shape\n",
        "\n",
        "    Returns:\n",
        "    A -- Post-activation parameter, of the same shape as Z\n",
        "    cache -- a python dictionary containing \"A\" ; stored for computing the backward pass efficiently\n",
        "    \"\"\"\n",
        "    \n",
        "    A = np.maximum(0,Z)\n",
        "    \n",
        "    assert(A.shape == Z.shape)\n",
        "    \n",
        "    cache = Z \n",
        "    return A, cache\n",
        "\n",
        "\n",
        "def relu_backward(dA, cache):\n",
        "    \"\"\"\n",
        "    Implement the backward propagation for a single RELU unit.\n",
        "\n",
        "    Arguments:\n",
        "    dA -- post-activation gradient, of any shape\n",
        "    cache -- 'Z' where we store for computing backward propagation efficiently\n",
        "\n",
        "    Returns:\n",
        "    dZ -- Gradient of the cost with respect to Z\n",
        "    \"\"\"\n",
        "    \n",
        "    Z = cache\n",
        "    dZ = np.array(dA, copy=True) # just converting dz to a correct object.\n",
        "    \n",
        "    # When z <= 0, you should set dz to 0 as well. \n",
        "    dZ[Z <= 0] = 0\n",
        "    \n",
        "    assert (dZ.shape == Z.shape)\n",
        "    \n",
        "    return dZ\n",
        "\n",
        "def sigmoid_backward(dA, cache):\n",
        "    \"\"\"\n",
        "    Implement the backward propagation for a single SIGMOID unit.\n",
        "\n",
        "    Arguments:\n",
        "    dA -- post-activation gradient, of any shape\n",
        "    cache -- 'Z' where we store for computing backward propagation efficiently\n",
        "\n",
        "    Returns:\n",
        "    dZ -- Gradient of the cost with respect to Z\n",
        "    \"\"\"\n",
        "    \n",
        "    Z = cache\n",
        "    \n",
        "    s = 1/(1+np.exp(-Z))\n",
        "    dZ = dA * s * (1-s)\n",
        "    \n",
        "    assert (dZ.shape == Z.shape)\n",
        "    \n",
        "    return dZ"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTB4qnf42KYE",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: Create and initialize the parameters of the 2-layer neural network. The model's structure is: *LINEAR -> RELU -> LINEAR -> SIGMOID*. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iYcIIfwO2KYF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def initialize_parameters(n_x, n_h, n_y):\n",
        "    \"\"\"\n",
        "    Argument:\n",
        "    n_x -- size of the input layer\n",
        "    n_h -- size of the hidden layer\n",
        "    n_y -- size of the output layer\n",
        "    \n",
        "    Returns:\n",
        "    parameters -- python dictionary containing your parameters:\n",
        "                    W1 -- weight matrix of shape (n_h, n_x)\n",
        "                    b1 -- bias vector of shape (n_h, 1)\n",
        "                    W2 -- weight matrix of shape (n_y, n_h)\n",
        "                    b2 -- bias vector of shape (n_y, 1)\n",
        "    \"\"\"\n",
        "    np.random.seed(1)\n",
        "\n",
        "    W1 = np.random.randn(n_h, n_x)*0.01\n",
        "    b1 = np.zeros((n_h, 1))\n",
        "    W2 = np.random.randn(n_y, n_h)*0.01\n",
        "    b2 = np.zeros((n_y,1))\n",
        "    \n",
        "    parameters = {\"W1\": W1,\n",
        "                  \"b1\": b1,\n",
        "                  \"W2\": W2,\n",
        "                  \"b2\": b2}\n",
        "    \n",
        "    return parameters    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GTk5HnNr2KYR",
        "colab_type": "text"
      },
      "source": [
        "L-layer Neural Network\n",
        "\n",
        "The initialization for a deeper L-layer neural network is more complicated because there are many more weight matrices and bias vectors. \n",
        "<table style=\"width:100%\">\n",
        "    <tr>\n",
        "        <td>  </td> \n",
        "        <td> **Shape of W** </td> \n",
        "        <td> **Shape of b**  </td> \n",
        "        <td> **Activation** </td>\n",
        "        <td> **Shape of Activation** </td> \n",
        "    <tr>\n",
        "    <tr>\n",
        "        <td> **Layer 1** </td> \n",
        "        <td> $(n^{[1]}, 12288)$ </td> \n",
        "        <td> $(n^{[1]}, 1)$ </td> \n",
        "        <td> $Z^{[1]} = W^{[1]}  X + b^{[1]} $ </td> \n",
        "        <td> $(n^{[1]}, 209)$ </td> \n",
        "    <tr>\n",
        "    <tr>\n",
        "        <td> **Layer 2** </td> \n",
        "        <td> $(n^{[2]}, n^{[1]})$  </td> \n",
        "        <td> $(n^{[2]},1)$ </td> \n",
        "        <td>$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$ </td> \n",
        "        <td> $(n^{[2]}, 209)$ </td> \n",
        "    <tr>\n",
        "       <tr>\n",
        "        <td> $\\vdots$ </td> \n",
        "        <td> $\\vdots$  </td> \n",
        "        <td> $\\vdots$  </td> \n",
        "        <td> $\\vdots$</td> \n",
        "        <td> $\\vdots$  </td>\n",
        "    <tr>\n",
        "    <tr>\n",
        "        <td> **Layer L-1** </td> \n",
        "        <td> $(n^{[L-1]}, n^{[L-2]})$ </td> \n",
        "        <td> $(n^{[L-1]}, 1)$  </td> \n",
        "        <td>$Z^{[L-1]} =  W^{[L-1]} A^{[L-2]} + b^{[L-1]}$ </td> \n",
        "        <td> $(n^{[L-1]}, 209)$ </td> \n",
        "   <tr>\n",
        "   <tr>\n",
        "        <td> **Layer L** </td> \n",
        "        <td> $(n^{[L]}, n^{[L-1]})$ </td> \n",
        "        <td> $(n^{[L]}, 1)$ </td>\n",
        "        <td> $Z^{[L]} =  W^{[L]} A^{[L-1]} + b^{[L]}$</td>\n",
        "        <td> $(n^{[L]}, 209)$  </td> \n",
        "    <tr>\n",
        "</table>\n",
        "\n",
        "$W X + b$ in python, if: \n",
        "\n",
        "$$ W = \\begin{bmatrix}\n",
        "    j  & k  & l\\\\\n",
        "    m  & n & o \\\\\n",
        "    p  & q & r \n",
        "\\end{bmatrix}\\;\\;\\; X = \\begin{bmatrix}\n",
        "    a  & b  & c\\\\\n",
        "    d  & e & f \\\\\n",
        "    g  & h & i \n",
        "\\end{bmatrix} \\;\\;\\; b =\\begin{bmatrix}\n",
        "    s  \\\\\n",
        "    t  \\\\\n",
        "    u\n",
        "\\end{bmatrix}\\tag{2}$$\n",
        "\n",
        "Then $WX + b$ will be:\n",
        "\n",
        "$$ WX + b = \\begin{bmatrix}\n",
        "    (ja + kd + lg) + s  & (jb + ke + lh) + s  & (jc + kf + li)+ s\\\\\n",
        "    (ma + nd + og) + t & (mb + ne + oh) + t & (mc + nf + oi) + t\\\\\n",
        "    (pa + qd + rg) + u & (pb + qe + rh) + u & (pc + qf + ri)+ u\n",
        "\\end{bmatrix}\\tag{3}  $$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x0AOh-lu2KYT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def initialize_parameters_deep(layer_dims):\n",
        "    \"\"\"\n",
        "    Arguments:\n",
        "    layer_dims -- python array (list) containing the dimensions of each layer in our network\n",
        "    \n",
        "    Returns:\n",
        "    parameters -- python dictionary containing your parameters \"W1\", \"b1\", ..., \"WL\", \"bL\":\n",
        "                    Wl -- weight matrix of shape (layer_dims[l], layer_dims[l-1])\n",
        "                    bl -- bias vector of shape (layer_dims[l], 1)\n",
        "    \"\"\"\n",
        "    \n",
        "    np.random.seed(3)\n",
        "    parameters = {}\n",
        "    L = len(layer_dims)            # number of layers in the network\n",
        "\n",
        "    for l in range(1, L):\n",
        "        parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l-1])*0.01\n",
        "        parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))\n",
        "\n",
        "    return parameters"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AVbtpttM2KYf",
        "colab_type": "text"
      },
      "source": [
        "Linear Forward \n",
        "\n",
        "Computes the following equations:\n",
        "\n",
        "$$Z^{[l]} = W^{[l]}A^{[l-1]} +b^{[l]}\\tag{4}$$\n",
        "\n",
        "where $A^{[0]} = X$. \n",
        "\n",
        "Build the linear part of forward propagation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wcryvtVL2KYg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def linear_forward(A, W, b):\n",
        "    \"\"\"\n",
        "    Implement the linear part of a layer's forward propagation.\n",
        "\n",
        "    Arguments:\n",
        "    A -- activations from previous layer (or input data): (size of previous layer, number of examples)\n",
        "    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)\n",
        "    b -- bias vector, numpy array of shape (size of the current layer, 1)\n",
        "\n",
        "    Returns:\n",
        "    Z -- the input of the activation function, also called pre-activation parameter \n",
        "    cache -- a python tuple containing \"A\", \"W\" and \"b\" ; stored for computing the backward pass efficiently\n",
        "    \"\"\"\n",
        "    Z = np.dot(W,A)+b\n",
        "    cache = (A, W, b)\n",
        "    \n",
        "    return Z, cache"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gnda3anC2KYs",
        "colab_type": "text"
      },
      "source": [
        "Implement the forward propagation of the *LINEAR->ACTIVATION* layer. Mathematical relation is: $A^{[l]} = g(Z^{[l]}) = g(W^{[l]}A^{[l-1]} +b^{[l]})$ where the activation \"g\" can be sigmoid() or relu(). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1VbTUnmm2KYt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def linear_activation_forward(A_prev, W, b, activation):\n",
        "    \"\"\"\n",
        "    Implement the forward propagation for the LINEAR->ACTIVATION layer\n",
        "\n",
        "    Arguments:\n",
        "    A_prev -- activations from previous layer (or input data): (size of previous layer, number of examples)\n",
        "    W -- weights matrix: numpy array of shape (size of current layer, size of previous layer)\n",
        "    b -- bias vector, numpy array of shape (size of the current layer, 1)\n",
        "    activation -- the activation to be used in this layer, stored as a text string: \"sigmoid\" or \"relu\"\n",
        "\n",
        "    Returns:\n",
        "    A -- the output of the activation function, also called the post-activation value \n",
        "    cache -- a python tuple containing \"linear_cache\" and \"activation_cache\";\n",
        "             stored for computing the backward pass efficiently\n",
        "    \"\"\"\n",
        "    \n",
        "    if activation == \"sigmoid\":\n",
        "        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n",
        "        Z, linear_cache = linear_forward(A_prev, W, b)\n",
        "        A, activation_cache = sigmoid(Z)\n",
        "    elif activation == \"relu\":\n",
        "        # Inputs: \"A_prev, W, b\". Outputs: \"A, activation_cache\".\n",
        "        Z, linear_cache = linear_forward(A_prev, W, b)\n",
        "        A, activation_cache = relu(Z)\n",
        "    \n",
        "    assert (A.shape == (W.shape[0], A_prev.shape[1]))\n",
        "    cache = (linear_cache, activation_cache)\n",
        "\n",
        "    return A, cache"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EB7XefIt2KZD",
        "colab_type": "text"
      },
      "source": [
        "L-Layer Model. Implement the forward propagation. In the code below, the variable `AL` will denote $A^{[L]} = \\sigma(Z^{[L]}) = \\sigma(W^{[L]} A^{[L-1]} + b^{[L]})$.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ibxoGZur2KZE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def L_model_forward(X, parameters):\n",
        "    \"\"\"\n",
        "    Implement forward propagation for the [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID computation\n",
        "    \n",
        "    Arguments:\n",
        "    X -- data, numpy array of shape (input size, number of examples)\n",
        "    parameters -- output of initialize_parameters_deep()\n",
        "    \n",
        "    Returns:\n",
        "    AL -- last post-activation value\n",
        "    caches -- list of caches containing:\n",
        "                every cache of linear_activation_forward() (there are L-1 of them, indexed from 0 to L-1)\n",
        "    \"\"\"\n",
        "\n",
        "    caches = []\n",
        "    A = X\n",
        "    L = len(parameters) // 2      # number of layers in the neural network\n",
        "    \n",
        "    # Implement [LINEAR -> RELU]*(L-1). Add \"cache\" to the \"caches\" list.\n",
        "    for l in range(1, L):\n",
        "        A_prev = A \n",
        "        A, cache = linear_activation_forward(A_prev, parameters['W'+str(l)], parameters['b'+str(l)], activation='relu')\n",
        "        caches.append(cache)\n",
        "    \n",
        "    # Implement LINEAR -> SIGMOID. Add \"cache\" to the \"caches\" list.\n",
        "    AL, cache =  linear_activation_forward(A,parameters['W'+str(L)],parameters['b'+str(L)], activation='sigmoid')\n",
        "    caches.append(cache)\n",
        "    \n",
        "    assert(AL.shape == (1,X.shape[1]))\n",
        "            \n",
        "    return AL, caches"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUazQeHP2KZP",
        "colab_type": "text"
      },
      "source": [
        "Compute the cross-entropy cost $J$, using the following formula: $$-\\frac{1}{m} \\sum\\limits_{i = 1}^{m} (y^{(i)}\\log\\left(a^{[L] (i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[L](i)}\\right)) \\tag{7}$$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rpjBFUjO2KZQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def compute_cost(AL, Y):\n",
        "    \"\"\"\n",
        "    Implement the cost function defined by equation (7).\n",
        "\n",
        "    Arguments:\n",
        "    AL -- probability vector corresponding to your label predictions, shape (1, number of examples)\n",
        "    Y -- true \"label\" vector (for example: containing 0 if non-cat, 1 if cat), shape (1, number of examples)\n",
        "\n",
        "    Returns:\n",
        "    cost -- cross-entropy cost\n",
        "    \"\"\"\n",
        "    m = Y.shape[1]\n",
        "\n",
        "    # Compute loss from aL and y.\n",
        "    cost = (-1/m)*np.sum(Y*np.log(AL)+(1-Y)*np.log(1-AL))\n",
        "    cost = np.squeeze(cost)\n",
        "    \n",
        "    assert(cost.shape == ())\n",
        "    \n",
        "    return cost"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eps5SRaY2KZb",
        "colab_type": "text"
      },
      "source": [
        "### 6.1 - Linear backward\n",
        "\n",
        "For layer $l$, the linear part is: $Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}$ (followed by an activation).\n",
        "\n",
        "Suppose you have already calculated the derivative $dZ^{[l]} = \\frac{\\partial \\mathcal{L} }{\\partial Z^{[l]}}$. You want to get $(dW^{[l]}, db^{[l]}, dA^{[l-1]})$.\n",
        "\n",
        "The three outputs $(dW^{[l]}, db^{[l]}, dA^{[l-1]})$ are computed using the input $dZ^{[l]}$.Here are the formulas you need:\n",
        "$$ dW^{[l]} = \\frac{\\partial \\mathcal{J} }{\\partial W^{[l]}} = \\frac{1}{m} dZ^{[l]} A^{[l-1] T} \\tag{8}$$\n",
        "$$ db^{[l]} = \\frac{\\partial \\mathcal{J} }{\\partial b^{[l]}} = \\frac{1}{m} \\sum_{i = 1}^{m} dZ^{[l](i)}\\tag{9}$$\n",
        "$$ dA^{[l-1]} = \\frac{\\partial \\mathcal{L} }{\\partial A^{[l-1]}} = W^{[l] T} dZ^{[l]} \\tag{10}$$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RDDbyz22KZd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def linear_backward(dZ, cache):\n",
        "    \"\"\"\n",
        "    Implement the linear portion of backward propagation for a single layer (layer l)\n",
        "\n",
        "    Arguments:\n",
        "    dZ -- Gradient of the cost with respect to the linear output (of current layer l)\n",
        "    cache -- tuple of values (A_prev, W, b) coming from the forward propagation in the current layer\n",
        "\n",
        "    Returns:\n",
        "    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev\n",
        "    dW -- Gradient of the cost with respect to W (current layer l), same shape as W\n",
        "    db -- Gradient of the cost with respect to b (current layer l), same shape as b\n",
        "    \"\"\"\n",
        "    A_prev, W, b = cache\n",
        "    m = A_prev.shape[1]\n",
        "\n",
        "    dW = (1/m)*np.dot(dZ, A_prev.T)\n",
        "    db = (1/m)*np.sum(dZ, axis=1, keepdims=True)\n",
        "    dA_prev = np.dot(W.T, dZ)\n",
        "    \n",
        "    assert (dA_prev.shape == A_prev.shape)\n",
        "    assert (dW.shape == W.shape)\n",
        "    assert (db.shape == b.shape)\n",
        "    \n",
        "    return dA_prev, dW, db"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0jpYBYb62KZm",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "If $g(.)$ is the activation function, \n",
        "`sigmoid_backward` and `relu_backward` compute $$dZ^{[l]} = dA^{[l]} * g'(Z^{[l]}) \\tag{11}$$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_pVPd0MO2KZn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def linear_activation_backward(dA, cache, activation):\n",
        "    \"\"\"\n",
        "    Implement the backward propagation for the LINEAR->ACTIVATION layer.\n",
        "    \n",
        "    Arguments:\n",
        "    dA -- post-activation gradient for current layer l \n",
        "    cache -- tuple of values (linear_cache, activation_cache) we store for computing backward propagation efficiently\n",
        "    activation -- the activation to be used in this layer, stored as a text string: \"sigmoid\" or \"relu\"\n",
        "    \n",
        "    Returns:\n",
        "    dA_prev -- Gradient of the cost with respect to the activation (of the previous layer l-1), same shape as A_prev\n",
        "    dW -- Gradient of the cost with respect to W (current layer l), same shape as W\n",
        "    db -- Gradient of the cost with respect to b (current layer l), same shape as b\n",
        "    \"\"\"\n",
        "    linear_cache, activation_cache = cache\n",
        "    \n",
        "    if activation == \"relu\":\n",
        "        dZ = relu_backward(dA,activation_cache)\n",
        "        dA_prev, dW, db = linear_backward(dZ, linear_cache)\n",
        "        \n",
        "    elif activation == \"sigmoid\":\n",
        "        dZ = sigmoid_backward(dA, activation_cache)\n",
        "        dA_prev, dW, db = linear_backward(dZ, linear_cache)\n",
        "    \n",
        "    return dA_prev, dW, db"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FbGYQqE_2KZw",
        "colab_type": "text"
      },
      "source": [
        "To backpropagate through this network, we know that the output is, \n",
        "$A^{[L]} = \\sigma(Z^{[L]})$. Your code thus needs to compute `dAL` $= \\frac{\\partial \\mathcal{L}}{\\partial A^{[L]}}$.\n",
        "```python\n",
        "dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL)) # derivative of cost with respect to AL\n",
        "```\n",
        "\n",
        "$$grads[\"dW\" + str(l)] = dW^{[l]}\\tag{15} $$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0GTYPeHr2KZx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# GRADED FUNCTION: L_model_backward\n",
        "\n",
        "def L_model_backward(AL, Y, caches):\n",
        "    \"\"\"\n",
        "    Implement the backward propagation for the [LINEAR->RELU] * (L-1) -> LINEAR -> SIGMOID group\n",
        "    \n",
        "    Arguments:\n",
        "    AL -- probability vector, output of the forward propagation (L_model_forward())\n",
        "    Y -- true \"label\" vector (containing 0 if non-cat, 1 if cat)\n",
        "    caches -- list of caches containing:\n",
        "                every cache of linear_activation_forward() with \"relu\" (it's caches[l], for l in range(L-1) i.e l = 0...L-2)\n",
        "                the cache of linear_activation_forward() with \"sigmoid\" (it's caches[L-1])\n",
        "    \n",
        "    Returns:\n",
        "    grads -- A dictionary with the gradients\n",
        "             grads[\"dA\" + str(l)] = ... \n",
        "             grads[\"dW\" + str(l)] = ...\n",
        "             grads[\"db\" + str(l)] = ... \n",
        "    \"\"\"\n",
        "    grads = {}\n",
        "    L = len(caches) # the number of layers\n",
        "    m = AL.shape[1]\n",
        "    Y = Y.reshape(AL.shape) # after this line, Y is the same shape as AL\n",
        "    \n",
        "    # Initializing the backpropagation\n",
        "    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL)) \n",
        "    \n",
        "    # Lth layer (SIGMOID -> LINEAR) gradients. Inputs: \"dAL, current_cache\". Outputs: \"grads[\"dAL-1\"], grads[\"dWL\"], grads[\"dbL\"]\n",
        "    current_cache = caches[L-1]\n",
        "    grads[\"dA\" + str(L-1)], grads[\"dW\" + str(L)], grads[\"db\" + str(L)] = linear_activation_backward(dAL,current_cache,'sigmoid')\n",
        "    \n",
        "    # Loop from l=L-2 to l=0\n",
        "    for l in reversed(range(L-1)):\n",
        "        # lth layer: (RELU -> LINEAR) gradients.\n",
        "        # Inputs: \"grads[\"dA\" + str(l + 1)], current_cache\". Outputs: \"grads[\"dA\" + str(l)] , grads[\"dW\" + str(l + 1)] , grads[\"db\" + str(l + 1)] \n",
        "        current_cache = caches[l]\n",
        "        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads[\"dA\" + str(l + 1)], current_cache, 'relu')\n",
        "        grads[\"dA\" + str(l)] = dA_prev_temp\n",
        "        grads[\"dW\" + str(l + 1)] = dW_temp\n",
        "        grads[\"db\" + str(l + 1)] = db_temp\n",
        "\n",
        "    return grads"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bd5Aqr152KZ8",
        "colab_type": "text"
      },
      "source": [
        "Update the parameters of the model, using gradient descent: \n",
        "\n",
        "$$ W^{[l]} = W^{[l]} - \\alpha \\text{ } dW^{[l]} \\tag{16}$$\n",
        "$$ b^{[l]} = b^{[l]} - \\alpha \\text{ } db^{[l]} \\tag{17}$$\n",
        "\n",
        "where $\\alpha$ is the learning rate. After computing the updated parameters, store them in the parameters dictionary. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kP924cxo2KZ9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def update_parameters(parameters, grads, learning_rate):\n",
        "    \"\"\"\n",
        "    Update parameters using gradient descent\n",
        "    \n",
        "    Arguments:\n",
        "    parameters -- python dictionary containing your parameters \n",
        "    grads -- python dictionary containing your gradients, output of L_model_backward\n",
        "    \n",
        "    Returns:\n",
        "    parameters -- python dictionary containing your updated parameters \n",
        "                  parameters[\"W\" + str(l)] = ... \n",
        "                  parameters[\"b\" + str(l)] = ...\n",
        "    \"\"\"\n",
        "    \n",
        "    L = len(parameters) // 2 # number of layers in the neural network\n",
        "\n",
        "    # Update rule for each parameter. Use a for loop.\n",
        "    for l in range(L):\n",
        "        parameters[\"W\" + str(l+1)] = parameters[\"W\" + str(l+1)] - learning_rate * grads[\"dW\" + str(l+1)]\n",
        "        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - learning_rate * grads[\"db\" + str(l+1)]\n",
        "    return parameters"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}