code/day8 sigmoid function.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab: Logistic Regression\n",
    "\n",
    "In this ungraded lab, you will \n",
    "- explore the sigmoid function (also known as the logistic function)\n",
    "- explore logistic regression; which uses the sigmoid function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib widget\n",
    "import matplotlib.pyplot as plt\n",
    "from plt_one_addpt_onclick import plt_one_addpt_onclick\n",
    "from lab_utils_common import draw_vthresh\n",
    "plt.style.use('./deeplearning.mplstyle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoid or Logistic Function\n",
    "<img align=\"left\" src=\"./images/C1_W3_LogisticRegression_left.png\"     style=\" width:300px; padding: 10px; \" >As discussed in the lecture videos, for a classification task, we can start by using our linear regression model, $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = \\mathbf{w} \\cdot  \\mathbf{x}^{(i)} + b$, to predict $y$ given $x$. \n",
    "- However, we would like the predictions of our classification model to be between 0 and 1 since our output variable $y$ is either 0 or 1. \n",
    "- This can be accomplished by using a \"sigmoid function\" which maps all input values to values between 0 and 1. \n",
    "\n",
    "\n",
    "Let's implement the sigmoid function and see this for ourselves.\n",
    "\n",
    "## Formula for Sigmoid function\n",
    "\n",
    "The formula for a sigmoid function is as follows -  \n",
    "\n",
    "$g(z) = \\frac{1}{1+e^{-z}}\\tag{1}$\n",
    "\n",
    "In the case of logistic regression, z (the input to the sigmoid function), is the output of a linear regression model. \n",
    "- In the case of a single example, $z$ is scalar.\n",
    "- in the case of multiple examples, $z$ may be a vector consisting of $m$ values, one for each example. \n",
    "- The implementation of the sigmoid function should cover both of these potential input formats.\n",
    "Let's implement this in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy has a function called [`exp()`](https://numpy.org/doc/stable/reference/generated/numpy.exp.html), which offers a convenient way to calculate the exponential ( $e^{z}$) of all elements in the input array (`z`).\n",
    " \n",
    "It also works with a single number as an input, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input to exp: [1 2 3]\n",
      "Output of exp: [ 2.72  7.39 20.09]\n",
      "Input to exp: 1\n",
      "Output of exp: 2.718281828459045\n"
     ]
    }
   ],
   "source": [
    "# Input is an array. \n",
    "input_array = np.array([1,2,3])\n",
    "exp_array = np.exp(input_array)\n",
    "\n",
    "print(\"Input to exp:\", input_array)\n",
    "print(\"Output of exp:\", exp_array)\n",
    "\n",
    "# Input is a single number\n",
    "input_val = 1  \n",
    "exp_val = np.exp(input_val)\n",
    "\n",
    "print(\"Input to exp:\", input_val)\n",
    "print(\"Output of exp:\", exp_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sigmoid` function is implemented in python as shown in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    \"\"\"\n",
    "    Compute the sigmoid of z\n",
    "\n",
    "    Args:\n",
    "        z (ndarray): A scalar, numpy array of any size.\n",
    "\n",
    "    Returns:\n",
    "        g (ndarray): sigmoid(z), with the same shape as z\n",
    "         \n",
    "    \"\"\"\n",
    "\n",
    "    g = 1/(1+np.exp(-z))\n",
    "   \n",
    "    return g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the output of this function is for various value of `z`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input (z), Output (sigmoid(z))\n",
      "[[-1.000e+01  4.540e-05]\n",
      " [-9.000e+00  1.234e-04]\n",
      " [-8.000e+00  3.354e-04]\n",
      " [-7.000e+00  9.111e-04]\n",
      " [-6.000e+00  2.473e-03]\n",
      " [-5.000e+00  6.693e-03]\n",
      " [-4.000e+00  1.799e-02]\n",
      " [-3.000e+00  4.743e-02]\n",
      " [-2.000e+00  1.192e-01]\n",
      " [-1.000e+00  2.689e-01]\n",
      " [ 0.000e+00  5.000e-01]\n",
      " [ 1.000e+00  7.311e-01]\n",
      " [ 2.000e+00  8.808e-01]\n",
      " [ 3.000e+00  9.526e-01]\n",
      " [ 4.000e+00  9.820e-01]\n",
      " [ 5.000e+00  9.933e-01]\n",
      " [ 6.000e+00  9.975e-01]\n",
      " [ 7.000e+00  9.991e-01]\n",
      " [ 8.000e+00  9.997e-01]\n",
      " [ 9.000e+00  9.999e-01]\n",
      " [ 1.000e+01  1.000e+00]]\n"
     ]
    }
   ],
   "source": [
    "# Generate an array of evenly spaced values between -10 and 10\n",
    "z_tmp = np.arange(-10,11)\n",
    "\n",
    "# Use the function implemented above to get the sigmoid values\n",
    "y = sigmoid(z_tmp)\n",
    "\n",
    "# Code for pretty printing the two arrays next to each other\n",
    "np.set_printoptions(precision=3) \n",
    "print(\"Input (z), Output (sigmoid(z))\")\n",
    "print(np.c_[z_tmp, y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values in the left column are `z`, and the values in the right column are `sigmoid(z)`. As you can see, the input values to the sigmoid range from -10 to 10, and the output values range from 0 to 1. \n",
    "\n",
    "Now, let's try to plot this function using the `matplotlib` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a06cf1747b4b40348f86661ed0e21ec9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot z vs sigmoid(z)\n",
    "fig,ax = plt.subplots(1,1,figsize=(5,3))\n",
    "ax.plot(z_tmp, y, c=\"b\")\n",
    "\n",
    "ax.set_title(\"Sigmoid function\")\n",
    "ax.set_ylabel('sigmoid(z)')\n",
    "ax.set_xlabel('z')\n",
    "draw_vthresh(ax,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the sigmoid function approaches  `0` as `z` goes to large negative values and approaches `1` as `z` goes to large positive values.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "<img align=\"left\" src=\"./images/C1_W3_LogisticRegression_right.png\"     style=\" width:300px; padding: 10px; \" > A logistic regression model applies the sigmoid to the familiar linear regression model as shown below:\n",
    "\n",
    "$$ f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = g(\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b ) \\tag{2} $$ \n",
    "\n",
    "  where\n",
    "\n",
    "  $g(z) = \\frac{1}{1+e^{-z}}\\tag{3}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n",
    "Let's apply logistic regression to the categorical data example of tumor classification.  \n",
    "First, load the examples and initial values for the parameters.\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x_train = np.array([0., 1, 2, 3, 4, 5])\n",
    "y_train = np.array([0,  0, 0, 1, 1, 1])\n",
    "\n",
    "w_in = np.zeros((1))\n",
    "b_in = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try the following steps:\n",
    "- Click on 'Run Logistic Regression' to find the best logistic regression model for the given training data\n",
    "    - Note the resulting model fits the data quite well.\n",
    "    - Note, the orange line is '$z$' or $\\mathbf{w} \\cdot \\mathbf{x}^{(i)} + b$  above. It does not match the line in a linear regression model.\n",
    "Further improve these results by applying a *threshold*. \n",
    "- Tick the box on the 'Toggle 0.5 threshold' to show the predictions if a threshold is applied.\n",
    "    - These predictions look good. The predictions match the data\n",
    "    - Now, add further data points in the large tumor size range (near 10), and re-run logistic regression.\n",
    "    - unlike the linear regression model, this model continues to make correct predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9e5b061389ce4ecb8818a79d30951011",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.close('all') \n",
    "addpt = plt_one_addpt_onclick( x_train,y_train, w_in, b_in, logistic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "You have explored the use of the sigmoid function in logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}