11. Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "\n",
    "© 2018 Daniel Voigt Godoy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "Logistic Regression is used for ***classification***, even though its called ***regression***.\n",
    "\n",
    "Therefore, it works on ***categorical labels***, namely, 0 and 1 for ***binary classification***. \n",
    "\n",
    "The ***Logistic Regression*** is a model that makes predictions in the [0, 1] interval, denoting ***probabilities***. Labels of the ***negative class*** are associated with 0, as labels of the ***positive class*** are associated with 1. So, the output is the ***probability of being a sample of the positive class***.\n",
    "\n",
    "Why is it called ***regression*** then? It actually fits a ***linear regression*** on the features and ***squishes*** the outputs using a ***Logistic / Sigmoid*** function.\n",
    "\n",
    "$$\n",
    "\\hat{p} = \\sigma(z) = \\frac{1}{1 + e^{-z}} = \\frac{1}{1 + e^{-(b + w_1x_1 + w_2x_2 + \\dots + w_nx_n)}}\n",
    "$$\n",
    "\n",
    "![sigmoid](https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/320px-Logistic-curve.svg.png)\n",
    "<center>Source: Wikipedia</center>\n",
    "\n",
    "Since its output is a ***probability***, we need to ***threshold*** it to get the predicted class. The default threshold is 0.5:\n",
    "\n",
    "$$\n",
    "\\hat{y} = \n",
    "\\begin{cases} 0 &\\mbox{if } \\hat{p} \\lt 0.5 \\\\\n",
    "1 & \\mbox{if } \\hat{p} \\geq 0.5\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "It is possible to ***change the threshold*** to achieve different goals, as reducing ***false positive*** or ***false negatives*** for instance - the next lesson on evaluation metrics will cover this topic in more depth.\n",
    "\n",
    "You can observe this behavior on the ***interactive example*** below. The sliders allow you to control the ***bias*** and ***coefficient*** of the simple linear regression that is going to be ***squished*** by the ***sigmoid function***."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.supervised.classification.LogisticRegression import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdcd5c6532c74ea0b42bb8bfcaba1943",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'line': {'color': 'black'},\n",
       "              'name': 'Sigmoid',\n",
       "     …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plotLogistic.plot_sigmoid_curve(x=np.linspace(-3, 3, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Loss Function\n",
    "\n",
    "How do we train the model? Differently from a linear regression, the ***Logistic Regression*** uses ***binary cross entropy*** (also called ***log loss***) as its loss function.\n",
    "\n",
    "What does it mean? It takes the ***log*** of the probability of ***correctly classifying*** a sample as positive or negative and then average it over all samples. For a single instance:\n",
    "\n",
    "$$\n",
    "loss = \n",
    "\\begin{cases} -log(\\hat{p}) &\\mbox{if } y = 1 \\\\\n",
    "-log(1-\\hat{p}) & \\mbox{if } y = 0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "And, for all $m$ instances:\n",
    "\n",
    "$$\n",
    "J={-\\frac{1}{m}\\sum_{i=1}^{m}{y^{(i)} \\cdot log(\\hat{p}(y^{(i)})) + (1-y^{(i)}) \\cdot log(1-\\hat{p}(y^{(i)}))}}\n",
    "$$\n",
    "\n",
    "I've written a very thorough explanation of this loss function on Towards Data Science: [Understanding binary cross-entropy / log loss: a visual explanation](https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself!\n",
    "\n",
    "You have 8 data points, either ***green (positive)*** or ***red (negative)***.\n",
    "\n",
    "There is only ***one feature***, represented on the horizontal axis. The ***y axis*** is the probability output of your ***Logistic Regression***.\n",
    "\n",
    "You want to start training your logistic regression, so you need to find both the ***bias*** (intercept) $b$ and the ***single weight*** $w_1$ that minimize the ***log loss***.\n",
    "\n",
    "The sliders below allow you to change both values, and you can observe the effect they have on the distribution of losses (on the upper right plot), as well as the ***log loss***.\n",
    "\n",
    "Use the slider to play with different configurations and answer the ***questions*** below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylr = plotLogistic(x=(-1, 0), n_samples=8, betas=(2, 1))\n",
    "vb1 = VBox(build_figure_fit(mylr))\n",
    "vb1.layout.align_items = 'center'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "697145cc678343578f80d8eb23d189bb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'line': {'color': 'red', 'dash': 'dash', 'width': 3},\n",
       "            …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. What happens to the probabilities as you increase $w_1$? What about the losses?\n",
    "\n",
    "2. What happens to the probabilities as you increase $b$? What about the losses?\n",
    "\n",
    "3. Try to ***minimize*** the log loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Scikit-Learn\n",
    "\n",
    "[Logistic Regression](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression)\n",
    "\n",
    "Please check Aurelién Geron's \"Hand-On Machine Learning with Scikit-Learn and Tensorflow\" notebook on Linear Models [here](http://nbviewer.jupyter.org/github/ageron/handson-ml/blob/master/04_training_linear_models.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. More Resources\n",
    "\n",
    "[InfoGraphic](https://github.com/Avik-Jain/100-Days-Of-ML-Code/blob/master/Info-graphs/Day%204.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Keras\n",
    "\n",
    "Just like we did with Linear Regression, we can also build a simple one neuron network to train a ***Logistic Regression***. The model has ***two*** differences:\n",
    "\n",
    "1. It has a ***sigmoid activation*** (instead of linear)\n",
    "2. It uses ***binary cross-entropy*** as loss (instead of MSE)\n",
    "\n",
    "Effectively, it computes:\n",
    "\n",
    "$$\n",
    "z = \\sigma(b + w_1x)\n",
    "$$\n",
    "\n",
    "If you compare the ***weights*** of a model trained using Keras versus a model trained using Scikit-Learn, you'll see they are somewhat different.\n",
    "\n",
    "This is due to the fact that Scikit-Learn uses a different optimizer and uses regularization by default. This is ***not*** the case of the simple neural network we built using Keras.\n",
    "\n",
    "```python\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(input_dim=1, units=1, activation='sigmoid', kernel_initializer='glorot_uniform'))\n",
    "model.compile(loss='binary_crossentropy', optimizer=SGD(lr=0.1))\n",
    "model.fit(m.x, m.y, epochs=100)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 1)                 2         \n",
      "=================================================================\n",
      "Total params: 2\n",
      "Trainable params: 2\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(input_dim=1, units=1, activation='sigmoid', kernel_initializer='glorot_uniform'))\n",
    "model.compile(loss='binary_crossentropy', optimizer=SGD(lr=0.1))\n",
    "model.fit(mylr.x, mylr.y, epochs=100, verbose=False)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "model.get_weights()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[1.2090513]], dtype=float32), array([-0.45385373], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "print(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}