Machine-Learning-Fundamentals/regression_logistic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "---\n",
    "- Author: Diego Inácio\n",
    "- GitHub: [github.com/diegoinacio](https://github.com/diegoinacio)\n",
    "- Notebook: [regression_logistic.ipynb](https://github.com/diegoinacio/machine-learning-notebooks/blob/master/Machine-Learning-Fundamentals/regression_logistic.ipynb)\n",
    "---\n",
    "Overview and implementation of *Logistic Regression* analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from regression__utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\large\n",
    "h_{\\theta}(x)=g(\\theta^Tx)=\\frac{e^{\\theta^Tx}}{1+e^{\\theta^Tx}}=\\frac{1}{1+e^{-\\theta^Tx}}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$ \\large\n",
    "\\theta^Tx=\n",
    "\\begin{bmatrix}\n",
    "    \\theta_0 \\\\\n",
    "    \\theta_1 \\\\\n",
    "    \\vdots \\\\\n",
    "    \\theta_i\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "    1 & x_{11} & \\cdots & x_{1i} \\\\\n",
    "    1 & x_{21} & \\cdots & x_{2i} \\\\\n",
    "    \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "    1 & x_{n1} & \\cdots & x_{ni}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $\\large h_\\theta(x)$ is the hypothesis;\n",
    "- $\\large g(z)$ is the logistic function or <em>sigmoid</em>;\n",
    "- $\\large \\theta_i$ is the parameters (or <em>weights</em>)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def arraycast(f):\n",
    "    '''\n",
    "    Decorator for vectors and matrices cast\n",
    "    '''\n",
    "    def wrap(self, *X, y=[]):\n",
    "        X = np.array(X)\n",
    "        X = np.insert(X.T, 0, 1, 1)\n",
    "        if list(y):\n",
    "            y = np.array(y)[np.newaxis]\n",
    "            return f(self, X, y)\n",
    "        return f(self, X)\n",
    "    return wrap\n",
    "\n",
    "class logisticRegression(object):\n",
    "    def __init__(self, rate=0.001, iters=1024):\n",
    "        self._rate = rate\n",
    "        self._iters = iters\n",
    "        self._theta = None\n",
    "    @property\n",
    "    def theta(self):\n",
    "        return self._theta\n",
    "    def _sigmoid(self, Z):\n",
    "        return 1/(1 + np.exp(-Z))\n",
    "    def _dsigmoid(self, Z):\n",
    "        return self._sigmoid(Z)*(1 - self._sigmoid(Z))\n",
    "    @arraycast\n",
    "    def fit(self, X, y=[]):\n",
    "        self._theta = np.ones((1, X.shape[-1]))\n",
    "        for i in range(self._iters):\n",
    "            thetaTx = np.dot(X, self._theta.T)\n",
    "            h = self._sigmoid(thetaTx)\n",
    "            delta = h - y.T\n",
    "            grad = np.dot(X.T, delta).T\n",
    "            self._theta -= grad*self._rate\n",
    "    @arraycast\n",
    "    def pred(self, x):\n",
    "        return self._sigmoid(np.dot(x, self._theta.T)) > 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Synthetic data 5\n",
    "x1, x2, y = synthData5()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logistic regression data](output/regression_logistic_data.png \"Logistic Regression Data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Training\n",
    "rlogb = logisticRegression(rate=0.001, iters=512)\n",
    "rlogb.fit(x1, x2, y=y)\n",
    "rlogb.pred(x1, x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logistic regression training](output/regression_logistic_gradDesc.gif \"Logistic Regression Training\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the boundary line components:\n",
    "\n",
    "$$ \\large\n",
    "    \\theta_0+\\theta_1 x_1+\\theta_2 x_2=0\n",
    "$$\n",
    "\n",
    "Considering $\\large x_2$ as the dependent variable:\n",
    "\n",
    "$$ \\large\n",
    "    x_2=-\\frac{\\theta_0+\\theta_1 x_1}{\\theta_2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prediction\n",
    "w0, w1, w2 = rlogb.theta[0]\n",
    "f = lambda x: - (w0 + w1*x)/w2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![regressão logística prediction](output/regression_logistic_pred.png \"Logistic Regression Prediction\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}