Machine-Learning-Fundamentals/regression_polynomial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Polynomial Regression\n",
    "---\n",
    "- Author: Diego Inácio\n",
    "- GitHub: [github.com/diegoinacio](https://github.com/diegoinacio)\n",
    "- Notebook: [regression_polynomial.ipynb](https://github.com/diegoinacio/machine-learning-notebooks/blob/master/Machine-Learning-Fundamentals/regression_polynomial.ipynb)\n",
    "---\n",
    "Overview and implementation of *Polynomial Regression* analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from regression__utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the function:\n",
    "\n",
    "$$ \\large\n",
    "    f(x)=x^3-3x^2+x+1+\\epsilon\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Synthetic data 6\n",
    "x, y = synthData6()\n",
    "\n",
    "# Predicting with Linear Regression\n",
    "# lrs = linearRegression_simple()\n",
    "# lrs.fit(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![polynomial data and linear regression](output/regression_polynomial_linear.png \"Polynomial data and Linear Regression\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algorithm\n",
    "---\n",
    "$$ \\large\n",
    "\\vec{y}=\\mathbf{X}\\vec{\\mathbf{\\beta}}+\\vec{\\epsilon}\n",
    "$$\n",
    "\n",
    "where $\\large \\mathbf{X}$ (or $\\large \\mathbf{V}$) is the *Vandermonde's matrix* of the independent variable, parametrised by the maximum degree $\\large m$, a response vector $\\large \\vec{y}$, a parameter vector $\\large \\vec{\\mathbf{\\beta}}$ and a random error vector $\\large \\vec{\\epsilon}$. In the form of a system of linear equations, we have:\n",
    "\n",
    "$$ \\large\n",
    "\\begin{bmatrix}\n",
    "    y_1 \\\\\n",
    "    y_2 \\\\\n",
    "    y_3 \\\\\n",
    "    \\vdots \\\\\n",
    "    y_n\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "    1 & x_1 & x_1^2 &\\cdots & x_1^m \\\\\n",
    "    1 & x_2 & x_2^2 & \\cdots & x_2^m \\\\\n",
    "    1 & x_3 & x_3^2 & \\cdots & x_3^m \\\\\n",
    "    \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "    1 & x_n & x_n^2 & \\cdots & x_n^m\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "    \\beta_1 \\\\\n",
    "    \\beta_2 \\\\\n",
    "    \\beta_3 \\\\\n",
    "    \\vdots \\\\\n",
    "    \\beta_m\n",
    "\\end{bmatrix}\n",
    "+\n",
    "\\begin{bmatrix}\n",
    "    \\epsilon_1 \\\\\n",
    "    \\epsilon_2 \\\\\n",
    "    \\epsilon_3 \\\\\n",
    "    \\vdots \\\\\n",
    "    \\epsilon_n\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "By means of the Least Squares Method, the estimated coefficient vector is given by:\n",
    "\n",
    "$$ \\large\n",
    "\\widehat{\\vec{\\mathbf{\\beta}}}=(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\vec{y}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def arraycast(f):\n",
    "    '''\n",
    "    Decorador para conversão de vetores e matrizes\n",
    "    '''\n",
    "    def wrap(self, X, y=[]):\n",
    "        X = np.array(X)\n",
    "        if list(y):\n",
    "            y = np.array(y)\n",
    "            return f(self, X, y)\n",
    "        return f(self, X)\n",
    "    return wrap\n",
    "\n",
    "class polynomialRegression(object):\n",
    "    def __init__(self, degree=1):\n",
    "        self._degree = degree\n",
    "        self._beta = None\n",
    "    @property\n",
    "    def beta(self):\n",
    "        return self._beta\n",
    "    @arraycast\n",
    "    def fit(self, X, y=[]):\n",
    "        V = np.stack([X**i for i in range(self._degree + 1)], axis=0).T\n",
    "        VTV = np.dot(V.T, V)\n",
    "        VTV_i = np.linalg.inv(VTV)\n",
    "        Vi = np.dot(VTV_i, V.T)\n",
    "        self._beta = np.dot(Vi, y)\n",
    "    @arraycast\n",
    "    def pred(self, x):\n",
    "        V = np.stack([x**i for i in range(self._degree + 1)], axis=0).T\n",
    "        return np.dot(V, self._beta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that our class has an attribute called <em>degree</em> which is the maximum degree of our function $\\large f(x)$. In our example it should be $\\large m=3$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "polreg = polynomialRegression(3)\n",
    "polreg.fit(x, y=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![polynomial regression](output/regression_polynomial_pred.png \"Polynomial Regression\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}