Linear_Regression_Theory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear Regression\n",
    "========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept and Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regression analysis explores the relationship between a quantitative response variable and one or more explanatory variables. If there is only one explanatory variable, it is called **Simple Linear Regression**. If there are more explanatory variables, it is called **Multiple Linear Regression**.<p>\n",
    "Linear Regression is a statistical method that enables us to summarize and study the relationship between two variables: <br>\n",
    "* **Dependent variable**: This is the variable whose value we want to explain or forecast. Its values depend on something else. We will denote the dependent variable as $y$.\n",
    "* **Independent variable**: This is the variable that explains the other one. Its values are independent. It will be denote as $x$.\n",
    "\n",
    "Linear Regression can be used to make predictions given two or more related variables. Suppose we have two variables that are related according to the relation: \n",
    "$$\\begin{align*} y = \\beta_0 + \\beta_1 x &&\\text{[1]} \\end{align*}$$\n",
    "<p>Given a set of ($x$,$y$) points, we could represent this set by a simple line connecting all those points given the relation in equation [1]. But if the set of points does not fit exactly a line? Well, at least we can find the best line that fits this model by using **Simple Linear Regression** to find it.\n",
    "\n",
    "In this notebook I will demonstrate how we can find the line that best fits a two-variable model. Here will denote the dependent variable as **$y$** and the independent variable as $x$. Our goal is to find $\\beta_0$ and $\\beta_1$.\n",
    "\n",
    "Suppose we have a set of data representing the height (meters) and weight (kg) of a certain population given by the points below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-25T00:59:33.667028Z",
     "start_time": "2017-08-25T00:59:33.661024Z"
    },
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "height = [1.47, 1.50, 1.52, 1.55, 1.57, 1.60, 1.63, 1.65, 1.68, 1.70, 1.73, 1.75, 1.78, 1.80, 1.83]\n",
    "weight = [52.21, 53.12, 54.48, 55.84, 57.20, 58.57, 59.93, 61.29, 63.11, 64.47, 66.28, 68.10, 69.92, 72.19, 74.46]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot these points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-25T00:59:33.858181Z",
     "start_time": "2017-08-25T00:59:33.669029Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEKCAYAAADw2zkCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFZlJREFUeJzt3X+UZ3V93/HnS37IFpAfYaWAoQspLtHmuJiBREELEoP2\nKELCabE9VkmOtDXFopYj8bTa2JweEjFKkxPMQkCTKLbyU5CAoII1qcTlhyyKFCWgLD92ScrPrvLD\nd//43oXvzs585zsze+/cmXk+zpkz873fe7/z3s/Ozmu/n3vv+5OqQpK0vL1ooQuQJC08w0CSZBhI\nkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSgB0XuoBx7LPPPrVq1aqFLkOSFpWbb775kapaOc6+iyIM\nVq1axbp16xa6DElaVJLcN+6+rYVBktXA/xjadDDwYWBP4N3Apmb7h6rq6rbqkCTNrLUwqKq7gDUA\nSXYANgCXAacAn6iqs9v63pKk2enqBPKxwA+qauy3LJKk7nQVBicDFw09Pi3J7UkuSLJXRzVIkqbR\nehgk2Rk4HvhCs+lcBucP1gAPAh+f5rhTk6xLsm7Tpk1T7SJJ2k66uJrozcAtVfUwwJbPAEnOA66a\n6qCqWgusBZiYmHAFHklLyuW3buBj197FA49uZv89V3DGcas54bADFqyeLsLg7QxNESXZr6oebB6e\nCNzRQQ2S1BuX37qB3750PZufeQ6ADY9u5rcvXQ+wYIHQ6jRRkl2BNwKXDm3+/STrk9wOHAO8r80a\nJKlvPnbtXc8HwRabn3mOj1171wJV1PI7g6p6CviZSdve0eb3lKS+e+DRzbPa3gV7E0lSx/bfc8Ws\ntnfBMJCkjp1x3GpW7LTDVttW7LQDZxy3eoEqWiS9iSRpKdlykni5XU0kSZrkhMMOWNBf/pM5TSRJ\nMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScKupZI0pb4tWN82\nw0CSJunjgvVtc5pIkibp44L1bTMMJGmSPi5Y3zbDQJIm6eOC9W0zDCRpkj4uWN82TyBL0iR9XLC+\nbYaBJE2hbwvWt81pIkmSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkmgxDJKsTnLb\n0MfjSU5PsneS65Lc3Xzeq60aJEnjaS0MququqlpTVWuAXwT+H3AZcCbwlao6BPhK81iStIC6alR3\nLPCDqrovyduAo5vtnwFuAD7YUR2SlonltobxfHUVBicDFzVf71tVDzZfPwTs21ENkpaJ5biG8Xy1\nfgI5yc7A8cAXJj9XVQXUNMedmmRdknWbNm1quUpJS8lyXMN4vrq4mujNwC1V9XDz+OEk+wE0nzdO\ndVBVra2qiaqaWLlyZQdlSloqluMaxvPVRRi8nRemiAC+CLyz+fqdwBUd1CBpGVmOaxjPV6thkGRX\n4I3ApUObzwLemORu4Feax5K03SzHNYznq9UTyFX1FPAzk7b9HYOriySpFctxDeP5cg1kSUvSclvD\neL5sRyFJMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJeNOZpAXiegP9YhhI6pzrDfSP00SSOud6\nA/1jGEjqnOsN9I9hIKlzrjfQP4aBpM653kD/eAJZUudcb6B/DANJC8L1BvrFaSJJkmEgSTIMJEkY\nBpIkDANJEoaBJAnDQJKEYSBJwpvOJM2R6xEsLYaBpFlzPYKlZ2QYJNkFeAvwOmB/YDNwB/ClqvpO\n++VJ6qNR6xEYBovTtGGQ5HcYBMENwE3ARmAX4OXAWU1QfKCqbu+gTkk94noES8+odwZ/U1Ufmea5\nP0jyUuDAFmqS1HP777mCDVP84nc9gsVr2quJqupLow6sqo1VtW77lySp71yPYOmZ8QRykiuBmrT5\nMWAd8CdV9eM2CpPUX65HsPSMczXRPcBK4KLm8b8AnmBw7uA84B3tlCapz1yPYGkZJwxeW1WHDz2+\nMsm3qurwJF5RJElLwDh3IO+W5PkTxc3XuzUPn26lKklSp8Z5Z/AB4BtJfgAEOAh4T5Jdgc+0WZwk\nqRvjhMFfAocAhzaP7wKqqn4CfHLUgUn2BM4H/gmDk9C/ARwHvBvY1Oz2oaq6evalS5K2l3Gmif60\nqn5SVd+uqm8DOwDj/vI+B7imqg4FXgXc2Wz/RFWtaT4MAklaYOOEwYYkfwyQZC/gOuAvZjooyR7A\n64E/Baiqp6vq0XnUKklqyYxhUFX/GXgyyaeALwMfr6oLx3jtgxhMBV2Y5NYk5zfnGQBOS3J7kgua\ngNlGklOTrEuybtOmTVPtIknaTqYNgyS/tuWDQW+iXwZuBarZNpMdgVcD51bVYcBTwJnAucDBwBrg\nQeDjUx1cVWuraqKqJlauXDmbP5MkaZZGnUB+66THtwI7NdsLuHSG174fuL+qbmoeXwycWVUPb9kh\nyXnAVbOqWNJYXG9AszFtGFTVKfN54ap6KMmPkqyuqruAY4HvJtmvqh5sdjuRQUtsSduR6w1otkZN\nE/2n6ebzm+ffkOQtM7z+acBnk9zOYFrovwG/n2R9s+0Y4H1zqFvSCKPWG5CmMmqaaD1wVZIfA7cw\nOBm8C4N7DtYA1zP45T6tqroNmJi02V5GUstcb0CzNWqa6ArgiiSHAEcC+wGPM7is9NSq8qdK6inX\nG9BszXgHclXdDdzdQS2StpMzjlu91TkDcL0BjTZOOwpJi4zrDWi2DANpiXK9Ac3GjHcgJzlynG2S\npMVrnN5EfzjmNknSIjXtNFGS1wCvBVYmef/QUy9h0LlUkrREjDpnsDODFc12BHYf2v44cFKbRUmS\nujXqPoMbgRuTfLqq7uuwJknYW0jdGudqohcnWQusGt6/qt7QVlHScmdvIXVtnDD4AvApBstXPjfD\nvpK2g1G9hQwDtWGcMHi2qs5tvRJJz7O3kLo2qmvp3kn2Bq5M8p4k+23Z1myX1JLpegjZW0htGfXO\n4GYGi9ikeXzG0HPFYLUySS2wt5C6NupqooO6LETSC+wtpK7NeM5gmvWOHwPWV9XG7V+SJLC3kLo1\nzgnk3wReA3yteXw0gymkg5J8tKr+vKXaJEkdGScMdgR+fstC9kn2Bf4M+CXg64BhIEmL3DiN6n52\nSxA0Njbb/h54pp2yJEldGuedwQ1JrmJw8xnArzfbdgUeba0ySVJnxgmD32IQAFvWMPgz4JKqKuCY\ntgqTJHVnnDWQC7i4+ZAkLUGj1jP4RlUdleQJBjeZPf8Ug4x4SevVSZI6Meqms6Oaz7tPt48kaWkY\n55wBSY4CDqmqC5PsA+xeVX/bbmlSf7nWgJaace5A/ggwAawGLmSwAtpf8MIJZWlZca0BLUXj3Gdw\nInA88BRAVT3A1stgSsvKqLUGpMVqnDB4urmiqACa+wukZcu1BrQUjRMG/zPJnwB7Jnk3cD1wXrtl\nSf3lWgNaimYMg6o6m8E9BpcwOG/w4ar6w7YLk/rqjONWs2KnHbba5loDWuxG3WdwOvDXwC1VdR1w\nXWdVST3mWgNaikZdTfQy4JPAoUnWA3/FIBz+umlSJy1brjWgpWbUTWf/ESDJzgwuLX0tcAqwNsmj\nVfWKbkqUJLVtnJvOVgAvAfZoPh4A1rdZlCSpW6POGawFXgk8AdzEYIroD6rq/3ZUmySpI6OuJjoQ\neDHwELABuJ9Zrl+QZM8kFyf5XpI7k7wmyd5Jrktyd/N5r7mXL0naHqYNg6p6E3A4cHaz6QPAt5J8\nOcnvjPn65wDXVNWhwKuAO4Ezga9U1SHAV5rHUqcuv3UDR571VQ4680scedZXufzWDQtdkrSgRp4z\naO48viPJo8BjzcdbgCOAj4w6NskewOuBdzWv9TTwdJK3AUc3u30GuAH44Fz/ANJs2VtI2ta07wyS\nvDfJ55P8ELiRQQh8D/g1YO8xXvsgYBNwYZJbk5zftLLYt6oebPZ5CNh3Xn8CaZbsLSRta9Q7g1UM\n1j1+39Av79m+9quB06rqpiTnMGlKqKoqSU11cJJTgVMBDjzwwDl8e2lq9haStjXqnMH7q+qSOQYB\nDE44319VNzWPL2YQDg8n2Q+g+bxxmu+/tqomqmpi5cqVcyxB2pa9haRtjdOobk6q6iHgR0m2NGw5\nFvgu8EXgnc22dwJXtFWDNBV7C0nbGmuls3k4DfhscxfzPQzuYH4Rg06ovwncB/zzlmuQtmJvIWlb\nGVww1G8TExO1bt26hS5DkhaVJDdX1cQ4+7Y2TSRJWjwMA0mSYSBJMgwkSbR/NZE0J5ffusGrfaQO\nGQbqHXsHSd1zmki9Y+8gqXuGgXrH3kFS9wwD9Y69g6TuGQbqHXsHSd3zBLJ6x95BUvcMA/XSCYcd\n4C9/qUNOE0mSDANJkmEgScIwkCRhGEiS8GoizZGN5KSlxTDQrNlITlp6nCbSrNlITlp6DAPNmo3k\npKXHMNCs2UhOWnoMA82ajeSkpccTyJo1G8lJS49hoDmxkZy0tDhNJEkyDCRJhoEkCcNAkoQnkJct\newtJGmYYLEP2FpI0mdNEy5C9hSRNZhgsQ/YWkjSZYbAM2VtI0mSGwTJkbyFJk7V6AjnJvcATwHPA\ns1U1keS/AO8GNjW7faiqrm6zDm3N3kKSJuviaqJjquqRSds+UVVnd/C9NQ17C0ka5jSRJKn1MCjg\n+iQ3Jzl1aPtpSW5PckGSvVquQZI0g7bD4KiqWgO8GfitJK8HzgUOBtYADwIfn+rAJKcmWZdk3aZN\nm6baRZK0nbQaBlW1ofm8EbgMOKKqHq6q56rqp8B5wBHTHLu2qiaqamLlypVtlilJy15rJ5CT7Aq8\nqKqeaL7+VeCjSfarqgeb3U4E7mirhsXM3kGSutTm1UT7Apcl2fJ9PldV1yT58yRrGJxPuBf4Ny3W\nsCjZO0hS11oLg6q6B3jVFNvf0db3XCpG9Q4yDCS1wUtLe8jeQZK6Zhj0kL2DJHXNMOghewdJ6pqL\n2/SQvYMkdc0w6Cl7B0nqktNEkiTDQJJkGEiSMAwkSXgCuTX2FpK0mBgGLbC3kKTFxmmiFozqLSRJ\nfWQYtMDeQpIWG8OgBfYWkrTYGAYtsLeQpMXGE8gtsLeQpMXGMGiJvYUkLSbLMgy8B0CStrbswsB7\nACRpW8vuBLL3AEjStpZdGHgPgCRta9mFgfcASNK2ll0YeA+AJG1r2Z1A9h4ASdrWsgsD8B4ASZps\n2U0TSZK2ZRhIkgwDSZJhIEnCMJAkAamqha5hRkk2AfdN8dQ+wCMdlzNbfa/R+uan7/VB/2u0vvkZ\nVd8/qqqV47zIogiD6SRZV1UTC13HKH2v0frmp+/1Qf9rtL752V71OU0kSTIMJEmLPwzWLnQBY+h7\njdY3P32vD/pfo/XNz3apb1GfM5AkbR+L/Z2BJGk76GUYJLkgycYkd8yw3+FJnk1y0tC2e5OsT3Jb\nknULUV+So5M81tRwW5IPDz33piR3Jfl+kjPbqG871LjgYzhU421JvpPkxqHtrY/hPOtrffzGqTHJ\nGUN/v3ckeS7J3s1zCz6GM9S34D+DSfZIcmWSbzd/x6cMPdeH8RtV3+zHr6p69wG8Hng1cMeIfXYA\nvgpcDZw0tP1eYJ+FrA84Grhqmpp/ABwM7Ax8G3hFn2rs0RjuCXwXOLB5/NIux3Cu9XU1fuPUOGnf\ntwJf7dMYTldfj34GPwT8XvP1SuDvm/HqxfhNV99cx6+X7wyq6usM/mCjnAZcAmxsv6KtjVnfVI4A\nvl9V91TV08Dngbdt1+Ia86ixE2PU9y+BS6vqh83+W/6eOxnDedTXmVn+Hb8duKj5ui9jOF19nRij\nvgJ2TxJgt2bfZ+nP+E1X35z0MgxmkuQA4ETg3CmeLuD6JDcnObXbyrby2iS3J/nLJK9sth0A/Gho\nn/ubbQtlqhqhH2P4cmCvJDc0dfzrZntfxnC6+qAf4/e8JP8AeBOD/zxBf8YQmLI+6McY/hHw88AD\nwHrgP1TVT+nP+E1XH8xh/Bbr4jafBD5YVT8dhOJWjqqqDUleClyX5HtNwnbpFgbTB08m+WfA5cAh\nHdcwk1E19mEMdwR+ETgWWAH87yTf7LiGUaasr6r+D/0Yv2FvBf6qqvr6TnGq+vowhscBtwFvAH6u\nqeN/dVzDKFPWV1WPM4fxW5TvDIAJ4PNJ7gVOAv44yQkAVbWh+bwRuIzBW7pOVdXjVfVk8/XVwE5J\n9gE2AD87tOvLmm2dG1FjL8aQwf+2rq2qp6rqEeDrwKvozxhOV19fxm/YyWw9BdOXMdxicn19GcNT\nGEwFVlV9H/hb4FD6M37T1Ten8VuUYVBVB1XVqqpaBVwMvKeqLk+ya5LdAZLsCvwqMPKKpDYk+YfN\nPB5JjmAwzn8HfAs4JMlBSXZm8I/gi13XN6rGvowhcAVwVJIdm2mEXwLupD9jOGV9PRo/mhr2AP5p\nU+8WfRnDKevr0Rj+kME7P5LsC6wG7qE/4zdlfXMdv15OEyW5iMHVLvskuR/4CLATQFV9asSh+wKX\nNb/jdgQ+V1XXLEB9JwH/LsmzwGbg5Bqc4n82yb8HrmVwRcIFVfWd7V3ffGpsfqgWfAyr6s4k1wC3\nAz8Fzq+qO5pjWx/DudaX5GA6GL9xamx2OxH4clU9teW4qurk53Cu9dGff8f/Ffh0kvVAGExNP9Ic\n24fxm7K+uf4MegeyJGlxThNJkrYvw0CSZBhIkgwDSRKGgSQJw0BLWJInJz1+V5I/muGY4zNDF8oM\nupVeNc1zpzf3HUx37MXNpX9jSfILST497v7SXBkG0pCq+mJVnTWPlzgdmDIMMuj/tENV3TOLetYD\nL0ty4DxqkmZkGGhZSrIyySVJvtV8HNlsf/7dQ5KfS/LNDPrC/+6kdxq7Nf/L/16Sz2bgvcD+wNeS\nfG2Kb/uv2PpO2yeTfCyDXvTXJzkig8Z39yQ5fui4Kxnc5Sq1xjDQUrYiLyyechvw0aHnzgE+UVWH\nA78OnD/F8ecA51TVLzDoRTTsMAbvAl7BoK/9kVX13xl0kDymqo6Z4vWOBG4eerwrgx7+rwSeAH4X\neCODu3KHa10HvG6cP7A0V71sRyFtJ5uras2WB0nexaDJIcCvAK/IC11vX5Jkt0nHvwY4ofn6c8DZ\nQ8/9TVXd37zubcAq4Bsz1LMfsGno8dPAljYB64GfVNUzTXuBVUP7bWTwjkNqjWGg5epFwC9X1Y+H\nN2bblujT+cnQ188x3r+lzcAuQ4+fqRf6wfx0y2s2rdmHX2+X5lipNU4Tabn6MoPV8gBIsmaKfb7J\nYAoJxp+zfwLYfZrn7gT+8bgFDnk5C9j5VMuDYaDl6r3ARAYrvX0X+LdT7HM68P4ktzP4Jf7YGK+7\nFrhmmhPIX2LQhXK2jmmOlVpj11JpGs39Apub1t4nA2+vqjmvdZtkBfA1BiebnxvzmBcDNzJYuWrO\n69tKMzEMpGkkeR2DdWYDPAr8RrOi1Hxe8zjgzqr64Zj7HwIcUFU3zOf7SjMxDCRJnjOQJBkGkiQM\nA0kShoEkCcNAkoRhIEkC/j8eQPhZJrafTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28c06b0cd30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.scatter(height, weight)\n",
    "plt.xlabel('Height (m)')\n",
    "plt.ylabel('Weight (kg)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there is a **positive correlation** between the height and weight of these population and we could approximate our set as a straight line. This line could be used to predict the weight of someone among this population who is not in our data, let's say someone who is 1.90m tall.<p>\n",
    "This is a linear regression problem, once we are trying to find **which line best fits these points.** <p>\n",
    "We will explore this problem using two different approaches: **Least Squares Error minimization** and a single **neuron**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Least Squares Error minimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In the begining of the 18th century, the French mathematician Adrian-Marie Legendre and the German mathematician Johan Carl Friedrich Gauss claimed to be the first one to solve the problem of fitting linear equations to data. Regardless who won this dispute, this given solution was known as **Least Squares**. In this section we will demonstrate this approach with a very simple example that can be extended to our height vs. weight problem.<p>\n",
    "Given the set of $N=7$ points in the image below, the best line that fits these points is the blue line.<p>\n",
    "![](figures/linear-regression_1.png)\n",
    "We can see that the green lines **vertically** connect each point to the blue line and they represent the distance between each point $i$ to our target blue line. Thus, we could model our total error $E$ as the sum of the squared distances $e_i$, which is given by:  \n",
    "\n",
    "$$\\begin{align*} \n",
    "    E = \\sum_{i=1}^N (e_i)^2 = \\sum_{i=1}^N (y_i-f(x_i, \\beta))^2 &&\\text{[2]} \n",
    "\\end{align*}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Our model function $f(x_i,\\beta)$ represents the function of the line that best fits our model.  $\\beta$ is a vector containing the parameters ($\\beta_0$ and $\\beta_1$) that are adjustable to give different lines. In other words, we can say that:\n",
    "\n",
    "$$\\begin{align*} \n",
    "    f(x,\\beta) = \\beta_0+\\beta_1 x  &&\\text{[3]} \n",
    "\\end{align*}$$\n",
    "\n",
    "Our goal is to find the parameters $\\beta_0$ and $\\beta_1$ that describe our line. Thus, we want to minimize the error $E$ that is obtained gathering the equations [2] and [3] resulting:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\min_{\\beta_0,\\beta_1} E = \\min_{\\beta_0, \\beta_1} \\sum_{i=1}^N (y_i - \\beta_0 - \\beta_1 x_i)^2  &&\\text{[4]} \n",
    "\\end{align*}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to minimize our error, we will derive the equation above with respect to $\\beta_0$ and $\\beta_1$ and set them equal to 0. This give us:\n",
    "\n",
    "$$\\begin{align*} \n",
    "\\frac{\\partial E}{\\partial \\beta_0} = \\sum_{i=1}^N -2(y_i - \\beta_0 - \\beta_1 x_i) = 0  &&\\text{[5]} \n",
    "\\end{align*}$$\n",
    "\n",
    "and\n",
    "\n",
    "$$\\begin{align*} \n",
    "\\frac{\\partial E}{\\partial \\beta_1} = \\sum_{i=1}^N -2x_i(y_i - \\beta_0 - \\beta_1 x_i) = 0  &&\\text{[6]} \n",
    "\\end{align*}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's solve equations [5] and [6] with the **First Order Conditions** for $\\beta_0$ and $\\beta_1$.<p> \n",
    "Starting with $\\beta_0$ from equation [5]:\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "&\\sum_{i=1}^N -2(y_i - \\beta_0 - \\beta_1 x_i) = 0 \\nonumber\\\\\n",
    "&\\sum_{i=1}^N (y_i - \\beta_0 - \\beta_1 x_i) = 0 \\nonumber\\\\\n",
    "&\\sum_{i=1}^N y_i - \\sum_{i=1}^N \\beta_0 - \\sum_{i=1}^N \\beta_1 x_i = 0 \\nonumber\\\\\n",
    "&\\sum_{i=1}^N y_i - N\\beta_0 - \\beta_1\\sum_{i=1}^N x_i = 0\\nonumber\\\\\n",
    "&\\beta_0 = \\frac{1}{N}\\sum_{i=1}^N y_i - \\frac{\\beta_1}{N}\\sum_{i=1}^N x_i\\nonumber\\\\\n",
    "&\\boxed{\\beta_0 = \\bar{y} - \\beta_1\\bar{x}}  &&\\text{[7]} \\nonumber\n",
    "\\end{eqnarray*}\n",
    "\n",
    "Where $\\bar{y}$ and $\\bar{x}$ are the means of $y$ and $x$ values respectively.<p> \n",
    "Now, let's solve for $\\beta_1$ from equation [6]:\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "&\\sum_{i=1}^N -2x_i(y_i - \\beta_0 - \\beta_1 x_i) = 0\\nonumber\\\\\n",
    "&\\sum_{i=1}^N x_i(y_i - \\beta_0 - \\beta_1 x_i) = 0\\nonumber\\\\\n",
    "&\\sum_{i=1}^N x_iy_i - \\sum_{i=1}^N x_i\\beta_0 - \\sum_{i=1}^N \\beta_1 x_i^2 = 0\\nonumber\\\\\n",
    "&\\sum_{i=1}^N x_iy_i - \\beta_0\\sum_{i=1}^N x_i - \\beta_1\\sum_{i=1}^N x_i^2 = 0\\nonumber\\\\\n",
    "\\end{eqnarray*}\n",
    "\n",
    "Substituting in the equation above the definition of $\\beta_0$ from equation [7] we have:\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "&\\sum_{i=1}^N x_iy_i - \\left(\\frac{1}{N}\\sum_{i=1}^N y_i - \\frac{\\beta_1}{N}\\sum_{i=1}^N x_i\\right)\\sum_{i=1}^N x_i - \\beta_1\\sum_{i=1}^N x_i^2 = 0\\nonumber\\\\\n",
    "&\\sum_{i=1}^N x_iy_i - \\frac{1}{N}\\sum_{i=1}^N x_i \\sum_{i=1}^N y_i = \\beta_1 \\left(-\\frac{1}{N}\\sum_{i=1}^N x_i \\sum_{i=1}^N x_i + \\sum_{i=1}^N x_i^2 \\right)\\nonumber\\\\\n",
    "&\\beta_1 = \\frac{\\sum_{i=1}^N x_iy_i - \\frac{1}{N}\\sum_{i=1}^N x_i \\sum_{i=1}^N y_i}{\\frac{1}{N} \\sum_{i=1}^N x_i \\sum_{i=1}^N x_i + \\sum_{i=1}^N x_i^2}\\nonumber\\\\\n",
    "&\\beta_1 = \\frac{N \\sum_{i=1}^N x_iy_i - \\sum_{i=1}^N x_i \\sum_{i=1}^N y_i}{-\\left(\\sum_{i=1}^N x_i\\right)^2 + N \\sum_{i=1}^N x_i^2}\\nonumber\\\\\n",
    "&\\boxed{\\beta_1 = \\frac{\\sum_{i=1}^N \\left(x_i - \\bar{x}\\right)\\left(y_i - \\bar{y}\\right)}{\\sum_{i=1}^N \\left(x_i - \\bar{x}\\right)^2}}  &&\\text{[8]}\\nonumber\\\\\n",
    "\\end{eqnarray*}\n",
    "\n",
    "Where $\\bar{y}$ and $\\bar{x}$ are the means of $y$ and $x$ values respectively.\n",
    "\n",
    "Therefore, the line that best fits our model is given by:\n",
    "\n",
    "$$\\begin{align*} \n",
    "\\boxed{y = \\beta_0 + \\beta_1 x \\text{ , where }\n",
    "            \\left\\{\n",
    "                \\begin{array}{ll}\n",
    "                  \\beta_0 = \\bar{y} - \\beta_1\\bar{x}\\\\\n",
    "                  \\beta_1 = \\frac{\\sum_{i=1}^N \\left(x_i - \\bar{x}\\right)\\left(y_i - \\bar{y}\\right)}{\\sum_{i=1}^N \\left(x_i - \\bar{x}\\right)^2}\n",
    "                \\end{array}\\right.} &&\\text{[9]} \n",
    "\\end{align*} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Generalizing\n",
    "\n",
    "So far we have been working with $N$ samples and trying to find the best fitting line. Our line had 2 coefficients or **degrees of freedom** ($\\beta_0$ and $\\beta_1$) that were the descriptors of our line. Alternatively, we could generalize this problem to have $p$ degrees of freedom ($\\beta_0$, $\\beta_1$, ..., $\\beta_{p-1}$) and make our line more complicated raising $x$ to different powers. For example:<br>\n",
    "* **Line** (1st degree equation): $\\beta_0 + \\beta_1x$ with 2 degrees of freedom ($p$=2)\n",
    "* **Quadratic function** (2nd degree equation): $\\beta_0 + \\beta_1x + \\beta_2x^2$ with 3 degrees of freedom ($p$=3)\n",
    "* **Cubic function** (3rd degree equation): $\\beta_0 + \\beta_1x + \\beta_2x^2 + \\beta_3x^3$ with 4 degrees of freedom ($p$=4)\n",
    "* **Quartic function** (4th degree equation): $\\beta_0 + \\beta_1x + \\beta_2x^2 + \\beta_3x^3 + \\beta_4x^4$ with 5 degrees of freedom ($p$=5)<br>\n",
    "\n",
    "**Attention:** if we start raising $x$ to different powers, we will be going beyond Linear Regression and start doing **Polynomial Regression** (because then instead of a straight line, our fitting model will  be a polynomial of higher degree). In order to keep with Linear Regression, we will increase the degrees of freedom of our linear model by adding $\\beta$'s to our equation considering the fact that now our data has more independent variables ($x$'s). Therefore, as we have more independent variables (features), our line will become a hyperplane.<p>\n",
    "\n",
    "The equation below is just an expansion of the previous particular linear case where we had only two degrees of freedom. Now we will have $p+1$ degrees of freedom in a **system of $N$ linear equations** taken over _i_ from 1 to $N$ (one for each sample). The number of independent variables of our samples is also given by $p$.\n",
    "\n",
    "$$\\begin{align*} \n",
    "    y_i = \\beta_0 + \\sum_{j=1}^p \\beta_{j}x_{j}   &&\\text{[10]} \n",
    "\\end{align*}$$\n",
    "\n",
    "We need to solve this linear equations system to obtain the best line (or a hyperplane if we have more dimensions) to fit our model, but suppose our linear model is an **overdetermined system**. Let's say we have more linearly independent equations than unknowns (betas). This kind of system has no solution, so what would be the best line to fit our model?\n",
    "\n",
    "To answer this question, we will do exactly the same approach as before: We have to minimize our total error $E$, that now is the sum of the error of all $N$ samples: \n",
    "\n",
    "$$\\begin{align*} \n",
    "    E = \\sum_{i=1}^N (y_i - \\beta_0 - \\sum_{j=1}^p \\beta_{j} x_j)^2  &&\\text{[11]} \n",
    "\\end{align*}$$\n",
    "\n",
    "The equation above is also known as Residual Sum of Squared Errors (RSS or RSSE).\n",
    "\n",
    "Before we minimize our error $E$ from equation [11], let's simplify it using matrix notation. We can also remove $\\beta_0$ by appending a column vector of 1 values to our $X$ matrix ($X_{i,1}=1$) and increasing the length of $\\beta$ by one. Thus:\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "&E = \\bigl\\| Y - X\\beta \\bigr\\|^2 \\nonumber\\\\\n",
    "&E = Y^{T}Y - 2\\beta^{T}X^{T}Y + \\beta^{T}X^{T}X\\beta &&\\text{[12]} \\nonumber\\\\\n",
    "\\end{eqnarray*}\n",
    "\n",
    "Where:\n",
    "\n",
    "$$Y = \\begin{bmatrix} y_1 \\\\ y_2 \\\\ \\vdots \\\\ y_N \\end{bmatrix},\n",
    "X = \\begin{bmatrix}\n",
    "1 & X_{1,2} & X_{1,3} & \\cdots & X_{1,p+1} \\\\ \n",
    "1 & X_{2,2} & X_{2,3} & \\cdots & X_{2,p+1} \\\\ \n",
    "\\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ \n",
    "1 & X_{N,2} & X_{N,3} & \\cdots & X_{N,p+1}\n",
    "\\end{bmatrix}, \n",
    "B = \\begin{bmatrix} \\beta_0 \\\\ \\beta_2 \\\\ \\vdots \\\\ \\beta_{p+1} \\end{bmatrix}$$\n",
    "\n",
    "Remeber that $N$ is our number of samples and $p$ is our independent variables.\n",
    "\n",
    "Now let's derive our error equation [12] with respect to our matrix $\\beta$, set it equal to 0 and divide by 2 (as we did before by using First Order Conditions).\n",
    "\n",
    "\\begin{eqnarray*} \n",
    "&\\frac{\\partial E}{\\partial \\beta} = \\frac{\\partial \\left(Y^{T}Y - 2\\beta^{T}X^{T}Y + \\beta^{T}X^{T}X\\beta\\right)}{\\partial \\beta} = 0 \\nonumber\\\\\n",
    "&X^{T}Y+X^{T}X\\beta = 0\\nonumber\\\\\n",
    "&\\boxed{\\beta = (X^{T}X)^{-1}X^{T}Y} &&\\text{[13]} \\nonumber\\\\\n",
    "\\end{eqnarray*}\n",
    "\n",
    "Note that if $(X^{T}X)^{-1}X^{T}$ is the pseudo-inverse of $X$. But if $(X^{T}X)^{-1}X^{T}$ is not invertible (singular), iterative solvers methods can be used to overcome this problem. However we still could use the pseudo-inverse of $X$ to have our solution, but it would not be the unique fitting line (or plane) for our solution.\n",
    "\n",
    "In the end of this notebook, we will solve the initial problem (height vs. weight) using our obtained equations with different degrees of freedom to see the behavior of the fitting line. :)\n",
    "\n",
    "**Observation:** We could have applied Linear Algebra using the concept of projections to get the same result. But I think it would be too much information on this notebook. We would end up obtaining the same result though. If I am not mistaken, I am pretty sure I have seen this proof in some Linear Algebra open courses. It might have been in a course by professor Strang (MIT) or Stephen Boyd (Stanford). This is a trivial proof, but in case you don't find those classes please send me an email and I will post it here. :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-23T22:47:58.399566Z",
     "start_time": "2017-08-23T22:47:58.396584Z"
    }
   },
   "source": [
    "## Representing linear equation with a neuron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The following image shows an example where we could use a single neuron to represent a linear equation.<p>\n",
    "![](figures/linear_regression_neuron.png)\n",
    "Note that we have:\n",
    "* 1 input $x_0$\n",
    "* 2 parameters to be adjusted $\\beta_0$ and $\\beta_1$\n",
    "* 1 output $y$\n",
    "\n",
    "We can see that our output $y$ is given exactly like our linear model seen before: $y=\\beta_0+\\beta_1x$\n",
    "\n",
    "Let's now model equation [10] but with only one entry (sample) $N=1$: \n",
    "$$ y = \\beta_0 + \\sum_{j=1}^p \\beta_{j}x_{j}$$\n",
    "\n",
    "Our equation is a linear combination of its $p$+1 entries with the $\\beta$'s and can be represented with the single neuron as shown in the image below:\n",
    "\n",
    "![](figures/linear_regression_neuron_2.png)\n",
    "\n",
    "That's why we combine neurons in layers in order to obtain more degrees of freedom as we saw previously with **Least Squares Error minimization**.<p>\n",
    "**Observation:** As we are dealing with Linear Regression, in these examples we are considering that our neuron has a linear activation function."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "119px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}