MSE is unbiased proof .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This is the proof that in multivariate linear regression $MSE = \\frac{SSE}{(DoF)} = (n-2)\\sigma^2 $ is unbiased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "Under the usual notation, \n",
    "\n",
    "$$  Y = X\\beta + \\epsilon \\\\\n",
    "$$ \n",
    "$$\n",
    "\\\\\\\\ X \\quad is \\quad n*(p+1) \\quad matrix\\\\ \\text{essentially, there are n rows of datapoints and (p+1) variables} \\\\\n",
    "$$\n",
    "$$ \\hat Y = X\\hat\\beta \\\\\n",
    "$$ \n",
    "$$  \\hat\\beta = (X'X)^{-1}X'Y \\\\\n",
    "\\implies \\hat\\beta' = Y'X(X'X)^{-1}\n",
    "$$ \n",
    "\n",
    "\n",
    "\\begin{align}\n",
    "\\Sigma (Y_i - \\hat Y_i)^2  & = (Y_i - \\hat Y_i)'(Y_i - \\hat Y_i) \\\\\n",
    " & = (X(\\beta - \\hat \\beta) + \\epsilon)' (X(\\beta - \\hat \\beta) + \\epsilon)\\\\\n",
    " & = \\underbrace {(\\beta - \\hat \\beta)'X'X(\\beta - \\hat \\beta)}_{term1} + \\underbrace {\\epsilon'X (\\beta - \\hat \\beta)}_{term2}\\\\ \n",
    " & + \\underbrace {(\\beta - \\hat \\beta)'X'\\epsilon}_{term3} + \\epsilon'\\epsilon \\\\\n",
    "\\end{align}\n",
    "\n",
    "Simplifying the individual terms\n",
    "\n",
    "Term 1: \n",
    "\\begin{align}\n",
    "(\\beta - \\hat \\beta)'X'X(\\beta - \\hat \\beta) &= (\\beta - (X'X)^{-1}X'Y)'X'X(\\beta - (X'X)^{-1}X'Y)\\\\\n",
    "& = (\\beta' - Y'X(X'X)^{-1})X'X(\\beta - (X'X)^{-1}X'Y) \\\\\n",
    "& = \\beta'X'X\\beta - Y'X\\beta - \\beta'(X'X)(X'X)^{-1}X'Y + Y'X(X'X)^{-1}X'Y \\\\\n",
    "& = \\beta'X'X\\beta - (\\beta'X' + \\epsilon')X\\beta \\\\ \n",
    "&-\\beta'(X'X)(X'X)^{-1}X'Y \\\\ \n",
    "& + (\\beta'X' + \\epsilon')X(X'X)^{-1}X'Y  \\quad \\text{substituting the value of }Y' \\\\\n",
    "& = - \\epsilon'X\\beta + \\epsilon'X(X'X)^{-1}X'Y \\quad \\text {some terms get cancelled} \\\\\n",
    "& = - \\epsilon'X\\beta + \\epsilon'X(X'X)^{-1}X'( X\\beta + \\epsilon) \\quad \\text {substituting the value of } Y \\\\\n",
    "& = \\epsilon'X(X'X)^{-1}X'\\epsilon \\\\\n",
    "\\end{align}\n",
    "\n",
    "Term 2 : \n",
    "\\begin{align}\n",
    "\\epsilon'X (\\beta - \\hat \\beta) &= \\epsilon'X(\\beta - (X'X)^{-1}X'Y)\\\\\n",
    "& = \\epsilon'X(\\beta - (X'X)^{-1}X'(X\\beta+\\epsilon))\\quad \\text {substituting the value of } Y \\\\\\\\\n",
    "& = -\\epsilon'X(X'X)^{-1}X'\\epsilon\n",
    "\\end{align}\n",
    "\n",
    "Similarly, \n",
    "Term 3 is also same as Term 2, i.e. $ -\\epsilon'X(X'X)^{-1}X'\\epsilon$\n",
    "\n",
    "\n",
    "$$\n",
    "\\Sigma (Y_i - \\hat Y_i)^2\\\\=\\epsilon'X(X'X)^{-1}X'\\epsilon-\\epsilon'X(X'X)^{-1}X'\\epsilon-\\epsilon'X(X'X)^{-1}X'\\epsilon+\\epsilon'\\epsilon\\\\=\\epsilon'\\epsilon-\\epsilon'X(X'X)^{-1}X'\\epsilon\\\\=\\epsilon'\\epsilon-\\epsilon'P\\epsilon\n",
    "$$\n",
    "\n",
    "$P$ is the projection matrix which is symmetric and idempotent\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let's calculate the expectation now, \n",
    "\n",
    "$$\n",
    "E[\\Sigma (Y_i - \\hat Y_i)^2]\\\\=E(\\epsilon'\\epsilon-\\epsilon'P\\epsilon)\\\\=E(\\epsilon'\\epsilon)-E(\\epsilon'P\\epsilon)\\\\=n\\sigma^2-\\sigma^2trace(P) \\\\\\text{(proof that trace is (p+1) is given below)}\\\\\n",
    "=(n-(p+1))\\sigma^2\n",
    "\\\\\\\\\n",
    "\\therefore \\frac{\\Sigma (Y_i - \\hat Y_i)^2}{n-(p+1)} \\text{is an unbiased estimator of }\\sigma^2\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Proof that $\\text{(trace(P)}=p+1)$\n",
    "\n",
    "$$\n",
    "\\text{trace operator is commutative}\\\\\n",
    "\\implies trace(AB)= trace(BA) \\\\\n",
    "\\therefore trace(\\underbrace{X}_{A}\\underbrace{(X'X)^{-1}X'}_{B})=trace((X'X)^{-1}X'X) = (p+1)\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}