slr-rescorr.md

layout	mathjax	author	affiliation	e_mail	date	title	chapter	section	topic	theorem	sources	proof_id	shortcut	username
proof	true	Joram Soch	BCCN Berlin	joram.soch@bccn-berlin.de	2021-10-27 06:07:00 -0700	The residuals and the covariate are uncorrelated in simple linear regression	Statistical Models	Univariate normal data	Simple linear regression	Correlation with covariate is zero	authors year title in pages url Wikipedia 2021 Simple linear regression Wikipedia, the free encyclopedia retrieved on 2021-10-27 https://en.wikipedia.org/wiki/Simple_linear_regression#Numerical_properties	P277	slr-rescorr	JoramSoch

Theorem: In simple linear regression, the residuals and the covariate are uncorrelated when estimated using ordinary least squares.

Proof: The residuals are defined as the estimated error terms

$$ \label{eq:slr-res} \hat{\varepsilon}_i = y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i $$

where $\hat{\beta}_0$ and $\hat{\beta}_1$ are parameter estimates obtained using ordinary least squares:

$$ \label{eq:slr-ols} \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} \quad \text{and} \quad \hat{\beta}1 = \frac{s{xy}}{s_x^2} ; . $$

With that, we can calculate the inner product of the covariate and the residuals vector:

$$ \label{eq:slr-rescorr} \begin{split} \sum_{i=1}^n x_i \hat{\varepsilon}i &= \sum{i=1}^n x_i (y_i - \hat{\beta}0 - \hat{\beta}1 x_i) \ &= \sum{i=1}^n \left( x_i y_i - \hat{\beta}0 x_i - \hat{\beta}1 x_i^2 \right) \ &= \sum{i=1}^n \left( x_i y_i - x_i (\bar{y} - \hat{\beta}1 \bar{x}) - \hat{\beta}1 x_i^2 \right) \ &= \sum{i=1}^n \left( x_i (y_i - \bar{y}) + \hat{\beta}1 (\bar{x} x_i - x_i^2 \right) \ &= \sum{i=1}^n x_i y_i - \bar{y} \sum{i=1}^n x_i - \hat{\beta}1 \left( \sum{i=1}^n x_i^2 - \bar{x} \sum{i=1}^n x_i \right) \ &= \left( \sum{i=1}^n x_i y_i - n \bar{x} \bar{y} - n \bar{x} \bar{y} + n \bar{x} \bar{y} \right) - \hat{\beta}1 \left( \sum{i=1}^n x_i^2 - 2 n \bar{x} \bar{x} + n \bar{x}^2 \right) \ &= \left( \sum_{i=1}^n x_i y_i - \bar{y} \sum_{i=1}^n x_i - \bar{x} \sum_{i=1}^n y_i + n \bar{x} \bar{y} \right) - \hat{\beta}1 \left( \sum{i=1}^n x_i^2 - 2 \bar{x} \sum_{i=1}^n x_i + n \bar{x}^2 \right) \ &= \sum_{i=1}^n \left( x_i y_i - \bar{y} x_i - \bar{x} y_i + \bar{x} \bar{y} \right) - \hat{\beta}1 \sum{i=1}^n \left( x_i^2 - 2 \bar{x} x_i + \bar{x}^2 \right) \ &= \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) - \hat{\beta}1 \sum{i=1}^n (x_i - \bar{x})^2 \ &= (n-1) s_{xy} - \frac{s_{xy}}{s_x^2} (n-1) s_x^2 \ &= (n-1) s_{xy} - (n-1) s_{xy} \ &= 0 ; . \end{split} $$

Because an inner product of zero also implies zero correlation, this demonstrates that residuals and covariate values are uncorrelated under ordinary least squares.