diff --git a/I/ToC.md b/I/ToC.md index 092182ac..4cdfdaf0 100644 --- a/I/ToC.md +++ b/I/ToC.md @@ -520,21 +520,22 @@ title: "Table of Contents"    1.3.5. **[Expectation of estimates](/P/slr-olsmean)**
   1.3.6. **[Variance of estimates](/P/slr-olsvar)**
   1.3.7. **[Distribution of estimates](/P/slr-olsdist)**
-    1.3.8. **[Effects of mean-centering](/P/slr-meancent)**
-    1.3.9. *[Regression line](/D/regline)*
-    1.3.10. **[Regression line includes center of mass](/P/slr-comp)**
-    1.3.11. **[Projection of data point to regression line](/P/slr-proj)**
-    1.3.12. **[Sums of squares](/P/slr-sss)**
-    1.3.13. **[Transformation matrices](/P/slr-mat)**
-    1.3.14. **[Weighted least squares](/P/slr-wls)** (1)
-    1.3.15. **[Weighted least squares](/P/slr-wls2)** (2)
-    1.3.16. **[Maximum likelihood estimation](/P/slr-mle)** (1)
-    1.3.17. **[Maximum likelihood estimation](/P/slr-mle2)** (2)
-    1.3.18. **[Sum of residuals is zero](/P/slr-ressum)**
-    1.3.19. **[Correlation with covariate is zero](/P/slr-rescorr)**
-    1.3.20. **[Residual variance in terms of sample variance](/P/slr-resvar)**
-    1.3.21. **[Correlation coefficient in terms of slope estimate](/P/slr-corr)**
-    1.3.22. **[Coefficient of determination in terms of correlation coefficient](/P/slr-rsq)**
+    1.3.8. **[Correlation of estimates](/P/slr-olsdist)**
+    1.3.9. **[Effects of mean-centering](/P/slr-meancent)**
+    1.3.10. *[Regression line](/D/regline)*
+    1.3.11. **[Regression line includes center of mass](/P/slr-comp)**
+    1.3.12. **[Projection of data point to regression line](/P/slr-proj)**
+    1.3.13. **[Sums of squares](/P/slr-sss)**
+    1.3.14. **[Transformation matrices](/P/slr-mat)**
+    1.3.15. **[Weighted least squares](/P/slr-wls)** (1)
+    1.3.16. **[Weighted least squares](/P/slr-wls2)** (2)
+    1.3.17. **[Maximum likelihood estimation](/P/slr-mle)** (1)
+    1.3.18. **[Maximum likelihood estimation](/P/slr-mle2)** (2)
+    1.3.19. **[Sum of residuals is zero](/P/slr-ressum)**
+    1.3.20. **[Correlation with covariate is zero](/P/slr-rescorr)**
+    1.3.21. **[Residual variance in terms of sample variance](/P/slr-resvar)**
+    1.3.22. **[Correlation coefficient in terms of slope estimate](/P/slr-corr)**
+    1.3.23. **[Coefficient of determination in terms of correlation coefficient](/P/slr-rsq)**
1.4. Multiple linear regression
   1.4.1. *[Definition](/D/mlr)*
diff --git a/P/slr-meancent.md b/P/slr-meancent.md index 6097c5b4..07490bd8 100644 --- a/P/slr-meancent.md +++ b/P/slr-meancent.md @@ -21,7 +21,7 @@ username: "JoramSoch" --- -**Theorem:** In [simple linear regression](/D/slr), when the independent variable $y$ and/or the dependent variable $x$ are [mean-centered](/D/mean), the [ordinary least squares](/P/slr-ols) estimate for the intercept changes, but that of the slope does not. +**Theorem:** In [simple linear regression](/D/slr), when the dependent variable $y$ and/or the independent variable $x$ are [mean-centered](/D/mean), the [ordinary least squares](/P/slr-ols) estimate for the intercept changes, but that of the slope does not. **Proof:** @@ -98,4 +98,4 @@ $$ \label{eq:slr-ols-meancent-xy} \end{split} $$ -and we can see that $\hat{\beta}_1(\tilde{x},\tilde{y}) = \hat{\beta}_1(x,y)$, but $\hat{\beta}_0(\tilde{x},\tilde{y}) \neq \hat{\beta}_0(x,y)$, specifically $\beta_0$ is now forced to become zero. \ No newline at end of file +and we can see that $\hat{\beta}_1(\tilde{x},\tilde{y}) = \hat{\beta}_1(x,y)$, but $\hat{\beta}_0(\tilde{x},\tilde{y}) \neq \hat{\beta}_0(x,y)$, specifically $\beta_0$ is now forced to become zero. diff --git a/P/slr-olscorr.md b/P/slr-olscorr.md new file mode 100644 index 00000000..72617b89 --- /dev/null +++ b/P/slr-olscorr.md @@ -0,0 +1,49 @@ +--- +layout: proof +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-04-14 17:17:00 + +title: "Parameter estimates for simple linear regression are uncorrelated after mean-centering" +chapter: "Statistical Models" +section: "Univariate normal data" +topic: "Simple linear regression" +theorem: "Correlation of estimates" + +sources: + +proof_id: "P320" +shortcut: "slr-olscorr" +username: "JoramSoch" +--- + + +**Theorem:** In [simple linear regression](/D/slr), when the independent variable $x$ is [mean-centered](/D/mean), the [ordinary least squares](/P/slr-ols) estimates for slope and intercept are [uncorrelated](/D/corr). + + +**Proof:** The [parameter estimates for simple linear regression are bivariate normally distributed under ordinary least squares](/P/slr-olsdist): + +$$ \label{eq:slr-olsdist} +\left[ \begin{matrix} \hat{\beta}_0 \\ \hat{\beta}_1 \end{matrix} \right] \sim \mathcal{N}\left( \left[ \begin{matrix} \beta_0 \\ \beta_1 \end{matrix} \right], \, \frac{\sigma^2}{(n-1) \, s_x^2} \cdot \left[ \begin{matrix} x^\mathrm{T}x/n & -\bar{x} \\ -\bar{x} & 1 \end{matrix} \right] \right) +$$ + +Because the [covariance matrix](/D/covmat) of the [multivariate normal distribution](/D/mvn) contains the pairwise covariances of the [random variables](/D/rvar), we can deduce that the [covariance](/D/cov) of $\hat{\beta}_0$ and $\hat{\beta}_1$ is: + +$$ \label{eq:slr-olscov} +\mathrm{Cov}\left( \hat{\beta}_0, \hat{\beta}_1 \right) = -\frac{\sigma^2 \, \bar{x}}{(n-1) \, s_x^2} +$$ + +where $\sigma^2$ is the [noise variance](/D/slr), $s_x^2$ is the [sample variance](/D/var-samp) of $x$ and $n$ is the number of observations. When $x$ is mean-centered, we have $\bar{x} = 0$, such that: + +$$ \label{eq:slr-olscov-meancent} +\mathrm{Cov}\left( \hat{\beta}_0, \hat{\beta}_1 \right) = 0 \; . +$$ + +Because [correlation is equal to covariance divided by standard deviations](/D/corr), we can conclude that the correlation of $\hat{\beta}_0$ and $\hat{\beta}_1$ is also zero: + +$$ \label{eq:slr-olscorr-qed} +\mathrm{Corr}\left( \hat{\beta}_0, \hat{\beta}_1 \right) = 0 \; . +$$ \ No newline at end of file