diff --git a/D/covmat-cross.md b/D/covmat-cross.md new file mode 100644 index 00000000..9a90132f --- /dev/null +++ b/D/covmat-cross.md @@ -0,0 +1,44 @@ +--- +layout: definition +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-09-26 09:45:00 + +title: "Cross-covariance matrix" +chapter: "General Theorems" +section: "Probability theory" +topic: "Covariance" +definition: "Cross-covariance matrix" + +sources: + - authors: "Wikipedia" + year: 2022 + title: "Cross-covariance matrix" + in: "Wikipedia, the free encyclopedia" + pages: "retrieved on 2022-09-26" + url: "https://en.wikipedia.org/wiki/Cross-covariance_matrix#Definition" + +def_id: "D176" +shortcut: "covmat-cross" +username: "JoramSoch" +--- + + +**Definition:** Let $X = [X_1, \ldots, X_n]^\mathrm{T}$ and $Y = [Y_1, \ldots, Y_m]^\mathrm{T}$ be two [random vectors](/D/rvec) that can or cannot be of equal size. Then, the cross-covariance matrix of $X$ and $Y$ is defined as the $n \times m$ matrix in which the entry $(i,j)$ is the [covariance](/D/cov) of $X_i$ and $Y_j$: + +$$ \label{eq:covmat-cross} +\Sigma_{XY} = +\begin{bmatrix} +\mathrm{Cov}(X_1,Y_1) & \ldots & \mathrm{Cov}(X_1,Y_m) \\ +\vdots & \ddots & \vdots \\ +\mathrm{Cov}(X_n,Y_1) & \ldots & \mathrm{Cov}(X_n,Y_m) +\end{bmatrix} = +\begin{bmatrix} +\mathrm{E}\left[ (X_1-\mathrm{E}[X_1]) (Y_1-\mathrm{E}[Y_1]) \right] & \ldots & \mathrm{E}\left[ (X_1-\mathrm{E}[X_1]) (Y_m-\mathrm{E}[Y_m]) \right] \\ +\vdots & \ddots & \vdots \\ +\mathrm{E}\left[ (X_n-\mathrm{E}[X_n]) (Y_1-\mathrm{E}[Y_1]) \right] & \ldots & \mathrm{E}\left[ (X_n-\mathrm{E}[X_n]) (Y_m-\mathrm{E}[Y_m]) \right] +\end{bmatrix} \; . +$$ \ No newline at end of file diff --git a/I/ToC.md b/I/ToC.md index 9120ac1b..465cf2fb 100644 --- a/I/ToC.md +++ b/I/ToC.md @@ -128,17 +128,23 @@ title: "Table of Contents"    1.9.1. *[Definition](/D/cov)*
   1.9.2. *[Sample covariance](/D/cov-samp)*
   1.9.3. **[Partition into expected values](/P/cov-mean)**
-    1.9.4. **[Covariance under independence](/P/cov-ind)**
-    1.9.5. **[Relationship to correlation](/P/cov-corr)**
-    1.9.6. **[Law of total covariance](/P/cov-tot)**
-    1.9.7. *[Covariance matrix](/D/covmat)*
-    1.9.8. *[Sample covariance matrix](/D/covmat-samp)*
-    1.9.9. **[Covariance matrix and expected values](/P/covmat-mean)**
-    1.9.10. **[Invariance under addition of vector](/P/covmat-inv)**
-    1.9.11. **[Scaling upon multiplication with matrix](/P/covmat-scal)**
-    1.9.12. **[Covariance matrix and correlation matrix](/P/covmat-corrmat)**
-    1.9.13. *[Precision matrix](/D/precmat)*
-    1.9.14. **[Precision matrix and correlation matrix](/P/precmat-corrmat)**
+    1.9.4. **[Symmetry](/P/cov-symm)**
+    1.9.5. **[Self-covariance](/P/cov-var)**
+    1.9.6. **[Covariance under independence](/P/cov-ind)**
+    1.9.7. **[Relationship to correlation](/P/cov-corr)**
+    1.9.8. **[Law of total covariance](/P/cov-tot)**
+    1.9.9. *[Covariance matrix](/D/covmat)*
+    1.9.10. *[Sample covariance matrix](/D/covmat-samp)*
+    1.9.11. **[Covariance matrix and expected values](/P/covmat-mean)**
+    1.9.12. **[Symmetry](/P/covmat-symm)**
+    1.9.13. **[Positive semi-definiteness](/P/covmat-psd)**
+    1.9.14. **[Invariance under addition of vector](/P/covmat-inv)**
+    1.9.15. **[Scaling upon multiplication with matrix](/P/covmat-scal)**
+    1.9.16. *[Cross-covariance matrix](/D/covmat-cross)*
+    1.9.17. **[Covariance matrix of a sum](/P/covmat-sum)**
+    1.9.18. **[Covariance matrix and correlation matrix](/P/covmat-corrmat)**
+    1.9.19. *[Precision matrix](/D/precmat)*
+    1.9.20. **[Precision matrix and correlation matrix](/P/precmat-corrmat)**
1.10. Correlation
   1.10.1. *[Definition](/D/corr)*
diff --git a/P/cov-symm.md b/P/cov-symm.md new file mode 100644 index 00000000..23236150 --- /dev/null +++ b/P/cov-symm.md @@ -0,0 +1,51 @@ +--- +layout: proof +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-09-26 12:14:00 + +title: "Symmetry of the covariance" +chapter: "General Theorems" +section: "Probability theory" +topic: "Covariance" +theorem: "Symmetry" + +sources: + - authors: "Wikipedia" + year: 2022 + title: "Covariance" + in: "Wikipedia, the free encyclopedia" + pages: "retrieved on 2022-09-26" + url: "https://en.wikipedia.org/wiki/Covariance#Covariance_of_linear_combinations" + +proof_id: "P353" +shortcut: "cov-symm" +username: "JoramSoch" +--- + + +**Theorem:** The [covariance](/D/cov) of two [random variables](/D/rvar) is a symmetric function: + +$$ \label{eq:cov-symm} +\mathrm{Cov}(X,Y) = \mathrm{Cov}(Y,X) \; . +$$ + + +**Proof:** The [covariance](/D/cov) of [random variables](/D/rvar) $X$ and $Y$ is defined as: + +$$ \label{eq:cov} +\mathrm{Cov}(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \; . +$$ + +Switching $X$ and $Y$ in \eqref{eq:cov}, we can easily see: + +$$ \label{eq:cov-symm-qed} +\begin{split} +\mathrm{Cov}(Y,X) &\overset{\eqref{eq:cov}}{=} \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X]) \right] \\ +&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \\ +&= \mathrm{Cov}(X,Y) \; . +\end{split} +$$ \ No newline at end of file diff --git a/P/cov-var.md b/P/cov-var.md new file mode 100644 index 00000000..1b16b8c0 --- /dev/null +++ b/P/cov-var.md @@ -0,0 +1,51 @@ +--- +layout: proof +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-09-26 12:08:00 + +title: "Self-covariance equals variance" +chapter: "General Theorems" +section: "Probability theory" +topic: "Covariance" +theorem: "Self-covariance" + +sources: + - authors: "Wikipedia" + year: 2022 + title: "Covariance" + in: "Wikipedia, the free encyclopedia" + pages: "retrieved on 2022-09-26" + url: "https://en.wikipedia.org/wiki/Covariance#Covariance_with_itself" + +proof_id: "P352" +shortcut: "cov-var" +username: "JoramSoch" +--- + + +**Theorem:** The [covariance](/D/cov) of a [random variable](/D/rvar) with itself is equal to the [variance](/D/var): + +$$ \label{eq:cov-var} +\mathrm{Cov}(X,X) = \mathrm{Var}(X) \; . +$$ + + +**Proof:** The [covariance](/D/cov) of [random variables](/D/rvar) $X$ and $Y$ is defined as: + +$$ \label{eq:cov} +\mathrm{Cov}(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \; . +$$ + +Inserting $X$ for $Y$ in \eqref{eq:cov}, the result is the [variance](/D/var) of $X$: + +$$ \label{eq:cov-var-qed} +\begin{split} +\mathrm{Cov}(X,X) &\overset{\eqref{eq:cov}}{=} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X]) \right] \\ +&= \mathrm{E}\left[ (X-\mathrm{E}[X])^2 \right] \\ +&= \mathrm{Var}(X) \; . +\end{split} +$$ \ No newline at end of file diff --git a/P/covmat-psd.md b/P/covmat-psd.md new file mode 100644 index 00000000..98a84ab1 --- /dev/null +++ b/P/covmat-psd.md @@ -0,0 +1,91 @@ +--- +layout: proof +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-09-26 11:26:00 + +title: "Positive semi-definiteness of the covariance matrix" +chapter: "General Theorems" +section: "Probability theory" +topic: "Covariance" +theorem: "Positive semi-definiteness" + +sources: + - authors: "hkBattousai" + year: 2013 + title: "What is the proof that covariance matrices are always semi-definite?" + in: "StackExchange Mathematics" + pages: "retrieved on 2022-09-26" + url: "https://math.stackexchange.com/a/327872" + - authors: "Wikipedia" + year: 2022 + title: "Covariance matrix" + in: "Wikipedia, the free encyclopedia" + pages: "retrieved on 2022-09-26" + url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties" + +proof_id: "P351" +shortcut: "covmat-psd" +username: "JoramSoch" +--- + + +**Theorem:** Each [covariance matrix](/D/covmat) is positive semi-definite: + +$$ \label{eq:covmat-symm} +a^\mathrm{T} \Sigma_{XX} a \geq 0 \quad \text{for all} \quad a \in \mathbb{R}^n \; . +$$ + + +**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows + +$$ \label{eq:covmat} +\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] +$$ + +A positive semi-definite matrix is a matrix whose eigenvalues are all non-negative or, equivalently, + +$$ \label{eq:psd} +M \; \text{pos. semi-def.} \quad \Leftrightarrow \quad x^\mathrm{T} M x \geq 0 \quad \text{for all} \quad x \in \mathbb{R}^n \; . +$$ + +Here, for an arbitrary real column vector $a \in \mathbb{R}^n$, we have: + +$$ \label{eq:covmat-symm-s1} +a^\mathrm{T} \Sigma_{XX} a \overset{\eqref{eq:covmat}}{=} a^\mathrm{T} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] a \; . +$$ + +Because the [expected value is a linear operator](/P/mean-lin), we can write: + +$$ \label{eq:covmat-symm-s2} +a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ a^\mathrm{T} (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} a \right] \; . +$$ + +Now define the [scalar random variable](/D/rvar) + +$$ \label{eq:Y-X} +Y = a^\mathrm{T} (X-\mu_X) \; . +$$ + +where $\mu_X = \mathrm{E}[X]$ and note that + +$$ \label{eq:YT-Y} +a^\mathrm{T} (X-\mu_X) = (X-\mu_X)^\mathrm{T} a \; . +$$ + +Thus, combing \eqref{eq:covmat-symm-s2} with \eqref{eq:Y-X}, we have: + +$$ \label{eq:covmat-symm-s3} +a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ Y^2 \right] \; . +$$ + +Because $Y^2$ is a random variable that cannot become negative and the [expected value of a strictly non-negative random variable is also non-negative](/P/mean-nonneg), we finally have + +$$ \label{eq:covmat-symm-s4} +a^\mathrm{T} \Sigma_{XX} a \geq 0 +$$ + +for any $a \in \mathbb{R}^n$. \ No newline at end of file diff --git a/P/covmat-sum.md b/P/covmat-sum.md new file mode 100644 index 00000000..b67c9892 --- /dev/null +++ b/P/covmat-sum.md @@ -0,0 +1,60 @@ +--- +layout: proof +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-09-26 10:37:00 + +title: "Covariance of the sum of two random vectors" +chapter: "General Theorems" +section: "Probability theory" +topic: "Covariance" +theorem: "Covariance matrix of a sum" + +sources: + - authors: "Wikipedia" + year: 2022 + title: "Covariance matrix" + in: "Wikipedia, the free encyclopedia" + pages: "retrieved on 2022-09-26" + url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties" + +proof_id: "P349" +shortcut: "covmat-sum" +username: "JoramSoch" +--- + + +**Theorem:** The [covariance matrix](/D/covmat) of the sum of two [random vectors](/D/rvec) of the same dimension equals the sum of the covariances of those random vectors, plus the sum of their [cross-covariances](/D/covmat-cross): + +$$ \label{eq:covmat-sum} +\Sigma(X+Y) = \Sigma_{XX} + \Sigma_{YY} + \Sigma_{XY} + \Sigma_{YX} \; . +$$ + + +**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows + +$$ \label{eq:covmat} +\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] +$$ + +and the [cross-covariance matrix](/D/covmat-cross) of $X$ and $Y$ can similarly be written as + +$$ \label{eq:covmat-cross} +\Sigma_{XY} = \Sigma(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] +$$ + +Using this and the [linearity of the expected value](/P/mean-lin) as well as the definitions of [covariance matrix](/D/covmat) and [cross-covariance matrix](/D/covmat-cross), we can derive \eqref{eq:covmat-sum} as follows: + +$$ \label{eq:covmat-sum-qed} +\begin{split} +\Sigma(X+Y) &\overset{\eqref{eq:covmat}}{=} \mathrm{E}\left[ ([X+Y]-\mathrm{E}[X+Y]) ([X+Y]-\mathrm{E}[X+Y])^\mathrm{T} \right] \\ +&= \mathrm{E}\left[ ([X-\mathrm{E}(X)] + [Y-\mathrm{E}(Y)]) ([X-\mathrm{E}(X)] + [Y-\mathrm{E}(Y)])^\mathrm{T} \right] \\ +&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} + (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} + (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} + (Y-\mathrm{E}[Y]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] \\ +&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] + \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] \\ +&\overset{\eqref{eq:covmat}}{=} \Sigma_{XX} + \Sigma_{YY} + \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} \right] \\ +&\overset{\eqref{eq:covmat-cross}}{=} \Sigma_{XX} + \Sigma_{YY} + \Sigma_{XY} + \Sigma_{YX} \; . +\end{split} +$$ \ No newline at end of file diff --git a/P/covmat-symm.md b/P/covmat-symm.md new file mode 100644 index 00000000..349a51b1 --- /dev/null +++ b/P/covmat-symm.md @@ -0,0 +1,70 @@ +--- +layout: proof +mathjax: true + +author: "Joram Soch" +affiliation: "BCCN Berlin" +e_mail: "joram.soch@bccn-berlin.de" +date: 2022-09-26 10:54:00 + +title: "Symmetry of the covariance matrix" +chapter: "General Theorems" +section: "Probability theory" +topic: "Covariance" +theorem: "Symmetry" + +sources: + - authors: "Wikipedia" + year: 2022 + title: "Covariance matrix" + in: "Wikipedia, the free encyclopedia" + pages: "retrieved on 2022-09-26" + url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties" + +proof_id: "P350" +shortcut: "covmat-symm" +username: "JoramSoch" +--- + + +**Theorem:** Each [covariance matrix](/D/covmat) is symmetric: + +$$ \label{eq:covmat-symm} +\Sigma_{XX}^\mathrm{T} = \Sigma_{XX} \; . +$$ + + +**Proof:** The [covariance matrix](/D/covmat) of a [random vector](/D/rvec) $X$ is defined as + +$$ \label{eq:covmat} +\Sigma_{XX} = +\begin{bmatrix} +\mathrm{Cov}(X_1,X_1) & \ldots & \mathrm{Cov}(X_1,X_n) \\ +\vdots & \ddots & \vdots \\ +\mathrm{Cov}(X_n,X_1) & \ldots & \mathrm{Cov}(X_n,X_n) +\end{bmatrix} \; . +$$ + +A symmetric matrix is a matrix whose transpose is equal to itself. The transpose of $\Sigma_{XX}$ is + +$$ \label{eq:covmat-trans} +\Sigma_{XX}^\mathrm{T} = +\begin{bmatrix} +\mathrm{Cov}(X_1,X_1) & \ldots & \mathrm{Cov}(X_n,X_1) \\ +\vdots & \ddots & \vdots \\ +\mathrm{Cov}(X_1,X_n) & \ldots & \mathrm{Cov}(X_n,X_n) +\end{bmatrix} \; . +$$ + +Because the [covariance is a symmetric function](/P/cov-symm), i.e. $\mathrm{Cov}(X,Y) = \mathrm{Cov}(Y,X)$, this matrix is equal to + +$$ \label{eq:covmat-symm-qed} +\Sigma_{XX}^\mathrm{T} = +\begin{bmatrix} +\mathrm{Cov}(X_1,X_1) & \ldots & \mathrm{Cov}(X_1,X_n) \\ +\vdots & \ddots & \vdots \\ +\mathrm{Cov}(X_n,X_1) & \ldots & \mathrm{Cov}(X_n,X_n) +\end{bmatrix} +$$ + +which is equivalent to our original definition in \eqref{eq:covmat}. \ No newline at end of file