added 1 definition and 5 proofs

D/covmat-cross.md

@@ -0,0 +1,44 @@
+---
+layout: definition
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2022-09-26 09:45:00
+
+title: "Cross-covariance matrix"
+chapter: "General Theorems"
+section: "Probability theory"
+topic: "Covariance"
+definition: "Cross-covariance matrix"
+
+sources:
+  - authors: "Wikipedia"
+    year: 2022
+    title: "Cross-covariance matrix"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2022-09-26"
+    url: "https://en.wikipedia.org/wiki/Cross-covariance_matrix#Definition"
+
+def_id: "D176"
+shortcut: "covmat-cross"
+username: "JoramSoch"
+---
+
+
+**Definition:** Let $X = [X_1, \ldots, X_n]^\mathrm{T}$ and $Y = [Y_1, \ldots, Y_m]^\mathrm{T}$ be two [random vectors](/D/rvec) that can or cannot be of equal size. Then, the cross-covariance matrix of $X$ and $Y$ is defined as the $n \times m$ matrix in which the entry $(i,j)$ is the [covariance](/D/cov) of $X_i$ and $Y_j$:
+
+$$ \label{eq:covmat-cross}
+\Sigma_{XY} =
+\begin{bmatrix}
+\mathrm{Cov}(X_1,Y_1) & \ldots & \mathrm{Cov}(X_1,Y_m) \\
+\vdots & \ddots & \vdots \\
+\mathrm{Cov}(X_n,Y_1) & \ldots & \mathrm{Cov}(X_n,Y_m)
+\end{bmatrix} =
+\begin{bmatrix}
+\mathrm{E}\left[ (X_1-\mathrm{E}[X_1]) (Y_1-\mathrm{E}[Y_1]) \right] & \ldots & \mathrm{E}\left[ (X_1-\mathrm{E}[X_1]) (Y_m-\mathrm{E}[Y_m]) \right] \\
+\vdots & \ddots & \vdots \\
+\mathrm{E}\left[ (X_n-\mathrm{E}[X_n]) (Y_1-\mathrm{E}[Y_1]) \right] & \ldots & \mathrm{E}\left[ (X_n-\mathrm{E}[X_n]) (Y_m-\mathrm{E}[Y_m]) \right]
+\end{bmatrix} \; .
+$$

I/ToC.md

@@ -128,17 +128,23 @@ title: "Table of Contents"
    &emsp;&ensp; 1.9.1. *[Definition](/D/cov)* <br>
    &emsp;&ensp; 1.9.2. *[Sample covariance](/D/cov-samp)* <br>
    &emsp;&ensp; 1.9.3. **[Partition into expected values](/P/cov-mean)** <br>
-   &emsp;&ensp; 1.9.4. **[Covariance under independence](/P/cov-ind)** <br>
-   &emsp;&ensp; 1.9.5. **[Relationship to correlation](/P/cov-corr)** <br>
-   &emsp;&ensp; 1.9.6. **[Law of total covariance](/P/cov-tot)** <br>
-   &emsp;&ensp; 1.9.7. *[Covariance matrix](/D/covmat)* <br>
-   &emsp;&ensp; 1.9.8. *[Sample covariance matrix](/D/covmat-samp)* <br>
-   &emsp;&ensp; 1.9.9. **[Covariance matrix and expected values](/P/covmat-mean)** <br>
-   &emsp;&ensp; 1.9.10. **[Invariance under addition of vector](/P/covmat-inv)** <br>
-   &emsp;&ensp; 1.9.11. **[Scaling upon multiplication with matrix](/P/covmat-scal)** <br>
-   &emsp;&ensp; 1.9.12. **[Covariance matrix and correlation matrix](/P/covmat-corrmat)** <br>
-   &emsp;&ensp; 1.9.13. *[Precision matrix](/D/precmat)* <br>
-   &emsp;&ensp; 1.9.14. **[Precision matrix and correlation matrix](/P/precmat-corrmat)** <br>
+   &emsp;&ensp; 1.9.4. **[Symmetry](/P/cov-symm)** <br>
+   &emsp;&ensp; 1.9.5. **[Self-covariance](/P/cov-var)** <br>
+   &emsp;&ensp; 1.9.6. **[Covariance under independence](/P/cov-ind)** <br>
+   &emsp;&ensp; 1.9.7. **[Relationship to correlation](/P/cov-corr)** <br>
+   &emsp;&ensp; 1.9.8. **[Law of total covariance](/P/cov-tot)** <br>
+   &emsp;&ensp; 1.9.9. *[Covariance matrix](/D/covmat)* <br>
+   &emsp;&ensp; 1.9.10. *[Sample covariance matrix](/D/covmat-samp)* <br>
+   &emsp;&ensp; 1.9.11. **[Covariance matrix and expected values](/P/covmat-mean)** <br>
+   &emsp;&ensp; 1.9.12. **[Symmetry](/P/covmat-symm)** <br>
+   &emsp;&ensp; 1.9.13. **[Positive semi-definiteness](/P/covmat-psd)** <br>
+   &emsp;&ensp; 1.9.14. **[Invariance under addition of vector](/P/covmat-inv)** <br>
+   &emsp;&ensp; 1.9.15. **[Scaling upon multiplication with matrix](/P/covmat-scal)** <br>
+   &emsp;&ensp; 1.9.16. *[Cross-covariance matrix](/D/covmat-cross)* <br>
+   &emsp;&ensp; 1.9.17. **[Covariance matrix of a sum](/P/covmat-sum)** <br>
+   &emsp;&ensp; 1.9.18. **[Covariance matrix and correlation matrix](/P/covmat-corrmat)** <br>
+   &emsp;&ensp; 1.9.19. *[Precision matrix](/D/precmat)* <br>
+   &emsp;&ensp; 1.9.20. **[Precision matrix and correlation matrix](/P/precmat-corrmat)** <br>
 
    1.10. Correlation <br>
    &emsp;&ensp; 1.10.1. *[Definition](/D/corr)* <br>

P/cov-symm.md

@@ -0,0 +1,51 @@
+---
+layout: proof
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2022-09-26 12:14:00
+
+title: "Symmetry of the covariance"
+chapter: "General Theorems"
+section: "Probability theory"
+topic: "Covariance"
+theorem: "Symmetry"
+
+sources:
+  - authors: "Wikipedia"
+    year: 2022
+    title: "Covariance"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2022-09-26"
+    url: "https://en.wikipedia.org/wiki/Covariance#Covariance_of_linear_combinations"
+
+proof_id: "P353"
+shortcut: "cov-symm"
+username: "JoramSoch"
+---
+
+
+**Theorem:** The [covariance](/D/cov) of two [random variables](/D/rvar) is a symmetric function:
+
+$$ \label{eq:cov-symm}
+\mathrm{Cov}(X,Y) = \mathrm{Cov}(Y,X) \; .
+$$
+
+
+**Proof:** The [covariance](/D/cov) of [random variables](/D/rvar) $X$ and $Y$ is defined as:
+
+$$ \label{eq:cov}
+\mathrm{Cov}(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \; .
+$$
+
+Switching $X$ and $Y$ in \eqref{eq:cov}, we can easily see:
+
+$$ \label{eq:cov-symm-qed}
+\begin{split}
+\mathrm{Cov}(Y,X) &\overset{\eqref{eq:cov}}{=} \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X]) \right] \\
+&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \\
+&= \mathrm{Cov}(X,Y) \; .
+\end{split}
+$$

P/cov-var.md

@@ -0,0 +1,51 @@
+---
+layout: proof
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2022-09-26 12:08:00
+
+title: "Self-covariance equals variance"
+chapter: "General Theorems"
+section: "Probability theory"
+topic: "Covariance"
+theorem: "Self-covariance"
+
+sources:
+  - authors: "Wikipedia"
+    year: 2022
+    title: "Covariance"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2022-09-26"
+    url: "https://en.wikipedia.org/wiki/Covariance#Covariance_with_itself"
+
+proof_id: "P352"
+shortcut: "cov-var"
+username: "JoramSoch"
+---
+
+
+**Theorem:** The [covariance](/D/cov) of a [random variable](/D/rvar) with itself is equal to the [variance](/D/var):
+
+$$ \label{eq:cov-var}
+\mathrm{Cov}(X,X) = \mathrm{Var}(X) \; .
+$$
+
+
+**Proof:** The [covariance](/D/cov) of [random variables](/D/rvar) $X$ and $Y$ is defined as:
+
+$$ \label{eq:cov}
+\mathrm{Cov}(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y]) \right] \; .
+$$
+
+Inserting $X$ for $Y$ in \eqref{eq:cov}, the result is the [variance](/D/var) of $X$:
+
+$$ \label{eq:cov-var-qed}
+\begin{split}
+\mathrm{Cov}(X,X) &\overset{\eqref{eq:cov}}{=} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X]) \right] \\
+&= \mathrm{E}\left[ (X-\mathrm{E}[X])^2 \right] \\
+&= \mathrm{Var}(X) \; .
+\end{split}
+$$

P/covmat-psd.md

@@ -0,0 +1,91 @@
+---
+layout: proof
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2022-09-26 11:26:00
+
+title: "Positive semi-definiteness of the covariance matrix"
+chapter: "General Theorems"
+section: "Probability theory"
+topic: "Covariance"
+theorem: "Positive semi-definiteness"
+
+sources:
+  - authors: "hkBattousai"
+    year: 2013
+    title: "What is the proof that covariance matrices are always semi-definite?"
+    in: "StackExchange Mathematics"
+    pages: "retrieved on 2022-09-26"
+    url: "https://math.stackexchange.com/a/327872"
+  - authors: "Wikipedia"
+    year: 2022
+    title: "Covariance matrix"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2022-09-26"
+    url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
+
+proof_id: "P351"
+shortcut: "covmat-psd"
+username: "JoramSoch"
+---
+
+
+**Theorem:** Each [covariance matrix](/D/covmat) is positive semi-definite:
+
+$$ \label{eq:covmat-symm}
+a^\mathrm{T} \Sigma_{XX} a \geq 0 \quad \text{for all} \quad a \in \mathbb{R}^n \; .
+$$
+
+
+**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows
+
+$$ \label{eq:covmat}
+\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right]
+$$
+
+A positive semi-definite matrix is a matrix whose eigenvalues are all non-negative or, equivalently,
+
+$$ \label{eq:psd}
+M \; \text{pos. semi-def.} \quad \Leftrightarrow \quad x^\mathrm{T} M x \geq 0 \quad \text{for all} \quad x \in \mathbb{R}^n \; .
+$$
+
+Here, for an arbitrary real column vector $a \in \mathbb{R}^n$, we have:
+
+$$ \label{eq:covmat-symm-s1}
+a^\mathrm{T} \Sigma_{XX} a \overset{\eqref{eq:covmat}}{=} a^\mathrm{T} \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] a \; .
+$$
+
+Because the [expected value is a linear operator](/P/mean-lin), we can write:
+
+$$ \label{eq:covmat-symm-s2}
+a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ a^\mathrm{T} (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} a \right] \; .
+$$
+
+Now define the [scalar random variable](/D/rvar)
+
+$$ \label{eq:Y-X}
+Y = a^\mathrm{T} (X-\mu_X) \; .
+$$
+
+where $\mu_X = \mathrm{E}[X]$ and note that
+
+$$ \label{eq:YT-Y}
+a^\mathrm{T} (X-\mu_X) = (X-\mu_X)^\mathrm{T} a \; .
+$$
+
+Thus, combing \eqref{eq:covmat-symm-s2} with \eqref{eq:Y-X}, we have:
+
+$$ \label{eq:covmat-symm-s3}
+a^\mathrm{T} \Sigma_{XX} a = \mathrm{E}\left[ Y^2 \right] \; .
+$$
+
+Because $Y^2$ is a random variable that cannot become negative and the [expected value of a strictly non-negative random variable is also non-negative](/P/mean-nonneg), we finally have
+
+$$ \label{eq:covmat-symm-s4}
+a^\mathrm{T} \Sigma_{XX} a \geq 0
+$$
+
+for any $a \in \mathbb{R}^n$.

P/covmat-sum.md

@@ -0,0 +1,60 @@
+---
+layout: proof
+mathjax: true
+
+author: "Joram Soch"
+affiliation: "BCCN Berlin"
+e_mail: "joram.soch@bccn-berlin.de"
+date: 2022-09-26 10:37:00
+
+title: "Covariance of the sum of two random vectors"
+chapter: "General Theorems"
+section: "Probability theory"
+topic: "Covariance"
+theorem: "Covariance matrix of a sum"
+
+sources:
+  - authors: "Wikipedia"
+    year: 2022
+    title: "Covariance matrix"
+    in: "Wikipedia, the free encyclopedia"
+    pages: "retrieved on 2022-09-26"
+    url: "https://en.wikipedia.org/wiki/Covariance_matrix#Basic_properties"
+
+proof_id: "P349"
+shortcut: "covmat-sum"
+username: "JoramSoch"
+---
+
+
+**Theorem:** The [covariance matrix](/D/covmat) of the sum of two [random vectors](/D/rvec) of the same dimension equals the sum of the covariances of those random vectors, plus the sum of their [cross-covariances](/D/covmat-cross):
+
+$$ \label{eq:covmat-sum}
+\Sigma(X+Y) = \Sigma_{XX} + \Sigma_{YY} + \Sigma_{XY} + \Sigma_{YX} \; .
+$$
+
+
+**Proof:** The [covariance matrix](/D/covmat) of $X$ [can be expressed](/P/covmat-mean) in terms of [expected values](/D/mean) as follows
+
+$$ \label{eq:covmat}
+\Sigma_{XX} = \Sigma(X) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right]
+$$
+
+and the [cross-covariance matrix](/D/covmat-cross) of $X$ and $Y$ can similarly be written as
+
+$$ \label{eq:covmat-cross}
+\Sigma_{XY} = \Sigma(X,Y) = \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right]
+$$
+
+Using this and the [linearity of the expected value](/P/mean-lin) as well as the definitions of [covariance matrix](/D/covmat) and [cross-covariance matrix](/D/covmat-cross), we can derive \eqref{eq:covmat-sum} as follows:
+
+$$ \label{eq:covmat-sum-qed}
+\begin{split}
+\Sigma(X+Y) &\overset{\eqref{eq:covmat}}{=} \mathrm{E}\left[ ([X+Y]-\mathrm{E}[X+Y]) ([X+Y]-\mathrm{E}[X+Y])^\mathrm{T} \right] \\
+&= \mathrm{E}\left[ ([X-\mathrm{E}(X)] + [Y-\mathrm{E}(Y)]) ([X-\mathrm{E}(X)] + [Y-\mathrm{E}(Y)])^\mathrm{T} \right] \\
+&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} + (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} + (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} + (Y-\mathrm{E}[Y]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] \\
+&= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] + \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] \\
+&\overset{\eqref{eq:covmat}}{=} \Sigma_{XX} + \Sigma_{YY} + \mathrm{E}\left[ (X-\mathrm{E}[X]) (Y-\mathrm{E}[Y])^\mathrm{T} \right] + \mathrm{E}\left[ (Y-\mathrm{E}[Y]) (X-\mathrm{E}[X])^\mathrm{T} \right] \\
+&\overset{\eqref{eq:covmat-cross}}{=} \Sigma_{XX} + \Sigma_{YY} + \Sigma_{XY} + \Sigma_{YX} \; .
+\end{split}
+$$