Orthogonality in Linear Algebra

The final piece in our theoretical module deals with orthogonality. Orthogonal vectors are our preferred vectors for most uses, so we need to look more closely at them. Everything in here is about orthogonality and how we can get it or use it. It's not just a random list of facts about orthogonality, though. The ideas hopefully build on each other in a logical way.

As a reminder, our key fact about orthogonality is:

Two vectors are orthogonal if their dot product is 0.

Orthogonal Subspaces

We'll start with a definition:

Definition: Let $\large W$ be a subspace of $\large \mathbb{R}^n$. The set of all vectors orthogonal to $\large W$ is called the orthogonal complement of $\large W$, and we write $\large W^\perp$.

As an example, let $\large W$ be a plane through the origin in $\large \mathbb{R}^3$ (we've seen that a plane is a subspace of $\large \mathbb{R}^3$). What is $\large W^\perp$? What vectors are orthogonal to the plane? The image reminds us that there is a single direction perpendicular to a plane:

There is a single basis vector orthogonal to the plane (the blue vector). The subspace is the set of all linear combinations of that basis vector. In other words, $\large W^\perp$ is a line in $\large\mathbb{R}^3$ (the red line).

$\large W$ and $\large W^\perp$ breakdown

Here's a neat result with orthogonal subspaces. Suppose we have a subspace $\large W$ of $\large \mathbb{R}^n$. Every vector $\large \vec{x}\in \mathbb{R}^n$ is the sum of a vector $\large \hat{x}\in W$ and a vector $\large \vec{z}\in W^\perp$. That is $\large \vec{x}=\hat{x}+\vec{z}$.

Note

The intuition here is that a vector space can be broken up into two independent or separate subpaces, $\large W$ and $\large W^\perp$. In data, this breaks the data into two uncorrelated bunches, two nonoverlapping sets of information. In the theorem $\large \hat{x}$ is the part of $\large \vec{x}$ in our subspace, and the part not in our subspace, $\large \vec{z}$, is completely orthogonal to the part in our subspace.

The Four Subspace Theorem

There is a theorem that we will use in optimization problems in our next module:

Theorem: Let $\large A$ be an $\large m\times n$ matrix. Then $\large \left( \text{Row}(A)\right)^\perp=\text{Null}(A)$ and $\large \left( \text{Col}(A)\right)^\perp=\text{Null}(A^T)$.

At first glance, this looks like the kind of thing you'd need to think hard about to understand. It's not that bad though. For the first statement, let's remember that $\large \text{Null}(A)$ is made up of solutions of $\large A\vec{x}=\vec{0}$, and $\large A\vec{x}$ is just a bunch of dot products between the rows of $\large A$ and $\large \vec{x}$. So, yeah, the rows (and linear combinations of the rows) of $\large A$ are orthogonal to anything in $\large \text{Null}(A)$.

The second statement is just the first statement applied to the matrix $\large A^T$ (since the rows of $\large A$ are the columns of $\large A^T$).

Note

Here's how we think of this theorem: we have an $\large m\times n$ matrix $\large A$. If we look at the product $\large A\vec{x}$ for any $\large \vec{x} \in \mathbb{R}^n$, we can think of part of $\vec{x}$ getting sent by $\large A$ to zero and another, orthogonal part getting sent to something nonzero.

Orthogonal Sets of Vectors

For reasons we will see throughout the rest of the class, orthogonal vectors are the best vectors to work with!

Definition: A set of vectors $\large \vec{v_1},\vec{v_2},\dots,\vec{v_m}$ in $\large \mathbb{R}^n$ is an orthogonal set of vectors if each pair of vectors is orthogonal (if $\large \vec{v_i}\cdot\vec{v_j}=0$ for all $\large i\ne j$).

Example: Show that the following is an orthogonal set of vectors:

$$\large \vec{v_1}=\begin{bmatrix} 3 \\ 1\\ 1\\ \end{bmatrix}, \vec{v_2}=\begin{bmatrix} -1 \\ 2\\ 1\\ \end{bmatrix}, \vec{v_3}=\begin{bmatrix} -1/2 \\ -2\\ 7/2\\ \end{bmatrix} $$

We'll see in the video that these do make up an orthogonal set.

Now, a theorem that illustrates why we care about orthogonal sets of vectors:

Theorem: An orthogonal set of nonzero vectors is linearly independent.

The proof is short and is a good example of how these types of arguments are made. Please watch it in the video:

Note

One consequence of the theorem is that a set of orthogonal vectors necessarily forms a basis for whatever subspace they span!

Definition: An orthogonal basis for a subspace is a basis that is also an orthogonal set.

Now, in general if we want to write a given vector as a linear combination of basis vectors, we need to solve a system of linear equations (Gaussian elimination...yay!) to find the coefficients in the linear combination. But if we have an orthogonal basis, it's much easier!

Theorem: If $\large \lbrace \vec{v_1},\vec{v_2},\dots,\vec{v_m}\rbrace$ is an orthogonal basis for $\large W$, then any $\large \vec{u}\in W$ can be written $\large \vec{u} = c_1\vec{v_1}+c_2\vec{v_2}+\dots+c_m\vec{v_m}$ where the coefficients are:

$$\large c_j = \frac{\vec{u}\cdot\vec{v_j}}{\vec{v_j}\cdot\vec{v_j}}$$

Note

The proof of this theorem mirrors the proof in the video above. Just start with $\large \vec{u} = c_1\vec{v_1}+c_2\vec{v_2}+\dots+c_m\vec{v_m}$, take the dot product of $\large \vec{v_j}$ with both sides of the equation, and solve for $\large c_j$.

Here's an example of how to use the theorem. We found earlier that the set:

$$\large \vec{v_1}=\begin{bmatrix} 3 \\ 1\\ 1\\ \end{bmatrix}, \vec{v_2}=\begin{bmatrix} -1 \\ 2\\ 1\\ \end{bmatrix}, \vec{v_3}=\begin{bmatrix} -1/2 \\ -2\\ 7/2\\ \end{bmatrix} $$

is an orthogonal set in $\large \mathbb{R}^3$. Since it's orthogonal, it's linearly independent. A linearly independent set of 3 vectors in $\large \mathbb{R}^3$ is necessarily a basis for $\large \mathbb{R}^3$. So we have an orthogonal basis. Write the vector $\large \left<6,1,-8\right>$ as a linear combination of these basis vectors. The solution is in the video:

Orthonormal

We can repeat all of the above with unit vectors (vectors of length one, $\large \lVert\vec{v}\rVert=1$), and we get definitions for orthonormal set and orthonornal basis. If we have an orthonormal basis the formula for coefficients in a linear combination of basis vectors is even simpler. Since $\large \vec{v_j}\cdot\vec{v_j}=\lVert\vec{v_j}\rVert^2=1$, we get

$$\large c_j = \vec{u}\cdot\vec{v_j}$$

Note

An orthonormal basis is the best kind of basis. But there is a conservation of misery principle at work here: it's easier to use an orthonormal basis, but it's harder to get one!

Orthogonal Projections

In Calc II, we learned about the projection of one vector onto another. The projection of $\large \vec{u}$ onto $\large \vec{v}$ is a measure of how much of $\large \vec{u}$ points in the same direction as $\large \vec{v}$. It breaks $\large \vec{u}$ into a vector parallel to $\large \vec{v}$ and a vector orthogonal to $\large \vec{v}$. In the pictures, we see examples of projections of vectors $\large \vec{u}$ (green) onto vectors $\large \vec{v}$ (blue). The red vectors are the projections, the components of $\large \vec{u}$ in the direction of $\large \vec{v}$. The dotted black vectors are the components of $\large \vec{u}$ orthogonal to $\large \vec{v}$.

The formula we learned in calculus for the projection of $\large \vec{u}$ onto $\large \vec{v}$ is:

$$\large \text{proj}_{\vec{v}}\vec{u}=\frac{\vec{u}\cdot\vec{v}}{\lVert \vec{v}\rVert ^2}\vec{v}$$

Even though we can only visualize projections in $\large \mathbb{R}^2$ and $\large \mathbb{R}^3$, we can still calculate them in $\large \mathbb{R}^n$. The formula is the same! Again, this will break up a vector into components that are parallel and orthogonal to a given direction.

Projection onto a Subspace $\large W$

Now, here's a new idea, and a big one. Instead of the projection of $\large \vec{u}$ onto a vector, we will define the orthogonal projection of $\large \vec{u}$ onto an entire subspace! We can visualize this in $\large \mathbb{R}^3$. Let $\large W$ be a plane (which, as we've seen, is a subspace). The projection of $\large \vec{u}$ onto this subspace is shown (vector is blue, projection onto $\large W$ is red, orthogonal component is purple):

The idea of the projection is that any vector $\large \vec{u}\in \mathbb{R}^n$ can be split up into two orthogonal components, one that is in the subspace $\large W$ and one that is in $\large W^\perp$. That is, $\large \vec{u}=\hat{u}+\vec{z}$, where $\large \hat{u}\in W$ and $\large \vec{z}\in W^\perp$. We call $\large \hat{u}$ the orthogonal projection of $\large \vec{u}$ onto $\large W$, and we write $\large \hat{u}=\text{proj}_W\vec{u}$.

Note

In this situation, we have a vector $\large \vec{u}$ that is not in $\large W$. With $\large \text{proj}_W\vec{u}$, we get the vector in $\large W$ that is closest to $\large \vec{u}$. In fact, this is a key fact of orthogonal projection onto a subspace:

Theorem: Let $\large \hat{u}$ be the projection of $\large \vec{u}$ onto a subspace $\large W$, $\large \hat{u}=\text{proj}_W\vec{u}$. Then $\large\lVert \vec{u}-\hat{u}\rVert \le \lVert \vec{u}-\vec{v}\rVert$ for any $\large \vec{v}\in W$.

Note

Whenever we see superlatives in math - closest, smallest, best - we should think about optimization. This case is no different. We will use the theorem above about projections onto a subspace in the next module, which is all about a key optimization technique known as Least Squares.

We now have a picture and an idea. We'd like to project a vector onto a subspace; we'd like to find the vector in $\large W$ closest to $\large \vec{u}$. How do we do this? The idea is to strip away, one at a time, every part of $\large \vec{u}$ that is orthogonal to the vectors in $\large W$. If we can do that, we will be left with a vector $\large \hat{u}=\text{proj}_W\vec{u}$ that is parallel to everything in $\large W$.

Here's our situation and formulas, which are based on the idea of subtracting off each orthogonal direction from $\large \vec{u}$:

Theorem: Let $\large \vec{u}$ be a vector in $\large \mathbb{R}^n$, and let $\large W$ be a subspace of $\large \mathbb{R}^n$ with orthogonal basis $\large \lbrace \vec{v_1}, \vec{v_2},\dots,\vec{v_m} \rbrace$. Then there are vectors $\large \hat{u}\in W$ and $\large \vec{z}\in W^\perp$ such that $\large \vec{u}=\hat{u}+\vec{z}$. The formulas for $\large \hat{u}$ and $\large \vec{z}$ are:

$$\large \begin{split} \vec{z}&=\vec{u}-\frac{\vec{u}\cdot\vec{v_1}}{\lVert \vec{v_1} \rVert ^2} \vec{v_1} -\frac{\vec{u}\cdot\vec{v_2}}{\lVert \vec{v_2} \rVert ^2} \vec{v_2} -\dots -\frac{\vec{u}\cdot\vec{v_m}}{\lVert \vec{v_m} \rVert ^2} \vec{v_m}\\ \hat{u}&=\text{proj}_W\vec{u}=\frac{\vec{u}\cdot\vec{v_1}}{\lVert \vec{v_1} \rVert ^2} \vec{v_1} +\frac{\vec{u}\cdot\vec{v_2}}{\lVert \vec{v_2} \rVert ^2} \vec{v_2} +\dots +\frac{\vec{u}\cdot\vec{v_m}}{\lVert \vec{v_m} \rVert ^2} \vec{v_m}\\ \end{split} $$

Note

In the formula above, we start with $\large \vec{u}$ and we substract off, one at a time, the projection of $\large \vec{u}$ onto each of the basis vectors of $\large W$. We peel off every part of $\large \vec{u}$ that lies in $\large W$, and we are left with the component of $\large \vec{u}$ that lies in $\large W^\perp$.

This is a lot, and it's and the end of one of our longest lectures all semester. Let's do an example that helpfully makes it more clear. The vectors $\large \vec{v_1}=\left< 2,5,-1\right>$ and $\large \vec{v_2}=\left<-2, 1, 1\right>$ form an orthogonal (test them!) basis for $\large W=\text{Span}\lbrace \vec{v_1},\vec{v_2}\rbrace$, a plane in $\large \mathbb{R}^3$. Find the orthogonal projection of $\large \vec{u}=\left<1,2,3\right>$ onto this plane. The work can be seen in the video:

The image below shows the plane and vectors in this particular example:

Section Outcomes

I can:

• Recall and use the definitions of orthogonal set, orthogonal basis, orthogonal subspace, and orthogonal projection.
• Recognize an othogonal/orthonormal set of vectors.
• Recall and use theorems related to orthogonal/orthonormal sets/bases.
• Write a vector as a linear combination of an orthogonal/orthonormal basis.
• Find the orthogonal projection of a vector onto a subspace.