Linear Algebra Crash Course

👨‍🔬 Linear Algebra Crash Course

Please cross-reference the linked Wikipedia articles as you read; it is good to get comfortable with other notations!

Also try coming up with small examples and working them out on paper as you go. 🙂👍

Notation and Conventions

Indexing

I will be "zero-indexing" my vectors and matrices, i.e. the top entry in vector $\mathbf{b}$ is $b_0$ and the top-left entry in matrix $\mathbf{A}$ is $A_{0, 0}$. Zero-indexing is not as typical as one-indexing, but it simplifies translating between LaTeX and C/C++.

Scalars

Let's use normal type for scalars, e.g., real number $k \in \mathbb{R}$ or integer $n \in \mathbb{Z}$.

Vectors

Let's use bold lowercase for vectors like $\mathbf{b}$. We can say $\mathbf{b}$ has $n$ entries, or is an $n$-vector.

$$\mathbf{b} = \begin{bmatrix}b_0\\\vdots\\b_{n-1}\end{bmatrix}.$$

By convention, we will assume "vector" refers to a column vector, i.e., a vector with one column and any number of rows. A row vector is the transpose of a column vector. We read $\mathbf{b}^T$ as "b transpose."

$$\mathbf{b}^T = \begin{bmatrix}b_0 & \cdots & b_{n-1}\end{bmatrix}$$

• The zero vector $\mathbf{0}$ is a vector of all zeros. My choice of notation doesn't specify the number of entries, but it's typically clear from the context.

Matrices

Let's use bold uppercase for matrices like $\mathbf{A}$. We can say $\mathbf{A}$ has $R$ rows and $C$ columns, or is an $R \times C$-matrix for short. We read $R \times C$ as "R by C."

$$\mathbf{A} = \begin{bmatrix} A_{0, 0} & \cdots & A_{0, C - 1}\\\ \vdots & \ddots & \vdots\\\ A_{R-1, 0} & \cdots & A_{R-1, C - 1} \end{bmatrix}$$

• $A_{r, c}$ is the entry in $\mathbf{A}$'s $r$-th row and $c$-th column.
• $\mathbf{A}$ has $R*C$ entries.
• If $C = 1$, then $\mathbf{A}$ is also a column vector.
• If $R = 1$, then $\mathbf{A}$ is also a row vector.
• If $R = C = 1$, then $\mathbf{A}$ is also a scalar.
• If $R = C$, then $\mathbf{A}$ is a square matrix. We could also say, "A is square."
• The identity matrix $\mathbf{I}$ is a square matrix with the property $\mathbf{AI}=\mathbf{IA}=\mathbf{A}$. My choice of notation doesn't specify the number of rows/cols, but it's typically clear from the context.

Block Notation

It's sometimes convenient to think about matrices and vectors as being broken up into sections called "blocks."

E.g., if

$$\mathbf{R} = \begin{bmatrix} R_{0, 0} & R_{0, 1} & R_{0, 2} \\\ R_{1, 0} & R_{1, 1} & R_{1, 2} \\\ R_{2, 0} & R_{2, 1} & R_{2, 2} \end{bmatrix}$$

and

$$\mathbf{t} = \begin{bmatrix} t_0 \\\ t_1 \\\ t_2 \end{bmatrix},$$

then instead of writing

$$\mathbf{C} = \begin{bmatrix} R_{0, 0} & R_{0, 1} & R_{0, 2} & t_0 \\\ R_{1, 0} & R_{1, 1} & R_{1, 2} & t_1 \\\ R_{2, 0} & R_{2, 1} & R_{2, 2} & t_2 \\\ 0 & 0 & 0 & 1 \end{bmatrix},$$

we could write

$$\mathbf{C} = \begin{bmatrix}\mathbf{R}&\mathbf{t}\\\mathbf{0}^T&1\end{bmatrix}.$$

Matrix-Matrix Multiplication

Consider $R \times K$-matrix $\mathbf{A}$ and $K \times C$-matrix $\mathbf{B}$. We can multiply $\mathbf{A}\mathbf{B}$ if and only if the number of columns in $\mathbf{A}$ is equal to the number of rows in $\mathbf{B}$. The result of this multiplication will have the same number of rows as $\mathbf{A}$ and the same number of columns as $\mathbf{B}$.

In summary, $[R \times K][K \times C]\rightarrow[R \times C].$

The $(i, j)$-th entry of matrix $\mathbf{C} = \mathbf{A}\mathbf{B}$ is

$$C_{i, j} = \sum_{k = 0}^{K - 1}A_{i, k}B_{k, j}$$

Matrix multiplication is associative, i.e. $(\mathbf{A}\mathbf{B})\mathbf{C} = \mathbf{A}(\mathbf{B}\mathbf{C}).$

Matrix multiplication is NOT commutative, i.e. $\mathbf{A}\mathbf{B}$ does NOT, in general, equal $\mathbf{B}\mathbf{A}.$

Square Matrix-Vector

If $\mathbf{A}$ is a square matrix and $\mathbf{b}$ is a vector with as many entries as $\mathbf{A}$ has rows/columns, then $\mathbf{A}\mathbf{b}=\mathbf{c}$ is a vector with the same number of entries as $\mathbf{b}$.

Inner Product (Dot Product)

If $\mathbf{a}$ and $\mathbf{b}$ are vectors with the same number of entries, then $\mathbf{a}^T \mathbf{b} = c$ is a scalar. Note that $\mathbf{a}^T\mathbf{b} = \mathbf{a} \cdot \mathbf{b}.$

Outer Product

If $\mathbf{a}$ and $\mathbf{b}$ are vectors with the same number of entries, then $\mathbf{a} \mathbf{b}^T = \mathbf{C}$ is a square matrix with as many rows as $\mathbf{a}$ has entries.

Other Vector Operations

Scalar Multiplication

Scalar multiplication scales a vector, i.e. preserves its direction and changes its length. E.g.,

$$a\mathbf{b} = \begin{bmatrix}ab_0\\\vdots\\ab_{n-1}\end{bmatrix}.$$

Component-wise Multiplication of Two Vectors

This is a fairly uncommon operation.

$$\mathbf{a} \circ \mathbf{b} = \begin{bmatrix}a_0b_0\\\vdots\\a_{n-1}b_{n-1}\end{bmatrix}$$

Length (The Euclidean Norm)

The length of a vector $\mathbf{a}$, also known as its magnitude or norm, is denoted by $|\mathbf{a}|$ (or $||\mathbf{a}||$) and is defined as the square root of the sum of the squares of its components.

$$|\mathbf{a}| = \sqrt{\sum_i {a_i}^2}$$

Direction (Normalizing a Vector)

To normalize a vector $\mathbf{a}$, you need to divide each of its components by its length $| \mathbf{a} |$, so that the resulting vector has length $1$, i.e., is a unit vector or direction vector. This is equivalent to multiplying $\mathbf{a}$ by the scalar $\frac{1}{|\mathbf{a}|}$. You cannot normalize the zero vector $\mathbf{0}$.

We denote a unit vector with a "hat," e.g. $\hat{\mathbf{a}}$ is a unit vector.

$$\hat{\mathbf{a}} = \frac{\mathbf{a}}{|\mathbf{a}|}$$

We read $\hat{\mathbf{a}}$ as "a hat."

Other Matrix Operations

Transpose

The transpose of a matrix $\mathbf{A}$ is denoted by $\mathbf{A}^T$ and is defined as a matrix where the rows of $\mathbf{A}$ become the columns of $\mathbf{A}^T$ and the columns of $\mathbf{A}$ become the rows of $\mathbf{A}^T$.

If $\mathbf{A}$ is $R\times C$, then $\mathbf{A}^T$ is $C\times R$.

We read $\mathbf{A}^{T}$ as "A transpose."

Inverse

The inverse of a square matrix $\mathbf{A}$ is denoted by $\mathbf{A}^{-1}$ and satisfies the equation

$$\mathbf{A} \mathbf{A}^{-1} = \mathbf{A}^{-1} \mathbf{A} = \mathbf{I}.$$

We read $\mathbf{A}^{-1}$ as "A inverse."

Handy Tricks

Linear Combination of Vectors -> Matrix-Vector Product

$$a_0\mathbf{b_0}+a_1\mathbf{b_1}+a_2\mathbf{b_2}+\cdots = \begin{bmatrix}\mathbf{b_0}&\mathbf{b_1}&\mathbf{b_2}&\cdots\end{bmatrix}\begin{bmatrix}a_0\\a_1\\a_2\\\vdots\end{bmatrix}$$