discussions/03-pca/notebook/discussion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discussion 03\n",
    "\n",
    "## Principal Components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to Discussion 03. In this discussion, we'll gain a deeper understanding of principal components and how they are used for dimensionality reduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn.cluster\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (7,7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by generating some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = np.array([\n",
    "    [3, -2],\n",
    "    [-2, 3]\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "X = np.random.multivariate_normal([0,0], C, 300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(*X.T)\n",
    "plt.gca().set_aspect(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 01**. What is the *top* eigenvector of the covariance matrix, $C$? What is the *second* eigenvector?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 02**. Plot both eigenvectors on top of the data. Using color, distinguish the top eigenvector from the second eigenvector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 03**. For each data point $\\vec x^{(i)}$, project it onto the first eigenvector, $\\vec u^{(1)}$ to get a new vector: $(\\vec x{(i)} \\cdot \\vec u{(i)}) \\, \\vec u{(i)}$. Plot your new points on top of the previous plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 04**. Now let's create a new dataset, $Z$, in the following way. Given a vector $\\vec x = (x_1, x_2)^T$, we produce a new representation $\\vec z = (z_1, z_2)^T$, where $z_1 = \\vec x \\cdot \\vec u^{(1)}$ and $z_2 = \\vec x \\cdot \\vec u^{(2)}$. Plot $Z$ as a scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell will download the MNIST digit dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should notice that your new scatter plot is a rotated version of the original scatter plot of $X$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 05**. Compute the covariance matrix for $Z$. What do you notice about the off-diagonal entries? What does this mean, informally?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try this on a more interesting data set. The cell below will download the MNIST digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "if [[ ! -e \"mnist.npz\" ]]; then\n",
    "    wget 'https://f000.backblazeb2.com/file/jeldridge-data/mnist.npz'\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = np.load('mnist.npz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use only the \"training\" data -- there is no training and testing in PCA, it's all just \"data\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = mnist['train'].T\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 60,000 images in 784 dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 06**. Compute the covariance matrix. What should be its size? Check to make sure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 07**. Compute the eigenvectors of the covariance matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 08**. Each of the eigenvectors is a unit vector -- a direction in 784 dimensions. We can think of each eigenvector as a vector of \"mixing coefficients\" which creates a particular mixture of the features in the original image (i.e., the pixel intensities). That is, if $\\vec u = (u_1, u_2, \\ldots, u_{784})^T$, we can think of $u_1$ as being the \"mixture coefficient\" of pixel 1, $u_2$ is the coefficient of pixel 2, and so forth.\n",
    "\n",
    "This interpretation allows us to visualize the 784-dimensional eigenvectors as images by reshaping them into 28 x 28 arrays.\n",
    "\n",
    "Visualize the top 5 eigenvectors of the covariance matrix as images. Do the same, but for the *bottom* five eigenvectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}