assignments/assignment_01/syde556_assignment_01_template.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SYDE 556/750 --- Assignment 1\n",
    "**Student ID: 00000000**\n",
    "\n",
    "*Note:* Please include your numerical student ID only, do *not* include your name.\n",
    "\n",
    "*Note:* Refer to the [PDF](https://github.com/astoeckel/syde556-w20/raw/master/assignments/assignment_01/syde556_assignment_01.pdf) for the full instructions (including some hints), this notebook contains abbreviated instructions only. Cells you need to fill out are marked with a \"writing hand\" symbol. Of course, you can add new cells in between the instructions, but please leave the instructions intact to facilitate marking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import numpy and matplotlib -- you shouldn't need any other libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy.optimize # For question 2.1b)\n",
    "\n",
    "# Fix the numpy random seed for reproducible results\n",
    "np.random.seed(18945)\n",
    "\n",
    "# Some formating options\n",
    "%config InlineBackend.figure_formats = ['svg']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Representation of Scalars\n",
    "\n",
    "## 1.1 Basic encoding and decoding\n",
    "\n",
    "**a) Computing gain and bias.** In general, for a neuron model $a = G[J]$ (and assuming that the inverse $J = G^{-1}[a]$ exists), solve the following system of equations to compute the gain $\\alpha$, and the bias $J^\\mathrm{bias}$ given a maximum rate $a^\\mathrm{max}$ and an $x$-intercept $\\xi$.\n",
    "\n",
    "$$a^\\mathrm{max} = G[\\alpha + J^\\mathrm{bias}] \\,, \\quad\\quad 0 = G[\\alpha \\xi + J^\\mathrm{bias}] \\,.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, simplify these equations for the specific case $G[J] = \\max(J, 0)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b) Neuron tuning curves.** Plot the neuron tuning curves $a_i(x)$ for 16 randomly generated neurons following the intercept and maximum rate distributions described above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c) Computing identity decoders.** Compute the optimal identity decoder $\\vec d$ for those 16 neurons (as shown in class). Report the value of the individual decoder coefficients. Compute $d$ using the matrix notation mentioned in the course notes. Do not apply any regularization. $A$ is the matrix of activities (the same data used to generate the plot in 1.1b)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d) Evaluating decoding errors.** Compute and plot $\\hat{x}=\\sum_i d_i a_i(x)$. Overlay on the plot the line $y=x$. Make a separate plot of $x-\\hat{x}$ to see what the error looks like. Report the Root Mean Squared Error (RMSE) value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e) Decoding under noise.** Now try decoding under noise. Add random normally distributed noise to $a$ and decode again. The noise is a random variable with mean $\\mu=0$ and standard deviation of $\\sigma=0.2 \\max(A)$ (where $\\max(A)$ is the maximum firing rate of all the neurons). Resample this variable for every different $x$ value for every different neuron. Create all the same plots as in part d). Report the RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f) Accounting for decoder noise.** Recompute the decoder $\\vec d$ taking noise into account (i.e., apply the appropriate regularization, as shown in class). Show how these decoders behave when decoding both with and without noise added to $a$ by making the same plots as in d) and e). Report the RMSE for all cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g) Interpretation.** Show a 2x2 table of the four RMSE values reported in parts d), e), and f). This should show the effects of adding noise and whether the decoders $d$ are computed taking noise into account. Write a few sentences commenting on what the table shows, i.e., what the effect of adding noise to the activities is with respect to the measured error and why accounting for noise when computing the decoders increases/decreases/does not change the measured RMSE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Exploring sources of error\n",
    "\n",
    "**a) Exploring error due to distortion and noise.** Plot the error due to distortion $E_\\mathrm{dist}$ and the error due to noise $E_\\mathrm{noise}$ as a function of $n$, the number of neurons. Generate two different loglog plots (one for each type of error) with $n$ values of at least $[4, 8, 16, 32, 64, 128, 256, 512]$. For each $n$ value, do at least $5$ runs and average the results. For each run, different $\\alpha$, $J^\\mathrm{bias}$, and $e$ values should be generated for each neuron. Compute $d$ taking noise into account, with $\\sigma = 0.1 \\max(A)$. Show visually that the errors are proportional to $1/n$ or $1/n^2$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b) Adapting the noise level.** Repeat part a) with $\\sigma = 0.01 \\max(A)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c) Interpretation.** What does the difference between the graphs in a) and b) tell us about the sources of error in neural populations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Leaky Integrate-and-Fire neurons\n",
    "\n",
    "**a) Computing gain and bias.** As in the second part of 1.1a), given a maximum firing rate $a^\\mathrm{max}$ and a bias $J^\\mathrm{bias}$, write down the equations for computing $\\alpha$ and the $J^\\mathrm{bias}$ for this specific neuron model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b) Neuron tuning curves.** Generate the same plot as in 1.1b). Use $\\tau_\\mathrm{ref}=2 \\mathrm{ms}$ and $\\tau_{RC}=20 \\mathrm{ms}$. Use the same distribution of $x$-intercepts and maximum firing rates as in 1.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c) Impact of noise.** Generate the same four plots as in 1.1f) (adding/not adding noise to $A$, accounting/not accounting for noise when computing $\\vec d$), and report the RMSE both with and without noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Reperesentation of Vectors\n",
    "\n",
    "## 2.1 Vector tuning curves\n",
    "\n",
    "**a) Plotting 2D tuning curves.** Plot the tuning curve of an LIF neuron whose 2D preferred direction vector is at an angle of $\\theta=-\\pi/4$, has an $x$-intercept at the origin $(0,0)$, and has a maximum firing rate of $100 \\mathrm{Hz}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b) Plotting the 2D tuning curve along the unit circle.** Plot the tuning curve for the same neuron as in a), but only considering the points around the unit circle, i.e., sample the activation for different angles $\\theta$. Fit a curve of the form $c_1 \\cos(c_2\\theta+c_3)+c_4$ to the tuning curve and plot it as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c) Discussion.** What makes a cosine a good choice for the curve fit in 2.1b? Why does it differ from the ideal curve?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Vector representation\n",
    "\n",
    "**a) Choosing encoding vectors.** Generate a set of $100$ random unit vectors uniformly distributed around the unit circle. These will be the encoders $\\vec e$ for $100$ neurons. Plot these vectors with a quiver or line plot (i.e., not just points, but lines/arrows to the points)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b) Computing the identity decoder.** Use LIF neurons with the same properties as in question 1.3. When computing the decoders, take into account noise with $\\sigma = 0.2\\max(A)$. Plot the decoders in the same way you plotted the encoders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c) Discussion.** How do these decoding vectors compare to the encoding vectors?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d) Testing the decoder.** Generate 20 random $\\vec x$ values throughout the unit circle (i.e.,~with different directions and radiuses). For each $\\vec x$ value, determine the neural activity $a_i$ for each of the 100 neurons. Now decode these values (i.e. compute $\\hat{x} = D \\vec a$) using the decoders from part b). Plot the original and decoded values on the same graph in different colours, and compute the RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e) Using encoders as decoders.** Repeat part d) but use the *encoders* as decoders. This is what Georgopoulos used in his original approach to decoding information from populations of neurons. Plot the decoded values and compute the RMSE. In addition, recompute the RMSE in both cases, but ignore the magnitude of the decoded vectors by normalizing before computing the RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ✍ <YOUR SOLUTION HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f) Discussion.** When computing the RMSE on the normalized vectors, using the encoders as decoders should result in a larger, yet still surprisingly small error. Thinking about random unit vectors in high dimensional spaces, why is this the case? What are the relative merits of these two approaches to decoding?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✍ \\<YOUR SOLUTION HERE\\>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}