w6-understanding/w6-understanding-exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 6 - Understanding neural networks - Exercise\n",
    "\n",
    "This weeks videos explained how we can try to understand neural networks by observing, analysing and manipulating their activity. \n",
    "\n",
    "In this exercise, we'll apply the same techniques to an artificial neural network to see what we can learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a model\n",
    "\n",
    "First, we need a model to interrogate. \n",
    "\n",
    "In [week 5's exercise](https://github.com/neuro4ml/exercises/blob/main/w5-snn/w5-snn-exercise.ipynb) we learned how to use surrogate gradient descent to train spiking neural networks (SNNs), and trained a SNN on a sound localisation task. \n",
    "\n",
    "We're going to use the same task this week, and the notebook below has all of the code you'll need. \n",
    "\n",
    "Note that last weeks notebook had gaps to fill, which are filled here, so turn back if you want to avoid spoilers! \n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comob-project/snn-sound-localization/blob/main/research/Starting-Notebook.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aim \n",
    "\n",
    "Once you have a trained model, your aim is to learn something about how it works.\n",
    "\n",
    "Below we've outlined two approaches to this (observing and manipulating unit activity). \n",
    "\n",
    "You can try these in any order or combination you like, or take a totally different approach! For example, the provided notebook ends with some analysis of the networks weight matrix - which could provide you with some ideas.\n",
    "\n",
    "Remember that visualising your data and analysis is always helpful!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach 1 - Observing unit activity \n",
    "\n",
    "Once you have a trained model, you could try to observe its unit activity.\n",
    "\n",
    "To do that you'll need to edit the snn function to return the hidden layers activity, then pass inputs to the network and record this. A matrix of: hidden units x (time and trials) or a tensor of hidden units x time x trials would be good!\n",
    "\n",
    "Once you have this data try to learn something about the hidden units. Based on the material in W6-V1 you could try: \n",
    "\n",
    "* Calculating some summary statistics - like how specifically each unit responds to each class. \n",
    "* Decoding the input class from the hidden layer - this [paper](https://doi.org/10.1523/ENEURO.0506-19.2020) and associated [code](https://github.com/kordinglab/neural_decoding) could help with that. \n",
    "* Grouping the hidden units into functional ensembles - with a clustering algorithm or the ensemble method from the lecture: [paper](https://doi.org/10.1016/j.neuron.2018.05.015) + [code](https://github.com/neurostatslab/tensortools).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach 2 - Manipulating unit activity\n",
    "\n",
    "Once you have a trained model, you could try to manipulate its unit activity. \n",
    "\n",
    "To do that, you'll need to edit the snn function so that you can set some weights or units to zero during a forward pass, and then check how that alters the accuracy. As a sanity check, silencing all of the hidden units should reduce your networks performance to chance. \n",
    "\n",
    "Once you can silence the units or weights you could: \n",
    "* See if silencing each element reduces performance on one or many classes. \n",
    "* Try over-activating units instead to see if you can force the network to make incorrect decisions. \n",
    "* Try the multi-lesion approach from the lecture: [paper](https://doi.org/10.1371/journal.pcbi.1010250​) + [code](https://kuffmode.github.io/msa/). \n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}