Part 4 Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "674a9328-c364-437f-9af2-bfd6a44d73f5",
   "metadata": {},
   "source": [
    "# E01\n",
    "\n",
    "I did not get around to seeing what happens when you initialize all weights and biases to zero. Try this and train the neural net. You might think either that 1) the network trains just fine or 2) the network doesn't train at all, but actually it is 3) the network trains but only partially, and achieves a pretty bad final performance. \n",
    "\n",
    "Inspect the gradients and activations to figure out what is happening and why the network is only partially training, and what part is being trained exactly.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f84c8c0b-2c18-4e7b-be82-21d103d132ad",
   "metadata": {},
   "source": [
    "I used this structure:\n",
    "\n",
    "\n",
    "    layers = [\n",
    "        Linear(n_embd * block_size, n_hidden), Tanh(),\n",
    "        Linear(n_hidden,            n_hidden), Tanh(),\n",
    "        Linear(n_hidden,            n_hidden), Tanh(),\n",
    "        Linear(n_hidden,            n_hidden), Tanh(),\n",
    "        Linear(n_hidden,            n_hidden), Tanh(),\n",
    "        Linear(n_hidden, vocab_size), \n",
    "    ]\n",
    "\n",
    "where all weights and biases of all linear layers are set to 0 during initialization. This means, the output of any linear layer is just 0, including the last one. Tanh sets any 0 input to 0. Hence, the last layer always receives 0 as input. The form of the model is therefore\n",
    "\n",
    "    softmax(W*0 + b)\n",
    "\n",
    "and the biases of the last layer are the only parameters being trained"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae8f3ee-7040-4b04-8344-75e72fbceb4a",
   "metadata": {},
   "source": [
    "# E02\n",
    "\n",
    "BatchNorm, unlike other normalization layers like LayerNorm/GroupNorm etc. has the big advantage that after training, the batchnorm gamma/beta can be \"folded into\" the weights of the preceeding Linear layers, effectively erasing the need to forward it at test time. Set up a small 3-layer MLP with batchnorms, train the network, then \"fold\" the batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new W2, b2 and erasing the batch norm. Verify that this gives the same forward pass during inference. i.e. we see that the batchnorm is there just for stabilizing the training, and can be thrown out after training is done! pretty cool."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e2bfe0b-c09e-45bf-afc6-a78c798d98e7",
   "metadata": {},
   "source": [
    "We have the following pseudo-calculation:\n",
    "\n",
    "    hpreact = embcat @ W1 + b1\n",
    "    -> hpreact = bngain * (hpreact - bnmean) / bnstd + bnbias\n",
    "    -> hpreact = bngain * (embcat @ W1 + b1 - bnmean) / bnstd + bnbias\n",
    "    -> hpreact = (embcat @ (bngain * W1) / bnstd + ((bngain * b1) - (bngain * bnmean)) / bnstd )+ bnbias\n",
    "    -> hpreact = embcat @ W2 + b2 \n",
    "\n",
    "where\n",
    "\n",
    "    W2 = (bngain * W1) / bnstd \n",
    "    b2 = ((bngain * b1) - (bngain * bnmean)) / bnstd) + bnbias\n",
    "\n",
    "the dimensions are:\n",
    "* bngain, bnstd, bnbias: n_hidden\n",
    "* W1: (n_embd * block_size, n_hidden) or (n_hidden, n_hidden) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11803515-5f15-418e-8c71-bde253682958",
   "metadata": {},
   "source": [
    "# E03 (self-imposed)\n",
    "\n",
    "Read these papers:\n",
    "Useful links:\n",
    "- \"Kaiming init\" paper: https://arxiv.org/abs/1502.01852\n",
    "- \"PReLu\" where the part < 0 is for example -0.1*x, but the -0.1 is a free parameter\n",
    "- BatchNorm paper: https://arxiv.org/abs/1502.03167\n",
    "- Good paper illustrating some of the problems with batchnorm in practice: https://arxiv.org/abs/2105.07576\n",
    "- Write a comment on the iterative updating of running mean and standard deviation. Convergence properties of moving exponential averages are well known, given 'momentum' and distributional properties of the quantity that is supposed to be updated.\n",
    "- Look at adam optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0490c5b8-3907-49b7-a3a3-ba3351fef0a3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c94fe54b-cc01-4e68-8a22-a816ffe4c9e4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c49cc22-e72b-45ef-9024-b0dbd220b8e8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}