examples/01_copying_task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copying Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspired on the task described in the following paper: [https://arxiv.org/pdf/1511.06464.pdf](https://arxiv.org/pdf/1511.06464.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The copying task is one of the simplest benchmark tasks for recurrent neural networks.\n",
    "The general idea of the task is to reproduce a random sequence of symbols with length\n",
    "`len_sequence` chosen from an alphabet of size `num_symbols` after a certain waiting\n",
    "period `len_wait`.\n",
    "\n",
    "Assuming the waiting symbol is `0`, the symbols chosen for the sequence are chosen from\n",
    "the alphabet `{1,2,3}` and the stop waiting symbol is `4`; an example input and target for a\n",
    "waiting time of 20 symbols and a sequence length of 5 can be given by:\n",
    "```\n",
    "    213310000000000000000000400000\n",
    "    000000000000000000000000021331\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As discussed in the [paper](https://arxiv.org/pdf/1511.06464.pdf), it is always useful\n",
    "to compare the loss of a certain implementation to the baseline loss of guessing.\n",
    "Assuming one uses the categorical cross-entropy loss, one can describe a baseline by\n",
    "predicting the waiting symbol for the first `len_wait + len_sequence` timesteps, followed by a random sampling for the remaining `len_sequence` positions out of\n",
    "the alphabet of symbols `{a1,...,an}` with `num_symbols` elements. This baseline cross entropy loss boils down to\n",
    "```\n",
    "    len_sequence*log(n_symbols)/(len_wait + 2*len_sequence)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sys; sys.path.append('..')\n",
    "from torch_eunn import EURNN\n",
    "\n",
    "torch.manual_seed(24)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline = 0.17328679513998632\n"
     ]
    }
   ],
   "source": [
    "# Training parameters\n",
    "num_steps = 500\n",
    "batch_size = 128\n",
    "test_size = 100\n",
    "valid_size = 100\n",
    "\n",
    "# Data Parameters\n",
    "len_wait = 100#0 # very slow if len_wait=1000\n",
    "num_symbols = 8\n",
    "len_sequence = 10\n",
    "\n",
    "# RNN Parameters\n",
    "capacity = 2\n",
    "num_layers_rnn = 1\n",
    "num_hidden_rnn = 128\n",
    "\n",
    "# Cuda\n",
    "cuda = True\n",
    "device = torch.device('cuda' if cuda else 'cpu')\n",
    "\n",
    "# Baseline Error\n",
    "baseline = len_sequence*np.log(num_symbols)/(len_wait+2*len_sequence)\n",
    "print(f'baseline = {baseline}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data(len_wait, n_data, len_sequence, num_symbols):\n",
    "    seq = np.random.randint(1, high=(num_symbols+1), size=(n_data, len_sequence))\n",
    "    zeros1 = np.zeros((n_data, len_wait-1))\n",
    "    zeros2 = np.zeros((n_data, len_wait))\n",
    "    marker = (num_symbols+1) * np.ones((n_data, 1))\n",
    "    zeros3 = np.zeros((n_data, len_sequence))\n",
    "    x = torch.tensor(np.concatenate((seq, zeros1, marker, zeros3), axis=1), dtype=torch.int64, device=device)\n",
    "    y = torch.tensor(np.concatenate((zeros3, zeros2, seq), axis=1), dtype=torch.int64, device=device)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[7, 4, 5, 7, 3, 8, 5, 5, 7, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],\n",
      "       device='cuda:0')\n",
      "tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 4, 5, 7, 3, 8, 5, 5, 7, 2]],\n",
      "       device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "x,y = data(len_wait, 1, len_sequence, num_symbols)\n",
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Model, self).__init__()\n",
    "        self.embedding = torch.nn.Embedding(len_wait+2*len_sequence, num_symbols+2)\n",
    "        self.rnn = EURNN(num_symbols+2, num_hidden_rnn, capacity, batch_first=True)\n",
    "        self.fc = torch.nn.Linear(num_hidden_rnn, num_symbols+1)\n",
    "        \n",
    "        # optimizers and criterion\n",
    "        self.lossfunc = torch.nn.CrossEntropyLoss()\n",
    "        self.optimizer = torch.optim.Adam(self.parameters(), lr=0.03)\n",
    "        \n",
    "        # move to device\n",
    "        self.to(device)\n",
    "        \n",
    "    def forward(self, data):\n",
    "        data = self.embedding(data)\n",
    "        rnn_out, _ = self.rnn(data)\n",
    "        out = self.fc(rnn_out)\n",
    "        return out\n",
    "    \n",
    "    def loss(self, data, labels):\n",
    "        return self.lossfunc(self(data).view(-1, num_symbols+1), labels.view(-1))\n",
    "    \n",
    "    def accuracy(self, data, labels):\n",
    "        return torch.mean((torch.argmax(self(data), -1).view(-1) == labels.view(-1)).float())\n",
    "    \n",
    "    def prediction(self, data):\n",
    "        return torch.argmax(self(data), -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step     0\t Valid. Loss. = 20.2436\n",
      "Step   100\t Valid. Loss. = 0.0035\n",
      "Step   200\t Valid. Loss. = 0.0008\n",
      "Step   300\t Valid. Loss. = 0.0009\n",
      "Step   400\t Valid. Loss. = 0.0002\n",
      "Step   499\t Valid. Loss. = 0.0001\n",
      "CPU times: user 3min 14s, sys: 91.3 ms, total: 3min 14s\n",
      "Wall time: 3min 14s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for step in range(num_steps):\n",
    "    # reset gradients\n",
    "    model.optimizer.zero_grad()\n",
    "    \n",
    "    # calculate validation accuracy and loss\n",
    "    if step %100 == 0 or step == num_steps -1:\n",
    "        valid_data, valid_labels = data(len_wait, valid_size, len_sequence, num_symbols)\n",
    "        loss = model.loss(valid_data, valid_labels).item()\n",
    "        print(f'Step {step:5.0f}\\t Valid. Loss. = {loss:5.4f}')\n",
    "        \n",
    "    # train\n",
    "    batch_data, batch_labels = data(len_wait, batch_size, len_sequence, num_symbols)\n",
    "    loss = model.loss(batch_data, batch_labels)\n",
    "    loss.backward()\n",
    "    model.optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test result: Loss= 0.000193, Accuracy= 1.00000\n",
      "baseline = 0.173287\n"
     ]
    }
   ],
   "source": [
    "test_data, test_labels = data(len_wait, test_size, len_sequence, num_symbols)\n",
    "test_loss = model.loss(test_data, test_labels).item()\n",
    "test_acc = model.accuracy(test_data, test_labels).item()\n",
    "print(\"Test result: Loss= \" + \"{:.6f}\".format(test_loss) + \", Accuracy= \" + \"{:.5f}\".format(test_acc))\n",
    "print('baseline = %f'%baseline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The baseline is clearly broken. Note that this would not be the case if we switch out the EUNN for an LSTM."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}