Tutorials/CNTK_202_Language_Understanding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNTK 202: Language Understanding with Recurrent Networks\n",
    "\n",
    "This tutorial shows how to implement a recurrent network to process text,\n",
    "for the [Air Travel Information Services](https://catalog.ldc.upenn.edu/LDC95S26) \n",
    "(ATIS) task of slot tagging (tag individual words to their respective classes, \n",
    "where the classes are provided as labels in the training data set).\n",
    "We will start with a straight-forward (linear) embedding of the words followed by a recurrent LSTM.\n",
    "This will then be extended to include neighboring words and run bidirectionally.\n",
    "Lastly, we will turn this system into an intent classifier.  \n",
    "\n",
    "The techniques you will practice are:\n",
    "* model description by composing layer blocks, a convenient way to compose \n",
    "  networks/models without requiring the need to write formulas,\n",
    "* creating your own layer block\n",
    "* variables with different sequence lengths in the same network\n",
    "* training the network\n",
    "\n",
    "We assume that you are familiar with basics of deep learning, and these specific concepts:\n",
    "* recurrent networks ([Wikipedia page](https://en.wikipedia.org/wiki/Recurrent_neural_network))\n",
    "* text embedding ([Wikipedia page](https://en.wikipedia.org/wiki/Word_embedding))\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "We assume that you have already [installed CNTK](https://github.com/Microsoft/CNTK/wiki/Setup-CNTK-on-your-machine).\n",
    "This tutorial requires CNTK V2. We strongly recommend to run this tutorial on a machine with\n",
    "a capable CUDA-compatible GPU. Deep learning without GPUs is not fun."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Downloading the data\n",
    "\n",
    "In this tutorial we are going to use a (lightly preprocessed) version of the ATIS dataset. You can download the data automatically by running the cells below or by executing the manual instructions.\n",
    "\n",
    "#### Fallback manual instructions\n",
    "Download the ATIS [training](https://github.com/Microsoft/CNTK/blob/v2.0/Tutorials/SLUHandsOn/atis.train.ctf) \n",
    "and [test](https://github.com/Microsoft/CNTK/blob/v2.0/Tutorials/SLUHandsOn/atis.test.ctf) \n",
    "files and put them at the same folder as this notebook. If you want to see how the model is \n",
    "predicting on new sentences you will also need the vocabulary files for \n",
    "[queries](https://github.com/Microsoft/CNTK/blob/v2.0/Examples/LanguageUnderstanding/ATIS/BrainScript/query.wl) and\n",
    "[slots](https://github.com/Microsoft/CNTK/blob/v2.0/Examples/LanguageUnderstanding/ATIS/BrainScript/slots.wl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reusing locally cached: atis.test.ctf\n",
      "Reusing locally cached: atis.train.ctf\n",
      "Reusing locally cached: query.wl\n",
      "Reusing locally cached: slots.wl\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)\n",
    "import requests\n",
    "import os\n",
    "\n",
    "def download(url, filename):\n",
    "    \"\"\" utility function to download a file \"\"\"\n",
    "    response = requests.get(url, stream=True)\n",
    "    with open(filename, \"wb\") as handle:\n",
    "        for data in response.iter_content():\n",
    "            handle.write(data)\n",
    "\n",
    "locations = ['Tutorials/SLUHandsOn', 'Examples/LanguageUnderstanding/ATIS/BrainScript']\n",
    "\n",
    "data = {\n",
    "  'train': { 'file': 'atis.train.ctf', 'location': 0 },\n",
    "  'test': { 'file': 'atis.test.ctf', 'location': 0 },\n",
    "  'query': { 'file': 'query.wl', 'location': 1 },\n",
    "  'slots': { 'file': 'slots.wl', 'location': 1 }\n",
    "}\n",
    "\n",
    "for item in data.values():\n",
    "    location = locations[item['location']]\n",
    "    path = os.path.join('..', location, item['file'])\n",
    "    if os.path.exists(path):\n",
    "        print(\"Reusing locally cached:\", item['file'])\n",
    "        # Update path\n",
    "        item['file'] = path\n",
    "    elif os.path.exists(item['file']):\n",
    "        print(\"Reusing locally cached:\", item['file'])\n",
    "    else:\n",
    "        print(\"Starting download:\", item['file'])\n",
    "        url = \"https://github.com/Microsoft/CNTK/blob/v2.0/%s/%s?raw=true\"%(location, item['file'])\n",
    "        download(url, item['file'])\n",
    "        print(\"Download completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Importing CNTK and other useful libraries\n",
    "\n",
    "CNTK's Python module contains several submodules like `io`, `learner`, and `layers`. We also use NumPy in some cases since the results returned by CNTK work like NumPy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "\n",
    "import cntk as C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select the right target device when this notebook is being tested:\n",
    "if 'TEST_DEVICE' in os.environ:\n",
    "    if os.environ['TEST_DEVICE'] == 'cpu':\n",
    "        C.device.try_set_default_device(C.device.cpu())\n",
    "    else:\n",
    "        C.device.try_set_default_device(C.device.gpu(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task and Model Structure\n",
    "\n",
    "The task we want to approach in this tutorial is slot tagging.\n",
    "We use the [ATIS corpus](https://catalog.ldc.upenn.edu/LDC95S26).\n",
    "ATIS contains human-computer queries from the domain of Air Travel Information Services,\n",
    "and our task will be to annotate (tag) each word of a query whether it belongs to a\n",
    "specific item of information (slot), and which one.\n",
    "\n",
    "The data in your working folder has already been converted into the \"CNTK Text Format.\"\n",
    "Let us look at an example from the test-set file `atis.test.ctf`:\n",
    "\n",
    "    19  |S0 178:1 |# BOS      |S1 14:1 |# flight  |S2 128:1 |# O\n",
    "    19  |S0 770:1 |# show                         |S2 128:1 |# O\n",
    "    19  |S0 429:1 |# flights                      |S2 128:1 |# O\n",
    "    19  |S0 444:1 |# from                         |S2 128:1 |# O\n",
    "    19  |S0 272:1 |# burbank                      |S2 48:1  |# B-fromloc.city_name\n",
    "    19  |S0 851:1 |# to                           |S2 128:1 |# O\n",
    "    19  |S0 789:1 |# st.                          |S2 78:1  |# B-toloc.city_name\n",
    "    19  |S0 564:1 |# louis                        |S2 125:1 |# I-toloc.city_name\n",
    "    19  |S0 654:1 |# on                           |S2 128:1 |# O\n",
    "    19  |S0 601:1 |# monday                       |S2 26:1  |# B-depart_date.day_name\n",
    "    19  |S0 179:1 |# EOS                          |S2 128:1 |# O\n",
    "\n",
    "This file has 7 columns:\n",
    "\n",
    "* a sequence id (19). There are 11 entries with this sequence id. This means that sequence 19 consists\n",
    "of 11 tokens;\n",
    "* column `S0`, which contains numeric word indices; the input data is encoded in one-hot vectors.  There are 943 words in the vocabulary, so each word is a 943 element vector of all 0 with a 1 at a vector index chosen to represent that word.  For example the word \"from\" is represented with a 1 at index 444 and zero everywhere else in the vector. The word \"monday\" is represented with a 1 at index 601 and zero everywhere else in the vector.\n",
    "* a comment column denoted by `#`, to allow a human reader to know what the numeric word index stands for;\n",
    "Comment columns are ignored by the system. `BOS` and `EOS` are special words\n",
    "to denote beginning and end of sentence, respectively;\n",
    "* column `S1` is an intent label, which we will only use in the last part of the tutorial;\n",
    "* another comment column that shows the human-readable label of the numeric intent index;\n",
    "* column `S2` is the slot label, represented as a numeric index; and\n",
    "* another comment column that shows the human-readable label of the numeric label index.\n",
    "\n",
    "The task of the neural network is to look at the query (column `S0`) and predict the\n",
    "slot label (column `S2`).\n",
    "As you can see, each word in the input gets assigned either an empty label `O`\n",
    "or a slot label that begins with `B-` for the first word, and with `I-` for any\n",
    "additional consecutive word that belongs to the same slot.\n",
    "\n",
    "The model we will use is a recurrent model consisting of an embedding layer,\n",
    "a recurrent LSTM cell, and a dense layer to compute the posterior probabilities:\n",
    "\n",
    "\n",
    "    slot label   \"O\"        \"O\"        \"O\"        \"O\"  \"B-fromloc.city_name\"\n",
    "                  ^          ^          ^          ^          ^\n",
    "                  |          |          |          |          |\n",
    "              +-------+  +-------+  +-------+  +-------+  +-------+\n",
    "              | Dense |  | Dense |  | Dense |  | Dense |  | Dense |  ...\n",
    "              +-------+  +-------+  +-------+  +-------+  +-------+\n",
    "                  ^          ^          ^          ^          ^\n",
    "                  |          |          |          |          |\n",
    "              +------+   +------+   +------+   +------+   +------+   \n",
    "         0 -->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->...\n",
    "              +------+   +------+   +------+   +------+   +------+   \n",
    "                  ^          ^          ^          ^          ^\n",
    "                  |          |          |          |          |\n",
    "              +-------+  +-------+  +-------+  +-------+  +-------+\n",
    "              | Embed |  | Embed |  | Embed |  | Embed |  | Embed |  ...\n",
    "              +-------+  +-------+  +-------+  +-------+  +-------+\n",
    "                  ^          ^          ^          ^          ^\n",
    "                  |          |          |          |          |\n",
    "    w      ------>+--------->+--------->+--------->+--------->+------... \n",
    "                 BOS      \"show\"    \"flights\"    \"from\"   \"burbank\"\n",
    "\n",
    "Or, as a CNTK network description. Please have a quick look and match it with the description above:\n",
    "(descriptions of these functions can be found at: [the layers reference](http://cntk.ai/pythondocs/layerref.html))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# number of words in vocab, slot labels, and intent labels\n",
    "vocab_size = 943 ; num_labels = 129 ; num_intents = 26    \n",
    "\n",
    "# model dimensions\n",
    "input_dim  = vocab_size\n",
    "label_dim  = num_labels\n",
    "emb_dim    = 150\n",
    "hidden_dim = 300\n",
    "\n",
    "# Create the containers for input feature (x) and the label (y)\n",
    "x = C.sequence.input_variable(vocab_size)\n",
    "y = C.sequence.input_variable(num_labels)\n",
    "\n",
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):\n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim, name='embed'),\n",
    "            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),\n",
    "            C.layers.Dense(num_labels, name='classify')\n",
    "        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to create a model and inspect it. \n",
    "\n",
    "The model attributes are fully accessible from Python. The first layer named `embed` is an Embedding layer. Here we use the CNTK default, which is linear embedding. It is a simple matrix with dimension (input word encoding x output projected dimension). You can access its parameter `E` (where the embeddings are stored) like any other attribute of a Python object. Its shape contains a `-1` which indicates that this parameter (with input dimension) is not fully specified yet, while the output dimension is set to `emb_dim` ( = 150 in this tutorial). \n",
    "\n",
    "Additionally we also inspect the value of the bias vector in the `Dense` layer named `classify`. The `Dense` layer is a fundamental compositional unit of a Multi-Layer Perceptron (as introduced in CNTK 103C tutorial). The `Dense` layer has both `weight` and `bias` parameters, one each per `Dense` layer. Bias terms are by default initialized to 0 (but there is a way to change that if you need). As you create the model, one should name the layer component and then access the parameters as shown here. \n",
    "\n",
    "**Suggested task**: What should be the expected dimension of the `weight` matrix from the layer named `classify`? Try printing the weight matrix of the `classify` layer? Does it match with your expected size?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-1, 150)\n",
      "[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "  0.  0.  0.]\n"
     ]
    }
   ],
   "source": [
    "# peek\n",
    "z = create_model()\n",
    "print(z.embed.E.shape)\n",
    "print(z.classify.b.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our case we have input as one-hot encoded vector of length 943 and the output dimension `emb_dim` is set to 150. In the code below we pass the input variable `x` to our model `z`. This binds the model with input data of known shape. In this case, the input shape will be the size of the input vocabulary. With this modification, the parameter returned by the embed layer is completely specified (943, 150). **Note**: You can initialize the Embedding matrix with pre-computed vectors using [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) or [GloVe](https://en.wikipedia.org/wiki/GloVe_(machine_learning))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(943, 150)\n"
     ]
    }
   ],
   "source": [
    "# Pass an input and check the dimension\n",
    "z = create_model()\n",
    "print(z(x).embed.E.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNTK Configuration\n",
    "\n",
    "To train and test a model in CNTK, we need to create a model and specify how to read data and perform training and testing. \n",
    "\n",
    "In order to train we need to specify:\n",
    "\n",
    "* how to read the data \n",
    "* the model function, its inputs, and outputs\n",
    "* hyper-parameters for the learner such as the learning rate\n",
    "\n",
    "[comment]: <> (For testing ...)\n",
    "\n",
    "### A Brief Look at Data and Data Reading\n",
    "\n",
    "We already looked at the data.\n",
    "But how do you generate this format?\n",
    "For reading text, this tutorial uses the `CNTKTextFormatReader`. It expects the input data to be\n",
    "in a specific format, as described [here](https://github.com/Microsoft/CNTK/wiki/CNTKTextFormat-Reader).\n",
    "\n",
    "For this tutorial, we created the corpora by two steps:\n",
    "* convert the raw data into a plain text file that contains of TAB-separated columns of space-separated text. For example:\n",
    "\n",
    "  ```\n",
    "  BOS show flights from burbank to st. louis on monday EOS (TAB) flight (TAB) O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-depart_date.day_name O\n",
    "  ```\n",
    "\n",
    "  This is meant to be compatible with the output of the `paste` command.\n",
    "* convert it to CNTK Text Format (CTF) with the following command:\n",
    "\n",
    "  ```\n",
    "  python [CNTK root]/Scripts/txt2ctf.py --map query.wl intent.wl slots.wl --annotated True --input atis.test.txt --output atis.test.ctf\n",
    "  ```\n",
    "  where the three `.wl` files give the vocabulary as plain text files, one word per line.\n",
    "\n",
    "In these CTF files, our columns are labeled `S0`, `S1`, and `S2`.\n",
    "These are connected to the actual network inputs by the corresponding lines in the reader definition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_reader(path, is_training):\n",
    "    return C.io.MinibatchSource(C.io.CTFDeserializer(path, C.io.StreamDefs(\n",
    "         query         = C.io.StreamDef(field='S0', shape=vocab_size,  is_sparse=True),\n",
    "         intent_unused = C.io.StreamDef(field='S1', shape=num_intents, is_sparse=True),  \n",
    "         slot_labels   = C.io.StreamDef(field='S2', shape=num_labels,  is_sparse=True)\n",
    "     )), randomize=is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['intent_unused', 'query', 'slot_labels'])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# peek\n",
    "reader = create_reader(data['train']['file'], is_training=True)\n",
    "reader.streams.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trainer\n",
    "\n",
    "We also must define the training criterion (loss function), and also an error metric to track. In most tutorials, we know the input dimensions and the corresponding labels. We directly create the loss and the error functions. In this tutorial we will do the same. However, we take a brief detour and learn about placeholders. This concept would be useful for Task 3. \n",
    "\n",
    "**Learning note**: Introduction to `placeholder`: Remember that the code we have been writing is not actually executing any heavy computation it is just specifying the function we want to compute on data during training/testing. And in the same way that it is convenient to have names for arguments when you write a regular function in a programming language, it is convenient to have placeholders that refer to arguments (or local computations that need to be reused). Eventually, some other code will replace these placeholders with other known quantities in the same way that in a programming language the function will be called with concrete values bound to its arguments. \n",
    "\n",
    "Specifically, the input variables you have created above `x = C.sequence.input_variable(vocab_size)` holds data pre-defined by `vocab_size`. In the case where such instantiations are challenging or not possible, using `placeholder` is a logical choice. Having the `placeholder` only allows you to defer the specification of the argument at a later time when you may have the data.\n",
    "\n",
    "Here is an example below that illustrates the use of `placeholder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Composite(Combine): Input('Input2293', [#, *], [129]), Placeholder('labels', [???], [???]) -> Output('Block2263_Output_0', [#, *], [1]), Output('Block2283_Output_0', [#, *], [])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def create_criterion_function(model):\n",
    "    labels = C.placeholder(name='labels')\n",
    "    ce   = C.cross_entropy_with_softmax(model, labels)\n",
    "    errs = C.classification_error      (model, labels)\n",
    "    return C.combine ([ce, errs]) # (features, labels) -> (loss, metric)\n",
    "\n",
    "criterion = create_criterion_function(create_model())\n",
    "criterion.replace_placeholders({criterion.placeholders[0]: C.sequence.input_variable(num_labels)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the cell above works well when one has input parameters defined at network creation, it compromises readability. Hence we prefer creating functions as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_criterion_function_preferred(model, labels):\n",
    "    ce   = C.cross_entropy_with_softmax(model, labels)\n",
    "    errs = C.classification_error      (model, labels)\n",
    "    return ce, errs # (model, labels) -> (loss, error metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(reader, model_func, max_epochs=10):\n",
    "    \n",
    "    # Instantiate the model function; x is the input (feature) variable \n",
    "    model = model_func(x)\n",
    "    \n",
    "    # Instantiate the loss and error function\n",
    "    loss, label_error = create_criterion_function_preferred(model, y)\n",
    "\n",
    "    # training config\n",
    "    epoch_size = 18000        # 18000 samples is half the dataset size \n",
    "    minibatch_size = 70\n",
    "    \n",
    "    # LR schedule over epochs \n",
    "    # In CNTK, an epoch is how often we get out of the minibatch loop to\n",
    "    # do other stuff (e.g. checkpointing, adjust learning rate, etc.)\n",
    "    # (we don't run this many epochs, but if we did, these are good values)\n",
    "    lr_per_sample = [0.003]*4+[0.0015]*24+[0.0003]\n",
    "    lr_per_minibatch = [lr * minibatch_size for lr in lr_per_sample]\n",
    "    lr_schedule = C.learning_rate_schedule(lr_per_minibatch, C.UnitType.minibatch, epoch_size)\n",
    "    \n",
    "    # Momentum schedule\n",
    "    momentum_as_time_constant = C.momentum_as_time_constant_schedule(700)\n",
    "    \n",
    "    # We use a the Adam optimizer which is known to work well on this dataset\n",
    "    # Feel free to try other optimizers from \n",
    "    # https://www.cntk.ai/pythondocs/cntk.learner.html#module-cntk.learner\n",
    "    learner = C.adam(parameters=model.parameters,\n",
    "                     lr=lr_schedule,\n",
    "                     momentum=momentum_as_time_constant,\n",
    "                     gradient_clipping_threshold_per_sample=15, \n",
    "                     gradient_clipping_with_truncation=True)\n",
    "\n",
    "    # Setup the progress updater\n",
    "    progress_printer = C.logging.ProgressPrinter(tag='Training', num_epochs=max_epochs)\n",
    "    \n",
    "    # Uncomment below for more detailed logging\n",
    "    #progress_printer = ProgressPrinter(freq=100, first=10, tag='Training', num_epochs=max_epochs) \n",
    "\n",
    "    # Instantiate the trainer\n",
    "    trainer = C.Trainer(model, (loss, label_error), learner, progress_printer)\n",
    "\n",
    "    # process minibatches and perform model training\n",
    "    C.logging.log_number_of_parameters(model)\n",
    "\n",
    "    t = 0\n",
    "    for epoch in range(max_epochs):         # loop over epochs\n",
    "        epoch_end = (epoch+1) * epoch_size\n",
    "        while t < epoch_end:                # loop over minibatches on the epoch\n",
    "            data = reader.next_minibatch(minibatch_size, input_map={  # fetch minibatch\n",
    "                x: reader.streams.query,\n",
    "                y: reader.streams.slot_labels\n",
    "            })\n",
    "            trainer.train_minibatch(data)               # update model with it\n",
    "            t += data[y].num_samples                    # samples so far\n",
    "        trainer.summarize_training_progress()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running it\n",
    "\n",
    "You can find the complete recipe below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 721479 parameters in 6 parameter tensors.\n",
      "Learning rate per minibatch: 0.21\n",
      "Finished Epoch[1 of 10]: [Training] loss = 0.787482 * 18010, metric = 15.61% * 18010 5.471s (3291.9 samples/s);\n",
      "Finished Epoch[2 of 10]: [Training] loss = 0.223525 * 18051, metric = 5.25% * 18051 4.958s (3640.8 samples/s);\n",
      "Finished Epoch[3 of 10]: [Training] loss = 0.154852 * 17941, metric = 3.68% * 17941 4.933s (3636.9 samples/s);\n",
      "Finished Epoch[4 of 10]: [Training] loss = 0.106380 * 18059, metric = 2.64% * 18059 4.796s (3765.4 samples/s);\n",
      "Learning rate per minibatch: 0.105\n",
      "Finished Epoch[5 of 10]: [Training] loss = 0.069279 * 17957, metric = 1.65% * 17957 4.706s (3815.8 samples/s);\n",
      "Finished Epoch[6 of 10]: [Training] loss = 0.061887 * 18021, metric = 1.50% * 18021 4.919s (3663.5 samples/s);\n",
      "Finished Epoch[7 of 10]: [Training] loss = 0.054078 * 17980, metric = 1.29% * 17980 4.862s (3698.1 samples/s);\n",
      "Finished Epoch[8 of 10]: [Training] loss = 0.050230 * 18025, metric = 1.30% * 18025 4.760s (3786.8 samples/s);\n",
      "Finished Epoch[9 of 10]: [Training] loss = 0.030962 * 17956, metric = 0.86% * 17956 4.775s (3760.4 samples/s);\n",
      "Finished Epoch[10 of 10]: [Training] loss = 0.033263 * 18039, metric = 0.90% * 18039 4.715s (3825.9 samples/s);\n"
     ]
    }
   ],
   "source": [
    "def do_train():\n",
    "    global z\n",
    "    z = create_model()\n",
    "    reader = create_reader(data['train']['file'], is_training=True)\n",
    "    train(reader, z)\n",
    "do_train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows how learning proceeds over epochs (passes through the data).\n",
    "For example, after four epochs, the loss, which is the cross-entropy criterion, has reached 0.11 as measured on the ~18000 samples of this epoch,\n",
    "and that the error rate is 2.6% on those same 18000 training samples.\n",
    "\n",
    "The epoch size is the number of samples--counted as *word tokens*, not sentences--to\n",
    "process between model checkpoints.\n",
    "\n",
    "Once the training has completed (a little less than 2 minutes on a Titan-X or a Surface Book),\n",
    "you will see an output like this\n",
    "```\n",
    "Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039\n",
    "```\n",
    "which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.\n",
    "\n",
    "On a CPU-only machine, it can be 4 or more times slower. You can try setting\n",
    "```python\n",
    "emb_dim    = 50 \n",
    "hidden_dim = 100\n",
    "```\n",
    "to reduce the time it takes to run on a CPU, but the model will not fit as well as when the \n",
    "hidden and embedding dimension are larger. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the model\n",
    "\n",
    "Like the train() function, we also define a function to measure accuracy on a test set by computing the error over multiple minibatches of test data. For evaluating on a small sample read from a file, you can set a minibatch size reflecting the sample size and run the test_minibatch on that instance of data. To see how to evaluate a single sequence, we provide an instance later in the tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate(reader, model_func):\n",
    "    \n",
    "    # Instantiate the model function; x is the input (feature) variable \n",
    "    model = model_func(x)\n",
    "    \n",
    "    # Create the loss and error functions\n",
    "    loss, label_error = create_criterion_function_preferred(model, y)\n",
    "\n",
    "    # process minibatches and perform evaluation\n",
    "    progress_printer = C.logging.ProgressPrinter(tag='Evaluation', num_epochs=0)\n",
    "\n",
    "    while True:\n",
    "        minibatch_size = 500\n",
    "        data = reader.next_minibatch(minibatch_size, input_map={  # fetch minibatch\n",
    "            x: reader.streams.query,\n",
    "            y: reader.streams.slot_labels\n",
    "        })\n",
    "        if not data:                                 # until we hit the end\n",
    "            break\n",
    "\n",
    "        evaluator = C.eval.Evaluator(loss, progress_printer)\n",
    "        evaluator.test_minibatch(data)\n",
    "     \n",
    "    evaluator.summarize_test_progress()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can measure the model accuracy by going through all the examples in the test set and using the ``test_minibatch`` method of the trainer created inside the evaluate function defined above. At the moment (when this tutorial was written) the Trainer constructor requires a learner (even if it is only used to perform ``test_minibatch``) so we have to specify a dummy learner. In the future it will be allowed to construct a Trainer without specifying a learner as long as the trainer only calls ``test_minibatch``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished Evaluation [1]: Minibatch[1-23]: metric = 0.31% * 10984;\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-0.03496236, -0.10450421, -0.05187245,  0.00315526, -0.03009692,\n",
       "       -0.00325531, -0.01932414, -0.07887443,  0.00732914, -0.04862531,\n",
       "        0.00964272,  0.03630733, -0.06187062,  0.01710773, -0.03572332,\n",
       "       -0.02587368, -0.10132953, -0.07231853,  0.00561448, -0.12136029,\n",
       "       -0.0191984 , -0.10075094,  0.04011377,  0.02298491, -0.00923854,\n",
       "        0.009691  ,  0.0097783 , -0.06790726, -0.00056537, -0.07296011,\n",
       "       -0.04167816,  0.01845982, -0.03122563,  0.02366866,  0.05708881,\n",
       "        0.07603112,  0.02723131, -0.02462533, -0.04524489, -0.0555689 ,\n",
       "       -0.10288978, -0.07713016,  0.06880754, -0.05441185, -0.07832464,\n",
       "       -0.0793924 ,  0.01641271,  0.01577672, -0.00071338, -0.08438327,\n",
       "       -0.04914922, -0.07169627,  0.00832607, -0.04796996,  0.06898551,\n",
       "        0.02447084, -0.06484294, -0.04137091, -0.02579352, -0.03093053,\n",
       "        0.02354763, -0.00530715,  0.04292115, -0.01890544, -0.00403343,\n",
       "        0.00891688, -0.07418539, -0.05906755, -0.01341417, -0.11452992,\n",
       "       -0.0419256 , -0.04264174, -0.01602072, -0.01675532, -0.08315146,\n",
       "        0.00711557,  0.00813489,  0.04468995,  0.01084688, -0.02816842,\n",
       "       -0.08520442, -0.04315411, -0.04462098, -0.18218465, -0.09093954,\n",
       "       -0.03417725, -0.04131652,  0.02496975, -0.05490182, -0.08519867,\n",
       "       -0.03578508, -0.10880258, -0.09149118, -0.14748524, -0.04622766,\n",
       "       -0.1308063 ,  0.0984766 , -0.03518417, -0.06054863, -0.06158594,\n",
       "       -0.01769979, -0.07496347, -0.08180398, -0.11138144, -0.01515801,\n",
       "        0.02478061, -0.10648742, -0.08123869, -0.08370569, -0.04487203,\n",
       "       -0.09033024, -0.01348432, -0.01635014, -0.0092629 ,  0.01535111,\n",
       "       -0.09441991,  0.02300696, -0.00615608,  0.01199632, -0.15448859,\n",
       "       -0.11188693, -0.07685462, -0.06171533,  0.02005708, -0.03582112,\n",
       "       -0.04907253, -0.03334751, -0.09121794,  0.09192816], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def do_test():\n",
    "    reader = create_reader(data['test']['file'], is_training=False)\n",
    "    evaluate(reader, z)\n",
    "do_test()\n",
    "z.classify.b.value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following block of code illustrates how to evaluate a single sequence. Additionally we show how one can pass in the information using NumPy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[178, 429, 444, 619, 937, 851, 752, 179]\n",
      "(8, 129)\n",
      "[128 128 128  48 110 128  78 128]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('BOS', 'O'),\n",
       " ('flights', 'O'),\n",
       " ('from', 'O'),\n",
       " ('new', 'B-fromloc.city_name'),\n",
       " ('york', 'I-fromloc.city_name'),\n",
       " ('to', 'O'),\n",
       " ('seattle', 'B-toloc.city_name'),\n",
       " ('EOS', 'O')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load dictionaries\n",
    "query_wl = [line.rstrip('\\n') for line in open(data['query']['file'])]\n",
    "slots_wl = [line.rstrip('\\n') for line in open(data['slots']['file'])]\n",
    "query_dict = {query_wl[i]:i for i in range(len(query_wl))}\n",
    "slots_dict = {slots_wl[i]:i for i in range(len(slots_wl))}\n",
    "\n",
    "# let's run a sequence through\n",
    "seq = 'BOS flights from new york to seattle EOS'\n",
    "w = [query_dict[w] for w in seq.split()] # convert to word indices\n",
    "print(w)\n",
    "onehot = np.zeros([len(w),len(query_dict)], np.float32)\n",
    "for t in range(len(w)):\n",
    "    onehot[t,w[t]] = 1\n",
    "\n",
    "#x = C.sequence.input_variable(vocab_size)\n",
    "pred = z(x).eval({x:[onehot]})[0]\n",
    "print(pred.shape)\n",
    "best = np.argmax(pred,axis=1)\n",
    "print(best)\n",
    "list(zip(seq.split(),[slots_wl[s] for s in best]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying the Model\n",
    "\n",
    "In the following, you will be given tasks to practice modifying CNTK configurations.\n",
    "The solutions are given at the end of this document... but please try without!\n",
    "\n",
    "### A Word About [`Sequential()`](https://www.cntk.ai/pythondocs/layerref.html#sequential)\n",
    "\n",
    "Before jumping to the tasks, let's have a look again at the model we just ran.\n",
    "The model is described in what we call *function-composition style*.\n",
    "```python\n",
    "        Sequential([\n",
    "            Embedding(emb_dim),\n",
    "            Recurrence(LSTM(hidden_dim), go_backwards=False),\n",
    "            Dense(num_labels)\n",
    "        ])\n",
    "```\n",
    "You may be familiar with the \"sequential\" notation from other neural-network toolkits.\n",
    "If not, [`Sequential()`](https://www.cntk.ai/pythondocs/layerref.html#sequential) is a powerful operation that,\n",
    "in a nutshell, allows to compactly express a very common situation in neural networks\n",
    "where an input is processed by propagating it through a progression of layers.\n",
    "`Sequential()` takes an list of functions as its argument,\n",
    "and returns a *new* function that invokes these functions in order,\n",
    "each time passing the output of one to the next.\n",
    "For example,\n",
    "```python\n",
    "\tFGH = Sequential ([F,G,H])\n",
    "    y = FGH (x)\n",
    "```\n",
    "means the same as\n",
    "```\n",
    "    y = H(G(F(x))) \n",
    "```\n",
    "This is known as [\"function composition\"](https://en.wikipedia.org/wiki/Function_composition),\n",
    "and is especially convenient for expressing neural networks, which often have this form:\n",
    "\n",
    "         +-------+   +-------+   +-------+\n",
    "    x -->|   F   |-->|   G   |-->|   H   |--> y\n",
    "         +-------+   +-------+   +-------+\n",
    "\n",
    "Coming back to our model at hand, the `Sequential` expression simply\n",
    "says that our model has this form:\n",
    "\n",
    "         +-----------+   +----------------+   +------------+\n",
    "    x -->| Embedding |-->| Recurrent LSTM |-->| DenseLayer |--> y\n",
    "         +-----------+   +----------------+   +------------+"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: Add Batch Normalization\n",
    "\n",
    "We now want to add new layers to the model, specifically batch normalization.\n",
    "\n",
    "Batch normalization is a popular technique for speeding up convergence.\n",
    "It is often used for image-processing setups, for example our other [hands-on lab on image\n",
    "recognition](./Hands-On-Labs-Image-Recognition).\n",
    "But could it work for recurrent models, too?\n",
    "\n",
    "> Note: training with Batch Normalization is currently only supported on GPU.\n",
    "\n",
    "So your task will be to insert batch-normalization layers before and after the recurrent LSTM layer.\n",
    "If you have completed the [hands-on labs on image processing](https://github.com/Microsoft/CNTK/blob/v2.0/Tutorials/CNTK_201B_CIFAR-10_ImageHandsOn.ipynb),\n",
    "you may remember that the [batch-normalization layer](https://www.cntk.ai/pythondocs/layerref.html#batchnormalization-layernormalization-stabilizer) has this form:\n",
    "```\n",
    "    BatchNormalization()\n",
    "```\n",
    "So please go ahead and modify the configuration and see what happens.\n",
    "\n",
    "If everything went right, you will notice improved convergence speed (`loss` and `metric`)\n",
    "compared to the previous configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your task: Add batch normalization\n",
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):\n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim),\n",
    "            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),\n",
    "            C.layers.Dense(num_labels)\n",
    "        ])\n",
    "\n",
    "# Enable these when done:\n",
    "z = create_model()\n",
    "#do_train()\n",
    "#do_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: Add a Lookahead \n",
    "\n",
    "Our recurrent model suffers from a structural deficit:\n",
    "Since the recurrence runs from left to right, the decision for a slot label\n",
    "has no information about upcoming words. The model is a bit lopsided.\n",
    "Your task will be to modify the model such that\n",
    "the input to the recurrence consists not only of the current word, but also of the next one\n",
    "(lookahead).\n",
    "\n",
    "Your solution should be in function-composition style.\n",
    "Hence, you will need to write a Python function that does the following:\n",
    "\n",
    "* takes no input arguments\n",
    "* creates a placeholder (sequence) variable\n",
    "* computes the \"next value\" in this sequence using the `sequence.future_value()` operation and\n",
    "* concatenates the current and the next value into a vector of twice the embedding dimension using `splice()`\n",
    "\n",
    "and then insert this function into `Sequential()`'s list right after the embedding layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your task: Add lookahead\n",
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):\n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim),\n",
    "            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),\n",
    "            C.layers.Dense(num_labels)\n",
    "        ])\n",
    "    \n",
    "# Enable these when done:\n",
    "z = create_model()\n",
    "#do_train()\n",
    "#do_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Bidirectional Recurrent Model\n",
    "\n",
    "Aha, knowledge of future words help. So instead of a one-word lookahead,\n",
    "why not look ahead until all the way to the end of the sentence, through a backward recurrence?\n",
    "Let us create a bidirectional model!\n",
    "\n",
    "Your task is to implement a new layer that\n",
    "performs both a forward and a backward recursion over the data, and\n",
    "concatenates the output vectors.\n",
    "\n",
    "Note, however, that this differs from the previous task in that\n",
    "the bidirectional layer contains learnable model parameters.\n",
    "In function-composition style,\n",
    "the pattern to implement a layer with model parameters is to write a *factory function*\n",
    "that creates a *function object*.\n",
    "\n",
    "A function object, also known as [*functor*](https://en.wikipedia.org/wiki/Function_object), is an object that is both a function and an object.\n",
    "Which means nothing else that it contains data yet still can be invoked as if it was a function.\n",
    "\n",
    "For example, `Dense(outDim)` is a factory function that returns a function object that contains\n",
    "a weight matrix `W`, a bias `b`, and another function to compute \n",
    "`input @ W + b.` (This is using \n",
    "[Python 3.5 notation for matrix multiplication](https://docs.python.org/3/whatsnew/3.5.html#whatsnew-pep-465).\n",
    "In Numpy syntax it is `input.dot(W) + b`).\n",
    "E.g. saying `Dense(1024)` will create this function object, which can then be used\n",
    "like any other function, also immediately: `Dense(1024)(x)`. \n",
    "\n",
    "Let's look at an example for further clarity: Let us implement a new layer that combines\n",
    "a linear layer with a subsequent batch normalization. \n",
    "To allow function composition, the layer needs to be realized as a factory function,\n",
    "which could look like this:\n",
    "\n",
    "```python\n",
    "def DenseLayerWithBN(dim):\n",
    "    F = Dense(dim)\n",
    "    G = BatchNormalization()\n",
    "    x = placeholder()\n",
    "    apply_x = G(F(x))\n",
    "    return apply_x\n",
    "```\n",
    "\n",
    "Invoking this factory function will create `F`, `G`, `x`, and `apply_x`. In this example, `F` and `G` are function objects themselves, and `apply_x` is the function to be applied to the data.\n",
    "Thus, e.g. calling `DenseLayerWithBN(1024)` will\n",
    "create an object containing a linear-layer function object called `F`, a batch-normalization function object `G`,\n",
    "and `apply_x` which is the function that implements the actual operation of this layer\n",
    "using `F` and `G`. It will then return `apply_x`. To the outside, `apply_x` looks and behaves\n",
    "like a function. Under the hood, however, `apply_x` retains access to its specific instances of `F` and `G`.\n",
    "\n",
    "Now back to our task at hand. You will now need to create a factory function,\n",
    "very much like the example above.\n",
    "You shall create a factory function\n",
    "that creates two recurrent layer instances (one forward, one backward), and then defines an `apply_x` function\n",
    "which applies both layer instances to the same `x` and concatenate the two results.\n",
    "\n",
    "Alright, give it a try! To know how to realize a backward recursion in CNTK,\n",
    "please take a hint from how the forward recursion is done.\n",
    "Please also do the following:\n",
    "* remove the one-word lookahead you added in the previous task, which we aim to replace; and\n",
    "* make sure each LSTM is using `hidden_dim//2` outputs to keep the total number of model parameters limited."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your task: Add bidirectional recurrence\n",
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):  \n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim),\n",
    "            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),\n",
    "            C.layers.Dense(num_labels)\n",
    "        ])\n",
    "\n",
    "# Enable these when done:\n",
    "#do_train()\n",
    "#do_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Works like a charm! This model achieves 0.32%, better than the lookahead model above.\n",
    "The bidirectional model has 40% less parameters than the lookahead one. However, if you go back and look closely\n",
    "you may find that the lookahead one trained about 30% faster.\n",
    "This is because the lookahead model has both less horizontal dependencies (one instead of two\n",
    "recurrences) and larger matrix products, and can thus achieve higher parallelism."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution 1: Adding Batch Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 721479 parameters in 6 parameter tensors.\n",
      "Learning rate per minibatch: 0.21\n",
      "Finished Epoch[1 of 10]: [Training] loss = 0.776891 * 18010, metric = 15.20% * 18010 4.989s (3609.9 samples/s);\n",
      "Finished Epoch[2 of 10]: [Training] loss = 0.227451 * 18051, metric = 5.15% * 18051 4.807s (3755.1 samples/s);\n",
      "Finished Epoch[3 of 10]: [Training] loss = 0.152620 * 17941, metric = 3.55% * 17941 5.105s (3514.4 samples/s);\n",
      "Finished Epoch[4 of 10]: [Training] loss = 0.105561 * 18059, metric = 2.57% * 18059 5.239s (3447.0 samples/s);\n",
      "Learning rate per minibatch: 0.105\n",
      "Finished Epoch[5 of 10]: [Training] loss = 0.066330 * 17957, metric = 1.53% * 17957 5.102s (3519.6 samples/s);\n",
      "Finished Epoch[6 of 10]: [Training] loss = 0.060731 * 18021, metric = 1.45% * 18021 5.167s (3487.7 samples/s);\n",
      "Finished Epoch[7 of 10]: [Training] loss = 0.050680 * 17980, metric = 1.26% * 17980 4.825s (3726.4 samples/s);\n",
      "Finished Epoch[8 of 10]: [Training] loss = 0.045435 * 18025, metric = 1.21% * 18025 4.956s (3637.0 samples/s);\n",
      "Finished Epoch[9 of 10]: [Training] loss = 0.030330 * 17956, metric = 0.86% * 17956 5.162s (3478.5 samples/s);\n",
      "Finished Epoch[10 of 10]: [Training] loss = 0.032149 * 18039, metric = 0.89% * 18039 4.835s (3730.9 samples/s);\n",
      "Finished Evaluation [1]: Minibatch[1-23]: metric = 0.37% * 10984;\n"
     ]
    }
   ],
   "source": [
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):\n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim),\n",
    "            #C.layers.BatchNormalization(),\n",
    "            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),\n",
    "            #C.layers.BatchNormalization(),\n",
    "            C.layers.Dense(num_labels)\n",
    "        ])\n",
    "\n",
    "do_train()\n",
    "do_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution 2: Add a Lookahead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 901479 parameters in 6 parameter tensors.\n",
      "Learning rate per minibatch: 0.21\n",
      "Finished Epoch[1 of 10]: [Training] loss = 0.735173 * 18010, metric = 14.15% * 18010 5.506s (3271.0 samples/s);\n",
      "Finished Epoch[2 of 10]: [Training] loss = 0.199272 * 18051, metric = 4.64% * 18051 5.187s (3480.0 samples/s);\n",
      "Finished Epoch[3 of 10]: [Training] loss = 0.133776 * 17941, metric = 2.98% * 17941 5.012s (3579.6 samples/s);\n",
      "Finished Epoch[4 of 10]: [Training] loss = 0.089028 * 18059, metric = 2.07% * 18059 5.185s (3482.9 samples/s);\n",
      "Learning rate per minibatch: 0.105\n",
      "Finished Epoch[5 of 10]: [Training] loss = 0.050588 * 17957, metric = 1.25% * 17957 5.235s (3430.2 samples/s);\n",
      "Finished Epoch[6 of 10]: [Training] loss = 0.045545 * 18021, metric = 0.98% * 18021 5.147s (3501.3 samples/s);\n",
      "Finished Epoch[7 of 10]: [Training] loss = 0.042307 * 17980, metric = 0.90% * 17980 5.364s (3352.0 samples/s);\n",
      "Finished Epoch[8 of 10]: [Training] loss = 0.031965 * 18025, metric = 0.82% * 18025 5.191s (3472.4 samples/s);\n",
      "Finished Epoch[9 of 10]: [Training] loss = 0.019296 * 17956, metric = 0.45% * 17956 4.963s (3618.0 samples/s);\n",
      "Finished Epoch[10 of 10]: [Training] loss = 0.019359 * 18039, metric = 0.50% * 18039 5.060s (3565.0 samples/s);\n",
      "Finished Evaluation [1]: Minibatch[1-23]: metric = 0.46% * 10984;\n"
     ]
    }
   ],
   "source": [
    "def OneWordLookahead():\n",
    "    x = C.placeholder()\n",
    "    apply_x = C.splice(x, C.sequence.future_value(x))\n",
    "    return apply_x\n",
    "\n",
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):\n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim),\n",
    "            OneWordLookahead(),\n",
    "            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),\n",
    "            C.layers.Dense(num_labels)        \n",
    "        ])\n",
    "\n",
    "do_train()\n",
    "do_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution 3: Bidirectional Recurrent Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training 541479 parameters in 9 parameter tensors.\n",
      "Learning rate per minibatch: 0.21\n",
      "Finished Epoch[1 of 10]: [Training] loss = 0.773435 * 18010, metric = 14.75% * 18010 9.278s (1941.2 samples/s);\n",
      "Finished Epoch[2 of 10]: [Training] loss = 0.189657 * 18051, metric = 4.34% * 18051 8.793s (2052.9 samples/s);\n",
      "Finished Epoch[3 of 10]: [Training] loss = 0.113594 * 17941, metric = 2.65% * 17941 8.689s (2064.8 samples/s);\n",
      "Finished Epoch[4 of 10]: [Training] loss = 0.066348 * 18059, metric = 1.59% * 18059 9.097s (1985.2 samples/s);\n",
      "Learning rate per minibatch: 0.105\n"
     ]
    }
   ],
   "source": [
    "def BiRecurrence(fwd, bwd):\n",
    "    F = C.layers.Recurrence(fwd)\n",
    "    G = C.layers.Recurrence(bwd, go_backwards=True)\n",
    "    x = C.placeholder()\n",
    "    apply_x = C.splice(F(x), G(x))\n",
    "    return apply_x \n",
    "\n",
    "def create_model():\n",
    "    with C.layers.default_options(initial_state=0.1):\n",
    "        return C.layers.Sequential([\n",
    "            C.layers.Embedding(emb_dim),\n",
    "            BiRecurrence(C.layers.LSTM(hidden_dim//2), \n",
    "                                  C.layers.LSTM(hidden_dim//2)),\n",
    "            C.layers.Dense(num_labels)\n",
    "        ])\n",
    "\n",
    "do_train()\n",
    "do_test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}