NNGraph Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NNGraph\n",
    "\n",
    "A graph based container for creating deep learning models.\n",
    "\n",
    "One can use nngraph to create simple networks too. Its syntax provides a more natural interface. It provides a function-like calling mechanism to connect nodes of computational units together. Every node has variable number of inputs and variable number of outputs.\n",
    "\n",
    "The graph has to be fully defined, all connections have to be connected to a node and all inputs to a graph should be provided at runtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "require 'nngraph';"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Simple\n",
    "\n",
    "We can create a simple feedforward neural network easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- it is common style to mark inputs with identity nodes for clarity.\n",
    "input = nn.Identity()()\n",
    "\n",
    "-- each hidden layer is achieved by connecting the previous one\n",
    "-- here we define a single hidden layer network\n",
    "h1 = nn.Tanh()(nn.Linear(20, 10)(input))\n",
    "output = nn.Linear(10, 1)(h1)\n",
    "mlp = nn.gModule({input}, {output})\n",
    "\n",
    "x = torch.rand(20)\n",
    "dx = torch.rand(1)\n",
    "mlp:updateOutput(x)\n",
    "mlp:updateGradInput(x, dx)\n",
    "mlp:accGradParameters(x, dx)\n",
    "\n",
    "-- draw graph (the forward graph, '.fg')\n",
    "-- this will produce an SVG in the runtime directory\n",
    "graph.dot(mlp.fg, 'MLP', 'MLP')\n",
    "itorch.image('MLP.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"MLP.svg\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Node Names\n",
    "\n",
    "The name of the nodes are important. When we draw complicated graphs, we want to be able to match the graph to our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "local function get_network()\n",
    "    -- it is common style to mark inputs with identity nodes for clarity.\n",
    "    local input = nn.Identity()()\n",
    "\n",
    "    -- each hidden layer is achieved by connecting the previous one\n",
    "    -- here we define a single hidden layer network\n",
    "    local h1 = nn.Linear(20, 10)(input)\n",
    "    local h2 = nn.Sigmoid()(h1)\n",
    "    local output = nn.Linear(10, 1)(h2)\n",
    "    \n",
    "    -- the following function call inspects the local variables in this\n",
    "    -- function and finds the nodes corresponding to local variables.\n",
    "    nngraph.annotateNodes()\n",
    "    return nn.gModule({input}, {output}) \n",
    "end\n",
    "mlp = get_network()\n",
    "x = torch.rand(20)\n",
    "dx = torch.rand(1)\n",
    "mlp:updateOutput(x)\n",
    "mlp:updateGradInput(x, dx)\n",
    "mlp:accGradParameters(x, dx)\n",
    "\n",
    "-- draw graph (the forward graph, '.fg')\n",
    "-- this will produce an SVG in the runtime directory\n",
    "graph.dot(mlp.fg, 'MLP', 'MLP_Annotated')\n",
    "itorch.image('MLP_Annotated.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"MLP_Annotated.svg\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identifying Errors at Runtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- We need to set debug flag to true\n",
    "nngraph.setDebug(true)\n",
    "\n",
    "local function get_network()\n",
    "    -- it is common style to mark inputs with identity nodes for clarity.\n",
    "    local input = nn.Identity()()\n",
    "\n",
    "    -- each hidden layer is achieved by connecting the previous one\n",
    "    -- here we define a single hidden layer network\n",
    "    local h1 = nn.Linear(20, 10)(input)\n",
    "    local h2 = nn.Sigmoid()(h1)\n",
    "    local output = nn.Linear(10, 1)(h2)\n",
    "    \n",
    "    -- the following function call inspects the local variables in this\n",
    "    -- function and finds the nodes corresponding to local variables.\n",
    "    nngraph.annotateNodes()\n",
    "    return nn.gModule({input}, {output}) \n",
    "end\n",
    "mlp = get_network()\n",
    "mlp.name = 'MyMLPError'\n",
    "x = torch.rand(15) -- note that this input will cause runtime error\n",
    "\n",
    "-- We do protected call to avoid real error interrupting the notebook\n",
    "local o, err = pcall(function() mlp:updateOutput(x) end)\n",
    "itorch.image('MyMLPError.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But at the same time, an svg file with the name 'MyMLPError.svg' is produced where the node where the error occured is marked in red. One can easily see that the calculation of 'h1' was the problem.\n",
    "\n",
    "\n",
    "<img src=\"MyMLPError.svg\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A More Complete Example\n",
    "\n",
    "Now we will create the core of an RNN module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "function get_rnn(input_size, rnn_size)\n",
    "  \n",
    "    -- there are n+1 inputs (hiddens on each layer and x)\n",
    "    local input = nn.Identity()()\n",
    "    local prev_h = nn.Identity()()\n",
    "\n",
    "    -- RNN tick\n",
    "    local i2h = nn.Linear(input_size, rnn_size)(input)\n",
    "    local h2h = nn.Linear(rnn_size, rnn_size)(prev_h)\n",
    "    local added_h = nn.CAddTable()({i2h, h2h})\n",
    "    local next_h = nn.Tanh()(added_h)\n",
    "    \n",
    "    nngraph.annotateNodes()\n",
    "    return nn.gModule({input, prev_h}, {next_h})\n",
    "end\n",
    "\n",
    "local rnn_net = get_rnn(128, 128)\n",
    "graph.dot(rnn_net.fg, 'rnn_net', 'rnn_net')\n",
    "itorch.image('rnn_net.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"rnn_net.svg\" width=\"400\" height=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect in time\n",
    "Now, let's connect these RNN cores in time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "local function get_rnn2(input_size, rnn_size)\n",
    "    local input1 = nn.Identity()()\n",
    "    local input2 = nn.Identity()()\n",
    "    local prev_h = nn.Identity()()\n",
    "    local rnn_net1 = get_rnn(128, 128)({input1, prev_h})\n",
    "    local rnn_net2 = get_rnn(128, 128)({input2, rnn_net1})\n",
    "    nngraph.annotateNodes()\n",
    "    return nn.gModule({input1, input2, prev_h}, {rnn_net2})\n",
    "end\n",
    "local rnn_net2 = get_rnn2(128, 128)\n",
    "graph.dot(rnn_net2.fg, 'rnn_net2', 'rnn_net2')\n",
    "itorch.image('rnn_net2.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"rnn_net2.svg\" width=\"400\" height=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Debug\n",
    "\n",
    "Even with variable naming, the graph becomes very complicated very quickly. One can then use custom annotations to mark certain paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "local function get_rnn2(input_size, rnn_size)\n",
    "    local input1 = nn.Identity()():annotate{graphAttributes = {style='filled', fillcolor='blue'}}\n",
    "    local input2 = nn.Identity()():annotate{graphAttributes = {style='filled', fillcolor='blue'}}\n",
    "    local prev_h = nn.Identity()():annotate{graphAttributes = {style='filled', fillcolor='blue'}}\n",
    "    local rnn_net1 = get_rnn(128, 128)({input1, prev_h}):annotate{graphAttributes = {style='filled', fillcolor='yellow'}}\n",
    "    local rnn_net2 = get_rnn(128, 128)({input2, rnn_net1}):annotate{graphAttributes = {style='filled', fillcolor='green'}}\n",
    "    nngraph.annotateNodes()\n",
    "    return nn.gModule({input1, input2, prev_h}, {rnn_net2})\n",
    "end\n",
    "local rnn_net3 = get_rnn2(128, 128)\n",
    "graph.dot(rnn_net3.fg, 'rnn_net3', 'rnn_net3')\n",
    "itorch.image('rnn_net3.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"rnn_net3.svg\" width=\"400\" height=\"200\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "iTorch",
   "language": "lua",
   "name": "itorch"
  },
  "language_info": {
   "name": "lua",
   "version": "5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}