3. Tensorflow programming model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "6ab1abd5-af7f-4e9e-96f1-0cf6c9d1b158"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tensorflow\n",
    "\n",
    "When starting off with deep learning, one of the first questions to ask is, which framework to learn?\n",
    "\n",
    "Common choices include Theano, TensorFlow, Torch, and Keras. All of these choices have their own pros and cons and have their own way of doing things.\n",
    "\n",
    "> From [**The Anatomy of Deep Learning Frameworks**](https://medium.com/@gokul_uf/the-anatomy-of-deep-learning-frameworks-46e2a7af5e47#.3ywhrk1st)\n",
    "\n",
    "> The core components of a deep learning framework we must consider are:\n",
    "\n",
    "> + How Tensor Objects are defined. At the heart of the framework is the tensor object. A tensor is a generalization of a matrix to n-dimensions. We need a Tensor Object that supports storing the data in form of tensors. Not just that, we would like the object to be able to convert other data types (images, text, video) into tensors and back, supporting indexing, overloading operators, having a space efficient way to store the data and so on.\n",
    "+ How Operations on the Tensor Object are defined. A neural network can be considered as a series of Operations performed on an input tensor to give an output. \n",
    "+ The use of a Computation Graph and its Optimizations. Instead of implementing operations as functions, they are usually implemented as **classes**. This allows us to store more information about the operation like calculated shape of the output (useful for sanity checks), how to compute the gradient or the gradient itself (for the auto-differentiation), have ways to be able to decide whether to compute the op on GPU or CPU and so on. The power of neural networks lies in the ability to chain multiple operations to form a powerful approximator. Therefore, the standard use case is that you can initialize a tensor, perform actions after actions on them and finally interpret the resulting tensor as labels or real values. Unfortunately, as you chain more and more operations together, several issues arise that can drastically slow down your code and introduce bugs as well. There are more such issues and it becomes necessary to be able to get a bigger picture to even notice that these issues exist. We need a way to optimize the resultant chain of operations for both space and time. A Computation Graph which is basically an object that contains links to the instances of various Ops and the relations between which operation takes the output of which operation as well as additional information. \n",
    "+ The use of Auto-differentiation tools. Another benefit of having the computational graph is that calculating gradients used in the learning phase becomes modular and straightforward to compute. \n",
    "+ The use of BLAS/cuBLAS and cuDNN extensions for maximizing performance. BLAS or Basic Linear Algebra Subprograms are a collection of optimized matrix operations, initially written in Fortran. These can be leveraged to do very fast matrix (tensor) operations and can provide significant speedups. There are many other software packages like Intel MKL, ATLAS which also perform similar functions. BLAS packages are usually optimized assuming that the instructions will be run on a CPU. In the deep learning situation, this is not the case and BLAS may not be able to fully exploit the parallelism offered by GPGPUs. To solve this issue, NVIDIA has released cuBLAS which is optimized for GPUs. This is now included with the CUDA toolkit.\n",
    "\n",
    "\n",
    "\n",
    "The computational model for Tensorflow (`tf`) is a **directed graph**.\n",
    "\n",
    "**Nodes** are *functions* (*operations* in `tf` terminology) and **edges** are *tensors*. \n",
    "\n",
    "**Tensor** are multidimensional data arrays. \n",
    "\n",
    "$$f(a,b) = (a*b) + (a+b)$$\n",
    "\n",
    "![alt text](images/t1.png)\n",
    "\n",
    "There are several reasons for this design:\n",
    "+ The most important is that is a good way to split up computation into small, **easily differentiable** pieces. `tf` uses automatic differentiation to automatically compute the derivative of every node with respect any other node that can affect the first node's output.\n",
    "+ The grah is also a convenient way for distributing computation across multiple CPUs, GPUs, etc.\n",
    "\n",
    "The primary API of `tf` (written in C++) is accessed through Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "47a4bdec-f25c-4842-9025-77aa6ace8805"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are different way of installing `tf`:\n",
    "\n",
    "+ Pip install: May impact existing Python programs on your machine.\n",
    "+ Virtualenv install: Install TensorFlow in its own directory, not impacting any existing Python programs on your machine.\n",
    "+ Anaconda install (Windows: only Python 3, not Python 2.7): Install TensorFlow in its own environment for those running the Anaconda Python distribution. Does not impact existing Python programs on your machine.\n",
    "+ Docker install: Run TensorFlow in a Docker container isolated from all other programs on your machine. It allows coexisting ``tf`` versions.\n",
    "+ Installing from sources: Install TensorFlow by building a pip wheel that you then install using pip.\n",
    "\n",
    "Our preferred way is Docker install."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "20019a20-ae7f-4beb-8cd4-d68150382b1e"
    }
   },
   "source": [
    "## Fundamentals\n",
    "\n",
    "`tf` computation graphs are described in code with `tf` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "6937c67a-43fa-4f1e-ab3d-89a199c83eb0"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "collapsed": false,
    "nbpresent": {
     "id": "aae85c45-c89c-401a-9131-01e07651e0af"
    }
   },
   "outputs": [],
   "source": [
    "# Basic constant operations = to assign a value to a tensor\n",
    "a = tf.constant(2)\n",
    "b = tf.constant(3)\n",
    "c = a+b \n",
    "d = a*b\n",
    "e = c+d\n",
    "\n",
    "# non interactive session\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(\"a=2\")\n",
    "    print(\"b=3\")\n",
    "    print(\"(a+b)+(a*b) = %i\" % sess.run(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create initialized tensors in many ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = tf.zeros([2,3], tf.int32)\n",
    "b = tf.ones([2,3], tf.int32)\n",
    "c = tf.fill([3,3], 23.9)\n",
    "d = tf.range(0,10,1)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(a))\n",
    "    print(sess.run(b))\n",
    "    print(sess.run(c))\n",
    "    print(sess.run(d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``tf`` sequences are not iterable!\n",
    "\n",
    "We can also generate random variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = tf.random_normal([2,2], 0.0, 1.0)\n",
    "b = tf.random_uniform([2,2], 0.0, 1.0)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(a))\n",
    "    print(sess.run(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to generate random shuffled number in tensorflow?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "idx = tf.constant(20)\n",
    "idx_list = tf.range(idx) # 0~19\n",
    "shuffle = tf.random_shuffle(idx_list)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    a, b = sess.run([idx_list, shuffle])\n",
    "    print a\n",
    "    print b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "collapsed": false,
    "nbpresent": {
     "id": "3fd06df9-1eec-447c-bed7-d88be2d8d4d9"
    }
   },
   "outputs": [],
   "source": [
    "# Basic operations with variable graph input\n",
    "\n",
    "a = tf.placeholder(tf.int16)\n",
    "b = tf.placeholder(tf.int16)\n",
    "c = tf.add(a,b) \n",
    "d = tf.mul(a,b)\n",
    "e = tf.add(c,d)\n",
    "\n",
    "values = feed_dict={a: 5, b: 3}\n",
    "\n",
    "# non interactive session\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print('a = %i' % sess.run(a, values))\n",
    "    print('b = %i' % sess.run(b, values))\n",
    "    print(\"(a+b)+(a*b) = %i\" % sess.run(e, values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "92384d6d-bed6-4e4e-bfc1-ddf88de0963e"
    }
   },
   "source": [
    "A computational graph is a series of functions chained together, each passing its output to zero, one or more functions further along the chain.\n",
    "\n",
    "In this way we can construct very complex transformations on data by using a library of simple functions.\n",
    "\n",
    "Nodes represent some sort of computation beign done in the graph context.\n",
    "\n",
    "Edges are the actual values (tensors) that get passed to and from nodes.\n",
    "\n",
    "![alt text](images/t2.png)\n",
    "\n",
    "+ The values flowing into the graph can come from different sources: from a different graph, from a file, entered by the client, etc. The *input* nodes simply pass on values given to them.\n",
    "+ The other nodes take values, apply an operation and output their result. \n",
    "\n",
    "Values running on edges are tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "a4071b69-ab10-4ccd-93ed-b445d8ba7299"
    }
   },
   "outputs": [],
   "source": [
    "# Basic operations with variable as graph input\n",
    "\n",
    "a = tf.placeholder(tf.int16,shape=[2])\n",
    "b = tf.placeholder(tf.int16,shape=[2])\n",
    "c = tf.add(a,b) \n",
    "d = tf.mul(a,b)\n",
    "e = tf.add(c,d)\n",
    "\n",
    "variables = feed_dict={a: [2,2], b: [3,3]}\n",
    "\n",
    "# non interactive session\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(a, variables))\n",
    "    print(sess.run(b, variables))\n",
    "    print(sess.run(e, variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "e8e3c449-56f2-4b1e-9112-6b013cfc6ec6"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Implement this computational graph:\n",
    "\n",
    "![alt text](images/t3.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "collapsed": false,
    "nbpresent": {
     "id": "77252053-4889-4a38-a497-49465a2e9ac1"
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "86c6424a-3e4a-4e6c-98f1-f9917cc659c8"
    }
   },
   "source": [
    "There are certein connections between nodes that are not allowed: you cannot create **circular dependencies**.\n",
    "\n",
    "> Dependency: Any node A that is required for the computation of a later node B is said to be a **dependency** of B.\n",
    "\n",
    "The main reason is that dependencies create endless feedback loops.\n",
    "\n",
    "There is one exception to this rule: recurrent neural networks. In this case `tf` simulate this kind of dependences by copying a **finite** number of versions of the graph, placing them side-by-side, and feeding them into another sequence. This process is referred as **unrolling** the graph.\n",
    "\n",
    "![alt text](images/t4.png)\n",
    "\n",
    "Keeping track of dependencies is a basic feature of `tf`. Let's suppose that we want to compute the output value of the `mul` node. We can see in the unrolled graph that is not necessary to compute the full graph to get the output of that node. But how to ensure that we only compute the necessary nodes?\n",
    "\n",
    "It's pretty easy:\n",
    "+ Build a list for each node with all nodes it directly depends on (not indirectly!). \n",
    "+ Initialize an empty stack, wich will eventually hold all the nodes we want to compute. \n",
    "+ Put the node you want to get the output from. \n",
    "+ Recursively, look at its dependency list and add to the stack the nodes it depends on, until there are no dependencies left to run and in this way we guarantee that we have all the nodes we need.\n",
    "\n",
    "The stack will be ordered in a way that we are guaranteed to be able to run each node in the stack as we iterate through it. \n",
    "\n",
    "The main thing to look out for is to keep track of nodes that were already computed and to store their value in memory.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f3a49756-9e22-421c-a0d3-8bc670856ab0"
    }
   },
   "source": [
    "As we have seen in previous code, `tf` workflow is a two-step process:\n",
    "\n",
    "+ Define the computation graph.\n",
    "+ Run the graph with data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "24af87db-9051-4768-9064-079dc5e9b965"
    }
   },
   "outputs": [],
   "source": [
    "# graph definition\n",
    "# we can assign a name to every node\n",
    "\n",
    "a = tf.placeholder(tf.int32, name='input_a')\n",
    "b = tf.placeholder(tf.int32, name='input_b')\n",
    "c = tf.add(a,b,name='add_1') \n",
    "d = tf.mul(a,b,name='mul_1')\n",
    "e = tf.add(c,d,name='add_2')\n",
    "\n",
    "values = feed_dict={a: 5, b: 3}\n",
    "\n",
    "# now we can run the graph in an interactive session\n",
    "\n",
    "sess = tf.Session()\n",
    "print(sess.run(e, values))\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "efe7c304-c8de-4d24-8bd0-dbfa078133b5"
    }
   },
   "source": [
    "`tf` has a very useful tool: `tensor-board`. Let's see how to use it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "c14452d1-3454-4704-b59a-e18b8251103b"
    }
   },
   "outputs": [],
   "source": [
    "# cleaning the tf graph space\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.placeholder(tf.int16, name='input_a')\n",
    "b = tf.placeholder(tf.int16, name='input_b')\n",
    "c = tf.add(a,b,name='add_1') \n",
    "d = tf.mul(a,b,name='mul_1')\n",
    "e = tf.add(c,d,name='add_2')\n",
    "\n",
    "values = feed_dict={a: 5, b: 3}\n",
    "\n",
    "# now we can run the graph\n",
    "\n",
    "# graphs are run by invoking Session objects\n",
    "session = tf.Session()\n",
    "\n",
    "# when you are passing an operation to 'run' you are \n",
    "# asking to run all operations necessary to compute that node\n",
    "\n",
    "# you can save the value of the node in a Python var\n",
    "output = session.run(e, values)\n",
    "print(output)\n",
    "\n",
    "# now let's visualize the graph\n",
    "\n",
    "# SummaryWriter is an object where we can save information\n",
    "# about the execution of the computational graph\n",
    "writer = tf.train.SummaryWriter('my_graph', session.graph)\n",
    "writer.close()\n",
    "\n",
    "# closing interactive session\n",
    "session.close()\n",
    "print output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "33897391-05a6-4b4e-a235-9e162427f843"
    }
   },
   "source": [
    "Open a terminal and type in:\n",
    "\n",
    "`tensorboard --logdir=\"my_graph\"`\n",
    "\n",
    "This starts a `tensorboard` server on port 6006. There, click on the `Graphs` link. You can see that each of the nodes is labeled based on the `name` parameter you passed into each operation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "757d1ac9-70da-430a-819a-903afd30e0bb"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Implement and visualize this graph for a constant tensor `[5,3]`:\n",
    "\n",
    "![alt text](images/t5.png)\n",
    "\n",
    "Check these functions in the `tf` official documentation (https://www.tensorflow.org/): `tf.reduce_prod`, `tf.reduce_sum`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "0cb9c1e0-21d6-4dfd-bfbe-582238143402"
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "1acdc391-57ed-4c09-854d-b7c8cb07701b"
    }
   },
   "source": [
    "### `tf` input data \n",
    "\n",
    "`tf` can take several Python var types that are automatically converted to tensors:\n",
    "\n",
    "`tf.constant([5,3], name='input_a')`\n",
    "\n",
    "But `tf` has a plethora of other data types: `tf_int16`, `tf_quint8`, etc.\n",
    "\n",
    "`tf` is tightly integrated with NumPy. In fact, `tf` data types are based on those from NumPy. Tensors returned from `Session.run` are NumPy arrays. NumPy arrays is the recommended way of specifying tensors.\n",
    "\n",
    "The `shape` of tensors describe both the number of dimensions in a tensor  as well as the length of each dimension. In addition to to being able to specify fixed lengths to each dimension, you can also assign a flexible length by passing in `None` as dimension's value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "a637e0a1-a62b-49a1-99e2-d8eeb974fada"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.placeholder(tf.int16, shape=[2,2], name='input_a')\n",
    "shape = tf.shape(a)\n",
    "\n",
    "session = tf.Session()\n",
    "print(session.run(shape))\n",
    "session.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can feed data points to placeholder by iterating through the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "list_a_values = [1,2,3]\n",
    "\n",
    "a = tf.placeholder(tf.int16)\n",
    "b = a * 2\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    for a_value in list_a_values:\n",
    "        print(sess.run(b,{a: a_value}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c4c8921f-ec0b-4302-bd4e-23251c7cd7de"
    }
   },
   "source": [
    "### `tf` operations\n",
    "\n",
    "`tf` overloads common mathematical operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "65a196d3-5416-4ccc-923c-29c06466427c"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.placeholder(tf.int16)\n",
    "b = tf.placeholder(tf.int16)\n",
    "c = a+b \n",
    "d = a*b\n",
    "e = c+d\n",
    "\n",
    "variables = feed_dict={a: 5, b: 3}\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(\"(a+b)+(a*b) = %i\" % sess.run(e, variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are more [Tensorflow Operations](https://www.tensorflow.org/api_guides/python/math_ops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "6c4f248a-2039-4d50-946e-934257e12d29"
    }
   },
   "source": [
    "### `tf` graphs\n",
    "\n",
    "Creating a graph is simple:\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "g = tf.Graph()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3d475195-dd8a-4ce5-8e90-f7bcd42008f4"
    }
   },
   "source": [
    "Once the graph is initialized we can attach operation to it by using the `Graph.as_default()` method:\n",
    "\n",
    "```python\n",
    "with g.as_default():\n",
    "    a = tf.mul(2,3)\n",
    "    ...\n",
    "```\n",
    "\n",
    "`tf` automatically creates a graph at the beginning and assigns it to be the default. Thus, if not using `Graph.as_default()` any operation will be automatically placed in the default graph.\n",
    "\n",
    "Creating multiple graphs can be useful if you are defining multiple models that do not have interdependencies:\n",
    "\n",
    "```python\n",
    "g1 = tf.Graph()\n",
    "g2 = tf.Graph()\n",
    "\n",
    "with g1.as_default():\n",
    "    ...\n",
    "    \n",
    "with g2.as_default():\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "e0b483f6-ccce-4f20-b082-0546e7f5f1e7"
    }
   },
   "source": [
    "### `tf` Variables\n",
    "\n",
    "`Tensor` and `Operation` objects are immutable, but we need a mechanism to save changing values over time. \n",
    "\n",
    "This is accomplished with `Variable` objects, which contain mutable tensor values that persist accross multiple calls to `Session.run()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "8f062cb0-836d-47b9-a5fd-a9291431e3dc"
    }
   },
   "source": [
    "Variables can be used anywhere you might use a tensor.\n",
    "\n",
    "`tf` has a number of helper operations to initialize variables: `tf-zeros()`, `tf_ones()`, `tf.random_uniform()`, `tf.random_normal()`, etc.\n",
    "\n",
    "`Variable` objects live in a `Graph` but their state is managed by `Session`. Because of these they need an extra step for inicialization:\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = tf.add(5,a)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.initialize_all_variables())\n",
    "    ...\n",
    "\n",
    "```\n",
    "\n",
    "In order to chage the value of a `Variable` we can use the `Variable.assign()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "e46c73f6-5607-4af4-b22f-41f3f4dcec9e"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = a.assign(tf.mul(2,a))  # variables are objects, not ops.\n",
    "\n",
    "# The statement a.assign(...) does not actually assign any to a, \n",
    "# but rather creates a tf.Operation that you have to explicitly \n",
    "# run to update the variable.\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.initialize_all_variables())\n",
    "    print \"a:\",a.eval()   # variables are objects, not ops.\n",
    "    print \"b:\", sess.run(b)\n",
    "    print \"b:\", sess.run(b)\n",
    "    print \"b:\", sess.run(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = a.assign(tf.mul(2,a))\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.initialize_all_variables())\n",
    "    print a.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = a.assign(tf.mul(2,a))\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.initialize_all_variables())\n",
    "    sess.run(b)\n",
    "    print a.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "defd975d-f087-4747-9b2f-06ccde7ee725"
    }
   },
   "source": [
    "We can increment and decrement variables: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "0eb291f0-5427-44ad-b55b-08fd1ce2ef01"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.initialize_all_variables())\n",
    "    print(sess.run(a.assign_add(1)))\n",
    "    print(sess.run(a.assign_sub(1)))\n",
    "    print(sess.run(a.assign_sub(1)))\n",
    "    sess.run(tf.initialize_all_variables())\n",
    "    print(sess.run(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "ca99d481-e62a-45c3-960c-ad87b458d8cb"
    }
   },
   "source": [
    "Some classes of `tf` (f.e. `Optimizer`) are able to automatically change variable values without explicitely asking to do so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensorflow sessions maintain values separately, each Session can have its own current value for a variable defined in the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(10)\n",
    "\n",
    "sess1 = tf.Session()\n",
    "sess2 = tf.Session()\n",
    "\n",
    "sess1.run(tf.initialize_all_variables())\n",
    "sess2.run(tf.initialize_all_variables())\n",
    "\n",
    "print sess1.run(a.assign_add(10))\n",
    "print sess2.run(a.assign_sub(2))\n",
    "\n",
    "sess1.close()\n",
    "sess2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "ee6da6ee-dd82-49bf-8a6f-b92a8413b396"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `tf` name scopes\n",
    "\n",
    "`tf` offers a tool to help organize your graphs: name scopes.\n",
    "\n",
    "Name scopes allows you to group operations into larger, named blocks. This is very usefu to visualize complex models with `tensorboard`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "ea824159-e736-4e2a-b024-524e337f6fac"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.name_scope(\"Scope_A\"):\n",
    "    a = tf.add(1, 2, name=\"A_add\")\n",
    "    b = tf.mul(a, 3, name=\"A_mul\")\n",
    "\n",
    "with tf.name_scope(\"Scope_B\"):\n",
    "    c = tf.add(4, 5, name=\"B_add\")\n",
    "    d = tf.mul(c, 6, name=\"B_mul\")\n",
    "\n",
    "e = tf.add(b, d, name=\"output\")\n",
    "\n",
    "writer = tf.train.SummaryWriter('./name_scope_1', graph=tf.get_default_graph())\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "236459a0-db19-4ca3-ab64-d36697e6180f"
    }
   },
   "source": [
    "We can start `tensorboard` to see the graph: `tensorboard --logdir=\"./name_scope_1\"`.\n",
    "\n",
    "You can expand the name scope boxes by clicking `+`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "58f4f8ea-e41a-469a-a078-bb1e1725a1d3"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Let's built and visualize and complex model:\n",
    "\n",
    "+ Our inputs will be placeholders.\n",
    "+ The model will take in a single vector of any lenght.\n",
    "+ The graph will be segmented in name scopes.\n",
    "+ We will accumulate the total value of all outputs over time.\n",
    "+ At each run, we are going to save the output of the graph, the accumulated total of all outputs, and the average value of all outputs to disk for use in `tensorboard`.\n",
    "\n",
    "\n",
    "![alt text](images/t6.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "collapsed": true,
    "nbpresent": {
     "id": "a8218e4e-4edc-48b5-a6a3-3af3169dee3f"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Explicitly create a Graph object\n",
    "graph = tf.Graph()\n",
    "\n",
    "with graph.as_default():\n",
    "    \n",
    "    with tf.name_scope(\"variables\"):\n",
    "# your code here\n",
    "\n",
    "    \n",
    "    # Primary transformation Operations\n",
    "    with tf.name_scope(\"transformation\"):\n",
    "        \n",
    "        # Separate input layer\n",
    "        with tf.name_scope(\"input\"):\n",
    "# your code here\n",
    "\n",
    "    \n",
    "        # Separate middle layer\n",
    "        with tf.name_scope(\"intermediate_layer\"):\n",
    "# your code here\n",
    "\n",
    "        \n",
    "        # Separate output layer\n",
    "# your code here\n",
    "        \n",
    "    with tf.name_scope(\"update\"):\n",
    "        # Increments the total_output Variable by the latest output\n",
    "# your code here\n",
    "\n",
    "    \n",
    "    # Summary Operations\n",
    "    with tf.name_scope(\"summaries\"):\n",
    "        avg = tf.div(update_total, tf.cast(increment_step, tf.float32), name=\"average\")\n",
    "        \n",
    "        # Creates summaries for output node\n",
    "        tf.scalar_summary(b'Output', output, name=\"output_summary\")\n",
    "        tf.scalar_summary(b'Sum of outputs over time', update_total, name=\"total_summary\")\n",
    "        tf.scalar_summary(b'Average of outputs over time', avg, name=\"average_summary\")\n",
    "    \n",
    "    # Global Variables and Operations\n",
    "    with tf.name_scope(\"global_ops\"):\n",
    "        # Initialization Op\n",
    "        init = tf.initialize_all_variables()    \n",
    "        # Merge all summaries into one Operation\n",
    "        merged_summaries = tf.merge_all_summaries()\n",
    "\n",
    "# Start an interactive Session, using the explicitly created Graph\n",
    "sess = tf.Session(graph=graph)\n",
    "\n",
    "# Open a SummaryWriter to save summaries\n",
    "writer = tf.train.SummaryWriter('./improved_graph', graph)\n",
    "\n",
    "# Initialize Variables\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "76d05e86-509b-494a-8364-953c74c1aadb"
    }
   },
   "source": [
    "Let's write a function to run the graph several times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "ac2057e4-ca62-4985-af03-04ecaf7c41b5"
    }
   },
   "outputs": [],
   "source": [
    "def run_graph(input_tensor):\n",
    "    \"\"\"\n",
    "    Helper function; runs the graph with given input tensor and saves summaries\n",
    "    \"\"\"\n",
    "    feed_dict = {a: input_tensor}\n",
    "    out, step, summary = sess.run([output, increment_step, merged_summaries], \n",
    "                                  feed_dict=feed_dict)\n",
    "    writer.add_summary(summary, global_step=step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "47fcb7a3-7d43-4dfc-9b03-2fc208b165e6"
    }
   },
   "outputs": [],
   "source": [
    "# Run the graph with various inputs\n",
    "run_graph([2,8])\n",
    "run_graph([3,1,3,3])\n",
    "run_graph([8])\n",
    "run_graph([1,2,3])\n",
    "run_graph([11,4])\n",
    "run_graph([4,1])\n",
    "run_graph([7,3,1])\n",
    "run_graph([6,3])\n",
    "run_graph([0,2])\n",
    "run_graph([4,5,6])\n",
    "\n",
    "# Write the summaries to disk\n",
    "writer.flush()\n",
    "\n",
    "# Close the SummaryWriter\n",
    "writer.close()\n",
    "\n",
    "# Close the session\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "4adbbc73-2fc9-4f83-8c54-081b32a3375e"
    }
   },
   "source": [
    "To start TensorBoard after running this code, run the following command:\n",
    "\n",
    "`tensorboard --logdir='./improved_graph'`   "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  },
  "nbpresent": {
   "slides": {
    "0eb9ee9c-5399-4988-94f3-8cee78537404": {
     "id": "0eb9ee9c-5399-4988-94f3-8cee78537404",
     "prev": null,
     "regions": {
      "4acc45f3-e023-4446-a93e-619fe0b22943": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "47a4bdec-f25c-4842-9025-77aa6ace8805",
        "part": "whole"
       },
       "id": "4acc45f3-e023-4446-a93e-619fe0b22943"
      }
     }
    },
    "10844e3a-543e-48a6-b696-d4dcfd40d55f": {
     "id": "10844e3a-543e-48a6-b696-d4dcfd40d55f",
     "prev": "5fc39987-5f96-4eef-88a0-b974113bfbab",
     "regions": {
      "af141f66-ba8f-4546-ab4a-e5e22cac1950": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "f3a49756-9e22-421c-a0d3-8bc670856ab0",
        "part": "whole"
       },
       "id": "af141f66-ba8f-4546-ab4a-e5e22cac1950"
      }
     }
    },
    "131dcdac-6159-44e0-9867-2c6521c536b9": {
     "id": "131dcdac-6159-44e0-9867-2c6521c536b9",
     "prev": "f788e7d5-5631-4090-b44e-03ea24aa1887",
     "regions": {
      "1ef1e8fa-cac5-4873-a3f7-d39435a2c89a": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "efe7c304-c8de-4d24-8bd0-dbfa078133b5",
        "part": "whole"
       },
       "id": "1ef1e8fa-cac5-4873-a3f7-d39435a2c89a"
      }
     }
    },
    "168f7f5f-f3e3-46ad-b488-436b5cc1fea0": {
     "id": "168f7f5f-f3e3-46ad-b488-436b5cc1fea0",
     "prev": "33131316-744b-45af-b499-9e1784ddd7ed",
     "regions": {
      "930c6824-963f-4cd5-bfde-65130261f1c7": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "a8218e4e-4edc-48b5-a6a3-3af3169dee3f",
        "part": "whole"
       },
       "id": "930c6824-963f-4cd5-bfde-65130261f1c7"
      }
     }
    },
    "2b9c3116-58b8-4773-8bb0-1bf8fe05296d": {
     "id": "2b9c3116-58b8-4773-8bb0-1bf8fe05296d",
     "prev": "3ff0f7df-82f5-43b2-be58-8ef1fea20437",
     "regions": {
      "e97963bc-5880-455f-b739-fe2035ff0725": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "ee6da6ee-dd82-49bf-8a6f-b92a8413b396",
        "part": "whole"
       },
       "id": "e97963bc-5880-455f-b739-fe2035ff0725"
      }
     }
    },
    "33131316-744b-45af-b499-9e1784ddd7ed": {
     "id": "33131316-744b-45af-b499-9e1784ddd7ed",
     "prev": "dfb0485f-4165-455b-a321-1368a5593f90",
     "regions": {
      "2f4480ab-af8e-4c1b-bbe7-e31da84b1df1": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "58f4f8ea-e41a-469a-a078-bb1e1725a1d3",
        "part": "whole"
       },
       "id": "2f4480ab-af8e-4c1b-bbe7-e31da84b1df1"
      }
     }
    },
    "39446710-2f8f-48cc-9800-22e7cbf9baa7": {
     "id": "39446710-2f8f-48cc-9800-22e7cbf9baa7",
     "prev": "564fcf1c-547b-4fc1-a39d-8998a92426ce",
     "regions": {
      "5682b394-0a6a-4d97-974f-ecaeb0e167fb": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "a637e0a1-a62b-49a1-99e2-d8eeb974fada",
        "part": "whole"
       },
       "id": "5682b394-0a6a-4d97-974f-ecaeb0e167fb"
      }
     }
    },
    "3c0bfcfd-edb6-4599-814d-85a7bdd827d3": {
     "id": "3c0bfcfd-edb6-4599-814d-85a7bdd827d3",
     "prev": "b8ef6042-0e70-43fe-a97b-c3c5e98e48a9",
     "regions": {
      "d00b6aa4-5b6d-4c1f-9db3-0f8e0c27be3b": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "6937c67a-43fa-4f1e-ab3d-89a199c83eb0",
        "part": "whole"
       },
       "id": "d00b6aa4-5b6d-4c1f-9db3-0f8e0c27be3b"
      }
     }
    },
    "3ff0f7df-82f5-43b2-be58-8ef1fea20437": {
     "id": "3ff0f7df-82f5-43b2-be58-8ef1fea20437",
     "prev": "8b8a4005-fafe-4345-b56a-ceea0afd3563",
     "regions": {
      "94792abe-ffb8-422b-89d2-7345f8cd9706": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "ca99d481-e62a-45c3-960c-ad87b458d8cb",
        "part": "whole"
       },
       "id": "94792abe-ffb8-422b-89d2-7345f8cd9706"
      }
     }
    },
    "42b53f50-3c32-4035-9006-72199a92dfa1": {
     "id": "42b53f50-3c32-4035-9006-72199a92dfa1",
     "prev": "99bfd62c-37d6-4513-975f-7453d45208bf",
     "regions": {
      "b3a398a7-4346-4a59-aea5-13fa483aab87": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "a4071b69-ab10-4ccd-93ed-b445d8ba7299",
        "part": "whole"
       },
       "id": "b3a398a7-4346-4a59-aea5-13fa483aab87"
      }
     }
    },
    "4d691fc4-2db6-4207-aafe-5534e2efcc31": {
     "id": "4d691fc4-2db6-4207-aafe-5534e2efcc31",
     "prev": "9169cd80-86a2-4764-bf37-e1c61e762403",
     "regions": {
      "1016787c-0369-47d3-8a83-316f76b88dd7": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "47fcb7a3-7d43-4dfc-9b03-2fc208b165e6",
        "part": "whole"
       },
       "id": "1016787c-0369-47d3-8a83-316f76b88dd7"
      }
     }
    },
    "50c442dc-bea5-4ccb-a16b-32b67ba7974d": {
     "id": "50c442dc-bea5-4ccb-a16b-32b67ba7974d",
     "prev": "39446710-2f8f-48cc-9800-22e7cbf9baa7",
     "regions": {
      "fdeaf8e6-6364-4c4d-a0b7-aeffa5ca4658": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "c4c8921f-ec0b-4302-bd4e-23251c7cd7de",
        "part": "whole"
       },
       "id": "fdeaf8e6-6364-4c4d-a0b7-aeffa5ca4658"
      }
     }
    },
    "564fcf1c-547b-4fc1-a39d-8998a92426ce": {
     "id": "564fcf1c-547b-4fc1-a39d-8998a92426ce",
     "prev": "65ab85a7-29e3-409f-8440-429ff4935863",
     "regions": {
      "d2bce45d-7487-41a5-b872-ecd420e8ada9": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "1acdc391-57ed-4c09-854d-b7c8cb07701b",
        "part": "whole"
       },
       "id": "d2bce45d-7487-41a5-b872-ecd420e8ada9"
      }
     }
    },
    "5bde732d-d4dd-4c0a-887b-8ecfbe878dbe": {
     "id": "5bde732d-d4dd-4c0a-887b-8ecfbe878dbe",
     "prev": "65c8bb38-d316-4bae-9a4f-a2853b16d1b7",
     "regions": {
      "af634b4f-62f6-4a77-af50-72798a8ab9fa": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "6c4f248a-2039-4d50-946e-934257e12d29",
        "part": "whole"
       },
       "id": "af634b4f-62f6-4a77-af50-72798a8ab9fa"
      }
     }
    },
    "5e7fc216-62a9-4e9e-aec1-86c77ce5811b": {
     "id": "5e7fc216-62a9-4e9e-aec1-86c77ce5811b",
     "prev": "b7abb62f-c11a-4375-a27e-e793c2795a3d",
     "regions": {
      "cd5f198c-ac67-440e-923f-4388a6d666b6": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "33897391-05a6-4b4e-a235-9e162427f843",
        "part": "whole"
       },
       "id": "cd5f198c-ac67-440e-923f-4388a6d666b6"
      }
     }
    },
    "5fc39987-5f96-4eef-88a0-b974113bfbab": {
     "id": "5fc39987-5f96-4eef-88a0-b974113bfbab",
     "prev": "c398a942-e2de-433f-b565-b6c258e5f76f",
     "regions": {
      "a5089fac-dd81-4121-81cb-617389f4448b": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "86c6424a-3e4a-4e6c-98f1-f9917cc659c8",
        "part": "whole"
       },
       "id": "a5089fac-dd81-4121-81cb-617389f4448b"
      }
     }
    },
    "65ab85a7-29e3-409f-8440-429ff4935863": {
     "id": "65ab85a7-29e3-409f-8440-429ff4935863",
     "prev": "8d834420-65a6-4be5-ab68-621de6fbe163",
     "regions": {
      "d0614420-c749-4937-ab00-52caeb7ba9b0": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "0cb9c1e0-21d6-4dfd-bfbe-582238143402",
        "part": "whole"
       },
       "id": "d0614420-c749-4937-ab00-52caeb7ba9b0"
      }
     }
    },
    "65c8bb38-d316-4bae-9a4f-a2853b16d1b7": {
     "id": "65c8bb38-d316-4bae-9a4f-a2853b16d1b7",
     "prev": "50c442dc-bea5-4ccb-a16b-32b67ba7974d",
     "regions": {
      "125c6623-7694-4973-beb8-17015254fbd6": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "65a196d3-5416-4ccc-923c-29c06466427c",
        "part": "whole"
       },
       "id": "125c6623-7694-4973-beb8-17015254fbd6"
      }
     }
    },
    "6b170754-f46e-4f7d-b3af-4fc2d6777247": {
     "id": "6b170754-f46e-4f7d-b3af-4fc2d6777247",
     "prev": "df917a16-e756-4db8-ae9a-506e790cf04c",
     "regions": {
      "d9e1fcf9-c339-49db-8020-de72925717c8": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "e46c73f6-5607-4af4-b22f-41f3f4dcec9e",
        "part": "whole"
       },
       "id": "d9e1fcf9-c339-49db-8020-de72925717c8"
      }
     }
    },
    "7675660a-4c0e-4fde-bedc-6bfdefe232d4": {
     "id": "7675660a-4c0e-4fde-bedc-6bfdefe232d4",
     "prev": "4d691fc4-2db6-4207-aafe-5534e2efcc31",
     "regions": {
      "46382fff-6b3a-40f9-8af5-25052be0b668": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "4adbbc73-2fc9-4f83-8c54-081b32a3375e",
        "part": "whole"
       },
       "id": "46382fff-6b3a-40f9-8af5-25052be0b668"
      }
     }
    },
    "79afa2d5-c641-43ae-8f08-3d64fdd52373": {
     "id": "79afa2d5-c641-43ae-8f08-3d64fdd52373",
     "prev": "defa79e9-d30e-49e9-b206-1dcf6736fd0b",
     "regions": {
      "e01a8964-f684-4035-9ddc-ef3516fdec90": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "e0b483f6-ccce-4f20-b082-0546e7f5f1e7",
        "part": "whole"
       },
       "id": "e01a8964-f684-4035-9ddc-ef3516fdec90"
      }
     }
    },
    "7fe8d340-580d-4293-bf70-3058295f0e5f": {
     "id": "7fe8d340-580d-4293-bf70-3058295f0e5f",
     "prev": "168f7f5f-f3e3-46ad-b488-436b5cc1fea0",
     "regions": {
      "ddfe5f89-e891-4b9d-9bdf-c0fc3fbc2c84": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "76d05e86-509b-494a-8364-953c74c1aadb",
        "part": "whole"
       },
       "id": "ddfe5f89-e891-4b9d-9bdf-c0fc3fbc2c84"
      }
     }
    },
    "8b8a4005-fafe-4345-b56a-ceea0afd3563": {
     "id": "8b8a4005-fafe-4345-b56a-ceea0afd3563",
     "prev": "c45f3e22-0112-491b-8708-cfb8fd55e67d",
     "regions": {
      "c09e06d3-9ce9-462e-b827-de824b2fd3eb": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "0eb291f0-5427-44ad-b55b-08fd1ce2ef01",
        "part": "whole"
       },
       "id": "c09e06d3-9ce9-462e-b827-de824b2fd3eb"
      }
     }
    },
    "8d834420-65a6-4be5-ab68-621de6fbe163": {
     "id": "8d834420-65a6-4be5-ab68-621de6fbe163",
     "prev": "5e7fc216-62a9-4e9e-aec1-86c77ce5811b",
     "regions": {
      "0709a4f5-d5b2-486f-a68d-508ec9b3118a": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "757d1ac9-70da-430a-819a-903afd30e0bb",
        "part": "whole"
       },
       "id": "0709a4f5-d5b2-486f-a68d-508ec9b3118a"
      }
     }
    },
    "8f3b675e-7fe6-447e-864b-074941b1619c": {
     "id": "8f3b675e-7fe6-447e-864b-074941b1619c",
     "prev": "42b53f50-3c32-4035-9006-72199a92dfa1",
     "regions": {
      "6b0c668f-bcd7-42e7-886d-8d8a18b8abe7": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "e8e3c449-56f2-4b1e-9112-6b013cfc6ec6",
        "part": "whole"
       },
       "id": "6b0c668f-bcd7-42e7-886d-8d8a18b8abe7"
      }
     }
    },
    "9169cd80-86a2-4764-bf37-e1c61e762403": {
     "id": "9169cd80-86a2-4764-bf37-e1c61e762403",
     "prev": "7fe8d340-580d-4293-bf70-3058295f0e5f",
     "regions": {
      "6f9bab17-12fc-4ffd-8ac8-aaf4f5b9666b": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "ac2057e4-ca62-4985-af03-04ecaf7c41b5",
        "part": "whole"
       },
       "id": "6f9bab17-12fc-4ffd-8ac8-aaf4f5b9666b"
      }
     }
    },
    "99bfd62c-37d6-4513-975f-7453d45208bf": {
     "id": "99bfd62c-37d6-4513-975f-7453d45208bf",
     "prev": "f2242224-82bb-45d0-948c-e0b58b8456c8",
     "regions": {
      "ec50533b-af5d-4fae-b186-c6dbd88efa04": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "92384d6d-bed6-4e4e-bfc1-ddf88de0963e",
        "part": "whole"
       },
       "id": "ec50533b-af5d-4fae-b186-c6dbd88efa04"
      }
     }
    },
    "a67d3ded-1c1a-43ec-ac9d-5fb0946652c2": {
     "id": "a67d3ded-1c1a-43ec-ac9d-5fb0946652c2",
     "prev": "3c0bfcfd-edb6-4599-814d-85a7bdd827d3",
     "regions": {
      "8b1efa67-24eb-47b1-b579-ec26c1e865ee": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "aae85c45-c89c-401a-9131-01e07651e0af",
        "part": "whole"
       },
       "id": "8b1efa67-24eb-47b1-b579-ec26c1e865ee"
      }
     }
    },
    "af606d45-02cf-4a7e-b91e-76d47fe00c40": {
     "id": "af606d45-02cf-4a7e-b91e-76d47fe00c40",
     "prev": "2b9c3116-58b8-4773-8bb0-1bf8fe05296d",
     "regions": {
      "e2bb3e5e-7372-4f9a-92a5-70d35bbcc850": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "ea824159-e736-4e2a-b024-524e337f6fac",
        "part": "whole"
       },
       "id": "e2bb3e5e-7372-4f9a-92a5-70d35bbcc850"
      }
     }
    },
    "b7abb62f-c11a-4375-a27e-e793c2795a3d": {
     "id": "b7abb62f-c11a-4375-a27e-e793c2795a3d",
     "prev": "131dcdac-6159-44e0-9867-2c6521c536b9",
     "regions": {
      "c41e06a6-86bf-436a-bbd5-25f4733cca6b": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "c14452d1-3454-4704-b59a-e18b8251103b",
        "part": "whole"
       },
       "id": "c41e06a6-86bf-436a-bbd5-25f4733cca6b"
      }
     }
    },
    "b8ef6042-0e70-43fe-a97b-c3c5e98e48a9": {
     "id": "b8ef6042-0e70-43fe-a97b-c3c5e98e48a9",
     "prev": "0eb9ee9c-5399-4988-94f3-8cee78537404",
     "regions": {
      "0393ad1f-f602-477d-9561-b9997b6ae0b1": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "20019a20-ae7f-4beb-8cd4-d68150382b1e",
        "part": "whole"
       },
       "id": "0393ad1f-f602-477d-9561-b9997b6ae0b1"
      }
     }
    },
    "c398a942-e2de-433f-b565-b6c258e5f76f": {
     "id": "c398a942-e2de-433f-b565-b6c258e5f76f",
     "prev": "8f3b675e-7fe6-447e-864b-074941b1619c",
     "regions": {
      "dd42546e-f361-4901-b289-853b870b5ef9": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "77252053-4889-4a38-a497-49465a2e9ac1",
        "part": "whole"
       },
       "id": "dd42546e-f361-4901-b289-853b870b5ef9"
      }
     }
    },
    "c45f3e22-0112-491b-8708-cfb8fd55e67d": {
     "id": "c45f3e22-0112-491b-8708-cfb8fd55e67d",
     "prev": "6b170754-f46e-4f7d-b3af-4fc2d6777247",
     "regions": {
      "d2c1656c-3044-4792-a198-e6148a0f7291": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "defd975d-f087-4747-9b2f-06ccde7ee725",
        "part": "whole"
       },
       "id": "d2c1656c-3044-4792-a198-e6148a0f7291"
      }
     }
    },
    "defa79e9-d30e-49e9-b206-1dcf6736fd0b": {
     "id": "defa79e9-d30e-49e9-b206-1dcf6736fd0b",
     "prev": "5bde732d-d4dd-4c0a-887b-8ecfbe878dbe",
     "regions": {
      "d6201f68-d926-464f-af69-b6358baa3169": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "3d475195-dd8a-4ce5-8e90-f7bcd42008f4",
        "part": "whole"
       },
       "id": "d6201f68-d926-464f-af69-b6358baa3169"
      }
     }
    },
    "df917a16-e756-4db8-ae9a-506e790cf04c": {
     "id": "df917a16-e756-4db8-ae9a-506e790cf04c",
     "prev": "79afa2d5-c641-43ae-8f08-3d64fdd52373",
     "regions": {
      "d9d17a48-00c5-4ef8-839f-11157b81e4d3": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "8f062cb0-836d-47b9-a5fd-a9291431e3dc",
        "part": "whole"
       },
       "id": "d9d17a48-00c5-4ef8-839f-11157b81e4d3"
      }
     }
    },
    "dfb0485f-4165-455b-a321-1368a5593f90": {
     "id": "dfb0485f-4165-455b-a321-1368a5593f90",
     "prev": "af606d45-02cf-4a7e-b91e-76d47fe00c40",
     "regions": {
      "9a34c164-a20e-4e3e-adae-493e61d54955": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "236459a0-db19-4ca3-ab64-d36697e6180f",
        "part": "whole"
       },
       "id": "9a34c164-a20e-4e3e-adae-493e61d54955"
      }
     }
    },
    "f2242224-82bb-45d0-948c-e0b58b8456c8": {
     "id": "f2242224-82bb-45d0-948c-e0b58b8456c8",
     "prev": "a67d3ded-1c1a-43ec-ac9d-5fb0946652c2",
     "regions": {
      "e947732f-650c-4a35-ad0a-869a3ba452fe": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "3fd06df9-1eec-447c-bed7-d88be2d8d4d9",
        "part": "whole"
       },
       "id": "e947732f-650c-4a35-ad0a-869a3ba452fe"
      }
     }
    },
    "f788e7d5-5631-4090-b44e-03ea24aa1887": {
     "id": "f788e7d5-5631-4090-b44e-03ea24aa1887",
     "prev": "10844e3a-543e-48a6-b696-d4dcfd40d55f",
     "regions": {
      "d441b345-210c-492c-8540-b0f3e833269c": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "24af87db-9051-4768-9064-079dc5e9b965",
        "part": "whole"
       },
       "id": "d441b345-210c-492c-8540-b0f3e833269c"
      }
     }
    }
   },
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}