1_TensorFlow_Basics/Tutorials/1_Graph_and_Session.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(You can also read this article on our website, [easy-tensorFlow](http://www.easy-tensorflow.com/basics/graph-and-session))\n",
    "\n",
    "Why do we need tensorflow? Why are people crazy about it? In a way, it is lazy computing and offers flexibility in the way you run your code. What is this thing with flexbility and laze computing? We are glad, you asked!\n",
    "\n",
    "Lazy Computing: TensorFlow is a way of representing computation without actually performing it until asked. The first step to learn Tensorflow is to understand its main key feature, the __\"computational graph\"__ approach. Basically, all Tensorflow codes contain two important parts:\n",
    "\n",
    "__Part 1:__ building the __GRAPH__, it represents the data flow of the computations\n",
    "\n",
    "__Part 2:__ running a __SESSION__, it executes the operations in the graph\n",
    "\n",
    "First step you create the graph i.e. what you want to do with the data, then you run it seperately using a session (don't struggle to wrap your head around it, it will come to you eventually). \n",
    "\n",
    "Flexibility: When you create a graph, you are not bound to run the whole graph and can control the parts of the graph that are executed separately. This provides a huge flexibility with your models. \n",
    "\n",
    "Bonus: One of the biggest advantages of TensorFlow is its visualizations of the computation graph. Its called TensorBoard and will be discussed in future. Now that we have discussed what and why about TensorFlow, lets dive in to the actual thing.\n",
    "\n",
    "TensorFlow separates the definition of computations from their execution. These two parts are explained in more detail in the following sections. Before that, remember that the first step is to import the Tensorflow library!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives Python access to all of TensorFlow's classes, methods, and symbols. Using this command, TensorFlow library will be imported under the alias __tf__ so that later we can use it instead of typing the whole term __tensorflow__ each time.\n",
    "\n",
    "__What is a Tensor?__\n",
    "TensorFlow programs use a data structure called tensor to represent all the data. Any type of data you plan to use for your model can be stored in Tensors. Simply put, a Tensor is a multi-dimensional array (0-D tensor: scalar, 1-D tensor: vector, 2-D tensor: matrix, and so on). Hence, TensorFlow is simply referring to the flow of the Tensors in the computational graph.\n",
    "<img src=\"files/files/1_1.gif\">\n",
    "___Fig1. ___ A sample computational graph in TensorFlow (Source: TensorFlow website)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GRAPH\n",
    "The biggest idea about Tensorflow is that all the numerical computations are expressed as a computational graph. In other words, the backbone of any Tensorflow program is a __Graph__. Anything that happens in your model is represented by the computational graph. This makes it, the to go place for anything related to your model. Quoted from the TensorFlow website, \"A __computational graph__ (or graph in short) is a series of TensorFlow operations arranged into a graph of nodes\". Basically, it means a graph is just an arrangement of nodes that represent the operations in your model. \n",
    "\n",
    "So First let's see what does a node and operation mean? The best way to explain it is by looking at a simple example. Suppose we want to write the code for function $f(x,y)=x^2y+y+2$. The Graph in TensorFlow will be something like:\n",
    "<img src=\"files/files/1_2.png\" width=\"500\" height=\"1000\" >\n",
    "\n",
    "___Fig2. ___ Schematic of the constructed computational graph in TensorFlow\n",
    "\n",
    "The graph is composed of a series of nodes connected to each other by edges (from the image above). Each __node__ in the graph is called __op__ (short for operation). So we'll have one node for each operation; either for operations on tensors (like math operations) or generating tensors (like variables and constants). Each node takes zero or more tensors as inputs and produces a tensor as an output.\n",
    "\n",
    "Now Let's build a simple computational graph.\n",
    "\n",
    "\n",
    "### Example 1:\n",
    "\n",
    "Let's start with a basic arithmatic operation like addition to demonstrate a graph. The code adds two values, say a=2 and b=3, using TensorFlow. To do so, we need to call __tf.add()__. From here on, we recommend you to check out the documentation of each method/class to get a clear idea of what it can do(documentation can be found at tensorflow.org or you can just use google to get to the required page in the documentation). The __tf.add()__ has three arugments 'x', 'y', and 'name' where x and y are the values to be added together and __name__ is the operation name, i.e. the name associated to the addition node on the graph.\n",
    "\n",
    "If we call the operation __\"Add\"__, the code will be as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Add:0\", shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "a = 2\n",
    "b = 3\n",
    "c = tf.add(a, b, name='Add')\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated graph and variables are:\n",
    "\n",
    "__*Note__: The graph is generated using __Tensorboard__. As discussed earlier, it is a visualization tool for the graph and will be discussed in detail in future.\n",
    "<img src=\"files/files/1_3.png\" width=\"800\" height=\"1500\">\n",
    "\n",
    "___Fig3. ___ __Left:__ generated graph visualized in Tensorboard, __Right:__ generated variables (screenshot captured from PyCharm debugger when running in debug mode)\n",
    "\n",
    "This code creates two input nodes (for inputs a=2 and b=3) and one output node for the addition operation (named Add). When we print out the variable __c__ (i.e. the output Tensor of the addition operation), it prints out the Tensor information; its name (Add), shape (__()__ means scalar), and type (32-bit integer). However, It does not spit out the result (2+3=5). Why?!\n",
    "\n",
    "Remember earlier in this post, we talked about the two parts of a TensorFlow code. First step is to create a graph and to actually evaluate the nodes, we must run the computational graph within a __Session__. In simple words, the written code only generates the graph which only determines the expected sizes of Tensors and operations to be executed on them. However, it doesn't assign a numeric value to any of the Tensors i.e. TensorFlow does not execute the graph unless it is specified to do so with a session. Hence, to assign these values and make them flow through the graph, we need to create and run a session.\n",
    "\n",
    "Therefore a TensorFlow Graph is something like a function definition in Python. It __WILL NOT__ do any computation for you (just like a function definition will not have any execution result). It __ONLY__ defines computation operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Session\n",
    "To compute anything, a graph must be launched in a session. Technically, session places the graph ops on hardware such as CPUs or GPUs and provides methods to execute them. In our example, to run the graph and get the value for c the following code will create a session and execute the graph by running 'c':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "print(sess.run(c))\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code creates a Session object (assigned to __sess__), and then (the second line) invokes its run method to run enough of the computational graph to evaluate __c__. This means that it only runs that part of the graph which is necessary to get the value of __c__ (remember the flexibility of using TensorFlow? In this simple example, it runs the whole graph). Remember to close the session at the end of the session. That is done using the last line in the above code. \n",
    "\n",
    "The following code does the same thing and is more commonly used. The only difference is that there is no need to close the session at the end as it gets closed automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at the created graph one more time. Don't you see anything weird?\n",
    "<img src=\"files/files/1_4.png\" width=\"500\" height=\"1000\">\n",
    "\n",
    "___Fig4. ___ The generated graph visualized by Tensorboard\n",
    "\n",
    "\n",
    "Exactly! What is x and y?! Where did these two thing come from? We didn't define any x or y variables!\n",
    "\n",
    "Well... To explain clearly, let's make up two names; say __\"Python-name\"__ and __\"TensorFlow-name\"__. In this piece of code, we generated 3 variables (look at the right panel of Fig. 3) with __\"Python-name\"__s of _a_, _b_, and _c_. Here, _a_ and _b_ are Python variables, thus have no __\"TensorFlow-name\"__; while _c_ is a Tensor with ___Add___ as its __\"TensorFlow-name\"__. \n",
    "\n",
    "Clear? Okay, let's get back to our question, what is x and y then?\n",
    "\n",
    "In an ideal Tensorflow case, __tf.add()__ receives two __Tensors__ with defined __\"TensorFlow-name\"__ as input (these names are separate from __Python-name__). For example, by writing $c = tf.add(a, b, name='Add')$, we're actually creating a variable (or Tensor) with __c__ as its Python-name and __Add__ as the TensorFlow-name. \n",
    "\n",
    "In the above code, we passed two Python variables (a=2 and b=3) which only have Python-names (a and b), but they have no TensorFlow-names. TensorFlow uses the TensorFlow-names for visualizing the graphs. Since a and b have no TensorFlow-names, it uses some default names, x and y. \n",
    "\n",
    "__*Note:__ This name mismatch can easily be solved by using tf.constant() for creating the input variables as Tensors instead of simply using Python variables (a=2, b=3). This is explained thoroughly in the next tutorial where we talk about TensorFlow DataTypes. \n",
    "\n",
    "For now, we'll continue using Python variables and change the Python variable names __a__ and __b__ into __x__ and __y__ to solve the name mismatch temporarily. \n",
    "\n",
    "Now let's look at a more complicated example.\n",
    "\n",
    "### Example 2:\n",
    "Creating a graph with multiple math operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "x = 2\n",
    "y = 3\n",
    "add_op = tf.add(x, y, name='Add')\n",
    "mul_op = tf.multiply(x, y, name='Multiply')\n",
    "pow_op = tf.pow(add_op, mul_op, name='Power')\n",
    "useless_op = tf.multiply(x, add_op, name='Useless')\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    pow_out, useless_out = sess.run([pow_op, useless_op])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The created graph and the defined variables (Tensors and Python variables) are:\n",
    "<img src=\"files/files/1_5.png\" width=\"1000\" height=\"2000\">\n",
    "\n",
    "___Fig5. ___ __Left:__ generated graph visualized in Tensorboard, __Right:__ generated variables (screenshot captured from PyCharm debugger when running in debug mode)\n",
    "\n",
    "\n",
    "I called one of the operations useless_op because it's output is not used by other operations. Lets talk about an __IMPORTANT__ point. Given this graph, if we fetch the __pow_op__ operation, it will first run the __add_op__ and __mul_op__ to get their output tensor and then run __pow_op__ on them to compute the required output value. In other words __useless_op__ will not be executed as it's output tensor is not used in executing the __pow_op__ operation. \n",
    "\n",
    "__This is one of the advantages of defining a graph and running a session on it! It helps running only the required operations of the graph and skip the rest (remember flexibility). This specially saves a significant amount of time for us when dealing with huge networks with hundreds and thousands of operations.__\n",
    "\n",
    "In the above code, in the defined session, we're fetching the value of two tensors (i.e. output tensors of __pow_op__ and __useless_op__) at the same time. This will run the whole graph to get the required output tensors.\n",
    "\n",
    "I hope this post has helped you to understand the concept of __Graph__ and __Session__ in TensorFlow. Thank you so much for reading! If you have any questions, feel free to leave a comment in our [webpage](http://www.easy-tensorflow.com/basics/graph-and-session). You can also send us feedback through the [__contacts__](http://www.easy-tensorflow.com/contacts) page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}