01-01-getting-started-with-tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started With TensorFlow\n",
    "[original documentation](https://www.tensorflow.org/get_started/get_started)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This guide gets you started programming in TensorFlow. Before using this guide, [install TensorFlow](https://www.tensorflow.org/install/). To get the most out of this guide, you should know the following:\n",
    "\n",
    "* How to program in Python.\n",
    "* At least a little bit about arrays.\n",
    "* Ideally, something about machine learning. However, if you know little or nothing about machine learning, then this is still the first guide you should read.\n",
    "\n",
    "TensorFlow provides multiple APIs. The lowest level API--TensorFlow Core-- provides you with complete programming control. We recommend TensorFlow Core for machine learning researchers and others who require fine levels of control over their models. The higher level APIs are built on top of TensorFlow Core. These higher level APIs are typically easier to learn and use than TensorFlow Core. In addition, the higher level APIs make repetitive tasks easier and more consistent between different users. A high-level API like tf.estimator helps you manage data sets, estimators, training and inference.\n",
    "\n",
    "This guide begins with a tutorial on TensorFlow Core. Later, we demonstrate how to implement the same model in tf.estimator. Knowing TensorFlow Core principles will give you a great mental model of how things are working internally when you use the more compact higher level API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The central unit of data in TensorFlow is the **tensor**. A tensor consists of a set of primitive values shaped into an array of any number of dimensions. A tensor's **rank** is its number of dimensions. Here are some examples of tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "3 # a rank 0 tensor; this is a scalar with shape []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.0, 2.0, 3.0]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[1., 2., 3.] # a rank 1 tensor; this is a vector with shape [3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[[1., 2., 3.], [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[[1.0, 2.0, 3.0]], [[7.0, 8.0, 9.0]]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[[[1., 2., 3.]], [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Core tutorial\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The canonical import statement for TensorFlow programs is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives Python access to all of TensorFlow's classes, methods, and symbols. Most of the documentation assumes you have already done this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Computational Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might think of TensorFlow Core programs as consisting of two discrete sections:\n",
    "\n",
    "1. Building the computational graph.\n",
    "2. Running the computational graph.\n",
    "\n",
    "A **computational graph** is a series of TensorFlow operations arranged into a graph of nodes. Let's build a simple computational graph. Each node takes zero or more tensors as inputs and produces a tensor as an output. One type of node is a constant. Like all TensorFlow constants, it takes no inputs, and it outputs a value it stores internally. We can create two floating point Tensors `node1` and `node2` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Const:0\", shape=(), dtype=float32) Tensor(\"Const_1:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "node1 = tf.constant(3.0, dtype=tf.float32)\n",
    "node2 = tf.constant(4.0) # also tf.float32 implicitly\n",
    "print(node1, node2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final print statement produces ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that printing the nodes does not output the values `3.0` and `4.0` as you might expect. Instead, they are nodes that, when evaluated, would produce 3.0 and 4.0, respectively. To actually evaluate the nodes, we must run the computational graph within a **session**. A session encapsulates the control and state of the TensorFlow runtime.\n",
    "\n",
    "The following code creates a `Session` object and then invokes its `run` method to run enough of the computational graph to evaluate `node1` and `node2`. By running the computational graph in a session as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.0, 4.0]\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "print(sess.run([node1, node2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we see the expected values of 3.0 and 4.0: ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can build more complicated computations by combining `Tensor` nodes with operations (Operations are also nodes). For example, we can add our two constant nodes and produce a new graph as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node3: Tensor(\"Add:0\", shape=(), dtype=float32)\n",
      "sess.run(node3): 7.0\n"
     ]
    }
   ],
   "source": [
    "node3 = tf.add(node1, node2)\n",
    "print(\"node3:\", node3)\n",
    "print(\"sess.run(node3):\", sess.run(node3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last two print statements produce ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow provides a utility called TensorBoard that can display a picture of the computational graph. Here is a screenshot showing how TensorBoard visualizes the graph:\n",
    "<img alt=\"TensorBoard screenshot\" src=\"https://www.tensorflow.org/images/getting_started_add.png\">\n",
    "As it stands, this graph is not especially interesting because it always produces a constant result. A graph can be parameterized to accept external inputs, known as **placeholders**. A **placeholder** is a promise to provide a value later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = tf.placeholder(tf.float32)\n",
    "adder_node = a + b  # + provides a shortcut for tf.add(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preceding three lines are a bit like a function or a lambda in which we define two input parameters (a and b) and then an operation on them. We can evaluate this graph with multiple inputs by using the feed_dict argument to the [run method](https://www.tensorflow.org/api_docs/python/tf/Session#run) to feed concrete values to the placeholders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.5\n",
      "[ 3.  7.]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(adder_node, {a: 3, b: 4.5}))\n",
    "print(sess.run(adder_node, {a: [1, 3], b: [2, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "resulting in the output ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorBoard, the graph looks like this:\n",
    "<img alt=\"TensorBoard screenshot\" src=\"https://www.tensorflow.org/images/getting_started_adder.png\">\n",
    "We can make the computational graph more complex by adding another operation. For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22.5\n"
     ]
    }
   ],
   "source": [
    "add_and_triple = adder_node * 3.\n",
    "print(sess.run(add_and_triple, {a: 3, b: 4.5}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "produces the output ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preceding computational graph would look as follows in TensorBoard:\n",
    "<img alt=\"TensorBoard screenshot\" src=\"https://www.tensorflow.org/images/getting_started_triple.png\">\n",
    "In machine learning we will typically want a model that can take arbitrary inputs, such as the one above. To make the model trainable, we need to be able to modify the graph to get new outputs with the same input. **Variables** allow us to add trainable parameters to a graph. They are constructed with a type and initial value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = tf.Variable([.3], dtype=tf.float32)\n",
    "b = tf.Variable([-.3], dtype=tf.float32)\n",
    "x = tf.placeholder(tf.float32)\n",
    "linear_model = W * x + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constants are initialized when you call `tf.constant`, and their value can never change. By contrast, variables are not initialized when you call `tf.Variable`. To initialize all the variables in a TensorFlow program, you must explicitly call a special operation as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to realize `init` is a handle to the TensorFlow sub-graph that initializes all the global variables. Until we call `sess.run`, the variables are uninitialized.\n",
    "\n",
    "Since `x` is a placeholder, we can evaluate `linear_model` for several values of `x` simultaneously as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.          0.30000001  0.60000002  0.90000004]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(linear_model, {x: [1, 2, 3, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to produce the output ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've created a model, but we don't know how good it is yet. To evaluate the model on training data, we need a `y` placeholder to provide the desired values, and we need to write a loss function.\n",
    "\n",
    "A loss function measures how far apart the current model is from the provided data. We'll use a standard loss model for linear regression, which sums the squares of the deltas between the current model and the provided data. `linear_model - y` creates a vector where each element is the corresponding example's error delta. We call `tf.square` to square that error. Then, we sum all the squared errors to create a single scalar that abstracts the error of all examples using `tf.reduce_sum`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23.66\n"
     ]
    }
   ],
   "source": [
    "y = tf.placeholder(tf.float32)\n",
    "squared_deltas = tf.square(linear_model - y)\n",
    "loss = tf.reduce_sum(squared_deltas)\n",
    "print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "producing the loss value ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could improve this manually by reassigning the values of `W` and `b` to the perfect values of -1 and 1. A variable is initialized to the value provided to `tf.Variable` but can be changed using operations like `tf.assign`. For example, `W=-1` and `b=1` are the optimal parameters for our model. We can change `W` and `b` accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n"
     ]
    }
   ],
   "source": [
    "fixW = tf.assign(W, [-1.])\n",
    "fixb = tf.assign(b, [1.])\n",
    "sess.run([fixW, fixb])\n",
    "print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final print shows the loss now is zero. ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We guessed the \"perfect\" values of `W` and `b`, but the whole point of machine learning is to find the correct model parameters automatically. We will show how to accomplish this in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tf.train API\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A complete discussion of machine learning is out of the scope of this tutorial. However, TensorFlow provides **optimizers** that slowly change each variable in order to minimize the loss function. The simplest optimizer is **gradient descent**. It modifies each variable according to the magnitude of the derivative of loss with respect to that variable. In general, computing symbolic derivatives manually is tedious and error-prone. Consequently, TensorFlow can automatically produce derivatives given only a description of the model using the function <mark>tf.gradients</mark>. For simplicity, optimizers typically do this for you. For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "train = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([-0.9999969], dtype=float32), array([ 0.99999082], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "sess.run(init) # reset values to incorrect defaults.\n",
    "for i in range(1000):\n",
    "  sess.run(train, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]})\n",
    "\n",
    "print(sess.run([W, b]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "results in the final model parameters: ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have done actual machine learning! Although doing this simple linear regression doesn't require much TensorFlow core code, more complicated models and methods to feed data into your model necessitate more code. Thus TensorFlow provides higher level abstractions for common patterns, structures, and functionality. We will learn how to use some of these abstractions in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complete program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The completed trainable linear regression model is shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W: [-0.9999969] b: [ 0.99999082] loss: 5.69997e-11\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Model parameters\n",
    "W = tf.Variable([.3], dtype=tf.float32)\n",
    "b = tf.Variable([-.3], dtype=tf.float32)\n",
    "# Model input and output\n",
    "x = tf.placeholder(tf.float32)\n",
    "linear_model = W * x + b\n",
    "y = tf.placeholder(tf.float32)\n",
    "\n",
    "# loss\n",
    "loss = tf.reduce_sum(tf.square(linear_model - y)) # sum of the squares\n",
    "# optimizer\n",
    "optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "train = optimizer.minimize(loss)\n",
    "\n",
    "# training data\n",
    "x_train = [1, 2, 3, 4]\n",
    "y_train = [0, -1, -2, -3]\n",
    "# training loop\n",
    "init = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(init) # reset values to wrong\n",
    "for i in range(1000):\n",
    "  sess.run(train, {x: x_train, y: y_train})\n",
    "\n",
    "# evaluate training accuracy\n",
    "curr_W, curr_b, curr_loss = sess.run([W, b, loss], {x: x_train, y: y_train})\n",
    "print(\"W: %s b: %s loss: %s\"%(curr_W, curr_b, curr_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When run, it produces ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the loss is a very small number (very close to zero). If you run this program, your loss may not be the exact same because the model is initialized with pseudorandom values.\n",
    "\n",
    "This more complicated program can still be visualized in TensorBoard\n",
    "<img alt=\"TensorBoard final model visualization\" src=\"https://www.tensorflow.org/images/getting_started_final.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tf.estimator\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>tf.estimator</mark> is a high-level TensorFlow library that simplifies the mechanics of machine learning, including the following:\n",
    "\n",
    "* running training loops\n",
    "* running evaluation loops\n",
    "* managing data sets\n",
    "\n",
    "tf.estimator defines many common models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how much simpler the linear regression program becomes with `tf.estimator`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp8qi36xc0\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp8qi36xc0', '_tf_random_seed': 1, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100}\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /tmp/tmp8qi36xc0/model.ckpt.\n",
      "INFO:tensorflow:loss = 14.0, step = 1\n",
      "INFO:tensorflow:global_step/sec: 1157.28\n",
      "INFO:tensorflow:loss = 0.205698, step = 101 (0.087 sec)\n",
      "INFO:tensorflow:global_step/sec: 1327.62\n",
      "INFO:tensorflow:loss = 0.0457045, step = 201 (0.076 sec)\n",
      "INFO:tensorflow:global_step/sec: 1200.17\n",
      "INFO:tensorflow:loss = 0.00413012, step = 301 (0.083 sec)\n",
      "INFO:tensorflow:global_step/sec: 1271.2\n",
      "INFO:tensorflow:loss = 0.000429388, step = 401 (0.079 sec)\n",
      "INFO:tensorflow:global_step/sec: 1163.81\n",
      "INFO:tensorflow:loss = 0.000361711, step = 501 (0.086 sec)\n",
      "INFO:tensorflow:global_step/sec: 1274.13\n",
      "INFO:tensorflow:loss = 2.80052e-05, step = 601 (0.079 sec)\n",
      "INFO:tensorflow:global_step/sec: 1300.4\n",
      "INFO:tensorflow:loss = 5.50731e-06, step = 701 (0.077 sec)\n",
      "INFO:tensorflow:global_step/sec: 1276.19\n",
      "INFO:tensorflow:loss = 1.48202e-06, step = 801 (0.079 sec)\n",
      "INFO:tensorflow:global_step/sec: 1289.04\n",
      "INFO:tensorflow:loss = 1.08629e-07, step = 901 (0.077 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmp8qi36xc0/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 4.24986e-08.\n",
      "INFO:tensorflow:Starting evaluation at 2017-09-26-05:57:52\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmp8qi36xc0/model.ckpt-1000\n",
      "INFO:tensorflow:Finished evaluation at 2017-09-26-05:57:53\n",
      "INFO:tensorflow:Saving dict for global step 1000: average_loss = 1.06284e-08, global_step = 1000, loss = 4.25137e-08\n",
      "INFO:tensorflow:Starting evaluation at 2017-09-26-05:57:53\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmp8qi36xc0/model.ckpt-1000\n",
      "INFO:tensorflow:Finished evaluation at 2017-09-26-05:57:53\n",
      "INFO:tensorflow:Saving dict for global step 1000: average_loss = 0.00253494, global_step = 1000, loss = 0.0101398\n",
      "train metrics: {'average_loss': 1.0628435e-08, 'loss': 4.251374e-08, 'global_step': 1000}\n",
      "eval metrics: {'average_loss': 0.0025349427, 'loss': 0.010139771, 'global_step': 1000}\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "# NumPy is often used to load, manipulate and preprocess data.\n",
    "import numpy as np\n",
    "\n",
    "# Declare list of features. We only have one numeric feature. There are many\n",
    "# other types of columns that are more complicated and useful.\n",
    "feature_columns = [tf.feature_column.numeric_column(\"x\", shape=[1])]\n",
    "\n",
    "# An estimator is the front end to invoke training (fitting) and evaluation\n",
    "# (inference). There are many predefined types like linear regression,\n",
    "# linear classification, and many neural network classifiers and regressors.\n",
    "# The following code provides an estimator that does linear regression.\n",
    "estimator = tf.estimator.LinearRegressor(feature_columns=feature_columns)\n",
    "\n",
    "# TensorFlow provides many helper methods to read and set up data sets.\n",
    "# Here we use two data sets: one for training and one for evaluation\n",
    "# We have to tell the function how many batches\n",
    "# of data (num_epochs) we want and how big each batch should be.\n",
    "x_train = np.array([1., 2., 3., 4.])\n",
    "y_train = np.array([0., -1., -2., -3.])\n",
    "x_eval = np.array([2., 5., 8., 1.])\n",
    "y_eval = np.array([-1.01, -4.1, -7, 0.])\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    {\"x\": x_train}, y_train, batch_size=4, num_epochs=None, shuffle=True)\n",
    "train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    {\"x\": x_train}, y_train, batch_size=4, num_epochs=1000, shuffle=False)\n",
    "eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    {\"x\": x_eval}, y_eval, batch_size=4, num_epochs=1000, shuffle=False)\n",
    "\n",
    "# We can invoke 1000 training steps by invoking the  method and passing the\n",
    "# training data set.\n",
    "estimator.train(input_fn=input_fn, steps=1000)\n",
    "\n",
    "# Here we evaluate how well our model did.\n",
    "train_metrics = estimator.evaluate(input_fn=train_input_fn)\n",
    "eval_metrics = estimator.evaluate(input_fn=eval_input_fn)\n",
    "print(\"train metrics: %r\"% train_metrics)\n",
    "print(\"eval metrics: %r\"% eval_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When run, it produces ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how our eval data has a higher loss, but it is still close to zero. That means we are learning properly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A custom model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tf.estimator` does not lock you into its predefined models. Suppose we wanted to create a custom model that is not built into TensorFlow. We can still retain the high level abstraction of data set, feeding, training, etc. of `tf.estimator`. For illustration, we will show how to implement our own equivalent model to `LinearRegressor` using our knowledge of the lower level TensorFlow API.\n",
    "\n",
    "To define a custom model that works with `tf.estimator`, we need to use `tf.estimator.Estimator`. `tf.estimator.LinearRegressor` is actually a sub-class of `tf.estimator.Estimator`. Instead of sub-classing `Estimator`, we simply provide `Estimator` a function `model_fn` that tells `tf.estimator` how it can evaluate predictions, training steps, and loss. The code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp2lwd0bu1\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp2lwd0bu1', '_tf_random_seed': 1, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100}\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /tmp/tmp2lwd0bu1/model.ckpt.\n",
      "INFO:tensorflow:loss = 8.66328258808, step = 1\n",
      "INFO:tensorflow:global_step/sec: 1357.72\n",
      "INFO:tensorflow:loss = 0.0107467976395, step = 101 (0.076 sec)\n",
      "INFO:tensorflow:global_step/sec: 1303.25\n",
      "INFO:tensorflow:loss = 0.00471162578938, step = 201 (0.078 sec)\n",
      "INFO:tensorflow:global_step/sec: 1314.9\n",
      "INFO:tensorflow:loss = 0.000431704739591, step = 301 (0.074 sec)\n",
      "INFO:tensorflow:global_step/sec: 1271.68\n",
      "INFO:tensorflow:loss = 1.64768906488e-05, step = 401 (0.079 sec)\n",
      "INFO:tensorflow:global_step/sec: 1231.27\n",
      "INFO:tensorflow:loss = 8.51231604409e-07, step = 501 (0.081 sec)\n",
      "INFO:tensorflow:global_step/sec: 1419.14\n",
      "INFO:tensorflow:loss = 2.79574175755e-07, step = 601 (0.070 sec)\n",
      "INFO:tensorflow:global_step/sec: 1285.15\n",
      "INFO:tensorflow:loss = 1.26902333109e-08, step = 701 (0.079 sec)\n",
      "INFO:tensorflow:global_step/sec: 1200.25\n",
      "INFO:tensorflow:loss = 2.14921581521e-09, step = 801 (0.082 sec)\n",
      "INFO:tensorflow:global_step/sec: 1317.22\n",
      "INFO:tensorflow:loss = 1.11960942995e-10, step = 901 (0.076 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/tmp2lwd0bu1/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 1.37598517241e-11.\n",
      "INFO:tensorflow:Starting evaluation at 2017-09-26-05:59:49\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmp2lwd0bu1/model.ckpt-1000\n",
      "INFO:tensorflow:Finished evaluation at 2017-09-26-05:59:49\n",
      "INFO:tensorflow:Saving dict for global step 1000: global_step = 1000, loss = 1.48093e-11\n",
      "INFO:tensorflow:Starting evaluation at 2017-09-26-05:59:49\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tmp2lwd0bu1/model.ckpt-1000\n",
      "INFO:tensorflow:Finished evaluation at 2017-09-26-05:59:50\n",
      "INFO:tensorflow:Saving dict for global step 1000: global_step = 1000, loss = 0.0101009\n",
      "train metrics: {'loss': 1.4809274e-11, 'global_step': 1000}\n",
      "eval metrics: {'loss': 0.010100947, 'global_step': 1000}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# Declare list of features, we only have one real-valued feature\n",
    "def model_fn(features, labels, mode):\n",
    "  # Build a linear model and predict values\n",
    "  W = tf.get_variable(\"W\", [1], dtype=tf.float64)\n",
    "  b = tf.get_variable(\"b\", [1], dtype=tf.float64)\n",
    "  y = W * features['x'] + b\n",
    "  # Loss sub-graph\n",
    "  loss = tf.reduce_sum(tf.square(y - labels))\n",
    "  # Training sub-graph\n",
    "  global_step = tf.train.get_global_step()\n",
    "  optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "  train = tf.group(optimizer.minimize(loss),\n",
    "                   tf.assign_add(global_step, 1))\n",
    "  # EstimatorSpec connects subgraphs we built to the\n",
    "  # appropriate functionality.\n",
    "  return tf.estimator.EstimatorSpec(\n",
    "      mode=mode,\n",
    "      predictions=y,\n",
    "      loss=loss,\n",
    "      train_op=train)\n",
    "\n",
    "estimator = tf.estimator.Estimator(model_fn=model_fn)\n",
    "# define our data sets\n",
    "x_train = np.array([1., 2., 3., 4.])\n",
    "y_train = np.array([0., -1., -2., -3.])\n",
    "x_eval = np.array([2., 5., 8., 1.])\n",
    "y_eval = np.array([-1.01, -4.1, -7, 0.])\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    {\"x\": x_train}, y_train, batch_size=4, num_epochs=None, shuffle=True)\n",
    "train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    {\"x\": x_train}, y_train, batch_size=4, num_epochs=1000, shuffle=False)\n",
    "eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    {\"x\": x_eval}, y_eval, batch_size=4, num_epochs=1000, shuffle=False)\n",
    "\n",
    "# train\n",
    "estimator.train(input_fn=input_fn, steps=1000)\n",
    "# Here we evaluate how well our model did.\n",
    "train_metrics = estimator.evaluate(input_fn=train_input_fn)\n",
    "eval_metrics = estimator.evaluate(input_fn=eval_input_fn)\n",
    "print(\"train metrics: %r\"% train_metrics)\n",
    "print(\"eval metrics: %r\"% eval_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When run, it produces ↑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the contents of the custom `model_fn()` function are very similar to our manual model training loop from the lower level API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have a working knowledge of the basics of TensorFlow. We have several more tutorials that you can look at to learn more. If you are a beginner in machine learning see [MNIST for beginners](https://www.tensorflow.org/get_started/mnist/beginners), otherwise see [Deep MNIST for experts](https://www.tensorflow.org/get_started/mnist/pros).\n",
    "<hr/>\n",
    "Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.\n",
    "\n",
    "Last updated 8월 17, 2017."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}