20_Convolutional_neural_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional neural network\n",
    "\n",
    "Convolutional neural network, also known as *convnets*, is a well-known\n",
    "method in computer vision applications. This type of architecture is\n",
    "dominant to recognize objects from a picture or video.\n",
    "\n",
    "In this tutorial, you will learn how to construct a convnet and how to\n",
    "use TensorFlow to solve the handwritten dataset.\n",
    "\n",
    "## The architecture of a Convolutional Neural Network\n",
    "\n",
    "Think about Facebook a few years ago, after you uploaded a picture to\n",
    "your profile, you were asked to add a name to the face on the picture\n",
    "manually. Nowadays, Facebook uses convnet to tag your friend in the\n",
    "picture automatically.\n",
    "\n",
    "A convolutional neural network is not very difficult to understand. An\n",
    "input image is processed during the convolution phase and later\n",
    "attributed a label.\n",
    "\n",
    "A typical convnet architecture can be summarized in the picture below.\n",
    "First of all, an image is pushed to the network; this is called the\n",
    "input image. Then, the input image goes through an infinite number of\n",
    "steps; this is the convolutional part of the network. Finally, the\n",
    "neural network can predict the digit on the image.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image001.png)\n",
    "\n",
    "An image is composed of an array of pixels with height and width. A\n",
    "grayscale image has only one channel while the color image has three\n",
    "channels (each one for Red, Green, and Blue). A channel is stacked over\n",
    "each other. In this tutorial, you will use a grayscale image with only\n",
    "one channel. Each pixel has a value from 0 to 255 to reflect the\n",
    "intensity of the color. For instance, a pixel equals to 0 will show a\n",
    "white color while pixel with a value close to 255 will be darker.\n",
    "\n",
    "Let’s have a look of an image stored in the [MNIST\n",
    "dataset](http://yann.lecun.com/exdb/mnist/). The picture below shows how\n",
    "to represent the picture of the left in a matrix format. Note that, the\n",
    "original matrix has been standardized to be between 0 and 1. For darker\n",
    "color, the value in the matrix is about 0.9 while white pixels have a\n",
    "value of 0.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image002.png)\n",
    "\n",
    "## Convolutional operation\n",
    "\n",
    "The most critical component in the model is the convolutional layer.\n",
    "This part aims at reducing the size of the image for faster computations\n",
    "of the weights and improve its generalization.\n",
    "\n",
    "During the convolutional part, the network keeps the essential features\n",
    "of the image and excludes irrelevant noise. For instance, the model is\n",
    "learning how to recognize an elephant from a picture with a mountain in\n",
    "the background. If you use a traditional neural network, the model will\n",
    "assign a weight to all the pixels, including those from the mountain\n",
    "which is not essential and can mislead the network.\n",
    "\n",
    "Instead, a convolutional neural network will use a mathematical\n",
    "technique to extract only the most relevant pixels. This mathematical\n",
    "operation is called convolution. This technique allows the network to\n",
    "learn increasingly complex features at each layer. The convolution\n",
    "divides the matrix into small pieces to learn to most essential elements\n",
    "within each piece.\n",
    "\n",
    "In every convnets, there are four components:\n",
    "\n",
    "1.  Convolution\n",
    "\n",
    "2.  Non Linearity (ReLU)\n",
    "\n",
    "3.  Pooling or Sub Sampling\n",
    "\n",
    "4.  Classification (Fully Connected Layer)\n",
    "\n",
    "-   Convolution\n",
    "\n",
    "The purpose of the convolution is to extract the features of the object\n",
    "on the image locally. It means the network will learn specific patterns\n",
    "within the picture and will be able to recognize it everywhere in the\n",
    "picture.\n",
    "\n",
    "Convolution is an element-wise multiplication. The concept is easy to\n",
    "understand. The computer will scan a part of the image, usually with a\n",
    "dimension of 3x3 and multiplies it to a filter. The output of the\n",
    "element-wise multiplication is called a feature map. This step is\n",
    "repeated until all the image is scanned. Note that, after the\n",
    "convolution, the size of the image is reduced.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image003.png)\n",
    "\n",
    "Below, there is a URL to see in action how convolution works.\n",
    "\n",
    "![](https://media.giphy.com/media/fV8esV6419OdEltpbO/giphy.gif)\n",
    "\n",
    "There are numerous channels available. Below, we listed some of the\n",
    "channels. You can see that each filter has a specific purpose. Note, in\n",
    "the picture below; the Kernel is a synonym of the filter.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image005.png)\n",
    "[Source](https://en.wikipedia.org/wiki/Kernel_(image_processing))\n",
    "\n",
    "*Arithmetic behind the convolution*\n",
    "\n",
    "The convolutional phase will apply the filter on a small array of pixels\n",
    "within the picture. The filter will move along the input image with a\n",
    "general shape of 3x3 or 5x5. It means the network will slide these\n",
    "windows across all the input image and compute the convolution. The\n",
    "image below shows how the convolution operates. The size of the patch is\n",
    "3x3, and the output matrix is the result of the element-wise operation\n",
    "between the image matrix and the filter.\n",
    "\n",
    "![](http://machinelearninguru.com/_images/topics/computer_vision/basics/convolutional_layer_1/stride1.gif)\n",
    "\n",
    "\n",
    "\n",
    "[Source](http://machinelearninguru.com/computer_vision/basics/convolution/convolution_layer.html)\n",
    "\n",
    "You notice that the width and height of the output can be different from\n",
    "the width and height of the input. It happens because of the border\n",
    "effect.\n",
    "\n",
    "**Border effect**\n",
    "\n",
    "Image has a 5x5 features map and a 3x3 filter. There is only one window\n",
    "in the center where the filter can screen an 3x3 grid. The output\n",
    "feature map will shrink by two tiles alongside with a 3x3 dimension.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image007.png)\n",
    "\n",
    "To get the same output dimension as the input dimension, you need to add\n",
    "padding. Padding consists of adding the right number of rows and columns\n",
    "on each side of the matrix. It will allow the convolution to center fit\n",
    "every input tile. In the image below, the input/output matrix have the\n",
    "same dimension 5x5\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image008.png)\n",
    "\n",
    "When you define the network, the convolved features are controlled by\n",
    "three parameters:\n",
    "\n",
    "1.  Depth: It defines the number of filters to apply during the\n",
    "    convolution. In the previous example, you saw a depth of 1, meaning\n",
    "    only one filter is used. In most of the case, there is more than one\n",
    "    filter. The picture below shows the operations done in a situation\n",
    "    with three filters\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image009.gif)\n",
    "\n",
    "1.  Stride: It defines the number of \"pixel's jump\" between two slices.\n",
    "    If the stride is equal to 1, the windows will move with a pixel's\n",
    "    spread of one. If the stride is equal to two, the windows will jump\n",
    "    by 2 pixels. If you increase the stride, you will have smaller\n",
    "    feature maps.\n",
    "\n",
    "Example stride 1\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image010.png)\n",
    "\n",
    "Image stride 2\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image011.png)\n",
    "\n",
    "1.  Zero-padding: A padding is an operation of adding a corresponding\n",
    "    number of rows and column on each side of the input features maps.\n",
    "    In this case, the output has the same dimension as the input.\n",
    "\n",
    "-   Non Linearity (ReLU)\n",
    "\n",
    "At the end of the convolution operation, the output is subject to an\n",
    "activation function to allow non-linearity. The usual activation\n",
    "function for convnet is the Relu. All the pixel with a negative value\n",
    "will be replaced by zero.\n",
    "\n",
    "-   Max-pooling operation\n",
    "\n",
    "This step is easy to understand. The purpose of the pooling is to reduce\n",
    "the dimensionality of the input image. The steps are done to reduce the\n",
    "computational complexity of the operation. By diminishing the\n",
    "dimensionality, the network has lower weights to compute, so it prevents\n",
    "overfitting.\n",
    "\n",
    "In this stage, you need to define the size and the stride. A standard\n",
    "way to pool the input image is to use the maximum value of the feature\n",
    "map. Look at the picture below. The \"pooling\" will screen a four\n",
    "submatrix of the 4x4 feature map and return the maximum value. The\n",
    "pooling takes the maximum value of a 2x2 array and then move this\n",
    "windows by two pixels. For instance, the first sub-matrix is\n",
    "\\[3,1,3,2\\], the pooling will return the maximum, which is 3.\n",
    "\n",
    "![](https://github.com/thomaspernet/Tensorflow/blob/master/tensorflow/20_CNN_v7_files/image012.png)\n",
    "\n",
    "There is another pooling operation such as the mean.\n",
    "\n",
    "This operation aggressively reduces the size of the feature map\n",
    "\n",
    "-   Fully connected layers\n",
    "\n",
    "The last step consists of building a traditional artificial neural\n",
    "network as you did in the previous tutorial. You connect all neurons\n",
    "from the previous layer to the next layer. You use a softmax activation\n",
    "function to classify the number on the input image.\n",
    "\n",
    "**Recap:** \n",
    "\n",
    "Convolutional Neural network compiles different layers before making a\n",
    "prediction. A neural network has:\n",
    "\n",
    "-   A convolutional layer\n",
    "\n",
    "-   Relu Activation function\n",
    "\n",
    "-   Pooling layer\n",
    "\n",
    "-   Densely connected layer\n",
    "\n",
    "The convolutional layers apply different filters on a subregion of the\n",
    "picture. The Relu activation function adds non-linearity, and the\n",
    "pooling layers reduce the dimensionality of the features maps.\n",
    "\n",
    "All these layers extract essential information from the images. At last,\n",
    "the features map are feed to a primary fully connected layer with a\n",
    "softmax function to make a prediction.\n",
    "\n",
    "# Train CNN with TensorFlow\n",
    "\n",
    "Now that you are familiar with the building block of a convnets, you are\n",
    "ready to build one with TensorFlow. We will use the MNIST dataset.\n",
    "\n",
    "The data preparation is the same as the previous tutorial. You can run\n",
    "the codes and jump directly to the architecture of the CNN.\n",
    "\n",
    "You will follow the steps below:\n",
    "\n",
    "Step 1: Upload Dataset\n",
    "\n",
    "Step 2: Input layer\n",
    "\n",
    "Step 3: Convolutional layer\n",
    "\n",
    "Step 4: Pooling layer\n",
    "\n",
    "Step 5: Second Convolutional Layer and Pooling Layer\n",
    "\n",
    "Step 6: Dense layer\n",
    "\n",
    "Step 7: Logit Layer\n",
    "\n",
    "\n",
    "**Step 1**: Upload Dataset\n",
    "\n",
    "The MNIST dataset is available at this [URL](https://www.dropbox.com/sh/jm9jo0d58oggeb9/AAAZrRHvHFGYdCHssXpEH2o1a?dl=0).\n",
    "\n",
    "Please download it and store it in Downloads. You can upload it with `fetch_mldata('MNIST original')`.\n",
    "\n",
    "**Create a train/test set**\n",
    "\n",
    "You need to split the dataset with `train_test_split\n",
    "\n",
    "**Scale the features**\n",
    "\n",
    "Finally, you can scale the feature with MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Thomas/anaconda3/envs/hello-tf/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "from sklearn.datasets import fetch_mldata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are a Windows user, run the following lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "#Change USERNAME by the username of your machine\n",
    "## Windows USER\n",
    "mnist = fetch_mldata('C:\\\\Users\\\\USERNAME\\\\Downloads\\\\MNIST original')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otherwise, you need to run this line\n",
    "```\n",
    "mnist = fetch_mldata('/Users/USERNAME/Downloads/MNIST original')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(70000, 784)\n",
      "(70000,)\n"
     ]
    }
   ],
   "source": [
    "## Mac User\n",
    "\n",
    "mnist = fetch_mldata('/Users/Thomas/Downloads/MNIST original')\n",
    "print(mnist.data.shape)\n",
    "print(mnist.target.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You split the dataset with 80 percent training and 20 percent testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(56000, 784) (56000,) (14000,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(mnist.data,\n",
    "                                                    mnist.target,\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=42)\n",
    "y_train  = y_train.astype(int)\n",
    "y_test  = y_test.astype(int)\n",
    "batch_size =len(X_train)\n",
    "\n",
    "print(X_train.shape, y_train.shape,y_test.shape )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finaly, you scale the data using the min/max scaler of scikit learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(784,)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## resclae\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "# Train\n",
    "X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))\n",
    "# test\n",
    "X_test_scaled = scaler.fit_transform(X_test.astype(np.float64))\n",
    "\n",
    "feature_columns = [\n",
    "      tf.feature_column.numeric_column('x', shape=X_train_scaled.shape[1:])]\n",
    "\n",
    "X_train_scaled.shape[1:]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Define the CNN**\n",
    "\n",
    "A CNN uses filters on the raw pixel of an image to learn details pattern\n",
    "compare to global pattern with a traditional neural net. To construct a\n",
    "CNN, you need to define:\n",
    "\n",
    "1. A convolutional layer: Apply n number of filters to the feature\n",
    "   map. After the convolution, you need to use a Relu activation\n",
    "   function to add non-linearity to the network.\n",
    "2. Pooling layer: The next step after the convolution is to downsample\n",
    "   the feature max. The purpose is to reduce the dimensionality of the\n",
    "   feature map to prevent overfitting and improve the computation\n",
    "   speed. Max pooling is the conventional technique, which divides the\n",
    "   feature maps into subregions (usually with a 2x2 size) and keeps\n",
    "   only the maximum values.\n",
    "3. Fully connected layers: All neurons from the previous layers are\n",
    "   connected to the next layers. The CNN will classify the label\n",
    "   according to the features from the convolutional layers and reduced\n",
    "   with the pooling layer.\n",
    "\n",
    "CNN architecture\n",
    "\n",
    "- Convolutional Layer: Applies 14 5x5 filters (extracting 5x5-pixel\n",
    "  subregions), with ReLU activation function\n",
    "- Pooling Layer: Performs max pooling with a 2x2 filter and stride of\n",
    "  2 (which specifies that pooled regions do not overlap)\n",
    "- Convolutional Layer: Applies 36 5x5 filters, with ReLU activation\n",
    "  function\n",
    "- Pooling Layer 2: Again, performs max pooling with a 2x2 filter and\n",
    "  stride of 2\n",
    "- 1,764 neurons, with dropout regularization rate of 0.4 (probability\n",
    "  of 0.4 that any given element will be dropped during training)\n",
    "- Dense Layer (Logits Layer): 10 neurons, one for each digit target\n",
    "  class (0–9).\n",
    "\n",
    "There are three important modules to use to create a CNN:\n",
    "\n",
    "- `conv2d()`. Constructs a two-dimensional convolutional layer with the\n",
    "  number of filters, filter kernel size, padding, and activation\n",
    "  function as arguments.\n",
    "- `max_pooling2d()`. Constructs a two-dimensional pooling layer using\n",
    "  the max-pooling algorithm.\n",
    "- `dense()`. Constructs a dense layer with the hidden layers and units\n",
    "\n",
    "You will define a function to build the CNN. Let's see in detail how to\n",
    "construct each building block before to wrap everything together in the\n",
    "function.\n",
    "\n",
    "**Step 2**: Input layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "def cnn_model_fn(features, labels, mode):\n",
    "    input_layer = tf.reshape(tensor = features[\"x\"],shape =[-1, 28, 28, 1])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to define a tensor with the shape of the data. For that, you can use the module tf.reshape. In this module, you need to declare the\n",
    "tensor to reshape and the shape of the tensor. The first argument is the\n",
    "features of the data, which is defined in the argument of the function.\n",
    "A picture has a height, a width, and a channel. The MNIST dataset is a\n",
    "monochronic picture with a 28x28 size. We set the batch size to -1 in\n",
    "the shape argument so that it takes the shape of the `features[\"x\"]`.\n",
    "The advantage is to make the batch size hyperparameters to tune. If the\n",
    "batch size is set to 7, then the tensor will feed 5,488 values (28*28*7).\n",
    "\n",
    "**Step 3**: Convolutional layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# first Convolutional Layer\n",
    "conv1 = tf.layers.conv2d(\n",
    "    inputs=input_layer,\n",
    "    filters=14,\n",
    "    kernel_size=[5, 5],\n",
    "    padding=\"same\",\n",
    "    activation=tf.nn.relu)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first convolutional layer has 14 filters with a kernel size of 5x5\n",
    "with the same padding. The same padding means both the output tensor and\n",
    "input tensor should have the same height and width. Tensorflow will add\n",
    "zeros to the rows and columns to ensure the same size.\n",
    "You use the Relu activation function. The output size will be\n",
    "[28, 28, 14].\n",
    "\n",
    "**Step 4**: Pooling layer\n",
    "The next step after the convolution is the pooling computation. The\n",
    "pooling computation will reduce the dimensionality of the data. You can\n",
    "use the module max_pooling2d with a size of 2x2 and stride of 2. You\n",
    "use the previous layer as input. The output size will be\n",
    "[batch_size, 14, 14, 14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# first Pooling Layer \n",
    "pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 5**: Second Convolutional Layer and Pooling Layer\n",
    "\n",
    "The second convolutional layer has 32 filters, with an output size of\n",
    "`[batch_size, 14, 14, 32]`. The pooling layer has the same size as\n",
    "before and the output shape is `[batch_size, 14, 14, 18]`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "conv2 = tf.layers.conv2d(\n",
    "      inputs=pool1,\n",
    "      filters=36,\n",
    "      kernel_size=[5, 5],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 6**: Dense layer\n",
    "Then, you need to define the fully-connected layer. The feature map has\n",
    "to be flatten before to be connected with the dense layer. You can use\n",
    "the module reshape with a size of 7*7*36.\n",
    "The dense layer will connect 1764 neurons. You add a Relu activation\n",
    "function. Besides, you add a dropout regularization term with a rate of\n",
    "0.3, meaning 30 percents of the weights will be set to 0. Note that, the\n",
    "dropout takes place only during the training phase. The function\n",
    "cnn_model_fn has an argument mode to declare if the model needs to\n",
    "be trained or to evaluate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 36])\n",
    "\n",
    "dense = tf.layers.dense(inputs=pool2_flat, units=7 * 7 * 36, activation=tf.nn.relu)\n",
    "dropout = tf.layers.dropout(\n",
    "      inputs=dense, rate=0.3, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 7**: Logit Layer\n",
    "\n",
    "Finally, you can define the last layer with the prediction of the model.\n",
    "The output shape is equal to the batch size and 10, the total number of\n",
    "images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# Logits Layer\n",
    "logits = tf.layers.dense(inputs=dropout, units=10)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create a dictionary containing the classes and the probability\n",
    "of each class. The module `tf.argmax()` with returns the highest value\n",
    "if the logit layers. The softmax function returns the probability of\n",
    "each class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "predictions = {\n",
    "      # Generate predictions\n",
    "      \"classes\": tf.argmax(input=logits, axis=1),\n",
    "      \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
    "  }\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You only want to return the dictionnary prediction when mode is set to\n",
    "prediction. You add this codes to dispay the predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step consists to compute the loss of the model. In the last\n",
    "tutorial, you learnt that the loss function for a multiclass model is\n",
    "cross entropy. The loss is easily computed with the following code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# Calculate Loss (for both TRAIN and EVAL modes)\n",
    "loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to optimize the model, that is to find the best values\n",
    "of the weights. For that, you use a Gradient descent optimizer with a\n",
    "learning rate of 0.001. The objective is to minimize the loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
    "train_op = optimizer.minimize(\n",
    "        loss=loss,\n",
    "        global_step=tf.train.get_global_step())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are done with the CNN. However, you want to display the performance\n",
    "metrics during the evaluation mode. The performance metrics for a\n",
    "multiclass model is the accuracy metrics. Tensorflow is equipped with\n",
    "a module accuracy with two arguments, the labels, and the predicted\n",
    "values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "eval_metric_ops = {\n",
    "      \"accuracy\": tf.metrics.accuracy(\n",
    "          labels=labels, predictions=predictions[\"classes\"])}\n",
    "return tf.estimator.EstimatorSpec(\n",
    "      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it. You created your first CNN and you are ready to wrap\n",
    "everything into a function in order to use it to train and evaluate the\n",
    "model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cnn_model_fn(features, labels, mode):\n",
    "  \"\"\"Model function for CNN.\"\"\"\n",
    "  # Input Layer\n",
    "  input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n",
    "\n",
    "  # Convolutional Layer\n",
    "  conv1 = tf.layers.conv2d(\n",
    "      inputs=input_layer,\n",
    "      filters=32,\n",
    "      kernel_size=[5, 5],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "\n",
    "  # Pooling Layer\n",
    "  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n",
    "\n",
    "  # Convolutional Layer #2 and Pooling Layer\n",
    "  conv2 = tf.layers.conv2d(\n",
    "      inputs=pool1,\n",
    "      filters=36,\n",
    "      kernel_size=[5, 5],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "  pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n",
    "\n",
    "  # Dense Layer\n",
    "  pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 36])\n",
    "  dense = tf.layers.dense(inputs=pool2_flat, units=7 * 7 * 36, activation=tf.nn.relu)\n",
    "  dropout = tf.layers.dropout(\n",
    "      inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
    "\n",
    "  # Logits Layer\n",
    "  logits = tf.layers.dense(inputs=dropout, units=10)\n",
    "\n",
    "  predictions = {\n",
    "      # Generate predictions (for PREDICT and EVAL mode)\n",
    "      \"classes\": tf.argmax(input=logits, axis=1),\n",
    "      \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
    "  }\n",
    "\n",
    "  if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
    "\n",
    "  # Calculate Loss\n",
    "  loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)\n",
    "\n",
    "  # Configure the Training Op (for TRAIN mode)\n",
    "  if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
    "    train_op = optimizer.minimize(\n",
    "        loss=loss,\n",
    "        global_step=tf.train.get_global_step())\n",
    "    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)\n",
    "\n",
    "  # Add evaluation metrics Evaluation mode\n",
    "  eval_metric_ops = {\n",
    "      \"accuracy\": tf.metrics.accuracy(\n",
    "          labels=labels, predictions=predictions[\"classes\"])}\n",
    "  return tf.estimator.EstimatorSpec(\n",
    "      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The steps below are the same as the previous tutorials.\n",
    "First of all, you define an estimator with the CNN model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'train/mnist_convnet_model', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x114288e48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "# Create the Estimator\n",
    "mnist_classifier = tf.estimator.Estimator(\n",
    "    model_fn=cnn_model_fn, model_dir=\"train/mnist_convnet_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A CNN takes many times to train, therefore, you create a Logging hook to\n",
    "store the values of the softmax layers every 50 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up logging for predictions\n",
    "tensors_to_log = {\"probabilities\": \"softmax_tensor\"}\n",
    "logging_hook = tf.train.LoggingTensorHook(\n",
    "      tensors=tensors_to_log, every_n_iter=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are ready to estimate the model. You set a batch size of 100 and\n",
    "shuffle the data. Note that we set training steps of 16.000, it can take\n",
    "lots of time to train. Be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into train/mnist_convnet_model/model.ckpt.\n",
      "INFO:tensorflow:probabilities = [[0.09477627 0.12005997 0.10586331 0.09869653 0.10132864 0.09234226\n",
      "  0.10487132 0.08388694 0.09745449 0.10072027]\n",
      " [0.09137405 0.10250511 0.10209611 0.09091399 0.11156436 0.08881138\n",
      "  0.10393392 0.09999351 0.1071749  0.10163269]\n",
      " [0.08873715 0.10550566 0.09641793 0.09746279 0.09736262 0.09782984\n",
      "  0.10728729 0.09973746 0.10353723 0.10612202]\n",
      " [0.08601869 0.11003186 0.08991204 0.08752326 0.11834793 0.09352052\n",
      "  0.10633367 0.10139539 0.10406768 0.10284896]\n",
      " [0.10315961 0.10586246 0.0936099  0.1030763  0.10913491 0.08971104\n",
      "  0.11866575 0.09154476 0.09029319 0.09494208]\n",
      " [0.11261123 0.10636197 0.09411069 0.11400822 0.09893055 0.08329154\n",
      "  0.10536935 0.08893343 0.09886362 0.0975194 ]\n",
      " [0.087216   0.10474014 0.09323981 0.10548251 0.09960875 0.09908232\n",
      "  0.10710015 0.08728469 0.10689161 0.10935402]\n",
      " [0.09037717 0.11850066 0.10845283 0.09036495 0.12497254 0.07775259\n",
      "  0.10303241 0.07735642 0.10383295 0.10535749]\n",
      " [0.09203806 0.11672669 0.0879255  0.09777382 0.1027937  0.08937059\n",
      "  0.1178918  0.0811695  0.09646051 0.11784983]\n",
      " [0.07744523 0.10778381 0.10223136 0.10253583 0.10028112 0.09320711\n",
      "  0.12182171 0.09189297 0.10289152 0.09990935]\n",
      " [0.0882757  0.11398443 0.10164383 0.09811232 0.10174763 0.10017468\n",
      "  0.10510596 0.08886497 0.092503   0.10958748]\n",
      " [0.08758337 0.10895424 0.10411318 0.10290048 0.11544459 0.08623699\n",
      "  0.10705958 0.1002896  0.0943812  0.09303677]\n",
      " [0.08732712 0.11717079 0.10824638 0.0902344  0.0912208  0.10232965\n",
      "  0.11455312 0.09052799 0.09793385 0.1004559 ]\n",
      " [0.10523345 0.10741371 0.10681214 0.09179917 0.10870784 0.08787969\n",
      "  0.0972807  0.08512054 0.09554896 0.1142038 ]\n",
      " [0.09370556 0.10347668 0.10248109 0.09057055 0.10204056 0.08957646\n",
      "  0.10910604 0.09741776 0.10945721 0.1021681 ]\n",
      " [0.08559919 0.10925838 0.09809111 0.09485994 0.11831627 0.0948895\n",
      "  0.10447654 0.08994441 0.10165867 0.10290599]\n",
      " [0.08695774 0.11110834 0.12553175 0.09454737 0.09933779 0.09213896\n",
      "  0.10230222 0.08699714 0.10435706 0.09672163]\n",
      " [0.09339443 0.11344254 0.08228982 0.09927023 0.11282496 0.08211148\n",
      "  0.12040265 0.07868681 0.10342638 0.1141507 ]\n",
      " [0.10192247 0.10112078 0.1070597  0.09799815 0.10749337 0.08882416\n",
      "  0.10689499 0.08262113 0.09794178 0.10812347]\n",
      " [0.10292241 0.10626735 0.10113504 0.09898753 0.10910281 0.08915134\n",
      "  0.10066633 0.09201657 0.09949737 0.10025325]\n",
      " [0.08234016 0.11065982 0.10599457 0.09893471 0.1154562  0.09548124\n",
      "  0.11062597 0.08619873 0.09608606 0.09822255]\n",
      " [0.10620914 0.11331085 0.10305813 0.08450326 0.10315709 0.10121783\n",
      "  0.10091058 0.08596787 0.09783431 0.10383094]\n",
      " [0.09282554 0.10528844 0.10290841 0.09464207 0.10711342 0.09167153\n",
      "  0.11588779 0.08543965 0.10173548 0.10248767]\n",
      " [0.07954638 0.11127387 0.10019367 0.10262637 0.10494642 0.09388408\n",
      "  0.10286555 0.08381825 0.11092973 0.10991569]\n",
      " [0.09269291 0.10610541 0.11435575 0.09660572 0.09658302 0.09044313\n",
      "  0.10300096 0.08724271 0.10728433 0.10568605]\n",
      " [0.09945339 0.13159333 0.09078286 0.08879604 0.11217698 0.08781617\n",
      "  0.10165422 0.08687326 0.08872112 0.11213263]\n",
      " [0.09979893 0.0986394  0.09311255 0.08952248 0.10784432 0.09032838\n",
      "  0.12129503 0.09352637 0.09846227 0.10747026]\n",
      " [0.09264918 0.10308595 0.10697635 0.09180093 0.10311106 0.10348497\n",
      "  0.11069563 0.07721728 0.09703295 0.1139457 ]\n",
      " [0.09040567 0.10560416 0.09215949 0.09262638 0.1178038  0.09279343\n",
      "  0.12636869 0.09000779 0.08668651 0.10554406]\n",
      " [0.09363284 0.10143781 0.09649093 0.10770173 0.1063482  0.09842086\n",
      "  0.09987485 0.08737357 0.10077317 0.10794605]\n",
      " [0.09701799 0.10658538 0.09822693 0.10435981 0.10577905 0.08457921\n",
      "  0.10843234 0.09868123 0.09314844 0.10318963]\n",
      " [0.09510654 0.11352345 0.10475464 0.09741106 0.09612477 0.09946139\n",
      "  0.09871102 0.08840924 0.09613517 0.11036271]\n",
      " [0.09867485 0.1127427  0.09876813 0.0990213  0.09565362 0.0929149\n",
      "  0.10601637 0.09274274 0.08837654 0.11508887]\n",
      " [0.09238018 0.11416183 0.10256469 0.09747151 0.10054805 0.10485937\n",
      "  0.11282266 0.08352242 0.09862266 0.09304663]\n",
      " [0.08536028 0.11293992 0.11094524 0.09755105 0.10119803 0.08418413\n",
      "  0.11391264 0.08937872 0.10526377 0.09926621]\n",
      " [0.09665235 0.09807382 0.09235908 0.09205186 0.1149671  0.09997211\n",
      "  0.10866256 0.09366969 0.10413439 0.09945704]\n",
      " [0.08572761 0.12158909 0.10609257 0.09993963 0.09759045 0.10110569\n",
      "  0.11393609 0.08311082 0.09528129 0.09562676]\n",
      " [0.09358929 0.11069349 0.11763485 0.08976709 0.09568804 0.08880975\n",
      "  0.10896682 0.08945567 0.11072045 0.09467456]\n",
      " [0.09090464 0.10126796 0.10187562 0.09535116 0.1067795  0.10292348\n",
      "  0.10361655 0.09905496 0.08335513 0.11487099]\n",
      " [0.09840156 0.12816868 0.08977151 0.09732473 0.10029128 0.09603746\n",
      "  0.11441581 0.08820783 0.09197304 0.09540811]\n",
      " [0.09775364 0.12660216 0.08813359 0.07575843 0.11491079 0.08334736\n",
      "  0.11218381 0.08847427 0.08981169 0.12302425]\n",
      " [0.0999305  0.09953814 0.10534713 0.10402522 0.10226403 0.09345784\n",
      "  0.09895859 0.10040292 0.09477609 0.10129955]\n",
      " [0.08185847 0.10322296 0.10781956 0.09069352 0.09805666 0.09655249\n",
      "  0.11233704 0.09977523 0.09738901 0.11229505]\n",
      " [0.0970194  0.10732862 0.0975212  0.08727585 0.11142121 0.08539802\n",
      "  0.11559209 0.10199942 0.09391653 0.10252766]\n",
      " [0.08822355 0.10551447 0.08922048 0.09574616 0.09642582 0.09616226\n",
      "  0.11333686 0.08082839 0.10215297 0.13238904]\n",
      " [0.09257094 0.11081256 0.10986823 0.08987674 0.10198785 0.07669328\n",
      "  0.11053754 0.08577755 0.10988467 0.11199062]\n",
      " [0.0904771  0.10059423 0.09837909 0.09569524 0.10544216 0.09549078\n",
      "  0.10976215 0.0935386  0.10283644 0.10778421]\n",
      " [0.09562387 0.11140071 0.09387314 0.09136051 0.12518403 0.08731371\n",
      "  0.09552952 0.08608692 0.10398556 0.10964204]\n",
      " [0.09632467 0.10348536 0.10496031 0.0896253  0.11315977 0.08444217\n",
      "  0.10701536 0.09720501 0.10265926 0.1011228 ]\n",
      " [0.10513389 0.1068675  0.09440215 0.09388577 0.10358219 0.09585685\n",
      "  0.10423435 0.08734773 0.10824443 0.10044514]\n",
      " [0.09731423 0.10644594 0.1090375  0.10537503 0.09918428 0.09294299\n",
      "  0.10998209 0.09315743 0.09086537 0.09569513]\n",
      " [0.10123907 0.10842714 0.10194908 0.09507789 0.10541448 0.07737227\n",
      "  0.11913865 0.08775285 0.10284508 0.10078349]\n",
      " [0.09992703 0.10836256 0.10275815 0.08588514 0.11088973 0.08286405\n",
      "  0.11894013 0.09130781 0.09004077 0.10902461]\n",
      " [0.09473495 0.09368717 0.11201667 0.08782815 0.10253646 0.08028022\n",
      "  0.12731079 0.0838974  0.10054756 0.11716062]\n",
      " [0.09943964 0.10161078 0.09541572 0.10538826 0.10446423 0.10150245\n",
      "  0.10609364 0.08820801 0.08459346 0.11328382]\n",
      " [0.09126713 0.1094009  0.10158526 0.10827031 0.11089621 0.09070689\n",
      "  0.10386672 0.08933751 0.09562739 0.09904169]\n",
      " [0.10611817 0.10741958 0.08909872 0.09795605 0.10369757 0.09332109\n",
      "  0.1123749  0.09584718 0.09613831 0.09802843]\n",
      " [0.09844941 0.10684991 0.09589166 0.0985519  0.10605975 0.09665397\n",
      "  0.10050706 0.09292148 0.09435212 0.10976273]\n",
      " [0.08947029 0.1129137  0.10726649 0.08869186 0.09640936 0.10546607\n",
      "  0.10527894 0.07070168 0.10361589 0.12018571]\n",
      " [0.0896296  0.10221787 0.09685163 0.09499216 0.10266155 0.09692681\n",
      "  0.11105229 0.09149451 0.1131144  0.1010592 ]\n",
      " [0.09214072 0.10523216 0.08562559 0.09916545 0.11532077 0.08861516\n",
      "  0.11836224 0.08832816 0.10084452 0.10636524]\n",
      " [0.10138659 0.11175099 0.10877353 0.08789372 0.103071   0.09989446\n",
      "  0.10843814 0.08871372 0.09211041 0.09796744]\n",
      " [0.11301389 0.08539265 0.0972022  0.09650226 0.10708489 0.09659861\n",
      "  0.11488098 0.08835761 0.10141414 0.09955277]\n",
      " [0.09832872 0.09654156 0.10051628 0.09305811 0.08420118 0.09734625\n",
      "  0.10503072 0.08961743 0.11231268 0.12304706]\n",
      " [0.0913598  0.09809444 0.10970738 0.09339366 0.10481746 0.09597608\n",
      "  0.10877693 0.0829438  0.10341574 0.11151471]\n",
      " [0.10119983 0.10297755 0.10273128 0.0921584  0.11279104 0.10248752\n",
      "  0.1050135  0.09856906 0.09930403 0.08276779]\n",
      " [0.08492543 0.10591415 0.1049501  0.09451202 0.10711533 0.09516922\n",
      "  0.1015731  0.08876956 0.10700501 0.11006609]\n",
      " [0.11070956 0.09855162 0.10019214 0.08212816 0.11074062 0.10026054\n",
      "  0.10042098 0.09158038 0.09608849 0.1093275 ]\n",
      " [0.08975872 0.10970234 0.09969618 0.09533209 0.10914912 0.10500528\n",
      "  0.11064774 0.07938824 0.09006236 0.11125794]\n",
      " [0.09248688 0.11752693 0.09857151 0.09815205 0.10265189 0.09007132\n",
      "  0.11193885 0.09398464 0.09646244 0.0981535 ]\n",
      " [0.09228167 0.10433102 0.1005711  0.1030877  0.09444394 0.10195967\n",
      "  0.11538947 0.07529703 0.10293715 0.10970123]\n",
      " [0.08876971 0.11136132 0.08942719 0.1054659  0.1227581  0.09667183\n",
      "  0.09837679 0.08515084 0.10277291 0.09924542]\n",
      " [0.09160986 0.10258748 0.11031057 0.09664901 0.11203503 0.09013819\n",
      "  0.10978149 0.08399416 0.10059852 0.10229568]\n",
      " [0.10001212 0.10083755 0.10983893 0.08555229 0.11453225 0.09162526\n",
      "  0.11794807 0.08849512 0.09182815 0.09933026]\n",
      " [0.08822069 0.13237664 0.10372681 0.10849863 0.08464736 0.09310339\n",
      "  0.11472311 0.07662932 0.09876478 0.09930927]\n",
      " [0.09707194 0.10866705 0.10111879 0.08723607 0.10445564 0.09349896\n",
      "  0.10503812 0.08726544 0.10018378 0.11546423]\n",
      " [0.09296612 0.10646117 0.10019705 0.08767922 0.1033056  0.09424386\n",
      "  0.1204897  0.090291   0.09063405 0.11373224]\n",
      " [0.09878406 0.10108925 0.10496195 0.09044451 0.10473652 0.09081131\n",
      "  0.11004237 0.09219706 0.08804857 0.11888441]\n",
      " [0.0913606  0.11352156 0.09372978 0.10086189 0.1027604  0.09334698\n",
      "  0.11730664 0.08789712 0.09434548 0.10486955]\n",
      " [0.09166942 0.11225881 0.09503555 0.0900662  0.10655183 0.09364416\n",
      "  0.11020344 0.08383904 0.10727295 0.1094586 ]\n",
      " [0.09270768 0.11237144 0.10653157 0.07985067 0.10058117 0.09455915\n",
      "  0.11014107 0.08545295 0.10926947 0.10853482]\n",
      " [0.09339526 0.11286006 0.09041484 0.10671464 0.12647774 0.08723654\n",
      "  0.09882879 0.08741674 0.09536593 0.10128946]\n",
      " [0.10315935 0.10085666 0.09503548 0.09163657 0.11351219 0.09518125\n",
      "  0.10684156 0.09645192 0.09584799 0.10147702]\n",
      " [0.07806656 0.10230957 0.10491866 0.0929223  0.10084099 0.09815357\n",
      "  0.1293502  0.09309497 0.10126271 0.09908048]\n",
      " [0.09443475 0.10504287 0.09983539 0.09391351 0.10139566 0.09360619\n",
      "  0.10005803 0.09424612 0.09569781 0.12176967]\n",
      " [0.09432892 0.12161017 0.08733891 0.10337046 0.10748676 0.09077028\n",
      "  0.11251068 0.07757936 0.10667618 0.09832827]\n",
      " [0.0957939  0.11206303 0.08935661 0.09871534 0.10832554 0.08941343\n",
      "  0.1043737  0.08764758 0.10561381 0.10869705]\n",
      " [0.1058721  0.10364363 0.10847021 0.07468409 0.10077738 0.09604683\n",
      "  0.10986363 0.10372065 0.09708114 0.09984034]\n",
      " [0.09755537 0.102055   0.09711649 0.10397008 0.10223868 0.09416995\n",
      "  0.10772427 0.0884591  0.0951144  0.11159667]\n",
      " [0.09313037 0.10421699 0.115703   0.1027037  0.09993006 0.09127614\n",
      "  0.09955402 0.08178753 0.10189641 0.10980179]\n",
      " [0.10815043 0.11021502 0.10954847 0.0833929  0.11431696 0.08237163\n",
      "  0.1140805  0.08440826 0.09659185 0.09692397]\n",
      " [0.09379273 0.10536435 0.09479776 0.08687428 0.11467929 0.08951769\n",
      "  0.11087991 0.09088559 0.10118368 0.11202471]\n",
      " [0.09840687 0.09487349 0.09018699 0.09746453 0.11124728 0.0969341\n",
      "  0.10857365 0.0935303  0.09562486 0.11315794]\n",
      " [0.08518515 0.11349864 0.10013819 0.08948862 0.09573815 0.0839393\n",
      "  0.11875229 0.0930909  0.1111873  0.10898145]\n",
      " [0.09210006 0.11799479 0.10455556 0.09840916 0.10490013 0.0859287\n",
      "  0.10000422 0.08303436 0.10441244 0.10866059]\n",
      " [0.10541176 0.11259707 0.09516838 0.09372374 0.10262216 0.08592701\n",
      "  0.11933541 0.09025378 0.09146277 0.10349792]\n",
      " [0.09423298 0.10866464 0.10681903 0.09867087 0.09099088 0.10391623\n",
      "  0.10836901 0.08540214 0.09765934 0.10527489]\n",
      " [0.10233208 0.10469262 0.09601465 0.09714619 0.11391051 0.08845449\n",
      "  0.10342303 0.09888098 0.093616   0.10152945]\n",
      " [0.08043491 0.12079332 0.10376781 0.09496202 0.10302038 0.08915791\n",
      "  0.10072855 0.08909667 0.1057519  0.11228654]\n",
      " [0.10153272 0.09865129 0.11723876 0.08610013 0.10806575 0.08399273\n",
      "  0.10006591 0.09639747 0.10848766 0.09946756]]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 2.297851085662842, step = 1\n",
      "INFO:tensorflow:probabilities = [[0.11054767 0.09275471 0.10013526 0.08835042 0.11395822 0.09137029\n",
      "  0.11880095 0.09505082 0.08695626 0.10207541]\n",
      " [0.10244074 0.10639194 0.09917687 0.0892521  0.11454935 0.09581779\n",
      "  0.1061843  0.07470831 0.11133587 0.10014272]\n",
      " [0.09699993 0.10831697 0.10654346 0.08630888 0.11071913 0.0836101\n",
      "  0.11320966 0.08499833 0.10292014 0.10637341]\n",
      " [0.0987331  0.09658351 0.09226895 0.10431514 0.10138343 0.10351134\n",
      "  0.10918573 0.10066592 0.08524862 0.10810425]\n",
      " [0.09117156 0.11203172 0.10274307 0.1035143  0.09821626 0.09624681\n",
      "  0.10442217 0.08876414 0.09710149 0.10578848]\n",
      " [0.08540867 0.12719263 0.11550417 0.09658033 0.0928353  0.09237116\n",
      "  0.1038759  0.0857867  0.09271943 0.10772572]\n",
      " [0.09545376 0.09604547 0.12127067 0.09799733 0.10155984 0.08310186\n",
      "  0.09298788 0.09923943 0.09127276 0.121071  ]\n",
      " [0.10535233 0.09774638 0.09067006 0.07460938 0.09765898 0.09005611\n",
      "  0.11580084 0.10369658 0.12084851 0.10356081]\n",
      " [0.09699483 0.11322552 0.10567522 0.088451   0.09886734 0.0860258\n",
      "  0.09617126 0.09492195 0.11068553 0.10898154]\n",
      " [0.09651341 0.09919401 0.10554811 0.10180538 0.09997488 0.08831122\n",
      "  0.10235445 0.09502503 0.09878763 0.11248587]\n",
      " [0.08732869 0.10130662 0.10040327 0.09206663 0.10063619 0.09498253\n",
      "  0.11635246 0.10162707 0.09983444 0.10546209]\n",
      " [0.08093264 0.11628664 0.09622939 0.09615391 0.11291239 0.0980408\n",
      "  0.10588902 0.08921966 0.09900837 0.10532717]\n",
      " [0.10564864 0.09800203 0.09423434 0.09455095 0.10492309 0.09829947\n",
      "  0.11414054 0.09205026 0.09090744 0.10724325]\n",
      " [0.08991361 0.0961521  0.10182456 0.09320452 0.09701647 0.09961248\n",
      "  0.10118399 0.08961189 0.1045735  0.12690688]\n",
      " [0.09950247 0.10149411 0.09499799 0.09363086 0.0972316  0.1013812\n",
      "  0.10372371 0.10640864 0.10817421 0.09345522]\n",
      " [0.08888175 0.11360589 0.08924509 0.08505765 0.12964987 0.07830585\n",
      "  0.10726287 0.08808018 0.10396537 0.11594549]\n",
      " [0.09352882 0.0980373  0.10415269 0.09798151 0.09621118 0.09959441\n",
      "  0.11105018 0.10341813 0.09582565 0.10020013]\n",
      " [0.08918021 0.10750787 0.09860381 0.10236182 0.11260553 0.08734359\n",
      "  0.09830921 0.09820526 0.10191805 0.10396466]\n",
      " [0.09336612 0.11226553 0.09725962 0.0914186  0.1122485  0.08443763\n",
      "  0.10191915 0.0892541  0.1213492  0.09648154]\n",
      " [0.09073882 0.10624945 0.10941111 0.09048919 0.1018954  0.09984449\n",
      "  0.11120417 0.0904655  0.09845178 0.10125009]\n",
      " [0.11756143 0.10065217 0.09810634 0.09328442 0.10473906 0.09695129\n",
      "  0.10728285 0.08722574 0.0898476  0.10434911]\n",
      " [0.10727526 0.10738638 0.1040475  0.08545933 0.10300324 0.09359847\n",
      "  0.10287855 0.08343182 0.10427349 0.10864597]\n",
      " [0.08734722 0.10239943 0.09485544 0.09793347 0.10437725 0.09115105\n",
      "  0.11544719 0.09568944 0.09767182 0.11312769]\n",
      " [0.10296955 0.0966298  0.10528476 0.08356832 0.1085544  0.09614503\n",
      "  0.11802921 0.08535601 0.10141442 0.10204848]\n",
      " [0.09977217 0.10614815 0.09982496 0.09145935 0.10116357 0.09224084\n",
      "  0.10314637 0.1019423  0.10341577 0.10088651]\n",
      " [0.1003985  0.10882035 0.09699257 0.09997417 0.10232225 0.0938822\n",
      "  0.10719781 0.10180235 0.09005321 0.0985566 ]\n",
      " [0.08153905 0.10990558 0.09086423 0.09664638 0.10777029 0.09849682\n",
      "  0.11024734 0.09448896 0.09955062 0.11049073]\n",
      " [0.09568528 0.10739655 0.10225965 0.08438599 0.09479487 0.0892621\n",
      "  0.11954678 0.08525123 0.10646027 0.11495727]\n",
      " [0.09230126 0.10791419 0.10749977 0.10165597 0.10760579 0.08953417\n",
      "  0.09230907 0.09174459 0.10326571 0.10616947]\n",
      " [0.11117819 0.12347667 0.09393357 0.09113163 0.10010051 0.08749111\n",
      "  0.113131   0.08538479 0.09209216 0.10208037]\n",
      " [0.09633596 0.1021606  0.10020448 0.08185333 0.10923835 0.09581074\n",
      "  0.11826381 0.09698565 0.09405001 0.10509707]\n",
      " [0.10414616 0.10451131 0.09504704 0.10686774 0.10067366 0.09032071\n",
      "  0.09994908 0.09109217 0.1088055  0.09858664]\n",
      " [0.0908034  0.10496255 0.09480937 0.09088523 0.1020242  0.09809842\n",
      "  0.11268478 0.09613345 0.10421537 0.10538323]\n",
      " [0.09968846 0.12050966 0.09626527 0.0889978  0.10767153 0.09670763\n",
      "  0.09883361 0.09480712 0.10317577 0.09334315]\n",
      " [0.08502082 0.10459097 0.10900421 0.09261328 0.11583278 0.08933185\n",
      "  0.10256135 0.08861438 0.10312703 0.10930333]\n",
      " [0.0966114  0.10143654 0.09874236 0.10686264 0.11350791 0.09568297\n",
      "  0.09955645 0.07593864 0.09241534 0.11924575]\n",
      " [0.09416571 0.11052389 0.09371057 0.10075998 0.10011773 0.11010613\n",
      "  0.1036441  0.09541164 0.08737423 0.10418601]\n",
      " [0.09918453 0.09736744 0.10818252 0.09488298 0.09640809 0.10543187\n",
      "  0.11779627 0.08594842 0.08652811 0.10826976]\n",
      " [0.08923933 0.11476752 0.08672682 0.09557922 0.0993956  0.08728334\n",
      "  0.10364002 0.08587507 0.10908978 0.1284033 ]\n",
      " [0.09052531 0.1142918  0.09915193 0.10708876 0.0912041  0.09674893\n",
      "  0.09117081 0.09045337 0.09329205 0.12607295]\n",
      " [0.10270112 0.10638882 0.09864537 0.09043204 0.10924151 0.08695869\n",
      "  0.10896569 0.09605948 0.09786586 0.10274142]\n",
      " [0.08984047 0.10631463 0.09749561 0.08534655 0.09271359 0.09685043\n",
      "  0.13667584 0.09269904 0.091485   0.11057883]\n",
      " [0.0746648  0.11663428 0.09881071 0.0952692  0.11410151 0.08355774\n",
      "  0.11290946 0.08607723 0.09100644 0.12696862]\n",
      " [0.09703568 0.09655291 0.10061069 0.09935633 0.10284953 0.09257421\n",
      "  0.11486331 0.09572251 0.0961096  0.10432523]\n",
      " [0.08599251 0.114817   0.09476137 0.09022602 0.10303524 0.09659765\n",
      "  0.10495694 0.09323939 0.10651815 0.10985573]\n",
      " [0.09603695 0.1033946  0.09233192 0.0991229  0.10085891 0.10002415\n",
      "  0.11536643 0.09472383 0.10913254 0.08900776]\n",
      " [0.09439564 0.10926794 0.10419202 0.09184058 0.09265544 0.09711812\n",
      "  0.11184701 0.09069192 0.11163041 0.09636092]\n",
      " [0.0836582  0.09911183 0.09705078 0.09466652 0.10257676 0.10065447\n",
      "  0.11962603 0.08144644 0.10389751 0.11731146]\n",
      " [0.0986985  0.10781028 0.09631101 0.09306591 0.10740994 0.10261088\n",
      "  0.10121307 0.08708154 0.10452269 0.10127617]\n",
      " [0.10222683 0.09774045 0.10066183 0.09144909 0.11435975 0.09333147\n",
      "  0.10700399 0.09033846 0.10399268 0.09889545]\n",
      " [0.0989466  0.10972112 0.10472712 0.09611863 0.09695597 0.09473424\n",
      "  0.10255013 0.09782372 0.08886579 0.10955669]\n",
      " [0.10171967 0.09821261 0.10348322 0.09262972 0.10003607 0.09854371\n",
      "  0.09914648 0.09744365 0.10496368 0.10382119]\n",
      " [0.10147624 0.0985585  0.08686892 0.10528283 0.10446172 0.0913368\n",
      "  0.10236899 0.08633387 0.11893253 0.10437962]\n",
      " [0.09066015 0.11449482 0.09339905 0.09486607 0.10803429 0.09429941\n",
      "  0.10626518 0.10494572 0.09796707 0.09506823]\n",
      " [0.09199115 0.10972414 0.11125485 0.082874   0.11937265 0.0907377\n",
      "  0.11074114 0.09286645 0.0983606  0.09207732]\n",
      " [0.09372918 0.10530688 0.11156863 0.10279177 0.10088053 0.08158192\n",
      "  0.10334728 0.10128432 0.09938703 0.10012246]\n",
      " [0.08736477 0.11175492 0.08637775 0.08318395 0.1354094  0.07773845\n",
      "  0.12585634 0.09586426 0.09895281 0.09749735]\n",
      " [0.10739638 0.1199887  0.09614643 0.09610395 0.09826621 0.08971826\n",
      "  0.11085609 0.08475369 0.09319785 0.10357246]\n",
      " [0.08646862 0.10610326 0.10633939 0.09519438 0.10082948 0.09507956\n",
      "  0.14066389 0.08289432 0.09352084 0.09290627]\n",
      " [0.08784315 0.09970521 0.10227884 0.10136532 0.10846408 0.08629864\n",
      "  0.1224002  0.09861584 0.0948234  0.09820531]\n",
      " [0.09069764 0.11001439 0.11257921 0.08110882 0.10450851 0.09831736\n",
      "  0.10707235 0.09623132 0.09984282 0.09962757]\n",
      " [0.09848169 0.10020652 0.0940028  0.08647385 0.09931926 0.0998603\n",
      "  0.11379268 0.08938337 0.10453085 0.11394868]\n",
      " [0.09478872 0.09896981 0.09988518 0.10407674 0.09304903 0.09553195\n",
      "  0.11131161 0.08262414 0.10575859 0.11400423]\n",
      " [0.09732468 0.10525604 0.09591805 0.10217074 0.11142718 0.08770186\n",
      "  0.11546666 0.0914317  0.09971993 0.09358316]\n",
      " [0.10904357 0.10900253 0.10104613 0.09260747 0.10423609 0.08636205\n",
      "  0.11112972 0.0922925  0.09682395 0.097456  ]\n",
      " [0.10182793 0.10612192 0.09883054 0.08655905 0.10643386 0.09291354\n",
      "  0.10749411 0.09023924 0.10353368 0.10604611]\n",
      " [0.10020928 0.10275912 0.09858788 0.09863098 0.09111058 0.0997418\n",
      "  0.09927632 0.0928059  0.1042708  0.11260733]\n",
      " [0.09584361 0.1150966  0.10624825 0.10437897 0.11395762 0.09648075\n",
      "  0.10134618 0.0917578  0.07904383 0.0958464 ]\n",
      " [0.09914049 0.11022338 0.09261473 0.1009097  0.11748748 0.09245828\n",
      "  0.10349465 0.0853908  0.10185309 0.0964274 ]\n",
      " [0.09929058 0.11425344 0.10573491 0.12477616 0.0897211  0.08433169\n",
      "  0.10787985 0.09721877 0.0956844  0.0811091 ]\n",
      " [0.10304052 0.10325044 0.10291668 0.100556   0.10949399 0.08867349\n",
      "  0.10943178 0.07991758 0.09912774 0.10359179]\n",
      " [0.09409581 0.10879428 0.0977519  0.09678794 0.1133723  0.10279263\n",
      "  0.09630179 0.0867876  0.09289413 0.11042161]\n",
      " [0.08458368 0.11783143 0.11273025 0.0952327  0.09391755 0.08213505\n",
      "  0.11142192 0.09586837 0.10854699 0.09773205]\n",
      " [0.0932893  0.11099327 0.0992195  0.09061003 0.1132391  0.09671827\n",
      "  0.10934087 0.08736459 0.09439596 0.10482911]\n",
      " [0.08718623 0.12094899 0.10486553 0.09416047 0.09759372 0.10409229\n",
      "  0.11492352 0.08854821 0.0905418  0.09713925]\n",
      " [0.09696081 0.11357626 0.09470854 0.08952153 0.10430566 0.08881127\n",
      "  0.11285009 0.09764233 0.08574575 0.11587776]\n",
      " [0.07970737 0.12418911 0.09692575 0.11851913 0.09622704 0.07776589\n",
      "  0.12048369 0.0885524  0.0965631  0.10106651]\n",
      " [0.0919177  0.11161371 0.09509719 0.09126718 0.10038122 0.09215625\n",
      "  0.11971486 0.0961978  0.10594154 0.09571255]\n",
      " [0.09505249 0.09959604 0.10327758 0.09524564 0.09435012 0.09032393\n",
      "  0.1243797  0.09736027 0.09488096 0.10553329]\n",
      " [0.08434841 0.11758393 0.09917844 0.10017636 0.09290306 0.09174812\n",
      "  0.1225284  0.08564164 0.10388074 0.10201091]\n",
      " [0.09543451 0.10122133 0.09977903 0.09095626 0.09987024 0.09383776\n",
      "  0.10561634 0.10154113 0.09714651 0.11459689]\n",
      " [0.08473758 0.1102581  0.09095183 0.10336697 0.11046259 0.09966544\n",
      "  0.09691041 0.08694054 0.09798408 0.11872245]\n",
      " [0.10094839 0.10529234 0.1030913  0.10142762 0.10466788 0.09302158\n",
      "  0.10071609 0.09263305 0.09533597 0.10286578]\n",
      " [0.0896825  0.11741371 0.09928015 0.1070854  0.11134567 0.07633802\n",
      "  0.11035052 0.08652921 0.10242319 0.09955163]\n",
      " [0.10114716 0.10444202 0.105094   0.09625159 0.10992912 0.08302146\n",
      "  0.11237167 0.07594263 0.10459222 0.10720814]\n",
      " [0.09987924 0.1199483  0.10120779 0.095109   0.09797886 0.08961029\n",
      "  0.1202701  0.09299198 0.09294377 0.09006067]\n",
      " [0.10213759 0.09843834 0.09740856 0.09714661 0.10394035 0.09531027\n",
      "  0.10680151 0.09378135 0.10430766 0.10072774]\n",
      " [0.09377904 0.10276363 0.09179743 0.09990853 0.12083361 0.09163772\n",
      "  0.10301661 0.09402004 0.09375603 0.10848736]\n",
      " [0.10603999 0.09869331 0.10227962 0.08964231 0.11262885 0.10188133\n",
      "  0.09805838 0.09609143 0.09937867 0.09530612]\n",
      " [0.10991298 0.11303709 0.09265513 0.09657162 0.11312515 0.08882619\n",
      "  0.10529958 0.08334859 0.09598744 0.10123624]\n",
      " [0.12483569 0.12220264 0.10394556 0.08486349 0.11290321 0.08542604\n",
      "  0.09956036 0.07444712 0.10260852 0.08920737]\n",
      " [0.09069218 0.10157267 0.0857737  0.11293905 0.12137694 0.09659296\n",
      "  0.11737734 0.08639982 0.08648113 0.10079421]\n",
      " [0.09680828 0.11275919 0.10602497 0.08676931 0.10626338 0.07975069\n",
      "  0.11187865 0.09264022 0.08896413 0.11814118]\n",
      " [0.10118069 0.11122252 0.10832978 0.08666232 0.10690419 0.07901704\n",
      "  0.10170137 0.0901566  0.10077018 0.11405532]\n",
      " [0.10119111 0.10652873 0.10062165 0.09749114 0.10418563 0.0905346\n",
      "  0.10396655 0.09569493 0.09553453 0.10425112]\n",
      " [0.10148103 0.11041767 0.10223254 0.08498054 0.10680266 0.09517347\n",
      "  0.09683505 0.0903085  0.0965633  0.11520523]\n",
      " [0.09148876 0.09840286 0.09747147 0.09753254 0.11696967 0.0869141\n",
      "  0.11003452 0.08864382 0.10440284 0.10813942]\n",
      " [0.09622218 0.09891395 0.09945357 0.09542664 0.11502789 0.09432139\n",
      "  0.10309017 0.09312188 0.09543707 0.10898526]\n",
      " [0.0930653  0.11054974 0.09618987 0.09619793 0.10705879 0.08780341\n",
      "  0.10170155 0.09441613 0.10215461 0.11086267]\n",
      " [0.0882316  0.10749353 0.0854144  0.09228727 0.13859635 0.08556007\n",
      "  0.12333822 0.07880148 0.09290195 0.10737512]] (19.430 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 2.62982\n",
      "INFO:tensorflow:probabilities = [[0.09779091 0.13565899 0.09770616 0.095411   0.10460225 0.10277529\n",
      "  0.09612318 0.07814849 0.09130824 0.10047547]\n",
      " [0.08314637 0.11291003 0.10187962 0.09921908 0.09288833 0.11500898\n",
      "  0.10429283 0.08099788 0.11138132 0.09827556]\n",
      " [0.10529038 0.1038397  0.10376705 0.10807484 0.10490608 0.08624273\n",
      "  0.09995768 0.09195606 0.08572645 0.11023902]\n",
      " [0.10792637 0.09719112 0.1017011  0.09304814 0.10655243 0.10512428\n",
      "  0.10660265 0.08814942 0.09070132 0.10300318]\n",
      " [0.1044911  0.10117117 0.09376086 0.106519   0.10203609 0.09516923\n",
      "  0.09423762 0.09972065 0.09807788 0.10481639]\n",
      " [0.09175901 0.1236866  0.10044358 0.08819327 0.097542   0.07747333\n",
      "  0.10762874 0.09329776 0.12006269 0.09991302]\n",
      " [0.0947483  0.10559257 0.08986076 0.10612343 0.09464517 0.0963543\n",
      "  0.11880008 0.08538435 0.10853712 0.09995391]\n",
      " [0.08661206 0.09738052 0.09092333 0.0966622  0.11212096 0.09477623\n",
      "  0.11236215 0.08964704 0.0987553  0.12076022]\n",
      " [0.09537663 0.11935516 0.0964144  0.08267498 0.10559988 0.08896369\n",
      "  0.11601085 0.08421206 0.11021059 0.10118175]\n",
      " [0.11451556 0.10409909 0.09486602 0.09807856 0.09887958 0.09353306\n",
      "  0.1118643  0.08886025 0.10439363 0.09090994]\n",
      " [0.09979061 0.10619726 0.08952374 0.10956133 0.11417096 0.09074245\n",
      "  0.1088932  0.0818211  0.10555698 0.09374239]\n",
      " [0.09439938 0.10263828 0.09884886 0.09806012 0.11061897 0.09627557\n",
      "  0.09552799 0.09911862 0.10682981 0.09768239]\n",
      " [0.10800968 0.1039876  0.09116798 0.0849143  0.09848712 0.08974826\n",
      "  0.10145899 0.10225826 0.10092401 0.1190438 ]\n",
      " [0.0941689  0.09598354 0.0895862  0.09497791 0.10684749 0.09207851\n",
      "  0.11761706 0.09871539 0.10591338 0.10411162]\n",
      " [0.0905313  0.10886604 0.10124651 0.08905306 0.10757225 0.09768457\n",
      "  0.10694807 0.0921307  0.10261793 0.10334957]\n",
      " [0.09708562 0.10148754 0.10108922 0.10113657 0.116177   0.09542947\n",
      "  0.10037385 0.08559286 0.09986911 0.10175875]\n",
      " [0.09183692 0.10997407 0.08564592 0.09474865 0.09663712 0.09449573\n",
      "  0.10363652 0.09573097 0.09701031 0.13028378]\n",
      " [0.1124959  0.10609615 0.09885341 0.09022708 0.09366332 0.09527963\n",
      "  0.10169407 0.09361589 0.10009647 0.10797808]\n",
      " [0.09420266 0.11305236 0.09460652 0.08648151 0.10336156 0.09410395\n",
      "  0.0938093  0.09584718 0.11493191 0.10960306]\n",
      " [0.0980584  0.10018187 0.10267142 0.09350969 0.10665868 0.08690385\n",
      "  0.1188909  0.10369141 0.09944995 0.08998384]\n",
      " [0.10359883 0.11117595 0.1054287  0.09665767 0.09838673 0.08719681\n",
      "  0.10209062 0.09040676 0.10055016 0.10450776]\n",
      " [0.09159476 0.09250153 0.09757083 0.09479299 0.10259298 0.10390594\n",
      "  0.10826209 0.09310457 0.0974631  0.1182112 ]\n",
      " [0.08122396 0.10393012 0.08625241 0.09560588 0.10103653 0.10501199\n",
      "  0.12698642 0.08352251 0.10987106 0.10655912]\n",
      " [0.09397625 0.11452813 0.09737421 0.09773371 0.10288914 0.10122814\n",
      "  0.10241811 0.08471533 0.0955111  0.10962588]\n",
      " [0.09239188 0.11052186 0.10250751 0.09366058 0.09583811 0.10660385\n",
      "  0.10035436 0.08670263 0.09639017 0.11502905]\n",
      " [0.1044539  0.1151024  0.0965653  0.09130865 0.10964574 0.08910983\n",
      "  0.11815419 0.10197975 0.08573949 0.08794075]\n",
      " [0.09188385 0.10962537 0.09730178 0.08918592 0.10777823 0.09315799\n",
      "  0.10626281 0.09201681 0.10523445 0.10755279]\n",
      " [0.09550755 0.10862979 0.08803579 0.09553154 0.10213688 0.0884909\n",
      "  0.11410283 0.09904535 0.10403965 0.10447971]\n",
      " [0.09521512 0.10132132 0.10423355 0.09007399 0.09640206 0.08918083\n",
      "  0.11084014 0.08481533 0.10453646 0.1233812 ]\n",
      " [0.08526295 0.09608917 0.09061173 0.11505567 0.11389106 0.0926889\n",
      "  0.10362813 0.08642153 0.09722852 0.11912234]\n",
      " [0.07396855 0.1103148  0.09508348 0.0968762  0.11366837 0.09464168\n",
      "  0.10716985 0.08770586 0.09492978 0.12564141]\n",
      " [0.08886303 0.11575272 0.08732203 0.09158052 0.11210833 0.10265178\n",
      "  0.11647526 0.0894429  0.10008232 0.09572111]\n",
      " [0.08965052 0.11199185 0.10507374 0.09276989 0.10638968 0.09543293\n",
      "  0.10863638 0.09445518 0.0951786  0.10042123]\n",
      " [0.09605399 0.11418501 0.08976658 0.09760738 0.10210148 0.10896917\n",
      "  0.11369458 0.09090873 0.09458935 0.09212372]\n",
      " [0.09295398 0.12157983 0.0966009  0.11234448 0.10350091 0.09448911\n",
      "  0.09073681 0.0889791  0.09538344 0.10343143]\n",
      " [0.0992292  0.10411416 0.10077048 0.09210735 0.10688191 0.09633086\n",
      "  0.10208047 0.08551825 0.10547996 0.10748737]\n",
      " [0.09292329 0.10720404 0.10647154 0.09826169 0.10599634 0.09654752\n",
      "  0.10201527 0.09039624 0.09831111 0.10187294]\n",
      " [0.09270798 0.09997264 0.10312681 0.09709978 0.11393955 0.09372155\n",
      "  0.12047115 0.08368083 0.09471804 0.10056167]\n",
      " [0.0863918  0.10445258 0.08959299 0.09814562 0.10706399 0.08538881\n",
      "  0.11800349 0.10073353 0.10955513 0.10067206]\n",
      " [0.08749469 0.09957681 0.11348129 0.10728215 0.09086827 0.09147892\n",
      "  0.10184625 0.08960641 0.10902185 0.10934335]\n",
      " [0.08707536 0.10238906 0.10628257 0.11272534 0.10499423 0.08756092\n",
      "  0.10777714 0.09965137 0.09545037 0.09609364]\n",
      " [0.10893197 0.10101914 0.09236178 0.10323309 0.10635331 0.0929814\n",
      "  0.10991328 0.08879271 0.09586776 0.10054556]\n",
      " [0.0969533  0.1070129  0.08613677 0.10095132 0.09842043 0.09344891\n",
      "  0.10713375 0.09842018 0.10217086 0.10935159]\n",
      " [0.09555021 0.10479591 0.09611552 0.10252688 0.10083071 0.09865398\n",
      "  0.11005369 0.09560355 0.09410776 0.10176179]\n",
      " [0.11384022 0.092363   0.10582292 0.09968034 0.08946071 0.09687642\n",
      "  0.10594417 0.10137866 0.09464192 0.09999165]\n",
      " [0.10139817 0.09798627 0.09650741 0.09847852 0.10199955 0.09949403\n",
      "  0.11783677 0.09268454 0.10055195 0.09306281]\n",
      " [0.09446877 0.10692553 0.09021398 0.09760961 0.09446337 0.10613143\n",
      "  0.10546649 0.09989559 0.09290416 0.11192106]\n",
      " [0.09731161 0.10603249 0.09940556 0.09253151 0.10832757 0.0909035\n",
      "  0.10209509 0.09481507 0.10199206 0.10658554]\n",
      " [0.10028537 0.09610658 0.09715567 0.10979483 0.10498327 0.09957377\n",
      "  0.1042187  0.09324852 0.09788918 0.0967441 ]\n",
      " [0.09454932 0.10817145 0.09686628 0.09696178 0.10347645 0.09126898\n",
      "  0.09919711 0.09305404 0.10580092 0.11065366]\n",
      " [0.07827387 0.1065571  0.09571014 0.10082965 0.11551979 0.09883065\n",
      "  0.11342947 0.09351312 0.0929614  0.10437481]\n",
      " [0.09590251 0.10348847 0.0996324  0.0978552  0.10072897 0.10229138\n",
      "  0.09687187 0.10347052 0.09873121 0.10102748]\n",
      " [0.11057171 0.10117211 0.10736092 0.09342645 0.10087998 0.09690783\n",
      "  0.10611133 0.09099696 0.09258132 0.09999139]\n",
      " [0.10454412 0.10395044 0.08921809 0.09067116 0.10573008 0.08873649\n",
      "  0.10628957 0.09970101 0.10109344 0.11006559]\n",
      " [0.09855295 0.11021116 0.09397325 0.10377473 0.1060722  0.08916308\n",
      "  0.09167295 0.08456185 0.10540898 0.11660887]\n",
      " [0.09728797 0.10453256 0.10185319 0.1104865  0.11023931 0.09203994\n",
      "  0.0935474  0.09416338 0.10186352 0.09398623]\n",
      " [0.09295906 0.11162004 0.09736292 0.09009586 0.11577565 0.09229629\n",
      "  0.1048711  0.08917237 0.09856902 0.10727769]\n",
      " [0.1024793  0.08940424 0.09413924 0.09636025 0.1218901  0.08557918\n",
      "  0.10766132 0.09306739 0.09644933 0.11296964]\n",
      " [0.09134337 0.12489963 0.09146204 0.09714808 0.09435443 0.08813218\n",
      "  0.10212342 0.10038661 0.11193452 0.09821573]\n",
      " [0.10113867 0.10878659 0.10133829 0.10174022 0.11297014 0.08514531\n",
      "  0.10819973 0.07617456 0.09541964 0.10908686]\n",
      " [0.10070698 0.10285231 0.10815808 0.10197219 0.09878374 0.08923299\n",
      "  0.09475313 0.10273015 0.09715025 0.10366018]\n",
      " [0.10038831 0.10868614 0.09722737 0.10448202 0.10334414 0.09895332\n",
      "  0.11798452 0.07967386 0.09454822 0.0947121 ]\n",
      " [0.10057622 0.12121428 0.0880874  0.08803828 0.11489456 0.08190068\n",
      "  0.10296746 0.08731879 0.11333585 0.10166647]\n",
      " [0.09600564 0.09100966 0.1005563  0.10327018 0.11472088 0.08946659\n",
      "  0.11948206 0.0899514  0.08921573 0.10632157]\n",
      " [0.09991464 0.09179221 0.0936804  0.10249418 0.10819188 0.10120367\n",
      "  0.11566852 0.09721698 0.10077859 0.08905894]\n",
      " [0.09646741 0.1079155  0.09789703 0.09571213 0.09721248 0.09391937\n",
      "  0.10749795 0.08280743 0.11577295 0.10479776]\n",
      " [0.09018735 0.09800921 0.10858019 0.10617689 0.10982273 0.08764764\n",
      "  0.11432342 0.07520227 0.10140945 0.10864085]\n",
      " [0.08887447 0.10932127 0.10059369 0.1119908  0.11376563 0.09361004\n",
      "  0.09507393 0.08856466 0.0991256  0.0990799 ]\n",
      " [0.09722756 0.10860642 0.1042145  0.08870128 0.09937376 0.09167104\n",
      "  0.11876757 0.08891786 0.09951743 0.10300258]\n",
      " [0.0903348  0.10332053 0.11061186 0.08918277 0.10249101 0.09458056\n",
      "  0.11962271 0.0984454  0.09055931 0.10085106]\n",
      " [0.09385718 0.09155867 0.10392607 0.10511389 0.1210901  0.07807379\n",
      "  0.11251689 0.09772972 0.10216335 0.09397035]\n",
      " [0.09261351 0.09763346 0.10000158 0.08757564 0.0956023  0.09441558\n",
      "  0.13046179 0.091022   0.10446231 0.10621182]\n",
      " [0.09315536 0.11470956 0.09584509 0.09361324 0.09882076 0.08776003\n",
      "  0.10295643 0.1047752  0.10043726 0.10792707]\n",
      " [0.0929582  0.11178608 0.09576806 0.10147624 0.10348126 0.0912181\n",
      "  0.1140677  0.08207485 0.0946342  0.11253532]\n",
      " [0.08827659 0.09063991 0.10385596 0.09931351 0.10007676 0.08917546\n",
      "  0.11083733 0.10590822 0.09557837 0.11633789]\n",
      " [0.10413843 0.08899973 0.10410738 0.09766492 0.1035809  0.09785805\n",
      "  0.09896637 0.08813108 0.10063815 0.11591499]\n",
      " [0.09758995 0.11820052 0.08867601 0.09930084 0.09936283 0.09926643\n",
      "  0.10963831 0.08550474 0.09734591 0.10511445]\n",
      " [0.09483543 0.11003301 0.10043655 0.09545149 0.09726846 0.09103154\n",
      "  0.11023423 0.0923225  0.09607449 0.1123123 ]\n",
      " [0.10448742 0.11044857 0.10445046 0.09537101 0.10056577 0.08737456\n",
      "  0.10182321 0.09732159 0.1011315  0.09702592]\n",
      " [0.09894384 0.09846927 0.10045283 0.09518238 0.10473858 0.10246491\n",
      "  0.09721852 0.09923841 0.09683206 0.10645919]\n",
      " [0.10004557 0.10672931 0.10314791 0.08824492 0.11308077 0.09093217\n",
      "  0.12165347 0.07175125 0.10646572 0.09794891]\n",
      " [0.08840455 0.11237755 0.08739058 0.10575499 0.11378668 0.09859786\n",
      "  0.10739769 0.10141958 0.08455898 0.10031155]\n",
      " [0.10950416 0.1101853  0.09012751 0.09297629 0.11263701 0.09681289\n",
      "  0.09250437 0.08523704 0.10711681 0.10289863]\n",
      " [0.10359152 0.10852063 0.09341937 0.09753452 0.11283373 0.09087059\n",
      "  0.09844876 0.09845599 0.09601428 0.10031062]\n",
      " [0.10538985 0.10636903 0.08982122 0.08673237 0.1128366  0.09508881\n",
      "  0.10763509 0.09634737 0.09480665 0.10497302]\n",
      " [0.09831592 0.12147545 0.10231501 0.09394292 0.10684014 0.09331947\n",
      "  0.10766518 0.08890819 0.09823216 0.08898555]\n",
      " [0.09856771 0.11278543 0.11235033 0.09441918 0.09413362 0.09524211\n",
      "  0.10160948 0.09748546 0.09319189 0.1002148 ]\n",
      " [0.09642306 0.10836466 0.09522329 0.09219291 0.11049547 0.08007528\n",
      "  0.11552695 0.09826111 0.09783968 0.10559759]\n",
      " [0.09411785 0.1146997  0.09739953 0.09193413 0.10020147 0.09970039\n",
      "  0.10854266 0.08522026 0.09556837 0.11261564]\n",
      " [0.0921915  0.0941947  0.11232039 0.1026331  0.09464175 0.10075026\n",
      "  0.11183544 0.09136612 0.0981962  0.10187053]\n",
      " [0.10266657 0.10619774 0.09883501 0.08974674 0.09794813 0.10330796\n",
      "  0.09549274 0.09488134 0.10021721 0.11070656]\n",
      " [0.10197632 0.10434283 0.10449218 0.09614237 0.10163906 0.09104328\n",
      "  0.10872928 0.08328536 0.10880819 0.09954114]\n",
      " [0.09961369 0.1184     0.08579806 0.10092974 0.10336376 0.09507896\n",
      "  0.10046585 0.0978044  0.09930835 0.09923721]\n",
      " [0.09566513 0.11502464 0.0979642  0.09125644 0.10029846 0.09905531\n",
      "  0.11207256 0.08477124 0.09679596 0.10709607]\n",
      " [0.08219521 0.10982269 0.10324277 0.10412451 0.09105328 0.11883177\n",
      "  0.09813625 0.08417882 0.09244371 0.11597099]\n",
      " [0.08890651 0.10995524 0.11812347 0.09883703 0.10404919 0.08426264\n",
      "  0.10539093 0.09968179 0.08513963 0.10565356]\n",
      " [0.10360387 0.09792383 0.09417884 0.0971562  0.09095821 0.1023554\n",
      "  0.11215807 0.09464408 0.10261455 0.10440695]\n",
      " [0.09772701 0.11227857 0.09904381 0.09436996 0.10051113 0.09548955\n",
      "  0.10081396 0.09901934 0.10253185 0.0982148 ]\n",
      " [0.10344517 0.08712689 0.08858476 0.10309397 0.09967446 0.10453612\n",
      "  0.12189656 0.09095669 0.09567224 0.10501313]\n",
      " [0.08482368 0.10467159 0.09211965 0.11009122 0.1012572  0.09420553\n",
      "  0.10822885 0.08789017 0.10593124 0.11078086]] (18.597 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 2.305206298828125, step = 101 (38.026 sec)\n",
      "INFO:tensorflow:probabilities = [[0.11161921 0.08057046 0.0994389  0.09346912 0.11075594 0.09434979\n",
      "  0.1120339  0.08882454 0.09737912 0.11155902]\n",
      " [0.09596016 0.10926549 0.09903363 0.10823471 0.10993965 0.09343465\n",
      "  0.09586058 0.09160284 0.09095405 0.10571425]\n",
      " [0.10227139 0.11006905 0.09801046 0.09957789 0.10438735 0.09181699\n",
      "  0.10525362 0.08816053 0.09450363 0.1059491 ]\n",
      " [0.09975787 0.11247331 0.09039749 0.0991134  0.10160263 0.09843848\n",
      "  0.1017482  0.09497388 0.09792441 0.10357033]\n",
      " [0.10305566 0.10108461 0.10605245 0.09733682 0.09768492 0.09180533\n",
      "  0.10126729 0.08430429 0.1079302  0.10947844]\n",
      " [0.11544857 0.09036746 0.10454513 0.08488042 0.099274   0.0949177\n",
      "  0.11203393 0.09910099 0.09572781 0.10370398]\n",
      " [0.10277044 0.11231499 0.09754805 0.10642093 0.10259929 0.09633377\n",
      "  0.0993438  0.08479529 0.09933239 0.09854105]\n",
      " [0.09547645 0.10823457 0.10347744 0.09912415 0.09726053 0.09262797\n",
      "  0.10986357 0.07888519 0.10534413 0.109706  ]\n",
      " [0.09786001 0.10070087 0.0934616  0.09918497 0.10678103 0.09697042\n",
      "  0.11446547 0.09309167 0.09833538 0.09914858]\n",
      " [0.10985016 0.10773832 0.09487821 0.09003923 0.10158127 0.07556493\n",
      "  0.10422655 0.09518913 0.10672372 0.11420848]\n",
      " [0.08776549 0.11284018 0.10103478 0.09260258 0.10276252 0.08897199\n",
      "  0.1031349  0.10316709 0.09814534 0.10957513]\n",
      " [0.10017351 0.11150094 0.10320115 0.09733921 0.10455124 0.09521346\n",
      "  0.10186594 0.09150483 0.09785833 0.09679139]\n",
      " [0.07876518 0.11123539 0.10965202 0.08718941 0.10575271 0.08992239\n",
      "  0.10766413 0.09802346 0.10364817 0.10814713]\n",
      " [0.09719698 0.11630887 0.09949187 0.0973434  0.10508053 0.09091719\n",
      "  0.10019774 0.09479102 0.10441416 0.09425824]\n",
      " [0.09539827 0.10233012 0.0972509  0.09430599 0.12292407 0.0992439\n",
      "  0.10011242 0.08711036 0.09814878 0.1031752 ]\n",
      " [0.07788094 0.09456552 0.1046588  0.11260943 0.10608863 0.08941814\n",
      "  0.11295667 0.07824124 0.09992115 0.12365947]\n",
      " [0.09793417 0.10557808 0.09275288 0.08749375 0.10661906 0.10316839\n",
      "  0.10354796 0.08720159 0.09898476 0.11671935]\n",
      " [0.10184451 0.10289647 0.1039014  0.09362969 0.10410685 0.09415209\n",
      "  0.09971717 0.08667326 0.10354973 0.10952881]\n",
      " [0.09860688 0.11875332 0.09737895 0.09728137 0.09674704 0.09691562\n",
      "  0.10188814 0.09623719 0.10022668 0.09596481]\n",
      " [0.10316743 0.10860897 0.09228717 0.10033245 0.10554635 0.09454717\n",
      "  0.10311949 0.09751778 0.09011196 0.10476123]\n",
      " [0.09772167 0.09042417 0.09956084 0.08432688 0.11672748 0.0988642\n",
      "  0.11021633 0.10050892 0.09369317 0.10795635]\n",
      " [0.10881696 0.09993882 0.1030933  0.10305809 0.09698876 0.10630361\n",
      "  0.10788972 0.07570445 0.08925389 0.1089524 ]\n",
      " [0.09047416 0.11198561 0.10446139 0.09647608 0.10300684 0.09358788\n",
      "  0.09933344 0.09680762 0.09560481 0.10826218]\n",
      " [0.08455178 0.1123099  0.10433438 0.10140989 0.09581517 0.0825908\n",
      "  0.11149891 0.09359452 0.1098224  0.10407225]\n",
      " [0.10141597 0.09704294 0.09571661 0.11693705 0.09446328 0.09994231\n",
      "  0.10733422 0.09660639 0.0955739  0.09496732]\n",
      " [0.10476671 0.10906807 0.09769183 0.08928141 0.09176254 0.08699003\n",
      "  0.09812904 0.09687354 0.11509862 0.11033821]\n",
      " [0.09334691 0.10390366 0.0879527  0.09449096 0.09918384 0.09921262\n",
      "  0.10433469 0.08537487 0.10686124 0.12533851]\n",
      " [0.07871267 0.105509   0.10194037 0.11410452 0.09955462 0.09439932\n",
      "  0.10509756 0.09764988 0.10396932 0.09906275]\n",
      " [0.11869335 0.09569907 0.11333275 0.10329745 0.09217544 0.0839959\n",
      "  0.10269956 0.09528278 0.10248928 0.09233442]\n",
      " [0.09339742 0.09693747 0.10251755 0.09766815 0.09862707 0.07856077\n",
      "  0.12492816 0.09667093 0.09644434 0.11424814]\n",
      " [0.11059268 0.10512396 0.10756971 0.09793971 0.09889011 0.0853519\n",
      "  0.10184427 0.08687944 0.10764175 0.09816647]\n",
      " [0.09607529 0.09699071 0.09318832 0.09834385 0.11597683 0.08767048\n",
      "  0.11164596 0.08597261 0.11111533 0.10302062]\n",
      " [0.09397345 0.10466477 0.10290946 0.0991876  0.10423602 0.09276109\n",
      "  0.0959682  0.08816891 0.11014665 0.10798385]\n",
      " [0.10761354 0.1128548  0.0918648  0.10872164 0.09178705 0.09470183\n",
      "  0.09501629 0.09620542 0.09736652 0.10386811]\n",
      " [0.08840972 0.12096136 0.10670604 0.09816697 0.10607057 0.09344852\n",
      "  0.09319143 0.09664839 0.09804748 0.09834953]\n",
      " [0.10492154 0.10425182 0.09768938 0.09874146 0.09673866 0.0911337\n",
      "  0.10599826 0.09510005 0.10754462 0.09788052]\n",
      " [0.09885091 0.0932974  0.09193874 0.09569682 0.11441029 0.09925322\n",
      "  0.1109186  0.08064063 0.11479976 0.10019362]\n",
      " [0.11906262 0.10448142 0.09029598 0.08038864 0.11659712 0.09952934\n",
      "  0.1007887  0.08346826 0.10773242 0.0976555 ]\n",
      " [0.08421596 0.10956927 0.10435974 0.10260153 0.10277808 0.10677052\n",
      "  0.10657972 0.08764903 0.09064759 0.10482855]\n",
      " [0.0949675  0.10672386 0.10109054 0.09721216 0.10592278 0.09199314\n",
      "  0.10618399 0.08811444 0.10618349 0.1016081 ]\n",
      " [0.10825162 0.10263069 0.09983715 0.09502044 0.103122   0.08786359\n",
      "  0.11243923 0.10070685 0.09014876 0.09997967]\n",
      " [0.10202035 0.0989255  0.10539001 0.10904336 0.09094814 0.10303239\n",
      "  0.10237913 0.07616014 0.11266673 0.09943425]\n",
      " [0.10917609 0.10528048 0.11653934 0.08758443 0.09524087 0.09222692\n",
      "  0.11274329 0.08943411 0.08509857 0.1066759 ]\n",
      " [0.09532955 0.10972529 0.09591192 0.094045   0.10705414 0.0839324\n",
      "  0.10535331 0.09855038 0.10974114 0.10035687]\n",
      " [0.08539806 0.11525411 0.08620681 0.1012888  0.10878868 0.08873976\n",
      "  0.09655622 0.09599337 0.10075927 0.1210149 ]\n",
      " [0.08810895 0.09426686 0.09418897 0.09835702 0.11090891 0.10030584\n",
      "  0.12154804 0.09346958 0.09595966 0.10288615]\n",
      " [0.10525276 0.11662954 0.10016523 0.08907795 0.09441882 0.10658649\n",
      "  0.11254127 0.08035086 0.09408831 0.10088877]\n",
      " [0.09933532 0.10937603 0.09982142 0.09706491 0.10476343 0.09739581\n",
      "  0.10423932 0.09927585 0.09485058 0.09387734]\n",
      " [0.09849846 0.09762784 0.10096708 0.09931202 0.10045985 0.09722284\n",
      "  0.11863643 0.10087849 0.09281205 0.09358495]\n",
      " [0.08867509 0.1111658  0.09130287 0.10367386 0.09897446 0.10961584\n",
      "  0.0996562  0.08060254 0.1066205  0.10971283]\n",
      " [0.10295453 0.11073569 0.10079014 0.08661759 0.10568937 0.08587435\n",
      "  0.1139303  0.09957991 0.08552191 0.1083062 ]\n",
      " [0.08342073 0.11384351 0.08574986 0.11042262 0.11982751 0.09312406\n",
      "  0.11106609 0.09425908 0.08798314 0.10030338]\n",
      " [0.10985861 0.10068375 0.10044735 0.10490249 0.10343483 0.09554008\n",
      "  0.10366734 0.08895097 0.0852031  0.10731147]\n",
      " [0.10155509 0.10925139 0.10994073 0.09572938 0.10088946 0.10075994\n",
      "  0.11311923 0.07078452 0.1000039  0.09796636]\n",
      " [0.09962698 0.09329988 0.11991321 0.08319362 0.11991871 0.11263042\n",
      "  0.10415203 0.07804052 0.08907276 0.10015189]\n",
      " [0.10913395 0.10027067 0.1002002  0.09579439 0.10317581 0.08390381\n",
      "  0.11500603 0.09015758 0.10559259 0.09676496]\n",
      " [0.09386499 0.10893426 0.09821975 0.0969244  0.1123161  0.09797696\n",
      "  0.10014929 0.10216183 0.09268627 0.09676613]\n",
      " [0.10058762 0.11927029 0.09458956 0.10035543 0.0872551  0.10334258\n",
      "  0.10554523 0.08328245 0.10289807 0.10287368]\n",
      " [0.09303302 0.10243072 0.09685658 0.09873124 0.10533772 0.10660159\n",
      "  0.1027143  0.0890042  0.10494645 0.10034418]\n",
      " [0.11154513 0.09164657 0.10529529 0.08751468 0.10426614 0.09394763\n",
      "  0.12115317 0.08917743 0.08196574 0.11348823]\n",
      " [0.11070806 0.09307651 0.08566563 0.0980089  0.10789738 0.10720589\n",
      "  0.10772186 0.08760816 0.0898073  0.11230031]\n",
      " [0.09149744 0.11898065 0.11024871 0.0979505  0.09949757 0.0919429\n",
      "  0.09935197 0.09163463 0.10255515 0.09634049]\n",
      " [0.08829023 0.09326542 0.10006089 0.10525205 0.11731187 0.10070425\n",
      "  0.11916972 0.07450254 0.09045014 0.11099288]\n",
      " [0.09710393 0.10824746 0.10631595 0.0979511  0.09886691 0.08232092\n",
      "  0.11647325 0.09506883 0.09430169 0.10334998]\n",
      " [0.08900149 0.10211129 0.10170007 0.0991644  0.09209735 0.09653108\n",
      "  0.10678535 0.08460674 0.10944435 0.11855789]\n",
      " [0.09714416 0.08898615 0.08853889 0.09176016 0.10351096 0.11382108\n",
      "  0.11491504 0.10541811 0.09158553 0.10431992]\n",
      " [0.11607719 0.09627682 0.09800479 0.08573327 0.1011409  0.08483598\n",
      "  0.11243311 0.08862961 0.10836906 0.10849928]\n",
      " [0.1070007  0.10515999 0.08962042 0.09193411 0.11025816 0.08998713\n",
      "  0.12071945 0.08218757 0.09463891 0.10849356]\n",
      " [0.09295159 0.10731795 0.10128881 0.09502911 0.11138154 0.08485148\n",
      "  0.11270277 0.09663929 0.09817521 0.09966225]\n",
      " [0.09248873 0.11756776 0.10498594 0.09082685 0.10527686 0.08910835\n",
      "  0.10694506 0.09278553 0.09727534 0.10273958]\n",
      " [0.10141401 0.09879574 0.0944611  0.09570293 0.11613281 0.10160084\n",
      "  0.09687147 0.09333462 0.11081686 0.09086963]\n",
      " [0.09162965 0.10311498 0.10057616 0.09783385 0.09803813 0.10395207\n",
      "  0.11843078 0.0895475  0.09474103 0.10213585]\n",
      " [0.10806361 0.11255052 0.09975894 0.09745676 0.09857271 0.09937971\n",
      "  0.09679935 0.09555203 0.09604159 0.09582478]\n",
      " [0.09251902 0.10007141 0.09900029 0.10053516 0.10600413 0.10288982\n",
      "  0.11012972 0.09373225 0.09626629 0.09885191]\n",
      " [0.09663516 0.1138746  0.08791787 0.10342956 0.11498346 0.08745116\n",
      "  0.10513418 0.08927929 0.10510822 0.0961865 ]\n",
      " [0.09401173 0.09743564 0.09466762 0.10819939 0.12204809 0.09061213\n",
      "  0.1022163  0.09241412 0.10436443 0.09403054]\n",
      " [0.0921019  0.10631664 0.09440919 0.09835972 0.10469996 0.08572202\n",
      "  0.11242313 0.10548427 0.09387672 0.10660645]\n",
      " [0.09049221 0.10132854 0.0964146  0.09614049 0.10139895 0.09494327\n",
      "  0.11687035 0.09587524 0.1009868  0.10554954]\n",
      " [0.10449447 0.09800022 0.11469175 0.09276384 0.08906331 0.0944725\n",
      "  0.10761363 0.0855949  0.10624688 0.1070585 ]\n",
      " [0.09769886 0.12092114 0.09479293 0.08735478 0.11390491 0.09458877\n",
      "  0.10359666 0.08810153 0.09970777 0.09933263]\n",
      " [0.09540786 0.11167137 0.10844408 0.10254094 0.09720516 0.09095939\n",
      "  0.09985735 0.09181125 0.1116773  0.09042529]\n",
      " [0.11178671 0.10502806 0.09816663 0.09682806 0.0936883  0.0929877\n",
      "  0.10695008 0.08687192 0.11076773 0.09692482]\n",
      " [0.11128913 0.09723096 0.09457755 0.10783296 0.10861085 0.08685733\n",
      "  0.1128015  0.0830101  0.10336577 0.09442385]\n",
      " [0.09810581 0.11688624 0.11674253 0.10369467 0.09110084 0.08435683\n",
      "  0.10863194 0.07633058 0.09673179 0.10741877]\n",
      " [0.10234977 0.10480113 0.09939018 0.10399816 0.10957945 0.09653818\n",
      "  0.10939141 0.09093462 0.09771308 0.08530402]\n",
      " [0.11245064 0.11792843 0.09009525 0.10033874 0.10120882 0.08611603\n",
      "  0.10055489 0.09030804 0.10266843 0.09833073]\n",
      " [0.10498877 0.10172763 0.10459588 0.09378875 0.10354305 0.09274023\n",
      "  0.10950606 0.08655922 0.09556267 0.10698774]\n",
      " [0.09280922 0.10440684 0.09496302 0.09273755 0.10455042 0.08612391\n",
      "  0.10476024 0.09983752 0.10928792 0.11052336]\n",
      " [0.09263556 0.11906534 0.09501645 0.09553706 0.10217977 0.08957103\n",
      "  0.10028858 0.09312991 0.11051525 0.10206105]\n",
      " [0.09578623 0.10023917 0.12160831 0.09202557 0.10875709 0.08194229\n",
      "  0.11309235 0.07970752 0.10544352 0.10139794]\n",
      " [0.09968781 0.10080362 0.10202571 0.09868785 0.09176862 0.09861799\n",
      "  0.10990692 0.08915324 0.11151257 0.09783566]\n",
      " [0.10582433 0.08636442 0.09507713 0.08829882 0.1174051  0.09515626\n",
      "  0.10572104 0.10794764 0.09367953 0.10452574]\n",
      " [0.1003096  0.10594337 0.09912345 0.0914579  0.10993621 0.10042351\n",
      "  0.10126056 0.09354604 0.09045255 0.10754679]\n",
      " [0.09524323 0.09549813 0.09717776 0.11042365 0.10694019 0.09552332\n",
      "  0.10281286 0.09686337 0.0948038  0.10471369]\n",
      " [0.08898164 0.10239967 0.09903575 0.0914782  0.10447193 0.09780541\n",
      "  0.11075357 0.09241482 0.10280808 0.10985093]\n",
      " [0.10477255 0.09982948 0.10922836 0.11010185 0.09976158 0.08291126\n",
      "  0.10028712 0.091319   0.09653388 0.10525492]\n",
      " [0.10182183 0.11181693 0.09360824 0.09802978 0.10253753 0.09572517\n",
      "  0.12200929 0.0813066  0.09235728 0.10078735]\n",
      " [0.09560319 0.09778755 0.10228929 0.08502724 0.12196217 0.08351053\n",
      "  0.09908935 0.09903056 0.10881937 0.10688076]\n",
      " [0.08773682 0.10566078 0.09750912 0.11381715 0.09784162 0.09660245\n",
      "  0.11267516 0.08967993 0.10155876 0.0969182 ]\n",
      " [0.08532762 0.1042257  0.09779287 0.08945929 0.11806857 0.0971001\n",
      "  0.09852431 0.09402216 0.11104264 0.10443674]] (19.963 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 2.49772\n",
      "INFO:tensorflow:probabilities = [[0.09618051 0.11333042 0.09681423 0.10213157 0.09754125 0.07517962\n",
      "  0.10612432 0.10738283 0.09960186 0.10571339]\n",
      " [0.11885299 0.09311653 0.11239283 0.09589802 0.10864241 0.09894172\n",
      "  0.09498059 0.09081223 0.08867836 0.09768431]\n",
      " [0.0851925  0.10295272 0.10020238 0.09144596 0.12191371 0.10631884\n",
      "  0.09848759 0.09264038 0.1030152  0.09783073]\n",
      " [0.10190065 0.09106947 0.11358515 0.10386782 0.09877861 0.07968473\n",
      "  0.10421464 0.09601401 0.10182035 0.10906457]\n",
      " [0.09843533 0.10299904 0.10726394 0.10401963 0.09897475 0.08264756\n",
      "  0.10430107 0.08448147 0.10445194 0.11242527]\n",
      " [0.09898605 0.10793448 0.10301802 0.09581081 0.08263798 0.09586277\n",
      "  0.10646926 0.09277058 0.11348585 0.10302419]\n",
      " [0.09874667 0.10791901 0.10422821 0.09205585 0.09750872 0.08964512\n",
      "  0.10634513 0.10639266 0.09601718 0.10114143]\n",
      " [0.10187298 0.10537817 0.10022145 0.08996586 0.09753465 0.09345467\n",
      "  0.10661632 0.10562289 0.10325421 0.0960788 ]\n",
      " [0.10194473 0.09261611 0.09609165 0.09644552 0.11200198 0.09942974\n",
      "  0.10386895 0.09864546 0.09607523 0.10288063]\n",
      " [0.09681634 0.13948313 0.09374399 0.09655018 0.09011181 0.08589811\n",
      "  0.10435807 0.0886598  0.11406999 0.09030858]\n",
      " [0.10389859 0.1123091  0.09511043 0.09337233 0.10116443 0.08851015\n",
      "  0.1238475  0.08886468 0.09247869 0.10044409]\n",
      " [0.10047134 0.10033728 0.1162208  0.09959163 0.09729794 0.10502352\n",
      "  0.11298649 0.0714467  0.0838151  0.11280921]\n",
      " [0.10185069 0.10208048 0.09897389 0.10239626 0.10336368 0.08895697\n",
      "  0.12244943 0.08532517 0.09698681 0.09761662]\n",
      " [0.08584505 0.10882213 0.09744507 0.09921132 0.10644786 0.10047726\n",
      "  0.10401076 0.07970933 0.10787407 0.11015714]\n",
      " [0.10315026 0.09880088 0.08968782 0.08599854 0.11381713 0.08986959\n",
      "  0.10208158 0.10101671 0.10443879 0.1111387 ]\n",
      " [0.09628198 0.10414554 0.10719913 0.10474119 0.09700784 0.08679062\n",
      "  0.10398949 0.09104155 0.10545289 0.10334976]\n",
      " [0.11035006 0.08974754 0.09916686 0.08782354 0.12834514 0.09245428\n",
      "  0.10764584 0.09020344 0.09626274 0.09800055]\n",
      " [0.10796649 0.09270994 0.10212944 0.09706486 0.11304152 0.0940077\n",
      "  0.10555126 0.0834184  0.10044966 0.10366074]\n",
      " [0.09995749 0.09785488 0.10050501 0.10246595 0.10031559 0.09615886\n",
      "  0.11126656 0.09379749 0.09635133 0.10132684]\n",
      " [0.09699869 0.10176308 0.09657833 0.1022564  0.09365304 0.0946173\n",
      "  0.10268616 0.10536014 0.09956896 0.1065179 ]\n",
      " [0.1003394  0.11959324 0.10764892 0.09524034 0.10208742 0.08647139\n",
      "  0.10768791 0.10075248 0.09134055 0.08883835]\n",
      " [0.10190408 0.09988024 0.10337806 0.10788584 0.09880259 0.09004448\n",
      "  0.10215296 0.08705572 0.10434344 0.10455259]\n",
      " [0.10596829 0.10726033 0.08609085 0.10024208 0.09842753 0.0995541\n",
      "  0.09464723 0.11239417 0.1036628  0.09175261]\n",
      " [0.09547941 0.10931487 0.09154712 0.09385693 0.11022605 0.09678933\n",
      "  0.11353335 0.08825273 0.09603997 0.10496025]\n",
      " [0.1175406  0.10279264 0.09838361 0.093013   0.10943484 0.08526512\n",
      "  0.12152694 0.09705489 0.08089698 0.09409137]\n",
      " [0.09920524 0.1108083  0.10897532 0.09759585 0.09423793 0.09330223\n",
      "  0.10424329 0.08629626 0.1014568  0.10387877]\n",
      " [0.10378664 0.09297281 0.10716715 0.10333068 0.09812628 0.08920725\n",
      "  0.10585101 0.08623156 0.1123439  0.10098275]\n",
      " [0.10121331 0.10101935 0.10271234 0.10623434 0.09992566 0.09368366\n",
      "  0.10551184 0.08654726 0.09493406 0.10821819]\n",
      " [0.08640139 0.10696379 0.11194438 0.09974544 0.10559647 0.10101501\n",
      "  0.1026151  0.07795982 0.10609268 0.10166591]\n",
      " [0.1120674  0.08927345 0.11280418 0.09016516 0.09167073 0.08313461\n",
      "  0.10613753 0.09016371 0.11041484 0.1141684 ]\n",
      " [0.09624743 0.08825596 0.10119763 0.11789844 0.10711622 0.08543491\n",
      "  0.10666294 0.08929277 0.09916438 0.10872933]\n",
      " [0.09620747 0.09997286 0.10199628 0.10539099 0.10672424 0.10421392\n",
      "  0.09475965 0.09651927 0.09670485 0.09751046]\n",
      " [0.09990495 0.10143626 0.09547243 0.10777951 0.11419049 0.0781187\n",
      "  0.11030441 0.09580198 0.08784594 0.10914533]\n",
      " [0.0947556  0.10418681 0.1077265  0.1045856  0.10536034 0.08966131\n",
      "  0.09512788 0.08716622 0.10129052 0.11013923]\n",
      " [0.09569385 0.09903431 0.10454994 0.10352079 0.10660521 0.08149032\n",
      "  0.11829598 0.09106069 0.09229765 0.10745124]\n",
      " [0.10520795 0.09589624 0.11822158 0.08744849 0.09040287 0.08042975\n",
      "  0.1107233  0.09505913 0.11539258 0.10121811]\n",
      " [0.10567681 0.10458103 0.10230054 0.09531295 0.09355027 0.08804716\n",
      "  0.11530539 0.0834709  0.10058577 0.11116919]\n",
      " [0.10206452 0.10254663 0.10561123 0.10333028 0.10102101 0.09785936\n",
      "  0.11921356 0.08436542 0.0912383  0.09274968]\n",
      " [0.10110402 0.10172191 0.09390056 0.0985851  0.10619507 0.10279633\n",
      "  0.10382448 0.09035655 0.09214735 0.10936863]\n",
      " [0.09511481 0.09962487 0.10493741 0.09467085 0.10044377 0.08790306\n",
      "  0.11432353 0.09296073 0.09603047 0.11399049]\n",
      " [0.08701386 0.09467847 0.11319948 0.10551415 0.10507774 0.0914449\n",
      "  0.10904725 0.09050032 0.09860801 0.10491583]\n",
      " [0.09747659 0.11884903 0.11612715 0.09557448 0.10506683 0.09072165\n",
      "  0.08891064 0.09338357 0.09191401 0.10197605]\n",
      " [0.10507719 0.0934043  0.09066115 0.08151309 0.11677983 0.10212559\n",
      "  0.1082215  0.08998011 0.1066935  0.10554374]\n",
      " [0.09976666 0.10277488 0.10212915 0.09258287 0.09972377 0.09329667\n",
      "  0.11355916 0.09093485 0.09545889 0.10977309]\n",
      " [0.08291657 0.09830007 0.09924978 0.11065487 0.10019926 0.10191158\n",
      "  0.10801961 0.1062465  0.09560615 0.09689561]\n",
      " [0.10461146 0.0900428  0.08102187 0.10452271 0.10780026 0.08979201\n",
      "  0.10988234 0.09109124 0.11448057 0.10675474]\n",
      " [0.089084   0.11032483 0.11039502 0.08634919 0.11585049 0.08515034\n",
      "  0.11385581 0.07735965 0.09937412 0.11225655]\n",
      " [0.09743344 0.09151799 0.11066318 0.09251268 0.12511515 0.07853269\n",
      "  0.12915951 0.07638783 0.08280899 0.11586855]\n",
      " [0.0967204  0.10592732 0.09915842 0.09134975 0.10762198 0.10866159\n",
      "  0.09856586 0.09355819 0.09811745 0.10031904]\n",
      " [0.0989644  0.10578814 0.11284196 0.09680296 0.09989971 0.08997489\n",
      "  0.10834185 0.09221948 0.09627081 0.0988958 ]\n",
      " [0.1080623  0.10027725 0.11762203 0.10111224 0.09217879 0.09371152\n",
      "  0.10317881 0.08048088 0.09803637 0.10533982]\n",
      " [0.09760663 0.09964254 0.10422319 0.08465553 0.10538521 0.09745054\n",
      "  0.11218815 0.09316602 0.10093995 0.10474222]\n",
      " [0.0973283  0.10816326 0.08879256 0.09522422 0.09677717 0.11192001\n",
      "  0.10842972 0.08586814 0.10079194 0.10670466]\n",
      " [0.10497078 0.09196954 0.10799908 0.09492959 0.09145741 0.09074806\n",
      "  0.11959477 0.08733327 0.0986817  0.1123158 ]\n",
      " [0.09105924 0.11403061 0.10041307 0.09967184 0.10507721 0.09003845\n",
      "  0.10385411 0.09274432 0.1002514  0.10285974]\n",
      " [0.09727947 0.09350588 0.10434481 0.0983979  0.11024759 0.08805222\n",
      "  0.10844464 0.09736873 0.08924413 0.11311464]\n",
      " [0.1246372  0.09732338 0.10274136 0.08329675 0.09959453 0.07934263\n",
      "  0.12009926 0.08751342 0.09885831 0.10659316]\n",
      " [0.11561513 0.10613446 0.10735531 0.0982569  0.09025642 0.08904533\n",
      "  0.11383133 0.08372838 0.10179195 0.09398477]\n",
      " [0.09557712 0.10057792 0.10744734 0.0999426  0.10154818 0.09043757\n",
      "  0.11358694 0.08278854 0.09661473 0.11147906]\n",
      " [0.08333049 0.10351926 0.10116435 0.09275531 0.10953975 0.09012791\n",
      "  0.10755636 0.09701327 0.10101016 0.11398315]\n",
      " [0.09920536 0.11253804 0.11442086 0.09372434 0.10644394 0.10290457\n",
      "  0.09709483 0.07865156 0.0888694  0.10614711]\n",
      " [0.10640364 0.09836483 0.09413923 0.08803644 0.11821585 0.09167422\n",
      "  0.11875486 0.09249325 0.08604208 0.1058756 ]\n",
      " [0.09636888 0.09695516 0.10732024 0.11204519 0.10163271 0.08509583\n",
      "  0.09723509 0.0847467  0.10054295 0.11805725]\n",
      " [0.10689968 0.10718925 0.09349567 0.09917087 0.09226612 0.10412045\n",
      "  0.09094785 0.09946749 0.1030521  0.10339053]\n",
      " [0.10155831 0.09795242 0.10423224 0.09809071 0.10833924 0.08646045\n",
      "  0.10044604 0.09020743 0.10204812 0.11066503]\n",
      " [0.08713351 0.10747791 0.09345602 0.0976688  0.10264187 0.08964077\n",
      "  0.11552745 0.10077807 0.09950603 0.10616957]\n",
      " [0.09714125 0.10956812 0.09007561 0.09221556 0.11209168 0.10163194\n",
      "  0.11376117 0.10772509 0.08303845 0.09275113]\n",
      " [0.10833939 0.10857636 0.08540585 0.0934673  0.10588805 0.10577881\n",
      "  0.11057908 0.08136485 0.10632268 0.09427764]\n",
      " [0.09965624 0.11749545 0.10447446 0.09772946 0.10353476 0.08285866\n",
      "  0.09991748 0.09725023 0.102111   0.09497225]\n",
      " [0.10728762 0.10004221 0.1071147  0.10889958 0.10076531 0.08710693\n",
      "  0.0979529  0.09076793 0.10037401 0.09968878]\n",
      " [0.09554518 0.09359338 0.11030391 0.0822164  0.10259901 0.09545223\n",
      "  0.1111594  0.09046404 0.10688627 0.11178018]\n",
      " [0.08372585 0.11038784 0.10972339 0.09029688 0.11034039 0.10159553\n",
      "  0.09177076 0.10293814 0.09875638 0.10046483]\n",
      " [0.1133167  0.09560023 0.1059905  0.08107938 0.10810633 0.07813058\n",
      "  0.12411968 0.09880307 0.09835691 0.0964966 ]\n",
      " [0.09977049 0.08052196 0.11413527 0.09296081 0.11139531 0.10078333\n",
      "  0.10291091 0.0849954  0.09854111 0.11398541]\n",
      " [0.09084265 0.09733619 0.10340384 0.10962292 0.09999113 0.07889725\n",
      "  0.12435856 0.0944164  0.09731309 0.10381798]\n",
      " [0.08029588 0.10103507 0.11626326 0.09049175 0.10187661 0.08241674\n",
      "  0.1292452  0.09863864 0.10440377 0.09533309]\n",
      " [0.11207631 0.08352259 0.10763993 0.08965158 0.1048536  0.09395015\n",
      "  0.11469738 0.08946918 0.08969924 0.11444005]\n",
      " [0.09603103 0.1258878  0.0940808  0.10025907 0.08436507 0.08765654\n",
      "  0.09963043 0.08755431 0.11239191 0.11214303]\n",
      " [0.09896868 0.09855483 0.09793126 0.09267096 0.11389706 0.09512645\n",
      "  0.11245219 0.09240697 0.09944622 0.09854536]\n",
      " [0.09058967 0.10404451 0.1063041  0.0988377  0.10153681 0.10384048\n",
      "  0.10838748 0.09004241 0.09621953 0.1001973 ]\n",
      " [0.09528571 0.09976553 0.10239049 0.09156712 0.10779773 0.09960122\n",
      "  0.11343732 0.0910773  0.09248025 0.10659733]\n",
      " [0.0955218  0.10681889 0.1030311  0.10124908 0.10164497 0.08426848\n",
      "  0.10382805 0.09871202 0.10211833 0.10280728]\n",
      " [0.09992259 0.10175689 0.0927768  0.08877899 0.09750067 0.11456765\n",
      "  0.10099129 0.08742216 0.09828914 0.11799382]\n",
      " [0.08678457 0.10621864 0.10968662 0.09944039 0.10811728 0.08867595\n",
      "  0.10415703 0.08317678 0.11118785 0.10255489]\n",
      " [0.08693616 0.10546367 0.10641086 0.10610282 0.10227357 0.10502789\n",
      "  0.09503906 0.08601372 0.10990044 0.09683182]\n",
      " [0.08997165 0.1112935  0.09428775 0.10318529 0.1034677  0.0954719\n",
      "  0.0999554  0.09355313 0.10533652 0.10347717]\n",
      " [0.1005768  0.08803098 0.11275298 0.09470705 0.0976579  0.08331007\n",
      "  0.12108135 0.08794291 0.10500578 0.10893418]\n",
      " [0.10068061 0.10083687 0.09904206 0.08903641 0.10552842 0.09872669\n",
      "  0.10896318 0.09912553 0.09691273 0.10114749]\n",
      " [0.09458253 0.10569738 0.10751051 0.095924   0.11229629 0.08984821\n",
      "  0.10382553 0.09618469 0.10111626 0.09301459]\n",
      " [0.09807382 0.09170031 0.10133122 0.10505963 0.09844861 0.09705483\n",
      "  0.09825094 0.10090939 0.10447956 0.10469171]\n",
      " [0.09761145 0.10387692 0.10899608 0.09643773 0.09404657 0.09242776\n",
      "  0.11271931 0.08856494 0.09932822 0.10599102]\n",
      " [0.09571252 0.10972289 0.10666961 0.08663137 0.10819354 0.08537377\n",
      "  0.10776247 0.10048494 0.10252607 0.09692281]\n",
      " [0.11518841 0.10603176 0.10390176 0.08433314 0.09301044 0.0953292\n",
      "  0.11340972 0.07558166 0.10671991 0.10649399]\n",
      " [0.09734532 0.11284657 0.10494042 0.10286095 0.10584587 0.09427996\n",
      "  0.10987156 0.09046151 0.08723748 0.09431037]\n",
      " [0.096113   0.09826398 0.10497515 0.10071107 0.09543048 0.10027875\n",
      "  0.10409639 0.09325287 0.10091055 0.10596775]\n",
      " [0.10176952 0.12288778 0.08533695 0.08420527 0.10824895 0.07513821\n",
      "  0.11792745 0.08668776 0.10975073 0.10804737]\n",
      " [0.10028548 0.10193485 0.1157365  0.08548433 0.1061184  0.08998762\n",
      "  0.12158511 0.10234137 0.08362097 0.09290538]\n",
      " [0.07781258 0.0972203  0.09711722 0.1074409  0.10340761 0.09121735\n",
      "  0.11607836 0.09119991 0.11106871 0.10743707]\n",
      " [0.09943346 0.08956617 0.10525019 0.09221022 0.10523672 0.10373673\n",
      "  0.09295907 0.08610306 0.10347428 0.1220301 ]\n",
      " [0.09909986 0.11493933 0.09665727 0.10245015 0.10271759 0.09476856\n",
      "  0.10118604 0.08891136 0.10522581 0.09404402]] (20.073 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 2.266510486602783, step = 201 (40.037 sec)\n",
      "INFO:tensorflow:probabilities = [[0.08740483 0.10220509 0.10154277 0.11332553 0.10139556 0.08944373\n",
      "  0.11141215 0.08346321 0.11501905 0.09478809]\n",
      " [0.10783568 0.09445036 0.10144717 0.0883547  0.09767817 0.09355765\n",
      "  0.10663487 0.1080292  0.09864307 0.10336912]\n",
      " [0.10261589 0.10171092 0.09599072 0.09412995 0.09108029 0.09985734\n",
      "  0.09911387 0.10225619 0.10101604 0.1122288 ]\n",
      " [0.09650049 0.09869481 0.11019421 0.10297574 0.10289639 0.10304986\n",
      "  0.09267563 0.09224545 0.10254738 0.09822004]\n",
      " [0.09640824 0.10602162 0.09291263 0.10333614 0.11200641 0.08912803\n",
      "  0.11117571 0.09001429 0.09357479 0.10542213]\n",
      " [0.0916957  0.09918697 0.09112835 0.10981116 0.10186257 0.11553579\n",
      "  0.10078084 0.08939906 0.09337671 0.10722286]\n",
      " [0.10108586 0.09176883 0.11027321 0.09762778 0.11309464 0.08905613\n",
      "  0.11250393 0.08939897 0.10287149 0.09231916]\n",
      " [0.0932304  0.09852946 0.11353489 0.09568931 0.10039062 0.09131\n",
      "  0.11460281 0.08696023 0.09439966 0.11135263]\n",
      " [0.09588254 0.09511577 0.1014692  0.12160704 0.09895744 0.09635098\n",
      "  0.10180188 0.09146871 0.09006229 0.10728414]\n",
      " [0.10023785 0.10519043 0.10873573 0.09137439 0.10453377 0.09490322\n",
      "  0.11305892 0.08684323 0.09289448 0.10222798]\n",
      " [0.08835866 0.10959404 0.10961895 0.10690936 0.12175797 0.08461701\n",
      "  0.10511941 0.07642764 0.09883004 0.09876691]\n",
      " [0.1052302  0.10144262 0.09205977 0.11429779 0.09619508 0.08911968\n",
      "  0.11875054 0.10021815 0.10956845 0.07311772]\n",
      " [0.09769001 0.11144527 0.10502694 0.09679897 0.09482931 0.09527452\n",
      "  0.10428596 0.09625377 0.09837572 0.10001955]\n",
      " [0.08624297 0.0870879  0.0997675  0.09543225 0.10838242 0.0967604\n",
      "  0.11824548 0.10864216 0.10195265 0.09748626]\n",
      " [0.09868805 0.09607338 0.09805666 0.09484785 0.11116066 0.10032701\n",
      "  0.10189339 0.09505142 0.09118524 0.11271633]\n",
      " [0.09370019 0.11206633 0.09740723 0.10514343 0.0952113  0.09085576\n",
      "  0.09671474 0.10136783 0.09620538 0.11132782]\n",
      " [0.10852478 0.09793441 0.0969508  0.11306739 0.10028413 0.10061448\n",
      "  0.11256373 0.07951757 0.09635098 0.09419173]\n",
      " [0.09610767 0.1105457  0.0952283  0.10013445 0.1062688  0.08430654\n",
      "  0.10124946 0.10347789 0.09057721 0.11210398]\n",
      " [0.08805687 0.09099497 0.10916584 0.1028474  0.09958737 0.10432479\n",
      "  0.0980157  0.1013497  0.10049307 0.10516429]\n",
      " [0.11554698 0.10051702 0.09539218 0.09447249 0.11469705 0.09608773\n",
      "  0.09660708 0.09598037 0.09545279 0.0952463 ]\n",
      " [0.10189379 0.09368019 0.10460884 0.09848357 0.10733508 0.09542065\n",
      "  0.10191198 0.09033023 0.09912635 0.10720932]\n",
      " [0.10228304 0.09850449 0.09642198 0.1029176  0.09415501 0.09543001\n",
      "  0.1046287  0.09392015 0.10203464 0.10970438]\n",
      " [0.08113466 0.11920355 0.10387674 0.10945164 0.08435471 0.11183615\n",
      "  0.10099368 0.08454621 0.08708613 0.11751654]\n",
      " [0.10032318 0.08358808 0.10445028 0.12162641 0.10087719 0.08472611\n",
      "  0.11942261 0.08771428 0.09336318 0.10390868]\n",
      " [0.10871928 0.09610235 0.09775966 0.10796088 0.09321699 0.09905878\n",
      "  0.10843952 0.08352622 0.10824171 0.09697462]\n",
      " [0.11629265 0.09860411 0.10429884 0.10005254 0.11215107 0.09284745\n",
      "  0.10872451 0.08994743 0.08794416 0.08913725]\n",
      " [0.09724553 0.10475668 0.11826099 0.09265688 0.09455267 0.08937996\n",
      "  0.11384806 0.07658954 0.10573629 0.10697341]\n",
      " [0.10771634 0.10490804 0.11114071 0.0860326  0.08559717 0.09862599\n",
      "  0.11846585 0.09618455 0.10012611 0.09120264]\n",
      " [0.11087574 0.11379467 0.09051366 0.09904188 0.09194088 0.1030935\n",
      "  0.10496668 0.08737232 0.1016148  0.09678586]\n",
      " [0.09634893 0.08508439 0.09821644 0.08977785 0.11290339 0.09836104\n",
      "  0.11782689 0.09165955 0.09056096 0.11926056]\n",
      " [0.10329202 0.10590701 0.09805169 0.08746072 0.10288731 0.08781964\n",
      "  0.1072772  0.09115239 0.11786978 0.09828225]\n",
      " [0.09427283 0.10636642 0.11835936 0.09777223 0.09616409 0.08843843\n",
      "  0.10732941 0.09228469 0.09021828 0.10879427]\n",
      " [0.08505699 0.09631136 0.10311623 0.0980593  0.11206105 0.09406766\n",
      "  0.11292663 0.08771691 0.10746145 0.10322242]\n",
      " [0.09430032 0.10186434 0.10675053 0.09613972 0.10686184 0.10197154\n",
      "  0.10074972 0.08979311 0.10095095 0.10061793]\n",
      " [0.09212461 0.09857665 0.11867496 0.09314549 0.09462786 0.0920809\n",
      "  0.11192929 0.0940144  0.092991   0.11183484]\n",
      " [0.10242478 0.11253405 0.08741924 0.11221432 0.08685782 0.08054382\n",
      "  0.10920898 0.09037719 0.10454872 0.11387107]\n",
      " [0.1119711  0.11298545 0.08478634 0.10110049 0.09627185 0.09500131\n",
      "  0.10306404 0.09150008 0.1047314  0.09858797]\n",
      " [0.11083399 0.09807926 0.09869896 0.10336487 0.09222381 0.08487442\n",
      "  0.13610541 0.09691518 0.08707865 0.09182544]\n",
      " [0.09569412 0.11427353 0.09367465 0.09704297 0.09776616 0.09570466\n",
      "  0.10068703 0.09242274 0.09492517 0.11780896]\n",
      " [0.10210835 0.09452903 0.10488129 0.08550027 0.09076464 0.09375083\n",
      "  0.12342738 0.08785956 0.09756605 0.1196126 ]\n",
      " [0.11251111 0.09530874 0.10250849 0.073526   0.12466414 0.09007459\n",
      "  0.1340772  0.0684494  0.09127241 0.10760794]\n",
      " [0.09558704 0.08603525 0.12594851 0.08525521 0.10463277 0.08826507\n",
      "  0.1156052  0.09322758 0.11108622 0.09435716]\n",
      " [0.10461771 0.10843144 0.11107815 0.09999597 0.09171819 0.08782211\n",
      "  0.10496214 0.07748526 0.09550108 0.11838795]\n",
      " [0.10228784 0.0885873  0.0891997  0.09161769 0.12783517 0.08860425\n",
      "  0.11334525 0.08906098 0.10074967 0.10871215]\n",
      " [0.09206447 0.11318386 0.10656791 0.09091594 0.10157975 0.10949229\n",
      "  0.09244817 0.10012889 0.08319535 0.11042337]\n",
      " [0.09194953 0.09634341 0.10843721 0.09111413 0.10426916 0.08835553\n",
      "  0.10159072 0.11359553 0.08963703 0.11470775]\n",
      " [0.09870581 0.09395874 0.09804551 0.10559398 0.09723415 0.09549315\n",
      "  0.10304034 0.10974579 0.09320233 0.10498019]\n",
      " [0.09098403 0.11546553 0.10529751 0.11221561 0.09771068 0.10419055\n",
      "  0.0984958  0.08015131 0.10431985 0.09116914]\n",
      " [0.10254977 0.09950377 0.1113152  0.0957526  0.09531762 0.10125676\n",
      "  0.10162223 0.09474424 0.09625692 0.1016809 ]\n",
      " [0.09159387 0.10877412 0.10008387 0.0931227  0.10057798 0.08396854\n",
      "  0.11805622 0.10419423 0.0866163  0.11301217]\n",
      " [0.09662431 0.10871919 0.10525008 0.10760472 0.09156293 0.1137897\n",
      "  0.07683725 0.08268283 0.10324604 0.11368295]\n",
      " [0.09067187 0.10429112 0.08763442 0.09799964 0.11782516 0.09994106\n",
      "  0.10315355 0.08401339 0.11061728 0.10385251]\n",
      " [0.09747394 0.10090512 0.11434661 0.09069325 0.0890477  0.08628125\n",
      "  0.10472085 0.10376342 0.09585959 0.11690827]\n",
      " [0.0968721  0.09181776 0.10583107 0.09216427 0.08733841 0.08865721\n",
      "  0.13663903 0.0941047  0.10182875 0.10474668]\n",
      " [0.09883061 0.09322856 0.10475173 0.08446253 0.11041647 0.09692613\n",
      "  0.10940645 0.08647268 0.10211497 0.11338987]\n",
      " [0.09941725 0.10355919 0.09381813 0.10236374 0.10766334 0.09735067\n",
      "  0.09999594 0.09468269 0.09467914 0.10646992]\n",
      " [0.09803747 0.11092791 0.09838133 0.09526591 0.1037956  0.09517017\n",
      "  0.10651283 0.09249697 0.10278973 0.09662206]\n",
      " [0.09679941 0.09968839 0.10150085 0.10455425 0.11079074 0.08203507\n",
      "  0.10061396 0.11054331 0.09473726 0.09873676]\n",
      " [0.1045121  0.09332501 0.09162018 0.09607544 0.10210058 0.10080157\n",
      "  0.12493497 0.09343615 0.11233408 0.0808599 ]\n",
      " [0.09163762 0.11212117 0.10799466 0.09418424 0.09905458 0.09893563\n",
      "  0.10261845 0.09472805 0.10343456 0.09529104]\n",
      " [0.08920849 0.10064898 0.10176002 0.0983072  0.11047576 0.09371177\n",
      "  0.1078506  0.10446151 0.0905194  0.10305625]\n",
      " [0.09148536 0.09399492 0.11959474 0.09569287 0.10903884 0.08821471\n",
      "  0.10585836 0.08785884 0.10506408 0.10319728]\n",
      " [0.098646   0.11105234 0.0991296  0.10266364 0.10736376 0.08953914\n",
      "  0.11067218 0.0922481  0.094564   0.09412124]\n",
      " [0.10738524 0.09914069 0.08888872 0.10536596 0.10777248 0.08514271\n",
      "  0.10487823 0.10134436 0.10586842 0.0942132 ]\n",
      " [0.10084763 0.09921311 0.09659885 0.09426227 0.11011203 0.10008017\n",
      "  0.09675194 0.10028038 0.10212376 0.09972986]\n",
      " [0.09555069 0.09522163 0.10050637 0.10529532 0.09906657 0.10520839\n",
      "  0.098738   0.08746993 0.10276005 0.11018305]\n",
      " [0.10685226 0.11334597 0.0906601  0.08816184 0.10773954 0.0873338\n",
      "  0.10452607 0.08780052 0.10651734 0.10706257]\n",
      " [0.10275299 0.09816222 0.12186453 0.09421658 0.10129667 0.09766428\n",
      "  0.11065591 0.07544668 0.1054344  0.09250576]\n",
      " [0.10731109 0.09041612 0.10041761 0.10257674 0.11492157 0.08306014\n",
      "  0.11218859 0.08702529 0.1046679  0.09741495]\n",
      " [0.09801253 0.10248403 0.09911468 0.10263617 0.09934645 0.09197143\n",
      "  0.10097917 0.09575637 0.10301907 0.1066801 ]\n",
      " [0.10393076 0.09528657 0.09316431 0.10186162 0.11469118 0.0918473\n",
      "  0.10857999 0.0981779  0.09317712 0.09928326]\n",
      " [0.11486213 0.10271929 0.09923124 0.09742108 0.09286596 0.08232926\n",
      "  0.11102773 0.08594143 0.10585559 0.10774629]\n",
      " [0.08794348 0.10871542 0.11699443 0.10317555 0.09891292 0.0764719\n",
      "  0.1139632  0.08305415 0.0999662  0.11080276]\n",
      " [0.09685098 0.1065347  0.10819509 0.09334013 0.09909161 0.10017654\n",
      "  0.1017936  0.09947928 0.09608355 0.09845451]\n",
      " [0.09806336 0.09428105 0.10310848 0.10128228 0.1061238  0.09244913\n",
      "  0.10479702 0.09020337 0.10995523 0.09973629]\n",
      " [0.09800042 0.10850982 0.11124436 0.09459933 0.10516848 0.08026019\n",
      "  0.10341329 0.08228007 0.10500114 0.11152292]\n",
      " [0.10636905 0.10431545 0.09433878 0.08738956 0.10785324 0.08545632\n",
      "  0.11210591 0.09483714 0.09737261 0.10996195]\n",
      " [0.10310463 0.08818741 0.1041201  0.0952688  0.10683896 0.08632256\n",
      "  0.12011172 0.08536955 0.10376216 0.10691411]\n",
      " [0.0952042  0.11417908 0.09868472 0.09916519 0.08622804 0.09630694\n",
      "  0.11073967 0.09247413 0.10446107 0.10255696]\n",
      " [0.08670308 0.09885688 0.09615267 0.08855731 0.11208459 0.11364321\n",
      "  0.11553289 0.09039349 0.09458378 0.10349211]\n",
      " [0.09245205 0.11641824 0.09854255 0.09656113 0.09526257 0.08944101\n",
      "  0.10331766 0.1000011  0.10490803 0.10309566]\n",
      " [0.11631209 0.10526207 0.10697044 0.10961986 0.08800284 0.10310599\n",
      "  0.09757255 0.09191743 0.08519173 0.096045  ]\n",
      " [0.10193874 0.0885323  0.09290255 0.09665877 0.11173973 0.08886196\n",
      "  0.11155964 0.10022099 0.10835273 0.09923259]\n",
      " [0.08864925 0.09318876 0.09741715 0.10126732 0.10699621 0.09586593\n",
      "  0.1097592  0.10663754 0.09541684 0.1048018 ]\n",
      " [0.1052311  0.10062229 0.10887455 0.09366566 0.09687175 0.08844974\n",
      "  0.12099802 0.09049556 0.10714162 0.08764972]\n",
      " [0.09844215 0.10446417 0.1059079  0.09441336 0.09843199 0.09473127\n",
      "  0.10443323 0.08840541 0.10125091 0.10951959]\n",
      " [0.11671506 0.10257485 0.09040353 0.1070702  0.10019227 0.10602973\n",
      "  0.09864026 0.08799056 0.08695315 0.1034304 ]\n",
      " [0.09424954 0.10737349 0.10352029 0.09649676 0.09916919 0.09034977\n",
      "  0.09972155 0.08317544 0.10979554 0.11614843]\n",
      " [0.11263989 0.09039776 0.10011293 0.09465413 0.11362546 0.0935201\n",
      "  0.10509293 0.10958671 0.08162706 0.09874302]\n",
      " [0.10619132 0.1116272  0.09926683 0.09513346 0.09704527 0.10331822\n",
      "  0.09633232 0.09750979 0.09119576 0.10237982]\n",
      " [0.09500816 0.12406288 0.0958208  0.10446704 0.0948781  0.08896599\n",
      "  0.10630217 0.10022657 0.09477551 0.09549278]\n",
      " [0.09620392 0.09613555 0.10207686 0.10297303 0.10321532 0.08853191\n",
      "  0.11398516 0.08187595 0.1026664  0.11233589]\n",
      " [0.09554705 0.10218812 0.09376019 0.11028172 0.10894909 0.08961696\n",
      "  0.10039409 0.08723499 0.09578918 0.11623862]\n",
      " [0.10248448 0.09210289 0.10076887 0.09811923 0.11129177 0.09235469\n",
      "  0.10924889 0.10042797 0.09241457 0.10078665]\n",
      " [0.09910196 0.11054722 0.10352755 0.09988768 0.09343363 0.09734437\n",
      "  0.09912498 0.09971097 0.09168486 0.10563679]\n",
      " [0.09601685 0.10320498 0.08548278 0.09979967 0.09373672 0.09860098\n",
      "  0.11526751 0.09332177 0.11460794 0.0999608 ]\n",
      " [0.0906621  0.10642329 0.10708361 0.0904106  0.10143641 0.09927992\n",
      "  0.10501952 0.09494189 0.09845365 0.10628902]\n",
      " [0.09836112 0.10386119 0.10356993 0.1063091  0.0980781  0.10282159\n",
      "  0.09489648 0.08844615 0.10560099 0.09805536]\n",
      " [0.09104907 0.09575659 0.11308509 0.08109021 0.11254065 0.09894557\n",
      "  0.10706907 0.08846498 0.09685065 0.11514813]\n",
      " [0.09656894 0.0781459  0.1096518  0.09423129 0.11360844 0.08176027\n",
      "  0.10138612 0.10152358 0.10808454 0.11503913]] (19.970 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 2.74781\n",
      "INFO:tensorflow:probabilities = [[0.09275079 0.09781953 0.10201835 0.10008664 0.11003606 0.08758336\n",
      "  0.09226309 0.09568198 0.11233258 0.10942761]\n",
      " [0.11545296 0.10298779 0.0998541  0.10120464 0.08954312 0.09110143\n",
      "  0.10608867 0.09680603 0.09244219 0.10451906]\n",
      " [0.10014649 0.10081567 0.11214886 0.107541   0.08731592 0.09723773\n",
      "  0.10361038 0.08312414 0.10188491 0.10617489]\n",
      " [0.11372872 0.10509682 0.10414652 0.09443506 0.11058112 0.0834747\n",
      "  0.10228564 0.10462599 0.08048178 0.10114367]\n",
      " [0.10499089 0.08267195 0.09753852 0.08725578 0.09892066 0.10101111\n",
      "  0.13854432 0.07955919 0.11225216 0.09725543]\n",
      " [0.07985565 0.10600006 0.11536836 0.10121217 0.11438446 0.1010733\n",
      "  0.11355437 0.08443012 0.08762246 0.09649905]\n",
      " [0.09408254 0.10071983 0.10091003 0.09706804 0.10504525 0.08435585\n",
      "  0.10218031 0.11100124 0.08832532 0.11631159]\n",
      " [0.10205579 0.10568917 0.09500247 0.0940452  0.09578024 0.08574331\n",
      "  0.11521807 0.09678406 0.10737587 0.10230583]\n",
      " [0.10611138 0.10058486 0.1036284  0.08970453 0.09806986 0.08756471\n",
      "  0.10466753 0.09581543 0.10412747 0.10972583]\n",
      " [0.10022597 0.09359904 0.09436333 0.09696795 0.1124594  0.11328627\n",
      "  0.08529792 0.10603688 0.09128823 0.10647501]\n",
      " [0.11453278 0.09407826 0.1021239  0.1015574  0.09966745 0.08862779\n",
      "  0.12995193 0.07833776 0.08766107 0.10346166]\n",
      " [0.11486272 0.08930018 0.11906416 0.08533002 0.0991761  0.09882947\n",
      "  0.10761612 0.08414793 0.11080868 0.09086462]\n",
      " [0.10023265 0.11360601 0.10190768 0.09476472 0.09993929 0.09241925\n",
      "  0.0982906  0.08871426 0.10428231 0.10584322]\n",
      " [0.10913001 0.10051446 0.10114402 0.10015635 0.09800353 0.08566358\n",
      "  0.10518071 0.09127842 0.10219108 0.10673783]\n",
      " [0.10924565 0.07966376 0.10843861 0.08420024 0.10900592 0.09439765\n",
      "  0.10529703 0.09332398 0.10727785 0.1091493 ]\n",
      " [0.10434282 0.10520528 0.09723439 0.10532531 0.11380073 0.09758543\n",
      "  0.10443899 0.09264522 0.08041213 0.09900971]\n",
      " [0.09445016 0.10666503 0.07955918 0.09222581 0.10981005 0.10971778\n",
      "  0.10490625 0.09884018 0.09964039 0.10418517]\n",
      " [0.08906626 0.09070575 0.10789925 0.12603092 0.11125655 0.10878769\n",
      "  0.09374647 0.09081451 0.0971194  0.08457321]\n",
      " [0.10275199 0.10105445 0.10612362 0.09018431 0.09658315 0.09553418\n",
      "  0.09817157 0.09233557 0.10486468 0.11239649]\n",
      " [0.11741534 0.09420371 0.10525699 0.12647053 0.09909856 0.09231767\n",
      "  0.09394087 0.08471958 0.09536551 0.09121123]\n",
      " [0.11572815 0.08765936 0.09392063 0.09236509 0.10806089 0.10266577\n",
      "  0.10740153 0.07863315 0.10697805 0.10658737]\n",
      " [0.08880028 0.10412311 0.10684467 0.0971818  0.10985704 0.09315974\n",
      "  0.10776113 0.08673021 0.10099867 0.10454333]\n",
      " [0.10218786 0.08694702 0.09618374 0.10871376 0.09875218 0.09006185\n",
      "  0.11515751 0.09557967 0.09490491 0.11151149]\n",
      " [0.10779103 0.08986572 0.10707979 0.0870215  0.10963529 0.08738602\n",
      "  0.10562969 0.09128468 0.10350678 0.11079949]\n",
      " [0.09797914 0.10424331 0.10993041 0.09493109 0.09716358 0.09675728\n",
      "  0.09586725 0.09667586 0.09499941 0.11145267]\n",
      " [0.10729273 0.09660651 0.09859091 0.10016578 0.10815571 0.08654051\n",
      "  0.10764363 0.11061574 0.09780934 0.08657915]\n",
      " [0.11176857 0.09637075 0.10553232 0.09108978 0.09532018 0.09528175\n",
      "  0.11968752 0.08950717 0.10633582 0.08910614]\n",
      " [0.10790833 0.09736875 0.0992449  0.10534561 0.08854177 0.09845337\n",
      "  0.0910125  0.09133336 0.11307151 0.10771991]\n",
      " [0.10760299 0.0994167  0.09511224 0.09622227 0.10546328 0.0929698\n",
      "  0.10628216 0.09249303 0.10365446 0.10078306]\n",
      " [0.11136036 0.09725838 0.10336405 0.10052968 0.09806201 0.09464989\n",
      "  0.117275   0.08188551 0.0921937  0.10342142]\n",
      " [0.08655615 0.10927523 0.10589565 0.11339991 0.10387722 0.10122761\n",
      "  0.09591328 0.08466956 0.10714517 0.09204021]\n",
      " [0.12304935 0.0919005  0.0900673  0.09990787 0.09342203 0.09219282\n",
      "  0.10061131 0.10180373 0.09588591 0.11115917]\n",
      " [0.10662454 0.09100491 0.10739752 0.09082289 0.11211046 0.07309977\n",
      "  0.10889693 0.08971198 0.09962503 0.12070597]\n",
      " [0.08668301 0.09956909 0.10555589 0.10485487 0.09801001 0.09849564\n",
      "  0.11578342 0.09214158 0.09874011 0.10016638]\n",
      " [0.09974656 0.09116599 0.11468515 0.09950939 0.10432842 0.09821319\n",
      "  0.11826859 0.08694505 0.08970638 0.09743128]\n",
      " [0.09753572 0.11160752 0.09044116 0.1021598  0.08816317 0.08843901\n",
      "  0.12533687 0.09325714 0.09986856 0.10319107]\n",
      " [0.10422732 0.09469702 0.10113244 0.08866461 0.09974115 0.10547803\n",
      "  0.10865805 0.08531013 0.10931237 0.10277888]\n",
      " [0.10350113 0.10930145 0.09266382 0.08657917 0.10996425 0.09448255\n",
      "  0.11165608 0.08888478 0.10027286 0.10269391]\n",
      " [0.09009472 0.11293061 0.11351513 0.0919602  0.10186904 0.09166513\n",
      "  0.12118708 0.08830049 0.09403451 0.09444309]\n",
      " [0.10667988 0.09672469 0.1100557  0.10889303 0.09217382 0.0806716\n",
      "  0.0938158  0.09664237 0.10518712 0.109156  ]\n",
      " [0.10316472 0.11339941 0.12052902 0.09111217 0.09903055 0.08361347\n",
      "  0.10842463 0.07636753 0.08878312 0.11557538]\n",
      " [0.10473894 0.08809293 0.10654402 0.08472275 0.10329422 0.09604479\n",
      "  0.1116414  0.09234368 0.1116026  0.10097467]\n",
      " [0.11175189 0.08391049 0.10594773 0.09026655 0.10656751 0.07694652\n",
      "  0.12976144 0.10148311 0.08795281 0.10541196]\n",
      " [0.09205657 0.10958101 0.10371458 0.09829105 0.08924219 0.08996869\n",
      "  0.10307468 0.08679917 0.11864784 0.10862419]\n",
      " [0.09717634 0.10933384 0.08974878 0.08647347 0.13420408 0.09606891\n",
      "  0.10749653 0.07821002 0.10428633 0.09700171]\n",
      " [0.09142849 0.09694549 0.09414195 0.11190246 0.10497155 0.09168785\n",
      "  0.10149637 0.09976797 0.09338749 0.11427038]\n",
      " [0.10082653 0.10184245 0.10159255 0.10225504 0.10524051 0.0981768\n",
      "  0.10066589 0.08084682 0.10179905 0.10675435]\n",
      " [0.0987003  0.10667162 0.08841434 0.10944163 0.10272422 0.09882767\n",
      "  0.09460605 0.10088373 0.10585262 0.09387782]\n",
      " [0.10137741 0.07806738 0.11283456 0.09375992 0.10945293 0.10461478\n",
      "  0.11518961 0.07979602 0.09431944 0.11058795]\n",
      " [0.09273418 0.1152195  0.09811601 0.1049883  0.10281302 0.09286357\n",
      "  0.09213619 0.09459767 0.09995269 0.10657885]\n",
      " [0.10423165 0.0944005  0.09512279 0.08288148 0.10897985 0.0979231\n",
      "  0.09813216 0.1065931  0.09918362 0.11255175]\n",
      " [0.10175105 0.10839672 0.10740871 0.10499579 0.10013165 0.09764617\n",
      "  0.09415187 0.09367813 0.09426905 0.09757085]\n",
      " [0.11045533 0.089374   0.10651572 0.08049103 0.10904914 0.09295903\n",
      "  0.11692037 0.07804749 0.10101104 0.11517687]\n",
      " [0.10682222 0.09347435 0.11142473 0.09666673 0.10819951 0.08703788\n",
      "  0.10975673 0.08523921 0.09449647 0.10688216]\n",
      " [0.10483984 0.08996328 0.092388   0.10263449 0.12122739 0.08831037\n",
      "  0.10403401 0.08472598 0.09320803 0.11866859]\n",
      " [0.11477111 0.09824974 0.0978155  0.11372505 0.09736227 0.08262166\n",
      "  0.1093197  0.10050285 0.09208547 0.09354663]\n",
      " [0.09729245 0.09791326 0.10986782 0.09215863 0.10094621 0.09656887\n",
      "  0.11061916 0.09505089 0.09706316 0.10251956]\n",
      " [0.11290532 0.09619782 0.10306329 0.11190964 0.0980081  0.07771602\n",
      "  0.10270672 0.11168736 0.08911897 0.09668676]\n",
      " [0.10506842 0.09038356 0.11901029 0.09548388 0.09715264 0.08590065\n",
      "  0.10386261 0.08548447 0.10024842 0.11740506]\n",
      " [0.09563747 0.09883294 0.11712754 0.10357712 0.09615855 0.09670919\n",
      "  0.09879157 0.08673251 0.10586704 0.10056606]\n",
      " [0.09905179 0.11775009 0.10469927 0.0887605  0.11342639 0.08562176\n",
      "  0.10685816 0.08821836 0.10519115 0.09042254]\n",
      " [0.09829273 0.09794934 0.10273236 0.09820391 0.0990088  0.0927175\n",
      "  0.09997764 0.1012395  0.09268382 0.1171944 ]\n",
      " [0.10905778 0.08191237 0.09141901 0.10246385 0.11855341 0.08985312\n",
      "  0.10480226 0.0958257  0.10411457 0.10199793]\n",
      " [0.08752193 0.10650011 0.09482989 0.10587991 0.10196872 0.09900259\n",
      "  0.10025351 0.09818291 0.10364827 0.10221216]\n",
      " [0.10444072 0.09186064 0.10495428 0.09667682 0.1182754  0.07856535\n",
      "  0.10272323 0.08955916 0.10834034 0.10460406]\n",
      " [0.09828712 0.08586116 0.08653732 0.08809825 0.11478158 0.09994499\n",
      "  0.10547837 0.09691509 0.09816257 0.12593355]\n",
      " [0.09247313 0.0964249  0.10000683 0.09355046 0.11000093 0.0979919\n",
      "  0.11148077 0.09571316 0.10452231 0.09783561]\n",
      " [0.1014917  0.08350659 0.1000451  0.09659075 0.10793668 0.09652505\n",
      "  0.11615829 0.08286534 0.09955741 0.1153231 ]\n",
      " [0.09636721 0.12233845 0.10562984 0.09729879 0.09044728 0.09044227\n",
      "  0.10275983 0.09164456 0.10222269 0.10084908]\n",
      " [0.11100603 0.09145572 0.08997765 0.10595614 0.09978798 0.09991027\n",
      "  0.09673083 0.09843276 0.09783804 0.10890459]\n",
      " [0.09732237 0.10341643 0.09652843 0.10165151 0.10532028 0.09004605\n",
      "  0.1123021  0.09486818 0.10361163 0.09493302]\n",
      " [0.09169281 0.08266813 0.09924053 0.097717   0.1178375  0.09140785\n",
      "  0.11035304 0.10668865 0.09194631 0.11044816]\n",
      " [0.10316712 0.10315585 0.09892452 0.09881655 0.10061486 0.09721211\n",
      "  0.10568241 0.09744762 0.09293395 0.10204503]\n",
      " [0.09308921 0.1138452  0.11130324 0.09751259 0.08127914 0.08831549\n",
      "  0.10157898 0.08602123 0.10955999 0.11749492]\n",
      " [0.09612103 0.11280515 0.08975713 0.10326816 0.09761845 0.09410822\n",
      "  0.11476687 0.09550837 0.10621479 0.08983184]\n",
      " [0.1187612  0.08202378 0.1115898  0.11028202 0.09924022 0.08395534\n",
      "  0.11659498 0.08231807 0.08767438 0.1075602 ]\n",
      " [0.10410525 0.11102715 0.10232224 0.08909157 0.09744191 0.09107533\n",
      "  0.11623893 0.08866166 0.09315269 0.10688327]\n",
      " [0.10409249 0.10571539 0.09142681 0.09370012 0.10759208 0.087701\n",
      "  0.11429121 0.09045583 0.09557434 0.10945074]\n",
      " [0.10157438 0.1115893  0.09757855 0.10124693 0.08885719 0.10925358\n",
      "  0.09793626 0.08423584 0.1044542  0.10327376]\n",
      " [0.10287263 0.09125391 0.10708095 0.10856277 0.11020115 0.08732178\n",
      "  0.10248461 0.09370346 0.08738594 0.10913279]\n",
      " [0.1031628  0.08533693 0.09460851 0.09157367 0.12065668 0.09360075\n",
      "  0.11167076 0.09754669 0.09222982 0.10961339]\n",
      " [0.09643135 0.10864841 0.09170696 0.10309538 0.10672457 0.09209071\n",
      "  0.10445681 0.10436526 0.09322797 0.09925258]\n",
      " [0.11009727 0.11359493 0.09540959 0.09199006 0.11695603 0.09062831\n",
      "  0.09699026 0.07786396 0.09591205 0.11055755]\n",
      " [0.10051629 0.10811555 0.10452584 0.10675256 0.10269908 0.09320896\n",
      "  0.10052795 0.09152223 0.0963959  0.09573564]\n",
      " [0.09323928 0.09414002 0.10586331 0.10532245 0.10906515 0.09824233\n",
      "  0.10247606 0.08248371 0.10262482 0.10654287]\n",
      " [0.10799546 0.09022972 0.09146509 0.10371642 0.11267207 0.09622341\n",
      "  0.1042205  0.0823432  0.1113966  0.09973751]\n",
      " [0.10248457 0.10910613 0.10879045 0.09312772 0.09697742 0.10436333\n",
      "  0.10438504 0.08309846 0.09750378 0.1001631 ]\n",
      " [0.10228015 0.09833115 0.10354892 0.09100728 0.10776348 0.08264753\n",
      "  0.115317   0.09844439 0.0965291  0.104131  ]\n",
      " [0.10800645 0.09083982 0.11621706 0.10400182 0.11545613 0.08648631\n",
      "  0.09898608 0.08204533 0.09528012 0.10268087]\n",
      " [0.1049034  0.09680435 0.11879523 0.11438753 0.10084878 0.08874846\n",
      "  0.0966088  0.08182303 0.08920247 0.10787795]\n",
      " [0.09360447 0.09442115 0.10984893 0.08983575 0.09224992 0.10396298\n",
      "  0.11630581 0.07659339 0.10269671 0.12048088]\n",
      " [0.09287594 0.1035582  0.10310934 0.10982848 0.09084785 0.10650915\n",
      "  0.09845526 0.08336914 0.11400867 0.09743797]\n",
      " [0.09493675 0.11108209 0.09613138 0.09720449 0.10118682 0.09631377\n",
      "  0.10947663 0.09088126 0.10456048 0.09822635]\n",
      " [0.10588005 0.09192092 0.09411255 0.08893893 0.10540702 0.08634706\n",
      "  0.11056819 0.09038073 0.12356683 0.10287772]\n",
      " [0.11887987 0.09265857 0.09872894 0.11294975 0.08851706 0.10668237\n",
      "  0.10636924 0.08773611 0.0916667  0.09581138]\n",
      " [0.09456386 0.08975842 0.10499846 0.09598939 0.11839243 0.08718249\n",
      "  0.11308974 0.10279261 0.09283882 0.10039379]\n",
      " [0.10736775 0.10854117 0.10938489 0.08330989 0.09817654 0.09340496\n",
      "  0.10183281 0.08172737 0.10789127 0.10836336]\n",
      " [0.101331   0.11486467 0.1061514  0.10974781 0.10110837 0.09255083\n",
      "  0.09960329 0.08552992 0.0970348  0.09207791]\n",
      " [0.10614707 0.09600034 0.11115061 0.09693158 0.10090961 0.08367854\n",
      "  0.12167913 0.09228432 0.08622134 0.10499747]\n",
      " [0.11476424 0.08889654 0.10032082 0.09570679 0.10372255 0.0936205\n",
      "  0.11108727 0.10306095 0.0943449  0.09447544]] (16.422 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 2.254169464111328, step = 301 (36.393 sec)\n",
      "INFO:tensorflow:probabilities = [[0.1050658  0.09359336 0.11288996 0.09263927 0.11123501 0.08779526\n",
      "  0.09171194 0.10530055 0.09520767 0.10456118]\n",
      " [0.1152992  0.08803308 0.11602868 0.09839093 0.0959201  0.08958415\n",
      "  0.11210812 0.08531573 0.10615357 0.09316644]\n",
      " [0.10433627 0.08969733 0.10240831 0.09740598 0.1062149  0.09310843\n",
      "  0.10328498 0.1045105  0.09704345 0.10198985]\n",
      " [0.09251677 0.085149   0.09478267 0.09379303 0.11946546 0.10614051\n",
      "  0.109974   0.09948232 0.09095115 0.10774509]\n",
      " [0.11366128 0.08452618 0.09686292 0.09923135 0.11486704 0.08678809\n",
      "  0.11077315 0.09991192 0.09400568 0.0993724 ]\n",
      " [0.13784295 0.09533166 0.08934016 0.08385327 0.11232967 0.09297074\n",
      "  0.11276561 0.08217103 0.1043056  0.08908931]\n",
      " [0.09953302 0.10522317 0.10318287 0.10449705 0.09902985 0.09427051\n",
      "  0.10843249 0.09477569 0.09911171 0.09194364]\n",
      " [0.08991633 0.10083876 0.09146703 0.10004696 0.11171964 0.09229909\n",
      "  0.10464963 0.10656835 0.09862925 0.10386495]\n",
      " [0.09631711 0.10399838 0.10098882 0.12513823 0.09579048 0.08699692\n",
      "  0.09166956 0.09139425 0.11250871 0.09519755]\n",
      " [0.10538403 0.09897644 0.12362465 0.11024392 0.0959717  0.07715619\n",
      "  0.1052995  0.08042353 0.10527197 0.09764806]\n",
      " [0.09759474 0.11405523 0.11001462 0.08431334 0.10598018 0.09362108\n",
      "  0.09456601 0.09768377 0.10831809 0.09385293]\n",
      " [0.0999492  0.10711794 0.09698487 0.1034419  0.0963631  0.1044156\n",
      "  0.08565816 0.11001817 0.09259789 0.10345318]\n",
      " [0.09495035 0.11451015 0.09479316 0.10722246 0.10280534 0.09777885\n",
      "  0.09405925 0.09129108 0.10119795 0.1013914 ]\n",
      " [0.12448667 0.07717104 0.10558356 0.10675682 0.09434125 0.10037828\n",
      "  0.11698255 0.0735621  0.1067517  0.09398604]\n",
      " [0.11359946 0.0913332  0.10072732 0.1097041  0.10815461 0.08078224\n",
      "  0.12183096 0.0941923  0.08528907 0.09438675]\n",
      " [0.11938649 0.09158107 0.10426832 0.10440049 0.08778278 0.10215177\n",
      "  0.11529105 0.08218258 0.08823246 0.10472299]\n",
      " [0.10488967 0.0962542  0.11298004 0.0984587  0.09393599 0.08874357\n",
      "  0.11431024 0.10610335 0.09827415 0.08605008]\n",
      " [0.10226563 0.09629179 0.09441866 0.09967561 0.11529632 0.09210964\n",
      "  0.12605971 0.09039609 0.09737916 0.08610739]\n",
      " [0.10282268 0.09582177 0.10617995 0.08923209 0.09287651 0.08414001\n",
      "  0.12997297 0.08098452 0.10606258 0.11190692]\n",
      " [0.09170752 0.10072216 0.0863557  0.08533327 0.12141707 0.08927234\n",
      "  0.10463822 0.10654354 0.09895907 0.11505112]\n",
      " [0.10166152 0.11767773 0.10659066 0.09811666 0.10418459 0.09256208\n",
      "  0.09870162 0.08926152 0.10244103 0.08880259]\n",
      " [0.08982491 0.09940562 0.09861425 0.10330483 0.10027851 0.0988138\n",
      "  0.11226887 0.11051624 0.09621898 0.09075399]\n",
      " [0.10902088 0.09111394 0.10591618 0.09899039 0.09776174 0.08455312\n",
      "  0.10963871 0.09934307 0.10447338 0.09918857]\n",
      " [0.10043448 0.1007981  0.08711212 0.10032818 0.09425049 0.0907144\n",
      "  0.11084585 0.11650397 0.09052946 0.10848295]\n",
      " [0.09431511 0.0857546  0.10805307 0.08957876 0.11336752 0.09826079\n",
      "  0.12377065 0.09052499 0.0978979  0.0984766 ]\n",
      " [0.10119079 0.10155271 0.10213923 0.10156539 0.09681139 0.08148929\n",
      "  0.11190624 0.11425107 0.09461328 0.09448061]\n",
      " [0.08650961 0.10938655 0.10001303 0.09680018 0.09434752 0.09917362\n",
      "  0.10495415 0.10770199 0.09500832 0.10610503]\n",
      " [0.11809475 0.09715376 0.10981754 0.09958386 0.09413834 0.08730387\n",
      "  0.1146965  0.0904698  0.09336476 0.09537683]\n",
      " [0.1046156  0.1070647  0.09765352 0.10316422 0.09783752 0.09607571\n",
      "  0.09644597 0.09086073 0.10502179 0.10126025]\n",
      " [0.10480653 0.08536541 0.08649443 0.10124024 0.10989964 0.0968864\n",
      "  0.12056458 0.1016213  0.09257528 0.10054618]\n",
      " [0.10694087 0.09730571 0.0908863  0.09546046 0.11412845 0.09066668\n",
      "  0.11159609 0.08416527 0.10176152 0.10708866]\n",
      " [0.10084264 0.1066361  0.09427219 0.09489457 0.09869509 0.10671958\n",
      "  0.10972225 0.09205065 0.09084646 0.10532047]\n",
      " [0.10503372 0.10610278 0.09869846 0.09782214 0.09567817 0.09721978\n",
      "  0.09836716 0.09937772 0.08950835 0.11219171]\n",
      " [0.09592973 0.11055343 0.10968916 0.09340391 0.09743725 0.08784778\n",
      "  0.10666938 0.09254451 0.1152463  0.09067856]\n",
      " [0.09532013 0.08832653 0.09542769 0.10358797 0.10000132 0.09745506\n",
      "  0.11728263 0.10428895 0.10533475 0.09297497]\n",
      " [0.09253763 0.09428635 0.09259177 0.08225706 0.11353654 0.0884979\n",
      "  0.09939824 0.10076657 0.10951749 0.12661043]\n",
      " [0.1136732  0.1048991  0.09811678 0.08249023 0.09948185 0.083596\n",
      "  0.10696537 0.09594616 0.10761584 0.10721545]\n",
      " [0.11097127 0.10768743 0.1016578  0.10026495 0.09826192 0.09556484\n",
      "  0.10176962 0.08371832 0.09854174 0.10156211]\n",
      " [0.10595977 0.10557597 0.11474816 0.10400138 0.09975411 0.08455915\n",
      "  0.09205469 0.08018638 0.11923241 0.09392797]\n",
      " [0.1178842  0.10233184 0.11852637 0.10010687 0.10408246 0.07748472\n",
      "  0.0939524  0.09132077 0.11141281 0.08289756]\n",
      " [0.10359521 0.08694252 0.10025696 0.09181401 0.09307605 0.10324418\n",
      "  0.12351557 0.07967342 0.09874188 0.11914021]\n",
      " [0.11043919 0.09005328 0.09836586 0.094506   0.09893843 0.09030945\n",
      "  0.12428497 0.09788089 0.10702967 0.08819227]\n",
      " [0.12308294 0.10544301 0.10018273 0.1002802  0.10128765 0.07506425\n",
      "  0.10626299 0.0885095  0.09799504 0.10189169]\n",
      " [0.09280307 0.10153173 0.10824218 0.09812658 0.1005798  0.1074938\n",
      "  0.09731897 0.09676474 0.08768239 0.10945675]\n",
      " [0.10330904 0.10209614 0.11050411 0.10078434 0.09553125 0.08692509\n",
      "  0.10170075 0.09288147 0.1033467  0.1029211 ]\n",
      " [0.09095352 0.10431241 0.11478959 0.12258813 0.09615458 0.08789227\n",
      "  0.10409472 0.08108015 0.09785699 0.10027763]\n",
      " [0.10889171 0.11836371 0.10006349 0.10061188 0.09399141 0.10487785\n",
      "  0.09475637 0.08116476 0.09722517 0.10005364]\n",
      " [0.12386102 0.09652328 0.08954168 0.09429244 0.08566077 0.08880798\n",
      "  0.12602802 0.07567883 0.09247419 0.12713179]\n",
      " [0.09362129 0.09357043 0.08387999 0.11167664 0.11177748 0.08974162\n",
      "  0.09520289 0.09981116 0.11009912 0.11061938]\n",
      " [0.11769608 0.10232629 0.1053298  0.09137711 0.095656   0.08378271\n",
      "  0.10839617 0.09145025 0.09202882 0.11195677]\n",
      " [0.10414178 0.10725803 0.10728825 0.09248529 0.09288361 0.09172205\n",
      "  0.11795221 0.09333853 0.09627448 0.09665576]\n",
      " [0.09412413 0.09821698 0.10301004 0.1036336  0.10784063 0.08603123\n",
      "  0.09567946 0.09800607 0.10855951 0.10489833]\n",
      " [0.12427183 0.09042244 0.09634533 0.11202639 0.10137096 0.08691208\n",
      "  0.09602007 0.08966539 0.10780164 0.09516387]\n",
      " [0.10107541 0.08157028 0.10319504 0.09138529 0.11482228 0.09725587\n",
      "  0.10519295 0.09348608 0.10832846 0.10368834]\n",
      " [0.10594215 0.11333313 0.10633958 0.09471973 0.10179856 0.09061074\n",
      "  0.11330042 0.08838219 0.09230121 0.0932723 ]\n",
      " [0.10972613 0.08684196 0.0970957  0.0954223  0.11026781 0.0870961\n",
      "  0.1290706  0.09147089 0.10157333 0.09143517]\n",
      " [0.10525065 0.09457865 0.08618619 0.11089911 0.08853905 0.09859342\n",
      "  0.10545653 0.07238182 0.12303829 0.11507629]\n",
      " [0.09868434 0.09899584 0.10131939 0.11097519 0.09782779 0.09095545\n",
      "  0.10345162 0.09180072 0.10471304 0.10127663]\n",
      " [0.1161838  0.10401975 0.12062066 0.0757915  0.09297848 0.06825932\n",
      "  0.10493357 0.08630107 0.10466412 0.12624774]\n",
      " [0.10719911 0.08722628 0.10862221 0.11530016 0.0971899  0.08443933\n",
      "  0.100922   0.10148746 0.10682748 0.09078606]\n",
      " [0.10945244 0.1024826  0.09820905 0.09796162 0.08665957 0.09584401\n",
      "  0.10810894 0.0969144  0.09867082 0.10569654]\n",
      " [0.09692074 0.11091664 0.09303108 0.09953572 0.09616809 0.08547959\n",
      "  0.10185899 0.09880874 0.10512121 0.1121592 ]\n",
      " [0.09963422 0.08161467 0.10125781 0.1038243  0.12516468 0.08581255\n",
      "  0.12417187 0.09140725 0.10330766 0.08380499]\n",
      " [0.09128408 0.09591609 0.11509591 0.09907551 0.08808257 0.09028746\n",
      "  0.10874719 0.10408892 0.11379343 0.09362884]\n",
      " [0.10219335 0.09006838 0.10420954 0.11370769 0.0963527  0.08124672\n",
      "  0.11925577 0.10071592 0.08448877 0.10776116]\n",
      " [0.09980027 0.09669944 0.11362808 0.09674466 0.10185591 0.08336517\n",
      "  0.10870317 0.10934573 0.08842644 0.10143113]\n",
      " [0.10993651 0.10201181 0.10581977 0.1035611  0.09771229 0.10406108\n",
      "  0.09826936 0.08449023 0.09305634 0.10108151]\n",
      " [0.09408952 0.09663849 0.09339187 0.09388001 0.10226483 0.10117545\n",
      "  0.11175491 0.09419772 0.09996811 0.1126391 ]\n",
      " [0.13239673 0.08612024 0.11480499 0.10458889 0.09174166 0.08370353\n",
      "  0.10897096 0.0814739  0.10691895 0.08928015]\n",
      " [0.07878421 0.09393553 0.1057066  0.08887845 0.12340419 0.09657347\n",
      "  0.10819715 0.10775079 0.09440356 0.10236604]\n",
      " [0.0979086  0.0976153  0.10296144 0.09318395 0.11276493 0.09454971\n",
      "  0.10244027 0.07815036 0.11538856 0.10503688]\n",
      " [0.10138903 0.10837624 0.10470557 0.09612085 0.0969401  0.08967812\n",
      "  0.10565925 0.0969026  0.1005176  0.09971064]\n",
      " [0.08137771 0.09726859 0.09458013 0.09756444 0.10477842 0.10833349\n",
      "  0.10958809 0.09544501 0.1036219  0.10744221]\n",
      " [0.09297864 0.10221401 0.10807913 0.09112663 0.1062926  0.09096908\n",
      "  0.10735031 0.09292402 0.10621151 0.10185406]\n",
      " [0.09696026 0.09650471 0.0847626  0.11646654 0.08799609 0.10540968\n",
      "  0.11588286 0.08677416 0.1154251  0.093818  ]\n",
      " [0.11217585 0.09073564 0.11372272 0.09385896 0.08976267 0.09325971\n",
      "  0.10489804 0.08928857 0.11294216 0.09935567]\n",
      " [0.10503371 0.09735571 0.09033526 0.10821405 0.09136036 0.09540238\n",
      "  0.10982985 0.10441565 0.10215847 0.09589455]\n",
      " [0.10236358 0.11092409 0.10159934 0.0997465  0.09924248 0.09743761\n",
      "  0.10665334 0.09829873 0.096698   0.08703633]\n",
      " [0.09492461 0.09109076 0.0924169  0.09233292 0.10596357 0.10346796\n",
      "  0.09924877 0.09804437 0.09976052 0.12274962]\n",
      " [0.09303445 0.10141033 0.1036464  0.09117419 0.10150028 0.10689867\n",
      "  0.10676699 0.0988079  0.0983036  0.09845717]\n",
      " [0.10816356 0.09755681 0.09285675 0.10771837 0.11088245 0.08615588\n",
      "  0.10559461 0.09837099 0.09517619 0.09752438]\n",
      " [0.08863101 0.11243352 0.11126372 0.09938066 0.09338328 0.08278407\n",
      "  0.10507045 0.09422329 0.10402339 0.10880659]\n",
      " [0.1137839  0.11679541 0.10537461 0.09395392 0.09003488 0.09772942\n",
      "  0.10122643 0.08678167 0.10135369 0.09296607]\n",
      " [0.09696467 0.09888485 0.10440548 0.08845534 0.10774126 0.07791178\n",
      "  0.11505642 0.10026535 0.10450123 0.10581362]\n",
      " [0.10981957 0.09942742 0.10468506 0.10452488 0.10179028 0.07815218\n",
      "  0.11371541 0.09585955 0.08632531 0.10570034]\n",
      " [0.09634265 0.10927606 0.10059072 0.10569506 0.09947973 0.09456652\n",
      "  0.10278932 0.10571195 0.10291451 0.08263348]\n",
      " [0.09509985 0.08899192 0.10253769 0.10683849 0.0990932  0.10376703\n",
      "  0.10471683 0.09616089 0.10601857 0.09677551]\n",
      " [0.10361352 0.08217523 0.11234754 0.08842253 0.10791    0.11199726\n",
      "  0.11152403 0.08355679 0.09280231 0.10565079]\n",
      " [0.09412725 0.10374485 0.096647   0.10761105 0.09946615 0.08738277\n",
      "  0.1058328  0.10322949 0.08855153 0.11340711]\n",
      " [0.10725445 0.12479743 0.10361026 0.08802363 0.08780755 0.07955032\n",
      "  0.10494392 0.10312726 0.10183243 0.09905275]\n",
      " [0.09875282 0.10368129 0.09625072 0.10775685 0.10079555 0.09559971\n",
      "  0.09801678 0.09716407 0.09849006 0.10349215]\n",
      " [0.09619507 0.10371264 0.11994924 0.08567311 0.08720489 0.0898561\n",
      "  0.11555277 0.09085722 0.10925115 0.10174781]\n",
      " [0.12331971 0.09377133 0.10972579 0.09505514 0.09727392 0.10019669\n",
      "  0.10495898 0.07380643 0.10438205 0.09750996]\n",
      " [0.09137841 0.10583524 0.10105215 0.09564183 0.10636123 0.10177894\n",
      "  0.09557526 0.09666135 0.0994632  0.10625238]\n",
      " [0.09533798 0.10399968 0.09634828 0.09899541 0.09890802 0.09141685\n",
      "  0.10822453 0.0989722  0.08469231 0.12310474]\n",
      " [0.08775325 0.1015587  0.11596545 0.10108005 0.10436921 0.09118536\n",
      "  0.0958849  0.09100691 0.10062145 0.11057473]\n",
      " [0.09961309 0.11302621 0.09569315 0.10386214 0.10468856 0.10010088\n",
      "  0.09599215 0.09244206 0.1020493  0.09253247]\n",
      " [0.09025142 0.09935954 0.09704267 0.10564836 0.11670779 0.08481687\n",
      "  0.10058006 0.09536263 0.11183781 0.09839285]\n",
      " [0.09467454 0.09653444 0.08919407 0.09853124 0.1087915  0.08806752\n",
      "  0.09593253 0.10041955 0.11209342 0.11576119]\n",
      " [0.09372746 0.1045122  0.09992153 0.10645166 0.09384341 0.08924892\n",
      "  0.10029817 0.098547   0.10603234 0.10741731]] (16.353 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 3.05531\n",
      "INFO:tensorflow:probabilities = [[0.09597762 0.0909486  0.12058421 0.10410072 0.099659   0.07758354\n",
      "  0.10627802 0.11648254 0.0769326  0.11145315]\n",
      " [0.10963587 0.09053114 0.13479191 0.09767633 0.08651705 0.08932541\n",
      "  0.10331992 0.08237692 0.10635014 0.09947531]\n",
      " [0.10397911 0.08333036 0.10077023 0.09792857 0.10290619 0.07712854\n",
      "  0.13626958 0.09095258 0.09763974 0.10909508]\n",
      " [0.10388879 0.08918134 0.10400142 0.09318527 0.10414206 0.0957871\n",
      "  0.12201979 0.09041223 0.09773254 0.09964945]\n",
      " [0.11870913 0.10231822 0.08833914 0.09680783 0.10971388 0.09126806\n",
      "  0.11002335 0.07492631 0.11399084 0.09390323]\n",
      " [0.08565493 0.10227403 0.09968803 0.09860474 0.11018276 0.11449807\n",
      "  0.09156754 0.09174301 0.08989833 0.11588855]\n",
      " [0.10981075 0.1183686  0.1132189  0.09458769 0.08500653 0.08478\n",
      "  0.10224897 0.09152189 0.10658724 0.09386945]\n",
      " [0.10203997 0.08620733 0.08903091 0.08759958 0.10962791 0.09202795\n",
      "  0.09395495 0.09671206 0.10027396 0.14252538]\n",
      " [0.09802255 0.09921866 0.10327087 0.09833464 0.10251698 0.08968727\n",
      "  0.10248265 0.10848519 0.09461884 0.10336236]\n",
      " [0.09920875 0.10313913 0.10344376 0.10039051 0.10896986 0.09183398\n",
      "  0.0966227  0.09437386 0.10070524 0.10131222]\n",
      " [0.09027425 0.12335    0.10195331 0.10791239 0.09997754 0.0785515\n",
      "  0.10608521 0.08916917 0.0978626  0.10486404]\n",
      " [0.09496553 0.10877582 0.08742742 0.09998529 0.10422268 0.09602026\n",
      "  0.09528837 0.09266967 0.09788928 0.12275567]\n",
      " [0.10018709 0.10452983 0.09979227 0.10454976 0.09554086 0.09999922\n",
      "  0.10799804 0.09654258 0.09231899 0.09854136]\n",
      " [0.10913523 0.09076074 0.09758129 0.09534677 0.09052722 0.09990643\n",
      "  0.08916824 0.07893687 0.1234107  0.12522653]\n",
      " [0.10564315 0.10300434 0.09849155 0.09256041 0.1007675  0.09614041\n",
      "  0.09785231 0.11253113 0.10113237 0.09187685]\n",
      " [0.10311684 0.09871888 0.10403771 0.09368425 0.10130434 0.08984861\n",
      "  0.10639019 0.10051608 0.09460608 0.10777702]\n",
      " [0.09050153 0.10992992 0.10803636 0.11828364 0.08904941 0.0792364\n",
      "  0.10174254 0.10389254 0.09653246 0.1027952 ]\n",
      " [0.10372353 0.0931939  0.10622838 0.08349066 0.1093457  0.09272636\n",
      "  0.09732416 0.09269531 0.10556735 0.11570464]\n",
      " [0.08285829 0.10291444 0.10636591 0.10366602 0.10442595 0.10018671\n",
      "  0.10607535 0.09755413 0.0858682  0.11008501]\n",
      " [0.10530676 0.09647519 0.09268305 0.11598164 0.10030245 0.09699405\n",
      "  0.09970681 0.10337947 0.09283422 0.09633636]\n",
      " [0.09626526 0.09383645 0.11874747 0.11098861 0.1098858  0.08542524\n",
      "  0.09645176 0.11090627 0.08037537 0.09711776]\n",
      " [0.08382992 0.08853971 0.09392503 0.10754074 0.12046597 0.08522752\n",
      "  0.11011477 0.09947068 0.10022424 0.1106614 ]\n",
      " [0.10630846 0.08417727 0.12272569 0.1093064  0.08163939 0.08350455\n",
      "  0.09718894 0.09915521 0.10924319 0.10675089]\n",
      " [0.11619466 0.09713772 0.09114352 0.10638204 0.08691963 0.09192287\n",
      "  0.10200892 0.09742787 0.10546337 0.10539939]\n",
      " [0.11717025 0.09600619 0.11624097 0.09034218 0.0893086  0.09027562\n",
      "  0.11232715 0.08440253 0.10282721 0.10109931]\n",
      " [0.10412965 0.08488757 0.11939661 0.0972583  0.08838822 0.09297517\n",
      "  0.11164295 0.09067968 0.11099383 0.09964802]\n",
      " [0.10505467 0.09701249 0.10000399 0.1017563  0.09729525 0.10348752\n",
      "  0.10158813 0.08798413 0.11241167 0.09340584]\n",
      " [0.10751224 0.10392026 0.10820468 0.09301574 0.11199089 0.09101979\n",
      "  0.08647455 0.10066036 0.10046519 0.09673629]\n",
      " [0.11226123 0.08214299 0.13504525 0.0902191  0.08939069 0.08087477\n",
      "  0.11296883 0.08888881 0.10503148 0.10317685]\n",
      " [0.11238276 0.08883003 0.12711461 0.10702357 0.09527889 0.08522667\n",
      "  0.10581985 0.07625138 0.09525171 0.10682052]\n",
      " [0.08784711 0.09380777 0.10783569 0.0975634  0.09898811 0.10710695\n",
      "  0.11025312 0.10763473 0.08787061 0.10109251]\n",
      " [0.09908476 0.10633835 0.10715451 0.09688727 0.10521484 0.0919445\n",
      "  0.09291881 0.09681494 0.10467236 0.09896965]\n",
      " [0.1063459  0.08778    0.10180496 0.09682663 0.10042768 0.09221105\n",
      "  0.10639038 0.09444392 0.10877682 0.10499266]\n",
      " [0.09812797 0.11845846 0.11094894 0.09247978 0.09064481 0.09766009\n",
      "  0.11014169 0.08954623 0.09168058 0.10031146]\n",
      " [0.14911123 0.09393476 0.10387798 0.10035124 0.08249794 0.09713689\n",
      "  0.11691777 0.08873908 0.09438665 0.07304646]\n",
      " [0.09451554 0.08337466 0.09081303 0.09447041 0.12662797 0.08719955\n",
      "  0.11297197 0.09859518 0.10717514 0.10425656]\n",
      " [0.091139   0.10266476 0.10344063 0.10106229 0.10800483 0.09025124\n",
      "  0.09621373 0.11300626 0.09340536 0.10081189]\n",
      " [0.12132025 0.09849247 0.11697458 0.11097613 0.0941364  0.09606231\n",
      "  0.09853622 0.08070944 0.10261707 0.08017513]\n",
      " [0.09729925 0.10876165 0.11400925 0.1102148  0.09123663 0.08108348\n",
      "  0.0972028  0.09543102 0.09927876 0.10548235]\n",
      " [0.10035027 0.11071469 0.09903293 0.10126556 0.11281545 0.10174673\n",
      "  0.09307525 0.08618212 0.10329724 0.09151976]\n",
      " [0.0846094  0.10288923 0.10487081 0.10029393 0.09139243 0.0824552\n",
      "  0.11536384 0.08594608 0.10591209 0.12626698]\n",
      " [0.09471375 0.10978627 0.09081023 0.12040922 0.09018169 0.09440434\n",
      "  0.10845317 0.09822803 0.09717001 0.09584328]\n",
      " [0.09459631 0.09051063 0.11591049 0.08912415 0.11931791 0.0877738\n",
      "  0.10649842 0.08231556 0.09995615 0.11399659]\n",
      " [0.10375793 0.1086023  0.10313953 0.09820946 0.10027128 0.09128055\n",
      "  0.09232692 0.09354018 0.10845641 0.10041544]\n",
      " [0.09452518 0.10869118 0.10727025 0.09176166 0.09621642 0.10304456\n",
      "  0.0916101  0.10618237 0.08968653 0.11101176]\n",
      " [0.10866614 0.09526975 0.10530086 0.09716271 0.10364267 0.09338512\n",
      "  0.1092228  0.09432842 0.09438924 0.09863229]\n",
      " [0.10467914 0.11064659 0.09107729 0.10757505 0.09748599 0.09168192\n",
      "  0.11709179 0.07754652 0.09659629 0.10561942]\n",
      " [0.10151648 0.07173404 0.08508935 0.09968809 0.0958514  0.09338341\n",
      "  0.1243764  0.09727135 0.10017788 0.13091159]\n",
      " [0.10297162 0.10379023 0.1112365  0.09146776 0.08795601 0.08863915\n",
      "  0.12420002 0.09047734 0.09431254 0.10494884]\n",
      " [0.10829706 0.09495179 0.09541348 0.09291517 0.10697675 0.08044776\n",
      "  0.1063662  0.09609362 0.1074633  0.11107486]\n",
      " [0.11319213 0.10821073 0.09264986 0.08650786 0.10092371 0.07559279\n",
      "  0.11773107 0.10157879 0.09705686 0.1065562 ]\n",
      " [0.08769616 0.10886408 0.10921332 0.10797865 0.10086481 0.08957553\n",
      "  0.09973064 0.08650046 0.10168699 0.10788935]\n",
      " [0.12186859 0.08311121 0.10773522 0.0928379  0.1125621  0.09181224\n",
      "  0.10593186 0.09652533 0.09466011 0.09295543]\n",
      " [0.09224112 0.11214973 0.10242747 0.12866656 0.1013868  0.08694116\n",
      "  0.09623115 0.10025331 0.09264675 0.08705593]\n",
      " [0.09264624 0.09825958 0.12300453 0.08895869 0.10113923 0.08683438\n",
      "  0.10872463 0.08408648 0.11753608 0.09881018]\n",
      " [0.11597054 0.11657633 0.10177548 0.11752752 0.09177991 0.08685841\n",
      "  0.09812258 0.07948893 0.10928654 0.08261376]\n",
      " [0.09441683 0.10750156 0.11208027 0.09132823 0.10097568 0.09053616\n",
      "  0.09322459 0.10130863 0.10363753 0.10499053]\n",
      " [0.10447435 0.09550019 0.10041095 0.10836298 0.0864332  0.10203253\n",
      "  0.10444107 0.09897045 0.09623267 0.1031416 ]\n",
      " [0.10145955 0.13245269 0.09996872 0.09779318 0.08979862 0.09154394\n",
      "  0.08819266 0.09054102 0.11035052 0.0978991 ]\n",
      " [0.0997396  0.10587163 0.09570723 0.09867023 0.1047826  0.08981288\n",
      "  0.10453646 0.09099908 0.10824376 0.10163653]\n",
      " [0.09588177 0.0814679  0.10031639 0.11606297 0.09331705 0.10253288\n",
      "  0.11838762 0.09333177 0.09917667 0.09952498]\n",
      " [0.10765368 0.09097976 0.09618156 0.11175566 0.0994987  0.08731182\n",
      "  0.12011152 0.09087441 0.09820618 0.09742671]\n",
      " [0.11940835 0.09583224 0.09955474 0.10101212 0.08606044 0.10474882\n",
      "  0.12470502 0.08553125 0.09252681 0.0906202 ]\n",
      " [0.10315053 0.08445444 0.10046502 0.1087546  0.10363283 0.09096679\n",
      "  0.11007454 0.10798737 0.09708759 0.09342629]\n",
      " [0.10409359 0.08888598 0.09528084 0.10588848 0.09576167 0.09663047\n",
      "  0.10164225 0.09819392 0.09783518 0.11578761]\n",
      " [0.09768812 0.10758707 0.09146689 0.09296035 0.09858452 0.1012378\n",
      "  0.11913249 0.09314692 0.09577865 0.10241718]\n",
      " [0.10653183 0.10375542 0.09927552 0.1212158  0.08613223 0.09252819\n",
      "  0.10301235 0.09292122 0.09322197 0.10140549]\n",
      " [0.13816977 0.08164083 0.11694247 0.10068249 0.11176274 0.08444545\n",
      "  0.09709738 0.0764375  0.08935768 0.1034637 ]\n",
      " [0.11402514 0.10406841 0.11098048 0.08260764 0.09984447 0.09559858\n",
      "  0.08946213 0.09586202 0.11108278 0.09646834]\n",
      " [0.09931285 0.08239943 0.10857027 0.10397401 0.08918712 0.09314491\n",
      "  0.0952296  0.11421333 0.10122345 0.11274502]\n",
      " [0.10693305 0.09412527 0.11166824 0.09192651 0.09636071 0.07408151\n",
      "  0.11640713 0.1069879  0.10616744 0.09534225]\n",
      " [0.1011494  0.1148369  0.11018347 0.10143957 0.09190591 0.08634987\n",
      "  0.09440729 0.09907402 0.09980503 0.10084854]\n",
      " [0.10382459 0.10185803 0.10185869 0.09116051 0.10371042 0.08242356\n",
      "  0.11541715 0.10167886 0.10184985 0.09621834]\n",
      " [0.10620006 0.10316661 0.10505076 0.09170717 0.09423519 0.09946765\n",
      "  0.10682853 0.08786486 0.10118232 0.10429684]\n",
      " [0.09771694 0.11352701 0.10240224 0.10256476 0.09504317 0.1028631\n",
      "  0.10019661 0.08647473 0.10541967 0.09379177]\n",
      " [0.10607261 0.09174914 0.10639634 0.11778017 0.0812407  0.09366012\n",
      "  0.10918182 0.10127937 0.09929081 0.09334892]\n",
      " [0.11610273 0.08899182 0.1012149  0.09402258 0.11125187 0.09203617\n",
      "  0.11333242 0.08452667 0.10716758 0.09135325]\n",
      " [0.09252057 0.09003394 0.11091647 0.10981293 0.10243839 0.09381915\n",
      "  0.09425984 0.10146755 0.08508861 0.11964254]\n",
      " [0.10062623 0.10472389 0.09665682 0.10323586 0.10172258 0.09018366\n",
      "  0.10261283 0.09112057 0.11162615 0.09749142]\n",
      " [0.11498788 0.08880865 0.10030813 0.0974972  0.10623805 0.10375334\n",
      "  0.10759514 0.09001564 0.10170205 0.08909392]\n",
      " [0.10198708 0.08782972 0.09594051 0.08103049 0.10916477 0.0971152\n",
      "  0.11970951 0.08995598 0.09439237 0.12287437]\n",
      " [0.10001996 0.10494754 0.09763045 0.10050148 0.10278074 0.10053625\n",
      "  0.10520462 0.09239109 0.10175044 0.09423743]\n",
      " [0.11565559 0.11278448 0.10281367 0.10849497 0.09584343 0.08915773\n",
      "  0.09458803 0.0927689  0.10592658 0.08196663]\n",
      " [0.10852585 0.08953653 0.08740538 0.10761926 0.09583344 0.09494816\n",
      "  0.11135588 0.09274766 0.09675061 0.11527722]\n",
      " [0.10884913 0.08746613 0.10215412 0.08769218 0.10424087 0.09101738\n",
      "  0.11497853 0.09250777 0.09329235 0.11780154]\n",
      " [0.09937387 0.12293727 0.11005565 0.09841616 0.10094191 0.08573736\n",
      "  0.09884041 0.09208557 0.09914175 0.09247005]\n",
      " [0.09497036 0.10092464 0.11876245 0.09457213 0.10127959 0.10011435\n",
      "  0.10689333 0.10051873 0.09404144 0.08792299]\n",
      " [0.09878765 0.10886802 0.10182131 0.09083432 0.10053975 0.0932657\n",
      "  0.10464995 0.09469121 0.11537741 0.09116469]\n",
      " [0.10163948 0.09362893 0.10966679 0.12186957 0.10902564 0.07974231\n",
      "  0.1004845  0.09068761 0.09517579 0.09807939]\n",
      " [0.08165537 0.10220698 0.08999828 0.09673963 0.1052714  0.0817531\n",
      "  0.10552568 0.12440375 0.08704972 0.1253961 ]\n",
      " [0.11796453 0.08352916 0.08950315 0.10072452 0.11887127 0.09002159\n",
      "  0.10945996 0.10149625 0.10207521 0.08635436]\n",
      " [0.09416033 0.11258997 0.11624149 0.08900922 0.10271034 0.08500689\n",
      "  0.10571924 0.08670274 0.10531725 0.10254253]\n",
      " [0.0962636  0.09503361 0.0880897  0.08929093 0.09969602 0.10318174\n",
      "  0.11593684 0.10597591 0.10127598 0.10525565]\n",
      " [0.09406139 0.0809058  0.10033619 0.10191719 0.10768757 0.11965666\n",
      "  0.10960304 0.0771052  0.08998296 0.11874401]\n",
      " [0.09194601 0.09584525 0.09484482 0.10424106 0.10192481 0.09219686\n",
      "  0.10428474 0.09869844 0.1185366  0.0974814 ]\n",
      " [0.10192555 0.09393381 0.08909514 0.10529193 0.0961927  0.10284197\n",
      "  0.11123325 0.0878707  0.10542467 0.10619029]\n",
      " [0.08560092 0.11719421 0.10650384 0.09701158 0.10679803 0.09584894\n",
      "  0.093101   0.09274691 0.10277689 0.10241767]\n",
      " [0.09338189 0.08494271 0.10026033 0.09756642 0.10378066 0.08844002\n",
      "  0.12311665 0.09234171 0.09880122 0.1173684 ]\n",
      " [0.12318263 0.09032418 0.12438054 0.08945016 0.10801508 0.0889578\n",
      "  0.09302875 0.08207306 0.08447048 0.11611732]\n",
      " [0.09651316 0.11078246 0.117602   0.10268608 0.09128937 0.08248613\n",
      "  0.10789611 0.0765717  0.09472508 0.11944792]] (16.378 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 2.234375, step = 401 (32.738 sec)\n",
      "INFO:tensorflow:probabilities = [[0.10925104 0.09653889 0.10431825 0.08467032 0.10086386 0.08627147\n",
      "  0.11630298 0.11062485 0.08954831 0.10161002]\n",
      " [0.09938521 0.08399853 0.12136011 0.10847069 0.10966809 0.08305985\n",
      "  0.10988242 0.08633995 0.09059924 0.1072359 ]\n",
      " [0.10220732 0.10345756 0.09625048 0.08527816 0.1015621  0.09477276\n",
      "  0.11312039 0.10197088 0.09421402 0.10716634]\n",
      " [0.09802776 0.1089209  0.1051896  0.0832979  0.10225636 0.0926843\n",
      "  0.0907352  0.10142474 0.10935258 0.10811067]\n",
      " [0.09239856 0.09341619 0.09808144 0.08604263 0.11372883 0.09269515\n",
      "  0.12414781 0.09271637 0.09827336 0.10849967]\n",
      " [0.11332475 0.12115515 0.11525958 0.09966035 0.09610272 0.07342646\n",
      "  0.09392155 0.07893969 0.10141926 0.10679046]\n",
      " [0.09248126 0.08352564 0.09328173 0.09296233 0.10924984 0.0779205\n",
      "  0.12711865 0.10182997 0.11599064 0.10563943]\n",
      " [0.09673758 0.08308309 0.10099019 0.12364542 0.09017135 0.08803624\n",
      "  0.10571979 0.09707048 0.11207459 0.10247126]\n",
      " [0.10208494 0.07537555 0.10117034 0.10068713 0.12662528 0.0884264\n",
      "  0.10896795 0.0885406  0.09559423 0.11252759]\n",
      " [0.10804117 0.09565398 0.10715713 0.09581058 0.08946644 0.11099297\n",
      "  0.09863447 0.09239364 0.11470834 0.08714129]\n",
      " [0.1006626  0.10660099 0.09809856 0.09281133 0.08472564 0.10908829\n",
      "  0.09907035 0.10575399 0.11176448 0.09142376]\n",
      " [0.1046777  0.10822953 0.09290074 0.08605089 0.11676117 0.10426149\n",
      "  0.10615619 0.08686318 0.09877302 0.09532609]\n",
      " [0.0997222  0.11396518 0.09466914 0.10702595 0.08863614 0.09148158\n",
      "  0.1069796  0.08944711 0.10338249 0.10469061]\n",
      " [0.08834564 0.11553467 0.09354199 0.09674064 0.09480212 0.10529677\n",
      "  0.08392889 0.10582583 0.10952708 0.10645637]\n",
      " [0.09304442 0.11182489 0.10396583 0.10826236 0.09280337 0.09980933\n",
      "  0.08461672 0.10800376 0.09290031 0.10476903]\n",
      " [0.11250623 0.10261934 0.10697953 0.10178841 0.09471385 0.08227177\n",
      "  0.11780025 0.09152094 0.09947804 0.09032163]\n",
      " [0.09817807 0.09557991 0.1057896  0.10577165 0.09470552 0.11732639\n",
      "  0.09152717 0.09934113 0.10389345 0.08788709]\n",
      " [0.10760349 0.08752278 0.09910238 0.0931268  0.10418361 0.09624686\n",
      "  0.1046735  0.10435837 0.0942621  0.10892011]\n",
      " [0.09587398 0.1066031  0.11162795 0.09386697 0.10280719 0.07050121\n",
      "  0.10392643 0.10139981 0.11084323 0.10255013]\n",
      " [0.10183381 0.11248348 0.10095256 0.10759095 0.09046773 0.07614211\n",
      "  0.10131532 0.09005322 0.11472519 0.10443562]\n",
      " [0.09198398 0.08595968 0.14279707 0.09341524 0.09587343 0.0884371\n",
      "  0.10999411 0.0790387  0.10360931 0.10889139]\n",
      " [0.09307934 0.10694002 0.0986629  0.09386028 0.11075882 0.07991074\n",
      "  0.10274216 0.09391251 0.10876737 0.11136586]\n",
      " [0.10985308 0.10234177 0.1060706  0.09329021 0.09459886 0.09646704\n",
      "  0.1069221  0.07819668 0.09932517 0.1129345 ]\n",
      " [0.12981363 0.09192619 0.12139943 0.09568135 0.07644775 0.09199365\n",
      "  0.09157327 0.0881479  0.11497123 0.0980456 ]\n",
      " [0.07413552 0.1194796  0.11431613 0.1135975  0.09750288 0.0871012\n",
      "  0.08084524 0.08881409 0.10315744 0.12105039]\n",
      " [0.10461471 0.10885101 0.10092662 0.0972525  0.0917916  0.08940053\n",
      "  0.10620925 0.09396148 0.10245248 0.10453982]\n",
      " [0.08851078 0.09818543 0.10303114 0.09095172 0.10470403 0.10257899\n",
      "  0.10987384 0.07732577 0.11078059 0.1140577 ]\n",
      " [0.09703259 0.11112942 0.0957085  0.09745339 0.0843902  0.0948946\n",
      "  0.08584983 0.11299793 0.09345207 0.12709147]\n",
      " [0.10998838 0.08392434 0.09315785 0.09250099 0.09921323 0.07670585\n",
      "  0.10589928 0.10462166 0.10455003 0.12943839]\n",
      " [0.12818051 0.0929146  0.11016355 0.08142619 0.10523145 0.09037212\n",
      "  0.1201421  0.09552966 0.08414504 0.09189478]\n",
      " [0.11480427 0.07854523 0.11452369 0.09385033 0.09525366 0.09174058\n",
      "  0.09710143 0.08796517 0.10502126 0.12119438]\n",
      " [0.10187233 0.08272047 0.07791445 0.10010444 0.1036258  0.10267285\n",
      "  0.11244183 0.11220968 0.10298385 0.10345429]\n",
      " [0.09306342 0.08410608 0.13003214 0.07781345 0.10285153 0.07566408\n",
      "  0.12245286 0.08725024 0.1211907  0.10557549]\n",
      " [0.11271798 0.09184406 0.0837263  0.10820541 0.11067427 0.09082406\n",
      "  0.11300037 0.0930812  0.08918496 0.1067414 ]\n",
      " [0.10236709 0.09608667 0.12213417 0.08847648 0.10080145 0.0826648\n",
      "  0.11707703 0.08823148 0.08600027 0.11616055]\n",
      " [0.09337323 0.11678249 0.10158751 0.10198855 0.08910941 0.10048864\n",
      "  0.0957388  0.09360859 0.10348606 0.10383673]\n",
      " [0.09065265 0.11175411 0.09562915 0.09959119 0.11651192 0.10282794\n",
      "  0.10058475 0.09106028 0.09163488 0.09975314]\n",
      " [0.09775753 0.11527511 0.10648455 0.09449095 0.09116502 0.09158013\n",
      "  0.10574525 0.09425318 0.10302209 0.1002262 ]\n",
      " [0.07972402 0.11787826 0.09714049 0.11655265 0.10500151 0.09606094\n",
      "  0.0925108  0.09372679 0.10210113 0.09930342]\n",
      " [0.09250909 0.10236039 0.09426029 0.09774067 0.09714455 0.1077656\n",
      "  0.11493988 0.10381493 0.09734675 0.09211785]\n",
      " [0.0898041  0.09483704 0.11242859 0.09675204 0.09986232 0.10066377\n",
      "  0.09584054 0.09955129 0.09340198 0.11685834]\n",
      " [0.09712689 0.11246792 0.09863871 0.09550054 0.11006803 0.08840639\n",
      "  0.10355609 0.10233072 0.09497962 0.0969251 ]\n",
      " [0.09133608 0.09953843 0.10163461 0.0962238  0.10400764 0.09269631\n",
      "  0.09671681 0.11289867 0.09955682 0.10539083]\n",
      " [0.10487785 0.09878482 0.10803305 0.09455734 0.10628717 0.08919287\n",
      "  0.10708695 0.08457674 0.10726676 0.09933645]\n",
      " [0.09662473 0.09934475 0.10566097 0.10040766 0.09777065 0.08714229\n",
      "  0.10530253 0.0989539  0.09792332 0.1108692 ]\n",
      " [0.09658876 0.10308763 0.0968225  0.09281584 0.09934645 0.0823094\n",
      "  0.10795695 0.11007662 0.10893221 0.10206363]\n",
      " [0.12165783 0.0859975  0.10832453 0.10969075 0.09694195 0.08943883\n",
      "  0.11734595 0.08847666 0.08058681 0.10153918]\n",
      " [0.1321206  0.09431814 0.09101745 0.07832081 0.09517563 0.07945165\n",
      "  0.12661174 0.07885966 0.11082812 0.1132962 ]\n",
      " [0.10696495 0.08777333 0.09908495 0.09369586 0.0909905  0.08911954\n",
      "  0.10537853 0.09923568 0.10802106 0.11973561]\n",
      " [0.11443128 0.10664781 0.07712351 0.10504359 0.10734905 0.0857858\n",
      "  0.10628882 0.08721112 0.10523103 0.10488799]\n",
      " [0.10356691 0.0850589  0.09013802 0.11248369 0.10708238 0.0943453\n",
      "  0.09834717 0.10096324 0.10458182 0.10343257]\n",
      " [0.09258053 0.1178367  0.10749497 0.09450302 0.09391698 0.10111388\n",
      "  0.09912977 0.08048896 0.10820717 0.10472801]\n",
      " [0.09741881 0.08441355 0.09982575 0.09757062 0.11073296 0.09504594\n",
      "  0.1159123  0.1029738  0.09450352 0.10160275]\n",
      " [0.10263381 0.09020086 0.10685654 0.08317983 0.11958486 0.08744186\n",
      "  0.10065319 0.08947054 0.09809782 0.12188071]\n",
      " [0.09321383 0.08996045 0.09576333 0.11979092 0.09130971 0.08906514\n",
      "  0.11180823 0.08030716 0.11244478 0.11633646]\n",
      " [0.09808133 0.10437768 0.10375619 0.09986261 0.10611096 0.10448351\n",
      "  0.0962903  0.07360111 0.11180898 0.10162732]\n",
      " [0.08511291 0.09085594 0.09296926 0.11320921 0.10960371 0.07701182\n",
      "  0.10116933 0.10953914 0.11483136 0.10569733]\n",
      " [0.10419464 0.07887697 0.09788375 0.10070603 0.09414506 0.09792456\n",
      "  0.11426532 0.09505686 0.10458187 0.11236492]\n",
      " [0.08437445 0.1018498  0.09975981 0.09351007 0.11455859 0.09338474\n",
      "  0.10707791 0.09934568 0.09220549 0.11393346]\n",
      " [0.10502955 0.08700608 0.09787817 0.11162158 0.11555353 0.084465\n",
      "  0.11011672 0.08673664 0.1057145  0.09587824]\n",
      " [0.11339007 0.09985619 0.1078359  0.07263275 0.10021305 0.08333156\n",
      "  0.12301086 0.09387259 0.10646081 0.09939621]\n",
      " [0.10552099 0.08921842 0.11722188 0.09703841 0.08465436 0.08544462\n",
      "  0.10995395 0.09823805 0.11644687 0.09626244]\n",
      " [0.10087855 0.07912128 0.10989637 0.10988141 0.10425416 0.0999067\n",
      "  0.09793502 0.09016903 0.10926881 0.09868868]\n",
      " [0.09827113 0.09555376 0.09660955 0.11508043 0.10173892 0.08722376\n",
      "  0.10977147 0.08283139 0.11076155 0.10215804]\n",
      " [0.09040409 0.09756922 0.09861336 0.107511   0.09930099 0.10357133\n",
      "  0.10226025 0.09159488 0.0941752  0.11499967]\n",
      " [0.09722238 0.07951729 0.08201377 0.10986518 0.09075867 0.12280207\n",
      "  0.09656689 0.08302214 0.12859124 0.10964037]\n",
      " [0.10445898 0.09417139 0.09761893 0.0855593  0.08705067 0.0987323\n",
      "  0.10765938 0.09202927 0.12156921 0.11115056]\n",
      " [0.0989389  0.0895463  0.10294398 0.0878153  0.09821876 0.09133769\n",
      "  0.12282528 0.10149516 0.10945832 0.09742033]\n",
      " [0.11287881 0.0989348  0.08683765 0.09828771 0.09699102 0.08931642\n",
      "  0.10878843 0.10475719 0.09217759 0.11103039]\n",
      " [0.1019343  0.08905994 0.08542779 0.10785826 0.08690577 0.07887655\n",
      "  0.11401403 0.08960921 0.12717449 0.11913966]\n",
      " [0.10351343 0.09740945 0.08948544 0.09082291 0.1065866  0.10070907\n",
      "  0.10979231 0.10082833 0.09788996 0.10296251]\n",
      " [0.09309656 0.10247924 0.11276899 0.11065714 0.09992815 0.08668995\n",
      "  0.09533981 0.0910731  0.11235839 0.09560866]\n",
      " [0.09849988 0.0953417  0.11691248 0.10294877 0.08965564 0.0949181\n",
      "  0.08408194 0.11560311 0.09845999 0.1035784 ]\n",
      " [0.09231257 0.09961833 0.12028975 0.12914332 0.08105471 0.07487731\n",
      "  0.11248011 0.08889687 0.10889806 0.09242897]\n",
      " [0.07939196 0.09691625 0.10929396 0.10167683 0.12362282 0.08995944\n",
      "  0.11916933 0.08256474 0.090359   0.10704567]\n",
      " [0.08316573 0.1128297  0.11380443 0.10530384 0.10229653 0.0995675\n",
      "  0.08663767 0.09230032 0.10182135 0.10227292]\n",
      " [0.13575363 0.08522079 0.11404419 0.08562204 0.09122791 0.08694012\n",
      "  0.09917169 0.10502589 0.10038415 0.09660959]\n",
      " [0.10010288 0.09668465 0.09757947 0.10674971 0.10168956 0.09784807\n",
      "  0.1044473  0.10115012 0.09282729 0.10092096]\n",
      " [0.1112359  0.08496836 0.11522662 0.08749071 0.11365627 0.08381708\n",
      "  0.1197141  0.09114007 0.08322622 0.10952467]\n",
      " [0.09681012 0.11370555 0.10292417 0.09025671 0.1086731  0.10306338\n",
      "  0.09703082 0.08782348 0.09417632 0.10553636]\n",
      " [0.11860756 0.09711233 0.10018525 0.10231091 0.10400832 0.07841043\n",
      "  0.11160912 0.08726287 0.1025779  0.09791531]\n",
      " [0.12299366 0.08728722 0.10799504 0.07457172 0.10623053 0.09529875\n",
      "  0.10122023 0.08399568 0.10425874 0.11614843]\n",
      " [0.11088051 0.08898702 0.10318203 0.09540083 0.09384633 0.07652624\n",
      "  0.13531199 0.08202191 0.11646067 0.09738247]\n",
      " [0.10658511 0.09155759 0.09687906 0.11633223 0.09412138 0.0935255\n",
      "  0.12375507 0.0873231  0.08968041 0.10024055]\n",
      " [0.1202895  0.09006985 0.09098189 0.11505992 0.10408776 0.09465629\n",
      "  0.11763448 0.07561611 0.09826202 0.09334217]\n",
      " [0.09518957 0.12531937 0.09025809 0.1120962  0.10893255 0.09820119\n",
      "  0.09123841 0.08377046 0.09416774 0.10082643]\n",
      " [0.1219505  0.09002388 0.10999681 0.09289957 0.10482234 0.08782701\n",
      "  0.10152169 0.09040006 0.10696904 0.0935891 ]\n",
      " [0.10505775 0.07614806 0.09986624 0.0962993  0.11895672 0.0977607\n",
      "  0.10683804 0.09570519 0.10719222 0.09617577]\n",
      " [0.08958614 0.09699698 0.09436832 0.08770871 0.10329747 0.07879161\n",
      "  0.10259763 0.10183323 0.11630504 0.12851487]\n",
      " [0.08563331 0.10983988 0.10397903 0.09808906 0.09688334 0.10479304\n",
      "  0.0918143  0.09961405 0.09975401 0.10959997]\n",
      " [0.10375516 0.08284412 0.11381693 0.10503154 0.10247767 0.08013755\n",
      "  0.11359703 0.08740724 0.1011989  0.10973387]\n",
      " [0.09293633 0.09446102 0.11681377 0.09259469 0.10453099 0.09207699\n",
      "  0.11760448 0.09139668 0.10479037 0.09279468]\n",
      " [0.10014272 0.08870379 0.10894551 0.10903374 0.09283482 0.0825362\n",
      "  0.10442478 0.10627689 0.10132826 0.10577328]\n",
      " [0.09975303 0.0896916  0.10240547 0.09554368 0.09134301 0.08328768\n",
      "  0.11402637 0.08301354 0.12988729 0.11104832]\n",
      " [0.10134459 0.11131687 0.10468834 0.09501454 0.09859264 0.09527829\n",
      "  0.09983865 0.09259996 0.1099091  0.09141703]\n",
      " [0.09042756 0.08028537 0.12468048 0.10459888 0.10940807 0.08478057\n",
      "  0.12602653 0.07743326 0.09730333 0.10505596]\n",
      " [0.09910292 0.09687155 0.10997362 0.09761424 0.10673202 0.09065564\n",
      "  0.10246141 0.11061277 0.09339277 0.09258308]\n",
      " [0.14038988 0.09739902 0.11674776 0.10564936 0.08853863 0.0815687\n",
      "  0.08564993 0.10199525 0.08553866 0.0965228 ]\n",
      " [0.11555217 0.08611188 0.10677012 0.11167415 0.11339578 0.08056304\n",
      "  0.11412184 0.09709716 0.0844687  0.09024515]\n",
      " [0.11368767 0.07673936 0.11231695 0.1049095  0.09827239 0.08576873\n",
      "  0.11657168 0.08449296 0.09856882 0.10867193]] (16.313 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 500 into train/mnist_convnet_model/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 2.2088098526000977.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.estimator.Estimator at 0x11427ccc0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train the model\n",
    "train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"x\": X_train_scaled},\n",
    "    y=y_train,\n",
    "    batch_size=100,\n",
    "    num_epochs=None,\n",
    "    shuffle=True)\n",
    "mnist_classifier.train(\n",
    "    input_fn=train_input_fn,\n",
    "    steps=500,\n",
    "    hooks=[logging_hook])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the model is train, you can evaluate it and print the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2018-09-04-12:46:51\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from train/mnist_convnet_model/model.ckpt-500\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2018-09-04-12:47:05\n",
      "INFO:tensorflow:Saving dict for global step 500: accuracy = 0.594, global_step = 500, loss = 2.1915197\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: train/mnist_convnet_model/model.ckpt-500\n",
      "{'accuracy': 0.594, 'loss': 2.1915197, 'global_step': 500}\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model and print results\n",
    "eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"x\": X_test_scaled},\n",
    "    y=y_test,\n",
    "    num_epochs=1,\n",
    "    shuffle=False)\n",
    "eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)\n",
    "print(eval_results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the current architecture, you get an accuracy of 97% (after 16.00 steps training). You can change the architecture, the batch size and the number of iteration to improve the accuracy. The CNN neural network has performed far better than ANN or logistic regression. In the tutorial on an artificial neural network, you had an accuracy of 96%, which is lower the CNN. The performances of the CNN are impressive with a larger image, both in term of speed computation and accuracy.\n",
    "\n",
    "## Summary\n",
    "\n",
    "A convolutional neural network works very well to evaluate picture. This\n",
    "type of architecture is dominant to recognize objects from a picture or\n",
    "video.\n",
    "To build a CNN, you need to follow six steps:\n",
    "\n",
    "**Step 1**: Input layer:\n",
    "This step reshapes the data. The shape is equal to the square root of\n",
    "the number of pixels. For instance, if a picture has 156 pixels, then\n",
    "the shape is 26x26. You need to specify if the picture has colour or\n",
    "not. If yes, then you had 3 to the shape- 3 for RGB-, otherwise 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "input_layer = tf.reshape(tensor = features[\"x\"],shape =[-1, 28, 28,\n",
    "\n",
    "1])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2**: Convolutional layer\n",
    "\n",
    "Next, you need to create the convolutional layers. You apply different\n",
    "filters to allow the network to learn important feature. You specify the\n",
    "size of the kernel and the amount of filters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "conv1 = tf.layers.conv2d(\n",
    "inputs=input_layer,\n",
    "filters=14,\n",
    "kernel_size=[5, 5],\n",
    "padding=\"same\",\n",
    "activation=tf.nn.relu)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3**: Pooling layer\n",
    "\n",
    "In the third step, you add a pooling layer. This layer decreases the\n",
    "size of the input. It does so by taking the maximum value of the a\n",
    "sub-matrix. For instance, if the sub-matrix is [3,1,3,2], the pooling\n",
    "will return the maximum, which is 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2],\n",
    "\n",
    "strides=2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 4**: Add Convolutional Layer and Pooling Layer\n",
    "\n",
    "In this step, you can add as much as you want conv layers and pooling\n",
    "layers. Google uses architecture with more than 20 conv layers.\n",
    "\n",
    "**Step 5**: Dense layer\n",
    "\n",
    "The step 5 flatten the previous to create a fully connected layers. In\n",
    "this step, you can use different activation function and add a dropout\n",
    "effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 36])\n",
    "\n",
    "dense = tf.layers.dense(inputs=pool2_flat, units=7 * 7 * 36, activation=tf.nn.relu)\n",
    "\n",
    "dropout = tf.layers.dropout(\n",
    "\n",
    "      inputs=dense, rate=0.3, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Step 6**: Logit Layer\n",
    "\n",
    "The final step is the prediction.\n",
    "\n",
    "`logits = tf.layers.dense(inputs=dropout, units=10)`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}