Neural Networks and TensorFlow - 18 - Convolutional Neural Networks - 6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# <center> TensorFlow </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "![](tflogo.png)\n",
    "\n",
    "*<p style='text-align:right'>By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons</p>*\n",
    "\n",
    "- one of the most popular tools for deep learning\n",
    "- released in 2015\n",
    "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n",
    "- tensors - multidimensional arrays\n",
    "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n",
    "\n",
    "![](sqcube.png)\n",
    "\n",
    "![](greyrgb.png)\n",
    "\n",
    "- image tensor flows through the computational graph\n",
    "\n",
    "### Computational graph:\n",
    "\n",
    "![](compgraph.png)\n",
    "\n",
    "- composed of nodes and edges\n",
    "\n",
    "*<p style='text-align:right'>Computational Graph. By Sebastian Raschka via Github</p>*\n",
    "\n",
    "### Advantages and Disadvantages (a few):\n",
    "\n",
    "- open source and continously developed\n",
    "- good visualization tools \n",
    "- scalability and portability\n",
    "- steeper learning curve to become good at it \n",
    "- implies solid knowledge of math\n",
    "- http://tensorflow.org\n",
    "\n",
    "### Types of Operations:\n",
    "\n",
    "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n",
    "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n",
    "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n",
    "- and more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "a = tf.constant('Hello')\n",
    "b = tf.constant(' World!')\n",
    "c = a+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"add:0\", shape=(), dtype=string)\n"
     ]
    }
   ],
   "source": [
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Hello World!'\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess: \n",
    "    runop = sess.run(c)\n",
    "    \n",
    "print(runop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Hello World!'\n"
     ]
    }
   ],
   "source": [
    "c = tf.add(a,b)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    runop = sess.run(c)\n",
    "    \n",
    "print(runop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 88  94]\n",
      "  [214 229]]\n",
      "\n",
      " [[484 508]\n",
      "  [642 674]]]\n"
     ]
    }
   ],
   "source": [
    "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n",
    "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n",
    "mul_c = tf.matmul(mat_a, mat_b)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    runop = sess.run(mul_c)\n",
    "    \n",
    "print(runop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### A typical TensorFlow project:\n",
    "\n",
    "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n",
    "- (2). executing the graph (within a session)\n",
    "- use tensorboard for visualization.\n",
    "\n",
    "### Placeholders:\n",
    "\n",
    "- allow for reusability\n",
    "- can be populated with different data every time you run the graph (feed_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#with tf.Session() as sess:\n",
    "#    sess.run(ops)\n",
    "#    sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n",
    "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n",
    "b = tf.fill((4,1), -1., name='bias')\n",
    "y = tf.matmul(x,w)+b\n",
    "s = tf.reduce_max(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output: 2.9670121669769287\n"
     ]
    }
   ],
   "source": [
    "x_data = np.random.randn(4,9)\n",
    "w_data = np.random.randn(9,1)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n",
    "    \n",
    "print('Output: {}'.format(out_p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Optimization\n",
    "\n",
    "- purpose of a typical project: learn associations between features and labels.\n",
    "\n",
    "### <center> $f(x_{i}) = w^{T}x_{i} + b$ </center> \n",
    "### <center> $y_{i} = f(x_{i}) + \\varepsilon_{i}$ </center>\n",
    "\n",
    "- **Loss Function**: how far the model is from the truth.\n",
    "- train the model to minimize loss (adjusting the parameters).\n",
    "- **MSE (Mean Squared Error)** - averages square distances (y_true, y_pred):\n",
    "\n",
    "## <center> $L(y, \\hat y) = \\frac{1}{n}\\ ∑_{i=1}^{n}(y_{i}-\\hat y_{i})^{2}$ </center>\n",
    "\n",
    "- **Cross Entropy** - similarity between distributions:\n",
    "\n",
    "## <center> $H(p,q)=-∑_{x}p(x)logq(x)$ </center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# X, Y, W, b\n",
    "\n",
    "x = tf.placeholder(tf.float32, shape=[None, 3])\n",
    "y_true = tf.placeholder(tf.float32, shape=None)\n",
    "w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n",
    "b = tf.Variable(0, dtype=tf.float32, name='b')\n",
    "\n",
    "# Output\n",
    "\n",
    "y_pred = tf.matmul(w, tf.transpose(x))+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# MSE (Mean Squared Error)\n",
    "\n",
    "loss = tf.reduce_mean(tf.square(y_true-y_pred))\n",
    "\n",
    "# Cross Entropy\n",
    "\n",
    "loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n",
    "loss = tf.reduce_mean(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Linear Regression \n",
    "\n",
    "- Batch Gradient Descent: entire dataset.\n",
    "- Stochastic Gradient Descent (SGD): mini-batches. \n",
    "- In TensorFlow:\n",
    "\n",
    "<pre>optimizer = tf.train.GradientDescentOptimizer(learning_rate)</pre>\n",
    "<pre>train = optimizer.minimize(loss)</pre>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "x_data = np.random.randn(2000,3)\n",
    "w_real = [0.4, 0.6, 0.2]\n",
    "b_real = -0.3 \n",
    "noise = np.random.randn(1,2000)*0.1 \n",
    "y_data = np.matmul(w_real, x_data.T)+b_real+noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [array([[ 0.4324733 ,  0.62102544,  0.21289678]], dtype=float32), -0.26101521]\n",
      "5 [array([[ 0.39815262,  0.59838271,  0.20108758]], dtype=float32), -0.30086589]\n",
      "10 [array([[ 0.39815292,  0.59838313,  0.20108768]], dtype=float32), -0.30086562]\n"
     ]
    }
   ],
   "source": [
    "num_iters = 10 \n",
    "g = tf.Graph()\n",
    "wb = []\n",
    "\n",
    "with g.as_default():\n",
    "    x = tf.placeholder(tf.float32, shape=[None, 3])\n",
    "    y_true = tf.placeholder(tf.float32, shape=None)\n",
    "    \n",
    "    with tf.name_scope('inference') as scope:\n",
    "        w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n",
    "        b = tf.Variable(0, dtype=tf.float32, name='b')\n",
    "        y_pred = tf.matmul(w, tf.transpose(x))+b\n",
    "        \n",
    "    with tf.name_scope('loss') as scope:\n",
    "        loss = tf.reduce_mean(tf.square(y_true-y_pred))\n",
    "        \n",
    "    with tf.name_scope('training') as scope:\n",
    "        lr = 0.5\n",
    "        optimizer = tf.train.GradientDescentOptimizer(lr)\n",
    "        train = optimizer.minimize(loss)\n",
    "        \n",
    "    init = tf.global_variables_initializer()\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        sess.run(init)\n",
    "        for step in range(num_iters):\n",
    "            sess.run(train, {x:x_data, y_true:y_data})\n",
    "            if(step%5==0):\n",
    "                print(step, sess.run([w,b]))\n",
    "                wb.append(sess.run([w,b]))\n",
    "                \n",
    "        print(10, sess.run([w,b]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Logistic Regression "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1/(1+np.exp(-x))\n",
    "\n",
    "x_data = np.random.randn(20000,3)\n",
    "w_real = [0.4, 0.6, 0.2]\n",
    "b_real = -0.3\n",
    "wb = np.matmul(w_real,x_data.T)+b_real\n",
    "\n",
    "y_data_bef_noise = sigmoid(wb)\n",
    "y_data = np.random.binomial(1, y_data_bef_noise)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [array([[ 0.04192033,  0.06381649,  0.02010243]], dtype=float32), -0.032525264]\n",
      "5 [array([[ 0.18624175,  0.28245297,  0.08911894]], dtype=float32), -0.1444058]\n",
      "10 [array([[ 0.26498973,  0.40079242,  0.12666152]], dtype=float32), -0.20510893]\n",
      "15 [array([[ 0.31052727,  0.46876729,  0.14835776]], dtype=float32), -0.23986366]\n",
      "20 [array([[ 0.33786425,  0.50934786,  0.16139235]], dtype=float32), -0.260488]\n",
      "25 [array([[ 0.35464621,  0.53414452,  0.16940555]], dtype=float32), -0.27300319]\n",
      "30 [array([[ 0.36508736,  0.54951191,  0.1743995 ]], dtype=float32), -0.28070512]\n",
      "35 [array([[ 0.3716363 ,  0.55911851,  0.17753723]], dtype=float32), -0.28548792]\n",
      "40 [array([[ 0.37576416,  0.56515634,  0.17951828]], dtype=float32), -0.28847569]\n",
      "45 [array([[ 0.37837368,  0.56896406,  0.18077263]], dtype=float32), -0.29034957]\n",
      "50 [array([[ 0.3797532 ,  0.57097304,  0.1814366 ]], dtype=float32), -0.29133382]\n"
     ]
    }
   ],
   "source": [
    "num_iters = 50\n",
    "g = tf.Graph()\n",
    "wb = []\n",
    "\n",
    "with g.as_default():\n",
    "    x = tf.placeholder(tf.float32, shape=[None, 3])\n",
    "    y_true = tf.placeholder(tf.float32, shape=None)\n",
    "    \n",
    "    with tf.name_scope('inference') as scope:\n",
    "        w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n",
    "        b = tf.Variable(0, dtype=tf.float32, name='b')\n",
    "        y_pred = tf.matmul(w, tf.transpose(x))+b\n",
    "        \n",
    "    with tf.name_scope('loss') as scope:\n",
    "        loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n",
    "        loss = tf.reduce_mean(loss)\n",
    "        \n",
    "    with tf.name_scope('training') as scope:\n",
    "        lr = 0.5\n",
    "        optimizer = tf.train.GradientDescentOptimizer(lr)\n",
    "        train = optimizer.minimize(loss)\n",
    "        \n",
    "    init = tf.global_variables_initializer()\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        sess.run(init)\n",
    "        \n",
    "        for step in range(num_iters):\n",
    "            sess.run(train, {x:x_data, y_true:y_data})\n",
    "            if(step%5==0):\n",
    "                print(step, sess.run([w,b]))\n",
    "                wb.append(sess.run([w,b]))\n",
    "                \n",
    "        print(50, sess.run([w,b]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Softmax Regression with MNIST\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "<center>![](mnist.png)</center>\n",
    "*<p style='text-align:right'>By Josef Steppan (Own work), via Wikimedia Commons</p>*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "datadir = '/data'\n",
    "num_iters = 1000\n",
    "minibatch_size = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /data\\train-images-idx3-ubyte.gz\n",
      "Extracting /data\\train-labels-idx1-ubyte.gz\n",
      "Extracting /data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting /data\\t10k-labels-idx1-ubyte.gz\n",
      "Accuracy: 91.66%\n"
     ]
    }
   ],
   "source": [
    "data = input_data.read_data_sets(datadir, one_hot=True)\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "W = tf.Variable(tf.zeros([784,10]))\n",
    "y_true = tf.placeholder(tf.float32, [None, 10])\n",
    "y_pred = tf.matmul(x, W)\n",
    "\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_pred, labels=y_true))\n",
    "optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
    "\n",
    "correct_pred = tf.equal(tf.argmax(y_pred,1), tf.argmax(y_true,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    for step in range(num_iters):\n",
    "        batch_xs, batch_ys = data.train.next_batch(minibatch_size)\n",
    "        sess.run(optimizer, feed_dict={x:batch_xs, y_true:batch_ys})\n",
    "        \n",
    "    testing = sess.run(accuracy, feed_dict={x:data.test.images, y_true:data.test.labels})\n",
    "    \n",
    "print('Accuracy: {:.4}%'.format(testing*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Convolutional Neural Networks\n",
    "\n",
    "<center>![](cnn.png)</center>\n",
    "*<p style='text-align:right'>By Aphex34 (Own work) [CC BY-SA 4.0], via Wikimedia Commons</p>*\n",
    "\n",
    "### Good resources:\n",
    "\n",
    "- CS231n from Stanford University\n",
    "- Convolutional Neural Networks (Course 4) by Prof. Andrew Ng - Deep Learning Specialization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Convolution in TensorFlow:\n",
    "\n",
    "<center><pre>tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')</pre></center>\n",
    "\n",
    "- x.shape is [None, 28, 28, 1]\n",
    "- W.shape is [5, 5, 1, 32]\n",
    "- strides - how the filter moves across the image\n",
    "- 'SAME' padding - output size = input size.\n",
    "\n",
    "### Activations\n",
    "\n",
    "- fully connected, covolutional layers - non-linear activations (sigmoid, tanh, relu).\n",
    "\n",
    "### Pooling \n",
    "\n",
    "- reduces the size of the data\n",
    "- reduces no. of hyper-params.\n",
    "\n",
    "<center><pre>tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')</pre></center>\n",
    "\n",
    "### Dropout\n",
    "\n",
    "- for regularization\n",
    "- turns off neurons or units in a given layer.\n",
    "\n",
    "<center><pre>tf.nn.dropout(layer, keep_prob=keep_prob)</pre></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def weight_variable(shape):\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def conv2d(x, W):\n",
    "    return tf.nn.conv2d(x, W, strides=[1,1,1,1], padding='SAME')\n",
    "\n",
    "def max_pool_2x2(x):\n",
    "    return tf.nn.max_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')\n",
    "\n",
    "def conv_layer(input, shape):\n",
    "    W = weight_variable(shape)\n",
    "    b = bias_variable([shape[3]])\n",
    "    return tf.nn.relu(conv2d(input, W)+b)\n",
    "\n",
    "def full_layer(input, size):\n",
    "    in_size = int(input.get_shape()[1])\n",
    "    W = weight_variable([in_size, size])\n",
    "    b = bias_variable([size])\n",
    "    return tf.matmul(input, W)+b "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### The Model (A CNN Approach)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, shape=[None, 784]) \n",
    "y_ = tf.placeholder(tf.float32, shape=[None, 10])\n",
    "x_image = tf.reshape(x, [-1, 28, 28, 1])\n",
    "\n",
    "conv1 = conv_layer(x_image, shape=[5,5,1,32])\n",
    "conv1_pool = max_pool_2x2(conv1)\n",
    "\n",
    "conv2 = conv_layer(conv1_pool, shape=[5,5,32,64])\n",
    "conv2_pool = max_pool_2x2(conv2)\n",
    "conv2_flat = tf.reshape(conv2_pool, [-1,7*7*64])\n",
    "\n",
    "full_1 = tf.nn.relu(full_layer(conv2_flat, 1024))\n",
    "keep_prob = tf.placeholder(tf.float32)\n",
    "full1_drop = tf.nn.dropout(full_1, keep_prob=keep_prob)\n",
    "y_conv = full_layer(full1_drop, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /data\\train-images-idx3-ubyte.gz\n",
      "Extracting /data\\train-labels-idx1-ubyte.gz\n",
      "Extracting /data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting /data\\t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "mnist = input_data.read_data_sets(datadir, one_hot=True)\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_))\n",
    "train_step = tf.train.AdamOptimizer(0.0001).minimize(cross_entropy)\n",
    "correct_prediction = tf.equal(tf.argmax(y_conv,1),tf.argmax(y_,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.03999999910593033\n",
      "step 10, training accuracy 0.18000000715255737\n",
      "step 20, training accuracy 0.5600000023841858\n",
      "step 30, training accuracy 0.47999998927116394\n",
      "step 40, training accuracy 0.7400000095367432\n",
      "step 50, training accuracy 0.6800000071525574\n",
      "step 60, training accuracy 0.8600000143051147\n",
      "step 70, training accuracy 0.8399999737739563\n",
      "step 80, training accuracy 0.7599999904632568\n",
      "step 90, training accuracy 0.7799999713897705\n",
      "step 100, training accuracy 0.8799999952316284\n",
      "step 110, training accuracy 0.800000011920929\n",
      "step 120, training accuracy 0.8799999952316284\n",
      "step 130, training accuracy 0.8999999761581421\n",
      "step 140, training accuracy 0.8999999761581421\n",
      "step 150, training accuracy 0.9200000166893005\n",
      "step 160, training accuracy 0.8799999952316284\n",
      "step 170, training accuracy 0.9399999976158142\n",
      "test accuracy: 0.9029000401496887\n"
     ]
    }
   ],
   "source": [
    "steps=180 \n",
    "                                                             \n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer()) \n",
    "    \n",
    "    for i in range(steps):\n",
    "        \n",
    "        batch = mnist.train.next_batch(50) \n",
    "        \n",
    "        if i % 10 == 0:\n",
    "            train_accuracy = sess.run(accuracy, feed_dict={x: batch[0],y_: batch[1],keep_prob: 1.0}) \n",
    "            print (\"step {}, training accuracy {}\".format(i, train_accuracy)) \n",
    "        sess.run(train_step, feed_dict={x: batch[0], y_: batch[1],keep_prob: 0.5}) \n",
    "\n",
    "        \n",
    "    X = mnist.test.images.reshape(10, 1000, 784)\n",
    "    Y = mnist.test.labels.reshape(10, 1000, 10) \n",
    "    test_accuracy = np.mean([sess.run(accuracy,feed_dict={x:X[i], y_:Y[i],keep_prob:1.0}) for i in range(10)])\n",
    "\n",
    "    \n",
    "print (\"test accuracy: {}\".format(test_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}