example/notebooks/cifar-100.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CIFAR-100 State-of-Art Model\n",
    "----\n",
    "\n",
    "In this example, we will show a new state-of-art result on CIFAR-100.\n",
    "We use a sub-Inception Network with Randomized ReLU (RReLU), and achieved 75.68% accuracy on CIFAR-100.\n",
    "\n",
    "We trained from raw pixel directly, only random crop from 3x28x28 from original 3x32x32 image with random flip, which is same to other experiments. \n",
    "\n",
    "We don't do any parameter search, all hyper-parameters come from ImageNet experience, and this work is just for fun. Definitely you can improve it.\n",
    "\n",
    "Train this network requires 3796MB GPU Memory.\n",
    "\n",
    "----\n",
    "\n",
    "\n",
    "| Model                       | Test Accuracy |\n",
    "| --------------------------- | ------------- |\n",
    "| **Sub-Inception + RReLU** [1], [2]   | **75.68%**       |\n",
    "| Highway Network  [3] | 67.76%        |\n",
    "| Deeply Supervised Network [4]   | 65.43%        |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import logging\n",
    "\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step we will set up basic Factories for Inception"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0), name=None, suffix=''):\n",
    "    conv = mx.symbol.Convolution(data=data, num_filter=num_filter, kernel=kernel, stride=stride, pad=pad, name='conv_%s%s' %(name, suffix))\n",
    "    bn = mx.symbol.BatchNorm(data=conv, name='bn_%s%s' %(name, suffix))\n",
    "    act = mx.symbol.LeakyReLU(data=bn, act_type='rrelu', name='rrelu_%s%s' %(name, suffix))\n",
    "    return act\n",
    "\n",
    "def InceptionFactoryA(data, num_1x1, num_3x3red, num_3x3, num_d3x3red, num_d3x3, pool, proj, name):\n",
    "    # 1x1\n",
    "    c1x1 = ConvFactory(data=data, num_filter=num_1x1, kernel=(1, 1), name=('%s_1x1' % name))\n",
    "    # 3x3 reduce + 3x3\n",
    "    c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), suffix='_reduce')\n",
    "    c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), name=('%s_3x3' % name))\n",
    "    # double 3x3 reduce + double 3x3\n",
    "    cd3x3r = ConvFactory(data=data, num_filter=num_d3x3red, kernel=(1, 1), name=('%s_double_3x3' % name), suffix='_reduce')\n",
    "    cd3x3 = ConvFactory(data=cd3x3r, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_0' % name))\n",
    "    cd3x3 = ConvFactory(data=cd3x3, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_1' % name))\n",
    "    # pool + proj\n",
    "    pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(1, 1), pad=(1, 1), pool_type=pool, name=('%s_pool_%s_pool' % (pool, name)))\n",
    "    cproj = ConvFactory(data=pooling, num_filter=proj, kernel=(1, 1), name=('%s_proj' %  name))\n",
    "    # concat\n",
    "    concat = mx.symbol.Concat(*[c1x1, c3x3, cd3x3, cproj], name='ch_concat_%s_chconcat' % name)\n",
    "    return concat\n",
    "\n",
    "def InceptionFactoryB(data, num_3x3red, num_3x3, num_d3x3red, num_d3x3, name):\n",
    "    # 3x3 reduce + 3x3\n",
    "    c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), suffix='_reduce')\n",
    "    c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), stride=(2, 2), name=('%s_3x3' % name))\n",
    "    # double 3x3 reduce + double 3x3\n",
    "    cd3x3r = ConvFactory(data=data, num_filter=num_d3x3red, kernel=(1, 1),  name=('%s_double_3x3' % name), suffix='_reduce')\n",
    "    cd3x3 = ConvFactory(data=cd3x3r, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), stride=(1, 1), name=('%s_double_3x3_0' % name))\n",
    "    cd3x3 = ConvFactory(data=cd3x3, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), stride=(2, 2), name=('%s_double_3x3_1' % name))\n",
    "    # pool + proj\n",
    "    pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(2, 2), pool_type=\"max\", name=('max_pool_%s_pool' % name))\n",
    "    # concat\n",
    "    concat = mx.symbol.Concat(*[c3x3, cd3x3, pooling], name='ch_concat_%s_chconcat' % name)\n",
    "    return concat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build Network by using Factories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def inception(nhidden, grad_scale):\n",
    "    # data\n",
    "    data = mx.symbol.Variable(name=\"data\")\n",
    "    # stage 2\n",
    "    in3a = InceptionFactoryA(data, 64, 64, 64, 64, 96, \"avg\", 32, '3a')\n",
    "    in3b = InceptionFactoryA(in3a, 64, 64, 96, 64, 96, \"avg\", 64, '3b')\n",
    "    in3c = InceptionFactoryB(in3b, 128, 160, 64, 96, '3c')\n",
    "    # stage 3\n",
    "    in4a = InceptionFactoryA(in3c, 224, 64, 96, 96, 128, \"avg\", 128, '4a')\n",
    "    in4b = InceptionFactoryA(in4a, 192, 96, 128, 96, 128, \"avg\", 128, '4b')\n",
    "    in4c = InceptionFactoryA(in4b, 160, 128, 160, 128, 160, \"avg\", 128, '4c')\n",
    "    in4d = InceptionFactoryA(in4c, 96, 128, 192, 160, 192, \"avg\", 128, '4d')\n",
    "    in4e = InceptionFactoryB(in4d, 128, 192, 192, 256, '4e')\n",
    "    # stage 4\n",
    "    in5a = InceptionFactoryA(in4e, 352, 192, 320, 160, 224, \"avg\", 128, '5a')\n",
    "    in5b = InceptionFactoryA(in5a, 352, 192, 320, 192, 224, \"max\", 128, '5b')\n",
    "    # global avg pooling\n",
    "    avg = mx.symbol.Pooling(data=in5b, kernel=(7, 7), stride=(1, 1), name=\"global_pool\", pool_type='avg')\n",
    "    # linear classifier\n",
    "    flatten = mx.symbol.Flatten(data=avg, name='flatten')\n",
    "    fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=nhidden, name='fc')\n",
    "    softmax = mx.symbol.SoftmaxOutput(data=fc1, name='softmax')\n",
    "    return softmax\n",
    "\n",
    "softmax = inception(100, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make data iterator. Note we convert original CIFAR-100 dataset into image format then pack into RecordIO in purpose of using our build-in image augmentation. For details about RecordIO, please refer ()[]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "\n",
    "train_dataiter = mx.io.ImageRecordIter(\n",
    "    shuffle=True,\n",
    "    path_imgrec=\"./data/train.rec\",\n",
    "    mean_img=\"./data/mean.bin\",\n",
    "    rand_crop=True,\n",
    "    rand_mirror=True,\n",
    "    data_shape=(3, 28, 28),\n",
    "    batch_size=batch_size,\n",
    "    prefetch_buffer=4,\n",
    "    preprocess_threads=2)\n",
    "\n",
    "test_dataiter = mx.io.ImageRecordIter(\n",
    "    path_imgrec=\"./data/test.rec\",\n",
    "    mean_img=\"./data/mean.bin\",\n",
    "    rand_crop=False,\n",
    "    rand_mirror=False,\n",
    "    data_shape=(3, 28, 28),\n",
    "    batch_size=batch_size,\n",
    "    prefetch_buffer=4,\n",
    "    preprocess_threads=2,\n",
    "    round_batch=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Make model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_epoch = 38\n",
    "model_prefix = \"model/cifar_100\"\n",
    "\n",
    "softmax = inception(100, 1.0)\n",
    "\n",
    "model = mx.model.FeedForward(ctx=mx.gpu(), symbol=softmax, num_epoch=num_epoch,\n",
    "                             learning_rate=0.05, momentum=0.9, wd=0.0001)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit first stage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Start training with [gpu(0)]\n",
      "INFO:root:Batch [200]\tSpeed: 157.49 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 144.49 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 142.79 samples/sec\n",
      "INFO:root:Iteration[0] Train-accuracy=0.184423\n",
      "INFO:root:Iteration[0] Time cost=342.269\n",
      "INFO:root:Iteration[0] Validation-accuracy=0.279757\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0001.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 142.65 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 141.95 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 141.44 samples/sec\n",
      "INFO:root:Iteration[1] Train-accuracy=0.363516\n",
      "INFO:root:Iteration[1] Time cost=352.763\n",
      "INFO:root:Iteration[1] Validation-accuracy=0.421775\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0002.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 142.58 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 141.47 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 141.47 samples/sec\n",
      "INFO:root:Iteration[2] Train-accuracy=0.464609\n",
      "INFO:root:Iteration[2] Time cost=353.075\n",
      "INFO:root:Iteration[2] Validation-accuracy=0.501891\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0003.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 142.24 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 141.21 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 141.11 samples/sec\n",
      "INFO:root:Iteration[3] Train-accuracy=0.529690\n",
      "INFO:root:Iteration[3] Time cost=353.893\n",
      "INFO:root:Iteration[3] Validation-accuracy=0.548368\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0004.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 142.36 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 141.10 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 141.13 samples/sec\n",
      "INFO:root:Iteration[4] Train-accuracy=0.572331\n",
      "INFO:root:Iteration[4] Time cost=354.524\n",
      "INFO:root:Iteration[4] Validation-accuracy=0.588973\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0005.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.93 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 141.05 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 141.14 samples/sec\n",
      "INFO:root:Iteration[5] Train-accuracy=0.610455\n",
      "INFO:root:Iteration[5] Time cost=354.183\n",
      "INFO:root:Iteration[5] Validation-accuracy=0.604299\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0006.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.44 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.94 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.65 samples/sec\n",
      "INFO:root:Iteration[6] Train-accuracy=0.634563\n",
      "INFO:root:Iteration[6] Time cost=355.035\n",
      "INFO:root:Iteration[6] Validation-accuracy=0.598229\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0007.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.57 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.94 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.60 samples/sec\n",
      "INFO:root:Iteration[7] Train-accuracy=0.662832\n",
      "INFO:root:Iteration[7] Time cost=355.113\n",
      "INFO:root:Iteration[7] Validation-accuracy=0.618432\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0008.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.51 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.69 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.63 samples/sec\n",
      "INFO:root:Iteration[8] Train-accuracy=0.677390\n",
      "INFO:root:Iteration[8] Time cost=355.880\n",
      "INFO:root:Iteration[8] Validation-accuracy=0.631270\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0009.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.22 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.60 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.58 samples/sec\n",
      "INFO:root:Iteration[9] Train-accuracy=0.695923\n",
      "INFO:root:Iteration[9] Time cost=355.619\n",
      "INFO:root:Iteration[9] Validation-accuracy=0.639431\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0010.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.21 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.70 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.57 samples/sec\n",
      "INFO:root:Iteration[10] Train-accuracy=0.712428\n",
      "INFO:root:Iteration[10] Time cost=355.604\n",
      "INFO:root:Iteration[10] Validation-accuracy=0.651373\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0011.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.07 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.58 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.48 samples/sec\n",
      "INFO:root:Iteration[11] Train-accuracy=0.729473\n",
      "INFO:root:Iteration[11] Time cost=355.796\n",
      "INFO:root:Iteration[11] Validation-accuracy=0.645502\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0012.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.22 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.43 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.54 samples/sec\n",
      "INFO:root:Iteration[12] Train-accuracy=0.739051\n",
      "INFO:root:Iteration[12] Time cost=356.473\n",
      "INFO:root:Iteration[12] Validation-accuracy=0.663217\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0013.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.15 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.58 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Iteration[13] Train-accuracy=0.752821\n",
      "INFO:root:Iteration[13] Time cost=355.815\n",
      "INFO:root:Iteration[13] Validation-accuracy=0.653961\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0014.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.89 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.35 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Iteration[14] Train-accuracy=0.759083\n",
      "INFO:root:Iteration[14] Time cost=356.155\n",
      "INFO:root:Iteration[14] Validation-accuracy=0.661027\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0015.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.13 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.52 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.38 samples/sec\n",
      "INFO:root:Iteration[15] Train-accuracy=0.770367\n",
      "INFO:root:Iteration[15] Time cost=355.945\n",
      "INFO:root:Iteration[15] Validation-accuracy=0.669984\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0016.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.21 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.57 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.44 samples/sec\n",
      "INFO:root:Iteration[16] Train-accuracy=0.781030\n",
      "INFO:root:Iteration[16] Time cost=356.440\n",
      "INFO:root:Iteration[16] Validation-accuracy=0.661027\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0017.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.00 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.44 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.44 samples/sec\n",
      "INFO:root:Iteration[17] Train-accuracy=0.787232\n",
      "INFO:root:Iteration[17] Time cost=356.027\n",
      "INFO:root:Iteration[17] Validation-accuracy=0.676652\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0018.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.77 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.56 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Iteration[18] Train-accuracy=0.796975\n",
      "INFO:root:Iteration[18] Time cost=356.066\n",
      "INFO:root:Iteration[18] Validation-accuracy=0.678145\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0019.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.01 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.42 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.42 samples/sec\n",
      "INFO:root:Iteration[19] Train-accuracy=0.805378\n",
      "INFO:root:Iteration[19] Time cost=356.019\n",
      "INFO:root:Iteration[19] Validation-accuracy=0.677548\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0020.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.17 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.52 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.57 samples/sec\n",
      "INFO:root:Iteration[20] Train-accuracy=0.808903\n",
      "INFO:root:Iteration[20] Time cost=356.454\n",
      "INFO:root:Iteration[20] Validation-accuracy=0.665207\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0021.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.94 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.29 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.28 samples/sec\n",
      "INFO:root:Iteration[21] Train-accuracy=0.815761\n",
      "INFO:root:Iteration[21] Time cost=356.311\n",
      "INFO:root:Iteration[21] Validation-accuracy=0.671377\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0022.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.83 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.27 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.30 samples/sec\n",
      "INFO:root:Iteration[22] Train-accuracy=0.822803\n",
      "INFO:root:Iteration[22] Time cost=356.518\n",
      "INFO:root:Iteration[22] Validation-accuracy=0.670482\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0023.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.10 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.24 samples/sec\n",
      "INFO:root:Iteration[23] Train-accuracy=0.827545\n",
      "INFO:root:Iteration[23] Time cost=356.127\n",
      "INFO:root:Iteration[23] Validation-accuracy=0.671178\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0024.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.91 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.28 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.30 samples/sec\n",
      "INFO:root:Iteration[24] Train-accuracy=0.833760\n",
      "INFO:root:Iteration[24] Time cost=356.974\n",
      "INFO:root:Iteration[24] Validation-accuracy=0.670382\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0025.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.00 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.41 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.41 samples/sec\n",
      "INFO:root:Iteration[25] Train-accuracy=0.839609\n",
      "INFO:root:Iteration[25] Time cost=356.055\n",
      "INFO:root:Iteration[25] Validation-accuracy=0.677946\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0026.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.04 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.42 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.43 samples/sec\n",
      "INFO:root:Iteration[26] Train-accuracy=0.841909\n",
      "INFO:root:Iteration[26] Time cost=356.007\n",
      "INFO:root:Iteration[26] Validation-accuracy=0.677946\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0027.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.88 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.39 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.14 samples/sec\n",
      "INFO:root:Iteration[27] Train-accuracy=0.846411\n",
      "INFO:root:Iteration[27] Time cost=356.341\n",
      "INFO:root:Iteration[27] Validation-accuracy=0.682126\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0028.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.16 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.15 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 139.99 samples/sec\n",
      "INFO:root:Iteration[28] Train-accuracy=0.847966\n",
      "INFO:root:Iteration[28] Time cost=357.334\n",
      "INFO:root:Iteration[28] Validation-accuracy=0.676652\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0029.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.74 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.37 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.35 samples/sec\n",
      "INFO:root:Iteration[29] Train-accuracy=0.860075\n",
      "INFO:root:Iteration[29] Time cost=356.321\n",
      "INFO:root:Iteration[29] Validation-accuracy=0.674363\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0030.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.75 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.35 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.34 samples/sec\n",
      "INFO:root:Iteration[30] Train-accuracy=0.856554\n",
      "INFO:root:Iteration[30] Time cost=356.349\n",
      "INFO:root:Iteration[30] Validation-accuracy=0.669686\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0031.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.06 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.47 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.46 samples/sec\n",
      "INFO:root:Iteration[31] Train-accuracy=0.861436\n",
      "INFO:root:Iteration[31] Time cost=355.920\n",
      "INFO:root:Iteration[31] Validation-accuracy=0.676254\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0032.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.03 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.28 samples/sec\n",
      "INFO:root:Iteration[32] Train-accuracy=0.858416\n",
      "INFO:root:Iteration[32] Time cost=357.042\n",
      "INFO:root:Iteration[32] Validation-accuracy=0.686405\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0033.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.66 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.16 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.12 samples/sec\n",
      "INFO:root:Iteration[33] Train-accuracy=0.868858\n",
      "INFO:root:Iteration[33] Time cost=356.653\n",
      "INFO:root:Iteration[33] Validation-accuracy=0.679041\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0034.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.62 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.55 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.50 samples/sec\n",
      "INFO:root:Iteration[34] Train-accuracy=0.870319\n",
      "INFO:root:Iteration[34] Time cost=356.121\n",
      "INFO:root:Iteration[34] Validation-accuracy=0.671676\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0035.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.81 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.39 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.39 samples/sec\n",
      "INFO:root:Iteration[35] Train-accuracy=0.874060\n",
      "INFO:root:Iteration[35] Time cost=356.216\n",
      "INFO:root:Iteration[35] Validation-accuracy=0.684813\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0036.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.11 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.49 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.48 samples/sec\n",
      "INFO:root:Iteration[36] Train-accuracy=0.872043\n",
      "INFO:root:Iteration[36] Time cost=356.771\n",
      "INFO:root:Iteration[36] Validation-accuracy=0.670581\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0037.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.93 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.48 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.46 samples/sec\n",
      "INFO:root:Iteration[37] Train-accuracy=0.875900\n",
      "INFO:root:Iteration[37] Time cost=355.997\n",
      "INFO:root:Iteration[37] Validation-accuracy=0.681330\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0038.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.09 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.45 samples/sec\n",
      "INFO:root:Iteration[38] Train-accuracy=0.879902\n",
      "INFO:root:Iteration[38] Time cost=355.928\n",
      "INFO:root:Iteration[38] Validation-accuracy=0.688694\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100-0039.params\"\n"
     ]
    }
   ],
   "source": [
    "model.fit(X=train_dataiter,\n",
    "          eval_data=test_dataiter,\n",
    "          eval_metric=\"accuracy\",\n",
    "          batch_end_callback=mx.callback.Speedometer(batch_size, 200),\n",
    "          epoch_end_callback=mx.callback.do_checkpoint(model_prefix))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without reducing learning rate, this model is able to achieve state-of-art result.\n",
    "\n",
    "Let's reduce learning rate to train a few more rounds.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Start training with [gpu(0)]\n",
      "INFO:root:Batch [200]\tSpeed: 147.84 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 139.77 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.17 samples/sec\n",
      "INFO:root:Iteration[0] Train-accuracy=0.951866\n",
      "INFO:root:Iteration[0] Time cost=353.261\n",
      "INFO:root:Iteration[0] Validation-accuracy=0.744924\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0001.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 141.02 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.35 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.39 samples/sec\n",
      "INFO:root:Iteration[1] Train-accuracy=0.976012\n",
      "INFO:root:Iteration[1] Time cost=356.142\n",
      "INFO:root:Iteration[1] Validation-accuracy=0.747213\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0002.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.77 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.49 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 139.74 samples/sec\n",
      "INFO:root:Iteration[2] Train-accuracy=0.983335\n",
      "INFO:root:Iteration[2] Time cost=356.680\n",
      "INFO:root:Iteration[2] Validation-accuracy=0.746716\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0003.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.64 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 140.16 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 140.17 samples/sec\n",
      "INFO:root:Iteration[3] Train-accuracy=0.987076\n",
      "INFO:root:Iteration[3] Time cost=356.758\n",
      "INFO:root:Iteration[3] Validation-accuracy=0.755971\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0004.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.58 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 139.97 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 139.89 samples/sec\n",
      "INFO:root:Iteration[4] Train-accuracy=0.989850\n",
      "INFO:root:Iteration[4] Time cost=358.025\n",
      "INFO:root:Iteration[4] Validation-accuracy=0.752090\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0005.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.18 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 139.61 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 139.32 samples/sec\n",
      "INFO:root:Iteration[5] Train-accuracy=0.991037\n",
      "INFO:root:Iteration[5] Time cost=358.366\n",
      "INFO:root:Iteration[5] Validation-accuracy=0.752886\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0006.params\"\n",
      "INFO:root:Batch [200]\tSpeed: 140.42 samples/sec\n",
      "INFO:root:Batch [400]\tSpeed: 139.11 samples/sec\n",
      "INFO:root:Batch [600]\tSpeed: 139.29 samples/sec\n",
      "INFO:root:Iteration[6] Train-accuracy=0.992858\n",
      "INFO:root:Iteration[6] Time cost=358.961\n",
      "INFO:root:Iteration[6] Validation-accuracy=0.756867\n",
      "INFO:root:Saved checkpoint to \"model/cifar_100_stage2-0007.params\"\n"
     ]
    }
   ],
   "source": [
    "# load params from saved model\n",
    "num_epoch = 38\n",
    "model_prefix = \"model/cifar_100\"\n",
    "tmp_model = mx.model.FeedForward.load(model_prefix, epoch)\n",
    "\n",
    "# create new model with params\n",
    "num_epoch = 6\n",
    "model_prefix = \"model/cifar_100_stage2\"\n",
    "model = mx.model.FeedForward(ctx=mx.gpu(), symbol=softmax, num_epoch=num_epoch,\n",
    "                             learning_rate=0.01, momentum=0.9, wd=0.0001,\n",
    "                             arg_params=tmp_model.arg_params, aux_params=tmp_model.aux_params,)\n",
    "\n",
    "\n",
    "model.fit(X=train_dataiter,\n",
    "          eval_data=test_dataiter,\n",
    "          eval_metric=\"accuracy\",\n",
    "          batch_end_callback=mx.callback.Speedometer(batch_size, 200),\n",
    "          epoch_end_callback=mx.callback.do_checkpoint(model_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reference**\n",
    "\n",
    "[1] Ioffe, Sergey, and Christian Szegedy. \"Batch normalization: Accelerating deep network training by reducing internal covariate shift.\" arXiv preprint arXiv:1502.03167 (2015).\n",
    "\n",
    "[2] Xu, Bing, et al. \"Empirical Evaluation of Rectified Activations in Convolutional Network.\" arXiv preprint arXiv:1505.00853 (2015).\n",
    "\n",
    "[3] Srivastava, Rupesh Kumar, Klaus Greff, and Jürgen Schmidhuber. \"Highway Networks.\" arXiv preprint arXiv:1505.00387 (2015).\n",
    "\n",
    "[4] Lee, Chen-Yu, et al. \"Deeply-supervised nets.\" arXiv preprint arXiv:1409.5185 (2014)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}