Tutorials/CNTK_103A_MNIST_DataLoader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# CNTK 103 Part A: MNIST Data Loader\n",
    "\n",
    "This tutorial is targeted to individuals who are new to CNTK and to machine learning. We assume you have completed or are familiar with CNTK 101 and 102. In this tutorial, we will download and pre-process the MNIST digit images to be used for building different models to recognize handwritten digits. We will extend CNTK 101 and 102 to be applied to this data set. Additionally we will introduce a convolutional network to achieve superior performance. This is the first example, where we will train and evaluate a neural network based model on read real world data.  \n",
    "\n",
    "CNTK 103 tutorial is divided into multiple parts:\n",
    "- Part A: Familiarize with the [MNIST][] database that will be used later in the tutorial\n",
    "- Subsequent parts in this 103 series would be using the MNIST data with different types of networks.\n",
    "\n",
    "\n",
    "[MNIST]: http://yann.lecun.com/exdb/mnist/\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the relevant modules to be used later\n",
    "from __future__ import print_function\n",
    "import gzip\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import shutil\n",
    "import struct\n",
    "import sys\n",
    "\n",
    "try: \n",
    "    from urllib.request import urlretrieve \n",
    "except ImportError: \n",
    "    from urllib import urlretrieve\n",
    "\n",
    "# Config matplotlib for inline plotting\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data download\n",
    "\n",
    "We will download the data into local machine. The MNIST database is a standard handwritten digits that has been widely used for training and testing of machine learning algorithms. It has a training set of 60,000 images and a test set of 10,000 images with each image being 28 x 28 pixels. This set is easy to use visualize and train on any computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Functions to load MNIST images and unpack into train and test set.\n",
    "# - loadData reads image data and formats into a 28x28 long array\n",
    "# - loadLabels reads the corresponding labels data, 1 for each image\n",
    "# - load packs the downloaded image and labels data into a combined format to be read later by \n",
    "#   CNTK text reader \n",
    "\n",
    "def loadData(src, cimg):\n",
    "    print ('Downloading ' + src)\n",
    "    gzfname, h = urlretrieve(src, './delete.me')\n",
    "    print ('Done.')\n",
    "    try:\n",
    "        with gzip.open(gzfname) as gz:\n",
    "            n = struct.unpack('I', gz.read(4))\n",
    "            # Read magic number.\n",
    "            if n[0] != 0x3080000:\n",
    "                raise Exception('Invalid file: unexpected magic number.')\n",
    "            # Read number of entries.\n",
    "            n = struct.unpack('>I', gz.read(4))[0]\n",
    "            if n != cimg:\n",
    "                raise Exception('Invalid file: expected {0} entries.'.format(cimg))\n",
    "            crow = struct.unpack('>I', gz.read(4))[0]\n",
    "            ccol = struct.unpack('>I', gz.read(4))[0]\n",
    "            if crow != 28 or ccol != 28:\n",
    "                raise Exception('Invalid file: expected 28 rows/cols per image.')\n",
    "            # Read data.\n",
    "            res = np.fromstring(gz.read(cimg * crow * ccol), dtype = np.uint8)\n",
    "    finally:\n",
    "        os.remove(gzfname)\n",
    "    return res.reshape((cimg, crow * ccol))\n",
    "\n",
    "def loadLabels(src, cimg):\n",
    "    print ('Downloading ' + src)\n",
    "    gzfname, h = urlretrieve(src, './delete.me')\n",
    "    print ('Done.')\n",
    "    try:\n",
    "        with gzip.open(gzfname) as gz:\n",
    "            n = struct.unpack('I', gz.read(4))\n",
    "            # Read magic number.\n",
    "            if n[0] != 0x1080000:\n",
    "                raise Exception('Invalid file: unexpected magic number.')\n",
    "            # Read number of entries.\n",
    "            n = struct.unpack('>I', gz.read(4))\n",
    "            if n[0] != cimg:\n",
    "                raise Exception('Invalid file: expected {0} rows.'.format(cimg))\n",
    "            # Read labels.\n",
    "            res = np.fromstring(gz.read(cimg), dtype = np.uint8)\n",
    "    finally:\n",
    "        os.remove(gzfname)\n",
    "    return res.reshape((cimg, 1))\n",
    "\n",
    "def try_download(dataSrc, labelsSrc, cimg):\n",
    "    data = loadData(dataSrc, cimg)\n",
    "    labels = loadLabels(labelsSrc, cimg)\n",
    "    return np.hstack((data, labels))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download the data\n",
    "\n",
    "The MNIST data is provided as train and test set. Training set has 60000 images while the test set has 10000 images. Let us download the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading train data\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Done.\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Done.\n",
      "Downloading test data\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Done.\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "# URLs for the train image and labels data\n",
    "url_train_image = 'http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz'\n",
    "url_train_labels = 'http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz'\n",
    "num_train_samples = 60000\n",
    "\n",
    "print(\"Downloading train data\")\n",
    "train = try_download(url_train_image, url_train_labels, num_train_samples)\n",
    "\n",
    "\n",
    "url_test_image = 'http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz'\n",
    "url_test_labels = 'http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz'\n",
    "num_test_samples = 10000\n",
    "\n",
    "print(\"Downloading test data\")\n",
    "test = try_download(url_test_image, url_test_labels, num_test_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image Label:  3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFfCAYAAACfj30KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJztnel24zrPrOnZztC9v/u/yXd3J56H82MfqEslgKQcu+Uk\n9azFJUpybFmOShAIgKPL5ZKEEEIMw3joAxBCiO+MRFgIIQZEIiyEEAMiERZCiAGRCAshxIBIhIUQ\nYkAkwkIIMSASYSGEGBCJsBBCDIhEWAghBmQ69AH8f5Q7LYT4ioxKL5AlLIQQAyIRFkKIAZEICyHE\ngEiEhRBiQCTCQggxIBJhIYQYEImwEEIMiERYCCEGRCIshBADIhEWQogBkQgLIcSASISFEGJAJMJC\nCDEgEmEhhBgQibAQQgyIRFgIIQZEIiyEEAMiERZCiAGRCAshxIBIhIUQYkAkwkIIMSASYSGEGBCJ\nsBBCDIhEWAghBkQiLIQQAyIRFkKIAZEICyHEgEiEhRBiQCTCQggxIBJhIYQYEImwEEIMiERYCCEG\nRCIshBADIhEWQogBkQgLIcSASISFEGJAJMJCCDEgEmEhhBgQibAQQgyIRFgIIQZEIiyEEAMiERZC\niAGRCAshxIBIhIUQYkAkwkIIMSASYSGEGBCJsBBCDIhEWAghBmQ69AEI0YfL5dJ7v23DfTXbouXf\nZjQaZbf3XXrvmdsn7otEWHxackJq/fP5nC6XS6vltpWW1u5JrZiOx+M0Go3SeDxu9XlpfWz2Pl4T\nfxeJsPh0eBaq1z+fz73a6XTq9L3lPUU4J5K8bTKZpPF4nCaTSdjHpSfYLNri7yMRFp8StkrZsjXr\n9XQ6tQSUG24/Ho9Vy/P5fLfvhSLrWbC4fTKZpOl02rTcugmuCbL1L5dLGo/Hrc8WfxeJsPi0eMKL\nAnw+n1sCmutbOxwO7hL7p9Ppbt8p51ZAER6Px2k6nabZbNZq0TazkLFdLpc0mUxSSv8J8FA+7++O\nRFh8OiLR5T6KbCSuuNzv9+lwOLT63vLeIowNRZe3zefzsM1ms9Y6Wsbs17b3kwgPg0RYfCo8fzAO\nnmEfBRgFNhLc3W6X9vt9cXk8HrPHGD3S14gcugu48b7FYpFt6OM2ixhvVnas9hpzS4i/i0RYfDpy\n7gceaDMRNrE1IeW22+3SbrdL2+3WXWL/cDi4x9XHn+oJslmkPKDmbZtMJmm5XLoN3SbeDYo/T1bw\nsEiExaclJ8Aswii03Pb7fdput2m73abNZtMsrc/bUYQ/OpDFoshRDl7kg7XVatVqZtGj+Nr7egKM\nos/7xd9DIiw+JZEvGMPJ2B2BFi1autbW63XabDbh0vr7/T6ldLukBhRLFtpoUG06naanp6f09PTU\nWOdo/bIIRxYwCrY1RUj8XSTC4tNR447AsDMT4cjiNQFer9fp/f3dXWJ/t9vdTahQdHEwjZfT6bTl\nHsGIDxRgs3QNFGCzgPlvxN9FIizuQt8LOpf1hv0o3tdr+/2+sWDRvcDbrL2/v7eaCS+vmyVcQ9/Q\nr0hwvW2eFYufi4LLljG6Itgd0ed4ZTV/HImw+CvwhZ1LL871ObGC435xH7sb0PL1tpvbAQfv0Mfa\nN0kDs9xqhY2TT+xzR6NRExpn74tW/na7bfmPURzP53NarVYdVwXHHNv5w+QN7/uI2yIRFnelpkBO\nLm2Y+5xcYYKLPlHbbuLk+YF5yQ3D0ex9+9SN8ASsjxCzq8XLZsOoD7OSWYDZVYPWMFrCs9ms8x0j\n/7DE+LZIhMXdiESXL3R0IaBFG1m8tZltXkRELkoCw9Ewjtgb7MqRq3pW8/eeAHugJYzhbPZ6fHpA\nSx5rT1iGnZeSzZXVTJQ1eHdbJMLirni1HXjds3A9UcWY36hxXDDGAXuxwV6ssDUe8PqIAOP+WiE2\nAWY3iJ03u9HkLGC7gXkWsAnwfD7viDAKObtVJMC3RSIs7o4XzYDNhIIFtSbLjZellGMU59y2yCec\nE9BaccoJseeKiF5zOBwad0IkwHYTQR8wDu7N5/NOeBuKbeQXlhjfDomwuBsl8UURRkvXs1C9LDev\nsaDWNM/CZgu8ZAn3FaSSReyJMIvzZDJJh8OhKMD2GvYBz+fz5jyxCNdY9OI2SITFXfCiIVhE2B3B\n8byez5ZFmes6YN8T09y6V1UNS15GlvC1ghQJsTdweblcmvRiq/NwPB5b9R5wEA6/23w+71Rem8/n\nTY0Js/ZZhL1aEhLf2yMRFnfF8wPzbBU48OZltmFYWTSo5g28sbjykrflag/fWoDx770IkpRSSxDZ\nRYADdhzCdzwe02w2a25qs9msI8Dz+Twtl0vXEjbx5aI+eKwS49shERZ3IxqE46VXaMdLqrBsNa+o\nTlRopySubOlyi0K8UrqdEHkWMQtzNLuG7ffqZXCBd6svvFgs0nK5bM6Rl/JslrAJsQbl7odEWNyV\nkhDj47NX48FShq1ZMoUX28txvywsLLa87h0rW/LGrcUIhdizjDlcDPfj9/HqTbALYrlcptVq1XHb\n4PnAympe7LAE+XZIhEU10WOztx4JnVfjAcUTrV6vnkOU8ea14/GYtWgjC7cEWqQ1r/POT+n1jHfu\nLYMOBzgnk0k6Ho+d6msmvpvNJq1Wq2bJsdH7/b7xP+MUSLbNjlMCfDskwqKK3MCat15b3+F0OoWV\ny7w+ux8wEoKL2LDgev5ppJSm22f9o7WFa16DUx7x6ziyAv3tmKZts3BgrDHPxMH1Ky6XSzNvnfg4\nEmFRDT/65h7tvfRiL/LgeDyGdXy9vjfbBYaWedEMfIPwXAyegF679Lb1sbSjJw7vpsGTgeLvZP3L\n5dIK/9tsNmmxWDRia4Jqboblctn4j02krW/HYO4KWcQfRyIsquGBtFJ6cW0sblTj1+tHscFsCXsV\nxjxrmMkNgHnr3t/wttpzm+t7SxZh/htMvGBLeL1et+pN4O+7Wq0akcbpklCAvSQScR0SYVENi3Au\n7rY2G42t2lI/SsKIBpg8C9hzRaTUFtlIfGtfg9vw/NWcY1tGfVt6FrgXVZFSu9jPdrtt5pzDdGf7\nbXe7XTN4x4kq5ieezWa9rHsRIxEW1eRCylgQSxNm8rZIoLlfKtoTWcGl5vl3ufGjf60w9zm/tvTc\nJrkbCL8PvwafOtAFYaALab/fp6enp5YA2/e3SAtZwrdDIiyqMRFG4SsVwsmVj7Rl5LLAdS+NOOpH\nlrB9h5yIRcKLApwTY28fnr+ac1zb+D0j10VKfwbmIhcE3li5ZoadF7OALepElvBtkAiLaqIL1ksv\n9rLdeEoh63uDdTmxzSVdYIse43M+4ZR8IUYxZqEtrfchZ8HXuFeidbSEPReEibTdFFFk0QWxWCw6\nJS/Fx5AIi2pyIhzNVswTZXoTZ3rZa9F6LiKDt+UsQ29bJL7cShYyL73zmDvHLLhYO8LbjpEQ0fuY\nyLIAowtit9s1IhsJsLmEJMK3QyIsqvFEGK1fL86XG06cac1KLZaSKaKwMy8Vmt0OLHylR+kaQWbB\nrRXhHN53QfG1aAcTQYuAsGNmyxlDBjEMjd1K9hvO5/NW2rLFBluyB7op5I64DRJhUU3JEkbx5Ukz\ncw0tr5oBqZrHb+v3pSS+PIOFJ8q83uf8ejcbFGITUXZ1cPYcPx0cj8fwN7TBNmtmdaMFbGnO5peX\nJXw7JMKiGu8CRivKRNjE9e3tLb29vYV9Wx/igs5FQ6Dgei1yU0SCXAtbwNw8cR2NRs2ThIkzv59Z\nvbgNBZhndR6Px01yhoWqWYw2hgDKEr4NEmFRDV7UXmIGR0dwIkWpLGQJE5i+f+slUvDSwq/MKvT6\n2CLr1xPjPpT83GzdemF6WOgdb3DeQJ9Z1yboKaWW0OYiTcRtkAiLavji9ZIzOGQtymTz/LY5UMzs\nkbz272qSKiaTSZOia/V2MW0XlyjCJTG+VoRrlhxHPZlM0n6/bwmwZ4mb8Jr4IpdLu+5HLulF3AaJ\nsOgFuiQ8S5gLrHNK8UcFGLfV/H0uoQIjG2y+NU7V9Zonwjm/cB9q6x9bZps1nuzT8wVHYW+efzmq\nsyzxvT0SYVFNNODDKcqcBfcRd0ROxEpCzNZvLrzMXA82CLVcLpvG6zZdUE0YW1+fcFSPw9tmfvgo\n/dib/ojFlz8fj0HuiL+DRFhUgxdhH0s4V2IyR40VWSPEUbQDrnPR86enp7RardwlztlWckVcI8K5\nKZnwnHMJypTaAsyzMNtnoBB7+3LuiL5PMaKMRFj04hqfcK6uQ0Sfx/gaIY4sVhMxc0fgzBPPz8+t\n9vT0lJ6fn9NisahyRfR1R6AIR2U/seUsYNyHA5olAbbEDs8VIXfEfZAIi2q86Ah2SXjREWzJlS7m\nvn5U+xt+z5wVjLNOmAijO8IE9+XlpWmvr6/p5eUlLZfLKgv4Gp8wV6TLVaozK5tDB6MaEZjMkdKf\nyAlOAvFcEd7gnLgNEmFRDVpRLMAYM5yzhEtW1TUCjH9bI8Q89Y8lJVhcrLkjnp+f0+vra9N+/PiR\nXl9f02q1qnZF9HVHcLGi3BJFFQUYZ8bguOGU2tEReI5MjL00cgnw/ZAIi16wNVyKE7YA/xp3xEcE\nGN/DE4nIFYGTYbJP2CzhHz9+pB8/fqSfP3+mnz9/tkS45Iq4RoRzhfBt+36/Tym1XRCYOMOWMH4G\nYvHBKMpRZITcEfdBIiyqiZI10CdcsoQjd8QtBBjfC1N7c9lwOB28ifBqtWp8wuaG+PnzZ/rnn3/S\nP//8k56fn6ss4Gt8wlGxem+7+W+9+g/sL7b3R+uZk1asXyoJKmv4tkiEvzF4Uda+3huY8yxhLsCe\nixOujYLgY6l5vSeWLMCWBRdZwuaK+PnzZ/q///u/9Pz8fFOfMP4GHNqXW55Op9bNb7PZNEklNZZw\ndA6jELWPivEtb7RfCYnwN8cuoujixP08Qu+tcyyr9xjL/lmGL9bo4i1d1GbhYvqxFanhvomuza9m\nlqR978PhkLbbbXp/f0+Xy6XjU871S98rOjf2PufzuZnlGH8PTKn26lp4VnANOb+/1Ry2GGUb3IsS\nYnCf8JEIi87F7fXN8uX4YF6vSU9mEc49GntL3h+tm9By6rG3bbVapZeXl44Ip5SaFOHdbpfW63Uj\nwmhJ49K+k73umu/hCTGfR65rgULsCd+1QuzFgaMIe9Y/9yXCMRLhb473eBm1WivYE2IDBcZzSbAF\n1WfJ2zAV2VwN0bq5ICwZYzabNW4CtIQtesAr9IPbkRrLkPehAHs3sZRS6xgiS/gaIY4sYXQ3oQjz\nEwAety1FjERYpJS6gfzc52iIkhWcS3dFoTFYhErr3t/wNov7rW3L5bJTHyKlP5Ywuiew/q4JME+A\nmXtE5+/N2zwh5v2RJRx9Fv7WOSH2/P4cgmgizE8F1senABwoFV0kwqIjwCi81mdfb40Y59wRLMA5\n8a3Zz9tSSo0IY90Ha7wdXRU4HTxawimlpo8WNUd8oIh6ff7+3nmwz0afOb8XHmutJYy/eW6ffW/P\nEkafsPmm7SmAfdd9Y6W/IxLhb44nwF4rDcqxGKOQsxVsAsDCdOs2m81aBXgs9MzrYz0IFDQTI1ta\nNtpisXBvNvYd2S9q202QsMwku1NQhL3t1ryBuZxP2PvtvddwKGLOJ5xSaj0F4DkYj8etbcJHIixS\nSrEYY6hSjQXMqcle9AU+ouI6W4K59dLrR6NRJwPOhNcrysN+XMS+p1nDk8nEdbmwUE6n03Q6nRpB\nNdH1UokjIeZtJt4mwizEbAV/VIijIk1mDdv58W62kU9btJEIi6wAc42Ikh/Ytnnia3j+W+/RvbSt\ntG7+XRRfC0Wzvq1Pp1O3boIXLzsa/UnttfOGx2AWKb6GH8lxYDInwLiOTxcYfudZwn0z9fAYagR4\ns9mk5XKZUkoda9eL6hAxEuFvTh8BrrWCTbDYXxstPZHNhTrlxBmX6PfFojzWXl5emv5kMumkBeP3\nwWw1s0i9QThM/rBqZohZvjxYya9J6Y9Q29+gcJ/PZ3dgrq8ljP8H9lp2R3hV8tAd4fn77SbE50l0\nkQiLq4U48g2bJYyimJIfAcF+Tk94S/Gn0X4sS+lVRcP1yWSStttt4/P0fMEmPil1rT/7bA5Zs/fA\n19kSnxbQmvb289IGxHhgjoW47/8B3iS8gTk7B4vFIq3X69bfowDbDUiWcBmJ8DcHL2wOR+NH0pL1\ni/tS+uP79MTYE9lIfEv7o9fnLGEsTfn6+toIqB3z8fhnVgqME16v1+4AlFnAJox4Xux1tmRrmOGb\nVvS71VjC1/w/1IaozWazzlMJPglE0TGijUT4m8PCiAM/GGx/uVya5AcUNa9usDdIVRLOjwhv1F8s\nFo3oWiYchp+l1J5SCIWGB6Ds8ZsTFFB45/N5p1iRCVokvp5Aefu4fz6fO8V9omiNvv8PLKrRd7Vk\nFx4gxBtCX7fId0QiLDoXHsenGlxpDAv14GM3W8B9hTcnyCU/MC7n83kzKGfhaFZhzAQRox6w+BBO\n02QCjFli6HZgAWYx9PzAkXWIlii7idhKNR9tn3Khpf8D73dD6xZno8ZEl1zMsgQ4j0RYuCLsXbxY\n+Bwvfn48NwH/qAXs7YsG5ry+xfOyWGBhHXvkRlFjAfYsYRalaEJTtILx3OZEOIrV5gGzaEZrTh6p\n/R/gdU+A8UZszUTZC5eTFVxGIvzNYevHC6ey/Vjq0S56T4AtIqCP8NaKMYst30BwPaqaZum0aAnb\n4BsLsSfC5o9Fi9Cb1NTE0M4r9iN3BFu7+D7YPx6P2clU0Xqu/R/AdYz24HKfUS0OPL+yhOuRCIuO\nkHn7LCvKLj684O11Zn1a8ZucT7ivGPN71TT2aXIEQUp/LGELS2MB9twRLMLsm2WXgFnDngB7sAB7\nZULxeK+dTNV+N2+b545gS9jSvz2/MNexEDES4W+OJ1zRPnRHYPwnC/B8Pu+I8K38wnhRe8Ibbfea\nWcLn87mTlluyhE2IvElNPZ9w5IqIws+8VHFuPIOJba8dmMuJI9/EWITRJYElQT2fcOmzvjsSYdES\nTHtkHo1GHX+mFcRBK8sT4OVymbWCP2IVe2Ib9SPR8wa8MBmhxh2B1mDkl2V3ROSSQDxXhIWI4RIH\nEaPPzQlxSYA9nzD+xuxn59Ke/JuJGImwSCl1s7QwcN/E2RIQsFaCN0nmarW6akCu1hVhx4t9b4nh\nZ15DoYsEOHJHLBaLZj9boyyE2DyLOOcTRssXxR6Pt3YeP/6tc/8LeM7REsZi+Fj2s09xedFGIvzN\nwQuErceU/oiCCTG6IPDixDoNu93uw9ZutI2FFvu8jdONrdn3wkEuLzIiZwkvl8vipKacrJCLirBj\nsqVnCXvWOn62ZwXzZ9UKIgqwF6KGljDP2edVdBMxEmHhCnFK3bnmMA6YXRAsdCiat7CA0b/Ixxxt\nM7+pNY6KsCXXy43cEWgJr1arVppzzhrNibGXqIHuCM5Y4xsFuyOiKnbROYv+H/DceyFq5nZaLBYt\nsba+N4gqfCTCIqXkW5bR6+xCm8/nYT2JnE/4I9tqjt/Y7/dps9m0wtLM8rX3wkG5nDuCLeHNZtPK\nGKzxzUYC7N3sOBW8ZAVHrpCa39Q7l6WBObSEczdSWcJlJMLfmD4XBw7C2cVp2+29bD+KcCTG1/b7\nHLM324RnofWJrjCfrvce0fvaeTLrlgcObft4PO4Iv9e35dvbW1qv142/GjMYOYa75rfH7x8Jqudu\niH4vWcB1SIRFbzxRuly6U/GweOaElZeeMN7qmHMCi8ISxRjbd80JEosRxgtblTmsWWGWOrs/cu3t\n7S29v7+n9XrdbPP8w3weuM/bIgH2BLbm95IQ55EIi2oi8Y1ewxdpTf+WwovHFH2HkrXHj+Q5EY7E\n2DBB5PhhPJ/sf/Z80p4Ibzabjo+4ZAnzebZljfB6+6LfUeSRCIteeELM+20ZPZ7XPsrj533kePm4\nvc+uffy2iSyjGr5sTbPrAd03fIwmwhgOh31evr+/Z0WYLWHvXHjrNcJbEuBb/HbfBYmw6A1fnNG+\nW7RbHS8vS4NJOUuXLWF8vfdeKL5onfLg3Pl8boSXhdjbvl6vm8buCLSEzZ/Pv0/p9yq5JWr866KM\nRFhUUyu+ti+6MHMX7b0u6JzIsOCwsHqWsBcXGwkUCrBXEc3WT6dTI66e+PI+towxXM6b1SM6J7VW\nb9Tu+bt9ByTCohcsmHYBo4/T+t7ro8fV3Ots20eOlbdFosOxrizAkU/Ys4bxM3JV0bCZyJaWGBGB\nIXVsCXsDc/zd+wpwzU009xuINhJh0Ru7qPBR2yw+C9/C13nL3D5v+dHjveZxG8XVBDjyCUeWsN2U\noqI8nJBh7gV2N3jbcBYQjCPmWU74d/POCft3S/5f3n6v3+47IBEW1fBFnCuO41mxkWXb57UfOfZa\nCzDnDy5FR0QCxSLM0xKZiJrA2oBbrvHfe0WEPHdE7qbE56LmpnXv3+6rIxEWvfAe71PqlmSMXv/R\nbdfguTlYWCKR8dwRKaVQfKMkhpRSR4QxA86y4DwRtj5v4/rCvMy5I7DPVm6fp4XI3SDxrUciLKow\ny9f6uPybx3Dt3+ASRbdGfEshapH4siVs1jBPIY8+XRNYDD+L+ibq6G/Gwb7cwFxf10xpYE5cj0RY\nVNNHBKO6CLn93nq0rebYTOwwUgDdJSY+nkXLFh6KqFf4Z7fbpc1m05n40uaCq2kmwugD5gE4rBUR\nRVlwPPJoNApdJ972l5eX9Pz8nJ6enjqzZ9jNR37f2yERFnejVLQmKmQTxdEykWvE+ugr9SbARBHO\nTcnDRXVsUlATTk+g7G82m407izNvQyFH8cWaEN738LLi8AZjNYztO3LxdW/958+f6fX1tSPG9j1N\nwCXAt0EiLO4CCiwLBm/j19dUG8sN5qEP1pv3Df9uMpmk8/kc+nHZ0sR6D/v9Pm23W9dCNNHGmalL\nzSudGc2ewecz8sVbBMtkMmlVQeMl9n/+/Jl+/PjRiPBqtWosYpxJWdwGibC4Gyxg3uOzJ9As1J7Q\nROFu2MeZKbwZJyJ3BFt4eMzohtjtdo0A299hFMThcGiJsAk3h5Phkt0T7IYoFW3Hc2N9TC7xpqvH\n9cVikX78+NERYXZJSIRvh0RY3AUW09IgEotxtDRwUInX2RpFCxaFC61EtPCiATWs77vf79NsNkvb\n7bYTAYGzdSwWi5bQ5lrOQo5qQvBTgomunQfr86wYy+WyaWbpWvvx40fjjnh6emr2yx1xHyTC4m54\nAhzN9VaylrHyWEp1NSr4JoBiziFZpUG5yBKOQtBMPHHWEZ6wM9qWE22vMA/juWrYEl6tVk0zobWl\nCfDr62vWJyxug0RY3IXIEuZ4Vk+Io0kyT6dTSillU2fZGsZj4b6JcEqptyV8OBw6r2OBNn8xZsTl\nprD39ntTF9n5iZIivAFL8wmjJfz09JSenp7S8/Nz056enho3hDX0CbP7RXwcibC4G9GgljcdElvL\n0TKlrgjnkg2MaCDPRDjyCfPNBC1dFmCzVne7XVosFmmz2aTpdNpJT871vRsUb+PqaOyC4O9oIoyT\nspob4uXlpdNMnNE6liV8PyTC4m5EVqRn4XmuCm9bSn9E2BPf3L6caOfihL2bCLsg0KdrUQgmWDlx\n5X25JwJez91kuKFP2ATYrGBzPZgvmH3EGCvMUSDi40iExV1gV0QkxBg+FokUbk+pLcJR9hv2Oa3Y\nrEYMSbNHbBZg/h527Cl105Cn02nabretmNvxeNy5oeT6XlSIF9qH38GIhBjjhNkdYSJsERE/f/5s\niS5HUcgdcXskwuJulCxhr+CM91iOy5SSK7TRtul02sQBT6fTziN7FKLmDczhcdl7Ho9Ht/ylLS1U\nruTrtj6fv1zfojvsuxgswHYs6BP2LOGfP3+mf/75pxWO5jW5I26LRFjchWhQDgXYK72YG7gqiTBb\nwlyCMre0bDUvNtdzFfD3NIvXuylEkR5eFIhnXUbb2NLl2he4zrHAHKKGzZIyOJsuF0strkciLO6C\nN5jlFa7BUCz2F3vbUoqL8HhuCa/uL6cpTyaTtN/v0//+97/077//pt+/f7dmMcbZKtgi9bL6MBMQ\nRTh6bU3CBfftO2JUR678JoqsuRvM4sWMOU5h9jIJvYFPcT0SYXE32BfMFrBncZYSGlJqi3CuXypU\ng9sPh0P6999/GxF+e3trLOP9ft+pz2s+WU9Yz+dzU7fBRLiUARgRRTyklDo3FBRO7mOUA4qw53pg\nyzcXQy0+jkRY3IVaS9hEDi1iTufFvpdokYt8YB9t5L89HA7p9+/fTUNLGFOG2RJGIU4pdQTY9peE\nOJeSzevWRxH23AfWn81mLSuYIx7QEvZcD+zekBDfFomwuAteZAT7g7E+AtdUiBqKcKlF/lFv2/F4\nTO/v7+nt7a1ZlkTYvicuU0qNANvAWVQLw/4uckd4AhxFPUSDaNYid0RkCXs3LLkh7oNEWNwNS2LI\nCTAPhHklHrGPFqbnn4yE2Gu4/3Q6deZxw4E6dEewaJq1y2B9h6jVEFn+nhWMli1WRrPUY2z8WosD\nns1m4Q2LXSLi40iExV3o447gWYOxcWUxr34E9nm9ZgBvNBql0+lULCXpWcL4XaMbQzQYF4lxZPVG\nVj4LMMf2zufzTvabRUigSwIt4VIYoLgdEmFxFzg8Da1hrp+72Ww65RtRmHEbi3BpWfIbWzufz1k3\niDdnW86SNVHG9ZwbouSS8G4q6BM2AeUwNOt7lrAnwBYHnLthySVxWyTC4m6wJRy5I3KNX4ciiELg\n9XP+VM9aLYXHRdER/J1xQA6355Z9jh1F0XNFcBwwxwJ7PmFvdpDSORO3QSIs7kIfS9isYW/J27wS\njpEoRG4Lb5lSqk4rxu+YE2LvnJT6fPyRBR9ZwuiO8BIxaixhjoDInUfxcSTC4i5EPmHzr6KrAedV\n48b7rX5EH1gwvPVc4kXOfxsJaGnQLbefrc1ooJFD1NgS9moGR3HCbAlHxyVuj0RY3A0Oz2LL2EtT\njmabMNG+RoRLoEviI5Qy3xCv9gJauDVLE1cuOxktMS2ZQ9M4MUP8PSTC4lOAj8EfFUvvff/Ge3vR\nG7zPq2thWee1AAAgAElEQVQR1bxgKzfX2BJmy1flKYdDIiweipwQ3FqIvc+6x3vnojJyA225DDgU\nYRNXb8nb0BfMVdEkwsMgERafko+KZUnsb/neLLS5Pk9HH/WtOHs0WWfU0BeMlrB9vvj7SITFp+Va\nsayx+G753tGgmpdKjQNlpT4Ka6mPf8fT16tI+7BIhMWnpq9Y9hGaW7w3ulByxYSsmbh6CRdR31vn\n7RhHjBY1WsKKAR4GibB4OPoKQa1YXiMwt3hvtoSj2sbT6bQRT3QvRH1OT+a+Ny0RLj0RFn8fibD4\nEpTE8iMC89H35kE4ju/1ZkK2OeByERCRu8LbFtUZxpuBRHgYJMLiIbml1XoLcfnIe7M/GKMgsHmJ\nFs/Pz00ssM0JZ312L3jr1o9qKXsV0sTfRSIsvhQslrcUlmvem33CniWMKcMowia2Ly8v6fn5udOi\nCApvPReVgdvE30ciLB6WawXUxPIelt017+25IqK6DzYwh7Mh44zILy8vTYuKuXvb8TisH20TfxeJ\nsLgbUUKC90jszf/m7UdKpSSvpfS31wwcRvV/ufoZJ1qwGwIt4VxCB/clsI+LRFjcBc/68yy+w+EQ\nTv8e1ZtIqSuUNRXKvPKRXh8FKyri3tcS9gTYCyczgeWqZ169h9yEnHItfB4kwuIueNYfC5AV6eFJ\nMFF4WYBZhHPCGhVRx8JC0YAbCnL03rVCjCnJ+P29jDYceItSjS3aIZqUk9OixWMjERZ3gy1Ar+4t\nzljhia9XdS2lWGhxW22zv8Fqaii+OaGuCU/zav+iEGP4GYswVz3LVT7TjMifE4mwuAsld4SVrjSh\nLbkgrE0mk2LdX9yGMxzn+gYKcE2iRh8h9gQYfb/mhkABjtwROTeEpiH6XEiExV0ouSNwtopIfL1t\nx+OxaNGy0Np7jMfj1jxxto+FyhPinCB7vmTvXNh5YBHGMLSo/KTnE2brFzPfFPf7eZAIi7uRiwrI\niTAKLxd9H4/HRfcCW7nY0Drk9ZxLoobIKh6NRq3vj1XQOBTNK0PpTUWEMyJ7TZbw50EiLO5CJMDs\ngijNusHrLMIouJEFfDqdsoLEA28oxPZdasXYE2I+DxgZwUkZufKTnIDBiRdoAUt8Pw8SYXEXTAi8\n7DAU4JRS6ANGETZLeDKZdKIpWHhxmwmwLRGMkODt+D36xhzze6I7gjPjbGDOLOFcpTT2CUdF4mUF\nfy4kwuJuRJEBPGtxNAhnIszuiCikzduGgsTz07H17NFXgPHv8LO9CTm5RsTLy0u2KhrXh4iy3ry+\neFwkwuIueO4ItoDtdZEVzCJ8OBw6Iuy5NdgK5uQLFN57ChW6MzxLGN0RWA8iVxkNLWG2tvn8i8+B\nRFjcBfRRcpgaipNZqDhQFw3YXS6XdDgcsgKM29iaZnGfTCatfSXXhheJ0edcROeEY6i9TDgvnVt8\nDSTC4m54ooOCZqAAo5/We4y35I6SG8JE2LOqo218A4j6eJwp5eOUU2q7Ww6HQ9rv92m326XtdpvW\n63Vj3Y7H48ZC5sFLO58cZhdFY+TWxWMhERZ3gwXYBGU6nTb7U0quAHPdXcywiyxWb1CuJLy4zbOc\nvSgNs5pLn2/Yun2WifBms2kVXR+Px+lwOLRuCnhDYtcOxjDjOee45ZqEEjEcEmFxF6LHb7QQbT9a\nkSzA7ENlAYzcBWYt1lrB5nNG/3PUr4lv5u+Fn4GWMM56MRqNWpY+W8DT6bTjOvGE2P5G4vs5kAiL\nu+EJMO5jEc4J8G63S8vlsuMOiBqKcI0VbMWErO33e7c/mUw67gvsp9QtKoTijCLMSRcptZ8KUurG\nGM9ms+Y1eB75vEuAPw8SYXEX2BK+XC6twST0+dp6JMD7/b5pJQFOKXWsz1or2D5jt9u1+rPZrFk3\nlwEnkFhhIfx8XOfP2W63nUk22Y3BLghLdsHwO/yMXLSEeFwkwuJuoAh722yQiQX4cDi0Cv2YFcqP\n6tHgmK338QkfDoe02+0aN4H1Z7NZ2u12LZeBie7xeHRD4PB72Tazyk2E8f284+dEl9ls1vEX1yZm\nSJAfG4mwuAueAOP28/ncWMgowMfjsZXejIka+BgeRSLgso9P2KzTzWaTFotF08dwMfsuJpCcIuzF\nH6MrAkWYp5lHKziqt2E+cTsPXuF2Ce7nQyIs7gYKMfY5msAE2IsX5vVIcA1PhGt8wvv9Pq3X67RY\nLNJms+nE65rg2WP/fr8PBZhntsBoDYuOYBeEfU/PBTGfzxu/NIfo8Y2O/e7i8ZEIi7tgAoAFd0x0\nuAgPZ9N5iRdeth32ed0T4ZwY73a7VnowCzAOINr38wTYEkB4P1rCOCDJURyRBWxTQbE7wm5qeJOw\n49Pg3OdAIizuBtYwiKxVW3q+3cjfy+/jwdanJ7y4z0TYEicwQ40F0xN7+yysamag0KLVitv3+31T\noAjrS+DAJLtlUIDxXEmAPxcS4YHICchHeJQLDwV4KKIEDG/pRSt4NSc4gYJjga3hwGNKbcG192Vf\n8el06kwCaoOE6I7AWGEUdX7qsON+lP8J4SMRfiA8YcbHS0YXVx68EfCjO4bMXS6XTrF19L3ae5ir\nYD6fp81mk7bbbaveAw+2pZRaNSK8wTgTb3stxiZjmJxFbWDDueW4b8etGhOPj0T4AcgNMPF6JLwS\nZB8O48JHd+R8PjdxyZiabO/BRdm5olkULcEJK1FEhL3WE2EUYhRjOx4MZcPvFn1X8VhIhAem5B9F\ncimqIiYSYOxjNhpnpHEZShRfzwJG1wW+B8f0onvDjgUz9Fh8TYCXy2UTwXE6nZrj4EHDKIxNPBYS\n4QEpDUzha0oCLN+fj2cJe/svl0uaz+duyjAXEVosFi0RjuKF2ZrGz7TXsc8YBdhzR1gInc3Tx0Xy\nvVBA8dhIhAcmFxkQhUR5SIBjPPHFfRj94LkguIYFFt3BrLeU2nUi2K/s3XRZ9CMBZmsYy11ybLBq\nDn8uJMIPQiTAXKiFH2klvjEosJ4lzNvYn8rxuiaQi8XCHYSLwuI4vI1LXmLERc4nbOJr/uhIgO3m\nIEv4cyARHphIeHlblKLKF6Hogokj5idFcY6EDIvmYEnL5XLpCjAXkccMNy5cb3+DlnNKXZ+wNyhn\nIuy5IMx9wr5m8bhIhAeEB4k8K4ldEl6KqvzCMXw+OFsPBZoFGON+sbD7brdzB9jwtSimlkFnYEQE\nWs4ppY4lHLkj5vN58/2i45Yl/DmQCD8Ank8YH11TSp0MKXu9sqPy8ICYN+hpTxp2fktF23e7XfN3\nUYEe8x1bJTh7PbpGWIQvl0sxRM1ik1GEeb46FGCJ8OMjER6YyBeMF3hK7UdpFm0JcZ4oMsEoDYxy\n2+12rd8oqkm83W6bQj/4e3pRFOa2iAR4Pp+3kkNsyntMBMGZN/B/Rzw2EuEbU2t5RILrNXs9Pkqz\nj9j24YVXEuWvLtqe+N6C3W7XGYBjAV4ul2m5XKb9fp9SarseuH4Euz2irDlMEpnP56EAyxL+XEiE\nb0guFImXfPFx48EczLgqNUwO4CX3xXWgFcoFd1arVVPvwYryeJlyp9PJ3WdL7+bs+ai9G7cE+PMg\nEb4xNY+zXL6QC5fztpwI8zauIeCtXy4XxZB+ALyZWcIGTka6XC7TarVqBuV4Bg2vfCVHWnj/L3yj\nxsiLSIAlxI+PRPjGePGf3tImfcSGj6LYPBH2+rZEYbA+prZalpe4Hjz3mM7MtX8jH7D9/l68sb3W\ns3A5BjkSYFnCnweJ8A2p8e+iFYQ+P29ABkOcPLH1liYKVt/AlrPZrBXSJkv4eswS9twRKMAW8WDw\nIB7OtMxCHVnCkRsrEmLx+EiE70B04eASJ5Y0/yHHg9o28yvmShdylS97NMaKYCgeukA/BteVQEsY\nIxRQSNHFhJN98lMJuhIiAWZLGD+PE0LEYyMRvjGRBYMXjT2K4sy+XCsWt0Ui7Pl9p9NpdiocpbPe\nBrSEsbYE/s74v4BuhP1+n61BnJLvjigN5MoS/pxIhG9INJDizeSAsaQ2s681XkcRjsTXBNimjEfL\niFNasVSj6I+XpWaWsJepxqFnu93OnULJqBmY46mOJMKfF4nwjckJMT+OmghvNpu0Xq/Ter1u9W3d\npkjHyl3cbJ/VmEUhQLHgyl7iOrwyl1GSBP/mXAzeK4OZC1Hjpyr8XBZg/c6Pj0T4xpREGAPxWYTf\n39+bJfYjEeZZFWzqHS5KzsVosM6t6A8PzHk3PozHNgHGKmhRQXiDxdgLT0PxzYWpicdGInwH+MJh\nAc6J8Pv7e3p7e0tvb29N3xNh6/PSqmtxPVwrxShL+DZ47gj73VGkR6NRx/VUmpcuF2VT8gkrRO3z\nIRG+ITVZTjkRXq/X6e3tLf3+/btZ/v79O+12u5blhEvetlgsWiLA4VM4aCSuxysbmVJXnMfjcRPt\nstlsGkvYRNgLUbMl/y9hJAS7JDgyQu6Iz4NEuED0T+xtxzAkrn5lfVtuNpv09vbW+H1xanN81EzJ\nt7rQmsL1xWKRnp+f0/Pzc1qtVmm1WrUufG82CNFPrDjhwgbbvN97s9k0riX7nTkGvM+TiUT16yER\n7gFeAF4fL0x+BPVC0eziXK/XzX4MLfMG1riIC7flcpleXl7Sy8tLI8TL5bI1L5rngxQ+nuhh3C/G\ne3u/9Xq9bp5sWIgj95CE9nshEa6EH++8JV6YGAfshZ1xOFruwsQ6BZwea0kZtr5cLhtL+OnpqRFh\nb0RexOREkTPfbNDN+43NxYTWsFnKZgnLPfS9kQhXwH66qI++X/b5eqFnnKwRWcI8Em8WrxWKsSU3\nE2GeIViWcB5PgHkb3nDR9YC/sQ22ogiz28luuLm4bQn010YiXIknul5MJ1+YZiFh9MP7+3sT/xvV\nikCfMPuDTVRNaE1s0f3ADX3CEuGYSHRxyYOtfMPF35l/c/t/uNYnLL4eEuGeRALMqamehYThZ5vN\nJqyehu4InAMN3RFm/T49PTXuB3NBmEhjDQm0hL16BcInevJhdwT6gO23Rl8wPwGh68lzR0iUvw8S\n4Uoiy5fXvYE5FuHfv3836cgcbsS1B1Lyp19nEX55eUmvr6/p6empU0ENK6nJHVEmEl5bYiU8vtma\nJYyhhjwGwD5hJVd8byTCFXhuCK6QlbOOWIR//fqVttttOCsCBt6n1B2YY3fE8/Nzen19TT9+/EjP\nz8+dGsJcX1gDcz656JfI/x9ZwijCHBnj+YQlwN8XiXAlkT8Yg+Rx1LzGEub389ZT6qYe48Dc09NT\nE5L2+vqaXl5eshXXODlAxES/jWcJY3QE/s6/f/9uxYzjMueOEN8HiXCBaIAmJ8CYFccibBfmZrNp\nhBALuHhLL044soRfXl6K88xJhGO8myBvj556vPTz379/t/4feNYUDcwJiXAF3qMpW0WRT9gbmDMR\nLk3UyZNzojvC8wn/+PEjvb6+ut9BotufmkHYyPVkv/OvX7/CuQM9d4TE+PshEa4ABTY3YwamrHLY\nGVtCNsEjTjM0Go3S5XLpTNzJFbdKTfxHLuHCu7FGBXN423a7Tb9+/Wp8vpj56KWgewOwUbEdfjrC\nG7I3pRX28W/4iUo8LhLhAnyBRiUFTYSj2F+cd8zadDptXWgpdQuGRwXc2XLWxRbj+XWtb0sujBMt\nT6dT2mw26X//+1/6999/W2KcE2Cv0lkUFZETYLwpRzdmFmLx2EiEK/D8vt4jJs8P582mjBclXmgc\nCeFZwRLg/uRcCuxWys1+jf3tdtsI8K9fv9y0ZPT55qYhQtiSZXGNCvpH/xP6v/gcSIQr4LAkFF68\nQD13RE6Ix+NxxxriC7D2YtNF51MT2325XDqzX+faZrNpBljNEi6lJbP7ynNFGJ4Ye1NZ8ROSN/Cq\n/4nHRyJcIHJH2EWLAhv5hPGCZBFGK9hgdwRbw7J6+hENonrhZlgRLepbGVIsvO+JMEY/1M5+kbOG\nI3eE/ic+NxLhCtBnyGFoNuDmxYGySLMbw2Y+9hIzcpZwZPWINrnkGm5R0SWugMepybi017Al7H1m\nzhJOKbX+B+SO+NpIhAt4ljDXh8BwNBbgnBCfTqc0nU47iRkcG1zjE9YFl4d/Q46AwN8ShdZrFoYW\nlSblAj1seXuDcpfLpWMFs1tK7oiviUS4AhTiaMJObqXICC9UyfDcEVHWm8KR8kRx3fhkw+4IrAGB\niRc4AWuuWD+XJPUs3xqfMMeMyxL+mkiEC+DFy6FpkRCXBuaikKWU6twRuuD6kRPgqCQlJ9dgGvJ6\nvXbdT1FaMofH5dwQNRESClH7WkiEK+jjjmD/cGQNsxAbeAFFA3MKU6snsoKjm2okwhaO9uvXr/T+\n/t65uUauJ6wLUVrib1iKksnVBJE74nMhEa4gNzCXc0dE4Wk8o0LOHSF/8PV4FqiX8ZhLPzYB/vff\nf5v2/v7uupe8lOTT6RQeV0RNnDD/X+TcVOKxkQhXkntE5DTSKI43ShKIGl/gXgF4DoWbTqfN8fLx\nR+t9+zV4QlPaxpZhaX9uyT78qNm53Ww2rQSMKCXZBt+irEn29fP5Yx8+9ieTiTsrik1X5a3jTNqe\ni0JC/PhIhCuIguaxTu/pdOpcCNFjol0YuRjkw+GQptNpR2h3u11aLBZpt9s1FdW2221T4tJ7FI0u\netzmreOI/bVEQsnbIn9pbjDLG/DCbew68pJsrGEChjWcIRnDznJZcLknG2+wDbdNp9NmdpSaZqKM\nM6ng/5+s4c+BRLiCyD+HAmwiXBJivChQPKJsvMlk0rF2UYCtsQhHwlpq0ePsNRezJ6ZR36upEG3j\nvieCHMnipSDjcrvdthIwMAvOm5KIoyv4hmDfzc6dF2rGT1Cz2axagG0aK7aGbR5BDFkTj41EuID9\nE3sj1FaEBy3hkggbuRF7FI/x+L8SlpEI22fOZrO03W47Ilq7jtl7uA3PwbWUwrNqRZVjfFEEvSWH\nE3rN9ueSMDD21xtUjZIw8H/I8/Fzv68I4xyC9v9g/3tyR3weJMIVeBELduHgxRhZI5F1GQkxCvB4\nPHYFeLfbtQTYWvSoGz0Go/iaAKeUmm3XXsR93Af8/T2x9W5UuYp2UWp5VFwd44O95lnCuZtGJMKT\nyaR1o8b+fD6vFuDVatV5EkIjQJbw50EiXEHOHYEXH15QnjWcE2EUjslkko7HY+vzIgHmCzAaOOQB\nQ29AcTKZdAbPzDd8LTUuBvavsrhyUkUpIsH2oeCWllwnghMx2CfsWemRO8KL8eXfzYr1W5H+GiH2\nbsLe/5x4bCTCBaKICBbgy+VSdEegENsFGrkj8LVoDXsWMFpVPICYi+Aw0bU+fmdbflSAbZnz67Il\nixYt9z0/b7TkQU0vmsTb5hVh8hIwvBtKjTsCb6DoSjARrm3RhK5oCYvHRyJcgWcJn8/n5qKyi84T\n4VxCRS46gl0gnijw51mReLTWo8EgLB5kYW38nT1B6UvJ/cD+2xort9bF4FWzi6rcmZWbqymMERW5\niI7ofwdFGP251mzi1tpWSt5QdMTnQCJcAV9IKF544XlWcJ/oiPF43BHglFIzco7iy/5EHJBhseVm\nxx+5H0ajUTqfz41f+Foiv3BOgHOhZLXWLboYoup2uZmPS9Y4xgCXoj5qRBjjf/uIcC5GXQNznweJ\ncAEUQhZiFij089WGqHmixLHEo9GoEeHZbJZ2u11RhHPZVehKyYmwCfU1eK6IKNLBE+LIEo1E1Fty\nP7d+OByKkRnY+Hvm+izCKMSYfNFXhL1ol1yYoXhMJMIVeO4IT7w894AJX84KZgHm19h7m/jaRYzr\nGKBvfdtuJTNRgNGK5ws459/sAwtxJMDWci4GHkDjhoXX++yzxoV28Lh5W1+86Ai2hFmAawbn7L3x\nc7y+eGwkwhWwCONgFl6UuYESHkyzKAgexeaIiZT+s8JNiHIhSObWQOu3tETXBrs5eFsfStEOuL1G\nfDGUzKvTwULr1fTIWcVYaKf2fwL/N3jd+pPJpCO2keVrwospyjhwx+nJ4mugX7ISFGIT4cvl0npc\n9wQXLyAchDGfa+TDMyFOKbUeyXe7Xeu1LNr7/T4s8OK5J0rCi8s+eMkUOREuJVV4/uCoap1XThT9\nyxxihr9xrRBHMdccSTOdTqvdCyzCmAmnsLOvi0S4APvb8NGSX8fChenELMTn89l9bxZhc0eYAHGY\nm4mJ+VMXi0U4KOdtjwTXE+U+sPDmhDiXUszbeDCu1Mell+12TVy0FzPO59bW+2bB5YrzSIS/JhLh\nSiJL2PaZ3zWyhDkUif2/DPogMTQLhRr9qyZY8/k8G5rGyxor+BoR9oQ3EuKaAjt9rOVoYM/eK5di\nbL9njRDjkwX65bHfJwsuckXY+VfEw9dEIlyBJ8C8zyzhkkvCBPl4PKaUynUVzNo9HA6ti48f580C\ntEdXz9XhJWvUWsG38gl766Vyk1Hj8LUaMS9Zwvi75vbZ/0PufNlv30eEvcposoS/NhLhCtgV4W1H\nEY5cEmgNY7ypFxaV0h9L19wRhg3AYX0ETGf2fJNe2nIfEb6lT9iziDlbzkvaKO0vvZ5jfkuRHzkh\nRvcDpiHzIFrfLDj8H+HBOC/KRnx+JMIF7EJkEbZYWi8Yv2ZwDgeITBRSarshMELCtpkAm9B44Woo\ntKXlIwzMeWLMfW+9b+NjylnC/PvzNrSE7RyheKKY4sBbLhZ4tVp1oiFkCX99JMIVoBCn1K4wZkJ8\nuVzcULQoOgKtNx6Mw0E3tNhsm4nv4XDohJ1xYkgpoP9eA3Oe6NWIcM1rMM446uM273Wc9Vbz+yNe\n3K/9xuZOMNeCF/0QibBX2Eci/LWRCFeCsZ8oiiig7BNmVwRbwl6kAwsyCwf7d9HP69UMiCIwrNWI\n8LU+4Rp/cK4fbWM3Drt0vH25Ze3vz8ktkSVs8cCchIEJGJ4Qr1ar1jln95aK8nxNJMIVsADjBYnL\nWlfEcrlsWcAp/Xl890T4dPqvroTtR6FlFwMLsC2jbff2CfcR3pxlyyJ7i4a/X+3/gb0+soQxIYMF\nlrPgvNfkUs1lCX9NJMKVsIil1L2A+yRreANtXrKG+UKjY4j6teuPIMIssrmG5x3Pf00/t60Wdumw\nJYwuiD7zxVlkRFSQh2uQiK+DRLhA7h+e95X8hLvdLq1Wq3Q4HJpsuyhl2dKaeT++zuB+7UVqInw6\n/RfjbL5m9DnfMjoicjP0dSv0hW9UpZtYtLT+bDbLFt15fn5uNXwdz5zMA3G5DDx+0hFfA4nwDclZ\nRzgrgwklBvWzfxgjIVh8S2LcR4jt9ewO4Au+r/iV3AuRAPdxG9R8xxqXjPWjKBIvogTFFq1eb1vN\nNPUsuuy7r/2+4vMhEb4xGKqGVjCGpKX0xwr1HjM5FC0nUJGFWCvE/D7sm8bX9KEUmRD5efl72ucj\npe/lWb41zRvk9JYowlHVM68Yj5eEgXUhIgGWEH9tJMI3xC4ib7CGBTjy87EvmNNsveV4PL6pEI9G\nIzc+uQ+ee8ET3miwre/nRv7vUrgeWp84GIZhf7zsUw8CY395Jo1aS9i+jwT4ayIRviEoruiOMAHG\nECfOYEupGxFhabdoRY5Go84AlW3H0XsjJ8Se+Ob213x/I7pxlMS3JqIh+kzPRx+5GXgbFjPKDVZO\np9O0WCxavmD09/K25XLZSb7wkjDYJeVZwdH3FJ8bifANYRGez+ehBWyPn4ZnAVtNCB7EMncB+m5z\n9BFib9/53H+GDXYteH7fnAj3JRpEK6Vwo/uBY7txiX0UYR6g4/XlcukmX2DznohybhPxtZAI3xi8\noNECRr+jWVQlC9gueospjny1txJiFmFzdZi7ow8l/3VuAC4aiGS8gTZeogDnQr/Qh+8teRsXZ4/a\nYrHoWNdew7oQnhtC4vt1kQjfEM8SNlCc7WLkATEsWYlV0ayCGvpqWcBqfb+Ry4F9y/Za2x69f42F\nXRJcHoDzBuSuEWIW4dyAGxdZwkE0TEPmiTk58oG3LRaLsKC+V+85+h65cy0+NxLhG4IDPCgaaAHb\nxT6dTlvuhdPpT01gb1p7toAnk0noy80RCXFKXd+y9dGd0oeS4JbENueW8HzBngXpWcGe+KEIo5jW\n9r11E+0ovdxb5+/jfUfxtZAI3xi7qHAQzsQXi/bMZrOWCPMURjxzM2IChq6CPo+sKMQohDnrq+a9\n+UbhLUv7+DhriR7hUYxzUz15Iuz5erHmb66h5exFYkTbovMp8f26SIRvCPp9U/ojyF72GIuyWcC7\n3S4tl8u03W5bI+gGuw+u9RdGQozfpeb71nxOzbZriCIHIlcECjGHnnkiHMUAW91fdk94rov5fF7l\nu8al+F5IhG+IXfS4bo/2JsZYA2G1WjWz/WKfJ6o8n8+NlTyZTNLhcOiksuJnXkONX5Y/g61n7xhu\nKSz2XrWxvzz9UNQ3/30u6437HPMbNRwXEMJDInwnolFuE2mOJbYRd0xvNrG2Og7RJJbYt2I/ueNC\nOFohF0LmPeZHLfq80vbS6yLrNgo7y4kv9m0aIo759Wo+cMYbvp9XTlSIHBLhO8ADW7bN/LcmEjhQ\nt1wum/nQ0Fo2n3I01TtuM7HmY8mRSyu2Qbko0STq89NAqd/neKOBtSjioCS+GMlSE/FgIswxxHgs\nuWgSIRiJ8J3gx3QUYHNPTCaTxg+JU7Kb6Jm42PxxXkMB9kS4RG5mC/YZs9Dmai3cyw9asm5rLV/u\nR7NieH5eE2HPEvYyIYXIIRG+MZ5wGSbAJrBsCWPFNPRpzufztN1u0263ay1tOwpA35KT3mSZJiBm\nleN3y0UaYPPcMfge3K89t+jGqamBXCPEdo69GVCixI0opRm/uxA1SITvROkijOpL2N+iSC8Wi7Td\nbtN2u02LxSJtNpvQCtvv99XHcrlcmsgM9mWiX5hdKjX+1lLIGG7vc+5QaEvpxSUh9gbnsPASv3/k\nB/YsYQmxqEUifCcicfEs4ai+BIZNbTabzoCQNwFkH0v4crk0Arzf71vHzGFwdmxsCbMlaP0+g3h9\nxJKxU8gAAAT+SURBVIrTi3nmEi6QUxJf7HvWdGRh57Lg5IoQfZAI3xgvfMu22zbzCdvFjQLMFrLF\nDJcsYAzLqsWOI7KAJ5NJZy48zxr2rNNcCBlv7wNGk5Qa3qRq3RI1+3lA0PORyxIWtUiE7wRHSKAA\n23I6nTYC7EVMYAYdz8YQhUNhZELu2FLqinBK7WgJFpTICvYe4b2ssKhfe7wp/SfCPGgWZat5boOc\nuPIAoye2UTgcD1pKgEUtEuE7kLsA+ZE/pfZsHDhAhgV9Ij8ni2iNCOPney4ITKWudUewW4DTcXPL\nPszn8zCMjNf7iDCex1wIXs6y9240QpSQCN8RfoxPqSu87AP2Jsfc7/cd8fUsrtpqat7rL5dLq5Qm\nRznYd8i5IjCqILISvX4fuJavV0zd1ufzebXbIQqri/reMtomRA6J8A3p82iNoWjmy42m/pnNZuH7\ncWGcvpawZ9VF74/+VoyZ9dZrHtc5saOGxWIRzmLB6yjCpWXf4xDiVkiEB4Z9xiZOFpmQUrcWsTdv\nnZETbOZyubRcCPYYb+Fw2+02bTabJjY5V/Sc42q9R/hoWx88dwRnsrHf3MtmuyYyQ4h7IBF+ALxH\nXRRifPyfz+ed2hL2d+PxuFfBGBNhi0PGTDxODjERzs2Xhus5Pypv64PdgLzMNp67zctiwybEIyAR\nHhCzgK2Py5T+CDEPgi0Wi1ZRdHNnTCaTtFgsqj//crk0ac9WxQ1rU/CSkyI4OQJbNIjlrfehFKIW\nRZDIEhaPikR4YFgEeH08HjfhbBhXjAN8aCl7GXM5aiqzWT+XyMDLnPB+RATZ4vaSNTwR9twhEmDx\nCEiEHwz0EWMznzAXXkd/scUW14Jpy9iibV6qLzce6MqJ7zVCyIkhkSXuRZFElrDEWAyJRHhgPEuY\nC6mjpcsuCIzT3e/3zUSgNZgI17TT6dQZ6PIGv2ybF96VW6+FbwJeER1rucFAia94FCTCA+L5hKP9\nk0l73joUZpsqyQoB9cHigqNylrgvyh7zfK9RXG0uxraG6CYQhZ3l3CLXfL4Qt0Yi/ABgiFq0zYri\noPh6oskhazkwFpkLu3v9UuwvRx7UJDP0FUF7AijdBNAa91wPsoTFozC61aSLH+QhDuJvkpva3dvH\ns15EiR19RNjeH98jtywNtkUWMPe99VrYpZCLQ46s3pxlLsSNKf5jSYQHonTePWHGltt2zXF478v9\nlOpnysiJ2kcFr8bfzJ9TuglIhMWdkAgLIcSAFEVYaUNCCDEgEmEhhBgQibAQQgyIRFgIIQZEIiyE\nEAMiERZCiAGRCAshxIBIhIUQYkAkwkIIMSASYSGEGBCJsBBCDIhEWAghBkQiLIQQAyIRFkKIAZEI\nCyHEgEiEhRBiQCTCQggxIBJhIYQYEImwEEIMiERYCCEGRCIshBADIhEWQogBkQgLIcSASISFEGJA\nJMJCCDEgEmEhhBgQibAQQgzIdOgD+P+Mhj4AIYQYAlnCQggxIBJhIYQYEImwEEIMiERYCCEGRCIs\nhBADIhEWQogBkQgLIcSASISFEGJAJMJCCDEgEmEhhBgQibAQQgyIRFgIIQZEIiyEEAMiERZCiAGR\nCAshxIBIhIUQYkAkwkIIMSASYSGEGBCJsBBCDIhEWAghBkQiLIQQAyIRFkKIAZEICyHEgEiEhRBi\nQCTCQggxIBJhIYQYEImwEEIMiERYCCEGRCIshBAD8v8AFLWioi3Qk3kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x198e18c55f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot a random image\n",
    "sample_number = 5001\n",
    "plt.imshow(train[sample_number,:-1].reshape(28,28), cmap=\"gray_r\")\n",
    "plt.axis('off')\n",
    "print(\"Image Label: \", train[sample_number,-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save the images\n",
    "\n",
    "Save the images in a local directory. While saving the data we flatten the images to a vector (28x28 image pixels becomes an array of length 784 data points).\n",
    "\n",
    "![mnist-input](https://www.cntk.ai/jup/cntk103a_MNIST_input.png)\n",
    "\n",
    "The labels are encoded as [1-hot][] encoding (label of 3 with 10 digits becomes `0001000000`, where the first index corresponds to digit `0` and the last one corresponds to digit `9`.\n",
    "\n",
    "![mnist-label](https://www.cntk.ai/jup/cntk103a_onehot.png)\n",
    "\n",
    "[1-hot]: https://en.wikipedia.org/wiki/One-hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the data files into a format compatible with CNTK text reader\n",
    "def savetxt(filename, ndarray):\n",
    "    dir = os.path.dirname(filename)\n",
    "\n",
    "    if not os.path.exists(dir):\n",
    "        os.makedirs(dir)\n",
    "\n",
    "    if not os.path.isfile(filename):\n",
    "        print(\"Saving\", filename )\n",
    "        with open(filename, 'w') as f:\n",
    "            labels = list(map(' '.join, np.eye(10, dtype=np.uint).astype(str)))\n",
    "            for row in ndarray:\n",
    "                row_str = row.astype(str)\n",
    "                label_str = labels[row[-1]]\n",
    "                feature_str = ' '.join(row_str[:-1])\n",
    "                f.write('|labels {} |features {}\\n'.format(label_str, feature_str))\n",
    "    else:\n",
    "        print(\"File already exists\", filename)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing train text file...\n",
      "Saving ..\\Examples\\Image\\DataSets\\MNIST\\Train-28x28_cntk_text.txt\n",
      "Writing test text file...\n",
      "Saving ..\\Examples\\Image\\DataSets\\MNIST\\Test-28x28_cntk_text.txt\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Save the train and test files (prefer our default path for the data)\n",
    "data_dir = os.path.join(\"..\", \"Examples\", \"Image\", \"DataSets\", \"MNIST\")\n",
    "if not os.path.exists(data_dir):\n",
    "    data_dir = os.path.join(\"data\", \"MNIST\")\n",
    "\n",
    "print ('Writing train text file...')\n",
    "savetxt(os.path.join(data_dir, \"Train-28x28_cntk_text.txt\"), train)\n",
    "\n",
    "print ('Writing test text file...')\n",
    "savetxt(os.path.join(data_dir, \"Test-28x28_cntk_text.txt\"), test)\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Suggested Explorations**\n",
    "\n",
    "One can do data manipulations to improve the performance of a machine learning system. I suggest you first use the data generated so far and run the classifier in subsequent parts of the CNTK 103 tutorial series. Once you have a baseline with classifying the data in its original form, now use the different data manipulation techniques to further improve the model.\n",
    "\n",
    "There are several ways data alterations can be performed. CNTK readers automate a lot of these actions for you. However, to get a feel for how these transforms can impact training and test accuracies, I strongly encourage individuals to try one or more of data perturbation.\n",
    "\n",
    "- Shuffle the training data (rows to create a different). Hint: Use `permute_indices = np.random.permutation(train.shape[0])`. Then run Part B of the tutorial with this newly permuted data.\n",
    "- Adding noise to the data can often improves [generalization error][]. You can augment the training set by adding  noise (generated with numpy, hint: use `numpy.random`) to the training images. \n",
    "- Distort the images with [affine transformation][] (translations or rotations)\n",
    "\n",
    "[generalization error]: https://en.wikipedia.org/wiki/Generalization_error\n",
    "[affine transformation]: https://en.wikipedia.org/wiki/Affine_transformation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}