convolutional_network_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Network \n",
    "\n",
    "## Demo\n",
    "\n",
    "We're going to build a Convolutional Neural Network using just numpy capable of detecting any alphanumeric character that a user draws. It will all be wrapped into a Flask web app so you can play with it in the browser!\n",
    "\n",
    "![alt text](http://i.imgur.com/8ysCoB5.png \"Logo Title Text 1\")\n",
    "\n",
    "## What inspired Convolutional Networks?\n",
    "\n",
    "CNNs are biologically-inspired models inspired by research by D. H. Hubel and T. N. Wiesel. They proposed an explanation for the way in which mammals visually perceive the world around them using a layered architecture of neurons in the brain, and this in turn inspired engineers to attempt to develop similar pattern recognition mechanisms in computer vision.\n",
    "\n",
    "![alt text](https://qph.ec.quoracdn.net/main-qimg-235acb60a481423eaf70c39b17bc914b.webp \"Logo Title Text 1\")\n",
    "\n",
    "\n",
    "In their hypothesis, within the visual cortex, complex functional responses generated by \"complex cells\" are constructed from more simplistic responses from \"simple cells'. \n",
    "\n",
    "For instances, simple cells would respond to oriented edges etc, while complex cells will also respond to oriented edges but with a degree of spatial invariance.\n",
    "\n",
    "Receptive fields exist for cells, where a cell responds to a summation of inputs from other local cells.\n",
    "\n",
    "The architecture of deep convolutional neural networks was inspired by the ideas mentioned above \n",
    "- local connections \n",
    "- layering  \n",
    "- spatial invariance (shifting the input signal results in an equally shifted output signal. , most of us are able to recognize specific faces under a variety of conditions because we learn abstraction These abstractions are thus invariant to size, contrast, rotation, orientation\n",
    " \n",
    "However, it remains to be seen if these computational mechanisms of convolutional neural networks are similar to the computation mechanisms occurring in the primate visual system\n",
    "\n",
    "- convolution operation\n",
    "- shared weights\n",
    "- pooling/subsampling \n",
    "\n",
    "## How does it work? \n",
    "\n",
    "![alt text](https://images.nature.com/w926/nature-assets/srep/2016/160610/srep27755/images_hires/srep27755-f1.jpg \"Logo Title Text 1\")\n",
    "![alt text](https://www.mathworks.com/content/mathworks/www/en/discovery/convolutional-neural-network/jcr:content/mainParsys/image_copy.adapt.full.high.jpg/1497876372993.jpg \"Logo Title Text 1\")\n",
    "\n",
    "### Step 1 - Prepare a dataset of images\n",
    "\n",
    "![alt text](http://xrds.acm.org/blog/wp-content/uploads/2016/06/Figure1.png \"Logo Title Text 1\")\n",
    "\n",
    "- Every image is a matrix of pixel values. \n",
    "- The range of values that can be encoded in each pixel depends upon its bit size. \n",
    "- Most commonly, we have 8 bit or 1 Byte-sized pixels. Thus the possible range of values a single pixel can represent is [0, 255]. \n",
    "- However, with coloured images, particularly RGB (Red, Green, Blue)-based images, the presence of separate colour channels (3 in the case of RGB images) introduces an additional ‘depth’ field to the data, making the input 3-dimensional. \n",
    "- Hence, for a given RGB image of size, say 255×255 (Width x Height) pixels, we’ll have 3 matrices associated with each image, one for each of the colour channels. \n",
    "- Thus the image in it’s entirety, constitutes a 3-dimensional structure called the Input Volume (255x255x3).\n",
    "\n",
    "Great training datasets are [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html) and [CoCo](http://mscoco.org/). We'll use CIFAR.\n",
    "\n",
    "### Step 2 - Convolution \n",
    "\n",
    "![alt text](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/more_images/Convolution_schematic.gif \"Logo Title Text 1\")\n",
    "\n",
    "![alt text](http://xrds.acm.org/blog/wp-content/uploads/2016/06/Figure_2.png \"Logo Title Text 1\")\n",
    "\n",
    "- A convolution is an orderly procedure where two sources of information are intertwined.\n",
    "\n",
    "- A kernel (also called a filter) is a smaller-sized matrix in comparison to the input dimensions of the image, that consists of real valued entries.\n",
    "\n",
    "- Kernels are then convolved with the input volume to obtain so-called ‘activation maps’ (also called feature maps).  \n",
    "- Activation maps indicate ‘activated’ regions, i.e. regions where features specific to the kernel have been detected in the input. \n",
    "\n",
    "- The real values of the kernel matrix change with each learning iteration over the training set, indicating that the network is learning to identify which regions are of significance for extracting features from the data.\n",
    "\n",
    "- We compute the dot product between the kernel and the input matrix. -The convolved value obtained by summing the resultant terms from the dot product forms a single entry in the activation matrix. \n",
    "\n",
    "- The patch selection is then slided (towards the right, or downwards when the boundary of the matrix is reached) by a certain amount called the ‘stride’ value, and the process is repeated till the entire input image has been processed. - The process is carried out for all colour channels.\n",
    "\n",
    "- instead of connecting each neuron to all possible pixels, we specify a 2 dimensional region called the ‘receptive field[14]’ (say of size 5×5 units) extending to the entire depth of the input (5x5x3 for a 3 colour channel input), within which the encompassed pixels are fully connected to the neural network’s input layer. It’s over these small regions that the network layer cross-sections (each consisting of several neurons (called ‘depth columns’)) operate and produce the activation map. (reduces computational complexity)\n",
    "\n",
    "![alt text](http://i.imgur.com/g4hRI6Z.png \"Logo Title Text 1\")\n",
    "![alt text](http://i.imgur.com/tpQvMps.jpg \"Logo Title Text 1\")\n",
    "![alt text](http://i.imgur.com/oyXkhHi.jpg \"Logo Title Text 1\")\n",
    "![alt text](http://xrds.acm.org/blog/wp-content/uploads/2016/06/Figure_5.png \"Logo Title Text 1\")\n",
    "\n",
    "Great resource on description of  convolution (discrete vs continous)  & the fourier transform\n",
    "\n",
    "http://timdettmers.com/2015/03/26/convolution-deep-learning/\n",
    "\n",
    "\n",
    "###  Step 3 - Pooling\n",
    "![alt text](http://xrds.acm.org/blog/wp-content/uploads/2016/06/Figure_6.png \"Logo Title Text 1\")\n",
    "\n",
    "- Pooling reducing the spatial dimensions (Width x Height) of the Input Volume for the next Convolutional Layer. It does not affect the depth dimension of the Volume.  \n",
    "- The transformation is either performed by taking the maximum value from the values observable in the window (called ‘max pooling’), or by taking the average of the values. Max pooling has been favoured over others due to its better performance characteristics.\n",
    "- also called downsampling\n",
    "\n",
    "###  Step 4 - Normalization (ReLU in our case)\n",
    "\n",
    "![alt text](http://xrds.acm.org/blog/wp-content/uploads/2016/06/CodeCogsEqn-3.png \"Logo Title Text 1\")\n",
    "\n",
    "Normalization (keep the math from breaking by turning all negative numbers to 0)  (RELU) a stack of images becomes a stack of images with no negative values. \n",
    "\n",
    "Repeat Steps 2-4 several times. More, smaller images (feature maps created at every layer)\n",
    "\n",
    "### Step 5 - Regularization \n",
    "\n",
    "- Dropout forces an artificial neural network to learn multiple independent representations of the same data by alternately randomly disabling neurons in the learning phase.\n",
    "- Dropout is a vital feature in almost every state-of-the-art neural network implementation.\n",
    "- To perform dropout on a layer, you randomly set some of the layer's values to 0 during forward propagation.\n",
    "\n",
    "See [this](http://iamtrask.github.io/2015/07/28/dropout/)\n",
    "\n",
    "![alt text](https://i.stack.imgur.com/CewjH.png \"Logo Title Text 1\")\n",
    "\n",
    "###  Step 6 - Probability Conversion\n",
    "\n",
    "At the very end of our network (the tail), we'll apply a softmax function to convert the outputs to probability values for each class. \n",
    "\n",
    "![alt text](https://1.bp.blogspot.com/-FHDU505euic/Vs1iJjXHG0I/AAAAAAABVKg/x4g0FHuz7_A/s1600/softmax.JPG \"Logo Title Text 1\")\n",
    "\n",
    "\n",
    "###  Step 7 - Choose most likely label (max probability value) \n",
    "\n",
    "argmax(softmax_outputs)\n",
    "\n",
    "These 7 steps are one forward pass through the network.\n",
    "\n",
    "## So how do we learn the magic numbers? \n",
    "\n",
    "- We can learn features and weight values through backpropagation\n",
    "\n",
    "![alt text](http://www.robots.ox.ac.uk/~vgg/practicals/cnn/images/cover.png \"Logo Title Text 1\")\n",
    "\n",
    "![alt text](https://image.slidesharecdn.com/cnn-toupload-final-151117124948-lva1-app6892/95/convolutional-neural-networks-cnn-52-638.jpg?cb=1455889178 \"Logo Title Text 1\")\n",
    "\n",
    "The other hyperparameters are set by humans and they are an active field of research (finding the optimal ones)\n",
    "\n",
    "i.e -  number of neurons, number of features, size of features, poooling window size, window stride\n",
    "\n",
    "\n",
    "## When is a good time to use it?\n",
    "\n",
    "- To classify images\n",
    "- To generate images (more on that later..)\n",
    "\n",
    "![alt text](https://nlml.github.io/images/convnet_diagram.png \"Logo Title Text 1\")\n",
    "\n",
    "But can also be applied to any any spatial 2D or 3D data. Images. Even sound and text. A rule of thumb is if you data is just as useful if you swap out the rows and columns, like customer data, then you can't use a CNN.\n",
    "\n",
    "\n",
    "## Good examples\n",
    "\n",
    "Robot learns to grasp (combining CNNs)\n",
    "\n",
    "![alt text](https://img.newatlas.com/youtube-robot-6.jpg?auto=format%2Ccompress&fit=max&h=670&q=60&w=1000&s=d003e42afa7e462fd711c6a99f21b51f \"Logo Title Text 1\")\n",
    "\n",
    "Tensorflow! https://github.com/upul/CarND-TensorFlow-Lab\n",
    "\n",
    "Adversarial CNNs https://github.com/michbad/adversarial-mnist\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle #saving and loading our serialized model \n",
    "import numpy as np #matrix math\n",
    "from app.model.preprocessor import Preprocessor as img_prep #image preprocessing\n",
    "\n",
    "#class for loading our saved model and classifying new images\n",
    "class LiteOCR:\n",
    "    \n",
    "\tdef __init__(self, fn=\"alpha_weights.pkl\", pool_size=2):\n",
    "        #load the weights from the pickle file and the meta data\n",
    "\t\t[weights, meta] = pickle.load(open(fn, 'rb'), encoding='latin1') #currently, this class MUST be initialized from a pickle file\n",
    "\t\t#list to store labels\n",
    "        self.vocab = meta[\"vocab\"]\n",
    "        \n",
    "        #how many rows and columns in an image\n",
    "\t\tself.img_rows = meta[\"img_side\"] ; self.img_cols = meta[\"img_side\"]\n",
    "        \n",
    "        #load our CNN\n",
    "\t\tself.CNN = LiteCNN()\n",
    "        #with our saved weights\n",
    "\t\tself.CNN.load_weights(weights)\n",
    "        #define the pooling layers size\n",
    "\t\tself.CNN.pool_size=int(pool_size)\n",
    "    \n",
    "    #classify new image\n",
    "\tdef predict(self, image):\n",
    "\t\tprint(image.shape)\n",
    "        #vectorize the image into the right shape for our network\n",
    "\t\tX = np.reshape(image, (1, 1, self.img_rows, self.img_cols))\n",
    "\t\tX = X.astype(\"float32\")\n",
    "        \n",
    "        #make the prediction\n",
    "\t\tpredicted_i = self.CNN.predict(X)\n",
    "        #return the predicted label\n",
    "\t\treturn self.vocab[predicted_i]\n",
    "\n",
    "class LiteCNN:\n",
    "\tdef __init__(self):\n",
    "        # a place to store the layers\n",
    "\t\tself.layers = [] \n",
    "        # size of pooling area for max pooling\n",
    "\t\tself.pool_size = None \n",
    "\n",
    "\tdef load_weights(self, weights):\n",
    "\t\tassert not self.layers, \"Weights can only be loaded once!\"\n",
    "        #add the saved matrix values to the convolutional network\n",
    "\t\tfor k in range(len(weights.keys())):\n",
    "\t\t\tself.layers.append(weights['layer_{}'.format(k)])\n",
    "\n",
    "\tdef predict(self, X):        \n",
    "        #here is where the network magic happens at a high level\n",
    "        h = self.cnn_layer(X, layer_i=0, border_mode=\"full\"); X= h\n",
    "        h = self.relu_layer(X); X = h;\n",
    "        h = self.cnn_layer(X, layer_i=2, border_mode=\"valid\"); X = h\n",
    "        h = self.relu_layer(X); X = h;\n",
    "        h = self.maxpooling_layer(X); X = h\n",
    "        h = self.dropout_layer(X, .25); X = h\n",
    "        h = self.flatten_layer(X, layer_i=7); X = h;\n",
    "        h = self.dense_layer(X, fully, layer_i=10); x = H\n",
    "        h = self.softmax_layer2D(X); x = h\n",
    "        max_i = self.classify(X)\n",
    "        return max_i[0]\n",
    "    \n",
    "    #given our feature map we've learned from convolving around the image\n",
    "    #lets make it more dense by performing pooling, specifically max pooling\n",
    "    #we'll select the max values from the image matrix and use that as our new feature map\n",
    "\tdef maxpooling_layer(self, convolved_features):\n",
    "        #given our learned features and images\n",
    "\t\tnb_features = convolved_features.shape[0]\n",
    "\t\tnb_images = convolved_features.shape[1]\n",
    "\t\tconv_dim = convolved_features.shape[2]\n",
    "\t\tres_dim = int(conv_dim / self.pool_size)       #assumed square shape\n",
    "\n",
    "        #initialize our more dense feature list as empty\n",
    "\t\tpooled_features = np.zeros((nb_features, nb_images, res_dim, res_dim))\n",
    "        #for each image\n",
    "\t\tfor image_i in range(nb_images):\n",
    "            #and each feature map\n",
    "\t\t\tfor feature_i in range(nb_features):\n",
    "                #begin by the row\n",
    "\t\t\t\tfor pool_row in range(res_dim):\n",
    "                    #define start and end points\n",
    "\t\t\t\t\trow_start = pool_row * self.pool_size\n",
    "\t\t\t\t\trow_end   = row_start + self.pool_size\n",
    "\n",
    "                    #for each column (so its a 2D iteration)\n",
    "\t\t\t\t\tfor pool_col in range(res_dim):\n",
    "                        #define start and end points\n",
    "\t\t\t\t\t\tcol_start = pool_col * self.pool_size\n",
    "\t\t\t\t\t\tcol_end   = col_start + self.pool_size\n",
    "                        \n",
    "                        #define a patch given our defined starting ending points\n",
    "\t\t\t\t\t\tpatch = convolved_features[feature_i, image_i, row_start : row_end,col_start : col_end]\n",
    "                        #then take the max value from that patch\n",
    "                        #store it. this is our new learned feature/filter\n",
    "\t\t\t\t\t\tpooled_features[feature_i, image_i, pool_row, pool_col] = np.max(patch)\n",
    "\t\treturn pooled_features\n",
    "\n",
    "    #convolution is the most important of the matrix operations here\n",
    "    #well define our input, lauyer number, and a border mode (explained below)\n",
    "\tdef cnn_layer(self, X, layer_i=0, border_mode = \"full\"):\n",
    "        #we'll store our feature maps and bias value in these 2 vars\n",
    "\t\tfeatures = self.layers[layer_i][\"param_0\"]\n",
    "\t\tbias = self.layers[layer_i][\"param_1\"]\n",
    "        #how big is our filter/patch?\n",
    "\t\tpatch_dim = features[0].shape[-1]\n",
    "        #how many features do we have?\n",
    "\t\tnb_features = features.shape[0]\n",
    "        #How big is our image?\n",
    "\t\timage_dim = X.shape[2] #assume image square\n",
    "        #R G B values\n",
    "\t\timage_channels = X.shape[1]\n",
    "        #how many images do we have?\n",
    "\t\tnb_images = X.shape[0]\n",
    "        \n",
    "        #With border mode \"full\" you get an output that is the \"full\" size as the input. \n",
    "        #That means that the filter has to go outside the bounds of the input by \"filter size / 2\" - \n",
    "        #the area outside of the input is normally padded with zeros.\n",
    "\t\tif border_mode == \"full\":\n",
    "\t\t\tconv_dim = image_dim + patch_dim - 1\n",
    "        #With border mode \"valid\" you get an output that is smaller than the input because \n",
    "        #the convolution is only computed where the input and the filter fully overlap.\n",
    "\t\telif border_mode == \"valid\":\n",
    "\t\t\tconv_dim = image_dim - patch_dim + 1\n",
    "        \n",
    "        #we'll initialize our feature matrix\n",
    "\t\tconvolved_features = np.zeros((nb_images, nb_features, conv_dim, conv_dim));\n",
    "        #then we'll iterate through each image that we have\n",
    "\t\tfor image_i in range(nb_images):\n",
    "            #for each feature \n",
    "\t\t\tfor feature_i in range(nb_features):\n",
    "                #lets initialize a convolved image as empty\n",
    "\t\t\t\tconvolved_image = np.zeros((conv_dim, conv_dim))\n",
    "                #then for each channel (r g b )\n",
    "\t\t\t\tfor channel in range(image_channels):\n",
    "                    #lets extract a feature from our feature map\n",
    "\t\t\t\t\tfeature = features[feature_i, channel, :, :]\n",
    "                    #then define a channel specific part of our image\n",
    "\t\t\t\t\timage   = X[image_i, channel, :, :]\n",
    "                    #perform convolution on our image, using a given feature filter\n",
    "\t\t\t\t\tconvolved_image += self.convolve2d(image, feature, border_mode);\n",
    "\n",
    "                #add a bias to our convoved image\n",
    "\t\t\t\tconvolved_image = convolved_image + bias[feature_i]\n",
    "                #add it to our list of convolved features (learnings)\n",
    "\t\t\t\tconvolved_features[image_i, feature_i, :, :] = convolved_image\n",
    "\t\treturn convolved_features\n",
    "\n",
    "    #In a dense layer, every node in the layer is connected to every node in the preceding layer.\n",
    "\tdef dense_layer(self, X, layer_i=0):\n",
    "        #so we'll initialize our weight and bias for this layer\n",
    "\t\tW = self.layers[layer_i][\"param_0\"]\n",
    "\t\tb = self.layers[layer_i][\"param_1\"]\n",
    "        #and multiply it by our input (dot product)\n",
    "\t\toutput = np.dot(X, W) + b\n",
    "\t\treturn output\n",
    "\n",
    "\t@staticmethod\n",
    "    \n",
    "    #so what does the convolution operation look like?, given an image and a feature map (filter)\n",
    "\tdef convolve2d(image, feature, border_mode=\"full\"):\n",
    "        #we'll define the tensor dimensions of the image and the feature\n",
    "\t\timage_dim = np.array(image.shape)\n",
    "\t\tfeature_dim = np.array(feature.shape)\n",
    "        #as well as a target dimension\n",
    "\t\ttarget_dim = image_dim + feature_dim - 1\n",
    "        #then we'll perform a fast fourier transform on both the input and the filter\n",
    "        #performing a convolution can be written as a for loop but for many convolutions\n",
    "        #this approach is too comp. expensive/slow. it can be performed orders of magnitude\n",
    "        #faster using a fast fourier transform. \n",
    "\t\tfft_result = np.fft.fft2(image, target_dim) * np.fft.fft2(feature, target_dim)\n",
    "        #and set the result to our target \n",
    "\t\ttarget = np.fft.ifft2(fft_result).real\n",
    "\n",
    "\t\tif border_mode == \"valid\":\n",
    "\t\t\t# To compute a valid shape, either np.all(x_shape >= y_shape) or\n",
    "\t\t\t# np.all(y_shape >= x_shape).\n",
    "            #decide a target dimension to convolve around\n",
    "\t\t\tvalid_dim = image_dim - feature_dim + 1\n",
    "\t\t\tif np.any(valid_dim < 1):\n",
    "\t\t\t\tvalid_dim = feature_dim - image_dim + 1\n",
    "\t\t\tstart_i = (target_dim - valid_dim) // 2\n",
    "\t\t\tend_i = start_i + valid_dim\n",
    "\t\t\ttarget = target[start_i[0]:end_i[0], start_i[1]:end_i[1]]\n",
    "\t\treturn target\n",
    "\n",
    "\tdef relu_layer(x):\n",
    "        #turn all negative values in a matrix into zeros\n",
    "\t\tz = np.zeros_like(x)\n",
    "\t\treturn np.where(x>z,x,z)\n",
    "\n",
    "\tdef softmax_layer2D(w):\n",
    "        #this function will calculate the probabilities of each\n",
    "        #target class over all possible target classes. \n",
    "\t\tmaxes = np.amax(w, axis=1)\n",
    "\t\tmaxes = maxes.reshape(maxes.shape[0], 1)\n",
    "\t\te = np.exp(w - maxes)\n",
    "\t\tdist = e / np.sum(e, axis=1, keepdims=True)\n",
    "\t\treturn dist\n",
    "\n",
    "    #affect the probability a node will be turned off by multiplying it\n",
    "    #by a p values (.25 we define)\n",
    "\tdef dropout_layer(X, p):\n",
    "\t\tretain_prob = 1. - p\n",
    "\t\tX *= retain_prob\n",
    "\t\treturn X\n",
    "\n",
    "    #get the largest probabililty value from the list\n",
    "\tdef classify(X):\n",
    "\t\treturn X.argmax(axis=-1)\n",
    "\n",
    "    #tensor transformation, less dimensions\n",
    "\tdef flatten_layer(X):\n",
    "\t\tflatX = np.zeros((X.shape[0],np.prod(X.shape[1:])))\n",
    "\t\tfor i in range(X.shape[0]):\n",
    "\t\t\tflatX[i,:] = X[i].flatten(order='C')\n",
    "\t\treturn flatX"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}