Visual_Question_Answering_Demo_in_python_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Question Answering Demo and Tutorial\n",
    "\n",
    "This is an online demo with explanation and tutorial on Visual Question Answering. This is no not a naive or hello-world model, this model returns close to state-of-the-art without using any attention models, memory networks (other than LSTM) and fine-tuning, which are essential recipe for current best results.\n",
    "\n",
    "I have tried to explain different parts, and their choices. This is meant to be an interactive tutorial, feel free to change the model parameters and experiment. If you have latest graphics card, execution time should be within a minute.\n",
    "\n",
    "All the files required to run this ipython notebook can be obtained from \n",
    "### <https://github.com/iamaaditya/VQA_Demo>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import os, argparse\n",
    "import cv2, spacy, numpy as np\n",
    "from keras.models import model_from_json\n",
    "from keras.optimizers import SGD\n",
    "from sklearn.externals import joblib\n",
    "from keras import backend as K\n",
    "from keras.utils.vis_utils import plot_model\n",
    "K.set_image_data_format('channels_first')\n",
    "#K.set_image_dim_ordering('th')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the models and weights files\n",
    "This does not load the models yet, but we are providing the files\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# File paths for the model, all of these except the CNN Weights are \n",
    "# provided in the repo, See the models/CNN/README.md to download VGG weights\n",
    "VQA_model_file_name      = 'models/VQA/VQA_MODEL.json'\n",
    "VQA_weights_file_name   = 'models/VQA/VQA_MODEL_WEIGHTS.hdf5'\n",
    "label_encoder_file_name  = 'models/VQA/FULL_labelencoder_trainval.pkl'\n",
    "CNN_weights_file_name   = 'models/CNN/vgg16_weights.h5'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Idea\n",
    "This uses a classical CNN-LSTM  model like shown below, where Image features and language features are computed separately and combined together and a multi-layer perceptron is trained on the combined features.\n",
    "\n",
    "Similar models have been presented at following links, this work takes ideas from them.\n",
    "1. <https://github.com/abhshkdz/neural-vqa>\n",
    "2. <https://github.com/avisingh599/visual-qa>\n",
    "3. https://github.com/VT-vision-lab/VQA_LSTM_CNN\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://i.imgur.com/Za5P1ZZ.png\">\n",
    "[Source](http://arxiv.org/pdf/1505.00468v4.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Pretrained VGG Net (VGG-16)\n",
    "\n",
    "While VGG Net is not the best CNN model for image features, GoogLeNet (winner 2014) and ResNet (winner 2015) have superior classification scores, but VGG Net is very versatile, simple, relatively small and more importantly portable to use. \n",
    "\n",
    "For reference here is the VGG 16 performance on ILSVRC-2012\n",
    "<img src=\"http://www.robots.ox.ac.uk/~vgg/research/very_deep/images/table_ILSVRC.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_image_model(CNN_weights_file_name):\n",
    "    ''' Takes the CNN weights file, and returns the VGG model update \n",
    "    with the weights. Requires the file VGG.py inside models/CNN '''\n",
    "    from models.CNN.VGG import VGG_16\n",
    "    image_model = VGG_16(CNN_weights_file_name)\n",
    "    image_model.layers.pop()\n",
    "    image_model.layers.pop()\n",
    "    # this is standard VGG 16 without the last two layers\n",
    "    sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)\n",
    "    # one may experiment with \"adam\" optimizer, but the loss function for\n",
    "    # this kind of task is pretty standard\n",
    "    image_model.compile(optimizer=sgd, loss='categorical_crossentropy')\n",
    "    return image_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the Model\n",
    "\n",
    "Keras has a function which allows you to visualize the model in block diagram. Let's do it !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_vgg = get_image_model(CNN_weights_file_name)\n",
    "plot_model(model_vgg, to_file='model_vgg.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"model_vgg.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Image features\n",
    "\n",
    "\n",
    "Extracting image features involves, taking a raw image, and running it through the model, until we reach the last layer. In this case our model is not 100% same as VGG Net, because we are not going to use the last two layer of the VGG. It is because the last layer of VGG Net is a 1000 way softmax and the second last layer is the Dropout.\n",
    "\n",
    "Thus we are extracting the 4096 Dimension image features from VGG-16\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_image_features(image_file_name):\n",
    "    ''' Runs the given image_file to VGG 16 model and returns the \n",
    "    weights (filters) as a 1, 4096 dimension vector '''\n",
    "    image_features = np.zeros((1, 4096))\n",
    "    # Magic_Number = 4096  > Comes from last layer of VGG Model\n",
    "\n",
    "    # Since VGG was trained as a image of 224x224, every new image\n",
    "    # is required to go through the same transformation\n",
    "    im = cv2.resize(cv2.imread(image_file_name), (224, 224))\n",
    "    im = im.transpose((2,0,1)) # convert the image to RGBA\n",
    "\n",
    "    \n",
    "    # this axis dimension is required because VGG was trained on a dimension\n",
    "    # of 1, 3, 224, 224 (first axis is for the batch size\n",
    "    # even though we are using only one image, we have to keep the dimensions consistent\n",
    "    im = np.expand_dims(im, axis=0) \n",
    "\n",
    "    image_features[0,:] = model_vgg.predict(im)[0]\n",
    "    return image_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings\n",
    "\n",
    "The question has to be converted into some form of word embeddings. Most popular is Word2Vec whereas these days state of the art uses [skip-thought vectors](https://github.com/ryankiros/skip-thoughts) or [positional encodings](https://en.wikipedia.org/wiki/Encoding_(memory).\n",
    "\n",
    "We will use Word2Vec from Stanford called [Glove](http://nlp.stanford.edu/projects/glove/). Glove reduces a given token into a 300 dimensional representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_question_features(question):\n",
    "    ''' For a given question, a unicode string, returns the time series vector\n",
    "    with each word (token) transformed into a 300 dimension representation\n",
    "    calculated using Glove Vector '''\n",
    "    word_embeddings = spacy.load('en_vectors_web_lg')\n",
    "    tokens = word_embeddings(question)\n",
    "    question_tensor = np.zeros((1, 30, 300))\n",
    "    for j in xrange(len(tokens)):\n",
    "        question_tensor[0,j,:] = tokens[j].vector\n",
    "    return question_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try the embeddings\n",
    "\n",
    "Let's see the embeddings, and their usage with sample words like this -\n",
    "1. Obama \n",
    "2. Putin\n",
    "3. Banana\n",
    "4. Monkey"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_embeddings = spacy.load('en', vectors='en_glove_cc_300_1m_vectors')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "obama = word_embeddings(u\"obama\")\n",
    "putin = word_embeddings(u\"putin\")\n",
    "banana = word_embeddings(u\"banana\")\n",
    "monkey = word_embeddings(u\"monkey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5735777213479505"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obama.similarity(putin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4924444357063912"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obama.similarity(banana)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5471796768752016"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "banana.similarity(monkey)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, obama and putin are very similar in representation than obama and banana. This shows you there is some semantic knowledge of the tokens embedded in the 300 dimensional representation. We can do cool arithmatics with these word2vec like 'Queen' - 'King' + 'Boy' = 'Girl'. See [this blog post](http://multithreaded.stitchfix.com/blog/2015/03/11/word-is-worth-a-thousand-vectors/) for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VQA Model\n",
    "\n",
    "VQA is a simple model which combines features from Image and Word Embeddings and runs a multiple layer perceptron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_VQA_model(VQA_model_file_name, VQA_weights_file_name):\n",
    "    ''' Given the VQA model and its weights, compiles and returns the model '''\n",
    "\n",
    "    # thanks the keras function for loading a model from JSON, this becomes\n",
    "    # very easy to understand and work. Alternative would be to load model\n",
    "    # from binary like cPickle but then model would be obfuscated to users\n",
    "    vqa_model = model_from_json(open(VQA_model_file_name).read())\n",
    "    vqa_model.load_weights(VQA_weights_file_name)\n",
    "    vqa_model.compile(loss='categorical_crossentropy', optimizer='rmsprop')\n",
    "    return vqa_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_vqa = get_VQA_model(VQA_model_file_name, VQA_weights_file_name)\n",
    "plot_model(model_vqa, to_file='model_vqa.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"model_vqa.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it can be seen above the model also runs a 3 layered LSTM on the word embeddings. To get a naive result it is sufficient to feed the word embeddings directly to the merge layer, but as mentioned above model is gives close to the state-of-the-art results.\n",
    "\n",
    "Also, four layers of fully connected layers might not be required to achieve a good enough results. But I settled on this model after some experimentation, and their results beat few layers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Asketh Away !\n",
    "\n",
    "Let's give a test image and a question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_file_name = 'test.jpg'\n",
    "question = u\"What vehicle is in the picture?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> What vehicle is in the picture ? </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"test.jpg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the image features\n",
    "image_features = get_image_features(image_file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the question features\n",
    "question_features = get_question_features(question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "78.72 %  yes\n",
      "21.28 %  no\n",
      "000.0 %  girl\n",
      "000.0 %  halloween\n",
      "000.0 %  left\n"
     ]
    }
   ],
   "source": [
    "y_output = model_vqa.predict([question_features, image_features])\n",
    "\n",
    "# This task here is represented as a classification into a 1000 top answers\n",
    "# this means some of the answers were not part of training and thus would \n",
    "# not show up in the result.\n",
    "# These 1000 answers are stored in the sklearn Encoder class\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "labelencoder = joblib.load(label_encoder_file_name)\n",
    "for label in reversed(np.argsort(y_output)[0,-5:]):\n",
    "    print str(round(y_output[0,label]*100,2)).zfill(5), \"% \", labelencoder.inverse_transform(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "I am copying the output of the previous command, so that you can validate if your results are same as mine.\n",
    "\n",
    "**\n",
    "51.87 %  train <br/>\n",
    "031.5 %  bicycle <br/>\n",
    "03.81 %  bike <br/>\n",
    "02.91 %  bus <br/>\n",
    "02.54 %  scooter <br/>\n",
    "**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo with image URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since cv2.imread cannot read an image from URL we will have to change our function `get_image_features`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_image_features(image_file_name):\n",
    "    ''' Runs the given image_file to VGG 16 model and returns the \n",
    "    weights (filters) as a 1, 4096 dimension vector '''\n",
    "    image_features = np.zeros((1, 4096))\n",
    "        \n",
    "    from skimage import io\n",
    "    # if you would rather not install skimage, then use cv2.VideoCapture which surprisingly can read from url\n",
    "    # see this SO answer http://answers.opencv.org/question/16385/cv2imread-a-url/?answer=16389#post-id-16389\n",
    "    im = cv2.resize(io.imread(image_file_name), (224, 224))\n",
    "    im = im.transpose((2,0,1)) # convert the image to RGBA\n",
    "\n",
    "    \n",
    "    # this axis dimension is required because VGG was trained on a dimension\n",
    "    # of 1, 3, 224, 224 (first axis is for the batch size\n",
    "    # even though we are using only one image, we have to keep the dimensions consistent\n",
    "    im = np.expand_dims(im, axis=0) \n",
    "\n",
    "    image_features[0,:] = model_vgg.predict(im)[0]\n",
    "    return image_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_file_name = \"http://www.newarkhistory.com/indparksoccerkids.jpg\"\n",
    "# get the image features\n",
    "image_features = get_image_features(image_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to change that url to any valid image, it can be any image format. Also try to use websites which have higher bandwidth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://www.newarkhistory.com/indparksoccerkids.jpg\">\n",
    "# <center> What are they playing? </center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = u\"What are they playing?\"\n",
    "\n",
    "# get the question features\n",
    "question_features = get_question_features(question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "78.72 %  yes\n",
      "21.28 %  no\n",
      "000.0 %  girl\n",
      "000.0 %  halloween\n",
      "000.0 %  left\n"
     ]
    }
   ],
   "source": [
    "y_output = model_vqa.predict([question_features, image_features])\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "for label in reversed(np.argsort(y_output)[0,-5:]):\n",
    "    print str(round(y_output[0,label]*100,2)).zfill(5), \"% \", labelencoder.inverse_transform(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Result\n",
    "Copying the result to validate your output.\n",
    "\n",
    "**55.44 %  frisbee<br/>\n",
    "18.91 %  tennis<br/>\n",
    "16.95 %  baseball<br/>\n",
    "08.31 %  soccer<br/>\n",
    "00.07 %  ball <br/>\n",
    "**\n",
    "\n",
    "\n",
    "As you can see, it got this wrong, but you can see why it could be harder to guess soccer and easier to guess frisbee, lack of soccer ball and lines at the edge.\n",
    "\n",
    "Let's ask another question for the same image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = u\"Are they playing Frisbee?\"\n",
    "\n",
    "# get the question features\n",
    "question_features = get_question_features(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://www.newarkhistory.com/indparksoccerkids.jpg\">\n",
    "# <center> Are they playing Frisbee? </center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "78.72 %  yes\n",
      "21.28 %  no\n",
      "000.0 %  girl\n",
      "000.0 %  halloween\n",
      "000.0 %  left\n"
     ]
    }
   ],
   "source": [
    "y_output = model_vqa.predict([question_features, image_features])\n",
    "\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "for label in reversed(np.argsort(y_output)[0,-5:]):\n",
    "    print str(round(y_output[0,label]*100,2)).zfill(5), \"% \", labelencoder.inverse_transform(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Result\n",
    "\n",
    "**\n",
    "78.72 %  yes <br />\n",
    "21.28 %  no <br />\n",
    "000.0 %  girl <br />\n",
    "000.0 %  halloween <br />\n",
    "000.0 %  left <br />\n",
    "**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, similar information about a Yes/No question elicits different response. This is an impertinent problem with `classification` tasks.\n",
    "\n",
    "Feel free to experiment with different types of questions, `count`, `color`, `location`.\n",
    "\n",
    "More interesting results are obtained when one takes a different crop of a image, instead of just scaling it to 224x224. This is again because we extract only the top level features of CNN model which was trained to classify one object in the image."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}