Visual Question Answering in Tensorflow.
Clone or download
Latest commit ae29b54 Jul 28, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
Data Update for VQA v2 Jul 28, 2017
.gitignore Update for VQA v2 Jul 28, 2017 Add Gitter badge Aug 16, 2016 Update for VQA v2 Jul 28, 2017 Update for v2 Jul 23, 2017 Training Started Aug 11, 2016 2 layer LSTM trained, fixed bugs in evaluate Aug 13, 2016 Update for VQA v2 Jul 28, 2017 2 layer LSTM trained, fixed bugs in evaluate Aug 13, 2016 Update for v2 Jul 23, 2017

Visual Question Answering in Tensorflow

Join the chat at

This is a Tensorflow implementation of the VIS + LSTM visual question answering model from the paper Exploring Models and Data for Image Question Answering by Mengye Ren, Ryan Kiros & Richard Zemel. The model architectures vaires slightly from the original - the image embedding is plugged into the last lstm step (after the question) instead of the first. The LSTM model uses the same hyperparameters as those in the Torch implementation of neural-VQA. Model architecture



  • Download the MSCOCO train+val images and VQA data using Data/ Extract all the downloaded zip files inside the Data folder.
  • Download the pretrained VGG-16 tensorflow model and save it in the Data folder.


  • Extract the fc-7 image features using:
python --split=train
python --split=val
  • Training

    • Basic usage python
    • Options
      • rnn_size: Size of LSTM internal state. Default is 512.
      • num_lstm_layers: Number of layers in LSTM
      • embedding_size: Size of word embeddings. Default is 512.
      • learning_rate: Learning rate. Default is 0.001.
      • batch_size: Batch size. Default is 200.
      • epochs: Number of full passes through the training data. Default is 50.
      • img_dropout: Dropout for image embedding nn. Probability of dropping input. Default is 0.5.
      • word_emb_dropout: Dropout for word embeddings. Default is 0.5.
      • data_dir: Directory containing the data h5 files. Default is Data/.
  • Prediction

    • python --image_path="sample_image.jpg" --question="What is the color of the animal shown?" --model_path = "Data/Models/model2.ckpt"
    • Models are saved during training after each of the complete training data in Data/Models. Supply the path of the trained model in model_path option.
  • Evaluation

    • run python with the same options as that in, if not the defaults.

Implementation Details

  • fc7 relu layer features from the pretrained VGG-16 model are used for image embeddings. I did not scale these features, and am not sure if that can make a difference.
  • Questions are zero padded for fixed length questions, so that batch training may be used. Questions are represented as word indices of a question word vocabulary built during pre processing.
  • Answers are mapped to 1000 word vocabulary, covering 87% answers across training and validation datasets.
  • The LSTM+VIS model is defined in The input tensors for training are fc7 features, Questions(Word indices upto 22 words), Answers(one hot encoding vector of size 1000). The model depicted in the figure is implemented with 2 LSTM layers by default(num_layers in configurable).


The model achieved an accuray of 50.8% on the validation dataset after 12 epochs of training across the entire training dataset.

Sample Predictions

The fun part! Try it for yourself. Make sure you have tensorflow installed. Download the data files/trained model from this link and save them in the Data/ directory. Also download the pretrained VGG-16 model and save it as Data/vgg16.tfmodel. You can test for any sample image using:

python --image_path="Data/sample.jpg" --question="Which animal is this?" --model_path="Data/model2.ckpt"
Image Question Top Answers (left to right)
What color is the signal? red, green, yellow
What animal is this? giraffe, cow, horse
What animal is this? cat, dog, giraffe
What color is the frisbee that is in the dog's mouth? white, brown, red
What color is the frisbee that is upside down? red, white, blue
What are they playing with? frisbee, soccer ball, soccer
What is in the standing person's hand? bat, glove, ball
What are they doing? surfing, swimming, parasailing
What sport is this? skateboarding, parasailing, surfing