Skip to content
A Keras implementation of VQA using the easy-VQA dataset.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
LICENSE
README.md
analyze.py
model.py
prepare_data.py
requirements.txt
train.py

README.md

easy-VQA-keras

A Keras implementation of a simple Visual Question Answering (VQA) architecture, using the easy-VQA dataset.

Methodology described in the official blog post.

Usage

Setup and Basic Usage

First, clone the repo and install the dependencies:

git clone https://github.com/vzhou842/easy-VQA-keras.git
cd easy-VQA-keras
pip install -r requirements.txt

To run the model,

python train.py

A typical run with should have results that look like this:

Epoch 1/8
loss: 0.8887 - accuracy: 0.6480 - val_loss: 0.7504 - val_accuracy: 0.6838
Epoch 2/8
loss: 0.7443 - accuracy: 0.6864 - val_loss: 0.7118 - val_accuracy: 0.7095
Epoch 3/8
loss: 0.6419 - accuracy: 0.7468 - val_loss: 0.5659 - val_accuracy: 0.7780
Epoch 4/8
loss: 0.5140 - accuracy: 0.7981 - val_loss: 0.4720 - val_accuracy: 0.8138
Epoch 5/8
loss: 0.4155 - accuracy: 0.8320 - val_loss: 0.3938 - val_accuracy: 0.8392
Epoch 6/8
loss: 0.3078 - accuracy: 0.8775 - val_loss: 0.3139 - val_accuracy: 0.8762
Epoch 7/8
loss: 0.1982 - accuracy: 0.9286 - val_loss: 0.2202 - val_accuracy: 0.9212
Epoch 8/8
loss: 0.1157 - accuracy: 0.9627 - val_loss: 0.1883 - val_accuracy: 0.9378 

Read the "Training" section for how you might improve the accuracy of the model--we were able to get it ot 99.5% validation accuracy!.

Training

The training script train.py has two optional arguments:

python train.py [--big-model] [--use-data-dir]

Optional arguments:
  --big-model     Use the bigger model with more conv layers
  --use-data-dir  Use custom data directory, at /data

The --big-model flag trains a slightly larger model, that we used to train a 99.5% accuracy model used in the following live demo.

Furthermore, instead of using the official easy-vqa package, you generate your own dataset using the easy-VQA repo and use that instead. After following the instructions in that repo, just copy the /data folder into the root directory of this repository, so that your files look like this:

easy-VQA-keras/
├── data/
  ├── answers.txt
  ├── test/
  ├── train/
├── analyze.py
├── model.py
├── prepare_data.py
└── train.py

For the 99.5% accuracy model, we used a custom dataset generated with double the images/questions as the official dataset (set NUM_TRAIN and NUM_TEST to 8000 and 2000, respectively, for the easy-VQA repo).

Other Files

In addition to the training script, we have three other files:

  • analyze.py, a script we used to debug our models. Run using a model weights file, and produce statistics about model outputs and confusion matrices to analyze model errors.
  • model.py, where the model architecture is specified
  • prepare_data.py, which reads and processes the data, either using the easy-vqa package or a custom data directory
You can’t perform that action at this time.