Skip to content
DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
Branch: master
Clone or download
Latest commit 35a6886 Apr 20, 2016
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
001_porting_VQA_data
002_extract_image_features
003_skipthoughts_porting
004_train_DPPnet_fixed_cnn
005_train_DPPnet_finetune_cnn
006_test_DPPnet
cache
data
model
utils
.gitignore
LICENSE
README.md
get_skip_thought_pkg.sh
prepare.sh

README.md

DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

Created by Hyeonwoo Noh, Paul Hongsuck Seo and Bohyung Han at POSTECH cvlab

Project page: [http://cvlab.postech.ac.kr/research/dppnet/]

Introduction

DPPnet is state-of-the-art Image Question Answering algorithm using dynamic parameter prediction to handle various types of questions.

Detailed description of the system will be provided by our technical report arXiv tech report

Citation

If you're using this code in a publication, please cite our papers.

@article{noh2015image,
  title={Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction},
  author={Noh, Hyeonwoo and Seo, Paul Hongsuck and Han, Bohyung},
  journal={arXiv preprint arXiv:1511.05756},
  year={2015}
}

Licence

This software is for research purpose only. Check LICENSE file for details.

System Requirements

  • This software is tested on Ubuntu 14.04 LTS (64bit).
  • At least 12GB gpu memory is required (NVIDIA tital-x gpu is used for training).

Dependencies

Setup

Run "setup.sh" for setting up.

Testing

Scripts for testing is in "006_test_DPPnet". Use following commands for testing. 0. Run ./gen_simulinks.sh 0. Run th vqa_test.lua 0. Results will be saved in "006_test_DPPnet/save_result_vqa_test/results/"

Training

Following steps are required for training. 0. Train DPPnet with fixed cnn feature (004_train_DPPnet_fixed_cnn) 0. Finetune CNN from the model trained in the previous step (005_train_DPPnet_finetune_cnn)

Directories

  • 001_porting_VQA_data: porting VQA data for torch implementation
  • 002_extract_image_features: extracting VGG16 features from MSCOCO
  • 003_skipthoughts_porting: porting model parameters for skipthoughts
  • 004_train_DPPnet_fixed_cnn: training DPPnet with extracted feature
  • 005_train_DPPnet_finetune_cnn: fine-tuning cnn from model trained 004
  • cache: We cache loaded vqa questions ans vocabularies to reduce the time for loading
  • data: data used for training / testing
  • model: trained model parameters, model definitions, layer implementations
  • utils: utilities (loading training data, loading models ...)
You can’t perform that action at this time.