Source code for training Gated Multimodal Units on MM-IMDb dataset
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
generators Initial commit Jan 17, 2017
.gitignore Initial commit Jan 17, 2017
README.md Update README.md Jan 10, 2018
concatenate.json Initial commit Jan 17, 2017
dataset.py
experiment.py Initial commit Jan 17, 2017
get_data.py Initial commit Jan 17, 2017
gmu.json Initial commit Jan 17, 2017
gmu.png upda gmu Jan 17, 2017
linear_sum.json
links.csv Initial commit Jan 17, 2017
make_dataset.py Initial commit Jan 17, 2017
maxoutmlp_w2v.json Initial commit Jan 17, 2017
model.py Initial commit Jan 17, 2017
moe.json Initial commit Jan 17, 2017
monitor.py
run.py Initial commit Jan 17, 2017
train.py
vgg.py Initial commit Jan 17, 2017
vgg_transfer.json Initial commit Jan 17, 2017

README.md

Source code for Gated Multimodal Units for Information Fusion.

GMU model

Dependencies

Make dataset

You can download a ready-to-use (multimodal_imdb.hdf5 and metadata.npy) version of the dataset in the Fuel format, or you can build it manually:

  • Get the following files and uncompress it in the root folder of this project:

  • Run the make script:

    python3 make_dataset.py gmu.json
    

Getting more movies

You can extend the dataset by adding more IMDb IDs to the links.csv file and run get_data.py script to crawl other movies.

Train and eval the model

Generate random configurations:

python3 generators/gmu.py gmu.json

Train the model and then report performance in test set (e.g. best conf for GMU model #23):

python3 run.py json/gmu_23.json