COMICS data / code / annotations
Switch branches/tags
Nothing to show
Clone or download
Varun Manjunatha
Varun Manjunatha adding some comments to text_cloze.py to clarify what variables stand…
… for and their dimensions
Latest commit 12bd5e4 Mar 16, 2018
Permalink
Failed to load latest commit information.
folds adding comics code and data Apr 9, 2017
models
preprocess
LICENSE
README.md
__init__.py
create_hdf5.py
run.sh
setup.sh

README.md

COMICS

code to download comics data and train models described in The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives.

email miyyer@umd.edu and varunm@cs.umd.edu with any comments/problems/questions/suggestions.

dependencies:

  • requires python 2.7, lasagne, theano, h5py, cv2, glob2

to download / unzip / preprocess COMICS data:

  • bash setup.sh (downloads raw panel images, OCR transcriptions, etc., and preprocesses them into an hdf5 file)
  • if you don't want to download everything at once, you can download individual files at https://obj.umiacs.umd.edu/comics/index.html.

to train models after preprocessing (example for text cloze):

  • python models/text_cloze.py (make sure to run on GPU; see run.sh for our theano flags)
  • see description of hyperparameters by running python models/text_cloze.py --help
  • note that low-quality data is only filtered out in dev/test data (by throwing out examples with too many UNK tokens). during training, all data is used.

results:

method text cloze easy text cloze hard visual cloze easy visual cloze hard character coherence
text only 63.4 52.9 55.9 48.4 68.2
image only 51.7 49.4 85.7 63.2 70.9
image text 68.6 61.0 81.3 59.1 69.3

if you use the COMICS data and/or code, please cite:

@InProceedings{Iyyer:Manjunatha-Comics2017,
    Title = {The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives},
    Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
    Author = {Mohit Iyyer and Varun Manjunatha and Anupam Guha and Yogarshi Vyas and Jordan Boyd-Graber and Hal {Daum\'{e} III} and Larry Davis},
    Year = {2017},
}