Interpretable Discovery in Large Image Datasets
Kiri Wagstaff, Jake Lee
in WHI 2018
This repository contains supplemental scripts and data used in the experiments presented in the paper.
Step-by-step instructions for running experiments
Download the imageset - It is recommended that the image filename include the class information. The images can be in class-related subfolders; if not, it can be placed in a single subfolder (ex.
Preprocess the imageset - We recommend scaling/center-cropping your images to 227x227 first. We used imagemagick:
$ mogrify -path imageset/# -format jpg -resize "227x227^" -gravity center -crop 227x227+0+0 +repage imageset/#/*.jpg
Download/install DEMUD - Available from https://github.com/wkiri/DEMUD
Extract features - Extract features from the images by using
DEMUD/scripts/cnn_feat_extraction/feat_csv.py. The extracted features will be saved as a CSV, with the first column being the image name.
Note: You will need to install Caffe and to specify the trained Caffe model from which the features will be extracted. We used Caffe's pre-trained network called
bvlc_reference_caffenet with a modified
deploy.prototxt. The modified prototxt is available in this repository.
Run DEMUD on features - Run DEMUD by adding the path to the feature CSV in
python demud.py -v --init-item=svd --k=50It will output
recon-cnn-k=50..., the reconstructed features,
resid-cnn-k50..., the residual features,
select-cnn-k=50..., the features of the selected image, and
selections-k50.csv, a list of selected image names (in corresponding order with the other outputs).
Download D&B CVPR 2016 - Available from https://lmb.informatik.uni-freiburg.de/resources/software.php
Visualize the output features - Using
src/demud_inversion_cvpr2016.pyprovided in this repo (see below for instructions)
Plot discovery rates (optional) - Using
src/plot/plot_exp.pyprovided in this repo
Organize your visualizations - Using
src/gen_html.pyprovided in this repo
src/gen_html.py collects the results from DEMUD and Dosovitskiy and Brox inversions into a single HTML file for convenient viewing. It requires the following paths to be edited into the appropriate variables:
outputname- The path of the html file to be generated
imageset_dir- The directory containing the full imageset, but not in subdirectories
dosov_dir- The directory containing the inversions generated by Dosovitskiy and Brox. It assumes the following subdirectories:
dosov_dir/select/- Inversions of the features in the
select-cnn-*.csvoutput from DEMUD
dosov_dir/recon/- Inversions of the features in the
recon-cnn-*.csvoutput from DEMUD
dosov_dir/resid/- Inversions of the features in the
resid-cnn-*.csvoutput from DEMUD
selections-kn.csvoutput file from DEMUD
perfplot_path- The path to the discovery performance plot, if it was generated.
src/plot/util_plot.py were used to generate the plots shown in the paper.
plot_exp.py requires the directory containing DEMUD result folders. Note that, since these scripts were used specifically for our set of experiments, they may not be easy to use for other experiments.
src/demud_inversion_cvpr2016.py is a modified version of
demo.py from Dosovitskiy and Brox, CVPR 2016. It reads each of the output CSVs from DEMUD
-v/--cnn (selected features, reconstructed features, and residual features), and inverts them for visualization. The required pretrained model definitions are available here
src/build_imageset/ext_from_train.py builds the balanced ImageNet dataset used in the experiments from the ILSVRC 2012 training set. The classes to be pulled are specified in
build_imageset/random_classes.txt. The files are renamed to match the pattern
classID_imageNO.jpg and placed in their respective subfolders; for example, an image may be
src/deploy.prototxt is a modified prototxt for the model
bvlc_reference_caffenet. It is modified to perform ReLU separately instead of in-place for the fully connected layers, allowing us to extract features before ReLU is applied. It requires more memory to run, however.
data/sun-class/ includes an additional "sun" class to the Mars image dataset available on Zenodo. These 21 additional images were included for the experiments presented in the paper.