##Simultaneous Detection and Segmentation
This is code for the ECCV Paper:
Simultaneous Detection and Segmentation
Bharath Hariharan, Pablo Arbelaez, Ross Girshick, Jitendra Malik
To appear in ECCV, 2014.
Installing caffe: The code comes bundled with a version of caffe that we have modified slightly for SDS. (These modifications might be merged into the public caffe version sometime in the future). To install caffe, follow the instructions on the caffe webpage. (You'll have to install some pre-requisites). After installing all prerequisites, cd into
After you have made caffe, you will also need to do
Downloading other external dependencies (MCG and liblinear): The extern folder has a script that downloads MCG and liblinear and compiles liblinear. After running the script, cd into
extern/MCG-PreTrainedand change the path in
root_dir.mto the path to the MCG-PreTrained directory.
Starting MATLAB: Start MATLAB and call
startup_sdsfrom the main SDS directory. This will compile all mexes in MCG and liblinear, and add all paths.
A few possible issues related to Caffe:
- You may need to add the path to CUDA libraries (usually in /usr/local/cuda/lib64)
LD_LIBRARY_PATHbefore starting MATLAB.
- When running the code, if you get an error saying:
/usr/lib/x86_64-linux-gnu/libharfbuzz.so.0: undefined symbol: FT_Face_GetCharVariantIndex, try adding
/usr/lib/x86_64-linux-gnu/libfreetype.so.6(or the equivalent library that your system may have) to the
LD_PRELOADenvironment variable before starting MATLAB.
- You may need to add the path to CUDA libraries (usually in /usr/local/cuda/lib64) to
###Using Pre-computed results
To get started you can look at precomputed results.
Download the precomputed results from this ftp link:
and untar it. The precomputed results contain results on VOC2012 val images (SDS, detection and segmentation).
You can visualize the precomputed results using the function
visualize_precomputed_results('/path/to/precomputed/results', '/path/to/VOC2012/VOCdevkit/VOC2012/JPEGImages', categ_id);
categ_id is the number of the category, for example 15 for person.
Note that you do not need to install Caffe or any of the external dependencies above if you want to simply visualize or use precomputed results.
###Testing Pre-trained models
Download the pretrained models from this ftp link:
and untar them in the main SDS directory.
demo_sds.m is a simple demo that uses the precomputed models to show the outputs we get on a single image. It takes no arguments.
It runs the trained models on an example image and displays the detections for the person category.
This function is a wrapper around the main function, which is called
###Benchmarking and evaluation
You can also run the benchmark demo,
demo_sds_benchmark, which tests our pipeline on a small 100 image subset of
VOC2012 val and then evaluates for the person category. You can call it as follows:
demo_sds_benchmark('/path/to/VOC2012/VOCdevkit/VOC2012/JPEGImages/', '/path/to/cachedir', '/path/to/SBD');
Here the cachedir is a directory where intermediate results will be stored. The function also requires the SBD (Semantic Boundaries Dataset), which you can get here. The function does the evaluation for both before refinement and after refinement, and reports an APr of 59.9 in the first case and 66.8 in the second case.
The main function for running the benchmark is
demo_sds_benchmark should point
you to how to run the benchmark.
###Evaluating on detection and segmentation
Detection: Look at
imagelist_to_det.mto see how to produce a bounding box detection output. In summary, after computing scores on all regions, we use
misc/box_nms.mto non-max suppress the boxes using box overlap.
misc/write_test_boxesthen writes the boxes out to a file that you can submit to PASCAL.
Semantic segmentation: Look at
imagelist_to_seg.mto see how we produce a semantic segmentation output. In summary, after we compute scores on all regions, we do
misc/region_nms.mto non-max suppress boxes, and use
misc/get_top_regions.mto get the top regions per category. For our experiments, we picked the top 5K regions for seg val and seg test. Then we call
[local_ids, labels, scores2] = paste_segments(topchosen, scores, region_meta_info, 2, 10, -1);
topchosenis the first output of
get_top_regions.m. These parameters above were tuned on seg val 2011. This function will pick out the segments to paste. To do the actual pasting, use
create_pasted_segmentations(if you don't want any refinement) or
create_pasted_segmentations_refined(if you want refinement). Refinement is a bit slower but works ~1 point better.
###SDS results format If you want to do more with our results, you may want to understand how we represent our results.
- Representing region candidates: Because we work with close to 2000 region candidates, saving them as full image-sized masks uses up a lot of space and requires a lot of memory to process. Instead, we save these region candidates using a superpixel representation: we save a superpixel map, containing the superpixel id for each pixel in the image, and we represent each region as a binary vector indicating which superpixels are present in the region. To allow this superpixel representation to be accessible to Caffe, we
- save the superpixel map as a text file, the first two numbers in which represent the size of the image and the rest of the file contains the superpixel ids of the pixels in MATLAB's column-major order (i.e, we first store the superpixel ids of the first column, then the second column and so on).
- stack the representation of each region as a matrix (each column representing a region) and save it as a png image.
read_sprep can read this representation into matlab.
Representing detections: After the regions have been scored and non-max suppressed, we store the chosen regions as a cell array, one cell per category. Each cell is itself a cell array, with as many cells as there are images, and each cell containing the region id of the chosen regions. The scores are stored in a separate cell array.
Representing refined detections: After refinement, the refined regions are stored as binary matrices in mat files, one for each image. The refined regions for different categories are stored in different directories
###Retraining region classifiers
To retrain region classifiers, you first need to save features for all regions including ground truth. You can look at the function
setup_svm_training.m. This function will save features and return a
region_meta_info struct, which has in it the overlaps of all the
regions with all the ground truth. The function expects a list of images, a number of paths to save stuff in, and a path to the
ground truth (SBD).
Once the features are saved you can use the
region_classification/train_svms.m function to train the detectors.
You can also train refinement models for each category using
###Retraining the network To retrain the network you will have to use caffe. You need two things: a prototxt specifying the architecture, and a window file specifying the data.
Window file: Writing the window file requires you to make a choice between using box overlap to define ground truth, or using region overlap to define ground truth. In the former case, use
feature_extractor/make_window_file_box.mand in the latter use
feature_extractor/make_window_file_box.m. Both functions require as input the image list,
setup_svm_trainingto see how to call it), sptextdir, regspimgdir (specifying the superpixels and regions) and the filename in which the output should go.
Prototxt: There are 3 prototxts that figure during training. One specifies the solver, and points to the other two: one for training and the other for testing. Training a single pathway network for boxes can be done with the
window_val, a single pathway network on regions can be done using
masked_window_val, and a two pathway network (net C) can be trained using
piwindow_val. (Here "pi" refers to the architecture of the network, which looks like the capital greek pi.) The train and val prototxts also specify which window file to use. The solver prototxt specifies the path to the train and val prototxts. It also specifies where the snapshots are saved. Make sure that path can be saved to.
Initialization: A final requirement for finetuning is to have an initial network, and also the imagenet mean. The latter you can get by running
extern/caffe/data/ilsvrc12/get_ilsvrc_aux.shThe initial network is the B network for net C. For everything else, it is the caffe reference imagenet model, which you can get by running
Finetuning: cd into caffe and use the following command to train the network (replace
caffe_reference_imagenet_modelby the appropriate initialization):
GLOG_logtostderr=1 ./build/tools/finetune_net.bin ../prototxts/pascal_finetune_solver.prototxt ./examples/imagenet/caffe_reference_imagenet_model 2>&1 | tee logdir/log.txt
Finally, extracting features requires a network with the two-pathway architecture. If you trained the box and region pathway separately, you can stitch them together using