We encourage authors to submit new or adapted superpixel algorithms to keep the benchmark alive. To submit an implementation, follow these steps:
- Preparing the Datasets
- Preparing the Implementation
- Performing Parameter Optimization
- Performing Evaluation
- Submit Implementation and Results
A full example:
**We are currently working on publishing the converted datasets in the data repository, davidstutz/superpixel-benchmark-data
- BSDS500 is still missing. **
For NYUV2, SBD, SUNRGBD and Fash, the following steps are sufficient to get started:
- Clone davidstutz/superpixel-benchmark-data;
- Extract the datasets into
- Check the directory structure - ideally, the NYUV2 dataset should be found in
Algorithms can comfortably evaluated on individual datasets, not all datasets need to be downloaded or extracted.
The bare minimum:
Evaluation itself only requires the superpixel segmentations as CSV files in a directory
where the names correspond to the names of the images. For example, see
For fairness, the generated superpixel segmentations are expected to be
post-processed using the provided connected components algorithms, i.e.
SuperpixelTools::relabelConnectedSuperpixels in C++ (see
sp_fast_connected_relabel in MatLab (see
If the algorithm is based on initial markers or a grid-like initialization,
SuperpixelTools::computeHeightWidthFromSuperpixels in C++ (see
sp_height_width_superpixels in MatLab (see
should be used.
Recommended: In addition to the above constraints, all implementations in the benchmark provide an easy-to-use command line tool. It is highly recommended to provide a similar command line tool with at least the following two command line options:
-h: display all available options with descriptions;
-i: accepts a directory containing multiple PNG, JPG or JPEG images;
-o: accepts a directory which is created and used to store the superpixel segmentations for each image in CSV format - the CSV files should be named according to the images and contain the superpixel labels as integers.
-v: accepts a directory which is created and used to store visualizations of the generated superpixel segmentations, e.g. using
-w: verbose mode detailing the number of superpixels generated for each image found in the directory provided by
Examples of simple C++ command line tools:
Examples of non-C++ command line tools:
Examples for converting superpixel boundaries to superpixel segmentations:
Parameters should be optimized or chosen on the validation sets. Depending on the algorithm/implementation to be submitted, there are two cases:
- Good parameters are already known (e.g. from theory, from similar experiments or by construction);
- or the parameters need to be optimized, i.e. found.
In the first case, this step can be skipped. Next step: Performing Evaluation.
For the second case,
eval_parameter_optimization_cli might be used to perform
parameter optimization. Details can be found in
or in the full examples: Example.
Evaluation consists of two steps. In the first step,
eval_summary_cli is used
to evaluate the generated superpixel segmentations against the ground truth.
The result is a evaluation summary consisting of several metrics and statistics.
eval_summary_cli can also be found in Executables.
To match the experiments presented in the paper, evaluation has to be run for
the following numbers of superpixels:
200, 300, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000, 2400, 2800, 3200, 3600, 4000, 4600, 5200
--append-file option, the results can be gathered in a single CSV file.
This file is required for the next step.
Given this single CSV file,
eval_average_cli is used to compute the average statistics
for Boundary Recall, Undersegmentation Error and Explained Variation. Note that
in the first draft, these metrics were called Average Boundary Recall, Average Undersegmentation
Error and Average Explained Variation. Note that in the latest version, these
metrics are called Average Miss Rate, Average Undersegmentation Error and
Average Unexplained Variation. Also see Executables for details.
eval_average_cli assumes results to be available in the interval
[200, 5200] as discussed above.
Examples for both
In the spirit of reproducible and fair research, a submission consists of the source code, the command line tool, a bash script used for evaluation as well as the results of the evaluation in form of the summary (as CSV file) and the average metrics (as CSV file). The source code and the command line tool should be accompanied with building instructions if applicable and the evaluation results provided should be reproducible when running the evaluation script.
As also pointed out above, the following aspects are critical for a fair comparison:
- Parameters where optimized on the validation sets;
- the computed superpixel segmentation are post-processed using one of the provided connected components algorithm;
- to choose a grid-like initialization (if applicable) the provided schemes are to be used;
- and evaluation is performed for the number of superpixels as discussed above in the range [200, 5200].
Please provide the source code and scripts, e.g. zipped, by mail. As we encourage
authors to also make their algorithms publicly available, the source code
and results can also be made available through a public repository (e.g. GitHub, BitBucket etc.).
The easiest way will be to work with a fork of this repository, i.e. davidstutz/superpixel-benchmark.
For up-to-date contact information see
or the repository's root