Permalink
Browse files

Added average metrics including easy-to-use cli, fixed some cmake iss…

…ues, updated documentation, added instructions how to integratenew algorithms, added several examples, updated Doxygen documentation.
  • Loading branch information...
davidstutz committed Apr 16, 2017
1 parent 1733f15 commit 41fe17410d2e1399fb9561a65169739b69c35bea
Showing 571 changed files with 8,812 additions and 6,846 deletions.
View
@@ -1 +1,2 @@
*~
/build/
View
@@ -71,6 +71,7 @@ add_subdirectory(eval_connected_relabel_cli)
add_subdirectory(eval_boundaries2labels_cli)
add_subdirectory(eval_parameter_optimization_cli)
add_subdirectory(eval_summary_cli)
add_subdirectory(eval_average_cli)
add_subdirectory(eval_visualization_cli)
add_subdirectory(eval_visualization_fuse_cli)
View
@@ -3,7 +3,10 @@
This is the documentation corresponding to [davidstutz/superpixel-benchmark](https://github.com/davidstutz/superpixel-benchmark),
presenting a large-scale comparison of superpixel algorithms conducted in [1].
**[ArXiv](https://arxiv.org/abs/1612.01601) | [Project Page](http://davidstutz.de/projects/superpixel-benchmark/) | [Datasets](https://github.com/davidstutz/superpixel-benchmark-data)**
**[ArXiv](https://arxiv.org/abs/1612.01601) |
[Project Page](http://davidstutz.de/projects/superpixel-benchmark/) |
[Datasets](https://github.com/davidstutz/superpixel-benchmark-data) |
[Doxygen Documentation](https://davidstutz.github.io/superpixel-benchmark/)**
**Also check Related Files in the navigation for further documentation.** Note however,
that some of the links might not work. For the links to work, navigate to the `docs/`
@@ -13,12 +16,14 @@ This repository subsumes earlier work on comparing superpixel algorithms:
[davidstutz/gcpr2015-superpixels](https://github.com/davidstutz/gcpr2015-superpixels),
[davidstutz/superpixels-revisited](https://github.com/davidstutz/superpixels-revisited).
**Please cite the following work if you use this benchmark:**
**Please cite the following work if you use this benchmark or the provided tools or implementations:**
[1] D. Stutz, A. Hermans, B. Leibe.
Superpixels: An Evaluation of the State-of-the-Art.
Computing Research Repository, abs/1612.01601.
Also make sure to cite additional papers when using datasets or superpixel algorithms.
## License
Note that part of the provided algorithms come with different licenses.
View
@@ -5,27 +5,41 @@
This repository contains the source code used for evaluation in [1], a large-scale
comparison of state-of-the-art superpixel algorithms.
**[ArXiv](https://arxiv.org/abs/1612.01601) | [Project Page](http://davidstutz.de/projects/superpixel-benchmark/) | [Datasets](https://github.com/davidstutz/superpixel-benchmark-data) | [Doxygen Documentation](https://davidstutz.github.io/superpixel-benchmark/)**
**[ArXiv](https://arxiv.org/abs/1612.01601) |
[Project Page](http://davidstutz.de/projects/superpixel-benchmark/) |
[Datasets](https://github.com/davidstutz/superpixel-benchmark-data) |
[Doxygen Documentation](https://davidstutz.github.io/superpixel-benchmark/)**
This repository subsumes earlier work on comparing superpixel algorithms:
[davidstutz/gcpr2015-superpixels](https://github.com/davidstutz/gcpr2015-superpixels),
[davidstutz/superpixels-revisited](https://github.com/davidstutz/superpixels-revisited).
**Please cite the following work if you use this benchmark:**
**Please cite the following work if you use this benchmark or the provided tools or implementations:**
[1] D. Stutz, A. Hermans, B. Leibe.
Superpixels: An Evaluation of the State-of-the-Art.
Computing Research Repository, abs/1612.01601.
Make also sure to cite additional papers when using datasets or superpixel algorithms.
Also make also sure to cite additional papers when using datasets or superpixel algorithms.
**Updates:**
* An implementation of the average metrics, i.e. Average Boundary Recall (called
Average Miss Rate in the updated paper), Average Undersegmentation Error
and Average Explained Variation (called Average Unexplained Variation in the updated paper)
is provided in `lib_eval/evaluation.h` and an easy-to-use command line tool is provided,
see `eval_average_cli` and the corresponding documentation and examples in
[Executables](docs/EXECUTABLES.md) and [Examples](docs/EXAMPLE.md) respectively.
* As of Mar 29, 2017 the paper was accepted for publication at
[CVIU](https://www.journals.elsevier.com/computer-vision-and-image-understanding/).
* The converted (i.e. pre-processed) NYUV2, SBD and SUNRGBD datasets are now available
in the [data repository](https://github.com/davidstutz/superpixel-benchmark-data).
* The source code of MSS has been added.
* The source code of PF and SEAW has been added.
* Doxygen documentation is now available [here](http://davidstutz.github.io/superpixel-benchmark/).
* The presented paper was in preparation for a longer period of time —
some recent superpixel algorithms are not included in the comparison. These include
[SCSP](https://github.com/freifeld/fastSCSP) and [LRW](https://github.com/shenjianbing/lrw14).
We are working on including these algorithms in future updates.
## Table of Contents
View
@@ -743,7 +743,7 @@ WARN_LOGFILE =
# spaces.
# Note: If this tag is empty the current directory is searched.
INPUT = . docs lib_eval ccs_cli lib_ccs/ccs_opencv.h lib_ccs/ccs_opencv.cpp cis_cli lib_cis/cis_opencv.h crs_cli lib_crs/crs_opencv.h cw_cli dasp_cli lib_dasp/dasp_opencv.h lib_dasp/dasp_opencv.cpp ergc_cli lib_ergc/ergc_opencv.h ers_cli lib_ers/ers_opencv.h lib_ers/ers_opencv.cpp etps_cli lib_etps/etps_opencv.h lib_etps/etps_opencv.cpp eval_boundaries2labels_cli eval_connected_relabel_cli eval_parameter_optimization_cli eval_summary_cli eval_visualization_cli eval_visualizatioN_fuse_cli fh_cli lib_fh/fh_opencv.h lsc_cli lib_lsc/lsc_opencv.h mss_cli lib_mss/mss_opencv.h pb_cli lib_pb/pb_opencv.h lib_pb/pb_opencv.cpp preslic_cli seeds_cli slic_cli lib_slic/slic_opencv.h lib_slic/slic_opencv.cpp vc_cli lib_vc/vc_opencv.h vccs_cli lib_vccs/vccs_opencv_pcl.h lib_vccs/vccs_opencv_pcl.cpp vlslic_cli lib_vlslic/vlslic_opencv.h w_cli lib_refh/lib lib_mss/mss_opencv.h mss_cli
INPUT = . docs lib_eval ccs_cli lib_ccs/ccs_opencv.h lib_ccs/ccs_opencv.cpp cis_cli lib_cis/cis_opencv.h crs_cli lib_crs/crs_opencv.h cw_cli dasp_cli lib_dasp/dasp_opencv.h lib_dasp/dasp_opencv.cpp ergc_cli lib_ergc/ergc_opencv.h ers_cli lib_ers/ers_opencv.h lib_ers/ers_opencv.cpp etps_cli lib_etps/etps_opencv.h lib_etps/etps_opencv.cpp eval_boundaries2labels_cli eval_connected_relabel_cli eval_parameter_optimization_cli eval_summary_cli eval_visualization_cli eval_visualization_fuse_cli fh_cli lib_fh/fh_opencv.h lsc_cli lib_lsc/lsc_opencv.h mss_cli lib_mss/mss_opencv.h pb_cli lib_pb/pb_opencv.h lib_pb/pb_opencv.cpp preslic_cli seeds_cli slic_cli lib_slic/slic_opencv.h lib_slic/slic_opencv.cpp vc_cli lib_vc/vc_opencv.h vccs_cli lib_vccs/vccs_opencv_pcl.h lib_vccs/vccs_opencv_pcl.cpp vlslic_cli lib_vlslic/vlslic_opencv.h w_cli lib_refh/lib lib_mss/mss_opencv.h mss_cli examples/cpp examples/submission/parameter_optimization.cpp examples/submission/command_line_tool.cpp eval_average_cli/main.cpp
# This tag can be used to specify the character encoding of the source files
# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
View
@@ -44,5 +44,5 @@ target_link_libraries(cw_cli
eval
cw
${Boost_LIBRARIES}
${OpenCV_LIBS}
${OpenCV_LIBRARIES}
)
View
@@ -1,13 +1,43 @@
# Algorithms
An overview of the evaluated algorithms can be found below. The table
is taken from the appendix in [35].
An overview of the evaluated algorithms can be found below. More details can be found
in the paper or in the original README found in the respective library directory.
Also check the corresponding web pages for author and license information. The
corresponding references are given below the table.
[35] D. Stutz, A. Hermans, B. Leibe.
Superpixels: An Evaluation of the State-of-the-Art.
Computing Research Repository, abs/1612.01601.
![Algorithm overview.](ALGORITHMS_TABLE.png?raw=true "Algorithm overview.")
Algorithm | Library | Executable | Implementation | Reference | Link
-------------|---------------|---------------|----------------|-----------|-----
CCS | `lib_ccs` | `ccs_cli` | C++ | [22,23] | [Web](http://www.emrahtasli.com/research/spextraction/)
CIS | `lib_cis` | `cis_cli` | C++ | [10] | [Web](http://www.csd.uwo.ca/faculty/olga/)
CRS | `lib_crs` | `crs_cli` | C++ | [13,14] | [Web](http://www.vsi.cs.uni-frankfurt.de/research/superpixel-segmentation/)
CW | `lib_cw` | `cw_cli` | C++ | [25] | [Web](https://www.tu-chemnitz.de/etit/proaut/forschung/cv/segmentation.html.en)
DASP | `lib_dasp` | `dasp_cli` | C++ | [17] | [Web](https://github.com/Danvil/dasp)
EAMS | `lib_eams` | `eams_cli` | MatLab | [2] | [Web](http://coewww.rutgers.edu/riul/research/code/EDISON/)
ERS | `lib_ers` | `ers_cli` | C++ | [15] | [Web](http://mingyuliu.net/)
FH | `lib_fh` | `fh_cli` | C++ | [4] | [Web](https://cs.brown.edu/~pff/segment/index.html)
reFH | `lib_refh` | `refh_cli` | C++ | -- | [Web](http://davidstutz.de/projects/superpixel-segmentation/)
MSS | `lib_mss` | `mss_cli` | C++ | [28] | --
PB | `lib_pb` | `pb_cli` | C++ | [16] | [Web](http://yuhang.rsise.anu.edu.au/yuhang/misc.html)
preSLIC | `lib_preslic` | `preslic_cli` | C++ | [25] | [Web](https://www.tu-chemnitz.de/etit/proaut/forschung/cv/segmentation.html.en)
SEEDS | `lib_seeds` | `seeds_cli` | C++ | [18] | [Web](http://www.mvdblive.org/seeds/)
reSEEDS | `lib_reseeds` | `reseeds_cli` | C++ | -- | [Web](http://davidstutz.de/projects/superpixel-segmentation/)
SEAW | `lib_seaw` | `seaw_cli` | MatLab | [34] | [Web](https://github.com/JohannStrassburg/InfluenceSegImageParsingCode)
SLIC | `lib_slic` | `slic_cli` | C++ | [11,12] | [Web](http://ivrl.epfl.ch/research/superpixels)
vlSLIC | `lib_clslic` | `vlslic_cli` | C++ | -- | [Web](http://www.vlfeat.org/overview/slic.html)
TP | `lib_tp` | `tp_cli` | MatLab | [9] | [Web](http://www.cs.toronto.edu/~babalex/research.html)
TPS | `lib_tps` | `tps_cli` | MatLab | [19,20] | [Web](http://hzfu.github.io/subpage/codes.html)
W | `lib_w` | `w_cli` | C++ | [1] | [Web](http://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html?highlight=watershed#watershed)
WP | `lib_wp` | `wp_cli` | Python | [29,30] | [Web](http://cmm.ensmp.fr/~machairas/waterpixels.html)
PF | `lib_pf` | `pf_cli` | Java | [8] | [Web](http://users.dickinson.edu/~jmac/publications/PathFinder.zip)
LSC | `lib_lsc` | `lsc_cli` | C++ | [32] | [Web](http://jschenthu.weebly.com/projects.html)
RW | `lib_rw` | `rw_cli` | MatLab | [5, 6] | [Web](http://cns.bu.edu/~lgrady/software.html)
QS | `lib_qs` | `qs_cli` | MatLab | [7] | [Web](http://www.vlfeat.org/overview/quickshift.html)
NC | `lib_nc` | `nc_cli` | C++ | [3] | [Web](http://www.cs.sfu.ca/~mori/research/superpixels)
VCCS | `lib_vccs` | `vccs_cli` | MatLab | [24] | [Web](http://pointclouds.org/documentation/tutorials/supervoxel_clustering.php)
POISE | `lib_poise` | `poise_cli` | MatLab | [33] | [Web](http://rehg.org/poise/)
VC | `lib_vc` | `vc_cli` | C++ | [21] | [Web](http://www-personal.umich.edu/~jwangumi/software.html)
ETPS | `lib_etps` | `etps_cli` | C++ | [31] | [Web](https://bitbucket.org/mboben/spixel)
ERGC | `lib_ergc` | `ergc_cli` | C++ | [26,27] | [Web](https://sites.google.com/site/pierrebuyssens/code/ergc)
[1] F. Meyer.
Color image segmentation.
@@ -110,43 +140,4 @@ is taken from the appendix in [35].
International Conference on Computer Vision, 2015, pp. 1600–1608.
[34] J. Strassburg, R. Grzeszick, L. Rothacker, G. A. Fink.
On the influence of superpixel methods for image parsing.
International Conference on Computer Vision Theory and Application, 2015, pp. 518–527.
## Licenses
As far as the provided algorithms included a `README.md` or `LICENSE.md` file, these
can be found in the corresponding subdirectories. For all algorithms, we include
the author's web pages below:
Algorithm | Link
-------------|-----
CCS | [Web](http://www.emrahtasli.com/research/spextraction/)
CIS | [Web](http://www.csd.uwo.ca/faculty/olga/)
CRS | [Web](http://www.vsi.cs.uni-frankfurt.de/research/superpixel-segmentation/)
CW | [Web](https://www.tu-chemnitz.de/etit/proaut/forschung/cv/segmentation.html.en)
DASP | [Web](https://github.com/Danvil/dasp)
EAMS | [Web](http://coewww.rutgers.edu/riul/research/code/EDISON/)
ERS | [Web](http://mingyuliu.net/)
FH | [Web](https://cs.brown.edu/~pff/segment/index.html)
MSS | --
PB | [Web](http://yuhang.rsise.anu.edu.au/yuhang/misc.html)
preSLIC | [Web](https://www.tu-chemnitz.de/etit/proaut/forschung/cv/segmentation.html.en)
reSEEDS | [Web](http://davidstutz.de/projects/superpixelsseeds/)
SEAW | [Web](https://github.com/JohannStrassburg/InfluenceSegImageParsingCode)
SEEDS | [Web](http://www.mvdblive.org/seeds/)
SLIC | [Web](http://ivrl.epfl.ch/research/superpixels)
TP | [Web](http://www.cs.toronto.edu/~babalex/research.html)
TPS | [Web](http://hzfu.github.io/subpage/codes.html)
vlSLIC | [Web](http://www.vlfeat.org/overview/slic.html)
W | [Web](http://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html?highlight=watershed#watershed)
WP | [Web](http://cmm.ensmp.fr/~machairas/waterpixels.html)
PF | [Web](http://users.dickinson.edu/~jmac/publications/PathFinder.zip)
LSC | [Web](http://jschenthu.weebly.com/projects.html)
RW | [Web](http://cns.bu.edu/~lgrady/software.html)
QS | [Web](http://www.vlfeat.org/overview/quickshift.html)
NC | [Web](http://www.cs.sfu.ca/~mori/research/superpixels)
VCCS | [Web](http://pointclouds.org/documentation/tutorials/supervoxel_clustering.php)
POISE | [Web](http://rehg.org/poise/)
VC | [Web](http://www-personal.umich.edu/~jwangumi/software.html)
ETPS | [Web](https://bitbucket.org/mboben/spixel)
ERGC | [Web](https://sites.google.com/site/pierrebuyssens/code/ergc)
International Conference on Computer Vision Theory and Application, 2015, pp. 518–527.
View
Binary file not shown.
View
@@ -1,12 +1,13 @@
# Benchmark
The benchmark combines metrics from different references focussing on different
aspects of good superpixel segmentations. Details can be found in [1]. In the
following, the relevant metrics are briefly introduced.
aspects of superpixel segmentations. Detailed equations can be found in the paper
or in the [Doxygen Documentation](https://davidstutz.github.io/superpixel-benchmark/),
specifically in the class [Evaluation](https://davidstutz.github.io/superpixel-benchmark/classEvaluation.html).
[1] D. Stutz, A. Hermans, B. Leibe.
Superpixels: An Evaluation of the State-of-the-Art.
Computing Research Repository, abs/1612.01601.
Note: Evaluation should be done using the executables discussed in [Executables](EXECUTABLES.md).
However, the metrics implemented in the [Evaluation](https://davidstutz.github.io/superpixel-benchmark/classEvaluation.html).
class can also be used directly from within C++.
## Boundary Recall
@@ -41,10 +42,8 @@ These pixels constitute the leakage of superpixels across the ground truth segme
Superpixel benchmark and comparison.
Forum Bildverarbeitung, 2012.
Note: `lib_eval/evaluation.h` includes three different implementations of Undersegmentation
Error: the definition by Neubert and Protzel, the definition by Levinshtein, and
a custom definition which can be understood as counterpart of a simplified Achievable
Segmentation Accuracy (which is called Oversegmentation Error in `lib_eval/evaluation.h`).
Note: `lib_eval/evaluation.h` includes two different implementations of Undersegmentation
Error: the definition by Neubert and Protzel and the definition by Levinshtein.
## Achievable Segmentation Accuracy
@@ -109,10 +108,8 @@ in the superpixel segmentation
Waterpixels.
Transactions on Image Processing 24 (11) (2015) 3707–3716.
## Average Boundary Recall, Average Undersegmentation Error and Average Explained Variation
## Average Metrics
In order to render Boundary Recall, Undersegmentation Error and Explained Variation independent
of the number of generated superpixels, the area above (and below and above)
the corresponding curves is used to summarize performance. To avoid confusion,
these metrics are called Average Recall, Average Undersegmentation Error and
Average Explained Variation.
of the number of generated superpixels, the area above (or below in the case of
Undersegmentation Error) the corresponding curves is used to summarize performance.
Oops, something went wrong.

0 comments on commit 41fe174

Please sign in to comment.