COCO-Stuff 10K dataset v1.1
- Results and Future Plans
- Semantic Segmentation Models
- Annotation Tool
- 10,000 complex images from COCO 
- Dense pixel-level annotations
- 91 thing and 91 stuff classes
- Instance-level annotations for things from COCO 
- Complex spatial context between stuff and things
- 5 captions per image from COCO 
- 06 Apr 2017: Dataset version 1.1: Modified label indices
- 31 Mar 2017: Published annotations in JSON format
- 09 Mar 2017: Added label hierarchy scripts
- 08 Mar 2017: Corrections to table 2 in arXiv paper 
- 10 Feb 2017: Added tools extract SLICO superpixels in annotation tool
- 12 Dec 2016: Dataset version 1.0 and arXiv paper  released
Results and Future Plans
The current release of COCO-Stuff-10K publishes both the training and test annotations and users report their performance individually. We invite users to report their results to us to complement this table. In the near future we will extend COCO-Stuff to the 200K images in COCO 2015 and organize an official challenge where the test annotations will only be known to the organizers.
|Method||Source||Class-average accuracy||Global accuracy||Mean IOU||FW IOU|
|Deeplab (no CRF) ||||38.1%||57.8%||26.9%||-|
|OHE + DC + FCN+ ||||45.8%||66.6%||34.3%||51.2%|
|W2V + DC + FCN+ ||||45.1%||66.1%||34.7%||51.0%|
|cocostuff-10k-v1.1.zip||COCO-Stuff dataset v. 1.1, images and annotations||2.0 GB|
|cocostuff-10k-v1.1.json||COCO-Stuff dataset v. 1.1, annotations in JSON format (optional)||62.3 MB|
|cocostuff-labels.txt||A list of the 1+91+91 classes in COCO-Stuff||2.3 KB|
|cocostuff-readme.txt||This document||6.5 KB|
|cocostuff-10k-v1.0.zip||COCO-Stuff dataset version 1.0, including images and annotations||2.6 GB|
To use the COCO-Stuff dataset, please follow these steps:
- Download or clone this repository using git:
git clone https://github.com/nightrome/cocostuff.git
- Open the dataset folder in your shell:
- If you have Matlab, run the following commands:
- Add the code folder to your Matlab path:
- Run the demo script in Matlab
- The script displays an image, its thing, stuff and thing+stuff annotations, as well as the image captions.
- Alternatively run the following Linux commands or manually download and unpack the dataset:
wget --directory-prefix=downloads http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-10k-v1.1.zip
unzip downloads/cocostuff-10k-v1.1.zip -d dataset/
Alternatively, we also provide annotations in the COCO-style JSON format above. These are created from the .mat file annotations using this Python script. They include stuff and thing annotations. The thing annotations are copied from COCO. We encode every stuff class present in an image as a single annotation using the RLE encoding format of COCO.
The hierarchy of labels is stored in
CocoStuffClasses. To visualize it, run
CocoStuffClasses.showClassHierarchyStuffThings() (also available for just stuff and just thing classes) in Matlab. The output should look similar to the following figure:
Label Names & Indices
To be compatible with COCO, version 1.1 of COCO-Stuff has 91 thing classes (1-91), 91 stuff classes (92-182) and 1 class "unlabeled" (0). Note that 11 of the thing classes from COCO 2015 do not have any segmentation annotations. The classes desk, door and mirror could be either stuff or things and therefore occur in both COCO and COCO-Stuff. To avoid confusion we add the suffix "-stuff" to those classes in COCO-Stuff. The full list of classes can be found here.
The older version 1.0 of COCO-Stuff had 80 thing classes (2-81), 91 stuff classes (82-172) and 1 class "unlabeled" (1).
The COCO-Stuff annotations are stored in separate .mat files per image. These files follow the same format as used by Tighe et al.. Each file contains the following fields:
- S: The pixel-wise label map of size [height x width].
- names: The names of the thing and stuff classes in COCO-Stuff. For more details see Label Names & Indices.
- captions: Image captions from  that are annotated by 5 distinct humans on average.
- regionMapStuff: A map of the same size as S that contains the indices for the approx. 1000 regions (superpixels) used to annotate the image.
- regionLabelsStuff: A list of the stuff labels for each superpixel. The indices in regionMapStuff correspond to the entries in regionLabelsStuff.
Semantic Segmentation Models
Note: This section will be updated to v. 1.1 soon...
To encourage further research of stuff and things we provide the trained semantic segmentation model (see Sect. 4.4 in ).
- Download deeplab-public-ver2:
git submodule update --init models/deeplab-public-ver2
- Compile and configure deeplab-public-ver2 following the author's instructions. Depending on your system setup you might have to install additional packages, but a minimum setup could look like this:
cp Makefile.config.example Makefile.config
- Download and unzip the model:
wget --directory-prefix=downloads http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-deeplab.zip
unzip downloads/cocostuff-deeplab.zip -d models/deeplab-public-ver2/
- Configure the COCO-Stuff dataset:
- Create a symbolic link to the images:
mkdir models/deeplab-public-ver2/cocostuff/data && ln -s ../../../../dataset/images models/deeplab-public-ver2/cocostuff/data/images
- Convert the annotations by running the Matlab script:
cd models/deeplab-public-ver2 && ./run_cocostuff.sh && cd ../..to train and test the network on COCO-Stuff.
In  we present a simple and efficient stuff annotation tool which was used to annotate the COCO-Stuff dataset. It uses a paintbrush tool to annotate SLICO superpixels (precomputed using the code of Achanta et al.) with stuff labels. These annotations are overlaid with the existing pixel-level thing annotations from COCO. We provide a basic version of our annotation tool:
- Prepare the required data:
- Specify a username in
- Create a list of images in
- Extract the thing annotations for all images in Matlab:
- Extract the superpixels for all images in Matlab:
- To enable or disable superpixels, thing annotations and polygon drawing, take a look at the flags at the top of
- Specify a username in
- Run the annotation tool in Matlab:
- The tool writes the .mat label files to
- To create a .png preview of the annotations, run
annotator/code/exportImages.min Matlab. The previews will be saved to
- The tool writes the .mat label files to
 COCO-Stuff: Thing and Stuff Classes in Context
H. Caesar, J. Uijlings, V. Ferrari,
In arXiv preprint arXiv:1612.03716, 2017.
 Microsoft COCO: Common Objects in Context
T.-Y. Lin, M. Maire, S. Belongie et al.,
In European Conference in Computer Vision (ECCV), 2014.
 Fully convolutional networks for semantic segmentation
J. Long, E. Shelhammer and T. Darrell,
In Computer Vision and Pattern Recognition (CVPR), 2015.
 Semantic image segmentation with deep convolutional nets and fully connected CRFs
L.-C. Chen, G. Papandreou, I. Kokkinos et al.,
In International Conference on Learning Representations (ICLR), 2015.
 LabelBank: Revisiting Global Perspectives for Semantic Segmentation
H. Hu, Z. Deng, G.-T. Zhou et al.
In arXiv preprint arXiv:1703.09891, 2017.
COCO-Stuff is a derivative work of the COCO dataset. The authors of COCO do not in any form endorse this work. Different licenses apply:
- COCO annotations: Creative Commons Attribution 4.0 License
- COCO-Stuff annotations & code: Creative Commons Attribution 4.0 License
If you have any questions regarding this dataset, please contact us at holger-at-it-caesar.com.