Train DeepLab for Semantic Image Segmentation
Branch: master
Clone or download
Latest commit 509419a Feb 18, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
DeepLab-LargeFOV-Semi-Bbox-Fixed modified names of local variables Apr 7, 2016
DeepLab-LargeFOV defaultly using DENSE_CRF layer Apr 19, 2016
code @ c9b5f64 wrong comment bugfix May 9, 2016
exper/voc12/list/original lists with original VOC 2012 training and validating data Mar 10, 2016
py_img_seg_eval @ c0bf978
.gitignore added .gitignore Apr 12, 2016
.gitmodules tools for evaluating trained net Mar 18, 2016
LICENSE MIT license Feb 19, 2017 tools for evaluating trained net Mar 18, 2016 updated paragraph about visual results Apr 19, 2016 convert ground truth labels from RGB notation to one-dimensional Mar 18, 2016
convert_weak_labels.m script for creating weak label lists Apr 7, 2016 evaluate deeplab from matlab output files Mar 24, 2016
faster_grep changed name of fast grep to faster grep Apr 12, 2016 added new parameter to script; given txt file is used for storing sub… Mar 25, 2016 apply for both datasets, PASCAL VOC 2012 and PASCAL VOC augmented Mar 25, 2016 solver and net descriptions of DeepLab_LargeFOV_Semi_Bbox_EM_Fixed Apr 7, 2016 store training and testing model descriptions and solver prototxt fil… Apr 5, 2016 clear any comment that exists within image Mar 22, 2016 changed number of mandatory script parameters May 9, 2016 typo in help string Mar 17, 2016 typo in name Apr 14, 2016 added comments Apr 5, 2016 fix integrity of settings for subset training Apr 5, 2016 applied faster grep Apr 14, 2016 defaultly using DENSE_CRF layer Apr 19, 2016
sub.sed modified names of local variables Apr 7, 2016 modified for usage of DENSE_CRF layer and added new parameters of script Apr 19, 2016 Fast version of convert_from_color_segmentation Jan 17, 2017

Train DeepLab for Semantic Image Segmentation

Martin Kersner,

This repository contains scripts for training DeepLab for Semantic Image Segmentation using strongly and weakly annotated data. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs and Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation papers describe training procedure using strongly and weakly annotated data, respectively.

git clone --recursive 

In following tutorial we use couple of shell variables in order to reproduce the same results without any obtacles.

  • $DEEPLAB denotes the main directory where repository is checked out
  • $DATASETS denotes path to directory where all necessary datasets are stored
  • $LOGNAME denotes name of log file stored in $DEEPLAB/exper/voc12/log directory
  • $DOWNLOADS denotes directory where downloaded files are stored


Install DeepLab caffe

You should follow instructions for installation. However, if you have already fulfilled all necessary dependencies running following commands from code/ directory should do the job.

cd $DEEPLAB/code
cp Makefile.config.example Makefile.config
# Adjust Makefile.config (for example, if using Anaconda Python, or if cuDNN is desired)
make all
make pycaffe
make test # NOT mandatory
make runtest # NOT mandatory

Compile DenseCRF

Go to $DEEPLAB/code/densecrf directory, modify Makefile if necessary and run make command. Or you can run following commands in sequential order.

cd $DEEPLAB/code/densecrf
# Adjust Makefile if necessary

Strong annotations

In this part of tutorial we train DCNN for semantic image segmentation using PASCAL VOC dataset with all 21 classes and also with limited number of them. As a training data we use only strong annotations (pixel level labels).


All necessary data for training are listed in $DEEPLAB/exper/voc12/list/original. Training scripts are prepared to employ either PASCAL VOC 2012 dataset or augmented PASCAL VOC dataset which contains more images.

# augmented PASCAL VOC
wget # 1.3 GB
tar -zxvf benchmark.tgz
mv benchmark_RELEASE VOC_aug

# original PASCAL VOC 2012
wget # 2 GB
tar -xvf VOCtrainval_11-May-2012.tar
mv VOCdevkit/VOC2012 VOC2012_orig && rm -r VOCdevkit

Data conversions

Unfortunately, ground truth labels within augmented PASCAL VOC dataset are distributed as Matlab data files, therefore we will have to convert them before we can start training itself.

cd $DATASETS/VOC_aug/dataset
mkdir cls_png
./ $DATASETS/VOC_aug/dataset/cls $DATASETS/VOC_aug/dataset/cls_png

Caffe softmax loss function can accept only one-channel ground truth labels. However, those labels in original PASCAL VOC 2012 dataset are defined as RGB images. Thus, we have to reduce their dimensionality.

cd $DATASETS/VOC2012_orig
mkdir SegmentationClass_1D

./ $DATASETS/VOC2012_orig/SegmentationClass/ \
  $DATASETS/VOC2012_orig/ImageSets/Segmentation/trainval.txt \

At last, part of code which computes DenseCRF is able to work only with PPM image files, hence we have to perform another conversion. This step is necessary only if we want to use DenseCRF separately and as one of Caffe layers.


# augmented PASCAL VOC
mkdir $DATASETS/VOC_aug/dataset/img_ppm
./ $DATASETS/VOC_aug/dataset/img $DATASETS/VOC_aug/dataset/img_ppm

# original PASCAL VOC 2012
mkdir $DATASETS/VOC2012_orig/PPMImages
./ $DATASETS/VOC2012_orig/JPEGImages $DATASETS/VOC2012_orig/PPMImages


Then we create symbolic links to training images and ground truth labels.

mkdir -p $DEEPLAB/exper/voc12/data
cd $DEEPLAB/exper/voc12/data

# augmented PASCAL VOC
ln -s $DATASETS/VOC_aug/dataset/img images_aug
ln -s $DATASETS/VOC_aug/dataset/cls_png labels_aug
ln -s $DATASETS/VOC_aug/dataset/img_ppm images_aug_ppm

# original PASCAL VOC 2012
ln -s $DATASETS/VOC2012_orig/JPEGImages images_orig
ln -s $DATASETS/VOC2012_orig/SegmentationClass_1D labels_orig
ln -s $DATASETS/VOC2012_orig/PPMImages images_orig_ppm

Download necessary files for training

Before the first training we have to download several files. Using the command below we download initialization model, definition its network and solver. It will also setup symbolic links in directories where those files are later expected during training.


In order to easily switch between datasets we will modify image lists appropriately.


Training with all classes can go through 4 different phases (twice training, twice testing), but I wouldn't recommend to run testing phases using this script. Actually, they are currently disabled. At lines 27 through 30, any of phase can be enabled (value 1) or disabled (value 0).

Finally, we can start training.


Plotting training information

Training script generates information which are printed to terminal and also stored in $DEEPLAB/exper/voc12/log/DeepLab-LargeFOV/ directory. For every printed iteration there are displayed loss and three different model evalutation metrics for currently employed batch. They denote pixel accuracy, average recall and average Jacard index, respectively. Even though those values are retrievd from training data, they possess important information about training and using the script below we can plot them as a graph. The script generates two graphs evaluation.png and loss.png.

./ exper/voc12/log/DeepLab-LargeFOV/`ls -t exper/voc12/log/DeepLab-LargeFOV/ | head -n 1` # for the newest log
#./ exper/voc12/log/DeepLab-LargeFOV/$LOGNAME # specified log 

Training with only 3 classes

If we want to train with limited number of classes we have to modify ground truth labels and also list of images that can be exploited for training. In at line 17 are specified classes that we are interested in (defaultly bird, bottle and chair).

# augmented PASCAL VOC 
mkdir -p $DATASETS/VOC_aug/dataset/cls_sub_png
cd $DEEPLAB/exper/voc12/data/
ln -s $DATASETS/VOC_aug/dataset/cls_sub_png labels_sub_aug
find exper/voc12/data/labels_aug/ -printf '%f\n' | sed 's/\.png//'  | tail -n +2 > all_aug_data.txt
python $DATASETS/VOC_aug/dataset/cls_png/ $DATASETS/VOC_aug/dataset/cls_sub_png/ all_aug_data.txt sub_aug_data.txt

# original PASCAL VOC 2012 
mkdir -p $DATASETS/VOC2012_orig/SegmentationClass_sub_1D
cd $DEEPLAB/exper/voc12/data/
ln -s $DATASETS/VOC2012_orig/SegmentationClass_sub_1D labels_sub_orig
find exper/voc12/data/labels_orig/ -printf '%f\n' | sed 's/\.png//'  | tail -n +2 > all_orig_data.txt
python $DATASETS/VOC2012_orig/SegmentationClass_1D/ $DATASETS/VOC2012_orig/SegmentationClass_sub_1D/ all_orig_data.txt sub_orig_data.txt


The number of classes that we plan to use is set at lines 13 and 14 in This number should be always higher by 1 than number of specified classes in script, because we also consider background as one of classes.

After, we can proceed to training.


We can also use the same script for plotting training information.


phase 1 (24,000 iter., no CRF) phase 2 (12,000 iter., no CRF)
pixel accuracy 0.8315 0.8523
mean accuracy 0.6807 0.6987
mean IU 0.6725 0.6937
frequency weighted IU 0.8182 0.8439

Visual results

Employed model was trained without CRF in phase 1 (24,000 iterations) and then in phase 2 (12,000 iterations), but results here exploited DENSE_CRF layer. Displayed images (bird: 2010_004994, bottle: 2007_000346, chair: 2008_000673) are part of validation dataset stored in $DEEPLAB/exper/voc12/list_subset/val.txt. Colors of segments differ from original ground truth labels because employed model was trained only for 3 classes + background.

Weak annotations

In a case we don't possess enough training data, weakly annotated ground truth labels can be exploited using DeepLab.


At first you should download and from link and run commands below to prepare data for use.

mv $DATASETS/VOC_aug/dataset/
mv $DATASETS/VOC_aug/dataset/

cd $DATASETS/VOC_aug/dataset

mv SegmentationClassBboxAug_Visualization/ SegClassBboxAug_RGB
mv SegmentationClassBboxErode20CRFAug_Visualization/ SegClassBboxErode20CRFAug_RGB

Downloaded weak annotations were created using Matlab and because of that labels are sometimes stored with one channel and other times with three channels. Similarly to strong annotations we have to convert all labels to the same one channel format. In order to cope with it I recommend you to use Matlab script convert_weak_labels.m (if anybody knows how to perform the same conversion using python I would be really interested) which is stored in $DEEPLAB directory. Before running script you have to specify path to datasets on line 3.

After script successfully finished we have to create symbolic links to be able to reach data during training.

cd $DEEPLAB/exper/voc12/data
ln -s $DATASETS/VOC_aug/dataset/SegClassBboxAug_1D/ labels_bbox
ln -s $DATASETS/VOC_aug/dataset/SegClassBboxErode20CRFAug_1D/ labels_bboxcrf

Create subsets

Training DeepLab using weak labels enables to employ datasets of different sizes. Following snippet creates those subsets of strong dataset and also necessary training lists with weak labels.


cd $DEEPLAB/exper/voc12/list
head -n 200  train.txt > train200.txt
head -n 500  train.txt > train500.txt
head -n 750  train.txt > train750.txt
head -n 1000 train.txt > train1000.txt

cp train_bboxcrf.txt trainval_bboxcrf.txt
cp train_bbox.txt trainval_bbox.txt


Training using weak annotations is similar to exploiting strong annotations. The only difference is the name of script which should be run.


Plotting is also same as for strong annotations.


5000 weak annotations and 200 strong annotations

phase 1 (6,000 iter., no CRF) phase 2 (8,000 iter., no CRF)
pixel accuracy 0.8688 0.8671
mean accuracy 0.7415 0.750
mean IU 0.6324 0.6343
frequency weighted IU 0.7962 0.7951


Init models are modified VGG-16 networks with changed kernel size from 7x7 to 4x4 or 3x3. There are two models that can be employed for initialization: vgg16_128, vgg16_20M.

The first fully connected layer of vgg16_128 has kernel size 4x4 and 4096 filters. It can be used for DeepLab basic model. In vgg16_20M, the first fully connected layer has kernel size 3x3 and 1024 filters. It can be used for DeepLab-LargeFOV.

Currently training is focused on DeepLab-LargeFOV.


At you can find frequently asked questions about DeepLab for semantic image segmentation.