Keras-tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation(Unfinished)
Branch: master
Clone or download


Fully convolutional networks and semantic segmentation with Keras.

Biker Image

Biker Ground Truth

Biker as classified by AtrousFCN_Resnet50_16s


Models are found in, and include ResNet and DenseNet based models. AtrousFCN_Resnet50_16s is the current best performer, with pixel mean Intersection over Union mIoU 0.661076, and pixel accuracy around 0.9 on the augmented Pascal VOC2012 dataset detailed below.


Useful setup scripts for Ubuntu 14.04 and 16.04 can be found in the robotics_setup repository. First use that to install CUDA, TensorFlow,

mkdir -p ~/src

cd ~/src
# install dependencies
pip install pillow keras sacred

# fork of keras-contrib necessary for DenseNet based models
git clone -b densenet-atrous
cd keras-contrib
sudo python install

# Install python coco tools
cd ~/src
git clone
cd coco
sudo python install

cd ~/src
git clone


Datasets can be downloaded and configured in an automated fashion via the ahundt-keras branch on a fork of the tf_image_segmentation repository.

For simplicity, the instructions below assume all repositories are in ~/src/, and datasets are downloaded to ~/.keras/ by default.

cd ~/src
git clone -b Keras-FCN

Pascal VOC + Berkeley Data Augmentation

Pascal VOC 2012 augmented with Berkeley Semantic Contours is the primary dataset used for training Keras-FCN. Note that the default configuration maximizes the size of the dataset, and will not in a form that can be submitted to the pascal VOC2012 segmentation results leader board, details are below.

# Automated Pascal VOC Setup (recommended)
export PYTHONPATH=$PYTHONPATH:~/src/tf-image-segmentation
cd path/to/tf-image-segmentation/tf_image_segmentation/recipes/pascal_voc/
python pascal_voc_setup

This downloads and configures image/annotation filenames pairs train/val splits from combined Pascal VOC with train and validation split respectively that has image full filename/ annotation full filename pairs in each of the that were derived from PASCAL and PASCAL Berkeley Augmented dataset.

The datasets can be downloaded manually as follows:

# Manual Pascal VOC Download (not required)

    # original PASCAL VOC 2012
    wget # 2 GB
    # berkeley augmented Pascal VOC
    wget # 1.3 GB

The setup utility has three type of train/val splits(credit matconvnet-fcn):

Let BT, BV, PT, PV, and PX be the Berkeley training and validation
sets and PASCAL segmentation challenge training, validation, and
test sets. Let T, V, X the final trainig, validation, and test
Mode 1::
      V = PV (same validation set as PASCAL)
Mode 2:: (default))
      V = PV \ BT (PASCAL val set that is not a Berkeley training
Mode 3::
      V = PV \ (BV + BT)
In all cases:
      S = PT + PV + BT + BV
      X = PX  (the test set is uncahgend)
      T = (S \ V) \ X (the rest is training material)


MS COCO support is very experimental, contributions would be highly appreciated.

Note that there any pixel can have multiple classes, for example a pixel which is point on a cup on a table will be classified as both cup and table, but sometimes the z-ordering is wrong in the dataset. This means saving the classes as an image will result in very poor performance.

export PYTHONPATH=$PYTHONPATH:~/src/tf-image-segmentation
cd ~/src/tf-image-segmentation/tf_image_segmentation/recipes/mscoco

# Initial download is 13 GB
# Extracted 91 class segmentation encoding
# npy matrix files may require up to 1TB

python coco_setup
python coco_to_pascal_voc_imageset_txt
python coco_image_segmentation_stats

# Train on coco
cd ~/src/Keras-FCN

Training and testing

The default configuration trains and evaluates AtrousFCN_Resnet50_16s on pascal voc 2012 with berkeley data augmentation.

cd ~/src/Keras-FCN
cd utils

# Generate pretrained weights

cd ~/src/Keras-FCN

# Run training

# Evaluate the performance of the network

Model weights will be in ~/src/Keras-FCN/Models, along with saved image segmentation results from the validation dataset.

Key files

    • contains model definitions, you can use existing models or you can define your own one.
    • The training script. Most parameters are set in the main function, and data augmentation parameters are where SegDataGenerator is initialized, you may change them according to your needs.
    • Used for infering segmentation results. It can be directly run and it's also called in
    • Used for evaluating perforance. It will save all segmentation results as images and calculate IOU. Outputs are not perfectly formatted so you may need to look into the code to see the meaning.

Most parameters of,, and are set in the main function.