Joint scene classification and semantic segmentation with FuseNet
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

FuseNet

[Caffe]

Joint scene classification and semantic segmentation using FuseNet architecture from FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture. Potential effects of additional scene classification loss on the overall semantic segmentation quality are tested.

Dependencies

Datasets

NYU

  1. Simply download the processed .h5py dataset with 40 annotations and 10 classes here: train + test set

  2. Scene mapping can be found here

SUNRGBD

  1. Download the dataset here

  2. Class weights are set in FuseNet.py. In case of an update on the training set, please adjust the weights, accordingly.

Training

  • To train Fusenet run Train_FuseNet.py. Dataset choice is manually implemented in the script for now. The dataset is taken and prepared by utils/data_utils_class.py, therefore make sure to give the correct path in the script.

  • Note: VGG weights are downloaded automatically at the beginning of the training process. Depth layers weights will also be initialized with their vgg16 equivalent layers. However, for 'conv1_1' the weights will be averaged to fit one channel depth input (3, 3, 3, 64) -> (3, 3, 1, 64)

Evaluation

Citing FuseNet

Caner Hazirbas, Lingni Ma, Csaba Domokos and Daniel Cremers, "FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture", in proceedings of the 13th Asian Conference on Computer Vision, 2016. (pdf)

@inproceedings{fusenet2016accv,
 author    = "C. Hazirbas and L. Ma and C. Domokos and D. Cremers",
 title     = "FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture",
 booktitle = "Asian Conference on Computer Vision",
 year      = "2016",
 month     = "November",
}