Corresponding implementations for the IROS 2018 paper "Modular Sensor Fusion for Semantic Segmentation"
Switch branches/tags
Nothing to show
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
experiments Merge branch 'publish' of… Jul 24, 2018
xview load a small part from testset for visualisations Jul 23, 2018
.flake8 update flake8 to linelength of 90 May 17, 2018
.gitignore also ignore notebooks Jul 16, 2018
Cityscapes.ipynb add more explanation to the notebooks Jul 25, 2018
Experimental Details.ipynb add more explanation to the notebooks Jul 25, 2018
Inference Time.ipynb add more explanation to the notebooks Jul 25, 2018
LICENSE add BSD 3clause license Jul 23, 2018 Merge branch 'publish' of… Jul 31, 2018
Synthia Rand Cityscapes Examples.ipynb add more explanation to the notebooks Jul 25, 2018
cityscapes.png add images for illustration Jul 23, 2018
requirements.txt add pyyaml as a requirement Jul 17, 2018

Corresponding implementations for the IROS 2018 paper "Modular Sensor Fusion for Semantic Segmentation" by Hermann Blum, Abel Gawel, Roland Siegwart and Cesar Cadena.

Fusion with Synthia RAND Fusion with Cityscapes
Reproduce Results Reproduce Results


Requires python 3

  1. optional but recommended steps: set up a virtual environment as follows Set up a new virtual environment: virtualenv --python=python3 <virtualenv-name> and activate it source <virtualenv-name>/bin/activate.

  2. clone the software into a new directory git clone and change into it cd modular_semantic_segmentation.

  3. Install all requirements and the repository software. pip install -r requirements.txt and pip install .

  4. Dependent on your hardware install tensorflow for CPU (pip install tensorflow) or GPU (pip install tensorflow-gpu).

  5. All following setup steps are shown in the jupyter notebooks: Synthia RAND, Cityscapes, Inference Time, Experimental Details


Reproducing Results

Reported Experiments can be reproduced in the following way:

python -m experiments.rerun with experiment_id=<insert here> -u

For all tables and examples in the paper, there are corresponging jupyter notebooks that show on which experiments they are based upon. You will find all the details behind the experiment using the notebook Experimental Details.


The package is devided into 3 parts:

  • xview/models: implementation of methods
  • xview/data: implementation of data interfaces
  • experiments: scripts that implement different functionalities for experiment, following the sacred framework.


Models are implemented following the sklearn interface, while context handling is necessary due to tensorflow semantics:

from xview.models.simple_fcn import SimpleFCN as FCN
from import Synthia
dataset = Synthia
# custom configs for the model
config = {'num_classes': 10,
            'dropout_probability': 0.2}
with FCN(data_description=dataset.get_data_description(), **config) as net:
    # Add data-loading to the graph.
    data = dataset(<config>)
    # Train the network for 10 iterations on data., 10)
    # Alternatively, load existing weigths.
    net.load_weights(<path to weights checkpoint>)
    net.import_weights(<path to npz file with stored weights>)
    # Now you can use it to produce classifications.
    semantic_map = net.predict({'rgb': <rgb image blob>, 'depth': <depth image blob>})
    evaluation = net.score(data.get_testset())

Any implementation of a model should inherit from, which implement basic sklean interfaces such as .fit(), .score(), .predict() etc. aswell as training procedures and graph building.

Model implementation should be split up into a method that defines a series of tensorflow-ops mapping input to output and a class inheriting from that handles all the functionality around this such as data piping etc. In this way, models can make use of the simple network definitions without building the whole model-object. See xview/models/ for an example.


Data interfaces return tf.dataset objects, which are documented here. Optionally, also numpy blobs can be produced (be careful with memory usage). All data interfaces inherit from xview/datasets/, which should be consulted for the interface details.


The experiments/ contains basic functions to interact with the sacred storage service, i.e. to load data from previous experiments and to store data from the current experiment.