Skip to content
Code for the paper "Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning" (ICRA 2018)
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Code for: Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning

This repository contains the code for our paper on generalizing spatial relations with end-to end metric learning, published at ICRA2018.


  • 24.06.2018 We won the ICRA2018 Best Paper Award in Robot Vision!


Install this repository (in development mode) using pip:

git clone
cd generalize_spatial_relations
pip install -e .

If you'd like to extend the code, I highly recommend to additionally install pandas as it reads in the point clouds in 3 seconds, while numpy takes over 30 seconds.

  • Download the dataset
  • Navigate to the relations dataset and create the point clouds from the .obj files (needs pcl): cd scripts; ./


To train all 15 splits run

CUDA_VISIBLE_DEVICES=0 python --logdir=$STORE_WEIGHTS_HERE --data_dir=$OBJECT_MODELS_ARE_HERE --more_augmentation=True

Adding the flag --more_augmentation applies stronger augmentation, i.e. clones a scene three times and applies stronger augmentation to the third clone. This leads to a better metric performance but might lead to less realistic generalizations.

To train a model on all the data add the flag --train_on_all_data.

Experiments (generalize relations)

To generalize relations from one scene to another, take a look at Here we picked a random subset of the scenes and use each scene as a reference to generalize the relation to all other scenes of this subset. The 3d visualization requires Mayavi to be installed and running it is extremely slow (~12 hours to generate all scenes.)

Integrate the model into your code

For a simple example of how you could use this model in your code, see SpatialRelationCNN/ Please note that the code currently runs on GPU only.

You can’t perform that action at this time.