Co-Fusion CRF 图像分割 --> 对象 --> 跟踪

我们使用多模型拟合方法，其中每个对象可以独立于背景移动，
并且仍然被有效地跟踪，并且其形状随着时间的推移，使用来自与该对象标签相关联的像素的信息融合。
处理动态场景通常将移动区域视为异常值，因此不会对其形状进行建模或跟踪其随时间的运动。
相比之下，我们使机器人能够为每个分段对象维护3D模型，并通过融合随时间改进它们。
因此，我们的系统可以使机器人能够在对象层面上保持场景描述，
这有可能允许与其工作环境进行交互;即使在动态场景的情况下。

This repository contains Co-Fusion, a dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects.

Crucially, we use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers that are of no interest to the robot, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system has the benefit to enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.

To run Co-Fusion in real-time, you have to use our approach based no motion cues. If you prefer to use semantic cues for segmentation, please pre-process the segmentation in advance and feed the resulting segmentation masks into Co-Fusion.

More information and the paper can be found here.

If you would like to see a short video comparing ElasticFusion and Co-Fusion, click on the following image:

Publication

Please cite this publication, when using Co-Fusion (bibtex can be found on project webpage):

• Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects, Martin Rünz and Lourdes Agapito, 2017 IEEE International Conference on Robotics and Automation (ICRA)

Building Co-Fusion

The script Scripts/install.sh shows step-by-step how Co-Fusion is build. A python-based install script is also available, see Scripts\install.py.

Dataset and evaluation tools

Tools

• Recorder for klg files: https://github.com/mp3guy/Logger2
• Viewer for klg files: https://github.com/mp3guy/LogView
• Images -> klg converter: https://github.com/martinruenz/dataset-tools/tree/master/convert_imagesToKlg
• klg -> images/pointclouds: https://github.com/martinruenz/dataset-tools/tree/master/convert_klg
• Evaluate segmentation (intersection-over-union): https://github.com/martinruenz/dataset-tools/tree/master/evaluate_segmentation
• Automated evaluation of Co-Fusion: https://github.com/martinruenz/dataset-tools/tree/master/automatisation
• Scripts to create synthetic datasets with blender: https://github.com/martinruenz/dataset-tools/tree/master/blender

Synthetic sequences:

• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/car4-noise.klg
• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/car4-full.tar.gz
• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/room4-noise.klg
• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/room4-full.tar.gz

Real (Asus Xtion) sequences, in klg format:

• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/teddy-handover.klg
• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/sliding-clock.klg
• http://visual.cs.ucl.ac.uk/pubs/cofusion/data/place-items.klg

Hardware

In order to run Co-Fusion smoothly, you need a fast GPU with enough memory to store multiple models simultaneously. We used an Nvidia TitanX for most experiments, but also successfully tested Co-Fusion on a laptop computer with an Nvidia GeForce™ GTX 960M. If your GPU memory is limited, the COFUSION_NUM_SURFELS CMake option can help reduce the memory footprint per model. While the tracking stage of Co-Fusion calls for a fast GPU, the motion based segmentation performance depends on the CPU and accordingly, having a nice processor helps as well.

Reformatting code:

The code-formatting rules for this project are defined .clang-format. Run:

clang-format -i -style=file Core/**/*.cpp Core/**/*.h Core/**/*.hpp GUI/**/*.cpp GUI/**/*.h GUI/**/*.hpp

ElasticFusion

The overall architecture and terminal-interface of Co-Fusion is based on ElasticFusion and the ElasticFusion readme file contains further useful information.

New command line parameters (see source-file)

• -run: Run dataset immediately (otherwise start paused).
• -static: Disable multi-model fusion.
• -confO: Initial surfel confidence threshold for objects (default 0.01).
• -confG: Initial surfel confidence threshold for scene (default 10.00).
• -segMinNew: Min size of new object segments (relative to image size)
• -segMaxNew: Max size of new object segments (relative to image size)
• -offset: Offset between creating models
• -keep: Keep all models (even bad, deactivated)
• -dir: Processes a log-directory (Default: Color####.png + Depth####.exr [+ Mask####.png])
• -depthdir: Separate depth directory (==dir if not provided)
• -maskdir: Separate mask directory (==dir if not provided)
• -exportdir: Export results to this directory, otherwise not exported
• -basedir: Treat the above paths relative to this one (like depthdir = basedir + depthdir, default "")
• -colorprefix: Specify prefix of color files (=="" or =="Color" if not provided)
• -depthprefix: Specify prefix of depth files (=="" or =="Depth" if not provided)
• -maskprefix: Specify prefix of mask files (=="" or =="Mask" if not provided)
• -indexW: Number of digits of the indexes (==4 if not provided)
• -nm: Ignore Mask####.png images as soon as the provided frame was reached.
• -es: Export segmentation
• -ev: Export viewport images
• -el: Export label images
• -em: Export models (point-cloud)
• -en: Export normal images
• -ep: Export poses after finishing run (just before quitting if '-q')
• -or: Outlier rejection strength (default 3).