Skip to content

Scan Annotation Pipeline

Jump to bottom
angelxuanchang edited this page Feb 26, 2018 · 4 revisions

The SSTK provides the annotation tools used for semantic segmentation labeling of the ScanNet and Matterport3D datasets. For ScanNet, we also provide an interface for aligning 3D CAD models to the reconstructed scan.

If you use either the semantic segmentation labeler or the scan to CAD model aligner, please cite:

@inproceedings{dai2017scannet,
    title={ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes},
    author={Dai, Angela and Chang, Angel X. and Savva, Manolis and Halber, Maciej and Funkhouser, Thomas and Nie{\ss}ner, Matthias},
    booktitle = {Proc. Computer Vision and Pattern Recognition (CVPR), IEEE},
    year = {2017}
}

An overview of the architecture of the annotation pipeline for ScanNet is shown below.

Preparing your assets for annotation

Please see Preparing assets for annotation for how to prepare your assets for annotation.

Below are some details of the input required for annotating scans.

Semantic segmentation of scans (3D Reconstructions)

The input is a reconstruction scan .ply that has been pre-segmented.

If you have a ply format mesh of the scan, use the ScanNet Segmentator to create a vertex-based segmentation using Felzenswalb and Huttenlocher's Graph Based Image Segmentation algorithm on computed mesh normals.

Once you have those, you can update the Basic metadata json example to point to them (see formats[0].path and surfaces.file). The asset should be ready for semantic segmentation annotation.

You also need to update server/proj/scannet/tasks/segment_annotation/default.config.yml to point to the assets you want to annotate by updating the following fields:

scansToAnnotate: <filename of your file with list of scans that you people to annotate>
idPrefix: <your asset source>

To test access, substitute for the modelId URL parameter with your assetId (<yourAssetName.assetId>) in this URL: http://localhost:8010/scans/segment-annotator-single?modelId=nyuv2.office_0001c

For exporting the semantic segmentation annotations with ssc/export-annotated-ply.js, the segment-annotations-manual field for the basic example specifies the following: segment-annotations-manual.files.segments for the over-segmentation and the segment-annotations-manual.files.segmentGroups for the annotated semantic segmentation.

Scan model alignment

For retrieving 3D CAD models and aligning them to your scan, you will first need to have a semantic segmentation. This can either be output from the manual semantic segmentation (above) or automatically generated segmentation. The semantic segmentation is specified in json format as described in Segment Annotation Formats.

Which semantic segmentation is used is specified as the 'segmentType' and segmentAnnotationId parameter to the scan model aligner. When the segmentAnnotationId is specified, it is taken directly from the database. The segmentation used is determined by the segmentType field. See Scan Model Aligner for more details on the scan model aligner UI.

A precomputed semantic segmentation can also be specified - this is also determined by the segmentType field (without any segmentAnnotationId specified). The precomputed semantic segmentation can be specified as two separate files (one for the initial over-segmentation, and one for the annotation) or as one combined file (see NYUv2 surfaces example).

The scan-model-alignments field for the basic example specifies the URLs for fetching list of annotated asset ids (annotatedAssetIds) and annotation ids for a given asset (annId) for exporting the scan model alignments.

Clone this wiki locally