3D Indoor furniture parsing. Segments the front face of a furniture item into more useful functional elements such as door, drawers and shelves.
C++ Makefile CMake Fortran Shell Python Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cli
evaluation_scripts
include
lib
oni2PNG
scripts
src
tests
CMakeLists.txt
README.md

README.md

The code is tested on Ubuntu 14.04.

Download RGB-D furniture models and their annotations from here.

Dependencies:

eigen, PCL, Gurobi, OpenCV

Compiling:

Following two commands to be run in the terminal from the build directory:

cmake -DCMAKE_BUILD_TYPE=Release ..
make

Note: change the required directory path in CMakeList file.

Training:

Train with all the images:

./bin/cli train ../data/depth/160/crossValidate/set4/train/

All the training models must be stored in the build directory before testing Each training can be separately enabled in parser.cpp

  1. trainEdgeDetector
  2. trainPartPrior
  3. trainAppearanceCodeBook
Parsing a single image:

for example to parse an image named '728.JPG' in the folder ../data/depth/160/crossValidate/set4/test

./bin/cli parse ../data/depth/160/crossValidate/set4/test/ 728

the resulting image will be stored with the name 'last_result.png' in the build directory.

Testing:

Test all the images:

'./bin/cli test ../data/depth/160/crossValidate/set4/test/'

The results will be saved in 'results' folder and the quantitative results in 'results.txt' in the build directory.

Parameter settings:

RDT(rectangleDetectionThreshold) and maxIOU are critical parameters for the segmentation.

  • RDT = 6 for WACV 17 synthetic dataset
  • RDT = 1 for WACV 17 real kinect dataset
  • RDT = 3 for WACV 16 dataset
  • maxIOU = 0.95

Note: RDT can be adjusted in parser.cpp and maxIOU can be adjusted in detector.h
Recommended: RDT = 1 for real images

More parameter options
  • DEBUG_MODE_ON: if 1, displays the debugging outputs in the command prompt

  • INCLUDE_DEPTH: if 1, includes depth information in the whole pipeline

  • DUMMY_MCMC_LOGIC: if 1, enables the 2 phase architecture and deactivtes the default architecture

  • ROI_CORRUPTION: if 1 enables corrpting the Region of interest bounding box, the extent of corruption (in pixels) can be specified using CORRUPT_PIXELS

  • SPLIT_MERGE_AUGMENT: if 1, enables proposal pool (split and merge) augmentation The split augment parameter can be specified using PROJ_PROF_THRESH and merging allowance (in pixels) can be specified using MERGE_ALLOWANCE

Initial probability of each (reversible) move can be specified using the following
  1. INIT_PROB_BIRTH
  2. INIT_PROB_DEATH
  3. INIT_PROB_SPLIT
  4. INIT_PROB_MERGE
  5. INIT_PROB_LABEL_DIFFUSE: switch move
  6. INIT_PROB_EXCHANGE_DATADRIVEN: data driven exchange
  7. INIT_PROB_UPDATE_CENTER
  8. INIT_PROB_UPDATE_WIDTH
  9. INIT_PROB_UPDATE_HEIGHT
  10. INIT_PROB_EXCHANGE_RANDOM: random exchange
Simulated Annealing Parameters (Default Architecture)
  1. MAX_TEMP // Starting temperature
  2. MIN_TEMP // ending temperature
  3. ALPHA // Geometric cooling parameter
  4. MAX_UPDATE_ITER // To check convergence
  5. NUM_INNER_LOOPS // iterations at each temperature
Simulated Annealing Parameters (2Phase Architecture)
# Phase 1 parameters
  1. MAX_TEMP_WARMUP
  2. MIN_TEMP_WARMUP
  3. ALPHA_WARMUP
  4. MAX_UPDATE_ITER_WARMUP
  5. NUM_INNER_LOOPS_WARMUP
# Phase 2 parameters
  1. MAX_TEMP_MASTER
  2. MIN_TEMP_MASTER
  3. ALPHA_MASTER
  4. MAX_UPDATE_ITER_MASTER
  5. NUM_INNER_LOOPS_MASTER
Sampling move indices
  1. EXCHANGE_MOVE_IDX
  2. BIRTH_MOVE_IDX
  3. DEATH_MOVE_IDX
  4. SPLIT_MOVE_IDX
  5. MERGE_MOVE_IDX
  6. LABEL_DIFFUSE_MOVE_IDX
  7. EXCHANGE_DD_MOVE_IDX
  8. UPDATE_CENTER_MOVE_IDX
  9. UPDATE_WIDTH_MOVE_IDX
  10. UPDATE_HEIGHT_MOVE_IDX
Diffusion in update moves
  1. WH_DIFFUSE_FACTOR : for diffusing width and height
  2. CENTER_DIFFUSE_FACTOR : for diffusing center location
Weighted sampling
  • ROULETTE_EXCHANGE: If 1, weighted sampling (russian roulette) is enabled
Redundancy removal IOU threshold
  • CLUSTER_MAX_IOU // measure to check similarity of rectangles in redundancy removal (proposal pool augmentation)
Upper Bound for appearance likelihood
  • numPartsCBUpperBound // Only if running in low specification systems
Maximum permitted Overlap above which penlising happens
  • MAX_OVERLAP
Augmentation in Appearance
  • APPEARANCE_AUGMENT: If this is true, then the parser generates debug output during the parsing process
Misc
  • VERBOSE_MODE: This is the size of the patches that are used for the edge detector
  • PATCH_SIZE: Steepness of Probabilistic SVM Sigmoid

Note on shape prior: Form factor was excluded in the evaluation of Proposed approach in 3d as 7 dimensions were enough. But for 2d, to strenthen the shape prior it was includedin the trained models.

Note on testing on real images: The current depth rectification is done using bilinear interpolation which works perfectly for synthetic images but not the best for real images especially for the ones with large amount of transformation. The authentic rectification is to implement the incomplete module using normal computation, clustering, gram schmidt orthonormalization and then cancelling the transformation based on the computed co-ordinate system. This section is incomplete.

To add noise to the synthetic images use use addKinectNoise.m in scripts folder To get real png images (RGB and depth) use oni2PNG.cpp