A Hand Motion-guided Articulation and Segmentation Estimation.

Articulation and segmentation estimation in RGB-D image (For Kinect v2) using hand motion.

Paper

Description

This repository offers the object articulation and segmentation estimation using human hand motion detected in RGB-D image sequence.

Requirement

Tested Environment: Windows 10

Dependency

• C++
  • OpenPose (https://github.com/CMU-Perceptual-Computing-Lab/openpose)
  • Kinect SDK V2 (https://www.microsoft.com/en-us/download/details.aspx?id=44561)
  • Point Cloud Library (http://www.pointclouds.org/)
  • Ceres solver (http://ceres-solver.org/)
• Python
  • Mask-RCNN (https://github.com/matterport/Mask_RCNN)
  • Hand Detection using TensorFlow (https://github.com/victordibia/handtracking)

Usage

• Capture (C++)
  • ArticulationDetection.exe --output_folder <capture_output_folder> --mode 0
  • push keys for capture background and manimulation scene (push 'b' once and 'c' twice)
    • c : start and stop recording
    • b : capture background
    • esc: quit program
• Human Masking (Python)
  • $ python masking.py </path/to/Mask_RCNN/> <capture_output_folder> <first_frame(int)> <last_frame(int)>
• Processing (C++)
  • ArticulationDetection.exe --model_folder </path/to/openpose/models> --input_folder <capture_output_folder> --output_folder <result_output_folder> --first_frame <first_frame(int)> --last_frame <last_frame(int)> --mode 1

Install

Dependency build

OpenPose

Clone repository

$ git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose

CMake & Build

Kinect SDK V2

Download SDK V2 (https://www.microsoft.com/en-us/download/details.aspx?id=44561) and install

Copy "FindKinectSDK2.cmake" in (https://gist.github.com/UnaNancyOwen/b7f8a543c3fa91a1a407) to CMake modules folder (C:/Program Files/CMake/share/cmake-version/Modules/)

Point Cloud Library

Download and Install All-in-One package (http://unanancyowen.com/en/pcl181/) (VS 2015 or 2017)

Ceres Solver

Follow the installation instruction (http://ceres-solver.org/installation.html).

CMake without GFLAGS and GLOG (check MINIGLOG and uncheck GFLAGS) to avoid to refer to duplicate libraries with OpenPose (OpenPose compiled in Windows refers to internal GFLAGS and GLOG).

Build "INSTALL" on Visual Studio. Install folder can be modified by configuring CMAKE_INSTALL_PREFIX.

Build this source

Clone this repository

$git clone https://github.com/cln515/articulation-detection

Set source code folder and build folder

Run "configure"

Set OpenPose repository folder to OpenPose_DIR

Set openpose.lib (/path/to/build_folder/src/openpose/Release/openpose.lib) to OpenPose_LIB

Run "configure" again and "Generate"

Licence

MIT