Image Stitching

OpenCV based Image Stitching applied various method in C++

1. Prerequisites

1.1 Dependencies

OpenCV 4.8.0, OpenCV Contrib 4.8.0, C++ 11 version \

• If you already use OpenCV Version 3.x.x, then only change cv::SIFT to cv::xfeatures2d::SIFT !!

1.2 Ubuntu and ROS

Follow ROS Installation

• This code tested Ubuntu 64-bit 18.04 (ROS Melodic Version).

1.3. OpenCV Installation

Follow OpenCV

• Install appropriate OpenCV version: Here.

1.4 Intel RealSense ROS package and SDK

Follow realsense-ros

2. Get Custom Dataset

Follow KAIST-DP Dataset

• Intel Realsense D455 x3
• Xsens MTI-300 IMU
• Velodyne LiDAR VLP 16

3. Methodology to apply image stitching

• Reference Codes
  • Feature Extraction & Descriptor & Matcher & Outlier Rejection
    • Use this Github Page → Feature Matching
  • Parallax
    • Use this Github Page → Parallax

---

• Histogram Equalization

  • [1] Vanilla
  • [2] CHALE

• Feature Extraction

  • [1] ORB
  • [2] goodFeaturesToTrack
  • [3] SIFT

• Feature Descriptor

  • [2] DAISY

• Feature Macher

  • [1] Brute-Force Matcher (BF)

• Matching

  • [1] knn match (KNN)

• Outlier Rejection

  • [1] findFundamentalMatrix (FM)
  • [2] Parallax
  • [3] Optical Flow

• Stitching Base

  • [1] Camera Parameter Setting
  • [2] Image Masking
  • [3] Pre-check OpenCV Errors

• Fast Stitching Method

  • Feature Matcher → cv::detail::BestOf2NearestMatcher
  • Camera Estimator → cv::detail::HomographyBasedEstimator
  • Bundle Adjustment → cv::detail::BundleAdjusterRay
  • Warping → cv::SphericalWarper
  • Exposure → cv::detail::GainCompensator
  • Seam → cv::detail::DpSeamFinder
  • Blender → cv::detail::MultiBandBlender

• Direct Stitching Method based Homography (in warping.h, imgproc.h)

  • OpenCV based stitching → cv::warpPerspective
  • Not use OpenCV stitching → cv::findHomography

4 Changing Parameters

Parameters in "paramsetting.h"

#define SEE_MASKING
/* Visualize masking image (opencv_setting.h)
   Not see masking result -> 0 & See masking result -> 1 */

#define MASKING_REAL
/* Visualize masking in real image or white image (visualization.h)
   To see results in white image -> 0 & To see results in real image -> 1 */

#define SAVE_SYNC_IMG_WITH_TIME 
/* Visualize sync image with time (visualization.h)
   Not see sync result -> 0 & See sync result -> 1 */

#define SAVE_IMGS_WITH_FEATURES
/* Visualize image with features (visualization.h) 
   Not see image result with features -> 0 & See image result with features -> 1 */

#define SAVE_IMGS_DESCRIPTOR_MATCHING 
/* Visualize image with feature matching (visualization.h) 
   Not see image matching result -> 0 & See image matching result -> 1 */

#define RESIZE 
/* Resize result of panorama image (stitching.cpp)
   Not resize panorama image -> 0 & Resize panorama image -> 1 */

#define FLOW_BACK
/* Optical flow reverse check (parallax.h)
   Not use reverse check -> 0 & Use reverse check -> 1 */

#define HIST_EQUA 
/* Use Histogram Equalization (preprocessing.h)
   Not use histogram equalization -> 0 & Use histogram equalization -> 1 */

#define USE_CHALE 
/* Use CHALE Histogram Equalization (preprocessing.h) 
   Not use CHALE histogram equalization -> 0 & Use CHALE histogram equalization -> 1 */

#define FAST_STITCHING 
/* Use Fast Stitching (stitch.h)
   Not use fast stitching -> 0 & Use fast stitching -> 1 */

#define METHOD 
/* Debug OpenCV Error (error_filter.h)
   ORB Feature + BF Matching w/ Hamming distance -> 0
   SURF Feature + FLANN Matching w/ KNN -> 1
   SURF Features + FLANN Matching w/ KNN + Masking -> 2 */

#define MODE 
/* Stitching Mode (stitch.h)
   Vanilla -> 0
   Camera Intrinsic -> 1
   Camera Intrinsic + Masking -> 2
   Camera Intrinsic + Masking in estimateTransform function -> 3 */

5. Build Stitching_Image

Clone the repository and catkin_make:

    $ cd ~/catkin_ws/src
    $ git clone https://github.com/SungJaeShin/Image_Stitching.git
    $ cd ../
    $ catkin build
    $ source ~/catkin_ws/devel/setup.bash

6. Run two cameras

    $ roslaunch image_stitching multi_cam.launch
    $ rosrun image_stitching stitching_node
    $ (option) rviz 

7. Results

• Test Pair (EuRoC dataset)

  Img1	Img2

• Comparing Stitching Time !!

  Method	Auto Stitching	Fast Stitching API	Only Homography Stitching	Only Homography Stitching (not use OpenCV Warp API)
  Time (sec)	4.00	0.90	0.02	0.37

• Feature Matching using test pair

  SIFT features + DAISY descriptors + BF Matcher + KNN Matching

• OpenCV Stitching using test pair

  • Original stitching (About 4.0 Sec) → Graph Cut Seam Finding Method + Block Gain Compensator Method
  • Fast stitching (About 0.9 Sec) → Dynamic Programming Seam Finding Method + Gain Compensator Method

  Original Auto Stitching

  Fast Stitching

• Img warping using test pair (additionally using Interpolation method)

  Img2 warp using cv::warpPerspective	Img2 not using cv::warpPerspective

• Direct Stitching using test pair

  • OpenCV Warp API stitching (About 0.02 Sec)
  • Not use OpenCV Warp API stitching (About 0.37 Sec)

  Img1 + Img2 warp using cv::warpPerspective	Img1 + Img2 not using cv::warpPerspective

• Convert Plane image to polar image using OpenCV API

  Polar Img (cv::linearPolar)	Log Polar Img (cv::logPolar)

8. Save panorama image in your PC

Change eight variables "save_match1_dir, save_match2_dir, save_orb_dir, save_sync_dir, save_img1_dir, save_img2_dir, save_img3_dir, save_pano_dir" in this package !!

9. Reference

[1] https://github.com/SungJaeShin/Feature_matching.git
[2] https://github.com/SungJaeShin/Parallax.git
[3] https://github.com/SungJaeShin/KAIST-DP.git
[4] https://docs.opencv.org/3.4/dc/d6b/group__video__track.html#ga473e4b886d0bcc6b65831eb88ed93323
[5] https://docs.opencv.org/3.2.0/da/d54/group__imgproc__transform.html#ga5bb5a1fea74ea38e1a5445ca803ff121
[6] https://docs.opencv.org/4.x/d6/dc7/group__imgproc__hist.html#ga7e54091f0c937d49bf84152a16f76d6e
[7] https://docs.opencv.org/4.x/d6/dc7/group__imgproc__hist.html#gad689d2607b7b3889453804f414ab1018
[8] Y. Xiong and K. Pulli, "Fast panorama stitching for high-quality panoramic images on mobile phones," in IEEE Transactions on Consumer Electronics, vol. 56, no. 2, pp. 298-306, May 2010, doi: 10.1109/TCE.2010.5505931.