1.introduction

This is a method for calibrating and aligning RGBD cameras.

2.Environment

• ubuntu 22.04
• GNU Tools 11.x
• OpenCV 4.x
• Eigen 3.x
• Boost

3.Example

calib --> align

3.1 build

To satisfy the environment,run command

sudo apt install  libopencv-dev \
                  libeigen3-dev \
                  libboost-all-dev \
                  build-essential \
                  cmake \
                  git

adn then clone the repository locally

git clone https://github.com/5p6/d2c.git

If the environment are linux,you need to change some varable in the CMakeLists ,which include the fmt and opencv path,the

cd project_path

and

bash build.sh

So you could make use of the executable file to calib

3.2 calibration

Run using the command line,for rgb images calibration

./build/rgbd_calib --rgb-dir /mnt/hgfs/share/dataset/calib/rgbd/calib/calib/rgb/ --depth-dir /mnt/hgfs/share/dataset/calib/rgbd/calib/calib/infrared/ -h 6 -w 7 -o . -s 20

this program will calculate the parameters,which needed by align program,those parameters will saved in align.yaml,like

%YAML:1.0
---
rgbK: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 1.0330670202081678e+03, 0., 9.3046861250618576e+02, 0.,
       1.0310455772701362e+03, 5.3662586300152179e+02, 0., 0., 1. ]
depthK: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 3.6013258447643722e+02, 0., 2.6048421452847896e+02, 0.,
       3.5982078122427777e+02, 2.1293927876427088e+02, 0., 0., 1. ]
R: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 9.9976239492351493e-01, -7.7578583427777136e-03,
       2.0370796025952997e-02, 7.8386493541675473e-03,
       9.9996171422686042e-01, -3.8891717345808990e-03,
       -2.0339844470890121e-02, 4.0479271747460499e-03,
       9.9978492937855823e-01 ]
t: !!opencv-matrix
   rows: 3
   cols: 1
   dt: d
   data: [ 4.0526717379990224e+01, 3.5935658368717966e+00,
       -4.0260805147425174e+00 ]

you could use the --help option to check the help context,like

./build/rgbd_calib --help

3.3 align

Finally, the alignmentcommand is as follows

./build/rgbd_align -r /mnt/hgfs/share/dataset/calib/rgbd/calib/test/kehu/rgb.png -d /mnt/hgfs/share/dataset/calib/rgbd/calib/test/kehu/depth.png -p ./Register.yaml

align by inputting images, calibration parameters, and extrinsic data.This program will show the rgb image and aligned depth image.

Appendix

The goal of alignmentis to convert $(u_{ir},v_{ir})$ to $(u_{rgb},v_{rgb})$,as

$$\begin{array}{r}H(u_{ir},v_{ir})=(u_{rgb},v_{rgb})\end{array}$$

so we need to find H to satisfy the equation as (1).the specific principle of RGBD is introduced, and its mathematical formula is as follows.The projection equations for RGB cameras and depth cameras are as follows,rgb :

$$\begin{array}{r}\left[\begin{array}{c}{u}_{rgb}{v}_{rgb}1\end{array}\right]=\frac{1}{Z_{rgb}}{K}_{rgb}[R_{rgb}|t_{rgb}]\left[\begin{array}{c}{X}_w{Y}_w{Z}_{w}1\end{array}\right]\end{array}$$

ir :

$$\begin{array}{r}\left[\begin{array}{c}{u}_{ir}{v}_{ir}1\end{array}\right]=\frac{1}{Z_{ir}}{K}_{ir}[R_{ir}|t_{ir}]\left[\begin{array}{c}{X}_w{Y}_w{Z}_{w}1\end{array}\right]\end{array}$$

The mathematical formula above allows for simultaneous equations,Assuming $M_{rgb},M_{ir}$ is a three-dimensional point in the coordinate system of relative camera RGB and depth

$$\begin{array}{r}\{\begin{array}{l}{M}_{rgb}=[R_{rgb}|t_{rgb}]\left[\begin{array}{c}{X}_w{Y}_w{Z}_{w}1\end{array}\right]=R_{rgb}{M}_w+t_{rgb}\\ M_{ir}=[R_{ir}|t_{ir}]\left[\begin{array}{c}{X}_w{Y}_w{Z}_{w}1\end{array}\right]=R_{ir}{M}_w+t_{ir}\end{array}\end{array}$$

which

$$\begin{array}{r}{M}_w=\left[\begin{array}{c}{X}_w{Y}_w{Z}_w\end{array}\right],M_{rgb}=\left[\begin{array}{c}{X}_{rgb}{Y}_{rgb}{Z}_{rgb}\end{array}\right],M_{ir}=\left[\begin{array}{c}{X}_{ir}{Y}_{ir}{Z}_{ir}\end{array}\right]\end{array}$$

We can obtain the following equation from equation (3)

$$\begin{array}{r}{M}_{ir}=R_{ir}{R}_{rgb}^T{M}_{rgb}+t_{ir}-R_{ir}{R}_{rgb}^T{t}_{rgb}\end{array}$$

Convert the factors in the above equation into the following parameters

$$\begin{array}{r}\{\begin{array}{l}R=R_{ir}{R}_{rgb}^T\\ t=t_{ir}-R_{ir}{R}_{rgb}^T{t}_{rgb}\end{array}\end{array}$$

Or convert it into the following equation

$$\begin{array}{r}\{\begin{array}{l}{M}_{rgb}=R{M}_{ir}+t\\ R=R_{rgb}{R}_{ir}^T\\ t=t_{rgb}-R_{rgb}^T{R}_{ir}{t}_{ir}\end{array}\end{array}$$

then we could get $M_{ir}$ like

$$\begin{array}{r}{M}_{ir}=\left[\begin{array}{c}\frac{d(u_{ir}-c_x^{ir})}{f_x^{ir}}\frac{d(v_{ir}-c_y^{ir})}{f_y^{ir}}d\end{array}\right]\end{array}$$

and convert the $M_{ir}$ to $u_{rgb},v_{rgb}$,

$$\begin{array}{r}{u}_{rgb}=\frac{f_x^{rgb}}{Z_{rgb}}{X}_{rgb}+c_x^{rgb}\\ v_{rgb}=\frac{f_y^{rgb}}{Z_{rgb}}{Y}_{rgb}+c_y^{rgb}\end{array}$$

so we could convert $(u_{ir},v_{ir})$ to $(u_{rgb},v_{rgb})$.To conclude,we

• calibrate the rgb and ir camera,get data $K_{rgb,}{R}_{rgb},t_{rgb},K_{ir},R_{ir},t_{ir}$
• calculate the $R$ and $t$

$$\begin{array}{r}\{\begin{array}{l}R=R_{rgb}{R}_{ir}^T\\ t=t_{rgb}-R_{rgb}{R}_{ir}^T{t}_{ir}\end{array}\end{array}$$

• calculate the Point $M_{ir}=(X_{ir},Y_{ir},Z_{ir})$

$$\begin{array}{r}\{\begin{array}{l}{X}_{ir}=\frac{d(u_{ir}-c_x^{ir})}{f_x^{ir}}\\ Y_{ir}=\frac{d(v_{ir}-c_y^{ir})}{f_y^{ir}}\\ Z_{ir}=d\end{array}\to M_{ir}\end{array}$$

• calculate the Point $M_{rgb}=(X_{rgb},Y_{rgb},Z_{rgb})$

$$\begin{array}{r}{M}_{rgb}=R{M}_{ir}+t\end{array}$$

• Finally, calculate the coordinates $(u_{rgb},v_{rgb})$

$$\begin{array}{r}{u}_{rgb}=\frac{f_x^{rgb}}{Z_{rgb}}{X}_{rgb}+c_x^{rgb}\\ v_{rgb}=\frac{f_y^{rgb}}{Z_{rgb}}{Y}_{rgb}+c_y^{rgb}\end{array}$$

remeber that $Z_{rgb}\approx d$ .