layout | title | mathjax | tags | |||
---|---|---|---|---|---|---|
mathpost |
Camera-Robot Extrinsic Calibration |
true |
|
These days we are seeing a lot of research in perception for grasping and manipulation of objects with robot arms. For any of these to work, pixels in the camera images need to be connected to 3D points in the robot's task space.
Camera-Robot Extrinsic calibration solves this problem, and in this blog post I want to talk about the fun matrix algebra behind it. Hand-eye calibration and Eye-on-hand calibration are related terms.
- Solve
$AX=XB$ for hand-eye calibration - Invert the poses used to construct
$A$ for eye-on-hand calibration! - Additionally invert the poses used to construct
$B$ for more interesting information!
Let us consider the following coordinate systems:
b
: robot basee
: robot end-effectort
: calibration target e.g. checkerboard, ArUco marker(s), April tag(s)
Symbols denoting poses follow the
[usual]({{ site.baseurl }}{% post_url 2019-12-29-gtsam_conventions %})
convention:
B
w.r.t. frame A
.
Calibration target is rigidly attached to the robot end effector. A static camera observes the end-effector (and calibration target) motion.
- Known:
$^cT_t$ (from marker detection in the camera images) and$^bT_e$ (from the robot controller) - Unknown:
$^bT_c$ and$^eT_t$
We are interested in
Looking at the figure above, which shows two motions of the robot
((1)
and (2)
), we can identify the kinematic "cycle":
Note how cleanly the
Calibration target stationary in the world. The camera is rigidly attached to the robot end effector. The camera observes the calibration target from various perspectives through robot arm motion.
- Known:
$^cT_t$ (from marker detection in the camera images) and$^bT_e$ (from the robot controller) - Unknown:
$^eT_c$ and$^bT_t$
We are interested in
Similar to above, we identify a kinematic "cycle":
Note how the equation is almost the same as the hand-eye calibration equation above, you just need to invert the robot kinematics data!
Let us try to solve for the pose of the calibration target w.r.t robot end-effector,
So if you invert both robot kinematics and marker detection data in the hand-eye calibration equations, you get the pose of the calibration target w.r.t. robot end effector!
The same reasoning applies to the pose of the calibration target w.r.t. robot base for the eye-on-hand scenario.
The paper I linked above and the code linked below solve the "$AX=XB$" problem in the least-squares sense. You might want to use that solution as initialization for a nonlinear optimization problem, like this paper does. You can use GTSAM or the Ceres solver for that. Why would this improve the solution?
Instead of casting this problem as
In spite of being a required basic ingredient for robot grasping research, I was surprised to find that none of the available code worked out of the box. I looked at the following:
There are probably many other repositories but I settled on MoveIt! Calibration because it integrates marker tracking, solver, and robot motion control all in one place. It can record the joint states of your robot that were used for calibration, and then play them back to re-calibrate later if you move your camera. This is a very useful under-appreaciated feature.
Anyway, in this blog I wanted to explain the maths behind the "$AX=XB$" used under the hood of all these repositories, so that you can modify the code yourself if you need to. Also, it is fun to see how you can get completely different output quantities by inverting various input quantities!