Pose Accuracy and Stability - Calibration

[JBBee]

---

Required Info
Camera Model	T265
Firmware Version	0.2.0.857
Operating System & Version	Win 10
Kernel Version (Linux Only)
Platform	PC
SDK Version	2.30.0
Language	C#
Segment	AR

Issue Description

Our AR application using the T265 overlays virtual objects on top of real world objects and we're having trouble with the virtual objects staying in place on top of the real objects. Apologies for the long post, but we're trying to see if there's anything we can do to improve on this issue.

During some runs the virtual object stays in place very well and during others it moves off the real world object a noticeable amount as we move around it. This is similar to #4789 #5207 #4368. We conducted some tests to try and characterize the accuracy and stability of the pose from the T265, and also turned dynamic calibration on/off and modified the extrinsics distance between the fisheyes using the calibration write api #4650 to try and achieve a more accurate pose. Each test run was with a freshly plugged in module, we then walk around the room twice, and then put it on a cart and wheel it exactly 5m in a straight line and measure the straight line distance between the start and end points. There's no sunlight in the room and the lighting and furniture (except for the cart) were kept static for all the runs. Below are the calibration values and the results

---

1. Factory calibration

a. Dynamic Calibration on

Run	T265 Distance	% error
1	4.734	5.4
2	4.93	1.4
3	4.85	3
4	4.90	2
5	4.79	4.2

b. Dynamic Calibration off

Run	T265 Distance	% error
1	4.41	11.8
2	4.93	1.4
3	4.85	3
4	4.90	2
5	4.94	1.2

---

2. Setting the fisheye extrinsics translation x value to 0.0658

a. Dynamic Calibration on

Run	T265 Distance	% error
1	4.99	.2
2	5.07	1.4
3	4.91	1.8
4	5.16	3.2
5	4.77	4.6

b. Dynamic Calibration off

Run	T265 Distance	% error
1	4.96	.8
2	4.99	.2
3	5.11	2.2
4	4.70	6
5	5.05	1

---

Number of runs with % error <= 2.5

Run	Number
1a	2
1b	3
2a	3
2b	4

So the number of runs where our application will look ok is highest with the calibration for 2b, that is with dynamic calibration off and the fisheye baseline adjusted high.

I understand the SLAM algorithm is always being improved for accuracy and stability, but I wonder if making calibration adjustments could help us in the near term.

• Do you see any issue with setting an artificially high distance between the fisheyes?
• Are there any other calibration adjustments we could make which could help us in this one environment?
• Is there a process to allow us to re-calibrate the IMU?
• Is there anything we can do to better stabilize the % error?

Also below are the factory and modified calibrations.

Thanks.

Factory

Width:        848
  Height:       800
  PPX:          430.163909912109
  PPY:          400.544311523438
  Fx:           284.657196044922
  Fy:           285.681304931641
  Distortion:   Kannala Brandt4
  Coeffs:       -0.000986896338872612   0.03599027171731        -0.0350222513079643     0.00552535010501742     0

  Width:        848
  Height:       800
  PPX:          429.614013671875
  PPY:          400.963409423828
  Fx:           285.275787353516
  Fy:           286.322204589844
  Distortion:   Kannala Brandt4
  Coeffs:       -0.00297739310190082    0.0402248986065388      -0.0377833284437656     0.00628961203619838     0

Bias Variances:         0.000000499999999  0.000000499999999  0.000000499999999
Noise Variances:        0.000005148030141  0.000005148030141  0.000005148030141
Sensitivity :
     0.993849       0.000000       0.000000       0.002005
     0.000000       0.997440       0.000000      -0.000646
     0.000000       0.000000       0.997373       0.000655


Bias Variances:         0.000099999997474  0.000099999997474  0.000099999997474
Noise Variances:        0.000066952452471  0.000066952452471  0.000066952452471
Sensitivity :
     1.010260       0.000000       0.000000      -0.061002
     0.000000       1.022954       0.000000       0.251849
     0.000000       0.000000       1.017220      -0.012597


Fisheye 1 to Fisheye 2 extrinsics :
 Rotation Matrix:
   0.999976         0.000451783     -0.00688393
  -0.000427367      0.999994         0.0035479
   0.00688548      -0.00354487       0.99997

 Translation Vector: -0.0646408051252365  0.000127978943055496  -9.49058885453269e-05

Fisheye 2 to Fisheye 1 extrinsics :
 Rotation Matrix:
   0.999976        -0.000427367      0.00688548
   0.000451783      0.999994        -0.00354487
  -0.00688393       0.0035479        0.99997

 Translation Vector: 0.0646399706602097  -9.91109918686561e-05  -0.000350533431628719

Modified

 Width:        848
  Height:       800
  PPX:          430.163909912109
  PPY:          400.544311523438
  Fx:           284.657196044922
  Fy:           285.681304931641
  Distortion:   Kannala Brandt4
  Coeffs:       -0.000986896338872612   0.03599027171731        -0.0350222513079643     0.00552535010501742     0

  Width:        848
  Height:       800
  PPX:          429.614013671875
  PPY:          400.963409423828
  Fx:           285.275787353516
  Fy:           286.322204589844
  Distortion:   Kannala Brandt4
  Coeffs:       -0.00297739310190082    0.0402248986065388      -0.0377833284437656     0.00628961203619838     0

Bias Variances:         0.000000499999999  0.000000499999999  0.000000499999999
Noise Variances:        0.000005148030141  0.000005148030141  0.000005148030141
Sensitivity :
     0.993849       0.000000       0.000000       0.002005
     0.000000       0.997440       0.000000      -0.000646
     0.000000       0.000000       0.997373       0.000655


Bias Variances:         0.000099999997474  0.000099999997474  0.000099999997474
Noise Variances:        0.000066952452471  0.000066952452471  0.000066952452471
Sensitivity :
     1.010260       0.000000       0.000000      -0.061002
     0.000000       1.022954       0.000000       0.251849
     0.000000       0.000000       1.017220      -0.012597


Fisheye 1 to Fisheye 2 extrinsics :
 Rotation Matrix:
   0.999976         0.000451782     -0.00688393
  -0.000427368      0.999994         0.0035479
   0.00688548      -0.00354487       0.99997

 Translation Vector: -0.0657961964607239  -0.000303193868603557  0.000452485022833571

Fisheye 2 to Fisheye 1 extrinsics :
 Rotation Matrix:
   0.999976        -0.000427368      0.00688548
   0.000451782      0.999994        -0.00354487
  -0.00688393       0.0035479        0.99997

 Translation Vector: 0.0657913759350777  0.000334521551849321  -0.000904331507626921

[radfordi]

Thanks for the data, @JBBee!

One quick question. Are you moving the device around before starting to take measurements? We highly recommend doing this.

[JBBee]

Sure @radfordi

We are moving the device around before taking measurements, we walk around the room twice before we put it on the cart to wheel it the 5m.

Thanks.

[JBBee]

As another data point: We continued to collect data with a second T265 and found that modifying the fisheye baseline to be lower than 64mm improved tracking on this second module.

[radfordi]

Thanks for the update, @JBBee. We are looking into your results and double checking our calibration procedure. Still, the path length variances you are seeing are larger than we've seen in our own testing. Would you be willing up make a recording (made with rs-record) available so we can see if maybe the device is having issues with something in your environment?

[dmirota]

@JBBee Could you provide the calibration of the second module?

[JBBee]

@radfordi , thanks. I'd be happy to provide some data: I did two recordings: One with the factory calibration and dynamic calibration on, and the second with the modified calibration and dynamic calibration off. I pause when placing it on the cart and also after pushing the cart so you can see the pose at the start and end of the 5m push. The straight line distance in gray_0p0658_no_dyn_calibration.bag is 5.04 and in gray_default_dyn_calibration_on.bag it's 4.64. You can grab the zipped .bags from this repository (which uses LFS): https://github.com/JBBee/IntelCalibration

@dmirota , the above recordings and calibrations are from our first rig which we're calling the gray rig. The second rig we have is the white rig and the factory and modified calibrations are below. On the white rig we saw the best results with the modified calibration below but with leaving dynamic calibration on.

Thanks again for the help.

White Factory

  Width:        848
  Height:       800
  PPX:          427.819610595703
  PPY:          405.811096191406
  Fx:           284.770111083984
  Fy:           285.836486816406
  Distortion:   Kannala Brandt4
  Coeffs:       -0.000712347915396094   0.0346902199089527      -0.0326500199735165     0.00464539602398872     0

  Width:        848
  Height:       800
  PPX:          427.181396484375
  PPY:          405.127685546875
  Fx:           285.041809082031
  Fy:           286.288299560547
  Distortion:   Kannala Brandt4
  Coeffs:       -0.0037984331138432     0.0436774492263794      -0.0422733016312122     0.00794548634439707     0

Bias Variances:         0.000000499999999  0.000000499999999  0.000000499999999
Noise Variances:        0.000005148030141  0.000005148030141  0.000005148030141
Sensitivity :
     0.993775       0.000000       0.000000       0.003035
     0.000000       0.998084       0.000000       0.002715
     0.000000       0.000000       1.000528      -0.000439


Bias Variances:         0.000099999997474  0.000099999997474  0.000099999997474
Noise Variances:        0.000066952452471  0.000066952452471  0.000066952452471
Sensitivity :
     1.010423       0.000000       0.000000      -0.009304
     0.000000       1.022351       0.000000       0.085985
     0.000000       0.000000       1.017702      -0.037644


Fisheye 1 to Fisheye 2 extrinsics :
 Rotation Matrix:
   0.999973        -0.00502772      -0.00534856
   0.00500874       0.999981        -0.0035558
   0.00536634       0.00352891       0.999979

 Translation Vector: -0.0642008781433105  0.000167761987540871  -0.000179022434167564

Fisheye 2 to Fisheye 1 extrinsics :
 Rotation Matrix:
   0.999973         0.00500874       0.00536634
  -0.00502772       0.999981         0.00352891
  -0.00534856      -0.0035558        0.999979

 Translation Vector: 0.064199261367321  -0.000489911122713238  -0.000163766962941736

White modified

  Width:        848
  Height:       800
  PPX:          427.819610595703
  PPY:          405.811096191406
  Fx:           284.770111083984
  Fy:           285.836486816406
  Distortion:   Kannala Brandt4
  Coeffs:       -0.000712347915396094   0.0346902199089527      -0.0326500199735165     0.00464539602398872     0

  Width:        848
  Height:       800
  PPX:          427.181396484375
  PPY:          405.127685546875
  Fx:           285.041809082031
  Fy:           286.288299560547
  Distortion:   Kannala Brandt4
  Coeffs:       -0.0037984331138432     0.0436774492263794      -0.0422733016312122     0.00794548634439707     0

Bias Variances:         0.000000499999999  0.000000499999999  0.000000499999999
Noise Variances:        0.000005148030141  0.000005148030141  0.000005148030141
Sensitivity :
     0.993775       0.000000       0.000000       0.003035
     0.000000       0.998084       0.000000       0.002715
     0.000000       0.000000       1.000528      -0.000439


Bias Variances:         0.000099999997474  0.000099999997474  0.000099999997474
Noise Variances:        0.000066952452471  0.000066952452471  0.000066952452471
Sensitivity :
     1.010423       0.000000       0.000000      -0.009304
     0.000000       1.022351       0.000000       0.085985
     0.000000       0.000000       1.017702      -0.037644


Fisheye 1 to Fisheye 2 extrinsics :
 Rotation Matrix:
   0.999973        -0.00502772      -0.00534856
   0.00500874       0.999981        -0.0035558
   0.00536633       0.00352891       0.999979

 Translation Vector: -0.0634965300559998  -0.000201145900064148  0.000602322979830205

Fisheye 2 to Fisheye 1 extrinsics :
 Rotation Matrix:
   0.999973         0.00500874       0.00536633
  -0.00502772       0.999981         0.00352891
  -0.00534856      -0.0035558        0.999979

 Translation Vector: 0.0634925961494446  -0.000120226119179279  -0.000942640705034137

[dmirota]

@JBBee Thank you for your prompt reply and for sharing the additional data. We will have a closer look and be in touch.

[dmirota]

@JBBee I was reviewing your data and I wanted to make sure I take the same measurement you are for the 5m segment. At beginning of the recordings, I see a tour around the table, the pause, and then the 5m motion on a cart. How did you select the starting point of the 5m motion? Is it simply one of the poses during the pause? Thanks.

[JBBee]

@dmirota Yes, here are the times in each bag file that I pause and write down the pose translation

gray_0p0658_no_dyn_calibration.bag

5m start	5m end
0:00:54	0:01:14

gray_default_dyn_calibration_on.bag

5m start	5m end
0:00:57	0:01:17

For each video I just calculate the distance between the starting and ending pose translations.

If you need anything else just let me know.

Thanks.

[dmirota]

Hi, @JBBee,
Thank you for the additional details. I have a new calibration that should provide an improvement in accuracy for the t265 you call gray. Attached is a python file update_calibration.py.txt.
The script uses Python 3.7 and the Python module pyrealsense2 to update the t265 calibration.
The Python module can be installed on most platforms using pip install pyrealsense2.
Rename the file to .py and call python update_calibration.py --write --confirm to update the calibration.
To restore the factory calibration call python update_calibration.py --reset

[JBBee]

Hi, @dmirota

Thank you, I followed the instructions and updated the calibration on our gray rig using the python script referenced above. Unfortunately though I'm seeing low distance measurements for the 5m test. I recorded bag files using this calibration with dynamic calibration both on and off. They're located here:
gray_intelcalibration_dyn_cal_off.zip
gray_intelcalibration_dyn_cal_on.zip

The straight line distances are:

File	distance
gray_intelcalibration_dyn_cal_off	4.42
gray_intelcalibration_dyn_cal_on	4.60

If you want to take a look at them the relevant times for the 5m start/end are as follows:

File	5m Start	5m End
gray_intelcalibration_dyn_cal_off	00:27	00:46
gray_intelcalibration_dyn_cal_on	00:30	00:46

Inside the zips is a dump of the calibration written by your python script to confirm.

We really appreciate the help, thanks again!

[dmirota]

@JBBee Thanks for your quick response. Just to confirm. Are you still using librealsense 2.30.0 and firmware 0.2.0.857?

[JBBee]

Sure @dmirota , actually I'm using 2.31.0 with firmware 0.2.0.879. Should I try the test with 2.30.0?

[dmirota]

No need to change back just yet. I just want to make sure my test environment is matching yours. Thanks.

[dmirota]

@JBBee How are you selecting the translation values to compute the length?

[JBBee]

@dmirota I just pause the bag file in the viewer when it's settled on the cart and write down the pose translation and then after it's moved the 5m and it's settled again I pause and do the same. Then I just calculate the distance between the two points.

For example, in gray_intelcalibration_dyn_cal_on I pause at 00:00:30 and write down (-1.705, 0.082, 1.058) and then I hit play until 00:00:46 when I pause again and write down (2.892, 0.018, 0.888). And the distance between those two points is 4.60m.

Do you see any issues with doing it this way?

[dmirota]

No, there should not be an issue with that. I just want to make sure I'm following all of the same steps you are.

[JBBee]

Hi @dmirota

Just checking in to see if there might be any new developments, or if there's anything else we can provide to help.

Thanks.

[dmirota]

Hi, @JBBee

Thanks for checking in. I found an inconsistency in how the calibration data is being updated and we are still investigating it further. I will provide further updates when available. We appreciate your help and I will let you know when we have an update for you or if I could use further data.

Thanks again,
Dan

[RealSenseSupport]

Thank you for highlighting the issue with calibration in T265. We have moved our focus to our next generation of products and consequently, we will not be addressing calibration changes in the T265.

[haldaazhim]

Hello @JBBee! I am interested in this topic and I want to ask the question about update_calibration.py script that you attached in this topic. I ran your script in ubuntu 20.04.5 LTS to change the extrinsics rotation and translation value of "fisheye 1" to "fisheye 2" into the value that you modified in your t265 camera. Instead of got a same value like yours, my extrinsic value is miss from what I input in R_p_1. In order to clarify, here is my customize R_p_1 value:

And here is the output data that I've got:

The value corresponding to input R_p_1 is
-extrinsics "fisheye 1" to "gyro" rotation matrix
-extrinsics "fisheye 1" to "accel" rotation matrix
-extrinsics "fisheye 1" to "pose" rotation matrix

the extrinsics "fisheye 1" to "fisheye 2" rotation matrix is also changed when I use your update_calibration.py but it changed not in accordance with the value that I put in your script. So, how to change the extrinsics rotation value of "fisheye 1" to "fisheye 2" as well as "fisheye 2" to "fisheye 1"?

[JBBee]

Hello @haldaazhim! I believe the problem is that Intel Realsense has removed the T265 functionality from the latest SDK release. Please try the script with python3.10 or lower and run pip3 install pyrealsense2==2.53.1.4623

If the write is successful you should see:

Write to device? [y/n]: y
Writing to device...
Finished

Since the T265 is discontinued do you know of any alternatives?

Thanks.