Kimera-VIO on TUM-VI dataset

[Quentin-9]

Hi, I used Kimera on Euroc with no issues and now I am trying to run Kimera on the TUM-VI dataset (https://vision.in.tum.de/data/datasets/visual-inertial-dataset). However I'm not sure how to make it work.

The only files that I change are [Left/Right]CameraParams.yaml and IMUParams.yaml following the calibration file "pinhole (512*512)" camera model that we can find in the dataset website.

When I run Kimera with those parameters, I have the following message error:
"StereoFrame.cpp:397 - undistortRectifyPoints: vector with zero depth".

Do I have to create a TUM-VI dataprovider.cpp as you did for Euroc and Kitti? Or maybe the equidistant distorsion model is not well handled? Did you already try to run Kimera with this dataset?
I am a bit confused...

Thank you!

Command:

./scripts/stereoVIOEuroc.bash -p "path_to_tumvi_dataset"

Console output:

I1020 16:39:26.891789 15725 Pipeline.cpp:402] Statistics                                                                                                                                                                                     
-----------                                  #  Log Hz  {avg     +- std    }    [min,max]                                                                                                                                                    
Data Provider [ms]                                  0                                                                                                                                                                                        
Display [ms]                                       76   3.90304 {248.921 +- 238.288}    [2,918]                                                                                                                                              
Lcd [ms]                                           71   3.61693 {12.9859 +- 3.91515}    [5,28]                                                                                                                                               
Mesher [ms]                                        71   3.61478 {8.71831 +- 8.42646}    [1,43]                                                                                                                                               
VioBackEnd [ms]                                    71   3.61798 {2.56338 +- 1.48066}    [0,8]                                                                                                                                                
VioFrontEnd Frame Rate [ms]                       231   12.0076 {2.09524 +- 0.300000}   [1,8]                                                                                                                                                
VioFrontEnd Keyframe Rate [ms]                     70   3.67238 {13.1286 +- 3.91179}    [8,29]                                                                                                                                               
VioFrontEnd [ms]                                  302   15.2532 {4.71854 +- 4.98032}    [1,29]                                                                                                                                               
Visualizer [ms]                                    71   3.61431 {3.61972 +- 1.80765}    [2,13]                                                                                                                                               
backend_input_queue Size [#]                       71   2.40791 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
data_provider_left_frame_queue Size [#]           304   10.2516 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
data_provider_right_frame_queue Size [#]          303   10.2312 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
display_input_queue Size [#]                      141   4.77831 {34.1348 +- 12.3463}    [1,64]                                                                                                                                               
lcd_backend_queue Size [#]                         71   3.61783 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
lcd_frontend_queue Size [#]                       302   15.2528 {2.63907 +- 1.25433}    [1,5]                                                                                                                                                
mesher_backend Size [#]                            71   2.40994 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
mesher_frontend Size [#]                          302   10.1903 {2.63907 +- 1.25433}    [1,5]                                                                                                                                                
stereo_frontend_input_queue Size [#]              302   10.1819 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
visualizer_backend_queue Size [#]                  71   2.40994 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
visualizer_frontend_queue Size [#]                302   10.1903 {2.63907 +- 1.25433}    [1,5]                                                                                                                                                
visualizer_mesher_queue Size [#]                   71   2.40851 {1.00000 +- 0.00000}    [1,1]                                                                                                                                                
W1020 16:39:26.925913 15718 Tracker.cpp:578] 1-point RANSAC (voting) could not find a solution.                                                                                                                                              
W1020 16:39:26.933882 15720 Mesher.cpp:1380] Missing landmark information to build 3D Mesh.                                                                                                                                                  
W1020 16:39:26.934303 15720 Mesher.cpp:1408] Missing landmark information for the Mesher!                                                                                                                                                    
W1020 16:39:26.934667 15720 Mesher.cpp:1608] List of Keypoints with associated Landmarks is empty.                                                                                                                                           
W1020 16:39:26.935127 15720 Mesher.cpp:1422] 2D Mesh is empty!                                                                                                                                                                               
W1020 16:39:26.935573 15720 Mesher.cpp:432] Missing landmark information for the Mesher: cannot generate 3D Mesh.                                                                                                                            
W1020 16:39:26.935958 15720 Mesher.cpp:523] Missing landmark information for the Mesher: cannot trim 3D mesh to time horizon.                                                                                                                
W1020 16:39:26.936389 15722 OpenCvVisualizer3D.cpp:479] No landmark information for Visualizer. Not displaying 3D points.                                                                                                                    
F1020 16:39:26.939332 15721 StereoFrame.cpp:397] undistortRectifyPoints: versor with zero depth                                                                                                                                              
*** Check failure stack trace: ***                                                                                                                                                                                                               
@     0x7f27451cc0cd  google::LogMessage::Fail()                                                                                                                                                                                             
@     0x7f27451cdf33  google::LogMessage::SendToLog()                                                                                                                                                                                        
@     0x7f27451cbc28  google::LogMessage::Flush()                                                                                                                                                                                            
@     0x7f27451ce999  google::LogMessageFatal::~LogMessageFatal()                                                                                                                                                                            
@     0x7f2746884387  VIO::StereoFrame::undistortRectifyPoints()                                                                                                                                                                             
@     0x7f274688602f  VIO::StereoFrame::sparseStereoMatching()                                                                                                                                                                               
@     0x7f27469d300a  VIO::LoopClosureDetector::rewriteStereoFrameFeatures()                                                                                                                                                                 
@     0x7f27469d6e82  VIO::LoopClosureDetector::processAndAddFrame()                                                                                                                                                                         
@     0x7f27469da6cd  VIO::LoopClosureDetector::detectLoop()                                                                                                                                                                                 
@     0x7f27469dae89  VIO::LoopClosureDetector::spinOnce()                                                                                                                                                                                   
@     0x7f27469c98f3  VIO::PipelineModule<>::spin()                                                                                                                                                                                          
@     0x7f27448cd66f  (unknown)                                                                                                                                                                                                              
@     0x7f27454076db  start_thread                                                                                                                                                                                                           
@     0x7f274431188f  clone                                                                                                                                                                                                              
./scripts/stereoVIOEuroc.bash: line 110: 15717 Aborted                 (core dumped) $BUILD_PATH/stereoVIOEuroc --dataset_type="$DATASET_TYPE" --dataset_path="$DATASET_PATH" --initial_k=50 --final_k=2000 --params_folder_path="$PARAMS_PATH" --use_lcd="$USE_LCD" --vocabulary_path="$VOCABULARY_PATH/ORBvoc.yml" --flagfile="$PARAMS_PATH/flags/stereoVIOEuroc.flags" --flagfile="$PARAMS_PATH/flags/Mesher.flags" --flagfile="$PARAMS_PATH/flags/VioBackEnd.flags" --flagfile="$PARAMS_PATH/flags/RegularVioBackEnd.flags" --flagfile="$PARAMS_PATH/flags/Visualizer3D.flags" --logtostderr=1 --colorlogtostderr=1 --log_prefix=1 --v=0 --vmodule=Pipeline*=00 --log_output="$LOG_OUTPUT" --log_euroc_gt_data="$LOG_OUTPUT" --save_frontend_images=1 --visualize_frontend_images=1 --output_path="$OUTPUT_PATH"

Additional files:

Please give also the following information:

• Kimera-Vio branch, tag or commit used: master
• GTSAM version used: Latest git clone https://github.com/borglab/gtsam
• OpenGV version used: Latest git clone https://github.com/laurentkneip/opengv
• OpenCV version used: 3.3.1
• Operating system and version (e.g. Ubuntu 16.04 or Windows 10): Ubuntu 18.04
• Did you change the source code? (yes / no): no

[ToniRV]

Hi @Quentin-9, let me try to help:

Did you already try to run Kimera with this dataset?
Or maybe the equidistant distorsion model is not well handled?

I haven't tried TUM-VI, and although we support equidistant, we don't have that part of the pipeline thoroughly tested.
We have some unit tests such as:

Kimera-VIO/tests/testStereoFrame.cpp

Line 1122 in b9d9ed3

TEST_F(StereoFrameFixture, DISABLED_undistortFisheyeStereoFrame) {

but as you can see, it is currently disabled, probably because we ran into some issues there.

See an example of equidistant calibration params yaml here: https://github.com/MIT-SPARK/Kimera-VIO/blob/b9d9ed38fc5d9e64f8e594ac18c10dfb43ea6099/tests/data/ForStereoFrame/left_sensor_fisheye.yaml

Do I have to create a TUM-VI dataprovider.cpp as you did for Euroc and Kitti?

Most likely yes, unless you transform the TUM-VI dataset to the Euroc format OR if TUM-VI is given as rosbags (you'll need Kimera-VIO-ROS in that case).
Mind that we do not support Kitti anymore, so do not follow the Kitti code, rather use the Euroc one, or have a look at the dataprovider in Kimera-VIO-ROS:

• RosOnlineDataProvider
• RosBagDataProvider

Hope this helps!

[Quentin-9]

Hi, thank you very much!
Your answer helps a lot.

[Viky397]

Hello @Quentin-9, I am trying to run my RGBD Realsense bag files through VIO and am running into issues.. would you please be able to share what changes you made to the launch files? Thank you!

[Quentin-9]

Hi @Viky397, unfortunately I have not yet been able to run Kimera on another dataset than Euroc.

I created a new DataProvider to fit the format of TUM-VI following the EurocDataProvider.cpp but it does not work.
I think the main problem is coming from the distorsion model (equidistant) or probably wrong camera/imu settings (extrinsics/intrinsics). I am not sure and I have not checked thoroughly to understand why I have this kind of error: "undistortRectifyPoints: versor with zero depth".

About your question, I am not using ROS but I think Kimera provides params for this RealSense sensor. I know that you can also use Kalibr to get your parameters. Though, I advise you to create a new issue with all the details about your problem so you can have a proper and useful answer.

[Viky397]

Thank you for the response! I'll open an issue.