Error messages and tuning when transplanting R-VIO to non-ROS setup

[jeffcyu]

Hello Zheng Haui and co!

Thanks for this awesome bit of code; I especially appreciate that it is very clean and easy to follow.

So, I'm trying to deploy R-VIO online on a custom, non-ROS setup. I have the camera calibration (intrinsics) and extrinsics (I think), but not the IMU noise properties. I get IMU readings at around 200Hz, and images at around 15-20Hz (good enough, I hope?). I have gotten the code to be plugged into my framework so it is now taking in data and outputting poses, but I am encountering some issues.

I turned on the print/debugging messages and noticed that I was getting a lot of:
Invalid inverse-depth feature estimate (1)!
Invalid inverse-depth feature estimate (1)!
...
Along with the occasional:
Failed in Mahalanobis distance test!
Followed by:
Too few measurements for update!
Followed by my position drifting off into the great beyond...

Digging a little deeper, I started looking into what the inverse depth bearing parameters were doing. I saw that the phi and psi are usually good and within the +-pi/2 bounds, but it was the rho that was always going negative after the LM solve. I looked into the solver iterations and they seems to converge fairly quickly (3-4 steps), with phi and psi not changing much but rho just jumping to some negative value. For the features that do end up with a positive rho value, they always fail the Mahalanobis test, with the distance being on the order of magnitude of 1e3~1e5.

Do you have any thoughts on what could be causing this? My initial reaction was to review my parameters, but looking at the values, the only parameter that has a large difference from the euroc ones are the extrinsics (and the IMU noise properties, which I left the same as the euroc ones). From what I can tell the extrinsics are used for the update step, so would you say that the symptoms I'm seeing are from bad extrinsics? I'm fairly confident with my rotation matrix, but the translation may be a little off, so how sensitive is the system to that (or bad IMU noise properties, for that matter)?

I noticed that in https://github.com/rpng/R-VIO/issues/8 you mentioned having correct parameters is of paramount importance and I totally agree, so I guess I'm inquiring to try and find the root cause of these messages so I can have an idea of which parameters I need to tune (or if the issue is something else completely!).

If you have additional insight or thoughts I'm all ears!

Thank you for your time,
~Jeff

[huaizheng]

Those messages are particular for informing, so if you keep seeing them then that means something out of this codebase may be wrong, as you said the extrinsic parameters (including the temporal offset). While the title of your issue @jeffcyu may confuse the other users because actually this is caused by your transplant attempt, not an issue about the R-VIO itself.

[jeffcyu]

Thanks for the very quick reply!

I apologize for the title, hopefully this new title is more reasonable.

So just to confirm what you're saying: the error messages that are showing are probably due to some incorrect parameter like the extrinsics or the temporal offset, or from some bug introduced when transplanting the code; is this correct?

Additionally, would you happen to have some intuition on which parameters tend to be the most sensitive? For example, do you find that the extrinsics and temporal offset the greatest culprits in most cases, or is it difficult to generalize?

Lastly, I was curious if extrinsics play a role in the feature tracking? I noticed that there is a rotation matrix that is created with them and then used for ransac. Does this mean that bad extrinsics would also result in a lot of features being tossed during ransac, meaning a symptom of bad extrinsics could be the need to constantly supplement the feature list?

Thanks again for your time,
~Jeff

[huaizheng]

Thanks @jeffcyu, I saw it. The importance of extrinsic parameters is that they connect the visual and inertial information in both front end (tracking) and back end (update). As you got negative depth estimate of feature, I may think this was caused by extrinsic calibration because the information for doing this feature recovery includes the relative IMU poses, feature tracking results, and extrinsic parameters. You need good extrinsic parameters to convert relative IMU poses to relative camera poses, or it would not be surprise to get a landmark being initialized behind the camera and to have those messages come out. Also, my 2-point ransac needs relative camera rotation information, while I can first get relative IMU rotation and then convert it to the camera frame using extrinsic parameters. So if those parameters are not reasonable enough, it may cause big problem, e.g., rejecting good tracks instead. Kalibr is the tool I used for calibration, which can give you both spatial and temporal parameters, while you need to know something about your sensor, such as IMU noise parameters, in order to get good calibration result. Also, please note that it is better to start R-VIO from stationary in order to avoid possible drift caused by my current initialization method.

[jeffcyu]

Hi @huaizheng, thanks for your good suggestions!

I ran my system through Kalibr and generated the intrinsics for the camera as well as the spatial and temporal extrinsics. Now, I'm noticing that the results of the tracking are quite good, but the feature depth parameter (rho) is still frequently becoming negative after the LM solve when performing updates.

I've also noticed that the timestamps from my camera are a bit inconsistent, and images aren't necessarily coming in at a fixed rate; could this be the root of the issue?

Lastly, regarding the Mahalanobis test, what would be the effects of relaxing the chi-squared threshold to something lower?

Thanks again for your time,
~Jeff

[huaizheng]

@jeffcyu, it is normal to have a triangulation result with negative depth. For example, a feature may be tracked from an edge, but this could be either the closest or the farthest edge of a plane depending on the view point, so the depth ambiguity would cause the above result, but it is not used by estimator.

Unless you use some very cheap camera which has very low frame rate, the inconsistency of timestamps may not become a big issue.

The mahalanobis test plays a role in outlier rejection, in addition to the ransac in tracker. Its value reflects the consistency of feature measurements as I show in the paper, and this is why the current value is the degrees of freedom of all the measurements of a single feature.