Event Feature Tracking
This repo contains MATLAB implementations of the event-based feature tracking methods described in Event-based Feature Tracking with Probabilistic Data Association and Event-based Visual Inertial Odometry. The code has been tested to run on MATLAB 2017b.
We provide functions in the
data folder to convert event data from ROS bag to MATLAB mat files. Currently, data from the Event Camera Dataset and the Multi Vehicle Stereo Event Camera dataset is tested. The
extract_ros_data.m and either the
generate_undistort_map_equi.py functions are necessary to generate the data and undistort map mat files. You must select the appropriate undistort map function depending on whether the camera calibration uses the radtan distortion model (e.g. the Event Camera Dataset) or the equidistant distortion model (e.g. MVSEC). Note that you will need the matlab_rosbag package, which has pre-compiled releases here. Assuming you have the
boxes_6dof.bag file in the data folder, the extraction code can be run as follows:
load_folder = 'data'; save_folder = 'data'; extract_ros_data('boxes_6dof', load_folder, save_folder)
For the radtan distortion model, the undistort maps are generated in MATLAB as follows:
load data/boxes_6dof.mat generate_undistort_map_radtan(cinfo, 'boxes_6dof')
For the equidistant distortion model, the undistort maps are generated from the terminal as follows:
python generate_undistort_map_radtan.py --camchain data/camchain-imucam-indoor_flying1.yaml
Note that the undistort_map can be shared across sequences with the same camera intrinsics.
Running the code
The code can be run from the script
main.m. Before you do so, you must set a few parameters, which are stored in the function
get_params.m. At a minimum, you will need to set the path to the generated data mat (
params.data_path), as well as the generated undistort_map mat (
params.undistort_data_path). This function also provides many other parameters, such as start and end times in the sequence, number of features and many more. For optimal performance, the main tracking loop can be run in parallel by setting
If you are using the IMU related features (for the EM2 affine warp or two point RANSAC), you will need the Robotics System Toolbox, or to define your own functions for
rotm2quat, which convert between a quaternion (w,x,y,z) and a 3x3 rotation matrix.
The feature points and their corresponding IDs are stored as the variables
valid_ids, and are updated at the end of each iteration in
params.headless is set to
false, the features will be plotted on top of the integrated event image in Figure 1.
Note that the label "EM1 flow" refers to the optical flow estimation step (Section IV.B in Event-based Feature Tracking with Probabilistic Data Association), and the label "EM2 affine" refers to the feature alignment step (Section IV.C in Event-based Feature Tracking with Probabilistic Data Association).
To debug the underlying EM algorithms, set
true. This will plot the optimization process at each step, and may help give you an intuition behind what is being minimized. Note that setting this to
true will automatically set
false, as plotting doesn't work properly inside a parfor loop.
For comparisons and benchmarks, the method proposed in "Event-based Feature Tracking with Probabilistic Data Association" can be run by setting
false. Setting these parameters to
true will run the full method proposed in "Event-based Visual Inertial Odometry".
If you use this code in an academic publication, please cite the following works:
Alex Zihao Zhu, Nikolay Atanasov and Kostas Daniilidis. "Event-based Feature Tracking with Probabilistic Data Association", IEEE International Conference on Robotics and Automation (ICRA), 2017.
Alex Zihao Zhu, Nikolay Atanasov, and Kostas Daniilidis. "Event-Based Visual Inertial Odometry." IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
This technology is the subject of a pending patent application: PCT/US2018/018196