Merge pull request #45 from rpng/develop_v2.0

Development v2.0
rpng · Apr 3, 2020 · e169a77 · e169a77
2 parents 092bafc + 7c3fd77
commit e169a77
Show file tree

Hide file tree

Showing 157 changed files with 680,503 additions and 2,574 deletions.
diff --git a/Doxyfile b/Doxyfile
@@ -2020,7 +2020,7 @@ INCLUDE_FILE_PATTERNS  =
 # recursively expanded use the := operator instead of the = operator.
 # This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
 
-PREDEFINED             =
+PREDEFINED             = DOXYGEN
 
 # If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this
 # tag can be used to specify a list of macro names that should be expanded. The

diff --git a/Doxyfile-mcss b/Doxyfile-mcss
@@ -7,7 +7,6 @@ HTML_EXTRA_STYLESHEET = \
     https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600&subset=latin-ext \
     docs/css/m-udel+documentation.compiled.css \
     docs/css/custom.css
-####    ../css/m-light+documentation.compiled.css \
 
 
 

diff --git a/ReadMe.md b/ReadMe.md
@@ -45,6 +45,7 @@ Please take a look at the feature list below for full details on what the system
     3. Anchored XYZ
     4. Anchored inverse depth
     5. Anchored MSCKF inverse depth
+    6. Anchored single inverse depth
 * Calibration of sensor intrinsics and extrinsics
     * Camera to IMU transform 
     * Camera to IMU time offset
@@ -79,6 +80,7 @@ Please take a look at the feature list below for full details on what the system
 This code was written by the [Robot Perception and Navigation Group (RPNG)](https://sites.udel.edu/robot/) at the University of Delaware.
 If you have any issues with the code please open an issue on our github page with relevant implementation details and references.
 For researchers that have leveraged or compared to this work, please cite the following:
+
 ```txt
 @Conference{Geneva2020ICRA,
   Title      = {OpenVINS: A Research Platform for Visual-Inertial Estimation},

diff --git a/docs/00-page-order.dox b/docs/00-page-order.dox
@@ -15,8 +15,7 @@
 @page simulation Visual-Inertial Simulator
 
 
-@page evaluation Estimator Evaluation Methods
-@page eval-metrics Estimator Evaluation Metrics
+@page evaluation System Evaluation
 
 
 @page dev-welcome Developer and Internals

diff --git a/docs/bib/extra.bib b/docs/bib/extra.bib
@@ -55,7 +55,8 @@ @article{Trawny2005TR
     journal = {University of Minnesota, Dept. of Comp. Sci. \& Eng., Tech. Rep},
     volume = {2},
     pages = {2005},
-    year = {2005}
+    year = {2005},
+    url = {http://mars.cs.umn.edu/tr/reports/Trawny05b.pdf},
 }
 
 
@@ -138,6 +139,15 @@ @inproceedings{Delmerico2018ICRA
     url = {http://rpg.ifi.uzh.ch/docs/ICRA18_Delmerico.pdf},
 }
 
+@inproceedings{Delmerico2019ICRA,
+    title={Are we ready for autonomous drone racing? the UZH-FPV drone racing dataset},
+    author={Delmerico, Jeffrey and Cieslewski, Titus and Rebecq, Henri and Faessler, Matthias and Scaramuzza, Davide},
+    booktitle={2019 International Conference on Robotics and Automation (ICRA)},
+    pages={6713--6719},
+    year={2019},
+    organization={IEEE},
+    url = {http://rpg.ifi.uzh.ch/docs/ICRA19_Delmerico.pdf},
+}
 
 @article{Eckenhoff2019IJRR,
     author    = {Kevin Eckenhoff and Patrick Geneva and Guoquan Huang},
@@ -150,3 +160,14 @@ @article{Eckenhoff2019IJRR
     url       = {https://doi.org/10.1177/0278364919835021}
 }
 
+@article{Qin2018TRO,
+    title={Vins-mono: A robust and versatile monocular visual-inertial state estimator},
+    author={Qin, Tong and Li, Peiliang and Shen, Shaojie},
+    journal={IEEE Transactions on Robotics},
+    volume={34},
+    number={4},
+    pages={1004--1020},
+    year={2018},
+    publisher={IEEE},
+    url = {https://arxiv.org/pdf/1708.03852.pdf},
+}
diff --git a/docs/dev-index.dox b/docs/dev-index.dox
@@ -42,7 +42,7 @@ As compared to passing a Jacobian matrix that is the full size of state wide, we
 For example, if we have a global 3D SLAM feature update (implemented in @ref ov_msckf::UpdaterSLAM) our Jacobian will only be a function of the newest clone and the feature.
 This means that our Jacobian is only of size 9 as compared to the size our state.
 In addition to the matrix containing the Jacobian elements, we need to know what order this Jacobian is in, thus we pass a vector of types which correspond to the state elements our Jacobian is a function of.
-Thus in our example, it would contain two types: our newest clone @ref ov_core::PoseJPL and current landmark feature @ref ov_core::Landmark with our Jacobian being the type size of the pose plus the type size of the landmark in width.
+Thus in our example, it would contain two types: our newest clone @ref ov_type::PoseJPL and current landmark feature @ref ov_type::Landmark with our Jacobian being the type size of the pose plus the type size of the landmark in width.
 
 
 

diff --git a/docs/evaluation.dox → docs/eval-error.dox b/docs/evaluation.dox → docs/eval-error.dox
@@ -1,14 +1,7 @@
 /**
 
 
-@page evaluation Filter Evaluation Methods
-
-The goal of our evaluation is to ensure fair comparison to other methods and our own.
-The actual metrics we use can be found on the @ref eval-metrics page.
-Using our metrics we wish to provide insight into *why* our method does better and in what ways (as no method will outperform in all aspects).
-We additionally need to account for the inherent randomness of the methods and thus should treat each estimated trajectory as a random output.
-The key steps that we follow are the following:  
-
+@page eval-error Filter Error Evaluation Methods
 
 
 @m_class{m-note m-warning}
@@ -34,16 +27,6 @@ Works with topics of type `PoseWithCovarianceStamped`, `PoseStamped`, `Transform
 @endcode
 
 
-To evaluate the computational load, we have a python script that leverages the [psutil](https://github.com/giampaolo/psutil) python package to record percent CPU and memory consumption.
-
-@code{.xml}
-<node name="recorder_timing" pkg="ov_eval" type="pid_ros.py" output="screen">
-    <param name="nodes"   type="str" value="/run_subscribe_msckf" />
-    <param name="output"  type="str" value="/home/user/data/time_log.txt" />
-</node>
-@endcode
-
-
 
 @section eval-ov-transform Transformation
 
@@ -63,7 +46,7 @@ We store trajectories by first their algorithm name and then a folder for each d
 The folder names of the datasets need to match the groundtruth trajectory files which should be in their own separate folder.
 Please see the example recorded datasets for how to structure your folders.
 
-@code{.text}
+@code{.txt}
 truth/
     dateset_name_1.txt
     dateset_name_2.txt
@@ -116,6 +99,7 @@ It will use the filename as the name in the legend, so you can change that to ch
 rosrun ov_eval plot_trajectories <align_mode> <file_gt.txt> ... <file_est9.txt>
 rosrun ov_eval plot_trajectories posyaw 1565371553_estimate.txt truths/V1_01_easy.txt
 @endcode
+
 @image html eval/plot_xy.png width=70%
 @image html eval/plot_z.png width=70%
 
@@ -133,7 +117,9 @@ To change the RPE distances you will need to edit the code currently.
 rosrun ov_eval error_singlerun <align_mode> <file_gt.txt> <file_est.txt>
 rosrun ov_eval error_singlerun posyaw 1565371553_estimate.txt truths/V1_01_easy.txt
 @endcode
+
 Example output:
+
 @code{.shell-session}
 [ INFO] [1583422165.069376426]: [COMP]: 2813 poses in 1565371553_estimate => length of 57.36 meters
 [ INFO] [1583422165.091423722]: ======================================
@@ -153,6 +139,7 @@ Example output:
 [ INFO] [1583422165.127934904]: seg 32 - median_ori = 0.715 | median_pos = 0.071 (1943 samples)
 [ INFO] [1583422165.127942178]: seg 40 - median_ori = 0.540 | median_pos = 0.063 (1792 samples)
 @endcode
+
 @image html eval/singlerun.png width=80%
 
 
@@ -168,7 +155,9 @@ To change the RPE distances you will need to edit the code currently.
 rosrun ov_eval error_dataset <align_mode> <file_gt.txt> <folder_algorithms>
 rosrun ov_eval error_dataset posyaw truths/V1_01_easy.txt algorithms/
 @endcode
+
 Example output:
+
 @code{.shell-session}
 [ INFO] [1583422654.333642977]: ======================================
 [ INFO] [1583422654.333915102]: [COMP]: processing mono_ov_slam algorithm
@@ -207,6 +196,7 @@ Example output:
 [ INFO] [1583422658.194763413]: ======================================
 ....
 @endcode
+
 @image html eval/dataset.png width=100%
 
 
@@ -227,7 +217,9 @@ To change the RPE distances you will need to edit the code currently.
 rosrun ov_eval error_comparison <align_mode> <folder_groundtruth> <folder_algorithms>
 rosrun ov_eval error_comparison posyaw truths/ algorithms/
 @endcode
+
 Example output:
+
 @code{.shell-session}
 [ INFO] [1583425216.054023187]: [COMP]: 2895 poses in V1_01_easy.txt => length of 58.35 meters
 [ INFO] [1583425216.092355692]: [COMP]: 16702 poses in V1_02_medium.txt => length of 75.89 meters
@@ -261,6 +253,7 @@ mono\_ov\_vio & 0.826 / 0.094 & 1.039 / 0.106 & 1.215 / 0.111 & 1.283 / 0.132 &
 ....
 [ INFO] [1583425262.514587296]: ============================================
 @endcode
+
 @image html eval/comparison.png width=100%
 
 

diff --git a/docs/eval-home.dox b/docs/eval-home.dox
@@ -0,0 +1,34 @@
+/**
+
+
+@page evaluation System Evaluation
+
+
+The goal of our evaluation is to ensure fair comparison to other methods and our own.
+The actual metrics we use can be found on the @ref eval-metrics page.
+Using our metrics we wish to provide insight into *why* our method does better and in what ways (as no method will outperform in all aspects).
+Since we are also interested in applying the systems to real robotic applications, the realtime performance is also a key metric we need to investigate.
+Timing of different system components is also key to removing bottlenecks and seeing where performance improvements or estimator approximations might help reduce complexity.
+
+The key metrics we are interested in evaluating are the following:
+
+- Absolute Trajectory Error (ATE)
+- Relative Pose Error (RPE)
+- Root Mean Squared Error (RMSE)
+- Normalized Estimation Error Squared (NEES)
+- Estimator Component Timing
+- System Hardware Usage (memory and computation)
+
+
+
+@section evaluation-more System Evaluation Guides
+
+-   @subpage eval-metrics --- Definitions of different metrics for estimator accuracy.
+-   @subpage eval-error --- Error evaluation methods for evaluating system performance.
+-   @subpage eval-timing --- Timing of estimator components and complexity.
+
+
+
+
+
+*/