_notebooks/2020-11-26-lessons-and-future-directions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Lessons learned and future directions\"\n",
    "> \"What I think is worthy to work on in wide baseline stereo\"\n",
    "- toc: false\n",
    "- image: images/yoy_cat.jpg\n",
    "- branch: master\n",
    "- badges: true\n",
    "- comments: true\n",
    "- hide: false\n",
    "- search_exclude: false"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would like to share some lessons I have learned about wide baseline stereo and propose some research directions, which are worth exploring in the short and longer term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lessons learned\n",
    "\n",
    "## 1. Benchmark = metrics + implementation + dataset\n",
    "\n",
    "In our paper \"[Image Matching across Wide Baselines: From Paper to Practice](https://arxiv.org/abs/2003.01587)\" we focused on the first two parts. Specifically, metrics -- if they are not \"downstream\" metrics, the improvements in the single component might not translate to the overall system improvements. \n",
    "And implementation -- implementing the simplest possible setup is, of course, a valuable tool, but one have to also incorporate the best known practices, e.g. matching, RANSAC tuning as so on.\n",
    "\n",
    "I hope, that we have delivered that message to the community. But the last component -- the dataset -- we have, perhaps, overlooked a bit ourself. The problem with dataset limitations, e.g. lack of illumination or seasonal changes is not that one does not properly address. Usually benchmark papers are adressing their limitations quite clearly.\n",
    "The problem is that researchers (including myself) have tendency to work on improving results, which are easily measurable, therefore the implicitly designing the methods, which solve only some specific problem, encoded in the form of the dataset.\n",
    "\n",
    "\n",
    "## 2. Trying to work for the \"most general\" case might be detrimental for the practical applications. \n",
    "\n",
    "This is kind of opposite side of the lesson 1. For example, the classical SIFT matching is rotation-invariant, because that was assumed to be the requirement to work in the \"real world\". However, the practice of image retrieval and then most of learned local features like R2D2, SuperPoint, DELF and so on, showed that \"up-is-up\" is a reasonable assumption to built on. The rotational invariance in lots of scenarios hurt more than helps. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. When borrow idea from classical paper, adapt it\n",
    "\n",
    "[HardNet](https://proceedings.neurips.cc/paper/2017/hash/831caa1b600f852b7844499430ecac17-Abstract.html) borrows the idea of the using [second nearest neighbor](https://medium.com/@ducha.aiki/how-to-match-to-learn-or-not-to-learn-part-2-1ab52ede2022)(SNN) descriptor from the SIFT. However, using SNN ratio for descriptor learning leads to inferior results. We had to modify it to the triplet margin loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## 4. Classic handcrafted algorithms are not dead and can be improved a lot\n",
    " \n",
    "Modern versions of RANSAC are still necessary for two view matching even when so complex methods as SuperGlue as used. Moreover, they still have quite a lot things for improvement, as proved by my colleague, [Daniel Barath](https://scholar.google.hu/citations?user=U9-D8DYAAAAJ&hl=en)\n",
    "\n",
    "They also have a benefit, that once you have an idea, you can make a paper out it faster, as you don't need gathering data and training the model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Future research directions\n",
    "\n",
    "## 1. Application-specific local features\n",
    "\n",
    "One thing about the current local features is that they are kind of universal. SIFT works reasonably well for lots of applications: 3d reconstruction, SLAM, image retrieval, etc. The learned features, like SuperPoint or R2D2, although are biased towards the data they are trained on, still don't have anything domain specific.\n",
    "\n",
    "Let me explain. There are different qualities about the local feature (detectors). It can be more or less robust to nuisance factors like the illumination and camera position. It can be more or less precisely localized. It can be more or less dense and/or evenly distributed over the image. \n",
    "\n",
    "For example, in image retrieval, one does not really care about precise localization, the robustness is much more important. For the 3d reconstruction one would like to have a lot of 3d points to get the reasonable reconstruction.\n",
    "On the other hand, for the SLAM/relocalization application, sparse features would be more advantageous because of smaller memory footprint and computational cost. \n",
    "\n",
    "There are, actually, some steps in that direction. Let me name a few. \n",
    "\n",
    "1. [HOW local features](https://arxiv.org/pdf/2007.13172.pdf) designed for the image retrieval\n",
    "![](2020-11-26-lessons-and-future-directions_files/att_00000.png \"Robust, but poorly localized HOW local features for imare retrieval, from 'Learning and aggregating deep local descriptors for instance-level recognition' paper\")\n",
    "\n",
    "2. [SIPs: Succinct Interest Points from Unsupervised Inlierness Probability Learning](http://rpg.ifi.uzh.ch/docs/3DV19_Cieslewski.pdf) sparse local features for the SLAM.\n",
    "<!--![](2020-11-26-lessons-and-future-directions_files/att_00001.png \"A small number of keypoints might be enough for estimating the camera pose. Figure from SIPs paper\") -->\n",
    "\n",
    "3. [Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task](https://arxiv.org/abs/1912.00623)\n",
    "![](2020-11-26-lessons-and-future-directions_files/att_00002.png \"Optimizing local features for non-differentiable high-level task, Figure from Reinforced Feature Points\")\n",
    "\n",
    "\n",
    "I believe, that it is only beginning and we are yet to experience AlphaZero moment for the local features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Rethinking overall wide baseline stereo pipeline, optimized for the specific application\n",
    "\n",
    "It is often perceived, that image matching across the unordered collection of the images is a task with quadratic complexity w.r.t. number of images. Some operations can be done separately, e.g. feature detection, but others, like feature matching and RANSAC cannot. Right?\n",
    "\n",
    "Not necessarily. It turns out, that one can avoid running feature matching and RANSAC for more than 90% of image pairs with clever preprocessing, ordering and re-using results from the previous matching.\n",
    "Moreover, in order to do the whole task faster (matching image collections), one may need to introduce additional steps, which are not necessary, or slowing things down for the two images case.\n",
    "\n",
    "That's what we done for the [intial pose estimation for the global SfM](https://arxiv.org/abs/2011.11986) in our paper, which reduced the matching runtime from 200 hours to 29.\n",
    "\n",
    "<!-- ![](2020-11-26-lessons-and-future-directions_files/att_00003.png \"Table from the Efficient Initial Pose-graph Generation for Global SfM\") -->\n",
    "\n",
    "Another example would be \"[SuperGlue: Learning Feature Matching with Graph Neural Networks](https://arxiv.org/abs/1911.11763)\", where authors abandoned traditional descriptor matching and instead leveraged all the information for keypoint and descriptors from both images altogether. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Rethinking and improving the process of training data generation for the WBS\n",
    "\n",
    "So far, all the local feature papers I have seen rely on one of the ground truth source.\n",
    "\n",
    "1. SfM data, obtained with COLMAP with, possibly, a cleaned depth information.\n",
    "2. Affine and color augmentation.\n",
    "3. Synthetic images (e.g. corners for SuperPoint).\n",
    "\n",
    "There are several problems with them. \n",
    "\n",
    "**SfM data** assumes that the data is matchable by the existing methods, at least for a some extent. That might not always be true for cross-seasonal, medical and other kind of data. It is also not applicable for historical photographies and other types of data. Moreover, SfM data takes quite a time for compute and space to store. I believe that we may do better. \n",
    "\n",
    "\n",
    "**Affine and color augmentation** can take us only this far -- we actually want our detectors and descriptors to be robust to the changes, which we don't know how to simulate/augment. \n",
    "\n",
    "\n",
    "**Synthetic images** as they are in, say, [CARLA](https://carla.org/) simulator lack fine details and photorealism. However, I am optimistic about using neural renderers and learned wide baseline stereo is a GAN-like self-improving loop. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Matching with On-Demand View Synthesis revisited\n",
    "\n",
    "I like the \"on-demand\" principle a lot and I think we can explore it much more that we are now. \n",
    "So far we have either [affine view synthesis (ASIFT, MODS)](https://ducha-aiki.github.io/wide-baseline-stereo-blog/2020/08/06/affine-view-synthesis.html), or [GAN-based stylizations](https://people.ee.ethz.ch/~timofter/publications/Anoosheh-ICRA-2019.pdf) for the day-night matching. \n",
    "\n",
    "That is why I am glad to see papers like [Single-Image Depth Prediction Makes Feature Matching Easier](https://arxiv.org/abs/2008.09497), which generate normalized views based on depth in order to help the matching. \n",
    "\n",
    "Why not go further? Combine viewpoint, illumination, season, sensor synthesis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Moar inputs!\n",
    "\n",
    "I have mentioned above that monocular depth may help the feature matching or [camera pose estimation](https://ducha-aiki.github.io/wide-baseline-stereo-blog/2020/07/17/affine-correspondences.html#Back-to-wide-baseline-stereo). However, why stop here? \n",
    "\n",
    "Let's use other networks as well, especially given that we will need them on robot or vehicle anyway. Semantic segmentation? Yes, please. Surface normals? Why not? Intrinsic images? Йой, най буде! {% fn 1 %} \n",
    "\n",
    "{{ \"Ukrainian, means let is be\" | fndetail: 1 }}\n",
    "![image.png](2020-11-26-lessons-and-future-directions_files/att_00004.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What do you think would be good idea for the WBS  research?\n",
    "\n",
    "Let me know in comments/twitter. I am also going to update this page from time to time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}