Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

#DeepI2P: Image-to-Point Cloud Registration via Deep Classification



PyTorch implementation for our CVPR 2021 paper DeepI2P. DeepI2P solves the problem of cross modality registration, i.e, solve the relative rotation R and translation t between the camera and the lidar.

DeepI2P: Image-to-Point Cloud Registration via Deep Classification
Jiaxin Li 1, Gim Hee Lee 2
1ByteDance, 2National University of Singapore


The intuition is to perform the Inverse Camera Projection, as shown in the images below. overview_1 overview_2

Repo Structure

  • data: Generate and process datasets
  • evaluation: Registration codes, include Inverse Camera Projection, ICP, PnP
    • frustum_reg: C++ codes of the Inverse Camera Projection, using Gauss-Newton Optimization. Installation method is shown below. It requires the Ceres Solver.
    python evaluation/frustum_reg/ install
    • icp: codes for ICP (Iterative Closest Point)
    • Python code for Inverse Camera Projection, which utilizes the per-point coarse classification prediction, and the frustum_reg solver.
    • Python code for PnP solver utilizing the per-point fine classification prediction.
  • kitti: Training codes for KITTI
  • nuscenes: Training codes for nuscenes
  • oxford: Training codes for Oxford Robotcar dataset
  • models: Networks and layers
    • 'index_max_ext': This is a custom operation from SO-Net, which is the backbone of our network. Installation:
    python models/index_max_ext/ install
    • Network to process images. It is a resnet-like structure.
    • Network to process point clouds, it is from SO-Net
    • Network to fuse information between point clouds and images.

Dataset and Models