Skip to content
/ awsome-3dcv-resources Public

总结3D视觉所涉及到的边边角角知识点，包括VSLAM、点云后处理、相机标定、深度学习等。

86 stars 18 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

qxiaofan/awsome-3dcv-resources

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
README.md
README.md
 
 

Repository files navigation

介绍

公众号：3D视觉工坊

主要关注：3D视觉算法、SLAM、vSLAM、计算机视觉、深度学习、自动驾驶、图像处理以及技术干货分享

运营者和嘉宾介绍：运营者来自国内一线大厂的算法工程师，深研3D视觉、深度学习、图像处理、自动驾驶、vSLAM等领域，特邀嘉宾包括国内外知名高校的博士硕士，旷视、商汤、百度、阿里等就职的算法大佬，欢迎一起交流学习

硬件

事件相机知识点汇总

相机标定

综述

  1. 线阵相机标定方法综述

单相机标定

  1. 相机标定误差因素分析
  2. Fully automatic camera calibration method based on circular markers基于圆形标志点的全自动相机标定方法
  3. Accurate camera calibration using iterative refinement of control points
  4. Accurate Feature Extraction and Control Point Correction for Camera Calibration with a Mono-Plane Target
  5. 基于主动红外辐射标定板的超广角红外相机标定
  6. 基于相位标靶的相机标定
  7. 基于广义成像模型的Scheimpflug相机标定方法
  8. 多几何约束下的鱼眼相机单像高精度标定

手眼标定

  1. 一种新的机器人手眼关系标定方法

其它

  1. 基于张正友标定法的红外靶标系统

3D视觉资源汇总

书籍

  1. 视觉测量[张广军]
  2. 机器人视觉测量与控制[徐德，谭民，李原]
  3. Machine Vision 2016: Automated Visual Inspection: Theory, Practice and Applications

资源

https://github.com/timzhang642/3D-Machine-Learning

https://github.com/sunglok/3dv_tutorial(涉及SLAM、多视图几何代码示例)

SLAM

优秀开源项目汇总

https://github.com/OpenSLAM/awesome-SLAM-list

https://github.com/tzutalin/awesome-visual-slam

Recent_SLAM_Research

https://github.com/youngguncho/awesome-slam-datasets

https://github.com/marknabil/SFM-Visual-SLAM

https://github.com/ckddls1321/SLAM_Resources

Books

Courses&&Lectures

Code

  1. ORB-SLAM
  2. LSD-SLAM
  3. ORB-SLAM2
  4. DVO: Dense Visual Odometry
  5. SVO: Semi-Direct Monocular Visual Odometry
  6. G2O: General Graph Optimization
  7. RGBD-SLAM
Project Language License
COSLAM C++ GNU General Public License
DSO-Direct Sparse Odometry C++ GPLv3
DTSLAM-Deferred Triangulation SLAM C++ modified BSD
LSD-SLAM C++/ROS GNU General Public License
MAPLAB-ROVIOLI C++/ROS Apachev2.0
OKVIS: Open Keyframe-based Visual-Inertial SLAM C++ BSD
ORB-SLAM C++ GPLv3
REBVO - Realtime Edge Based Visual Odometry for a Monocular Camera C++ GNU General Public License
SVO semi-direct Visual Odometry C++/ROS GNU General Public License

计算机视觉

资源汇总

Books

Courses

深度学习

Github link

1、https://github.com/ChristosChristofidis/awesome-deep-learning

2、https://github.com/endymecy/awesome-deeplearning-resources

机器学习

Github link

1、https://github.com/josephmisiti/awesome-machine-learning

3D点云

点云获取

传统的点云获取技术包括非接触式测量和接触式测量两种，它们的主要区别在于，在测量过程中测头是否与工件的表面相接触。

非接触式测量是利用光学原理的方法采集数据，例如结构光法、测距法以及干涉法等。该方法的优点在于测量速度较快、测量精度高，并且能够获得高密度点云数据，但其测量精度易受外界因素干扰，而且测量物 体表面的反射光与环境光对测量精度也有一定影响。

相反，接触式测量是通过将测头上的测量传感器与被测物体的外表面相接触，然后通过移动测头来读取物体表面点的三维坐标值。该方法的优点在于测头的结构相对固定，并且其测量结果不受被测物体表面的材料与表面特性等因素的影响。这种方法的不足在于，由于测头长期与被测物体表面相接触，易产生磨损，并且这种测量方式的测量速度较慢，不适合测量几何结构较复杂的物体。

点云应用场景

逆向工程、游戏人物重建、文物保护、数字博物馆、医疗辅助、三维城市建模

点云种类

不同的点云获取技术获取的点云数据类型不同，根据点云数据中点的分布情况可将点云数据划分为以下四种类型

散乱点云

散乱点云是指所有数据点在空间中以散乱状态分布，任意两点之间没有建立拓扑连接关系。一般而言，激光点测量系统获得的点云数据以及坐标测量机在随机扫描状态下获得的点云数据都为散乱点云数据。

扫描线点云

测量设备所获得的三维点云数据是由多条直线或曲线构成，点与点之间有一定的拓扑连接关系。一般而言，这种点云数据类型常见于扫描式点云数据中。

网格化点云

网格化点云是指点云数据中任意一点，均对应于其参数域所对应的一个均匀网格的顶点。当对空间散乱点云进行网格化插值时，所获得的点云数据即为网格化点云数据。

多边形点云

多边形点云是指分布在一组平面内的点云数据，该组平面内的平面两两互相平行，并且一个平面内距离最近的点连接起来可以形成平面多边形。这种点云数据常见于等高线测量、CT 测量等获得的点云数据中。

点云去噪&滤波

主要包括双边滤波、高斯滤波、条件滤波、直通滤波、随机采样一致滤波、VoxelGrid滤波等

三角网格去噪算法、

  1. 基于K-近邻点云去噪算法的研究与改进
  2. Point cloud denoising based on tensor Tucker decomposition
  3. 3D Point Cloud Denoising using Graph Laplacian Regularization of a Low Dimensional Manifold Model

有序点云去噪

孤立点排异法、曲线拟合法、弦高差法、全局能量法和滤波法.

孤立点排异法是通过观察点云数据，然后将与扫描线偏离较大的点剔除掉，从而达到去噪的目的。这类方法简单，可除去比较明显的噪声点，但缺点是只能对点云做初步的去噪处理，并不能滤除与真实点云数据混合在一起的噪声数据点。曲线拟合法是根据给定数据点的首末点，然后通过最小二乘等方法拟合一条曲线，通常为3到4 阶，最后计算中间的点到该曲线的距离，如果该距离大于给定阈值，则该点为噪声点，予以删 除，相反，如果该距离小于给定阈值，则该点为正常点，应该保留。弦高差法通过连接给定点集的首末点形成弦，然后求取中间每个点到该弦的距离，如果该距离小于给定阈值，则该点为正常点，予以保留，相反，若大于给定阈值，则该点为噪声点，予以删除。全局能量法通常用于网格式点云去噪，它通过建立整个曲面的能量方程，并求该方程在约束情况下的能量值的最小值。可以看出，这是一个全局最优化问题，因为网格数量比较大，因此会消耗大量的计算机资源与计算时间，而且由于约束方程是建立在整体网格的基础上，所以对于局部形状的去噪效果并不是很好。滤波法也是一种常用的有序点云去噪方法，它通过运用信号处理中的相关方法，使用合适的滤波函数对点云数据进行去噪处理，常用的滤波方法主要包括高斯滤波、均值滤波以及中值滤波法等。

无序点云去噪&空间散乱点云去噪算法

目前，针对空间散乱点云数据去噪方法，主要分为两类方法，即基于网格模型的去噪方法和直接对空间点云数据进行去噪的方法。

其中，基于网格模型的去噪方法需要首先建立点云的三角网格模型，然后计算所有三角面片的纵横比和顶点方向的曲率值，并将该值与相应的阈值进行比较，若小于阈值，则为正常点，予以保留，相反，则为噪声点，予以删除。由于该方法需要对空间点云数据进行三角网格剖分，所以，往往比较复杂，并需要大量计算。

点云精简

采用三维激光扫描仪获得的点云数据往往十分密集，点云数据中点的数量往往高达千万级甚至数亿级，即使对点云数据进行了去噪处理，点云数据中点的数量还是很多，所以往往不会直接使用这些原始点云数据进行曲面重建等工作，因为这会使后续处理过程变得耗时并且消耗过多的计算机资源，而且重构的曲面，其精度也不一定高，甚至出现更大的误差。所以，在进行空间点云曲面重建之前，往往需要对高密度的点云数据进 行点云精简操作。点云精简的目的是在保持原始点云的形状特征以及几何特征信息的前提下，尽量删除多余的数据点。

目前，空间散乱点云数据的精简方法主要分为两大类：基于三角网格模型的空间点云精简方法与直接基于数据点的空间点云精简方法。

其中，基于三角网格模型的空间点云精简方法需要先对点云数据进行三角剖分处理，建立其相应的三角网格拓扑结构，然后再对该三角网格进行处理，并将区域内那些形状变化较小的三角形进行合并，最后删除相关的三角网格顶点，从而达到点云数据精简的目的。这种方法需要对点云数据建立其相应的三角网格，该过程比较复杂，且因为需要存储网格数据，故需要消耗大量的计算机系统资源，并且该方法的抗噪能力较弱，对含有噪声的点云数据，构造的三角网格可能会出现变形等情况，因此精简后的点云数据经过曲面重建后的 模型与原始点云经过曲面重建后的模型可能大不相同。因此，目前关于直接基于点云数据的精简方法成为点云精简方法的主流。这种方法依据点云数据点之间的空间位置关系来建立点云的拓扑连接关系，并根据建立的拓扑连接关系计算点云数据中每个数据点的几何特征信息，最后根据这些特征信息来对点云数据进行点云精简处理。相比基于三角网格的空间点云精简方法，由于直接基于点云数据点的精简方法无需计算和存储复杂的三角网格结构，使得其精简的效率相对较高。因此，本章只研究直接基于空间点云数据的精简算法。

其中基于空间点云精简方法主要有：空间包围盒法、基于聚类的方法、法向偏差法、曲率精简法、平局点距法以及均匀栅格划分法。

Paper

  1. 点模型的几何图像简化法
  2. 基于相似性的点模型简化算法
  3. 基于最小曲面距离的快速点云精简算法
  4. 大规模点云选择及精简
  5. 一种基于模糊聚类的海量测量数据简化方法
  6. 基于均值漂移聚类的点模型简化方法
  7. 基于局部曲面拟合的散乱点云简化方法

点云关键点

常见的三维点云关键点提取算法有一下几种：ISS3D、Harris3D、NARF、SIFT3D，这些算法在PCL库中都有实现，其中NARF算法是用的比较多的

点云描述

如果要对一个三维点云进行描述，光有点云的位置是不够的，常常需要计算一些额外的参数，比如法线方向、曲率、文理特征等等。如同图像的特征一样，我们需要使用类似的方式来描述三维点云的特征。

常用的特征描述算法有：法线和曲率计算、特征值分析、PFH、FPFH、3D Shape Context、Spin Image等。PFH：点特征直方图描述子，FPFH：跨苏点特征直方图描述子，FPFH是PFH的简化形式。

点云线、面拟合

针对直线拟合：RANSAC算法、最小二乘法、平面相交法

针对曲线拟合：拉格朗日插值法、最小二乘法、Bezier曲线拟合法、B样条曲线法（二次、三次B样条曲线拟合）

针对平面拟合：主成成分分析、最小二乘法、粗差探测法、抗差估计法

针对曲面拟合：最小二乘法（正交最小二乘、移动最小二乘）、NURBS、 Bezier

  1. 三维激光扫描拟合平面自动提取算法
  2. 点云平面拟合新方法
  3. 海量散乱点的曲面重建算法研究
  4. 一种稳健的点云数据平面拟合方法
  5. 迭代切片算法在点云曲面拟合中的应用
  6. 基于最小二乘的点云叶面拟合算法研究
  7. 点云曲面边界线的提取

点云体积计算

基于三维点云求取物理模型体积的研究算法大致可分为以下 4 大类。

1.凸包算法：使用凸包模型近似表示不规则体，再通过把凸包模型切片分割进行累加、或将凸包模型分解为上下两个三角网格面，采用正投影法求取两者的投影体积，其差即所求体积。此方法适用于凸模型，非凸模型误差较大。

2.模型重建法：在得到点云数据后，使用三角面片构建物理模型的方法求得体积。该算法受点云密度、生成的三角网格数量、点精度影响较大，易产生孔洞。

3.切片法：将点云沿某一坐标轴方向进行切片处理，再计算切片上下两表面的面积，通过累加切片体积求得总体积。该方法受到切片厚度的影响，切片厚度越小，计算精度越高但会导致计算效率下降。

4.投影法：先将点云投影进行三角形剖分，再将投影点与其原对应点构建出五面体，通过累加五面体体积求得总体积。该算法同样容易产生孔洞。上述算法，无论是通过三维点云先构建物理模型再求体积、还是基于三维点云通过几何方法直接求体积，当激光雷达采集的三维点云存在密度不均匀、空间物体存在过渡带或过渡线等问题时，重建三维模型的误差较大，体积计算精度不高。

点云识别&分类

分类：基于点的分类，基于分割的分类，监督分类与非监督分类

  1. 3D ShapeNets: A Deep Representation for Volumetric Shapes
  2. PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding
  3. Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data
  4. Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models[ICCV2017]
  5. [ICCV2017] Colored Point Cloud Registration Revisited.
  6. [ICRA2017] SegMatch: Segment based place recognition in 3D point clouds.
  7. [IROS2017] 3D object classification with point convolution network.
  8. [CVPR2018] Pointwise Convolutional Neural Networks.
  9. [CVPR2018] SO-Net: Self-Organizing Network for Point Cloud Analysis.
  10. [CVPR2018] PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition.
  11. [CVPR2018] PointGrid: A Deep Network for 3D Shape Understanding.
  12. [CVPR2019] Spherical Fractal Convolutional Neural Networks for Point Cloud Recognition.
  13. [MM] MMJN: Multi-Modal Joint Networks for 3D Shape Recognition.

点云匹配&配准&对齐&注册

点云配准的概念也可以类比于二维图像中的配准，只不过二维图像配准获取得到的是x，y，alpha，beta等放射变化参数，三维点云配准可以模拟三维点云的移动和对齐，也就是会获得一个旋转矩阵和一个平移向量，通常表达为一个4×3的矩阵，其中3×3是旋转矩阵，1x3是平移向量。严格说来是6个参数，因为旋转矩阵也可以通过罗格里德斯变换转变成1*3的旋转向量。

常用的点云配准算法有两种：正太分布变换和著名的ICP点云配准，此外还有许多其它算法，列举如下：

ICP：稳健ICP、point to plane ICP、point to line ICP、MBICP、GICP

NDT 3D、Multil-Layer NDT

FPCS、KFPSC、SAC-IA

Line Segment Matching、ICL

  1. An ICP variant using a point-to-line metric
  2. Generalized-ICP
  3. Linear Least-Squares Optimization for Point-to-Plane ICP Surface Registration
  4. Metric-Based Iterative Closest Point Scan Matching for Sensor Displacement Estimation
  5. NICP: Dense Normal Based Point Cloud Registration
  6. Efficient Global Point Cloud Alignment using Bayesian Nonparametric Mixtures[CVPR2017]
  7. 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions[CVPR2017]
  8. [CVPR2018] Density Adaptive Point Set Registration.
  9. [CVPR2018] Inverse Composition Discriminative Optimization for Point Cloud Registration.
  10. [CVPR2018] PPFNet: Global Context Aware Local Features for Robust 3D Point Matching.
  11. [ECCV2018] Learning and Matching Multi-View Descriptors for Registration of Point Clouds.
  12. [ECCV2018] 3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration.
  13. [ECCV2018] Efficient Global Point Cloud Registration by Matching Rotation Invariant Features Through Translation Search.
  14. [IROS2018] Robust Generalized Point Cloud Registration with Expectation Maximization Considering Anisotropic Positional Uncertainties.
  15. [CVPR2019] PointNetLK: Point Cloud Registration using PointNet.
  16. [CVPR2019] SDRSAC: Semidefinite-Based Randomized Approach for Robust Point Cloud Registration without Correspondences.
  17. [CVPR2019] The Perfect Match: 3D Point Cloud Matching with Smoothed Densities.
  18. [CVPR] FilterReg: Robust and Efficient Probabilistic Point-Set Registration using Gaussian Filter and Twist Parameterization.
  19. [CVPR2019] 3D Local Features for Direct Pairwise Registration.
  20. [ICCV2019] DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration.
  21. [ICCV2019] Deep Closest Point: Learning Representations for Point Cloud Registration.
  22. [ICRA2019] 2D3D-MatchNet: Learning to Match Keypoints across 2D Image and 3D Point Cloud.
  23. [CVPR2019] The Perfect Match: 3D Point Cloud Matching with Smoothed Densities.
  24. [CVPR2019] 3D Local Features for Direct Pairwise Registration.
  25. [ICCV2019] Robust Variational Bayesian Point Set Registration.
  26. [ICRA2019] Robust low-overlap 3-D point cloud registration for outlier rejection.

点云匹配质量评估

  1. [IROS2017] Analyzing the quality of matched 3D point clouds of objects.

点云分割

点云的分割也算是一个大Topic了，这里因为多了一维就和二维图像比多了许多问题，点云分割又分为区域提取、线面提取、语义分割与聚类等。同样是分割问题，点云分割涉及面太广，确实是三言两语说不清楚的。只有从字面意思去理解了，遇到具体问题再具体归类。一般说来，点云分割是目标识别的基础。

分割主要有四种方法：基于边的区域分割、基于面的区域分割、基于聚类的区域分割、混合区域分割方法、深度学习方法

分割：区域声场、Ransac线面提取、NDT-RANSAC、K-Means（谱聚类）、Normalize Cut、3D Hough Transform(线面提取)、连通分析

  1. 基于局部表面凸性的散乱点云分割算法研究
  2. 三维散乱点云分割技术综述
  3. 基于聚类方法的点云分割技术的研究
  4. SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor
  5. From Planes to Corners: Multi-Purpose Primitive Detection in Unorganized 3D Point Clouds
  6. Learning and Memorizing Representative Prototypes for 3D Point Cloud Semantic and Instance Segmentation
  7. JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds
  8. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
  9. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
  10. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation,CVPR2017
  11. [ICRA2017] SegMatch: Segment based place recognition in 3D point clouds.
  12. [3DV2017] SEGCloud: Semantic Segmentation of 3D Point Clouds.
  13. [CVPR2018] Recurrent Slice Networks for 3D Segmentation of Point Clouds.
  14. [CVPR2018] SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation.
  15. [CVPR2018] Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs.
  16. [ECCV2018] 3D Recurrent Neural Networks with Context Fusion for Point Cloud Semantic Segmentation.
  17. [CVPR2019] JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields.
  18. [CVPR2019] PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation.
  19. [ICCV2019] 3D Instance Segmentation via Multi-Task Metric Learning.
  20. [IROS2019] PASS3D: Precise and Accelerated Semantic Segmentation for 3D Point Cloud.

点云目标检索

这是点云数据处理中一个偏应用层面的问题，简单说来就是Hausdorff距离常被用来进行深度图的目标识别和检索，现在很多三维人脸识别都是用这种技术来做的。

点云三维重建

我们获取到的点云数据都是一个个孤立的点，如何从一个个孤立的点得到整个曲面呢，这就是三维重建的topic。

在玩kinectFusion时候，如果我们不懂，会发现曲面渐渐变平缓，这就是重建算法不断迭代的效果。我们采集到的点云是充满噪声和孤立点的，三维重建算法为了重构出曲面，常常要应对这种噪声，获得看上去很舒服的曲面。

常用的三维重建算法和技术有：

泊松重建、Delauary triangulatoins(Delauary三角化)

表面重建，人体重建，建筑物重建，输入重建

实时重建：重建纸杯或者农作物4D生长台式，人体姿势识别，表情识别

  1. 改进的点云数据三维重建算法
  2. Scalable Surface Reconstruction from Point Clouds with Extreme Scale and Density Diversity,CVPR2017
  3. [ICCV2017] PolyFit: Polygonal Surface Reconstruction from Point Clouds.
  4. [ICCV2017] From Point Clouds to Mesh using Regression.
  5. [ECCV2018] Efficient Dense Point Cloud Object Reconstruction using Deformation Vector Fields.
  6. [ECCV2018] HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.
  7. [AAAI2018] Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction.
  8. [CVPR2019] Robust Point Cloud Based Reconstruction of Large-Scale Outdoor Scenes.
  9. [AAAI2019] CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision.
  10. [MM] L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention.
  11. SurfNet: Generating 3D shape surfaces using deep residual networks

点云其它

  1. [CVPR2018] Reflection Removal for Large-Scale 3D Point Clouds.
  2. [ICML2018] Learning Representations and Generative Models for 3D Point Clouds.
  3. [3DV] PCN: Point Completion Network.
  4. [CVPR2019] PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding.
  5. [CVPR2019] ClusterNet: Deep Hierarchical Cluster Network with Rigorously Rotation-Invariant Representation for Point Cloud Analysis.
  6. [ICCV2019] LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis.
  7. [ICRA2019] Speeding up Iterative Closest Point Using Stochastic Gradient Descent.

点云数据集

  1. [KITTI] The KITTI Vision Benchmark Suite.
  2. [ModelNet] The Princeton ModelNet .
  3. [ShapeNet] A collaborative dataset between researchers at Princeton, Stanford and TTIC.
  4. [PartNet] The PartNet dataset provides fine grained part annotation of objects in ShapeNetCore.
  5. [PartNet] PartNet benchmark from Nanjing University and National University of Defense Technology.
  6. [S3DIS] The Stanford Large-Scale 3D Indoor Spaces Dataset.
  7. [ScanNet] Richly-annotated 3D Reconstructions of Indoor Scenes.
  8. [Stanford 3D] The Stanford 3D Scanning Repository.
  9. [UWA Dataset] .
  10. [Princeton Shape Benchmark] The Princeton Shape Benchmark.
  11. [SYDNEY URBAN OBJECTS DATASET] This dataset contains a variety of common urban road objects scanned with a Velodyne HDL-64E LIDAR, collected in the CBD of Sydney, Australia. There are 631 individual scans of objects across classes of vehicles, pedestrians, signs and trees.
  12. [ASL Datasets Repository(ETH)] This site is dedicated to provide datasets for the Robotics community with the aim to facilitate result evaluations and comparisons.
  13. [Large-Scale Point Cloud Classification Benchmark(ETH)] This benchmark closes the gap and provides a large labelled 3D point cloud data set of natural scenes with over 4 billion points in total.
  14. [Robotic 3D Scan Repository] The Canadian Planetary Emulation Terrain 3D Mapping Dataset is a collection of three-dimensional laser scans gathered at two unique planetary analogue rover test facilities in Canada.
  15. [Radish] The Robotics Data Set Repository (Radish for short) provides a collection of standard robotics data sets.
  16. [IQmulus & TerraMobilita Contest] The database contains 3D MLS data from a dense urban environment in Paris (France), composed of 300 million points. The acquisition was made in January 2013.
  17. [Oakland 3-D Point Cloud Dataset] This repository contains labeled 3-D point cloud laser data collected from a moving platform in a urban environment.
  18. [Robotic 3D Scan Repository] This repository provides 3D point clouds from robotic experiments，log files of robot runs and standard 3D data sets for the robotics community.
  19. [Ford Campus Vision and Lidar Data Set] The dataset is collected by an autonomous ground vehicle testbed, based upon a modified Ford F-250 pickup truck.
  20. [The Stanford Track Collection] This dataset contains about 14,000 labeled tracks of objects as observed in natural street scenes by a Velodyne HDL-64E S2 LIDAR.
  21. [PASCAL3D+] Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild.
  22. [3D MNIST] The aim of this dataset is to provide a simple way to get started with 3D computer vision problems such as 3D shape recognition.
  23. [WAD] [ApolloScape] The datasets are provided by Baidu Inc.
  24. [nuScenes] The nuScenes dataset is a large-scale autonomous driving dataset.
  25. [PreSIL] Depth information, semantic segmentation (images), point-wise segmentation (point clouds), ground point labels (point clouds), and detailed annotations for all vehicles and people. [paper]
  26. [3D Match] Keypoint Matching Benchmark, Geometric Registration Benchmark, RGB-D Reconstruction Datasets.
  27. [BLVD] (a) 3D detection, (b) 4D tracking, (c) 5D interactive event recognition and (d) 5D intention prediction. [ICRA 2019 paper]
  28. [PedX] 3D Pose Estimation of Pedestrians, more than 5,000 pairs of high-resolution (12MP) stereo images and LiDAR data along with providing 2D and 3D labels of pedestrians. [ICRA 2019 paper]
  29. [H3D] Full-surround 3D multi-object detection and tracking dataset. [ICRA 2019 paper]
  30. [Matterport3D] RGB-D: 10,800 panoramic views from 194,400 RGB-D images. Annotations: surface reconstructions, camera poses, and 2D and 3D semantic segmentations. Keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and scene classification. [3DV 2017 paper] [code] [blog]
  31. [SynthCity] SynthCity is a 367.9M point synthetic full colour Mobile Laser Scanning point cloud. Nine categories.
  32. [Lyft Level 5] Include high quality, human-labelled 3D bounding boxes of traffic agents, an underlying HD spatial semantic map.
  33. [SemanticKITTI] Sequential Semantic Segmentation, 28 classes, for autonomous driving. All sequences of KITTI odometry labeled. [ICCV 2019 paper]
  34. [NPM3D] The Paris-Lille-3D has been produced by a Mobile Laser System (MLS) in two different cities in France (Paris and Lille).
  35. [The Waymo Open Dataset] The Waymo Open Dataset is comprised of high resolution sensor data collected by Waymo self-driving cars in a wide variety of conditions.
  36. [A*3D: An Autonomous Driving Dataset in Challeging Environments] A*3D: An Autonomous Driving Dataset in Challeging Environments.
  37. [PointDA-10 Dataset] Domain Adaptation for point clouds.
  38. [Oxford Robotcar] The dataset captures many different combinations of weather, traffic and pedestrians.

三维重建

单目图像

主要分为基于SfM三维重建和基于Deep learning的三维重建方法

paper

结构光

结构光投影三维成像目前是机器人3D 视觉感知的主要方式,结构光成像系统是由若干个投影仪和 相机组成, 常用的结构形式有: 单投影仪-单相机、单投影仪-双相机 、单投影仪-多相机、单相机-双投影 仪和单相机-多投影仪等典型结构形式.

结构光投影三维成像的基本工作原理是：投影仪向目标物体投射特定的结构光照明图案,由相机摄取被目标调制后的图像,再通过图像处理和视觉模型求出目标物体的三维信息. 常用的投影仪主要有下列几种类型:液晶投影(LCD)、数字光调制投影(DLP)[如数字微镜器件 (DMD)]、激光 LED图案直接投影. 根据结构光投影次数划分,结构光投影三维成像可以分成单次投影3D和多次投影3D方法.

按照扫描方式又可分为：线扫描结构光、面阵结构光

参考链接：https://zhuanlan.zhihu.com/p/29971801

结构光三维表面成像：综述（一）

结构光三维表面成像：综述（二）

结构光三维表面成像：综述（三）

综述

Structured-light 3D surface imaging: a tutorial

机器人视觉三维成像技术综述

Real-time structured light profilometry a review

A state of the art in structured light patterns for surface profilometry

Phase shifting algorithms for fringe projection profilometry: a review

Overview of the 3D profilometry of phase shifting fringe projection

Temporal phase unwrapping algorithms for fringe projection profilometry:a comparative review

Lectures&Video

  1. Build Your Own 3D Scanner: Optical Triangulation for Beginners
  2. https://github.com/nikolaseu/thesis
  3. CS6320 3D Computer Vision, Spring 2015

标定

  1. 高效线结构光视觉测量系统标定方法
  2. 一种新的线结构光标定方法
  3. 一种结构光三维成像系统的简易标定方法
  4. 基于单应性矩阵的线结构光系统简易标定方法
  5. 线结构光标定方法综述
  6. 三线结构光视觉传感器现场标定方法
  7. 单摄像机单投影仪结构光三维测量系统标定方法
  8. 超大尺度线结构光传感器内外参数同时标定
  9. 单摄像机单投影仪结构光三维测量系统标定方法
  10. 三维空间中线结构光与相机快速标定方法
  11. 线结构光传感系统的快速标定方法

单次投影成像

单次投影结构光主要采用空间复用编码和频率复用编码形式实现 ,常用的编码形式有:彩色编码 、灰度索引、 几何形状编码和随机斑点. 目前在机器人手眼系统应用中,对于三维测量精度要求不高的场合,如码垛、拆垛、三维抓取等,比较受欢迎的是投射伪随机斑点获得目标三维信息 。

  1. One-shot pattern projection for dense and accurate 3D acquisition in structured light

  2. A single-shot structured light means by encoding both color and geometrical features

  3. Dynamic 3D surface profilometry using a novel colour pattern encoded with a multiple triangular mode

  4. Review of single-shot 3D shape measurement by phase calculation-based fringe projection techniques

  5. Robust pattern decoding in shape-coded structured light

多次投影成像

多次投影3D方法主要采用时间复用编码方式实现,常用的图案编码形式有:二进制编码、多频相移编码和混合编码法(如格雷码＋相移条纹)等.

但是格雷码方法仅能在投射空间内进行离散的划分,空间分辨率受到成像器件的限制. 为了提高空间分辨率,需要增加投影条纹幅数,投射条纹宽度更小的格雷码条纹图,但条纹宽度过小会导致格雷码条纹的边缘效应,从而引 起解码误差.

正弦光栅条纹投影克服了格雷码空间离散划分的缺点,成为使用率最高的结构光类型之一. 众所周知,对于复杂外形,如有空洞、阶梯、遮挡等,采用正弦单频相移法条纹投影时,存在相位解包裹难题.另外为了能够从系列条纹图中求出相位绝对值,需要在条纹中插入特征点,比如一个点、一条线作为参考相位点,但是这些点或线特征标志有可能投影在物体的遮挡或阴影区域,或受到环境光等干扰等,发生丢失,影响测量结果的准确性. 因此,对于复杂轮廓的物体,常采用多频相移技术.

  1. 三维重建的格雷码-相移光编码技术研究
  2. Pattern codification strategies in structured light systems
  3. Binary coded linear fringes for three-dimensional shape profiling
  4. 3D shape measurement based on complementary Gray-code light
  5. Phase shifting algorithms for fringe projection profilometry: a review
  6. Overview of the 3D profilometry of phase shifting fringe projection
  7. Temporal phase unwrapping algorithms for fringe projection profilometry:a comparative review
  8. A multi-frequency inverse-phase error compensation method for projectornon linear in3D shape measurement
  9. 双目线激光扫描三维重建技术及其在工业测量中的应用
  10. Simple, Accurate, and Robust Projector-Camera Calibration
  11. High-accuracy, high-speed 3D structured light imaging techniques and potential applications to intelligent robotics
  12. Flexible 3D shape measurement using projector defocusing: extended measurement rang
  13. High-quality 3D shape measurement using saturated fringe patterns

偏折法成像

对于粗糙表面,结构光可以直接投射到物体表面进行视觉成像测量;但对于大反射率光滑表面和镜面物体3D 测量,结构光投影不能直接投射到被测表面,3D测量还需要借助镜面偏折技术 .

  1. Principles of shape from specular reflection
  2. Deflectometry: 3D-metrology from nanometer to meter
  3. Three-dimensional shape measurement of a highly reflected specular surface with structured light method
  4. Three-dimensional shape measurements of specular objects using phase-measuring deflectometry

由于单次投影曝光和测量时间短,抗振动性能好,适合运动物体的3D测量,如机器人实时运动引导,手眼机器人对生产线上连续运动产品进行抓取等操作. 但深度垂直方向上的空间分辨率受到目标视场、镜头倍率和相机像素等因素的影响,大视场情况下不容易提升.

多次投影方法(如多频条纹方法)具有较高空间分辨率,能有效地解决表面斜率阶跃变化和空洞等难题. 不足之处在于:① 对于连续相移投影方法,3D重构的精度容易受到投影仪、相机的非线性和环境变化的影响;②抗振动性能差,不合适测量连续运动的物体;③在 Eye-in-Hand视觉导引系统中,机械臂不易在连续运动时进行3D成像和引导;④实时性差,不过随着投影仪投射频率和 CCD/CMOS图像传感器采集速度的提高,多次投影方法实时3D 成像的性能也在逐步改进.

偏折法对于复杂面型的测量,通常需要借助多次投影方法,因此具有多次投影方法相同的缺点.另外偏折法对曲率变化大的表面测量有一定的难度,因为条纹偏折后的反射角的变化率是被测表面曲率变化率的２倍,因此对被测物体表面的曲率变化比较敏感,很容易产生遮挡难题.

Other Papers

  1. 基于面结构光的三维重建阴影补偿算法
  2. Enhanced phase measurement profilometry for industrial 3D inspection automation
  3. Profilometry of three-dimensional discontinuous solids by combining two-steps temporal phase unwrapping, co-phased profilometry and phase-shifting interferometry
  4. 360-Degree Profilometry of Discontinuous Solids Co-Phasing ２-Projectors and１-Camera
  5. Coherent digital demodulation of single-camera N-projections for 3D-object shape measurement Co-phased profilometr
  6. High-speed 3D image acquisition using coded structured light projection
  7. Accurate 3D measurement using a Structured Light System
  8. Structured light stereoscopic imaging with dynamic pseudo-random patterns
  9. Robust one-shot 3D scanning using loopy belief propagation
  10. Robust Segmentation and Decoding of a Grid Pattern for Structured Light
  11. Rapid shape acquisition using color structured light and multi-pass dynamic programming
  12. Improved stripe matching for colour encoded structured light
  13. Absolute phase mapping for one-shot dense pattern projection
  14. 3D digital stereophotogrammetry: a practical guide to facial image acquisition
  15. Method and apparatus for 3D imaging using light pattern having multiple sub-patterns
  16. High speed laser three-dimensional imager
  17. Three-dimensional dental imaging method and apparatus having a reflective member
  18. 3D surface profile imaging method and apparatus using single spectral light condition
  19. Three-dimensional surface profile imaging method and apparatus using single spectral light condition
  20. High speed three dimensional imaging method
  21. A hand-held photometric stereo camera for 3-D modeling
  22. High-resolution, real-time 3D absolute coordinate measurement based on a phase-shifting method
  23. A fast three-step phase shifting algorithm
  24. 基于相位测量轮廓术的三维重建系统标定研究

Code

  1. https://github.com/jakobwilm/slstudio
  2. https://github.com/phreax/structured_light
  3. https://github.com/nikolaseu/neuvision
  4. https://github.com/pranavkantgaur/3dscan

几个不错的开源网站：

  1. http://mesh.brown.edu/calibration/
  2. https://www.3dunderworld.org/about/

扫描3D成像

扫描3D成像方法可分为扫描测距、主动三角法、色散共焦法等。扫描3D成像的最大优点是测量精度高,其中 色散共焦法还有其他方法难以比拟的优点,即非常适合测量透明物体、高反与光滑表面的物体. 但缺点是速度慢、效率低;当用于机械手臂末端时,可实现高精度3D测量,但不适合机械手臂实时3D引导与定位,因此应用场合有限;另外主动三角扫描在测量复杂结构形貌时容易产生遮挡,需要通过合理规划末端路径与姿态来解决.

扫描测距

扫描测距是利用一条准直光束通过一维测距扫描整个目标表面实现3D测量，主要包括：单点飞行时间法、激光散射干涉法、 共焦法。

单点测距扫描3D方法中,单点飞行时间法适合远距离扫描,测量精度较低,一般在毫米量级. 其他几种单点扫描方法有:单点激光干涉法、共焦法和单点激光主动三角法,测量精度较高,但前者对环境要求高;线扫描精度适中,效率高. 比较适合于机械手臂末端执行3D测量的应是主动激光三角法和色散共焦法.

Paper
  1. Active optical range imaging sensor
  2. Active and passive range sensing for robotics

主动三角法

主动三角法是基于三角测量原理,利用准直光束、一条或多条平面光束扫描目标表面完成3D测量的. 光束常采用以下方式获得:激光准直、圆柱或二次曲面柱形棱角扩束,非相干光(如白光、LED 光源)通过小孔、狭缝(光栅)投影或相干光衍射等. 主动三角法可分为三种类型:单点扫描、单线扫描和多线扫描

Paper
  1. Review of different 3D scanners and scanning techniques
  2. 3D metrology using a collaborative robot with a laser triangulation sensor
  3. Introductory review on Flying Triangulation a motion-robust optical 3D measurement principle
  4. Flying triangulation an optical 3D sensor for the motion-robust acquisition of complex object
  5. Hand-Guided 3D Surface Acquisition by Combining Simple Light Sectioning with Real-Time Algorithms

色彩共焦法

色散共焦似乎可以扫描测量粗糙和光滑的不透明和透明物体,如反射镜面、透明玻璃面等,目前在手机盖板三维检测等领域广受欢迎。色散共焦扫描有三种类型:单点一维绝对测距扫描、多点阵列扫描和连续线扫描。

Paper
  1. Spectral characteristics of chromatic confocal imaging systems
  2. Spectrally multiplexed chromatic confocal multipoint sensing
  3. Chromatic confocal matrix sensor with actuated pinhole arrays
  4. Multiplex acquisition approach for high speed 3d measurements with a chromatic confocal microscope
  5. Fast 3D in line-sensor for specular and diffuse surfaces combining the chromatic confocal and triangulation principle
  6. Single-shot depth-section imaging through chromatic slit-scan confocal microscopy
  7. Three-dimensional surface profile measurement using a beam scanning chromatic confocal microscope

立体视觉3D成像

立体视觉字面意思是用一只眼睛或两只眼睛感知三维结构,一般情况下是指从不同的视点获取两 幅或多幅图像重构目标物体3D结构或深度信息. 深度感知视觉线索可分为 Monocular cues 和 Binocular cues(双目视差). 目前立体视觉3D 可以通过单目视觉、双目视觉、多 (目) 视觉、光场3D 成像(电子复眼或阵列相机)实现.

书籍

  1. 机器视觉 Robot Vision

教程

  1. 立体视觉书籍推荐&立体匹配十大概念综述---立体匹配算法介绍
  2. 【关于立体视觉的一切】立体匹配成像算法BM，SGBM，GC，SAD一览
  3. StereoVision--立体视觉（1）
  4. StereoVision--立体视觉（2）
  5. StereoVision--立体视觉（3）
  6. StereoVision--立体视觉（4）
  7. StereoVision--立体视觉（5）

综述

  1. Review of Stereo Vision Algorithms: From Software to Hardware
  2. 双目立体视觉的研究现状及进展

单目视觉成像

单目视觉深度感知线索通常有:透视、焦距差异 、多视觉成像、覆盖、阴影 、运动视差等.

  1. Depth map extracting based on geometric perspective an applicable２D to３D conversion technology
  2. Focus cues affect perceived depth
  3. 3D image acquisition system based on shape from focus technique
  4. Multi-view stereo: a tutorial
  5. 3D reconstruction from multiple images part1 principles
  6. Three-dimensional reconstruction of hybrid surfaces using perspective shape from shading
  7. Numerical methods for shape-from-shading a new survey with benchmarks
  8. The neural basis of depth perception from motion parallax
  9. Motion parallax in stereo 3D
  10. 3D image sensor based on parallax motion

双目视觉

在机器视觉里利用两个相机从两个视点对同一个目标场景获取两个视点图像,再计算两个视点图像中同名点的视差获得目标场景的3D深度信息. 典型的双目立体视觉计算过程包含下面四个步骤:图像畸变矫正、立体图像对校正、图像配准和三角法重投影视差图计算.

双目视觉的难点：

1、光照敏感，被动光

2、双目视觉系统估计视差没那么容易，立体匹配是计算机视觉典型的难题，基线宽得到远目标测距准，而基线短得到近目标测距结果好。谈到双目系统的难点，除了立体匹配，还有标定。标定后的系统会出现“漂移”的，所以在线标定是必须具有的。

综述

  1. 双目立体视觉匹配技术综述

视差和深度计算

  1. Real-time depth computation using stereo imaging
  2. Binocular disparity and the perception of depth

立体匹配

匹配方法分两种，全局法和局部法，实用的基本是局部法，因为全局法太慢。

（一）基于全局约束的立体匹配算法：在本质上属于优化算法，它是将立体匹配问题转化为寻找全局能量函数的最优化问题，其代表算法主要有图割算法、置信度传播算法和协同优化算法等．全局算法能够获得较低的总误匹配率，但算法复杂度较高,很难满足实时的需求，不利于在实际工程中使用，常见的算法有DP、BP 等。

（二）基于局部约束的立体匹配算法：主要是利用匹配点周围的局部信息进行计算，由于其涉及到的信息量较少，匹配时间较短，因此受到了广泛关注，其代表算法主要有 SAD、SSD、ZSAD、NCC等。

  1. DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch
  2. Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning
  3. PMSC: PatchMatch-Based Superpixel Cut for Accurate Stereo Matching
  4. Exact Bias Correction and Covariance Estimation for Stereo Vision
  5. Efficient minimal-surface regularization of perspective depth maps in variational stereo
  6. Event-Driven Stereo Matching for Real-Time 3D Panoramic Vision
  7. Leveraging Stereo Matching with Learning-based Confidence Measures
  8. Graph Cut based Continuous Stereo Matching using Locally Shared Labels
  9. Cross-Scale Cost Aggregation for Stereo Matching
  10. Fast Cost-Volume Filtering for Visual Correspondence and Beyond
  11. Constant Time Weighted Median Filtering for Stereo Matching and Beyond
  12. A non-local cost aggregation method for stereo matching
  13. On building an accurate stereo matching system on graphics hardware
  14. Efficient large-scale stereo matching
  15. Accurate, dense, and robust multiview stereopsis
  16. A constant-space belief propagation algorithm for stereo matching
  17. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling
  18. Cost aggregation and occlusion handling with WLS in stereo matching
  19. Stereo matching: An outlier confidence approach
  20. A region based stereo matching algorithm using cooperative optimization
  21. Multi-view stereo for community photo collections
  22. A performance study on different cost aggregation approaches used in real-time stereo matching
  23. Evaluation of cost functions for stereo matching
  24. Adaptive support-weight approach for correspondence search
  25. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure

多目视觉

多(目)视觉成像,也称多视点立体成像,用单个或多个相机从多个视点获取同一个目标场景的多幅图像,重构目标场景的三维信息.

  1. Adaptive structure from motion with a contrario model estimation
  2. A comparison and evaluation of multi-view stereo reconstruction algorithms
  3. Multiple view geometry in computer vision

光场成像

光场3D成像的原理与传统 CCD和 CMOS相机成像原理在结构原理上有所差异,传统相机成像是光线穿过镜头在后续的成像平面上直接成像,一般是2D图像;光场相机成像是在传感器平面前增加了一个微透镜阵列,将经过主镜头入射的光线再次穿过每个微透镜,由感光阵列接收,从而获得光线的方向与位置信息,使成像结果可在后期处理,达到先拍照,后聚焦的效果.

光场相机的优点是:单个相机可以进行3D成像,横向和深度方向的空间分辨率可以达到20μm到 mm 量级,景深比普通相机大好几倍,比较适合Eye-in-Hand系统3D测量与引导,但目前精度适中的商业化光场相机价格昂贵.

  1. Light field imaging models calibrations reconstructions and applications
  2. Extracting depth information from stereo vision system using a correlation and a feature based methods
  3. 基于微透镜阵列型光场相机的多目标快速测距方法
  4. 基于光场相机的四维光场图像水印及质量评价
  5. 基于光场相机的深度面光场计算重构
  6. 光场相机视觉测量误差分析
  7. 一种基于光场图像的聚焦光场相机标定方法
  8. 光场相机成像模型及参数标定方法综述

SFM

SFM（Structure From Motion），主要基于多视觉几何原理，用于从运动中实现3D重建，也就是从无时间序列的2D图像中推算三维信息，是计算机视觉学科的重要分支。

使用同一相机在其内参数不变的条件下,从不同视点获取多幅图像,重构目标场景的三维信息. 该技术常用 于跟踪目标场景中大量的控制点,连续恢复场景3D结构信息、相机的姿态和位置.

Incremental SfM

Global SfM

Hierarchical SfM

Multi-Stage SfM

Non Rigid SfM

Turtorial

  1. Open Source Structure-from-Motion. M. Leotta, S. Agarwal, F. Dellaert, P. Moulon, V. Rabaud. CVPR 2015 Tutorial (material).
  2. Large-scale 3D Reconstruction from Images](https://home.cse.ust.hk/~tshenaa/sub/ACCV2016/ACCV_2016_Tutorial.html). T. Shen, J. Wang, T.Fang, L. Quan. ACCV 2016 Tutorial.

Incremental SfM

  1. Photo Tourism: Exploring Photo Collections in 3D. N. Snavely, S. M. Seitz, and R. Szeliski. SIGGRAPH 2006.
  2. Towards linear-time incremental structure from motion. C. Wu. 3DV 2013.
  3. Structure-from-Motion Revisited. Schöenberger, Frahm. CVPR 2016.

Global SfM

  1. Combining two-view constraints for motion estimation V. M. Govindu. CVPR, 2001.
  2. Lie-algebraic averaging for globally consistent motion estimation. V. M. Govindu. CVPR, 2004.
  3. Robust rotation and translation estimation in multiview reconstruction. D. Martinec and T. Pajdla. CVPR, 2007.
  4. Non-sequential structure from motion. O. Enqvist, F. Kahl, and C. Olsson. ICCV OMNIVIS Workshops 2011.
  5. Global motion estimation from point matches. M. Arie-Nachimson, S. Z. Kovalsky, I. KemelmacherShlizerman, A. Singer, and R. Basri. 3DIMPVT 2012.
  6. Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. P. Moulon, P. Monasse and R. Marlet. ICCV 2013.
  7. A Global Linear Method for Camera Pose Registration. N. Jiang, Z. Cui, P. Tan. ICCV 2013.
  8. Global Structure-from-Motion by Similarity Averaging. Z. Cui, P. Tan. ICCV 2015.
  9. Linear Global Translation Estimation from Feature Tracks Z. Cui, N. Jiang, C. Tang, P. Tan, BMVC 2015.

Hierarchical SfM

  1. Structure-and-Motion Pipeline on a Hierarchical Cluster Tree. A. M.Farenzena, A.Fusiello, R. Gherardi. Workshop on 3-D Digital Imaging and Modeling, 2009.
  2. Randomized Structure from Motion Based on Atomic 3D Models from Camera Triplets. M. Havlena, A. Torii, J. Knopp, and T. Pajdla. CVPR 2009.
  3. Efficient Structure from Motion by Graph Optimization. M. Havlena, A. Torii, and T. Pajdla. ECCV 2010.
  4. Hierarchical structure-and-motion recovery from uncalibrated images. Toldo, R., Gherardi, R., Farenzena, M. and Fusiello, A.. CVIU 2015.

Multi-Stage SfM

  1. Parallel Structure from Motion from Local Increment to Global Averaging. S. Zhu, T. Shen, L. Zhou, R. Zhang, J. Wang, T. Fang, L. Quan. arXiv 2017.
  2. Multistage SFM : Revisiting Incremental Structure from Motion. R. Shah, A. Deshpande, P. J. Narayanan. 3DV 2014. -> Multistage SFM: A Coarse-to-Fine Approach for 3D Reconstruction, arXiv 2016.
  3. HSfM: Hybrid Structure-from-Motion. H. Cui, X. Gao, S. Shen and Z. Hu, ICCV 2017.

Non Rigid SfM

  1. Robust Structure from Motion in the Presence of Outliers and Missing Data. G. Wang, J. S. Zelek, J. Wu, R. Bajcsy. 2016.

Project&code

Project Language License
Bundler C++ GNU General Public License - contamination
Colmap C++ BSD 3-clause license - Permissive
TeleSculptor C++ BSD 3-Clause license - Permissive
MicMac C++ CeCILL-B
MVE C++ BSD 3-Clause license + parts under the GPL 3 license
OpenMVG C++ MPL2 - Permissive
OpenSfM Python Simplified BSD license - Permissive
TheiaSfM C++ New BSD license - Permissive

TOF

飞行时间 (TOF) 相机每个像素利用光飞行的时间差来获取物体的深度。TOF成像可用于大视野、远距离、低精度、低成本的3D图像采集. 其特点是:检测速度快、视野范围较大、工作距离远、价格便宜,但精度低,易受环境 光的干扰 。

分类

直接TOF

D-TOF通常用于单点测距系统, 为了实现面积范围3D成像,通常需要采用扫描技术 。

间接TOF

间接 TOF(I-TOF),时间往返行程是从光强度的时间选通测量中间接外推获得 ，I-TOF不需要精确的 计时,而是采用时间选通光子计数器或电荷积分器,它们可以在像素级实现.

教程

  1. ToF技术是什么？和结构光技术又有何区别?
  2. 3D相机--TOF相机

Paper

  1. https://arxiv.org/pdf/1511.07212.pdf)

Multi-view Stereo

多视角立体视觉（Multiple View Stereo，MVS）是对立体视觉的推广，能够在多个视角（从外向里）观察和获取景物的图像，并以此完成匹配和深度估计。某种意义上讲，SLAM/SFM其实和MVS是类似的，只是前者是摄像头运动，后者是多个摄像头视角。也可以说，前者可以在环境里面“穿行”，而后者更像在环境外“旁观”。

多视角立体视觉的pipelines如下：

  1. 收集图像；
  2. 针对每个图像计算相机参数；
  3. 从图像集和相应的摄像机参数重建场景的3D几何图形；
  4. 可选择地重建场景的形状和纹理颜色。

参考链接：多视角立体视觉MVS简介

paper

  1. Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume
  2. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
  3. Point-Based Multi-View Stereo Network
  4. Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference
  5. NRMVS: Non-Rigid Multi-View Stereo
  6. Multi-View Stereo 3D Edge Reconstruction
  7. Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference

综述

  1. Multi-view stereo: A tutorial
  2. State of the Art 3D Reconstruction Techniques N. Snavely, Y. Furukawa, CVPR 2014 tutorial slides. Introduction MVS with priors - Large scale MVS

Point cloud computation（点云计算）

  1. Accurate, Dense, and Robust Multiview Stereopsis. Y. Furukawa, J. Ponce. CVPR 2007. PAMI 2010
  2. State of the art in high density image matching. F. Remondino, M.G. Spera, E. Nocerino, F. Menna, F. Nex . The Photogrammetric Record 29(146), 2014.
  3. Progressive prioritized multi-view stereo. A. Locher, M. Perdoch and L. Van Gool. CVPR 2016.
  4. Pixelwise View Selection for Unstructured Multi-View Stereo. J. L. Schönberger, E. Zheng, M. Pollefeys, J.-M. Frahm. ECCV 2016.
  5. TAPA-MVS: Textureless-Aware PAtchMatch Multi-View Stereo. A. Romanoni, M. Matteucci. ICCV 2019

Surface computation & refinements（曲面计算与优化）

  1. Efficient Multi-View Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts. P. Labatut, J-P. Pons, R. Keriven. ICCV 2007
  2. Multi-View Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh. S. N. Sinha, P. Mordohai and M. Pollefeys. ICCV 2007.
  3. Towards high-resolution large-scale multi-view stereo. H.-H. Vu, P. Labatut, J.-P. Pons, R. Keriven. CVPR 2009.
  4. Refinement of Surface Mesh for Accurate Multi-View Reconstruction. R. Tylecek and R. Sara. IJVR 2010.
  5. High Accuracy and Visibility-Consistent Dense Multiview Stereo. H.-H. Vu, P. Labatut, J.-P. Pons, R. Keriven. Pami 2012.
  6. Exploiting Visibility Information in Surface Reconstruction to Preserve Weakly Supported Surfaces M. Jancosek et al. 2014.
  7. A New Variational Framework for Multiview Surface Reconstruction. B. Semerjian. ECCV 2014.
  8. Photometric Bundle Adjustment for Dense Multi-View 3D Modeling. A. Delaunoy, M. Pollefeys. CVPR2014.
  9. Global, Dense Multiscale Reconstruction for a Billion Points. B. Ummenhofer, T. Brox. ICCV 2015.
  10. Efficient Multi-view Surface Refinement with Adaptive Resolution Control. S. Li, S. Yu Siu, T. Fang, L. Quan. ECCV 2016.
  11. Multi-View Inverse Rendering under Arbitrary Illumination and Albedo, K. Kim, A. Torii, M. Okutomi, ECCV2016.
  12. Shading-aware Multi-view Stereo, F. Langguth and K. Sunkavalli and S. Hadap and M. Goesele, ECCV 2016.
  13. Scalable Surface Reconstruction from Point Clouds with Extreme Scale and Density Diversity, C. Mostegel, R. Prettenthaler, F. Fraundorfer and H. Bischof. CVPR 2017.
  14. Multi-View Stereo with Single-View Semantic Mesh Refinement, A. Romanoni, M. Ciccone, F. Visin, M. Matteucci. ICCVW 2017

Machine Learning based MVS

  1. Matchnet: Unifying feature and metric learning for patch-based matching, X. Han, Thomas Leung, Y. Jia, R. Sukthankar, A. C. Berg. CVPR 2015.
  2. Stereo matching by training a convolutional neural network to compare image patches, J., Zbontar, and Y. LeCun. JMLR 2016.
  3. Efficient deep learning for stereo matching, W. Luo, A. G. Schwing, R. Urtasun. CVPR 2016.
  4. Learning a multi-view stereo machine, A. Kar, C. Häne, J. Malik. NIPS 2017.
  5. Learned multi-patch similarity, W. Hartmann, S. Galliani, M. Havlena, L. V. Gool, K. Schindler.I CCV 2017.
  6. Surfacenet: An end-to-end 3d neural network for multiview stereopsis, Ji, M., Gall, J., Zheng, H., Liu, Y., Fang, L. ICCV2017.
  7. DeepMVS: Learning Multi-View Stereopsis, Huang, P. and Matzen, K. and Kopf, J. and Ahuja, N. and Huang, J. CVPR 2018.
  8. RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials, D. Paschalidou and A. O. Ulusoy and C. Schmitt and L. Gool and A. Geiger. CVPR 2018.
  9. MVSNet: Depth Inference for Unstructured Multi-view Stereo, Y. Yao, Z. Luo, S. Li, T. Fang, L. Quan. ECCV 2018.
  10. Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency, T. Khot, S. Agrawal, S. Tulsiani, C. Mertz, S. Lucey, M. Hebert. 2019.
  11. DPSNET: END-TO-END DEEP PLANE SWEEP STEREO, Sunghoon Im, Hae-Gon Jeon, Stephen Lin, In So Kweon. 2019.
  12. Point-based Multi-view Stereo Network, Rui Chen, Songfang Han, Jing Xu, Hao Su. ICCV 2019.

Multiple View Mesh Texturing（多视图网格纹理）

  1. Seamless image-based texture atlases using multi-band blending. C. Allène, J-P. Pons and R. Keriven. ICPR 2008.
  2. Let There Be Color! - Large-Scale Texturing of 3D Reconstructions. M. Waechter, N. Moehrle, M. Goesele. ECCV 2014

Texture Mapping(纹理贴图)

  1. 3D Textured Model Encryption via 3D Lu Chaotic Mapping

Courses

Software

Project&code

Project Language License
Colmap C++ CUDA BSD 3-clause license - Permissive (Can use CGAL -> GNU General Public License - contamination)
GPUIma + fusibile C++ CUDA GNU General Public License - contamination
HPMVS C++ GNU General Public License - contamination
MICMAC C++ CeCILL-B
MVE C++ BSD 3-Clause license + parts under the GPL 3 license
OpenMVS C++ (CUDA optional) AGPL3
PMVS C++ CUDA GNU General Public License - contamination
SMVS Shading-aware Multi-view Stereo C++ BSD-3-Clause license

3D人脸重建

1、Nonlinear 3D Face Morphable Model

2、On Learning 3D Face Morphable Model from In-the-wild Images

3、Cascaded Regressor based 3D Face Reconstruction from a Single Arbitrary View Image

4、JointFace Alignment and 3D Face Reconstruction

5、Photo-Realistic Facial Details Synthesis From Single Image

6、FML: Face Model Learning from Videos

7、Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric

8、Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network

9、Joint 3D Face Reconstruction and Dense Face Alignment from A Single Image with 2D-Assisted Self-Supervised Learning

10、Face Alignment Across Large Poses: A 3D Solution

纹理/材料分析与合成

  1. Texture Synthesis Using Convolutional Neural Networks (2015)[Paper]
  2. Two-Shot SVBRDF Capture for Stationary Materials (SIGGRAPH 2015) [Paper]
  3. Reflectance Modeling by Neural Texture Synthesis (2016) [Paper]
  4. Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks (2017)[Paper]
  5. High-Resolution Multi-Scale Neural Texture Synthesis (2017) [Paper]
  6. Reflectance and Natural Illumination from Single Material Specular Objects Using Deep Learning (2017) [Paper]
  7. Joint Material and Illumination Estimation from Photo Sets in the Wild (2017) [Paper]
  8. TextureGAN: Controlling Deep Image Synthesis with Texture Patches (2018 CVPR) [Paper]
  9. Gaussian Material Synthesis (2018 SIGGRAPH) [Paper]
  10. Non-stationary Texture Synthesis by Adversarial Expansion (2018 SIGGRAPH) [Paper]
  11. Synthesized Texture Quality Assessment via Multi-scale Spatial and Statistical Texture Attributes of Image and Gradient Magnitude Coefficients (2018 CVPR) [Paper]
  12. LIME: Live Intrinsic Material Estimation (2018 CVPR) [Paper]
  13. Learning Material-Aware Local Descriptors for 3D Shapes (2018) [Paper]

场景合成/重建

  1. Make It Home: Automatic Optimization of Furniture Arrangement (2011, SIGGRAPH) [Paper]
  2. Interactive Furniture Layout Using Interior Design Guidelines (2011) [Paper]
  3. Synthesizing Open Worlds with Constraints using Locally Annealed Reversible Jump MCMC (2012) [Paper]
  4. Example-based Synthesis of 3D Object Arrangements (2012 SIGGRAPH Asia) [Paper]
  5. Sketch2Scene: Sketch-based Co-retrieval and Co-placement of 3D Models (2013) [Paper]
  6. Action-Driven 3D Indoor Scene Evolution (2016) [Paper]
  7. The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes (2015) [Paper]
  8. Relationship Templates for Creating Scene Variations (2016) [Paper]
  9. IM2CAD (2017) [Paper]
  10. Predicting Complete 3D Models of Indoor Scenes (2017) [Paper]
  11. Complete 3D Scene Parsing from Single RGBD Image (2017) [Paper]
  12. Adaptive Synthesis of Indoor Scenes via Activity-Associated Object Relation Graphs (2017 SIGGRAPH Asia) [Paper]
  13. Automated Interior Design Using a Genetic Algorithm (2017) [Paper]
  14. SceneSuggest: Context-driven 3D Scene Design (2017) [Paper]
  15. A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition (2017)[Paper]
  16. Human-centric Indoor Scene Synthesis Using Stochastic Grammar (2018, CVPR)[Paper] [Supplementary] [Code]
  17. FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans (2018) [Paper] [Code]
  18. ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans (2018) [Paper]
  19. Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars (2018) [Paper]
  20. Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image (ECCV 2018) [Paper]
  21. Automatic 3D Indoor Scene Modeling from Single Panorama (2018 CVPR) [Paper]
  22. Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding (2019 CVPR) [Paper] [Code]
  23. 3D Scene Reconstruction with Multi-layer Depth and Epipolar Transformers (ICCV 2019) [Paper]

无序抓取

视觉伺服

综述

  1. Survey on Visual Servoing for Manipulation
  2. A review on vision-based control of robot manipulators

基于图像（2D）

  1. Kinematics-based incremental visual servo for robotic capture of non-cooperative target
  2. Position and attitude control of Eye-In-Hand System by visual servoing using Binocular Visual Space
  3. Progressive 3D Reconstruction of Unknown Objects using one Eye-in-Hand Camera

基于位置（3D）

  1. A hybrid positioning method for eye-in-hand industrial robot by using 3D reconstruction and IBVS

基于图像和位置（2.5D）

  1. Moment-Based 2.5-D Visual Servoing for Textureless Planar Part Grasping

深度图补全

1、HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion

2、Sparse and noisy LiDAR completion with RGB guidance and uncertainty

3、3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

4、Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completion

5、Confidence Propagation through CNNs for Guided Sparse Depth Regression

6、Learning Guided Convolutional Network for Depth Completion

7、DFineNet: Ego-Motion Estimation and Depth Refinement from Sparse, Noisy Depth Input with RGB Guidance

8、PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

9、Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints

3DCVer 会议/顶刊

SLAM

点云处理

3D视觉/机器视觉

计算机视觉

图像处理

深度学习

机器学习

自动驾驶

医疗影像

About

总结3D视觉所涉及到的边边角角知识点，包括VSLAM、点云后处理、相机标定、深度学习等。

Resources

Readme
Activity

Stars

86 stars

Watchers

2 watching

Forks

18 forks
Report repository

Releases

No releases published

Packages

No packages published