Neural network reconstruction of 3D semantic maps, AIRI x Sirius 2022
The goal is a 3D reconstruction of the bottom surface, with which we can then calculate the volume of the biotope (in our case crabs). At the entrance, we receive a monocular video extracted from online recordings of the seabed from youtube. As a result, we get a three-dimensional map in the form of a mesh with detected crabs and the volume of crabs found in the video
The project is relevant because it is necessary to automate detailed underwater research, which is mandatory in the Russian Federation. The areas of application can be:
- Exploration inspection work on water bodies
- Environmental monitoring during construction on the continental shelf
- Monitoring of fishing sites
Our pipeline includes the steps, described on the picture below.
The following figure contains a detailed description of the components of the Hierarchical-Localization network and the Poisson method implemented in open3d. As a result, we get the restored surface of the seabed from a series of frames.
As an input data we have this video from youtube.
Next, we get a point cloud with approximate camera positions. And using the Poisson method, we get a surface. We can pull the original image onto a surface where we can see crabs.
The following is another example obtained from a video recording using our approach.
In this part of the solution, we segment the images to find crabs on them, and then correlate the objects in the image and on the 3d mesh using the built-in localization in hloc.
We also tried neuralblox, which of the depth maps and camera positions can restore the surface, however, on our data, this method did not prove itself very well. Perhaps more static images of the same place are needed for a good result.
Another tried method is VDBFusion. This neural network, under certain parameters, gave good approximations of the surface, but the surface has noise, so we preferred a more stable classical version of surface reconstruction.
Despite the fact that we tried various non-classical methods of solving our problem, in conditions of limited time, we still could not implement all the ideas.
One of the unrealized ideas was to compare areas with 2D images from a video with 3D mesh to identify crabs using the ray tracing method. To find a specific 3D point on the mesh, we should find the intersections of a ray from camera center, through the 2d point on the image plane and the mesh.This can be implemented using the trimesh library. The example code can be viewed here.
Also, we have not tried many other approaches to restoring misha from a point cloud, for example, the following approaches:
It would also be interesting to explore how the newly released MobileNerf can be applied to the task.