Fisheye lens information loss during undistortion

[ekuznetsov139]

I am trying to use colmap to reconstruct a scene from a set of fisheye lens images, 2048x2048 with FOV 180 degrees (camera: RADIAL_FISHEYE 2048 2048 600 1024 1024 0 0).

Sparse reconstruction works well - I see the poses correctly reconstructed and much of the scene in place (apart from some outliers).

Problems begin when I move on to dense reconstruction. For some reason, colmap "undistorts" input images by slightly deforming them and cropping them to 409x409 pixels, keeping only the center portion of each image. Effectively 90% of the image is simply lost. The resulting mesh is very poor quality and missing large chunks.

I can achieve better results by manually transforming all images to pinhole projection and specifying the camera as SIMPLE_PINHOLE - in that case, colmap apparently skips the undistortion step and copies the images as-is. But that's inefficient (to keep the original resolution and to retain most of the original image, I need to convert them to SIMPLE_PINHOLE 3072 3072 512 1536 1536, which corresponds to the FOV of 140 degrees - more than double the number of pixels.) Not to mention that there's nothing in the documentation to suggest that manually converting images is a required or even a recommended step.

P.S. Colmap 3.8 commit 3021f1 on 2022-04-01 with CUDA on Linux. The same behavior is observed with GUI and with the command-line version.

[ekuznetsov139]

Here's what seems to be happening.

This code https://github.com/colmap/colmap/blob/dev/src/base/undistortion.cc#L826-L870 utterly fails with my fisheye lens parameters, because the input image includes the entire front hemisphere and then some. (All points on the perimeter are actually behind the camera.)

So, here https://github.com/colmap/colmap/blob/dev/src/base/undistortion.cc#L885 left_max_x is something like 4e+18, and here https://github.com/colmap/colmap/blob/dev/src/base/undistortion.cc#L893 both min_scale_x and max_scale_x are very small (and max_scale_x is actually negative); options.blank_pixels is 0, and scale_x is large and negative.

So, it sets the image scale to the greater of scale_x and options.min_scale, which is 0.20.

The entire code block incorrectly assumes that the undistorted image is bounded after pinhole projection, so there's no simple fix that I see.

[ahojnnes]

Yes, this is not a supported scenario. There is a fork of COLMAP based on a very old commit that addresses the issue across the entire codebase, but it will be a lot of work to apply those changes to the latest COLMAP commits.

[ekuznetsov139]

It should be mentioned in the documentation then. Here perhaps https://colmap.github.io/cameras.html

[vesor]

@ahojnnes could you find the old fork you mentioned? Are there any workaround to this, for example set min_scale to 0.8?