Sparse reconstruction incomplete / Ceres solver failure

[cr00lick]

Hi,
I tried to do a sparse reconstruction (according to instructions from github) using GLOMAP 1.0.0 (feature extraction and matching in COLMAP) but the result is just cameras, very oddly arranged in space.

Output from glomap mapper:

glomap mapper --database_path D:\colmap_projects\gerabox0_glomap\gearbox0.db --image_path D:\colmap_projects\gerabox0_glomap\images --output_path D:\colmap_projects\gerabox0_glomap\sparse_glomap
Loading Images 483 / 483
Loading Image Pair 104419 / 104419
I20240817 17:50:43.794697 4280 colmap_converter.cc:303] Pairs read done. 51413 / 104419 are invalid
I20240817 17:50:43.810030 4280 global_mapper.cc:80] Loaded database

Running preprocessing ...

I20240817 17:50:43.831224 4280 view_graph_manipulation.cc:246] Decompose relative pose for 0 pairs
I20240817 17:50:43.832840 4280 view_graph_manipulation.cc:295] Decompose relative pose done. 0 pairs are pure rotation
I20240817 17:50:43.832909 4280 timer.cc:87] Elapsed time: 0.02250 [seconds]

---

Running view graph calibration ...

---

I20240817 17:50:43.833095 4280 view_graph_calibration.cc:16] Start ViewGraphCalibrator
I20240817 17:50:44.563322 4280 view_graph_calibration.cc:155] 0 cameras are rejected in view graph calibration
I20240817 17:50:44.574276 4280 view_graph_calibration.cc:189] invalid / total number of two view geometry: 5629 / 95904

---

Running relative pose estimation ...

---

I20240817 17:50:44.584905 4280 image_undistorter.cc:17] Undistorting images..
I20240817 17:50:44.602915 4280 image_undistorter.cc:37] Image undistortion done
I20240817 17:50:44.612452 4280 relpose_estimation.cc:24] Estimating relative pose for 47377 pairs
Estimating relative pose: 100%
I20240817 17:53:16.775923 4280 relpose_estimation.cc:69] Estimating relative pose done
I20240817 17:53:18.317452 4280 relpose_filter.cc:46] Filtered 12715 relative poses with inlier number < 30
I20240817 17:53:18.325258 4280 relpose_filter.cc:63] Filtered 0 relative poses with inlier ratio < 0.25
I20240817 17:53:18.352928 4280 timer.cc:87] Elapsed time: 153.76804 [seconds]

---

Running rotation averaging ...

---

I20240817 17:53:19.115703 4280 relpose_filter.cc:31] Filtered 983 relative rotation with angle > 10 degrees
I20240817 17:53:19.838922 4280 relpose_filter.cc:31] Filtered 0 relative rotation with angle > 10 degrees
I20240817 17:53:19.862613 4280 global_mapper.cc:95] 483 / 483 images are within the connected component.
I20240817 17:53:19.863035 4280 timer.cc:87] Elapsed time: 1.50903 [seconds]

---

Running track establishment ...

---

Initializing pairs 104419 / 104419
Establishing pairs 104419 / 104419
Establishing tracks 376406 / 376406
I20240817 17:53:38.136008 4280 track_establishment.cc:149] Discarded 4559 tracks due to inconsistency
I20240817 17:53:39.382754 4280 global_mapper.cc:115] Before filtering: 376406, after filtering: 241976
I20240817 17:53:39.384061 4280 timer.cc:87] Elapsed time: 19.51780 [seconds]

---

Running global positioning ...

---

I20240817 17:53:39.811269 4280 image_undistorter.cc:17] Undistorting images..
I20240817 17:53:39.811641 4280 image_undistorter.cc:37] Image undistortion done
I20240817 17:53:39.811983 4280 global_positioning.cc:43] Setting up the global positioner problem
I20240817 17:53:49.956592 4280 global_positioning.cc:68] Solving the global positioner problem
W20240817 17:55:35.373184 4280 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240817 17:55:36.728027 4280 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240817 17:55:38.339985 4280 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240817 17:55:39.940677 4280 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240817 17:55:42.293521 4280 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
E20240817 17:55:42.293684 4280 trust_region_minimizer.cc:95] Terminating: Number of consecutive invalid steps more than Solver::Options::max_num_consecutive_invalid_steps: 5
I20240817 17:55:44.241090 4280 global_positioning.cc:77] Ceres Solver Report: Iterations: 5, Initial cost: 1.663638e+07, Final cost: 1.663638e+07, Termination: FAILURE
I20240817 17:55:49.725829 4280 global_mapper.cc:86] Reconstruction done in 305.916 seconds
I20240817 17:55:50.410956 4280 colmap_io.cc:27] Extracting colors ...
I20240817 17:56:12.889060 4280 global_mapper.cc:91] Export to COLMAP reconstruction done

[aW4KeNiNG]

Same problem with same results

[ahojnnes]

How does the same reconstruction look when using COLMAP?

[aW4KeNiNG]

It works fine. I use pinhole camera.

COLMAP:

GLOMAP:

[aW4KeNiNG]

I have uploaded the images/database and results here: https://mega.nz/file/j9pygKwJ#pge3CVpgWJJ--kDl1AVkE-qQVXMcyv4qI0kK1Kla2_4

Commands executed:

colmap feature_extractor --database_path colmap\database.db --image_path data\images_0.83 --ImageReader.single_camera 1 --ImageReader.camera_model PINHOLE --SiftExtraction.use_gpu 1

colmap exhaustive_matcher --database_path colmap\database.db --SiftMatching.use_gpu 1

glomap mapper --database_path glomap\database.db --image_path data\images_0.83 --output_path glomap\sparse
colmap mapper --database_path colmap\database.db --image_path data\images_0.83 --output_path colmap\sparse

I can see the next error/warning when glomap is executed:

-------------------------------------
Running global positioning ...
-------------------------------------
I20240819 21:47:47.331957 31644 image_undistorter.cc:17] Undistorting images..
I20240819 21:47:47.332085 31644 image_undistorter.cc:37] Image undistortion done
I20240819 21:47:47.332203 31644 global_positioning.cc:43] Setting up the global positioner problem
I20240819 21:47:49.996335 31644 global_positioning.cc:68] Solving the global positioner problem
W20240819 21:48:14.758614 31644 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240819 21:48:16.093164 31644 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240819 21:48:17.481188 31644 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240819 21:48:18.820373 31644 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240819 21:48:20.161022 31644 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
E20240819 21:48:20.161222 31644 trust_region_minimizer.cc:95] Terminating: Number of consecutive invalid steps more than Solver::Options::max_num_consecutive_invalid_steps: 5
I20240819 21:48:21.066863 31644 global_positioning.cc:77] Ceres Solver Report: Iterations: 5, Initial cost: 7.422136e+06, Final cost: 7.422136e+06, Termination: FAILURE

[hnzhw]

Same problem with same results

[lpanaf]

HI, @aW4KeNiNG, thanks for sharing the database!
Strangely, from my computer, I got

for the database.db in the glomap folder. Note that two databases seem to be the same in colmap and glomap folder. @hnzhw did you get the failed result with the shared database?

[hnzhw]

HI, @aW4KeNiNG, thanks for sharing the database! Strangely, from my computer, I got for the database.db in the glomap folder. Note that two databases seem to be the same in colmap and glomap folder. @hnzhw did you get the failed result with the shared database?

Yes, same result.
windows10 x64 / 128G / Intel(R) Xeon(R) Gold 6226R / RTX4090 24G
colmap 3.8 / glomap 1.0.0 for windows

All my image datasets have the same issue, and both of the shared database got the failed result.

D:\mygs\glomap\data\glomap_test\data>D:\mygs\glomap\bin\glomap mapper --database_path D:\mygs\glomap\data\glomap_test\data\distorted\database.db --image_path D:\mygs\glomap\data\glomap_test\data\images --output_path D:\mygs\glomap\data\glomap_test\data\distorted\sparse
Loading Images 306 / 306
Loading Image Pair 38507 / 38507
I20240824 01:16:14.717896 6308 colmap_converter.cc:303] Pairs read done. 26430 / 38507 are invalid
I20240824 01:16:14.736190 6308 global_mapper.cc:80] Loaded database

Running preprocessing ...

I20240824 01:16:14.750036 6308 view_graph_manipulation.cc:246] Decompose relative pose for 0 pairs
I20240824 01:16:14.759110 6308 view_graph_manipulation.cc:295] Decompose relative pose done. 0 pairs are pure rotation
I20240824 01:16:14.759255 6308 timer.cc:87] Elapsed time: 0.02112 [seconds]

Running view graph calibration ...

I20240824 01:16:14.761173 6308 view_graph_calibration.cc:16] Start ViewGraphCalibrator
I20240824 01:16:14.920437 6308 view_graph_calibration.cc:155] 0 cameras are rejected in view graph calibration
I20240824 01:16:14.925570 6308 view_graph_calibration.cc:189] invalid / total number of two view geometry: 1533 / 21768

Running relative pose estimation ...

I20240824 01:16:14.934751 6308 image_undistorter.cc:17] Undistorting images..
I20240824 01:16:14.970915 6308 image_undistorter.cc:37] Image undistortion done
I20240824 01:16:14.974728 6308 relpose_estimation.cc:24] Estimating relative pose for 10544 pairs
Estimating relative pose: 100%
I20240824 01:16:29.251929 6308 relpose_estimation.cc:69] Estimating relative pose done
I20240824 01:16:29.581938 6308 relpose_filter.cc:46] Filtered 2439 relative poses with inlier number < 30
I20240824 01:16:29.585076 6308 relpose_filter.cc:63] Filtered 0 relative poses with inlier ratio < 0.25
I20240824 01:16:29.590740 6308 timer.cc:87] Elapsed time: 14.65834 [seconds]

Running rotation averaging ...

I20240824 01:16:29.788226 6308 relpose_filter.cc:31] Filtered 54 relative rotation with angle > 10 degrees
I20240824 01:16:29.997088 6308 relpose_filter.cc:31] Filtered 0 relative rotation with angle > 10 degrees
I20240824 01:16:30.010057 6308 global_mapper.cc:95] 306 / 306 images are within the connected component.
I20240824 01:16:30.010211 6308 timer.cc:87] Elapsed time: 0.41760 [seconds]

Running track establishment ...

Initializing pairs 38507 / 38507
Establishing pairs 38507 / 38507
Establishing tracks 378130 / 378130
I20240824 01:16:36.485452 6308 track_establishment.cc:149] Discarded 1696 tracks due to inconsistency
I20240824 01:16:37.407418 6308 global_mapper.cc:115] Before filtering: 378130, after filtering: 208450
I20240824 01:16:37.407675 6308 timer.cc:87] Elapsed time: 7.39741 [seconds]

Running global positioning ...

I20240824 01:16:37.640145 6308 image_undistorter.cc:17] Undistorting images..
I20240824 01:16:37.640991 6308 image_undistorter.cc:37] Image undistortion done
I20240824 01:16:37.641621 6308 global_positioning.cc:43] Setting up the global positioner problem
I20240824 01:16:42.381556 6308 global_positioning.cc:68] Solving the global positioner problem
W20240824 01:17:27.850766 6308 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240824 01:17:29.123147 6308 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240824 01:17:30.414073 6308 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240824 01:17:31.654266 6308 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W20240824 01:17:32.870850 6308 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
E20240824 01:17:32.871136 6308 trust_region_minimizer.cc:95] Terminating: Number of consecutive invalid steps more than Solver::Options::max_num_consecutive_invalid_steps: 5
I20240824 01:17:34.172385 6308 global_positioning.cc:77] Ceres Solver Report: Iterations: 5, Initial cost: 8.892553e+06, Final cost: 8.892553e+06, Termination: FAILURE
I20240824 01:17:36.048224 6308 global_mapper.cc:86] Reconstruction done in 81.3241 seconds
I20240824 01:17:36.410897 6308 colmap_io.cc:27] Extracting colors ...
I20240824 01:17:46.277188 6308 global_mapper.cc:91] Export to COLMAP reconstruction done

[jonstephens85]

I also downloaded the dataset of the vehicle and got the same failures. I noticed that there was some bad stationary panning of the camera in the dataset. I removed some of those and I got a result to work.

The image set as uploaded also failed sparse when I tried Reality Capture.

[ichsan2895]

AFAIK, the latest glomap does not support colmap 3.8. It needs colmap 3.11-dev. Maybe thats the problem.

[aW4KeNiNG]

Could it be a Windows issue? I'm using Windows. What OS are you using?

If I use the first 60 images in the dataset, it works. I have other small dataset (43 images) where it works too.
I have check other dataset with 750 images and I have the same problem.

[JohannesKrueger]

Yes, the Dataset Size might be a Problem. I Succesfully processed a Dataset with 600 Images but it failed with with "error" for 800 Images. I also tried using the newest Colmap Version for preprocessing etc. to see if there might be the problem but it caused the same "error" with COLMAP (3.8, 3.9, 3.10) so this issue seems to be related to glomap.

[jonstephens85]

I have gotten a dataset of around 600 images to work, but also getting failures around 200 images. Doesn't seem to the number of images. Seems to be more sensitive to ensuring enough parallax movement in the images.

One note, when all of the datasets fail, I barely see any usage on my GPU.

[lpanaf]

I am using Ubuntu 22.04. In this case, it can be the problem of running out of memory. Can you maybe try decreasing --TrackEstablishment.max_num_tracks? Like set it to be 100000, etc.

[hnzhw]

I am using Ubuntu 22.04. In this case, it can be the problem of running out of memory. Can you maybe try decreasing --TrackEstablishment.max_num_tracks? Like set it to be 100000, etc.

Still the same issue.

[aW4KeNiNG]

Same issue. I'm running on 64gb RAM and 4090 24gb VRAM.

[jonstephens85]

I am finding data size might not be the problem for me. Most of my datasets I have tested have been images extracted from a video that was taken of a loop around an object. The rotation between each image is small. I found that if used every 4 images which increased the rotation between images, I could get the datasets to work.

I have 128 GB of RAM and an RTX 6000 Ada with 48GB or VRAM. None of the datasets I have tried have approached the limits of my memory.

[ichsan2895]

My result with glomap is also strange

Left = colmap 3.8 compiled with CUDA
Right = glomap, commit 816e06f

Additional info:

MXLinux KDE 19.4
Ceres-solver 2.1.0 (manually build with CUDA 11.4)
Cmake 3.28.4

Here is command that I use.

mkdir distorted

colmap feature_extractor \
    --image_path    input/ \
    --database_path distorted/database.db \
    --ImageReader.single_camera_per_folder 1

colmap exhaustive_matcher \
    --database_path distorted/database.db 

mkdir distorted/colmap
mkdir distorted/glomap

colmap mapper --database_path distorted/database.db \
    --image_path input/ \
    --output_path distorted/colmap

glomap mapper --database_path distorted/database.db \
    --image_path input/ \
    --output_path distorted/glomap

colmap model_converter \
  --input_path distorted/colmap/0 \
  --output_path distorted/colmap/0 \
  --output_type TXT

colmap model_converter \
  --input_path distorted/glomap/0 \
  --output_path distorted/glomap/0 \
  --output_type TXT

colmap image_undistorter \
  --image_path input \
  --input_path distorted/colmap/0 \
  --output_path distorted/colmap

colmap image_undistorter \
  --image_path input \
  --input_path distorted/glomap/0 \
  --output_path distorted/glomap

colmap model_converter \
  --input_path distorted/colmap/sparse \
  --output_path distorted/colmap/sparse \
  --output_type TXT

colmap model_converter \
  --input_path distorted/glomap/sparse \
  --output_path distorted/glomap/sparse \
  --output_type TXT

[hnzhw]

A small dataset, 167 images, a stone statue, it works.
So strange.

[lpanaf]

Hi @hnzhw @cr00lick @aW4KeNiNG, I have tried this example on my Windows computer. I got a similar failed result as you if I run with default parameters, but I can get some successful results by changing --TrackEstablishment.max_num_tracks to some smaller number, like 50000.

In this case, there should be some limit on the problem memory which causes this problem. You can maybe also use this workaround for now, and we will continue figuring out a solution for this.

[hnzhw]

Hi @hnzhw @cr00lick @aW4KeNiNG, I have tried this example on my Windows computer. I got a similar failed result as you if I run with default parameters, but I can get some successful results by changing --TrackEstablishment.max_num_tracks to some smaller number, like 50000. In this case, there should be some limit on the problem memory which causes this problem. You can maybe also use this workaround for now, and we will continue figuring out a solution for this.

Great, I tested four large datasets (around 1000 4K images) with --TrackEstablishment.max_num_tracks 50000, all worked.

[aW4KeNiNG]

Yes, it works for me too. Thanks!

[hnzhw]

There are still some issues.

colmap:

glomap:

[ichsan2895]

Unfortunatelly, it don't work properly with --TrackEstablishment.max_num_tracks 50000

Left = glomap mapper, commit 816e06f

Right = glomap mapper --TrackEstablishment.max_num_tracks 50000, commit 816e06f

Additional info:

MXLinux KDE 19.4
Colmap 3.8 with CUDA 11.4
Ceres-solver 2.1.0 (manually build with CUDA 11.4)
Cmake 3.28.4

[hnzhw]

Unfortunatelly, it don't work properly with --TrackEstablishment.max_num_tracks 50000

Left = glomap mapper, commit 816e06f

Right = glomap mapper --TrackEstablishment.max_num_tracks 50000, commit 816e06f

Additional info:

MXLinux KDE 19.4
Colmap 3.8 with CUDA 11.4
Ceres-solver 2.1.0 (manually build with CUDA 11.4)
Cmake 3.28.4

'--TrackEstablishment.max_num_tracks 50000' solves the following issue:
Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.

[ichsan2895]

Colmap (version 3.8) reconstruction using splatfacto (just 6k iters)
PSNR : 22.067, SSIM : 0.7425, LPIPS : 0.234

Glomap (commit 816e06f) reconstruction using splatfacto (just 6k iters)
PSNR : 19.604, SSIM : 0.6691, LPIPS : 0.351

[ichsan2895]

GREAT!, THIS DATASET SOLVED BY UPDATING COLMAP AND GLOMAP

Colmap (version 3.11-dev0, commit 66fd8e56) reconstruction using splatfacto (just 6k iters)
PSNR : 22.129, SSIM : 0.7404, LPIPS : 0.236

Glomap (version 1.0) reconstruction using splatfacto (just 6k iters)
PSNR : 22.271, SSIM : 0.7437, LPIPS : 0.233

Additional info:

MXLinux KDE 19.4
Ceres-solver 2.1.0 (manually build with CUDA 11.4)
Cmake 3.28.4

Command used:

mkdir distorted

colmap feature_extractor \
    --image_path    input/ \
    --database_path distorted/database.db \
    --ImageReader.single_camera_per_folder 1

colmap exhaustive_matcher \
    --database_path distorted/database.db 

mkdir distorted/colmap
mkdir distorted/glomap

colmap mapper --database_path distorted/database.db \
    --image_path input/ \
    --output_path distorted/colmap

glomap mapper --database_path distorted/database.db \
    --image_path input/ \
    --output_path distorted/glomap

colmap model_converter \
  --input_path distorted/colmap/1 \
  --output_path distorted/colmap/1 \
  --output_type TXT

colmap model_converter \
  --input_path distorted/glomap/0 \
  --output_path distorted/glomap/0 \
  --output_type TXT

colmap image_undistorter \
  --image_path input \
  --input_path distorted/colmap/1 \
  --output_path distorted/colmap

colmap image_undistorter \
  --image_path input \
  --input_path distorted/glomap/0 \
  --output_path distorted/glomap

colmap model_converter \
  --input_path distorted/colmap/sparse \
  --output_path distorted/colmap/sparse \
  --output_type TXT

colmap model_converter \
  --input_path distorted/glomap/sparse \
  --output_path distorted/glomap/sparse \
  --output_type TXT

[HungNgoCT]

Hi.,

I have the same problem. Not help when used --TrackEstablishment.max_num_tracks 50000, 100000 and even 1000000

[ahojnnes]

Could you please share database and images? Couls you please also specify the platform you are running on? Thank you.

[HungNgoCT]

Hi @ahojnnes .,

I am using Windows OS.

Please find the data here.

https://drive.google.com/file/d/1Yg68dHXxh1_JJTXxM9MQsLi8YcSeKVAt/view?usp=sharing

This is my experimental results: Good pose when using Colmap with exhaustive_matcher + Hierarchy_mapper, but faild when Glomap with exhaustive_matcher + Glomap. Looking forward your advice. Thank you.