Question about getting covariance using BACovariance and get_cam1_from_cam2_cov

[moh-ake]

Hello,

I am trying to obtain covariance matrices of poses using pycolmap. To do so, I followed the pycolmap documentation and the BACovariance unit test to write the Python code below. However, the 6x6 covariance matrices I get using get_cam1_from_cam2_cov seem to be incorrect.

Questions:

• Does ba_covariance.get_cam1_from_cam2_cov(id1, id2) return the covariance of camera pose 1 with camera pose 2 as a reference , or does it return the covariance of the projection of 3D points from camera 2 to camera 1 ?
• I would like to know if the code below correctly implements the process for running ba_covariance.get_cam1_from_cam2_cov(id1, id2), or if I'm missing any steps ?

################################################## reconstruction ##############################################
reconstruction = pycolmap.Reconstruction()
reconstruction.read("/Path/To/Reconstruction")

################################################## BA options ##############################################
ba_options = pycolmap.BundleAdjustmentOptions()

################################################## BA config ################################################
ba_config = pycolmap.BundleAdjustmentConfig()

# Add images
for image_id in reconstruction.reg_image_ids():
    ba_config.add_image(image_id)

# As mentioned in covariance_test.cc, we must fix
# the Gauge by always setting at least 3 points as constant
point_idx = 0
for point_id in reconstruction.point3D_ids():
    if point_idx >= 3:
        ba_config.add_variable_point(point_id)
    else:
        ba_config.add_constant_point(point_id)
        point_idx += 1

ba_config.set_constant_cam_intrinsics(1)

################################################## BA adjuster ################################################
bundle_adjuster = pycolmap.create_default_bundle_adjuster(options=ba_options, config=ba_config, reconstruction=reconstruction)
summary = bundle_adjuster.solve()
print(summary.IsSolutionUsable()) # Returns True

################################################## BA covariance ##############################################
ba_covariance_options = pycolmap.BACovarianceOptions()
ba_covariance_options.params = pycolmap.BACovarianceOptionsParams.POSES
ba_covariance = pycolmap.estimate_ba_covariance(
                                        options=ba_covariance_options,
                                        reconstruction=reconstruction,
                                        bundle_adjuster=bundle_adjuster)

ba_covariance.get_cam1_from_cam2_cov(1, 2)

Thank you

[B1ueber2y]

Hey. It should return the covariance of the relative pose cam1_from_cam2. How do you find it incorrect? Could you share more info? Thanks.

[moh-ake]

Thanks for you response @B1ueber2y,

I'm trying to correct a loop closure using pose graph optimization because in my case bundle adjustment does not correct it well.
In my pose graph, vertices are camera poses and edges are relative poses, for each edge i must give an information matrix which is the inverse of the covariance matrix. When i set the information matrix to the identity for all edges i get good results but once i replace it by the inverse of the covariance matrix of COLMAP, the graph optimization does not give good results (Please see images below)

Example of covariance matrix returned by COLMAP:

[[6.49629331e-03, 4.84738973e-05, 2.42546587e-03, 9.01098007e-06, 1.32283296e-05, 1.08578916e-05],
 [5.98033757e-05, 4.11657010e-03, 6.28971703e-05, -5.25375255e-05, -7.45648234e-07, -1.69786677e-05],
 [2.45528154e-03, 5.62601110e-05, 6.62852814e-03, -1.60119940e-05, -8.27462814e-06, -8.01427556e-06],
 [8.92782675e-06, -5.25197978e-05, -1.60211515e-05, 2.31358755e-04, 2.91027732e-06, 1.10493827e-04],
 [1.32282544e-05, -7.97829422e-07, -8.29537561e-06, 3.16902128e-06, 6.07864554e-05, 2.42531555e-06],
 [1.08425748e-05, -1.70470141e-05, -8.04794658e-06, 1.10944438e-04, 2.29020925e-06, 1.48035790e-04]]

Results without graph optimization

Results of graph optimization with information matrix as identity

Results of graph optimization with information matrix as the inverse of covariance matrix of COLMAP

Thank you

[B1ueber2y]

Hey. I am not sure how you set up the PGO. The covariance you get from pycolmap describes the uncertainty of the relative transformation from the bundle adjustment problem you input to the interface. This does not necessarily correspond to the uncertainty of the relative pose constraints you have in the PGO. Also, the uncertainty of the relative pose in PGO may require inflation in practice. Moreover, you need to handle the scale correctly in the covariance propagation since this is often ambiguous in the bundle adjustment but can largely constrain your PGO. Thus, I dont think what you showed has contradiction to the correctness of the covariance estimation in pycolmap.

[moh-ake]

Thank you for the explanation,

I understand that the covariance obtained from get_cam1_from_cam2_cov corresponds to the covariance within the bundle adjustment problem. However, I expected it to yield reasonably accurate results in the pose graph optimization as well, rather than completely incorrect ones.

I’ll look into inflating the covariance matrix. If you happen to have any example or references that outline the steps, I’d be grateful

Thank you

[ahojnnes]

This question has been addressed in #3211, which covers BACovariance usage and provides guidance. Closing as answered.