Dump some data association metrics #104

johnwlambert · 2021-02-24T05:50:08Z

Make RANSAC more modular in data association
Save json file with some basic DA stats
Make it so 3d point outliers dont get plotted

…ot-legend

johnwlambert · 2021-02-24T05:51:32Z

akshay-krishnan · 2021-02-24T06:49:02Z

gtsfm/data_association/data_assoc.py

+            "mean_track_length": mean_track_length,
+            "median_track_length": median_track_length,


Is there a good reason to have both median and mean? We should try to reduce numbers.
Also, would it be (more) useful to have median_ per_track_average_error?

I was just trying to dump as much as possible now, since space is cheap and performance is not great yet :-)

Do you mean median error within each track, or the median of all avg track errors @akshay-krishnan ?

I was referring to these two:
"mean_track_length": mean_track_length,
"median_track_length": median_track_length,
If they are the mean and median of the same quantity, we can drop the mean. It does provide a weak signal about outliers, but we can look at min and max if we need that.

We would need to change the SFMResult class to get the min and max stats, so i'd prefer to leave that for another PR

On second thought, i added that.

But "mean track length" is a metric I've seen in published papers like COLMAP so I'd like to keep it for now

Then we probably won't need median. We'll ultimately end up using one number for the central tendency (bias) and one for the range/variance. But its upto you.

akshay-krishnan · 2021-02-24T07:04:29Z

gtsfm/data_association/point3d_initializer.py


        # compute reprojection errors for each measurement
        reproj_errors = self.compute_track_reprojection_errors(inlier_track.measurements, triangulated_pt)

        # all the measurements should have error < threshold
        if not np.all(reproj_errors < self.reproj_error_thresh):
-            return None
+            return None, None


Since we have reprojection errors, cant we return the mean here?

That's a good point. I was a bit torn whether we should dump statistics of accepted tracks, or of any generated track, even if rejected.

@ayushbaid any thoughts here? maybe we should dump both for completeness now, since DA's failures are a bit of a mystery?

Yeah I think having more data right now is useful to debug our DA module.

But I would like a distinction on the errors when we dump it to the file. Like separate sections for rejected v accepted tracks.

Yeah, I think the error should include failures as well, unless we are tracking them separately. Tracking them separately (another metric) might be the better choice. Maybe accepted_tracks_ratio?

Cool, I added these extra metrics now

akshay-krishnan · 2021-02-24T07:06:20Z

gtsfm/multi_view_optimizer.py

            init_cameras_graph, v_corr_idxs_graph, keypoints_graph
        )
+        ba_input_graph = data_assoc_graph[0]
+        data_assoc_metrics_graph = data_assoc_graph[1]


ba_input_graph, data_assoc_metrics_graph = self.data_association_module.create_computation_graph( init_cameras_graph, v_corr_idxs_graph, keypoints_graph )

Isn't that better?

Dask won't allow this unfortunately:

Traceback (most recent call last): File "gtsfm/runner/run_scene_optimizer_argoverse.py", line 97, in <module> run_scene_optimizer(args) File "gtsfm/runner/run_scene_optimizer_argoverse.py", line 37, in run_scene_optimizer sfm_result_graph = scene_optimizer.create_computation_graph( File "/Users/johnlambert/Documents/gtsfm/gtsfm/scene_optimizer.py", line 184, in create_computation_graph (ba_input_graph, ba_output_graph, optimizer_metrics_graph, ) = self.multiview_optimizer.create_computation_graph( File "/Users/johnlambert/Documents/gtsfm/gtsfm/multi_view_optimizer.py", line 81, in create_computation_graph ba_input_graph, data_assoc_metrics_graph = self.data_association_module.create_computation_graph( File "/Users/johnlambert/anaconda3/envs/gtsfm-v1/lib/python3.8/site-packages/dask/delayed.py", line 562, in __iter__ raise TypeError("Delayed objects of unspecified length are not iterable") TypeError: Delayed objects of unspecified length are not iterable

This logic should be moved to DataAssociation's create_computation_graph.

I'm good with moving it, but is there a particular reason to put it in one place rather than the other?

I think its more clear seeing the semantics on what to expect. If a module has three Delayed being returned, we can create richer docstrings.

gtsfm/utils/geometry_comparisons.py

akshay-krishnan · 2021-02-26T02:45:09Z

gtsfm/data_association/data_assoc.py

+            "mean_track_length": mean_track_length,
+            "median_track_length": median_track_length,


I was referring to these two:
"mean_track_length": mean_track_length,
"median_track_length": median_track_length,
If they are the mean and median of the same quantity, we can drop the mean. It does provide a weak signal about outliers, but we can look at min and max if we need that.

akshay-krishnan · 2021-02-26T02:49:47Z

gtsfm/data_association/point3d_initializer.py


        # compute reprojection errors for each measurement
        reproj_errors = self.compute_track_reprojection_errors(inlier_track.measurements, triangulated_pt)

        # all the measurements should have error < threshold
        if not np.all(reproj_errors < self.reproj_error_thresh):
-            return None
+            return None, None


Yeah, I think the error should include failures as well, unless we are tracking them separately. Tracking them separately (another metric) might be the better choice. Maybe accepted_tracks_ratio?

gtsfm/utils/geometry_comparisons.py

gtsfm/data_association/point3d_initializer.py

akshay-krishnan · 2021-02-26T03:09:05Z

gtsfm/data_association/point3d_initializer.py

+                camera_estimates.append(self.track_camera_dict.get(i1))
+                camera_estimates.append(self.track_camera_dict.get(i2))
+
+                img_measurements = Point2Vector()


Cant we just do img_measurements = [uv1, uv2]

GTSAM requires the Point2Vector type as argument

akshay-krishnan · 2021-02-26T03:12:19Z

gtsfm/data_association/point3d_initializer.py

+                camera_estimates.append(self.track_camera_dict.get(i1))
+                camera_estimates.append(self.track_camera_dict.get(i2))


Cant we do camera_estimates = [self.track_camera_dict.get(i1), self.track_camera_dict.get(i2)]?

akshay-krishnan · 2021-02-26T03:17:50Z

gtsfm/data_association/point3d_initializer.py

+            k1, k2 = measurement_pairs[sample_idxs]
+
+            i1, uv1 = track_2d.measurements[k1]
+            i2, uv2 = track_2d.measurements[k2]
+
+            camera_estimates = CameraSetCal3Bundler()
+            # check for unestimated cameras
+            if self.track_camera_dict.get(i1) != None and self.track_camera_dict.get(i2) != None:
+                camera_estimates.append(self.track_camera_dict.get(i1))
+                camera_estimates.append(self.track_camera_dict.get(i2))
+
+                img_measurements = Point2Vector()
+                img_measurements.append(uv1)
+                img_measurements.append(uv2)
+
+                # triangulate point for track
+                try:
+                    triangulated_pt = gtsam.triangulatePoint3(
+                        camera_estimates,
+                        img_measurements,
+                        rank_tol=SVD_DLT_RANK_TOL,
+                        optimize=True,
+                    )
+                except RuntimeError:
+                    # TODO: handle cheirality exception properly?
+                    logger.info(
+                        "Cheirality exception from GTSAM's triangulatePoint3() likely due to outlier, skipping track"
+                    )
+                    continue
+
+                errors = self.compute_track_reprojection_errors(track_2d.measurements, triangulated_pt)


I think it would be more modular and cleaner if you made the above changes, and moved this code to a different function compute_reprojection_error_for_hypothesis(track_2d.measurements, k1, k2)

@akshay-krishnan can you point me to which line you were referring to?

I was referring to all the lines I selected. Basically, the code for computing the errors for each sample_idx could be moved to a different function as this function is getting a little too big. But it's fine if you'd like to keep it this way.

gtsfm/multi_view_optimizer.py

gtsfm/data_association/data_assoc.py

gtsfm/data_association/point3d_initializer.py

ayushbaid · 2021-02-24T21:31:28Z

gtsfm/data_association/point3d_initializer.py


        # compute reprojection errors for each measurement
        reproj_errors = self.compute_track_reprojection_errors(inlier_track.measurements, triangulated_pt)

        # all the measurements should have error < threshold
        if not np.all(reproj_errors < self.reproj_error_thresh):
-            return None
+            return None, None


Yeah I think having more data right now is useful to debug our DA module.

ayushbaid · 2021-02-24T21:32:26Z

gtsfm/data_association/point3d_initializer.py


        # compute reprojection errors for each measurement
        reproj_errors = self.compute_track_reprojection_errors(inlier_track.measurements, triangulated_pt)

        # all the measurements should have error < threshold
        if not np.all(reproj_errors < self.reproj_error_thresh):
-            return None
+            return None, None


But I would like a distinction on the errors when we dump it to the file. Like separate sections for rejected v accepted tracks.

gtsfm/data_association/point3d_initializer.py

ayushbaid · 2021-02-26T04:37:38Z

gtsfm/data_association/point3d_initializer.py

+                    )
+                except RuntimeError:
+                    # TODO: handle cheirality exception properly?
+                    logger.info(


do we have a way to log the counts of this exception?

…ot-legend

johnwlambert · 2021-03-05T03:24:59Z

Dumps something like:

{
    "mean_2d_track_length": 2.8,
    "accepted_tracks_ratio": 0.977,
    "track_cheirality_failure_ratio": 0.007,
    "num_accepted_tracks": 2498,
    "mean_3d_track_length": 2.404,
    "median_3d_track_length": 2.0,
    "per_rejected_track_avg_errors": [
        null,
        null,
        null,
...
}

gtsfm/data_association/data_assoc.py

ayushbaid · 2021-03-07T05:08:14Z

gtsfm/data_association/data_assoc.py

+            "num_len_10_tracks": int(np.sum(track_lengths_3d == 10)),
+            "per_rejected_track_avg_errors": per_rejected_track_avg_errors,
+            "per_accepted_track_avg_errors": per_accepted_track_avg_errors,
+            "points_3d": points_3d,


why dump just 3d points? Why not dump the track?

I was visualizing them in Mayavi on my local machine for debugging.

I recommend in a separate PR we dump the output in COLMAP-format. Then we could remove my points_3d dump later.

gtsfm/data_association/point3d_initializer.py

ayushbaid · 2021-03-07T05:17:32Z

gtsfm/multi_view_optimizer.py

            init_cameras_graph, v_corr_idxs_graph, keypoints_graph
        )
+        ba_input_graph = data_assoc_graph[0]
+        data_assoc_metrics_graph = data_assoc_graph[1]


This logic should be moved to DataAssociation's create_computation_graph.

gtsfm/utils/geometry_comparisons.py

ayushbaid · 2021-03-07T05:45:25Z

tests/utils/test_geometry_comparisons.py

+def test_get_points_within_radius_of_cameras():
+    """Verify that points that fall outside of 10 meter radius of two camera poses.
+
+    Cameras are placed at (0,0,0) and (10,0,0).
+    """
+    wTi0 = Pose3(Rot3(), np.zeros(3))
+    wTi1 = Pose3(Rot3(), np.array([10.0, 0, 0]))
+    wTi_list = [wTi0, wTi1]
+    points_3d = np.array([[-15, 0, 0], [0, 15, 0], [-5, 0, 0], [15, 0, 0], [25, 0, 0]])
+    radius = 10.0
+    nearby_points_3d = geometry_comparisons.get_points_within_radius_of_cameras(wTi_list, points_3d, radius)
+
+    expected_nearby_points_3d = np.array(
+        [
+            [-5, 0, 0],
+            [15, 0, 0],
+        ]
+    )
+    np.testing.assert_allclose(nearby_points_3d, expected_nearby_points_3d)
+
+
+def test_get_points_within_radius_of_cameras_negative_radius():
+    """Catch degenerate input."""
+    wTi0 = Pose3(Rot3(), np.zeros(3))
+    wTi1 = Pose3(Rot3(), np.array([10.0, 0, 0]))
+    wTi_list = [wTi0, wTi1]
+    points_3d = np.array([[-15, 0, 0], [0, 15, 0], [-5, 0, 0], [15, 0, 0], [25, 0, 0]])
+    radius = -5
+    nearby_points_3d = geometry_comparisons.get_points_within_radius_of_cameras(wTi_list, points_3d, radius)
+    assert nearby_points_3d is None, "Non-positive radius is not allowed"
+
+
+def test_get_points_within_radius_of_cameras_no_points():
+    """Catch degenerate input."""
+
+    wTi0 = Pose3(Rot3(), np.zeros(3))
+    wTi1 = Pose3(Rot3(), np.array([10.0, 0, 0]))
+    wTi_list = [wTi0, wTi1]
+    points_3d = np.zeros((0, 3))
+    radius = 10.0
+
+    nearby_points_3d = geometry_comparisons.get_points_within_radius_of_cameras(wTi_list, points_3d, radius)
+    assert nearby_points_3d is None, "At least one 3d point must be provided"
+
+
+def test_get_points_within_radius_of_cameras_no_poses():
+    """Catch degenerate input."""
+    wTi_list = []
+    points_3d = np.array([[-15, 0, 0], [0, 15, 0], [-5, 0, 0], [15, 0, 0], [25, 0, 0]])
+    radius = 10.0
+
+    nearby_points_3d = geometry_comparisons.get_points_within_radius_of_cameras(wTi_list, points_3d, radius)
+    assert nearby_points_3d is None, "At least one camera pose must be provided"
+
+


I think we should have these tests inside the class as we have done it in other tests.

I'm fine either way. Technically it's not a class, so i was going more for the functional, free-function approach

akshay-krishnan

Looks good John, adding some optional comments for improved readability.

gtsfm/data_association/point3d_initializer.py

gtsfm/data_association/data_assoc.py

akshay-krishnan · 2021-03-07T07:00:59Z

gtsfm/data_association/data_assoc.py

+                "per_rejected_track_avg_errors": per_rejected_track_avg_errors,
+                "per_accepted_track_avg_errors": per_accepted_track_avg_errors,


Just curious, why per_ here?

I don't have a good way of differentiating between names for
(1) average error within a track
(2) average of per-track avg errors

but i'm open to suggestions

ayushbaid · 2021-03-10T18:45:28Z

gtsfm/data_association/data_assoc.py


        logger.debug("[Data association] output number of tracks: %s", num_accepted_tracks)
        logger.debug("[Data association] output avg. track length: %s", mean_3d_track_length)

        # dump the 3d point cloud before Bundle Adjustment for offline visualization
        points_3d = [list(triangulated_data.track(j).point3()) for j in range(num_accepted_tracks)]
+        # bin edges are halfway between each integer
+        histogram_track_lengths, _ = np.histogram(track_lengths_3d, bins=np.linspace(-0.5, 10.5, 12))


track_lengths_histogram?

Also, why bin it from -0.5. We can use the np.histogram(track_lengths_3d, bins=8, range=(2, 10))

ayushbaid

Left some small suggestions. Looks good otherwise.

ayushbaid · 2021-03-10T18:46:47Z

gtsfm/data_association/data_assoc.py


        logger.debug("[Data association] output number of tracks: %s", num_accepted_tracks)
        logger.debug("[Data association] output avg. track length: %s", mean_3d_track_length)

        # dump the 3d point cloud before Bundle Adjustment for offline visualization
        points_3d = [list(triangulated_data.track(j).point3()) for j in range(num_accepted_tracks)]
+        # bin edges are halfway between each integer
+        histogram_track_lengths, _ = np.histogram(track_lengths_3d, bins=np.linspace(-0.5, 10.5, 12))


Also, why bin it from -0.5. We can use the np.histogram(track_lengths_3d, bins=8, range=(2, 10))

gtsfm/data_association/data_assoc.py

ayushbaid · 2021-03-10T18:54:35Z

gtsfm/multi_view_optimizer.py

            init_cameras_graph, v_corr_idxs_graph, keypoints_graph
        )
+        ba_input_graph = data_assoc_graph[0]
+        data_assoc_metrics_graph = data_assoc_graph[1]


I think its more clear seeing the semantics on what to expect. If a module has three Delayed being returned, we can create richer docstrings.

John Lambert added 10 commits February 24, 2021 00:43

restrict point cloud to nearby cameras

ee568cc

dump data association metrics

ac17a14

need equal axes scaling for pose plots

edc460f

add unit test on point-cam distance

926b84e

only render nearby points

41aea84

improve docstring

9cba17b

dump json metrics

62bf3b8

make ransac inside Point3Initializer more modular, was too long

aa17ae0

Merge branch 'master' of https://github.com/borglab/gtsfm into add-pl…

9d567c9

…ot-legend

rearrange fns to minimize diff

cd81a29

johnwlambert requested review from ayushbaid and akshay-krishnan February 24, 2021 05:50

johnwlambert requested a review from xwu4lab February 24, 2021 06:17

akshay-krishnan reviewed Feb 24, 2021

View reviewed changes

John Lambert added 4 commits February 24, 2021 10:33

fix docstring

2c074ef

zero radius not allowed

338abef

add test cases for degenerate input

4500101

add missing docstring

970b4b9

johnwlambert requested a review from akshay-krishnan February 25, 2021 23:01

akshay-krishnan requested changes Feb 26, 2021

View reviewed changes

ayushbaid reviewed Feb 26, 2021

View reviewed changes

John Lambert added 8 commits March 4, 2021 21:08

remove indent to simplify logic with early continue

2b11729

newline before args

08bdb55

clearer docstring

4e59918

switch += [] to .append

cf475a0

report cheirality failures

77818ab

add more helpful DA stats

ffffdad

clean up style

114286e

Merge branch 'master' of https://github.com/borglab/gtsfm into add-pl…

5fb979c

…ot-legend

johnwlambert requested review from akshay-krishnan and ayushbaid March 5, 2021 03:24

John Lambert added 4 commits March 4, 2021 22:34

report number of tracks at each threshold

7432a76

return all track lengths for further analysis

c100693

report min and max, since mean is not so useful

d3b69ab

add back mean 3d track length

abe6224

ayushbaid reviewed Mar 7, 2021

View reviewed changes

akshay-krishnan approved these changes Mar 7, 2021

View reviewed changes

gtsfm/data_association/point3d_initializer.py Outdated Show resolved Hide resolved

gtsfm/data_association/data_assoc.py Outdated Show resolved Hide resolved

gtsfm/data_association/data_assoc.py Outdated Show resolved Hide resolved

John Lambert added 3 commits March 7, 2021 01:34

replace duplicated code

13e164c

Remove newline

4624cbc

organize json hierarchically

f55d615

akshay-krishnan approved these changes Mar 7, 2021

View reviewed changes

make names less redundant in json file

de5d24b

johnwlambert requested a review from ayushbaid March 8, 2021 17:46

ayushbaid reviewed Mar 10, 2021

View reviewed changes

John Lambert added 4 commits March 10, 2021 15:56

re-order histogram dict to make easier to follow

43b48ca

make return args clearer

11916fd

make return args clearer

7f2f483

simplify data assoc stats dict creation

7ef9736

johnwlambert requested a review from ayushbaid March 10, 2021 22:02

ayushbaid approved these changes Mar 10, 2021

View reviewed changes

johnwlambert merged commit 0d6b0e2 into master Mar 11, 2021

johnwlambert deleted the add-plot-legend branch March 11, 2021 00:54

ayushbaid mentioned this pull request Mar 16, 2021

Adding expressive GTSFM data to allow non-consecutive cameras #113

Merged

		"mean_track_length": mean_track_length,
		"median_track_length": median_track_length,

		camera_estimates.append(self.track_camera_dict.get(i1))
		camera_estimates.append(self.track_camera_dict.get(i2))

		"per_rejected_track_avg_errors": per_rejected_track_avg_errors,
		"per_accepted_track_avg_errors": per_accepted_track_avg_errors,

Dump some data association metrics #104

Dump some data association metrics #104

Conversation

johnwlambert commented Feb 24, 2021

johnwlambert commented Feb 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnwlambert commented Mar 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshay-krishnan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ayushbaid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment