Post-BA Retriangulation (Structure-Refinement) - #1111
Merged
Conversation
kathirgounder
force-pushed
the
retri-densify-pr
branch
3 times, most recently
from
April 27, 2026 23:50
e854f95 to
b1a34cb
Compare
kathirgounder
changed the title
Adds an opt-in retriangulation + final-BA pass that runs after the existing BA
loop in `BundleAdjustmentOptimizer`. Recovers tracks dropped between union-find
(`CppDsfTracksEstimator`) and the BA filter passes — these are 3D points that
*could* be reconstructed once cameras are well-converged, but failed to triangulate
cleanly at the intermediate camera estimates from translation averaging.
**Default behavior is byte-identical to master.** All new flags default off;
`tracks_2d` defaults None. Existing callers see zero change.
## What it does
When `use_multi_view_retriangulation=True`, after the existing BA loop converges:
1. **Re-triangulate** the union-find 2D track set against the post-BA cameras
using multi-view RANSAC (via existing `Point3dInitializer`). Samples camera
pairs, triangulates with DLT, scores candidate inlier sets by reprojection
error across all track measurements.
2. **Run final BA** on the augmented track set. Cameras get a second pass of
refinement now that the geometry has more constraints.
That's it. Two ops. No filter, no loop.
## How to enable (config example)
Add `use_multi_view_retriangulation: true` to your `bundle_adjustment_module`
yaml config:
```yaml
bundle_adjustment_module:
_target_: gtsfm.bundle.bundle_adjustment.BundleAdjustmentOptimizer
reproj_error_thresholds: [10, 5, 3] # existing config — unchanged
use_multi_view_retriangulation: true # ← new flag, opt-in
# Optional tuning (sensible defaults; only override if you know what you want):
# mv_retri_min_track_length: 3
# mv_retri_reproj_error_thresh: 10.0
```
The `MultiViewOptimizer` automatically threads its existing `tracks2d_graph`
through to BA — no caller changes needed.
## Validated results (IMC phototourism scenes, sift_gp_single_pt config)
| Scene | num_matched, min_score | Without retri | With retri | Δ |
|----------------|------------------------|---------------|------------|--------|
| British Museum | (100, 0.20) | AUC@5 = 65.8 | AUC@5 = 69.5 | +3.7 |
| Brussels | (100, 0.15) | — | AUC@5 = 79.5 | (vs GLOMAP-full 81.2) |
| Pantheon | (15, 0.45) | — | AUC@5 = 79.9 | strong |
Wall-time: one extra BA pass (~10-15s on BM-scale).
## API summary
New public function:
`multi_view_retriangulate_from_2d_tracks(gtsfm_data, tracks_2d, ...) -> GtsfmData`
New `BundleAdjustmentOptimizer.__init__` flags (all default off / sensible):
`use_multi_view_retriangulation: bool = False`
`mv_retri_min_track_length: int = 3`
`mv_retri_reproj_error_thresh: float = 10.0`
Plumbing:
- `tracks_2d: Optional[List[SfmTrack2d]]` kwarg added to `_run_ba_and_evaluate`
and `create_computation_graph`
- `MultiViewOptimizer` passes its existing `tracks2d_graph` to
`ba_optimizer.create_computation_graph` (single-line change)
## How to review
1. Read `multi_view_retriangulate_from_2d_tracks` (~80 lines, single loop, well-commented).
2. Read the `if self._use_multi_view_retriangulation:` block in `_run_ba_and_evaluate` (~40 lines, the integration point).
3. Verify new `__init__` flags default off and `tracks_2d` defaults None (backwards compat).
4. Verify the `MultiViewOptimizer` change (one line: `tracks_2d=tracks2d_graph,` added to `ba_optimizer.create_computation_graph`).
Total diff: 165 lines added, 1 deleted, across 2 files.
## What this PR does NOT add (saved for follow-ups)
- **2-view densification** (visualization-only post-BA point-cloud densification).
Adds ~24% point cloud density on Sacre Coeur with no AUC impact. Will follow
as a separate PR if needed for visualization use cases.
- **Full GLOMAP `IterativeRetriangulateAndRefine` port** (DeleteAllPoints +
per-image `TriangulateImage` + 5-round refinement loop). Implemented and
validated correct on synthetic tests in the `gp-glomap-parity` experimental
branch but ~15x slower than C++/Ceres in Python/GTSAM, so uneconomical to ship.
Recoverable from git history if anyone pursues a Ceres-backed reimplementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kathirgounder
force-pushed
the
retri-densify-pr
branch
from
April 27, 2026 23:56
b1a34cb to
a659bbb
Compare
After merging master's BundleAdjustmentOptions / new __init__ params, both the pre-merge default (1e-8) and master's new default (0.0) ended up in the __init__ signature. The duplicate kwarg made the second definition (0.0) silently override the first — kept that one and removed the stale 1e-8 line. Also moved the retri opt-in flags to the bottom of the __init__ signature so the master-side BA params group cleanly first. Verified: - factor_weight_outlier_threshold: 0.0 (matches master default) - use_multi_view_retriangulation: False (default off — backwards compat) - opt-in works: BundleAdjustmentOptimizer(use_multi_view_retriangulation=True) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The function uses Point3dInitializer.triangulate, which samples camera PAIRS as RANSAC hypotheses (2-view DLT), scores inliers across the whole track, then computes the final 3D point via multi-view DLT on inliers. Calling it "multi-view RANSAC" oversold what the sampling step actually does. Also dropped the 'GLOMAP-style' framing (this is gtsfm code, no need to reference an experimental branch's lineage).
akshay-krishnan
approved these changes
Apr 28, 2026
Akshay's review feedback (PR borglab#1111): - max_num_hypotheses was hardcoded at 100 — now configurable via mv_retri_max_num_hypotheses on BundleAdjustmentOptimizer.__init__. Default kept at 100; users can lower for long tracks where the cap binds. - Re: mv_retri_reproj_error_thresh = 10.0 'seems way too high': Kept default at 10.0. This threshold serves both the RANSAC inlier check AND the final-track-acceptance gate (Point3dInitializer.triangulate rejects the entire track if any inlier measurement exceeds the threshold). 10.0 was validated to give the +3.7 AUC@5 lift on BM, and matches the data_association_module's existing convention. GLOMAP's analogous TriangulatorOptions actually uses 15.0px for complete/merge max reproj error during build phase, so 10.0 is on the tighter side of the regime. Made the threshold tunable via mv_retri_reproj_error_thresh — users can lower for stricter track acceptance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kathirgounder
marked this pull request as ready for review
April 28, 2026 02:23
Removed the loop for BA steps and replaced it with a single call to run_ba, simplifying the bundle adjustment process.
kathirgounder
added a commit
to kathirgounder/gtsfm
that referenced
this pull request
Apr 28, 2026
Mirrors the retri stage merged in borglab#1111 (BundleAdjustmentOptimizer) at the cluster-BA layer so VGGT-frontend pipelines also recover tracks dropped earlier. After the initial cluster BA + filter, optionally re-triangulates the union-find 2D tracks against the post-BA cameras (multi_view_retriangulate_from_2d_tracks) and runs a second cluster BA on the augmented set. Default off (use_multi_view_retriangulation: false) → no behavior change. ## Diff - `_run_cluster_ba` (shared by ClusterVGGT and ClusterVGGTWithFrontend) gains `tracks_2d` and `use_multi_view_retriangulation` kwargs. When the flag is on and tracks_2d is supplied, runs retri + a second BA + post-BA reproj filter. - `ClusterVGGTWithFrontend.__init__` adds `use_multi_view_retriangulation: bool = False` and passes the union-find tracks_2d_graph through to `_run_cluster_ba`. - 3 yamls (vggt_sift_frontend_megaloc, _phototourism, vggt_unified_frontend_megaloc) expose the flag with default false. ## How to enable ```yaml cluster_optimizer: optimizer: _target_: gtsfm.cluster_optimizer.VggtWithFrontend ... use_multi_view_retriangulation: true ``` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in post-BA retriangulation step. Default off — existing pipelines unchanged.
When
use_multi_view_retriangulation=True, after BA converges:Point3dInitializer). Recovers tracks the upstream pipeline dropped earlier when cameras weren't yet converged.How to enable
In your
bundle_adjustment_moduleyaml:The
MultiViewOptimizerthreads its existingtracks2d_graphthrough to BA automatically — no caller changes needed.Results on IMC scenes
Wall-time: one extra BA pass.
What's in the diff
multi_view_retriangulate_from_2d_tracksinbundle_adjustment.pyBundleAdjustmentOptimizer.__init__:use_multi_view_retriangulation,mv_retri_min_track_length,mv_retri_reproj_error_threshtracks_2dkwarg threaded throughcreate_computation_graphand_run_ba_and_evaluatemulti_view_optimizer.py:tracks_2d=tracks2d_graphpassed toba_optimizer.create_computation_graph