Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core Solver - Add Ceres Solver #174

Open
david-cattermole opened this issue Oct 30, 2020 · 8 comments
Open

Core Solver - Add Ceres Solver #174

david-cattermole opened this issue Oct 30, 2020 · 8 comments
Assignees
Labels
core solver Related to the low-level solver functions. maya plugin Maya API Plug-In (C++)
Milestone

Comments

@david-cattermole
Copy link
Owner

Feature

Ceres Solver is a state-of-the-art non-linear least squares solver and would likely show performance improvements over the currently supported CMinpack and LevMar solvers.

Ceres solver will be installed as part of #144, and so we should make sure mmSolver can use it as a solver type if available.

This will involve writing a new wrapper around our internal "solverFunc" for Ceres, and providing the maya.cmds.mmSolver Python command with the ability to use Ceres solver.

Software Versions

  • mmSolver version: v0.4.x
  • Maya version: All supported
  • Operating System (OS): All supported
@david-cattermole david-cattermole added core solver Related to the low-level solver functions. maya plugin Maya API Plug-In (C++) labels Oct 30, 2020
@david-cattermole david-cattermole added this to the v0.4.0 milestone Oct 30, 2020
@david-cattermole david-cattermole self-assigned this Oct 30, 2020
david-cattermole added a commit that referenced this issue Nov 27, 2021
This includes building gflags, glog and eigen.

We also include build recipies for CMinpack and LevMar, so we
can deprecate the older methods of building and keep a single
cross-platform CMake approach.

Issues #144 #150 #174

Change-Id: Id4a8fb7c29be44f58101fdf555444078da91fed6
@david-cattermole
Copy link
Owner Author

Ceres Solver has been added as a dependency in v0.4.0, however Ceres is not yet supported as a solver back-end.

Progress will continue to add support as a Solver back-end.

@david-cattermole david-cattermole modified the milestones: v0.4.0, v0.5.0 Nov 6, 2022
@bhack
Copy link

bhack commented Jul 30, 2023

What kind of features listed in the bullet points at #144 (comment)

are already covered by OpenMVG?

@david-cattermole
Copy link
Owner Author

Hello @bhack,

OpenMVG provides Fundamental Matrix solving, Essential Matrix Solving and Unconstrained Bundle Adjustment (via Ceres Solver). These have been used in the Camera solver in mmSolver, although I'm not very happy with my implementation as I don't find it very robust, and can provide vastly different results each run of the solver.

OpenMVG does not provide 2D pattern tracking, only libmv can provide that. OpenMVG only provides photogrammetry-style matching using SIFT-like image feature detection. 2D pattern tracking is a non-goal of MM Solver (at least in the immediate future), so not having that feature is a non-issue.

David

@bhack
Copy link

bhack commented Jul 30, 2023

These have been used in the Camera solver in mmSolver, although I'm not very happy with my implementation as I don't find it very robust, and can provide vastly different results each run of the solver.

Do you have any specific plan to improve on this?

I see that you also import tracks from 3Dequalizer. Have you compared with its own solver?

@david-cattermole
Copy link
Owner Author

I do not have a specific plan, however I have a general idea of what needs to be improved.

Generally, the main issue with quality is caused because the bundle adjustment step is only performed every 5 camera poses that are approximated. Changing this value to 1 improves the quality overall but results in a much slower camera solve. I have experimented with changing the camera solver algorithm to find a good compromise.

The bundle adjustment performance can probably be improved by using the Bundle Adjustment function inside OpenMVG (which uses Ceres). This is not yet implemented, and the Camera solver is currently using the mmSceneGraph and the mmSolver core for bundle adjustment.

Yes, importing 2D Marker data is supported from 3DEqualizer and Blender (which is using libmv for 2D tracking and solving) and any other software that can export the .uv file format (which is pretty simple).

I test against 3DEqualizer solved camera output, or any other software, as long as the markers and bundles match with a low error/deviation and is physically plausible, I consider that correct. Generally my goal is to solve an image sequence of say 250 frames with 20 to 40 markers/bundles in under 1 second.

You can see the current camera solver implementation here:
https://github.com/david-cattermole/mayaMatchMoveSolver/blob/be8813fcbb991eebe69fc10dca9df44d5729371b/python/mmSolver/_api/solvercamera.py
I have an uncommitted camera solver which improves the quality, with a big performance drop. Until I can speed up the camera solver (perhaps by using Ceres solver for bundle adjustment), I won't be releasing my changes.

Thanks,
David

@bhack
Copy link

bhack commented Jul 31, 2023

Thanks for the details..
do you have any empirical estimation (I don't know if you have small internal benchmark) about the 3d equalizer solver performances against what we currently have in mmsolve?

@david-cattermole
Copy link
Owner Author

mmSolver has a test suite to test for correctness, but there isn't any performance benchmark tests (other than me being annoyed that tests take too long to finish).
https://github.com/david-cattermole/mayaMatchMoveSolver/blob/master/tests/README.md

Anecdotally, 3DEqualizer's camera solver is the fastest and most robust solver for SfM tasks that I've used.

I am not intending mmSolver to compete directly with 3DEqualizer's camera solver, instead mmSolver provides a different type of solver and focuses on getting results that 3DEqualizer cannot solve; for example the core of mmSolver allows arbitrary transform hierarchies with connections between objects, utilizing the Maya scene graph and the ability to solve arbitrary attributes. The mmSolver camera solver is intended to provide an even lower number of minimum points (only 5 points needed per-frame, as opposed to 3DEqualizer's 6 points per-frame), and is also intended to be used to solve complex camera moves that need to be stitched together, and combined with the mmSolver core solver with arbitrary attributes and transform hierarchies.

@bhack
Copy link

bhack commented Aug 1, 2023

Ok thanks for the details. I subscribe to this ticket to check if we have some updates for the OpenMVG/Ceres bundle adjustment.

@david-cattermole david-cattermole modified the milestones: v0.5.0, v0.6.0 Sep 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core solver Related to the low-level solver functions. maya plugin Maya API Plug-In (C++)
Projects
None yet
Development

No branches or pull requests

2 participants