PnP issue 15647 #15650

tobycollins · 2019-10-05T15:40:31Z

This pullrequest changes

This pull request is designed to fix the problems discussed in #15647

A PnPSolver base class has been created according to the OpenCV algorithm-based design guidelines. This is derived by classes that implement a PnP solver algorithm, and also for specifying important properties of the PnP solver, implemented as virtual member functions. These include:

minPointNumber: specifies the minimal number of points that can be handled by the solver (e.g. EPnP requires 4 and DLT requires 6)
maxPointNumber: specifies the maximal number of points that can be handled by the solver (e.g. AP3P handles at most 4 points, while others can handle an arbitrary number of points)
requires3DObject: specifies if the PnP solver requires non co-planar model points
requiresPlanarObject: specifies if the PnP solver requires co-planar model points
requiresPlanarTagObject: specifies if the PnP solver requires co-planar model points arranged in a square tag formation
artificalDegeneracy: warns if the object points & image points will cause an artificial degeneracy

The PnPSolver class provides the function geometryWarn that tests the input object points and image points for whether they will will cause the solver to fail. This involves running basic tests (checking number of points, checking for co-linear points etc.) and also running specific checks for each algorithm, as implemented in artificalDegeneracy.
All the currently used PnP solvers in OpenCV have been implemented as derived classes of PnPSolver: EPnP, DLT, DLS, P3P, AP3P, Zhang, IPPE and IPPESquare.
A PnPRefiner base class has bee created according to the OpenCV algorithm-based design guidelines. This is derived by classes that implements refinement algorithms. All the currently used refinement algorithms in OpenCV have been implemented as derived classes: C API Levenberg Marquardt, C++ API Levenberg Marquardt and VVS.
The public interfaces: solvePnP and solvePnPRansac are unchanged. When these are called, PnPSolver and PnPRefiner objects are created according to the flags passed to SolvePnP and SolvePnPRansac respectively.
The public function: pnp has been created. This is used to solve a PnP problem using PnPSolver and PnPRefiner objects. This unifies all PnP functionality in OpenCV, including P3P, in a single interface.
A new test set has been created (test_pnp_precision.cpp) that does the following:

Tests pnp with all combinations of PnPSolver and PnPRefiner objects for solver precision, using all combinations of precision and formats for object points, image points, intrinsics and distortion coefficients. This exhaustive test makes testing a new PnPSolver or PnPRefiner classes easy.
Tests all PnPRefiner classes using an imprecise initial pose estimate.
Documents corner cases that cause one or more PnPSolver to fail (also called artificial degeneracies). This set can grow if more corner cases get included by the community.

Credit should also go to @catree and @alalek for the recent work on improving solvePnP #14431 #14362 #14181

alalek · 2019-10-05T15:42:26Z

Please don't reopen PRs about the same/similar things.

tobycollins · 2019-10-05T16:27:03Z

Sorry about the double PR.

minor fixes fix whitespace errors reverted to previous solvePnPGeneric interface removed whitespace errors fixed doc issues fixed doc issues 2 fixed doc issues 3 fixed minor warnings fixed minor warnings 2

paroj · 2019-10-06T14:18:06Z

modules/calib3d/include/opencv2/calib3d.hpp

+
+/** @brief create creates a pointer to a new instance of this class
+
+    @param withGeometricTests set this true if you want to execute geometryWarn before solving pnp.


this refers to the non existing geometryWarn (probably validProblem ) function. Use @ref to get an buildbot error for this.
Also I would recommend using @copydoc PnPSolver::PnPSolver for the create functions to avoid copy & paste

paroj · 2019-10-06T14:19:32Z

I think documenting the corner cases of the different algorithms alone is an invaluable contribution.

I just have some minor code-style suggestions.

paroj · 2019-10-06T14:22:46Z

modules/calib3d/include/opencv2/calib3d.hpp

+    @param _opoints
+    @return true if correct, false otherwise
+    */
+    CV_WRAP bool isPlanarTag(InputArray _opoints) const;


this does not use any members and thus can be static

paroj · 2019-10-06T14:27:47Z

modules/calib3d/include/opencv2/calib3d.hpp

+
+    @returns true if required, false otherwise
+     */
+    CV_WRAP virtual bool requires3DObject() const = 0;


how about using a single int getCaveats() const function returning a mask like PNP_CAVEAT_PLANAR | PNP_CAVEAT_TAG instead of the three functions below?

The same mask could be also returned from validProblem to signal which check failed.

alalek

Thank you for working on this improvement!

At first we need to keep library changes compatible. We should not break existing User applications. Or existing tests.
Try unchanged tests (reverted code changes) and run them - there are several failed tests:

[  FAILED  ] 5 tests, listed below:
[  FAILED  ] Calib3d_SolveP3P.accuracy
[  FAILED  ] Calib3d_SolvePnPRansac.double_support
[  FAILED  ] Calib3d_SolvePnP.generic
[  FAILED  ] Calib3d_SolvePnP.refine3pts
[  FAILED  ] Calib3d_SolvePnP.refine

In general, tests should be added only - not changed.

Suggestion from @paroj with flags for geometric restrictions sounds good.
Checks are applied for "object points" only. Sometimes it make sense to handle them once (move to ctor/create to pass them once?).
For example, factory with "object points" can analyze them and selects proper solver (additional to algorithm-specific factories).

alalek · 2019-10-07T11:51:08Z

modules/calib3d/include/opencv2/calib3d.hpp

-                              //!<   - point 1: [ squareLength / 2,  squareLength / 2, 0]
-                              //!<   - point 2: [ squareLength / 2, -squareLength / 2, 0]
-                              //!<   - point 3: [-squareLength / 2, -squareLength / 2, 0]
+    //!< This is a special case suitable for marker pose estimation.\n


Please revert these unnecessary whitespace changes (and many other below).

alalek · 2019-10-07T11:56:31Z

modules/calib3d/include/opencv2/calib3d.hpp

+class CV_EXPORTS_W PnPSolver : public Algorithm
+        #else
+class CV_EXPORTS_W PnPSolver : public virtual Algorithm
+        #endif


Why virtual is needed?

alalek · 2019-10-12T02:00:11Z

modules/calib3d/include/opencv2/calib3d.hpp

+
+    @returns Identity intrinsic matrix (3x3 single channel double)
+    */
+    CV_WRAP cv::Mat makeIdentityIntrinsic() const;


protected:
CV_WRAP

This should be removed from public API, so this can't be wrapped.

alalek · 2019-10-12T02:01:32Z

modules/calib3d/include/opencv2/calib3d.hpp

+    opoints are defined in object coordinates.
+    @return
+    */
+    CV_WRAP size_t getNumberOfPoints(InputArray opoints) const;


See Mat::checkVector()

alalek · 2019-10-12T02:05:01Z

modules/calib3d/include/opencv2/calib3d.hpp

+    the model coordinate system to the camera coordinate system.
+    @param tvecs Vector of output translation vectors (double)
+     */
+    CV_WRAP virtual void solve(InputArray opoints, InputArray ipoints, CV_OUT std::vector<Mat> & rvecs, CV_OUT std::vector<Mat> & tvecs) const = 0;


Why is this method protected? (I believe solveProblem() should be virtual instead)

alalek · 2019-10-12T02:06:53Z

modules/calib3d/include/opencv2/calib3d.hpp

+
+    @param withGeometricTests flag
+    */
+    CV_WRAP PnPSolverP3PComplete(bool withGeometricTests);


bool withGeometricTests = false to be consistent with create() interface

asmorkalov · 2019-11-07T05:55:47Z

@tobycollins Friendly reminder. Do you have a chance to fix notes?

asenyaev · 2021-04-07T20:14:53Z

jenkins cn please retry a build

initial commit

41f3cdf

minor fixes fix whitespace errors reverted to previous solvePnPGeneric interface removed whitespace errors fixed doc issues fixed doc issues 2 fixed doc issues 3 fixed minor warnings fixed minor warnings 2

tobycollins force-pushed the pip_issue_15647 branch from 1d75d81 to 41f3cdf Compare October 6, 2019 05:03

tobycollins added 2 commits October 6, 2019 08:02

fixed error on Win64 build

1bd3f19

more comments to test_pnp

477ecbc

tobycollins changed the title ~~Pip issue 15647~~ PnP issue 15647 Oct 6, 2019

tobycollins added 2 commits October 6, 2019 10:01

renamed test_pnp to test_pnp_precision

e60c7fd

fixed comments in test_pnp_precision

45f9f50

paroj reviewed Oct 6, 2019

View reviewed changes

alalek reviewed Oct 12, 2019

View reviewed changes

paroj mentioned this pull request Dec 10, 2019

Robust variant of solvePnPRefineLM #16114

Open

catree mentioned this pull request Aug 29, 2022

Add Robot-World/Hand-Eye calibration function #18203

Merged

6 tasks

catree mentioned this pull request Aug 17, 2024

Add a family of routines for evaluating confidence limits on parameters of the models optimized with SolvePnP and calibrateCamera #26020

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PnP issue 15647 #15650

PnP issue 15647 #15650

tobycollins commented Oct 5, 2019 •

edited

Loading

alalek commented Oct 5, 2019

tobycollins commented Oct 5, 2019

paroj Oct 6, 2019 •

edited

Loading

paroj commented Oct 6, 2019

paroj Oct 6, 2019

paroj Oct 6, 2019

alalek left a comment

alalek Oct 7, 2019

alalek Oct 7, 2019

alalek Oct 12, 2019

alalek Oct 12, 2019

alalek Oct 12, 2019

alalek Oct 12, 2019

asmorkalov commented Nov 7, 2019

asenyaev commented Apr 7, 2021


		/** @brief create creates a pointer to a new instance of this class

		@param withGeometricTests set this true if you want to execute geometryWarn before solving pnp.

PnP issue 15647 #15650

Are you sure you want to change the base?

PnP issue 15647 #15650

Conversation

tobycollins commented Oct 5, 2019 • edited Loading

This pullrequest changes

alalek commented Oct 5, 2019

tobycollins commented Oct 5, 2019

paroj Oct 6, 2019 • edited Loading

Choose a reason for hiding this comment

paroj commented Oct 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alalek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asmorkalov commented Nov 7, 2019

asenyaev commented Apr 7, 2021

tobycollins commented Oct 5, 2019 •

edited

Loading

paroj Oct 6, 2019 •

edited

Loading