Standard Deviation calculation of calibrateCamera incorrect

[Seneral]

System information (version)

• OpenCV => 4.5.1
• Operating System / Platform => Ubuntu 64 Bit
• Compiler => gcc

Detailed description

The standard deviation calculation of calibrateCamera is returning NaN for all standard deviations in all cases when provided with comparably low numbers of points.
This is due to this line, where the divisor and thus sigma2 can become negative. The variable total usually is nimages*maxPoints or less, nparams = nimages*6+18 (for standard calibration method), and nparams_nz < nparams.
For the case total < nparams_nz < nparams the resulting sigma2 is negative.
This can (and nearly always does) happen when maxPoints = 6 so total/nparams < 1.
This is likely also why it has been unnoticed so far.

Changing the total to nparams yields seemingly correct results. EDIT: Only for my case where maxPoints = 6! This might not be a general or even correct fix!

I've also opened a forum question previously, although I'm positive it is a bug.

Steps to reproduce

1. Generate a sample set of 22 6-point markers
2. Call calibrateCameraRO, requesting the standard deviations to be calculated

• One sample run yielded the following variable values: total=132 (22*6), nparams=150(22*6+18), nparams_nz=139

3. Observe the returned standard deviations to all be NaN

An example implementation can be found in this larger project (calibrateCameraRO call here).

Issue submission checklist

• [ X] I report the issue, it's not a question
• [X ] I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found solution
• [ X] I updated to latest OpenCV version and the issue is still there
• [ X] There is reproducer code and related data files: videos, images, onnx, etc

[sanketc001]

is this bug reproducible on windows or is this system dependent?

[Seneral]

System independent, the same code runs on linux and windows (though rn I'm away from my windows machine so I can't verify the fix works)

[Seneral]

I want to note that I overlooked one critical thing in my proposed fix:
nparams = total+18 is wrong (with total <= nimages*maxPoints), instead it is nparams = nimages*6+18
It just so happened that for me maxPoints = 6.
Which would imply for 'normal' calibrations (with checkerboards of up to 100 points each) the divisor would be drastically different now, by a factor of ~20.

So we'd need someone with knowledge of the algorithm to verify that it is indeed nparams that belongs there instead of total (implying standard deviation calculations were even more off than I had thought) or if this is an edge case that needs to be handled separately.
Hence I will not make a PR for now.

[alalek]

Perhaps standard deviation can't be estimated for that case with maxPoints = 6 (or other cases where number of unknown parameters is larger than inputs).

Not sure how Least Squares Estimation problem works for that case, so probably we should update this check.

/cc @catree

[lamm45]

I don't see any obvious errors, but I think the code is slightly relying on undefined behavior and numerical perfectness.

Let us consider three different situations:

1. In the typical case total > nparams_nz, the problem is overdetermined, that is, there are more equations than unknowns. Unless the input data (i.e., image points and world points) were acquired perfectly, it is generally not possible to satisfy all equations simultaneously. Therefore, even with an optimized parameter vector there will be nonzero reprojection errors. The inaccuracies in the input data can be propagated to parameter uncertainty through the reprojection errors and by suitably scaling with the Jacobian that describes the sensitivity between the input and output. This is exactly what the code does and the result is often called standard error (instead of standard deviation).

2. If total == nparams_nz, then the number of equations is equal to the number of unknowns, and thus we would expect to have exactly one parameter vector that gives zero reprojection errors. Because there are no redundant equations or redundant input data, it does not make sense to estimate input inaccuracies or compute uncertainty in parameters. In my opinion, NaN is the correct result.

3. If total < nparams_nz, then the problem is underdetermined, that is, there are more unknowns than equations. In this case, one would expect to have infinitely many parameter vectors that result in zero reprojection errors. Again, we cannot estimate inaccuracies in the input data, and thus computing uncertainty in the solution is impossible. I think NaN is again the correct result, although one could probably also argue that the standard error is infinite.

The above observations apply directly to linear, unconstrained least-squares problems. The camera calibration problem is nonlinear with some (non-negativity) constraints. However, the last stage of the calibration is optimization with Levenberg-Marquardt method which is based on linearization, so in practice the observations still hold. Let me know if you agree or disagree.

If you agree that the correct result in cases 2) and 3) is NaN, then I think it should be made more explicit in the code. In particular, case 2) is unlikely to result in NaN even in IEEE 754 conformant implementations, because reprojection errors in the numerator (here) are unlikely to be exactly zero, whereas the integer denominator will be zero. Consequently, the standard error value will be positive or negative infinity (I didn't yet try an example where this would actually happen).

[savuor]

I think that the problem is that in stddev calculation the total should be 2x bigger (each point has 2 projected screen coordinates and therefore gives 2 more residuals), see the discussion under #22992

[lamm45]

Yes. My post above was written under the (wrong) assumption that total denotes the number of equations in the least-squares problem. But if/when total is the number of 2D points, the number of equations becomes 2*total.

[savuor]

By the way, looks like we have the same bug here: https://github.com/opencv/opencv/blob/4.x/modules/calib3d/src/fisheye.cpp#L1597

Other calibration code should be checked for that too.