cv2.omnidir.calibrate - Assertion error(-215) - Python

[FilipSJohansson]

System information (version)

• OpenCV => 4.5.5
• Python => 3.9
• Operating System / Platform => Windows 64 Bit
• Compiler => Visual Studio 2019

Detailed description

When trying to calibrate a wide FOV camera using opencv-contrib-python the omnidir.calibrate() results in Assertion error (-215):
"OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv_contrib\modules\ccalib\src\omnidir.cpp:854: error: (-215:Assertion failed) !objectPoints.empty() && objectPoints.type() == CV_64FC3 in function 'cv::omnidir::internal::computeJacobian'"

Steps to reproduce

function inputs:
inputsDump.txt

load inputs:

file = open('inputsDump.txt', 'rb')
objectpoints, imagepoints, image_size, K, xi, D, calibration_flag, subpixel_criteria = pickle.load(file)

ret, K, xi, D, rvecs, tvecs, idx = cv2.omnidir.calibrate(objectpoints, imagepoints,image_size , K, xi, D, calibration_flag, subpixel_criteria)

Complete code

import cv2
assert cv2.__version__[0] >= '3', 'The fisheye module requires opencv version >= 3.0.0'
import numpy as np
import os
import glob
import matplotlib.pyplot as plt

# Globals
image_path = '../image_path/'
#output_path = '../output_path/'
image_extension = 'jpg'
image_size = (4032  , 3024)
checkerboard_size = (10,15)
subpixel_criteria = (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 300, 0.1)
fisheye_calibration_flags = cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC+cv2.fisheye.CALIB_CHECK_COND+cv2.fisheye.CALIB_FIX_SKEW
omnidir_calibration_flags = cv2.omnidir.CALIB_USE_GUESS + cv2.omnidir.CALIB_FIX_SKEW + cv2.omnidir.CALIB_FIX_CENTER + cv2.omnidir.CALIB_FIX_XI
draw_chessboard_marked_images = False

def getObjectAndImagePoints(imagePath, imageExtension, checkerboardSize, subpixelCriteria, drawChessboardMarkedImages):
    objp = np.zeros((1, checkerboardSize[0]*checkerboardSize[1], 3), np.float32)
    objp[0,:,:2] = np.mgrid[0:checkerboardSize[0], 0:checkerboardSize[1], ].T.reshape(-1, 2)
    _img_shape = None
    objpoints = [] # 3d point in real world space
    imgpoints = [] # 2d points in image plane.
 
    images = glob.glob(imagePath + '*.' + imageExtension)
    for fname in images:
        img = cv2.imread(fname)
        if _img_shape == None:
            _img_shape = img.shape[:2]
        else:
            assert _img_shape == img.shape[:2], "All images must share the same size."
        gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
        # Find the chess board corners
        ret, corners = cv2.findChessboardCorners(gray, checkerboardSize, cv2.CALIB_CB_ADAPTIVE_THRESH+cv2.CALIB_CB_FAST_CHECK+cv2.CALIB_CB_NORMALIZE_IMAGE)
        print('Nr of corners ' + str(corners.shape))
 
        # If found, add object points, image points (after refining them)
        if ret == True:
            corners = cv2.cornerSubPix(gray, corners, (11,11), (-1,-1), subpixelCriteria)
            corners = corners.reshape((150,2)) # This is necessary only for omnidirectional calibration
            imgpoints.append(corners)
            objpoints.append(objp)
 
            if drawChessboardMarkedImages:
                gray2 = gray
                cv2.drawChessboardCorners(gray2, checkerboardSize, corners, ret)
                cv2.imwrite(os.path.splitext(fname)[0] + '_' + 'CHESSCORNERS.png', gray2)
    return (objpoints, imgpoints)
 
 
def getOmnidirectionalCalibrationParams(objectpoints, imagepoints, calibration_flag, subpixel_criteria):
    K = np.zeros((3,3))
    D = np.zeros((1,4))
    xi = np.array([])
    idx = np.array([])
    no_images = len(imagepoints)
    no_points = len(imagepoints[0])
    
    #objectpoints = np.reshape(objectpoints, (no_images, 1, no_points, 3))
    objectpoints = np.array(objectpoints,dtype=np.float64).reshape(no_images, 1, no_points, 3)
    #imagepoints = np.reshape(imagepoints, (no_images, 1, no_points, 2))
    imagepoints = np.array(imagepoints,dtype=np.float64).reshape(no_images, 1, no_points, 2)
    print('shape of objectpoints ' + str(objectpoints.shape))
    print('shape of imagepoints ' + str(imagepoints.shape))
    
    print("size of objectpoints " + str(objectpoints.size))
    print("size of imagepoints " + str(imagepoints.size))
    ret, K, xi, D, rvecs, tvecs, idx = cv2.omnidir.calibrate(objectpoints, imagepoints,image_size , K, xi, D, calibration_flag, subpixel_criteria)
    
    return ret, K, xi, D, rvecs, tvecs, idx
 

# Find camera calibration parameters
objectpoints, imagepoints = getObjectAndImagePoints(image_path, image_extension, checkerboard_size, subpixel_criteria, draw_chessboard_marked_images)
         
ret, K, xi, D, rvecs, tvecs, idx = getOmnidirectionalCalibrationParams(objectpoints, imagepoints, omnidir_calibration_flags, subpixel_criteria)
 
print("Found " + str(len(imagepoints)) + " valid images for calibration")
print("K = np.array(" + str(K.tolist()) + ")")
print("D = np.array(" + str(D.tolist()) + ")")
print("xi = np.array(" + str(xi.tolist()) + ")")

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

[alalek]

Please provide complete minimal reproducer including necessary input data (nobody has access to your disk C:).

"minimal" means that we need the "broken" call only.
You could dump passed inputs of this call into a file and load them in reproducer code (e.g, using cv::FileStorage or Python pickle).

[FilipSJohansson]

I have now included a input dump aswell.

[alalek]

Thank you for update!
I can reproduce and confirm the problem.

Unfortunately this module doesn't have any tests. Probably its code is permanently broken.

---

Data is passed properly:

# InputArrayOfArrays: empty()=false kind=0x00050000 flags=0x01050000 total(-1)=8 dims(-1)=1 size(-1)=8x1 type(0)=CV_64FC3 dims(0)=2 size(0)=150x1
print(cv.utils.dumpInputArrayOfArrays(objectpoints))
# InputArrayOfArrays: empty()=false kind=0x00050000 flags=0x01050000 total(-1)=8 dims(-1)=1 size(-1)=8x1 type(0)=CV_64FC2 dims(0)=2 size(0)=150x1
print(cv.utils.dumpInputArrayOfArrays(imagepoints))

---

Algorithm has NaNs here:

• reprojectError: https://github.com/opencv/opencv_contrib/blob/4.5.5/modules/ccalib/src/omnidir.cpp#L680
• and here: https://github.com/opencv/opencv_contrib/blob/4.5.5/modules/ccalib/src/omnidir.cpp#L712

which causes empty _idx and empty loop here.

Finally it fails !objectPoints.empty() check and triggers the observed error.

---

gamma is NaN here due to negative sqrt() argument.

[alalek]

Similar problems are here: #2621 #804

[FilipSJohansson]

I guess this problem is related to release 4.5.5, and not due to python/images?

[gred0216]

I found a workaround to the assertion issue somewhere else. By expanding the dimensions of the imagepoints and objpoints array, they can be passed into the calibrate functions with no error.

objpoints_expand = np.expand_dims(np.asarray(objpoints), -2)

Here are the dimensions and type of the array.

print(objpoints[0].shape) # (88, 3)
print(objpoints_expand.shape) # (7, 88, 1, 3)
print(oe.dtype) # dtype('float64')

Hope this would help someone who has the same issue.

[maxrobic]

hey there,

same issue with the assertion using omnidir::calibration, even if I have already registered my objectPoints and my imagePoints with another detection algorithms. Did someone find a fix?

Best

[ManthanSh]

Hi, Finally!! This works now for me :) Thanks to @gred0216

objpoints_expand = np.expand_dims(np.asarray(objpoints,dtype=np.float32), -2)
imgpoints_expand = np.expand_dims(np.asarray(imgpoints,dtype=np.float32), -2)
subpixel_criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 50, 0.0001)
ret, K, xi, D, rvecs, tvecs,_ = cv2.omnidir.calibrate(objpoints_expand, imgpoints_expand, gray.shape[::-1], None,None,None,flags=cv2.omnidir.CALIB_FIX_SKEW,criteria=subpixel_criteria)

Basically, it needs objpoints_expand to be NxMx1x3 and imgpoints_expand to be NxMx1x2 shape.
Where,

• N: Number of poses used for calibration
• M: Total inner corners of checkerboard
  For this code specifically your objpoints and imgpoints should be of shape NxMx3 and NxMx2 respectively.

Although, the reprojection error (ret parameter) that I am getting from this is very high. Approximately ~20-30px. Let me know if there is something I am missing in this or if I need to add some more flags or parameters.

[wengflow]

I am still having this error:

error: (-215:Assertion failed) !objectPoints.empty() && objectPoints.type() == CV_64FC3 in function 'computeJacobian'

for running:

cv2.omnidir.calibrate(
    np.ones((20, 40, 1, 3), dtype=np.float32),
    np.ones((20, 40, 1, 2), dtype=np.float32),
    (1080, 1080),
    None, None, None,
    flags=cv2.omnidir.CALIB_FIX_SKEW,
    criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 50, 0.0001),
)

with Python 3.10.17, opencv-contrib-python 4.11.0.86 and numpy 2.2.6.

@ManthanSh May I know what Python, opencv-contrib-python and numpy version are you using? Does the code above work for you?

[harshit-aimonk]

@wengflow Did you get the solution to fix this?

[wengflow]

I don't remember the specifics, but I think the object points need to be a 1D list of (N, 1, 3) numpy arrays, and image points need to be a 1D list of (N, 1, 2) numpy arrays, as returned by cv2.findChessboardCorners() or cv2.cornerSubPix().
Try it directly on some real data, instead of the dummy np.ones((20, 40, 1, 2 or 3), dtype=np.float32) above, as I vaguely recall that it will fail due to degeneracy.

Hope this helps 😃

[harshit-aimonk]

@wengflow Thanks for this the calibration worked this time