Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sfm::reconstruct error with python error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef' #1675

Open
themantalope opened this issue Jun 27, 2018 · 22 comments

Comments

@themantalope
Copy link

@themantalope themantalope commented Jun 27, 2018

Hi Everyone,

I'm messing around with the sfm module and doing a 3D reconstruction from a video file. I've compiled the OpenCV project with the contributed modules on my machine (Mac OS X 10.11.6, python 3.6 Anaconda) with python bindings for sfm. Compilation was successful, and I am able to successfully run the example_sfm_scene_reconstruction program.

For the problem I'm having, the general program is as follows:

  1. Extract the location of ORB features for the first 8 images from the video
  2. Compile the features into a python list, where each element of the list is a 2XN (where N is the number of features detected in the image, typically 100 points so 2X100) numpy array.
  3. Load the camera calibration matrix from a previously stored file
  4. Call the sfm::reconstruct(InputArrayOfArrays points2d, OutputArray Ps, OutputArray points3d, InputOutputArray K, bool is_projective) via the python binding.

Code is as follows

# get all the keypoints, save them in a pandas dataframe
all_keypoints = []

for idx, im in tqdm(zip(range(idx_start, idx_stop),images),desc='gettingkeypoints',total=len(images)):
#     kps = fast_detector.detect(im)
    kps = orb_detector.detect(im)
    for kp in kps:
        all_keypoints.append({'frame':idx, 'x':kp.pt[0], 'y':kp.pt[1]})

# get the keypoints for the first 8 frames
samped_frames = list(set(kp_df.loc[:, 'frame'].values.tolist()))
samped_frames = samped_frames[0:8]
kps_for_recon = []
for idx in samped_frames:
    kps_data = kp_df.loc[kp_df['frame']==idx, ['x','y']]
    kps_for_recon.append(np.matrix(kps_data.values.transpose().astype(np.float32).copy()))
    
# get the camera calibration data
lcm = pd.read_csv(left_cam_cal_file, index_col=0)
lcm = lcm.values.astype(np.float32).copy()

# run scene reconstruction

cv2.sfm.reconstruct(kps_for_recon, lcm, None, None, True)

The code starts to run, then I get an error:
error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

Based on how I understand the sfm::reconstruct function works, I think the error is coming from the line linked below.

Mat(P).copyTo(Ps.getMatRef(i));

Camera matrix:

1685.72249 | 0.000000 | 689.894140
0.00000 | 1749.486859 | 767.726025
0.00000 | 0.000000 | 1.000000

Any help much appreciated. I'm really not sure where the error is coming from.

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

i don't have the sfm module (or python bindings for that), but:

if lcm is your camera Mat, it should go into the 4th argument(K), not the 2nd (Ps).
maybe you should use "named" arguments here, like:

ps, pts3d = cv2.sfm.reconstruct(kps_for_recon, K=lcm, is_projective=True)

can you also try a : help(cv2.sfm.reconstruct) , and show us the output ?
(so we can see, which signature was generated exactly)

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

Hey @berak

Here is the output of help(cv2.sfm.reconstruct):

Help on built-in function reconstruct:

reconstruct(...)
    reconstruct(points2d, K[, Ps[, points3d[, is_projective]]]) -> Ps, points3d, K
    .   @brief Reconstruct 3d points from 2d correspondences while performing autocalibration.
    .   @param points2d Input vector of vectors of 2d points (the inner vector is per image).
    .   @param Ps Output vector with the 3x4 projections matrices of each image.
    .   @param points3d Output array with estimated 3d points.
    .   @param K Input/Output camera matrix \f$K = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1}\f$. Input parameters used as initial guess.
    .   @param is_projective if true, the cameras are supposed to be projective.
    .   
    .   This method calls below signature and extracts projection matrices from estimated K, R and t.
    .   
    .   @note
    .   - Tracks must be as precise as possible. It does not handle outliers and is very sensible to them.

Changing the input arguments to named arguments resulted in the same error:

cv2.sfm.reconstruct(kps_for_recon, K=lcm,is_projective=True)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-96-c9a12494b6e8> in <module>()
----> 1 cv2.sfm.reconstruct(kps_for_recon, K=lcm,is_projective=True)

error: OpenCV(4.0.0-pre) /Users/<path>/opencv/modules/core/src/matrix_wrap.cpp:1735: error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

oh, thanks for the report !
(ok, K is mandatory, so the args get reordered, and your initial call is the same as with named args.)

hmm, you might be the 1st person ever, to try this from python, unfortunately.

any chance you still have the opencv build folder ? looking at build/modules/python_bindings_generator/pyopencv_generated_funcs.h (and finding the reconstruct() wrapper) might be helpful.

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

Ok found it, it's interesting:

static PyObject* pyopencv_cv_sfm_reconstruct(PyObject* , PyObject* args, PyObject* kw)
{
    using namespace cv::sfm;

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    Mat Ps;
    PyObject* pyobj_points3d = NULL;
    Mat points3d;
    PyObject* pyobj_K = NULL;
    Mat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }
    PyErr_Clear();

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    UMat Ps;
    PyObject* pyobj_points3d = NULL;
    UMat points3d;
    PyObject* pyobj_K = NULL;
    UMat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }

    return NULL;
}

It appears that the order of the keyword arguments in the python wrapper don't align with the order of arguments (passed?) to the C++ sfm::reconstruct function. However, this doesn't have any bearing on functionality and another error is thrown. Based on what I see, it looks like the python wrapper correctly maps the python keyword arguments to the C++ arguments.

I'm wondering if the problem is that I'm using the latest version of OpenCV (4.0.0-pre) and OpenCV contrib modules, and the behavior of some of the core functions is a bit different.

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

oh, that is really helpful !

It appears that the order of the keyword arguments in the python wrapper don't align with the order of arguments (passed?) to the C++ sfm::reconstruct function.

the python wrapper script has an internal rule for this:

  • InputArray's are mandatory, and thus go to the front of the args list
  • InputOutputArray's are mandatory, too, and also get returned, so they go in the middle
  • OutputArrays (usually) do not need to get initialized, thus go to the end of the list (and get returned from the python function)

I'm wondering if the problem is that I'm using the latest version of OpenCV (4.0.0-pre)

i don't think so. imho, there's a bug in the python generator(script), and not much changed there between 3.4 and 4.0

but here's the bug !

    PyObject* pyobj_Ps = NULL;
    Mat Ps;

remember the errormsg ? it should be vector_Mat Ps, not Mat Ps here !
(vector_Mat is just some lube/typedef, to make it smoothly though the python based parser)

you can easily reproduce it with 4 lines of c++:

    Mat v; // should be vector<Mat> but isn't.
    OutputArray o(v); // entering reconstruct() from python wrappers
    o.create(5,1,5); // assuming: nviews==5, depth==CV_32F
    Mat m = o.getMatRef(0);

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

@berak

Ok, so how does this get fixed? Which line of code in the reconstruct function should be changed?

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

Ok, so how does this get fixed?

good one ;)

i have no (immediate) idea.
no, the problem is not one of the reconstruct() overloads (or in c++)

we'll have to dive in here: https://github.com/opencv/opencv/tree/master/modules/python/src2

https://docs.opencv.org/master/df/da2/tutorial_py_table_of_contents_bindings.html

and again, you seem to be the 1st person to try this from python, aka -- "our man on the moon" !

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

Ok, so based on this issue, I changed the CV_EXPORTS macro in my reconstruct.hpp file to CV_EXPORTS_W:

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArray Ps, OutputArray points3d, InputOutputArray K,bool is_projective = false);

Is there another macro that needs to be changed to the Ps gets converted to the proper object type?

I could also try to change which overloaded method of reconstruct gets exported....

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

hmm, what if you change the c++ sig to:

void sfm::reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArray points3d, InputOutputArray K, bool is_projective);

( OutputArrayOfArrays should generate a vector instead of a single Mat (OutputArray) in python )

? (and rebuild the whole schlepp)

(same might be needed for points3d, idk.)

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

@berak well the points3d should just be the points corresponding to the reconstructed 3d scene, so I would expect them to be output in one array. The Ps should be the [R|t] matricies corresponding to the (relative) rotations and translations of the camera for each picture used in reconstruction.

Once more, unto the breach (of recompiling)!

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

yea, true.

all i was trying to point out was:

if the python generator script encounters an OutputArrayOfArrays , it will generate a vector<Mat> (correct for Ps), if it sees an OutputArray only it will generate a Mat (problem)

so, well, shift in the diagnosis again: imho, the c++ sig is wrong, so the py generator does the wrong thing

(and the Ps prob has to be mended, before going on to pts3d)

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

Ok, so I made the changes suggested and rebuilt. There are now modifications to the pyopencv_generated_funcs.h file:

static PyObject* pyopencv_cv_sfm_reconstruct(PyObject* , PyObject* args, PyObject* kw)
{
    using namespace cv::sfm;

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    vector_Mat Ps;
    PyObject* pyobj_points3d = NULL;
    Mat points3d;
    PyObject* pyobj_K = NULL;
    Mat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }
    PyErr_Clear();

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    vector_Mat Ps;
    PyObject* pyobj_points3d = NULL;
    UMat points3d;
    PyObject* pyobj_K = NULL;
    UMat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }

    return NULL;
}

And Psis now a vector mat. However, re-running the python code still gives the same error....

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

The other weird thing I'm now noticing is in the example scene reconstruction C++ example:

  bool is_projective = true;
  vector<Mat> Rs_est, ts_est, points3d_estimated;
  reconstruct(images_paths, Rs_est, ts_est, K, points3d_estimated, is_projective);

Here they are creating the Rs and ts vectors of matricies, and using them in the reconstruct function. However even in that version of the reconstruct function, shouldn't they be OutputArrayOfArrays?

The function declaration in the header file reconstruct.hpp is as such:

CV_EXPORTS
void
reconstruct(const std::vector<String> images, OutputArray Rs, OutputArray Ts,
            InputOutputArray K, OutputArray points3d, bool is_projective = false);

Additionally the reconstruction example works on my machine. Not sure what the exact issue is here.

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 27, 2018

Ok, strange behavior but I got it to work. I had to change the reconstruct.hpp file as such:

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArrayOfArrays points3d, InputOutputArray K, bool is_projective = false);

Seems like the object type of the input argument wasn't correct, and hence the python bindings were generating the incorrect object types for the function. I've updated the code on my forked version of the repo. @berak are you a maintainer for the sfm module in the opencv_contrib repo?

@berak
Copy link
Contributor

@berak berak commented Jun 27, 2018

no, i'm not the maintainer of this, i'm only out for a nice "detective story" , now & then.

do you feel apt to make a pr with the changes ? if so, -- please do ! (your bug, your fix, your honour.)

(and only 1 overload (the one marked CV_EXPORTS_W) will be wrapped into python/java/matlab, so only that one needs to be fixed, imho.)

@themantalope
Copy link
Author

@themantalope themantalope commented Jun 28, 2018

Yeah I don't have a problem making a PR, I have the changes saved on my fork of the repo. Just haven't contributed here before so just want to make sure I that whatever I do is in the style of the people who are maintaining the project.

@berak
Copy link
Contributor

@berak berak commented Jun 28, 2018

https://github.com/opencv/opencv/wiki/How_to_contribute (if you have not found it yet)

@pierslawrence
Copy link

@pierslawrence pierslawrence commented Nov 13, 2018

I have been trying to get the same issue sorted out. I am that your fork will do the trick.

I do get a warning that
warning: "CERES_FOUND" redefined`

coming from your insertion of

#define CERES_FOUND true

in sfm.hpp and reconstruct.hpp.

It seems that only when the python bindings are compiled that this flag is not set externally, do you know if there might be a better way around this?

I looked on the pull requests, but did not see this issue in there. I certainly would find this fix useful so that the reconstruct function will eventually make it to the masses.

@alalek
Copy link
Contributor

@alalek alalek commented Nov 13, 2018

python bindings are compiled that this flag is not set externally

Python (and other bindings) should use public OpenCV API only.
Public API should not require any extra flags.

@pierslawrence
Copy link

@pierslawrence pierslawrence commented Nov 13, 2018

Perhaps I don't fully understand the build process. Are there other examples in OpenCV where conditional compilation is done?

The main issue with the python bindings seems to be related to the following:

Ok, strange behavior but I got it to work. I had to change the reconstruct.hpp file as such:

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArrayOfArrays points3d, InputOutputArray K, bool is_projective = false);

Seems like the object type of the input argument wasn't correct, and hence the python bindings were generating the incorrect object types for the function. I've updated the code on my forked version of the repo. @berak are you a maintainer for the sfm module in the opencv_contrib repo?

@berak
Copy link
Contributor

@berak berak commented Nov 13, 2018

@pierslawrence no, i'm not. (as stated before, already :)

@pierslawrence
Copy link

@pierslawrence pierslawrence commented Nov 13, 2018

@berak I understand that is the case that you are not the maintainer. Sorry about that, I quoted the whole post.

Making the change to OutputArrayOfArrays for Ps and points3d in reconstruct.hpp together with the CV_EXPORTS_W works for me to get the python version correctly working.

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArrayOfArrays points3d,
InputOutputArray K, bool is_projective = false);

anibali added a commit to anibali/opencv_contrib that referenced this issue Apr 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants