attempt to add 0d/1d mat support to OpenCV #23473

vpisarev · 2023-04-08T20:58:06Z

updated
This is another attempt to add 0D/1D Mat support to OpenCV. In the era of deep learning, ChatGPT etc. we want to support ONNX well, so we want Mat/UMat to be able to represent 1D and 0D data w/o 0D/1D=>2D hack.

The previous patch is here:
#18594. Unlike in 18594, here we don't add any special "1d" flag to the header. Instead, for 1D Mat the header looks like:

Mat::dims == 1
Mat::rows == 1
Mat::cols == number_of_elements
Mat::step[0] == Mat::step.buf[1] = Mat::elemSize()
Mat::step.buf[0] == number_of_elements*Mat::step.buf[1]

for 0D Mat (not tested yet) there should be

Mat::dims == 0
Mat::rows == Mat::cols == 1
Mat::step[0] = Mat::step.buf[0] == Mat::step.buf[1] == Mat::elemSize()

In other words, most of the functions that could process 2D Mat's by using their Mat::rows, Mat::cols, data and step's should handle 1D/0D Mat's just fine, but the check dims == 2 (if any) should be changed with dims <= 2. Also, the functions should create output array more carefully, via nD versions of Mat::create().

so far, OpenCV core tests seem to pass.
Let's see whether any other tests fail

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

opencv-alalek · 2023-04-10T08:13:17Z

For 1D case I (and other people) expect m.size(0) == total() instead of m.size(1) == total().

total() by definition is multiply of m.size(i) where i in [0, dims) (for any ND tensor). No need to break that.

Also accessing values >= dims is a mess and should be prohibited to avoid coding mistakes.

No need to design "something" just take a look on numpy.

a = np.array([1,2,3], dtype=int)
print(a)
print(a.shape)     # (3,)
print(a.shape[0])  # 3
print(a.shape[1])  # usage ERROR

Problem is here:

Mat::rows == number_of_elements
Mat::cols == 1

It should be vise versa.

opencv-alalek · 2023-04-10T08:09:54Z

modules/ts/src/ts_func.cpp

@@ -515,7 +515,7 @@ void extract(const Mat& src, Mat& dst, int coi)
 void transpose(const Mat& src, Mat& dst)
 {
    CV_Assert(src.data != dst.data && "Inplace is not support in cvtest::transpose");
-    CV_Assert(src.dims == 2);
+    CV_Assert(src.dims <= 2);


transpose() for 1D/0D should be prohibited to avoid coding mistakes.

User should reshape mat to 2D first in explicit way.

opencv-alalek · 2023-04-10T08:15:11Z

modules/core/test/test_arithm.cpp

-    CV_Assert(src.dims == 2);
-    dst.create(src.size(), src.type());
+    CV_Assert(src.dims <= 2);
+    dst.create(src.dims, src.size.p, src.type());


createSameSize()

opencv-alalek · 2023-04-10T08:17:44Z

modules/core/test/test_mat.cpp

+    int n = 3;
+    Mat_<float> B(1, &n); B << 1, 2, 3;


n

use self-described names.

int shapeB[1] = { 3 };

opencv-alalek · 2023-04-10T08:24:04Z

modules/core/include/opencv2/core/mat.inl.hpp

@@ -726,7 +727,6 @@ const _Tp* Mat::ptr(int y) const
 inline
 uchar* Mat::ptr(int i0, int i1)
 {
-    CV_DbgAssert(dims >= 2);


There is no sense to use both arguments for 1D/0D cases.

vrabaud · 2023-05-10T13:49:36Z

I believe this supersedes #21059
Maybe the tests from that PR can be copied over ?

dmatveev · 2023-06-14T14:05:04Z

👀

… partly supported!

…t work * also fixed a few remaining failures (hopefully) in dnn & core

…ipeline

dmatveev

Looks invasive but will focus more deeply on the g-api changes.

dmatveev · 2023-09-20T09:16:50Z

modules/core/include/opencv2/core/mat.hpp

@@ -2134,6 +2147,7 @@ class CV_EXPORTS Mat
    int dims;
    //! the number of rows and columns or (-1, -1) when the matrix has more than 2 dimensions
    int rows, cols;
+    int dummy = 153;


I will remove it in one of the subsequent commits in a different pull request;
Let it stay here for a few weeks/months to catch the situation when Mat.size.p (when it points to rows or cols) is accessed outside of those 2 fields

opencv-alalek

In comparison with #18594 there are too much changes in tests (equals to broken user code).

opencv-alalek · 2023-09-01T06:48:22Z

modules/3d/misc/java/test/Cv3dTest.java

        Cv3d.undistortPoints(src, dst, cameraMatrix, distCoeffs);

-        assertEquals(src.size(), dst.size());
+        assertEquals(src.cols(), dst.rows());
+        assertEquals(src.rows(), dst.cols());


Why do we want to swap rows/cols?
This doesn't look like a convenient behavior.

opencv-alalek · 2023-09-01T06:51:07Z

modules/3d/perf/perf_pnp.cpp

@@ -40,7 +40,8 @@ PERF_TEST_P(PointsNum_Algo, solvePnP,
    projectPoints(points3d, rvec, tvec, intrinsics, distortion, points2d);

    //add noise
-    Mat noise(1, (int)points2d.size(), CV_32FC2);
+    int sz = (int)points2d.size();
+    Mat noise(1, &sz, CV_32FC2);


Requirement of test modification means that we would broke user code.
We could add new tests easily, but not to modify it.

We need compatibility code to support both inputs (old and new).

opencv-alalek · 2023-09-21T21:08:12Z

modules/core/test/test_mat.cpp

@@ -2432,6 +2439,7 @@ TEST(Mat1D, DISABLED_basic)
        Mat m(5, 100, CV_8UC1, Scalar::all(0));
        const Mat row2D = m.row(2);
        EXPECT_NO_THROW(m1.copyTo(row2D));
+        EXPECT_NO_THROW(row2D.copyTo(m1_copy));


Idea is to write set of simple independent tests with common predefined variables.
This line violates that rule as it modifies common variable (result affects checks below).

opencv-alalek · 2023-09-21T21:12:07Z

modules/core/test/test_mat.cpp

+{
+    std::string s;
+    s = cv::utils::dumpInputArray(noArray());
+    EXPECT_EQ(s, "InputArray: noArray()");


Signature for _EQ is ASSERT_EQ(expected_reference, actual_result);.
Correct order of parameters are required to emit valid error messages.

Reference: https://github.com/opencv/opencv/blob/4.0.0/modules/ts/include/opencv2/ts/ts_gtest.h#L8196-L8200

GTEST_API_ AssertionResult EqFailure(const char* expected_expression, const char* actual_expression, const std::string& expected_value, const std::string& actual_value, bool ignoring_case);

opencv-alalek · 2023-09-21T21:13:48Z

modules/dnn/include/opencv2/dnn/shape_utils.hpp

-    if (shape.empty())
-        return 0;
+    //if (shape.empty())
+    //    return 0;


Why not just return 1 for 0D tensor?

opencv-alalek · 2023-09-21T21:20:06Z

modules/dnn/include/opencv2/dnn/shape_utils.hpp

-    axis = (axis < 0) ? (dims + axis) : axis;
-    CV_DbgCheck(axis, axis >= 0 && axis < dims, "");
+    CV_Assert(dims >= 0);
+    CV_Check(axis, axis >= -dims && axis <= dims, "");


axis <= dims

We should not allow axis == dims.
It is not allowed in classic index/stride rules (e.g. in Python)

>>> a = [1, 2, 3] >>> a[-3] 1 >>> a[-2] 2 >>> a[-1] 3 >>> a[0] 1 >>> a[1] 2 >>> a[2] 3 >>> a[3] Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: list index out of range >>>

opencv-alalek · 2023-09-21T21:22:40Z

modules/dnn/test/test_onnx_importer.cpp

@@ -1899,7 +1905,7 @@ TEST_P(Test_ONNX_layers, Quantized_Convolution)

 TEST_P(Test_ONNX_layers, Quantized_MatMul)
 {
-    testONNXModels("quantized_matmul_uint8_weights", npy, 0.005, 0.007);
+    testONNXModels("quantized_matmul_uint8_weights", npy, 0.008, 0.015);


What is happened?

asmorkalov added the category: core label Apr 10, 2023

asmorkalov assigned opencv-alalek Apr 10, 2023

asmorkalov requested a review from opencv-alalek April 10, 2023 07:02

opencv-alalek reviewed Apr 10, 2023

View reviewed changes

asmorkalov added this to the 5.0 milestone May 5, 2023

vpisarev mentioned this pull request Jun 14, 2023

ONNX: Scalar values initialise with dimensions cause extra false dimensions #23719

Open

4 tasks

vpisarev mentioned this pull request Jun 14, 2023

attempt to add 0d/1d mat support opencv/opencv_contrib#3511

Merged

6 tasks

vpisarev force-pushed the 0d_1d_mat branch from c5665e8 to 424857b Compare July 11, 2023 11:28

vpisarev force-pushed the 0d_1d_mat branch from 424857b to 94203e7 Compare August 8, 2023 19:03

vpisarev added 17 commits August 31, 2023 11:28

attempt to add 0d/1d mat support to OpenCV

b22efac

revised the patch; now 1D mat is treated as 1xN 2D mat rather than Nx1.

64a5da1

a step towards 'green' tests

60fefa1

another little step towards 'green' tests

a8c98e2

calib test failures seem to be fixed now

d2c8b1e

more fixes _core & _dnn

8aaeb26

another step towards green ci; even 0D mat's (a.k.a. scalars) are now…

d2f1c39

… partly supported!

* fixed strange bug in aruco/charuco detector, not sure why it did no…

901ace8

…t work * also fixed a few remaining failures (hopefully) in dnn & core

disabled failing GAPI tests - too complex to dig into this compiler p…

ced94fb

…ipeline

hopefully fixed java tests

0a1a28c

trying to fix some more tests

d8af48e

quick followup fix

32ed0c1

continue to fix test failures and warnings

92d83a3

quick followup fix

9f45ae9

trying to fix some more tests

1075850

partly fixed support for 0D/scalar UMat's

ab60f13

use updated parseReduce() from upstream

296c284

vpisarev force-pushed the 0d_1d_mat branch from 94203e7 to 296c284 Compare August 31, 2023 08:32

vpisarev added 6 commits September 1, 2023 00:16

trying to fix the remaining test failures

cf6ed96

fixed [ch]aruco tests in Python

53f3706

still trying to fix tests

ab40691

revert "fix" in dnn's CUDA tensor

5032600

trying to fix dnn+CUDA test failures

0e85eee

fixed 1D umat creation

f72c2aa

dmatveev reviewed Sep 20, 2023

View reviewed changes

vpisarev added 2 commits September 21, 2023 10:02

hopefully fixed remaining cuda test failures

06097dc

removed training whitespaces

80c7907

vpisarev merged commit 416bf32 into opencv:5.x Sep 21, 2023
23 checks passed

opencv-alalek reviewed Sep 21, 2023

View reviewed changes

asmorkalov mentioned this pull request Oct 2, 2023

(5.x) Merge 4.x #24338

Merged

opencv-alalek mentioned this pull request Oct 21, 2023

(5.x) Merge 4.x #24416

Merged

opencv-alalek mentioned this pull request Feb 5, 2024

5.x merge 4.x #24958

Closed

opencv-alalek mentioned this pull request Mar 7, 2024

cv::Mat reshape does not support 1d and 0d arrays #25174

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

attempt to add 0d/1d mat support to OpenCV #23473

attempt to add 0d/1d mat support to OpenCV #23473

vpisarev commented Apr 8, 2023 •

edited

opencv-alalek commented Apr 10, 2023 •

edited

opencv-alalek Apr 10, 2023

opencv-alalek Apr 10, 2023

opencv-alalek Apr 10, 2023

opencv-alalek Apr 10, 2023

vrabaud commented May 10, 2023

dmatveev commented Jun 14, 2023

dmatveev left a comment

dmatveev Sep 20, 2023

vpisarev Sep 21, 2023

opencv-alalek left a comment

opencv-alalek Sep 1, 2023

opencv-alalek Sep 1, 2023

opencv-alalek Sep 21, 2023

opencv-alalek Sep 21, 2023

opencv-alalek Sep 21, 2023

opencv-alalek Sep 21, 2023

opencv-alalek Sep 21, 2023

attempt to add 0d/1d mat support to OpenCV #23473

attempt to add 0d/1d mat support to OpenCV #23473

Conversation

vpisarev commented Apr 8, 2023 • edited

Pull Request Readiness Checklist

opencv-alalek commented Apr 10, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vrabaud commented May 10, 2023

dmatveev commented Jun 14, 2023

dmatveev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opencv-alalek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vpisarev commented Apr 8, 2023 •

edited

opencv-alalek commented Apr 10, 2023 •

edited