Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python: support tuple src for cv::add()/subtract()/... #24074

Merged
merged 12 commits into from Sep 19, 2023

Conversation

Kumataro
Copy link
Contributor

fix #24057

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • [ x The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

Copy link
Contributor

@VadimLevin VadimLevin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add tests for NumPy array cases mentioned in the issue:

>>> cv2.subtract(mat, np.uint8([10, 10, 10]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\arithm.cpp:652: error: (-215:Assertion failed) type2 == CV_64F && (sz2.height == 1 || sz2.height == 4) in function 'cv::arithm_op'

>>> cv2.subtract(mat, np.array([10, 10, 10]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\arithm.cpp:652: error: (-215:Assertion failed) type2 == CV_64F && (sz2.height == 1 || sz2.height == 4) in function 'cv::arithm_op'

>>> cv2.subtract(mat, np.float64([10, 10, 10]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\arithm.cpp:652: error: (-215:Assertion failed) type2 == CV_64F && (sz2.height == 1 || sz2.height == 4) in function 'cv::arithm_op'

modules/python/src2/cv2.hpp Outdated Show resolved Hide resolved
modules/python/test/test_misc.py Outdated Show resolved Hide resolved
modules/python/test/test_misc.py Outdated Show resolved Hide resolved
@Kumataro
Copy link
Contributor Author

Kumataro commented Aug 8, 2023

Thank you very much for your review.

Please add tests for NumPy array cases mentioned in the issue:

I'm sorry but I was only thinking about tuple support, didn't care about array support.
Compared to PyTuple support, PyArray support requires slightly more complicated modifications.
I need some time. I will continue.

@Kumataro
Copy link
Contributor Author

Kumataro commented Aug 8, 2023

Currently I think new patch handles array for cv::add()/cv::subtract()/... well.

However, there is an additional problem.
The behaviour of dst = cv2.subtract(mat, 5) in python is different from the behaviour of cv2::subtract(mat, 5, dst ) in C++

I'll continue to investigate more(But it seems to be hard). I'm sorry.

memos

#24057 (comment)

Looking more closely at it, subtracting a number with

cv2.subtract(mat, 5)

only subtracts from the first channel of each pixel too.

I tired with C++.

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>

int main()
{
  cv::Mat src = (cv::Mat_<uchar>(3, 9) <<
      128, 192, 64, 128, 192, 64, 128, 192, 64,
      128, 192, 64, 128, 192, 64, 128, 192, 64,
      128, 192, 64, 128, 192, 64, 128, 192, 64 );
  src = src.reshape(3);
  std::cout << "<< src >>"
            << " cols=" << src.cols
            << " rows=" << src.rows << std::endl;
  std::cout << src << std::endl;

  cv::Mat dst;
  cv::subtract(src, 5, dst );
  std::cout << "<< cv::subtract(src, 5, dst ); >>" << std::endl;
  std::cout << "<< dst >>"
            << " cols=" << dst.cols
            << " rows=" << dst.rows << std::endl;
  std::cout << dst << std::endl;

  cv::subtract(src, cv::Scalar(5), dst );
  std::cout << "<< cv::subtract(src, cv::Scalar(5), dst ); >> " << std::endl;
  std::cout << "<< dst >>"
            << " cols=" << dst.cols
            << " rows=" << dst.rows << std::endl;
  std::cout << dst << std::endl;

  cv::subtract(src, cv::Scalar(5,0,0,0), dst );
  std::cout << "<< cv::subtract(src, cv::Scalar(5,0,0,0), dst ); >> " << std::endl;
  std::cout << "<< dst >>"
            << " cols=" << dst.cols
            << " rows=" << dst.rows << std::endl;
  std::cout << dst << std::endl;

  return 0;
}

Result is

<< src >> cols=3 rows=3
[128, 192,  64, 128, 192,  64, 128, 192,  64;
 128, 192,  64, 128, 192,  64, 128, 192,  64;
 128, 192,  64, 128, 192,  64, 128, 192,  64]
<< cv::subtract(src, 5, dst ); >>
<< dst >> cols=3 rows=3
[123, 187,  59, 123, 187,  59, 123, 187,  59;
 123, 187,  59, 123, 187,  59, 123, 187,  59;
 123, 187,  59, 123, 187,  59, 123, 187,  59]
<< cv::subtract(src, cv::Scalar(5), dst ); >>
<< dst >> cols=3 rows=3
[123, 192,  64, 123, 192,  64, 123, 192,  64;
 123, 192,  64, 123, 192,  64, 123, 192,  64;
 123, 192,  64, 123, 192,  64, 123, 192,  64]
<< cv::subtract(src, cv::Scalar(5,0,0,0), dst ); >>
<< dst >> cols=3 rows=3
[123, 192,  64, 123, 192,  64, 123, 192,  64;
 123, 192,  64, 123, 192,  64, 123, 192,  64;
 123, 192,  64, 123, 192,  64, 123, 192,  64]

python result

diff --git a/modules/python/test/test_misc.py b/modules/python/test/test_misc.py
index 3d1c5c4b94..ac2127b9ed 100644
--- a/modules/python/test/test_misc.py
+++ b/modules/python/test/test_misc.py
@@ -882,6 +882,10 @@ class Arguments(NewOpenCVTests):

         # Special case, vector length is not same as color channel of src mat
         ## bgr image
+        dst = cv.subtract(src1_mat, 32)
+        np.testing.assert_equal(dst[0][0], [32,32,32], "subtract(mat, vec1) [0][0]")
+        np.testing.assert_equal(dst[4][4], [32,32,32], "subtract(mat, vec1) [4][4]")
+
         dst = cv.subtract(src1_mat, (32))
         np.testing.assert_equal(dst[0][0], [32,64,64], "subtract(mat, vec1) [0][0]")
         np.testing.assert_equal(dst[4][4], [32,64,64], "subtract(mat, vec1) [4][4]")
======================================================================
FAIL: test_arithm_op_with_scalar_issue_24057 (__main__.Arguments.test_arithm_op_with_scalar_issue_24057)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtr/work/opencv/modules/python/test/test_misc.py", line 886, in test_arithm_op_with_scalar_issue_24057
    np.testing.assert_equal(dst[0][0], [32,32,32], "subtract(mat, vec1) [0][0]")
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 349, in assert_equal
    return assert_array_equal(actual, desired, err_msg, verbose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 985, in assert_array_equal
    assert_array_compare(operator.__eq__, x, y, err_msg=err_msg,
  File "/usr/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal
subtract(mat, vec1) [0][0]
Mismatched elements: 2 / 3 (66.7%)
Max absolute difference: 32
Max relative difference: 1.
 x: array([32, 64, 64], dtype=uint8)
 y: array([32, 32, 32])

@asmorkalov asmorkalov added this to the 4.9.0 milestone Aug 9, 2023
@Kumataro
Copy link
Contributor Author

Kumataro commented Aug 9, 2023

The behaviour of dst = cv2.subtract(mat, 5) in python is different from the behaviour of cv2::subtract(mat, 5, dst ) in C++

I think this problem is resolved, and they are same behavirour now.

modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
modules/python/src2/cv2_convert.cpp Outdated Show resolved Hide resolved
@@ -595,6 +602,8 @@ def parse_func_decl(self, decl_str, mat="Mat", docstring=""):

if arg_type == "InputArray":
arg_type = mat
if is_arithm_op_func and arg_name in { "src1", "src2" }:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check arg_name in { "src1", "src2" } is really necessary?
Name of the argument shouldn't impact its properties.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I belive it is needed. Because these functions have not only src1 src2 but also mask as InputArray.
I think that the mask image should be removed from the correction target.

https://docs.opencv.org/4.x/d2/de8/group__core__array.html#ga10ac1bfb180e2cfda1701d06c24fdbd6

void cv::add	(
   InputArray 	src1,
   InputArray 	src2,
   OutputArray 	dst,
   InputArray 	mask = noArray(),
    int 	dtype = -1 
)		
Python:
cv.add(	src1, src2[, dst[, mask[, dtype]]]	) ->	dst

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true, but the solution looks very fragile at this point, because it relies on things that cannot be checked rather during runtime nor generation time...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your review, I don't think there's much of some problem without checking for arg_names.
Due to future implications, I will remove the verification code.

modules/python/test/test_misc.py Outdated Show resolved Hide resolved
@VadimLevin
Copy link
Contributor

It looks like NumPy version in CI is lower than version when np.random.default_rng is introduced.
Please make corresponding updates to use old np.random functions for lower NumPy versions.

@Kumataro
Copy link
Contributor Author

Kumataro commented Aug 25, 2023

For the same test code, the test results are inconsistent between Python2 (NG) and Python3 (OK).
In this weekend, I'll build an execution environment for Python2 and test it. thank you very much.


memo:
It seems that the behaviours of "int / int" is changed between Python2 and Python3. The result of Python2 is integer. The result of Python3 is float. When using "//" operator, which is integer divide operator, the same result as python 2 is given with python 3.

However, on linux this test didn't work well with python2 and python3 too. I'll investigate tommorow.

@Kumataro
Copy link
Contributor Author

I hope it works well.

Test failed on Linux64

Currently, precommit test server for linux64 is running with Ubuntu 16.04

uname: Linux x86_64 x86_64 x86_64 GNU/Linux
getconf[LONG_BIT]: 64
NAME="Ubuntu"
VERSION="16.04.7 LTS (Xenial Xerus)"

And it uses numpy 1.11.0, it doesn't include following fix. So np.rad() causes fail.

https://docs.scipy.org/doc/numpy-1.13.0/release.html#support-for-returning-arrays-of-arbitrary-dimensions-in-apply-along-axis

I replaced to np.append() and np.fill(). In my ubuntu 16.04 environment, it is OK.

Test failed on python2

The behavious of int / int is changed between python2 and python3.

For example, 5 / 2

  • In python2 it returns 2 as int.
  • In python3 it return 2.5 as float.

I think a very simple workaround would be to multiply by 1.0 and not do integer arithmetic.

@Kumataro
Copy link
Contributor Author

Kumataro commented Aug 28, 2023

It seems that calicuration results are differ between release and debug. Maybe they has different optimization levels. I'll continue to investigate.

@Kumataro
Copy link
Contributor Author

I am very sorry that my investigation took so long.
CI failed is reproduced at raspi4 64bit ubuntu 20.04.

The inconsistency of calculation results in the Neon implementation of cv::divide() needs to be resolved.
(It's an issue that existed before I fixed the python binding).
It is similar(but not same) as #24163 .

For example, src1 = {{25,0,0}, {0,0,0} {0,0,0} }.
Run cv::divide(src1,2,dst).
{{12,0,0,},{0,0,0},{0,0,0}} is the expected value.
{{13,0,0},{0,0,0},{0,0,0}} is the actual value.

In addition, when dividing by 0 in armv6(arm11) /armv7(cortex a17), there is also a problem that the element of dst becomes 255 instead of 0, unlike the description in the specification. If src2 is scalar, 0 padding causes this problem.

After the situation is sorted out, I would like to issue a separate issue or comment on a related issue.

@VadimLevin
Copy link
Contributor

@Kumataro thank you for investigation.
For the current PR scope, let's add a tolerance for division allowing results to be off by 1.

@Kumataro
Copy link
Contributor Author

Kumataro commented Sep 1, 2023

Thank you and I'm sorry for waiting. I pushed it for it.

@Kumataro
Copy link
Contributor Author

Kumataro commented Sep 1, 2023

Hi, CI tells us this PR increases the build time.
Maybe there's something seems to go on that I'm not aware of.

I compare "Build OpenCV Contrib" time for this PR and other PR .
This PR makes x2-x60 slower than other PR.

The correction that comes to mind is that this PR contains changing core.hpp.
So I guess ccache doesn't work well ?

Is there anything I can do ?

This PR) https://github.com/opencv/opencv/actions/runs/6047365813/job/16417304163?pr=24074
Other PR) https://github.com/opencv/opencv/actions/runs/5968360484/job/16192028180?pr=24194

Environment This PR Other PR
Ubuntu2004-ARM64 15m42s 8m41s
Ubuntu2004-x64 17m11s 51s
Ubuntu2204-x64 21m53s 36s
Windows10-x64 33m33s 22m47s
macOS-ARM 34m41s 1m14s
maxOS-x64 Over 60m(Timeout) 1m2s

@asmorkalov
Copy link
Contributor

@VadimLevin @Kumataro I re-triggered the build and everything pass now. What are the next steps here?

@asmorkalov asmorkalov merged commit b870ad4 into opencv:4.x Sep 19, 2023
23 checks passed
@asmorkalov asmorkalov mentioned this pull request Sep 28, 2023
thewoz pushed a commit to thewoz/opencv that referenced this pull request Jan 4, 2024
Python: support tuple src for cv::add()/subtract()/... opencv#24074

fix opencv#24057

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ x The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants