Imgproc: templmatch: Add support for mask for all methods #15214

jumostedu · 2019-08-03T20:40:08Z

resolves #6919
resolves #14076

Current situation

There are multiple requests for supporting a mask in the matchTemplate() function, see #14076. Currently only the matching methods CV_TM_SQDIFF and CV_TM_SQDIFF support a mask, and there is justified doubt (#6919) about the correctness.

This pullrequest changes

This pull request adds support for masked template matching for all methods, adds the appropriate documentation, as well as tests (which pass). The mask works in an expectable way (the old implementation is scrubbed): The mask is multiplied to the template and to each image patch that is matched.

# disable_ipp:Mac=ON

force_builders=Linux AVX2,Custom
buildworker:Custom=linux-3
build_image:Custom=ubuntu:18.04
CPU_BASELINE:Custom=AVX512_SKX
disable_ipp=ON

jumostedu · 2019-08-03T23:09:05Z

So I see that some tests fail, suspiciously only on MacOSX, namely exactly all CCOEFF[_NORMED] tests with multiple channels. That seems very strange, and I think it might hint to some lower level issue.

I suspect, that maybe crossCorr() is not behaving identical. From my code the problem could be invoked here. I did not find any real documentation for crossCorr(), but on my build (linux), if the result Mat passed to crossCorr() has multiple channels just like the inputs, it does per channel correlation and saves every channel individually (it sums them up if the result has only one channel). Maybe this behavior of crossCorr() is not used anywhere else and somehow is faulty on MacOSX.

I did not have a deeper look into crossCorr(), I just used it as it is, hoping that it is an efficient low level DFT implementation.

Other than that I have no idea where these test errors on MacOSX come from.

asmorkalov · 2019-11-13T13:24:40Z

@jumostedu Do you have a chance to finish the patch? CI bot reports issues, please take a look.

jumostedu · 2019-11-13T19:26:17Z

@jumostedu Do you have a chance to finish the patch? CI bot reports issues, please take a look.

I have not found time to try to fix the errors on MacOSX so far, but I will try to have a look again the next week(s).

jumostedu · 2019-11-28T23:15:17Z

I can not say, what the problem is, that causes the tests to fail on MacOSX, my code does not contain anything platform specific. My guess would be that it's something inside the crossCorr() function (the one defined in templmatch.cpp), probably the hal::DFT2D. Is crossCorr() tested anywhere for multiple channels? filter2D() uses it, but the tests for it seem to be only randomly done for multiple channels.

I also don't have a Mac, so I can really only debug this by looking at the CI results/output here. This is not a great way to do it, so if someone has a Mac and wants to dive into this, they're welcome.

I've written short tests to narrow down if the problem is really crossCorr() with multiple channels. But I don't know how to restart the builds and tests, it says

Pending — Waiting for builds

Pushing to my branch or does not seem to do it. Neither does adding force_builders_only=Mac,linux to the PR description. On https://pullrequest.opencv.org/#/summary/opencv every build is

not_queued

some days now. @alalek , how can I retrigger the CI tests?

asmorkalov · 2019-11-29T09:14:52Z

@jumostedu I re-started CI for you. CI starts automatically every time when you push changes, so manual triggering usually is not required.

Jordak · 2020-02-10T22:24:35Z

As an interested party, has this PR seen any movement through CI recently?

VadimLevin · 2020-03-27T07:24:21Z

@alalek can you proceed with review? I resolved the problem with the clang build (I reproduced the original problem on Ubuntu 18.04 clang 9.0.0-2 Release) and removed crossCorr from the public interface. I think if it is required it should be done as separate PR.

asmorkalov · 2020-04-08T06:01:04Z

@VadimLevin @alalek are there any open items in the PR? Could we merge it?

jumostedu · 2020-04-08T07:34:27Z

@VadimLevin Thanks for fixing the problem (and prettying up the code :)), I did not realize it was a compiler dependent problem.

If you want, I can remove the test for CrossCorr (df07c97) (as it is removed by @VadimLevin 's commit anyway). Or we could leave it in, if someone ever wants to make a test they could use it as reference (if they find it in the git history).

alalek

Thank you for contribution!

Need to take a look on "mask" design more carefully.
/cc @vpisarev

modules/imgproc/test/test_templmatchmask.cpp

modules/imgproc/include/opencv2/imgproc.hpp

alalek · 2020-04-08T08:59:52Z

modules/imgproc/src/templmatch.cpp

+        // It does not matter what to use Mat/MatExpr, it should be evaluated to perform assign subtraction
+        Mat temp_res = img_mask_corr.mul(sum(templx_mask).div(mask_sum));


I believe there is some bug in MatExpr, need to prepare simple reproducer for tests with similar code.

Should probably be a separate issue. I'm did obviously not resolve this, so you may resolve this conversation yourself :)

VadimLevin · 2020-04-08T15:03:10Z

@jumostedu Are you planning to finalize this PR and pass the review or should I do this on my own?

jumostedu · 2020-04-08T17:21:17Z

@VadimLevin Yeah, I'll do it within the next few (two or three) days. Thanks for your help!

Add support for masked template matching. Fix/scrub old implementation for masked matching, as it did partly not even really do a meaningful masking, and only supported limited template matching methods. Add documentation including formulas for masked matching.

Test accuracy by comparing to naive implementation for one point. Test compatibility/correctness by comparing results without mask and with all ones mask. All tests are done for all methods, all supported depths, and for 1 and 3 channels.

Add a test for the crossCorr function in templmatch.cpp. crossCorr() had to be added to exported functions to be testable. This test can maybe help to identify the problem with template matching on MacOSX.

If it should be exported, it should be done as separate PR.

jumostedu · 2020-04-13T11:26:53Z

I just realized I can then extend the test to test to using 8U masks with 32F image and template, as well as single channel mask with multichannel image and template, so I did that.

asmorkalov · 2020-04-28T07:31:31Z

@jumostedu Do you have a chance to finalize the patch?

jumostedu · 2020-04-28T08:13:19Z

@jumostedu Do you have a chance to finalize the patch?

From my side I think I've resolved all open points. If you want me to rebase something, I can do that. Other than that there's a review pending by @vpisarev

asmorkalov · 2020-04-28T10:38:57Z

@vpisarev could you look at it again?

vpisarev · 2020-05-14T20:08:48Z

@jumostedu, @alalek, @asmorkalov, sorry for delay.
This is excellent job by the contributor and the reviewers, thank you! 👍

anwar6953 · 2020-09-15T08:56:23Z

I am using 4.4.0 and seeing unexpected behavior.
I'm using matchTemplate against the image itself, which should produce 100% match. It does indeed (I'm using cv2.TM_SQDIFF_NORMED).
However, when I pass in a mask that is a subset of the overall image, it then yields a nonzero value for the best match, which is unexpected.

In python:

minMaxLoc = cv2.minMaxLoc(cv2.matchTemplate(img, img, cv2.TM_SQDIFF_NORMED, mask=mask_white(img)))

This yields minMaxLoc[0] to be 0 if mask is None, but yields ~0.00000045 when I generate a mask.
The mask I generate simply sets non-white colors in the image to 0, and all other pixels to (255,255,255).

VadimLevin · 2020-09-15T09:07:59Z

@anwar6953 OpenCV uses (in most cases) single precision in computations involving floating point numbers and TM_SQDIFF_NORMED is one of them. According to the IEEE 754 (floating point standard) single precision float has 7 significant digits, but in my experience you can trust only up to 5-6 digits, so you have approximately zero, which is expected. The source of the difference in the result (0 vs approx 0) may be caused by different data flow in computations.

anwar6953 · 2020-09-15T21:51:08Z

Fair enough that it is approximately zero.
I just thought it was odd that adding a mask resulting in a nonzero, when a mask should only increase the match rate.

Thanks for the explanation.

* imgproc: templmatch: Add support for mask for all methods Add support for masked template matching. Fix/scrub old implementation for masked matching, as it did partly not even really do a meaningful masking, and only supported limited template matching methods. Add documentation including formulas for masked matching. * imgproc: test: Add tests for masked template matching Test accuracy by comparing to naive implementation for one point. Test compatibility/correctness by comparing results without mask and with all ones mask. All tests are done for all methods, all supported depths, and for 1 and 3 channels. * imgproc: test: templmatch: Add test for crossCorr Add a test for the crossCorr function in templmatch.cpp. crossCorr() had to be added to exported functions to be testable. This test can maybe help to identify the problem with template matching on MacOSX. * fix: Fixed wrong evaluations of the MatExpr on Clang * fix: removed crossCorr from public interface. If it should be exported, it should be done as separate PR. Co-authored-by: Vadim Levin <vadim.levin@xperience.ai>

opencv/opencv#15214

…error Implemented in OpenCV 4.4.0: opencv/opencv#15214

opencv/opencv#15214

* imgproc: templmatch: Add support for mask for all methods Add support for masked template matching. Fix/scrub old implementation for masked matching, as it did partly not even really do a meaningful masking, and only supported limited template matching methods. Add documentation including formulas for masked matching. * imgproc: test: Add tests for masked template matching Test accuracy by comparing to naive implementation for one point. Test compatibility/correctness by comparing results without mask and with all ones mask. All tests are done for all methods, all supported depths, and for 1 and 3 channels. * imgproc: test: templmatch: Add test for crossCorr Add a test for the crossCorr function in templmatch.cpp. crossCorr() had to be added to exported functions to be testable. This test can maybe help to identify the problem with template matching on MacOSX. * fix: Fixed wrong evaluations of the MatExpr on Clang * fix: removed crossCorr from public interface. If it should be exported, it should be done as separate PR. Co-authored-by: Vadim Levin <vadim.levin@xperience.ai>

jumostedu mentioned this pull request Aug 3, 2019

Is masked matchTemplate implementation correct? #6919

Closed

catree mentioned this pull request Oct 24, 2019

Strange results of matchTemplate using TM_CCORR_NORMED with mask #15768

Open

jumostedu force-pushed the matchtemplmask branch from 187aded to 5d64332 Compare November 15, 2019 23:42

jumostedu changed the title ~~imgproc: templmatch: Add support for mask for all methods~~ WIP: imgproc: templmatch: Add support for mask for all methods Nov 24, 2019

jumostedu force-pushed the matchtemplmask branch from 5d64332 to df07c97 Compare November 24, 2019 19:27

asmorkalov assigned VadimLevin Mar 13, 2020

VadimLevin force-pushed the matchtemplmask branch from a63ecfe to fa9da50 Compare March 26, 2020 10:18

asmorkalov changed the title ~~WIP: imgproc: templmatch: Add support for mask for all methods~~ Imgproc: templmatch: Add support for mask for all methods Apr 8, 2020

alalek reviewed Apr 8, 2020

View reviewed changes

jumostedu force-pushed the matchtemplmask branch from fa9da50 to dba9437 Compare April 11, 2020 19:00

jumostedu force-pushed the matchtemplmask branch from dba9437 to 342fed9 Compare April 12, 2020 20:18

jumostedu and others added 4 commits April 13, 2020 13:24

imgproc: test: templmatch: Add test for crossCorr

751509d

Add a test for the crossCorr function in templmatch.cpp. crossCorr() had to be added to exported functions to be testable. This test can maybe help to identify the problem with template matching on MacOSX.

fix: Fixed wrong evaluations of the MatExpr on Clang

53bcf22

fix: removed crossCorr from public interface.

43b2b2b

If it should be exported, it should be done as separate PR.

jumostedu force-pushed the matchtemplmask branch from 342fed9 to 43b2b2b Compare April 13, 2020 11:27

vpisarev self-assigned this Apr 28, 2020

vpisarev approved these changes May 14, 2020

View reviewed changes

alalek merged commit 5b095df into opencv:master May 14, 2020

Zebiano mentioned this pull request Mar 30, 2021

matchTemplate with mask error justadudewhohacks/opencv4nodejs#702

Closed

drothlis mentioned this pull request May 23, 2022

Ubuntu 22.04 support stb-tester/stb-tester#743

Merged

drothlis added a commit to stb-tester/stb-tester that referenced this pull request May 23, 2022

stbt.match: OpenCV 4.4.0 allows transparency mask with any match method

21b59cb

opencv/opencv#15214

drothlis added a commit to stb-tester/stb-tester that referenced this pull request May 23, 2022

Skip test_transparent_reference_image_with_sqdiff_normed_raises_value…

739ce30

…error Implemented in OpenCV 4.4.0: opencv/opencv#15214

drothlis added a commit to stb-tester/stb-tester that referenced this pull request May 23, 2022

stbt.match: OpenCV >=4.4.0 no longer normalises SQDIFF with mask

b1054ed

opencv/opencv#15214

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Imgproc: templmatch: Add support for mask for all methods #15214

Imgproc: templmatch: Add support for mask for all methods #15214

jumostedu commented Aug 3, 2019 •

edited by alalek

Loading

jumostedu commented Aug 3, 2019

asmorkalov commented Nov 13, 2019

jumostedu commented Nov 13, 2019

jumostedu commented Nov 28, 2019

asmorkalov commented Nov 29, 2019

Jordak commented Feb 10, 2020

VadimLevin commented Mar 27, 2020

asmorkalov commented Apr 8, 2020

jumostedu commented Apr 8, 2020

alalek left a comment

alalek Apr 8, 2020

jumostedu Apr 12, 2020

VadimLevin commented Apr 8, 2020

jumostedu commented Apr 8, 2020

jumostedu commented Apr 13, 2020

asmorkalov commented Apr 28, 2020

jumostedu commented Apr 28, 2020

asmorkalov commented Apr 28, 2020

vpisarev commented May 14, 2020

anwar6953 commented Sep 15, 2020 •

edited

Loading

VadimLevin commented Sep 15, 2020

anwar6953 commented Sep 15, 2020

		// It does not matter what to use Mat/MatExpr, it should be evaluated to perform assign subtraction
		Mat temp_res = img_mask_corr.mul(sum(templx_mask).div(mask_sum));

Imgproc: templmatch: Add support for mask for all methods #15214

Imgproc: templmatch: Add support for mask for all methods #15214

Conversation

jumostedu commented Aug 3, 2019 • edited by alalek Loading

Current situation

This pullrequest changes

jumostedu commented Aug 3, 2019

asmorkalov commented Nov 13, 2019

jumostedu commented Nov 13, 2019

jumostedu commented Nov 28, 2019

asmorkalov commented Nov 29, 2019

Jordak commented Feb 10, 2020

VadimLevin commented Mar 27, 2020

asmorkalov commented Apr 8, 2020

jumostedu commented Apr 8, 2020

alalek left a comment

Choose a reason for hiding this comment

alalek Apr 8, 2020

Choose a reason for hiding this comment

jumostedu Apr 12, 2020

Choose a reason for hiding this comment

VadimLevin commented Apr 8, 2020

jumostedu commented Apr 8, 2020

jumostedu commented Apr 13, 2020

asmorkalov commented Apr 28, 2020

jumostedu commented Apr 28, 2020

asmorkalov commented Apr 28, 2020

vpisarev commented May 14, 2020

anwar6953 commented Sep 15, 2020 • edited Loading

VadimLevin commented Sep 15, 2020

anwar6953 commented Sep 15, 2020

jumostedu commented Aug 3, 2019 •

edited by alalek

Loading

anwar6953 commented Sep 15, 2020 •

edited

Loading