Add detect qr with aruco #23264

AleksandrPanov · 2023-02-17T01:31:03Z

Using Aruco to detect finder patterns to search QR codes.

TODO (in next PR):

add single QR detect (update detect() and detectAndDecode())
need reduce full enumeration of finder patterns
need add finder pattern info to decode step
need to merge the pipeline of the old and new algorithm

Current results:
+20% total detect, +8% total decode in OpenCV QR benchmark

78.4% detect, 58.7% decode vs 58.5 detect, 50.5% decode in default

main.py.txt

add new info to google docs

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

asmorkalov

Please add:

The table with benchmark results for 4.7 and new version;
Performance test (parameterize existing) and it's result.

modules/objdetect/include/opencv2/objdetect.hpp

modules/objdetect/src/qrcode.cpp

modules/objdetect/include/opencv2/objdetect.hpp

modules/objdetect/src/qrcode.cpp

modules/objdetect/test/test_qrcode.cpp

modules/objdetect/CMakeLists.txt

modules/objdetect/include/opencv2/objdetect.hpp

modules/objdetect/src/qrcode.cpp

modules/objdetect/include/opencv2/objdetect.hpp

modules/objdetect/src/qrcode.cpp

modules/objdetect/include/opencv2/objdetect.hpp

modules/objdetect/test/test_qr_utils.hpp

modules/objdetect/test/test_qrcode.cpp

modules/objdetect/perf/perf_qrcode_pipeline.cpp

modules/objdetect/src/qrcode.cpp

asmorkalov · 2023-04-07T16:58:15Z

/build/precommit_linux64/4.x/opencv/modules/objdetect/perf/perf_qrcode_pipeline.cpp:101:111: warning: passing 'cv::QRDetectMethod' chooses 'int' over 'unsigned int' [-Wsign-promo]
/build/precommit_linux64/4.x/opencv/modules/objdetect/perf/perf_qrcode_pipeline.cpp:101:111: warning:   in call to 'std::__cxx11::string std::__cxx11::to_string(int)' [-Wsign-promo]
/build/precommit_linux64/4.x/opencv/modules/objdetect/test/test_qrcode.cpp:371:111: warning: passing 'cv::QRDetectMethod' chooses 'int' over 'unsigned int' [-Wsign-promo]
/build/precommit_linux64/4.x/opencv/modules/objdetect/test/test_qrcode.cpp:371:111: warning:   in call to 'std::__cxx11::string std::__cxx11::to_string(int)' [-Wsign-promo]

modules/objdetect/include/opencv2/objdetect.hpp

modules/objdetect/perf/perf_qrcode_pipeline.cpp

modules/objdetect/src/qrcode.cpp

asmorkalov · 2023-04-11T07:47:45Z

General notes:

Only ::detectMulti is extended with the new algorithm. ::detect, detectAndDecode, detectAndDecodeCurved are not covered.
It makes sense to decouple the old algorithm and new one as 2 internal classes with own virtual detect, detectMulti, etc. The only check point for algo type is impl pointer initialization.

modules/objdetect/src/qrcode.cpp

modules/objdetect/include/opencv2/objdetect.hpp

asmorkalov · 2023-05-17T14:51:50Z

@opencv-alalek Could you update ABI-complience-checker settings to cover API changes.

modules/objdetect/src/qrcode.cpp

asmorkalov · 2023-05-22T10:39:19Z

@opencv-alalek Friendly reminder.

opencv-alalek · 2023-04-28T08:48:53Z

modules/objdetect/include/opencv2/objdetect.hpp

+       /** @brief Aruco detector parameters are used to search for the finder patterns. */
+        CV_PROP_RW aruco::DetectorParameters arucoParams;


broken indentation

opencv-alalek · 2023-04-28T08:52:14Z

modules/objdetect/test/test_qrcode.cpp


    std::string image_path = findDataFile(root + name_current_image);
    Mat src = imread(image_path);
    ASSERT_FALSE(src.empty()) << "Can't read image: " << image_path;
 #ifdef HAVE_QUIRC
-    QRCodeDetector qrcode;
+    Ptr<QRCodeDetectorBase> qrcode;


PImpl doesn't need Ptr<>. They are already smart pointers.

removed Ptr

opencv-alalek · 2023-05-22T21:02:40Z

samples/cpp/qrcode.cpp

@@ -157,7 +160,7 @@ void drawQRCodeResults(Mat& frame, const vector<Point>& corners, const vector<cv

 static
 void runQR(
-    QRCodeDetector& qrcode, const Mat& input,
+    Ptr<QRCodeDetectorBase> qrcode, const Mat& input,


We don't need Ptr for PImpl which is already smart ptr.

Ptrs must be passed as "const reference" be default.

removed Ptr, added ref

opencv-alalek · 2023-05-22T21:05:27Z

modules/objdetect/include/opencv2/objdetect.hpp

+     @param epsX Epsilon neighborhood, which allows you to determine the horizontal pattern
+     of the scheme 1:1:3:1:1 according to QR code standard.
+    */
+    CV_WRAP void setEpsX(double epsX);


All "set" methods should return *this reference to allow construction chains.

these setters drop CI (Python bindings)
Maybe update the setters later?

In DNN pyopencv_cv_dnn_dnn_ClassificationModel_setEnableSoftmaxPostProcessing has been properly generated.

done.
The directive CV_EXPORTS_W_SIMPLE instead of CV_EXPORTS_W directive helped to add setter

first was added QRCodeDetectorAruco& setDetectorParameters
now were added QRCodeDetector& setEpsX, QRCodeDetector& setEpsY, QRCodeDetector& setUseAlignmentMarkers

now it's done)

modules/objdetect/include/opencv2/objdetect.hpp

opencv-alalek · 2023-05-30T16:50:28Z

modules/objdetect/include/opencv2/objdetect.hpp

@@ -763,29 +764,8 @@ class CV_EXPORTS_W QRCodeEncoder {

 };

-class CV_EXPORTS_W QRCodeDetector
-{
+class CV_EXPORTS_W QRCodeDetectorBase {
 public:


As mentioned on team meetings, we have to follow dnn:Model approach.

Constructors / destructors / assignment / move operators should be defined explicitly.
Also we should not have public default constructor (but we have issue with bindings - deprecation and comment should be on-place).

added with dnn::Model style:

//CV_DEPRECATED_EXTERNAL

Why is it commented?

removed the comments

opencv-alalek · 2023-05-30T16:52:17Z

modules/objdetect/src/qrcode.cpp

+    virtual bool detectAndDecodeMulti(InputArray img, std::vector<std::string>& decoded_info, OutputArray points,
+                              OutputArrayOfArrays straight_qrcode) const = 0;
+
+    virtual ~Impl() {}


It is better to declare virtual destructor first.

opencv-alalek · 2023-05-30T16:53:34Z

modules/objdetect/src/qrcode.cpp

+};
+
+bool QRCodeDetectorBase::detect(InputArray img, OutputArray points) const {
+    return p->detect(img, points);


->

All deference cases must be pre-checked for NULL values.

added CV_Assert(p != nullptr);

Just CV_Assert(p) as we don't really want to retrieve the pointer itself.

It is like container.size() == 0 vs container.empty().

added just CV_Assert(p)

opencv-alalek · 2023-05-30T16:54:57Z

modules/objdetect/src/qrcode.cpp

 };

-QRCodeDetector::QRCodeDetector() : p(new Impl) {}
+QRCodeDetector::QRCodeDetector() {
+    p = new ImplContour();


new

makePtr

opencv-alalek · 2023-05-30T16:57:03Z

modules/objdetect/src/qrcode.cpp

@@ -2832,7 +2881,7 @@ cv::String QRCodeDetector::decodeCurved(InputArray in, InputArray points,
    CV_Assert(src_points.size() == 4);
    CV_CheckGT(contourArea(src_points), 0.0, "Invalid QR code source points");

-    QRDecode qrdec(p->useAlignmentMarkers);
+    QRDecode qrdec((std::static_pointer_cast<ImplContour>)(p)->useAlignmentMarkers);


PImpl pattern classes should not have large method implementations.
All logic must go to Impl.

I propose to make this fix in another PR. This fix will erase the history and increase the diff.

I moved decodeCurved and detectAndDecodeCurved to ImplContour. I think I've fixed everything you asked for. Is there anything else that needs to be fixed?

opencv-alalek · 2023-05-30T17:03:57Z

modules/objdetect/include/opencv2/objdetect.hpp

+     @param epsX Epsilon neighborhood, which allows you to determine the horizontal pattern
+     of the scheme 1:1:3:1:1 according to QR code standard.
+    */
+    CV_WRAP void setEpsX(double epsX);


In DNN pyopencv_cv_dnn_dnn_ClassificationModel_setEnableSoftmaxPostProcessing has been properly generated.

opencv-alalek

LGTM 👍

opencv-alalek · 2023-06-02T04:13:07Z

modules/objdetect/include/opencv2/objdetect.hpp

@@ -763,29 +764,8 @@ class CV_EXPORTS_W QRCodeEncoder {

 };

-class CV_EXPORTS_W QRCodeDetector
-{
+class CV_EXPORTS_W QRCodeDetectorBase {
 public:


//CV_DEPRECATED_EXTERNAL

Why is it commented?

opencv-alalek · 2023-06-02T04:16:02Z

modules/objdetect/src/qrcode.cpp

+};
+
+bool QRCodeDetectorBase::detect(InputArray img, OutputArray points) const {
+    return p->detect(img, points);


Just CV_Assert(p) as we don't really want to retrieve the pointer itself.

It is like container.size() == 0 vs container.empty().

opencv-alalek · 2023-06-02T04:20:24Z

modules/objdetect/src/qrcode.cpp

-    vector<vector<Point2f>> alignmentMarkers;
-    vector<Point2f> updateQrCorners;
+    mutable vector<vector<Point2f>> alignmentMarkers;
+    mutable vector<Point2f> updateQrCorners;


suspicious code, need to be revised later.

opencv-alalek · 2023-06-02T04:24:24Z

modules/objdetect/src/qrcode.cpp


-QRCodeDetector::~QRCodeDetector() {}
+QRCodeDetector& QRCodeDetector::setEpsX(double epsX) {
+    (std::static_pointer_cast<ImplContour>)(p)->epsX = epsX;


no need braces here:

std::static_pointer_cast<ImplContour>(p)

checked dynamic_pointer_cast is more secure.

added dynamic_pointer_cast instead static_pointer_cast

opencv-alalek · 2023-06-02T04:26:19Z

modules/objdetect/src/qrcode.cpp

+    scaleTimingPatternScore = 0.9f;
+}
+
+struct FinderPatternInfo {


BTW, it makes sense to hide local internal classes (especially with some generic names) into inner anonymous namespace.

added FinderPatternInfo and QRCode into anonymous namespace

opencv-alalek · 2023-06-02T04:30:01Z

samples/cpp/qrcode.cpp

@@ -157,7 +160,7 @@ void drawQRCodeResults(Mat& frame, const vector<Point>& corners, const vector<cv

 static
 void runQR(
-    QRCodeDetector& qrcode, const Mat& input,
+    QRCodeDetectorBase& qrcode, const Mat& input,


Use const reference for inputs

added const reference and added const to decode methods

…ace, add const

…h_aruco Add detect qr with aruco opencv#23264 Using Aruco to detect finder patterns to search QR codes. TODO (in next PR): - add single QR detect (update `detect()` and `detectAndDecode()`) - need reduce full enumeration of finder patterns - need add finder pattern info to `decode` step - need to merge the pipeline of the old and new algorithm [Current results:](https://docs.google.com/spreadsheets/d/1ufKyR-Zs-IGXwvqPgftssmTlceVjiQX364sbrjr2QU8/edit#gid=1192415584) +20% total detect, +8% total decode in OpenCV [QR benchmark](https://github.com/opencv/opencv_benchmarks/tree/develop/python_benchmarks/qr_codes) ![res1](https://user-images.githubusercontent.com/22337800/231228556-191d3eae-a318-44e1-af99-e7d420bf6248.png) 78.4% detect, 58.7% decode vs 58.5 detect, 50.5% decode in default [main.py.txt](https://github.com/opencv/opencv/files/10762369/main.py.txt) ![res2](https://user-images.githubusercontent.com/22337800/231229123-ed7f1eda-159a-444b-a3ff-f107d8eb4a20.png) add new info to [google docs](https://docs.google.com/spreadsheets/d/1ufKyR-Zs-IGXwvqPgftssmTlceVjiQX364sbrjr2QU8/edit?usp=sharing) ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

AleksandrPanov added test feature category: objdetect labels Feb 17, 2023

AleksandrPanov self-assigned this Feb 17, 2023

asmorkalov requested changes Feb 17, 2023

View reviewed changes

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch 3 times, most recently from bdbe5de to 339e8a1 Compare March 23, 2023 22:40

asmorkalov reviewed Mar 24, 2023

View reviewed changes

AleksandrPanov marked this pull request as ready for review March 29, 2023 13:12

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch 3 times, most recently from 8602d7e to 04785da Compare April 5, 2023 15:41

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch from fa28b6f to 3e981d0 Compare April 6, 2023 08:02

asmorkalov assigned asmorkalov and unassigned AleksandrPanov Apr 7, 2023

asmorkalov reviewed Apr 7, 2023

View reviewed changes

asmorkalov added this to the 4.8.0 milestone Apr 7, 2023

asmorkalov reviewed Apr 7, 2023

View reviewed changes

asmorkalov reviewed Apr 10, 2023

View reviewed changes

asmorkalov requested changes Apr 11, 2023

View reviewed changes

modules/objdetect/perf/perf_qrcode_pipeline.cpp Outdated Show resolved Hide resolved

modules/objdetect/src/qrcode.cpp Outdated Show resolved Hide resolved

asmorkalov requested changes Apr 12, 2023

View reviewed changes

modules/objdetect/src/qrcode.cpp Outdated Show resolved Hide resolved

asmorkalov reviewed Apr 13, 2023

View reviewed changes

modules/objdetect/include/opencv2/objdetect.hpp Outdated Show resolved Hide resolved

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch 4 times, most recently from cb7a30b to 0d6eddf Compare April 14, 2023 13:10

Code review fixes.

fca3934

asmorkalov force-pushed the add_detect_qr_with_aruco branch from 3d9c6e5 to fca3934 Compare May 17, 2023 14:51

AleksandrPanov commented May 17, 2023

View reviewed changes

modules/objdetect/src/qrcode.cpp Outdated Show resolved Hide resolved

Restroed clone as analyzeFinderPatterns modfies user input.

1ec09d1

opencv-alalek reviewed May 22, 2023

View reviewed changes

use inner class Params, add Params to objc ignore list

e964404

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch from c162f4f to e964404 Compare May 25, 2023 21:48

AleksandrPanov added 2 commits May 26, 2023 01:50

removed ptr, add const ref to getter

e02369b

fix Ptr, fix docs

b9d1942

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch from 4a81fc3 to b9d1942 Compare May 26, 2023 08:37

mshabunin mentioned this pull request May 26, 2023

Moved barcode from opencv_contrib #23666

Merged

6 tasks

Documentation fix

e97cb72

opencv-alalek reviewed May 30, 2023

View reviewed changes

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch 2 times, most recently from e633c4c to f860430 Compare May 30, 2023 21:29

add default construcor, update setter

866289e

AleksandrPanov force-pushed the add_detect_qr_with_aruco branch from f860430 to 866289e Compare May 30, 2023 23:31

AleksandrPanov added 4 commits June 1, 2023 01:04

fixes

8d8edc7

update setters

563c239

move decodeCurved to ImplContour

0c3ec4a

move detectAndDecodeCurved to ImplContour

f2ff342

opencv-alalek approved these changes Jun 2, 2023

View reviewed changes

add CV_Assert(p), add std::dynamic_pointer_cast, add anonymous namesp…

c05217d

…ace, add const

asmorkalov merged commit 9fa014e into opencv:4.x Jun 2, 2023
21 checks passed

opencv-alalek mentioned this pull request Jun 3, 2023

opencv.js: QR code test failure #23737

Closed

asmorkalov mentioned this pull request Jul 12, 2023

(5.x) Merge 4.x #23979

Merged

opencv-alalek mentioned this pull request Feb 19, 2024

fix setArucoParameters in QRCodeDetectorAruco, add const to getArucoParameters #25046

Merged

6 tasks

		/** @brief Aruco detector parameters are used to search for the finder patterns. */
		CV_PROP_RW aruco::DetectorParameters arucoParams;

Add detect qr with aruco #23264

Add detect qr with aruco #23264

Conversation

AleksandrPanov commented Feb 17, 2023 • edited

Pull Request Readiness Checklist

asmorkalov left a comment

Choose a reason for hiding this comment

asmorkalov commented Apr 7, 2023

asmorkalov commented Apr 11, 2023

asmorkalov commented May 17, 2023

asmorkalov commented May 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AleksandrPanov May 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AleksandrPanov Jun 1, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AleksandrPanov May 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AleksandrPanov May 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opencv-alalek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AleksandrPanov commented Feb 17, 2023 •

edited

AleksandrPanov May 26, 2023 •

edited

AleksandrPanov Jun 1, 2023 •

edited

AleksandrPanov May 31, 2023 •

edited

AleksandrPanov May 31, 2023 •

edited