Bounding box fixes for GOTURN tracker #2620

ieliz · 2020-07-30T09:41:03Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under OpenCV (BSD) License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Me and @dkurt trying to fix GOTURN tracker issues:

Memory issue.
The issue with taking ROI out of the image.
Related links:
Memory problem with GOTURN tracker #2589
Goturn tracker fails to allocate 95GB of RAM #2426

modules/tracking/src/gtrTracker.cpp

dkurt · 2020-07-30T09:43:20Z

Hi! Please create a test with 2-3 images where we can catch observed error.

ieliz · 2020-07-30T13:01:30Z

Hi! Is it OK if I will use for reproducing the problem a video from the LaSOT dataset and provide drive.google link to it?

modules/tracking/src/gtrTracker.cpp

dkurt · 2020-07-31T08:11:29Z

You need to select minimal number of images where this error is reproduced. You need to open a separate pull request at https://github.com/opencv/opencv_extra/tree/master/testdata/cv/tracking (BTW, please check if existing videos can trigger it).

mshabunin · 2020-07-31T09:12:59Z

We should reuse existing videos if possible. Other way is to generate sequence of frames in the test.

ieliz · 2020-07-31T09:37:45Z

Sorry, but what do you mean by "generate a sequence of frames"? For reproducing I need a video, where an object is leaving the border of the frame, so it can not be random generated "dummy input".
So, which option do you mean?

Take a video sequence from the camera.
Create my own pictures.
Something else.

UPD: I looked at three existing videos in the repository:
https://github.com/opencv/opencv_extra/tree/master/testdata/cv/tracking
and there are no situations that cover the needed case.

mshabunin · 2020-07-31T10:26:33Z

For example, this generator: https://github.com/opencv/opencv_contrib/blob/master/modules/bgsegm/src/synthetic_seq.cpp, creates frames with waves on background and moving object (just an image) on foreground. It is used in some bgsegm tests.

Another example is here: https://github.com/opencv/opencv/blob/a2f7ef9d210e5be5741c6e3109c233672534a1ed/modules/videoio/test/test_precomp.hpp#L40-L58. It generates a sequence of frames with a moving background, static text and moving object (circle). Used in VideoWriter tests.

You can create similar generator (some static background) and an object (can be simple geometric figure) moving on whatever trajectory you want (leaving the frame).

ieliz · 2020-08-03T07:33:41Z

In the last few days, I tried again to use the videos from https://github.com/opencv/opencv_extra/tree/master/testdata/cv/tracking with 3.4 branch and find out that I was wrong in my last comment - with using tracker.py script, video "david" and 3.4 branch without any patches I successfully reproduced memory issue for GOTURN tracker.
Also, I have a suggestion about this PR - memory problem is related to GOTURN itself, but the ROI problem related to the benchmark. So, I think I should move ROI part of the patch in the tracker benchmark. Of all trackers in opencv-contrib, only the GOTURN tracker has ROI problem during initialization, but this case related to incorrect using of selectROI function (in ground-truth files of the LaSOT dataset exists bounding boxes with out of image coordinates). That fix in the benchmark should prevent the same problem for new trackers in the future.
I am so sorry for the inconvenience. What should I do in this situation?

dkurt · 2020-08-03T11:52:09Z

@ieliz , benchmark is nothing but one of possible tracker applications. Please fix a problem completely in this PR including a test. Thanks!

ieliz · 2020-08-03T12:03:10Z

OK, I will leave all changes in this PR.
But what should I do with the test now?

dkurt · 2020-08-04T10:22:26Z

with using tracker.py script, video "david" and 3.4 branch without any patches I successfully reproduced memory issue for GOTURN tracker.

But what should I do with the test now?

@ieliz, you need to run an algorithm on this video as a test application. Add your code in https://github.com/opencv/opencv_contrib/tree/master/modules/tracking/test.

modules/tracking/test/test_trackers.cpp

modules/tracking/src/gtrTracker.cpp

ieliz · 2020-08-11T14:08:00Z

I think I make this test work, but I have some concerns.
First time I faced the problem with goturn.caffemodel and goturn.prototxt when I tried to start GOTURN tracker from opencv_contrib.
These files are absent in opencv_contrib and opencv_extra, so I was forced to download them from the other source.
Test and tracker would work if only we put these files into the current folder (opencv_contrib/modules/tracking/src or /samples , opencv/build_dir/bin).
My question to @mshabunin: what should I do in a situation like that?

UPD 12.08.20: Also, I think it is worth mentioning - it is necessary to add environment variable OPENCV_TEST_DATA_PATH (path to opencv_extra/testdata) for proper work of the tracker's tests (not only GOTURN test).

3.4

ieliz · 2020-08-14T13:49:04Z

Issue related to absent GOTURN models: #941. No answer about why models are absent in opencv_extra. Someone provides a link to models, but I do not understand why it is not in the master or 3.4 branches.

ieliz · 2020-08-14T15:35:43Z

To be sure, that all changes are working properly I rebuild opencv with opencv-contrib using 3.4 branches, tried to start C++ and Python 3 samples of the GOTURN tracker, start tests with and without fixes.
When we trying to launch tracker without fixes - we catch:

C++ exception with description "OpenCV(3.4.11-dev) /home/user/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 85926161856 bytes in function 'OutOfMemoryError'
" thrown in the test body.

so if statement was redundant. My test is trying to run tracker on video from opencv_extra and if tracker trying to allocate too much memory - test is failed. I think it is enough for this kind of test.

modules/tracking/test/test_trackers.cpp

modules/tracking/src/gtrTracker.cpp

modules/tracking/test/test_trackers.cpp

modules/tracking/include/opencv2/tracking/tracker.hpp

modules/tracking/src/gtrTracker.cpp

modules/tracking/test/test_trackers.cpp

ieliz · 2020-09-04T12:06:45Z

For now, failed builds on win64 and win32:

[----------] 1 test from GOTURN
[ RUN      ] GOTURN.memory_usage
[ INFO:0] VIDEOIO: Enabled backends(5, sorted by priority): FFMPEG(1000); MSMF(990); DSHOW(980); CV_IMAGES(970); CV_MJPEG(960)
unknown file: error: C++ exception with description "OpenCV(3.4.11-dev) C:\build\precommit-contrib_windows64\3.4\opencv\modules\dnn\src\caffe\caffe_io.cpp:1121: error: (-2:Unspecified error) FAILED: fs.is_open(). Can't open "" in function 'cv::dnn::ReadProtoFromTextFile'
" thrown in the test body.
[  FAILED  ] GOTURN.memory_usage (28 ms)
[----------] 1 test from GOTURN (28 ms total)

@mshabunin , @alalek is it possible that these hosts do not have .prototxt and .caffemodel files for GOTURN tracker?
Test successful on Linux host and locally(Ubuntu 18.04).

alalek · 2020-09-04T14:00:25Z

@ieliz .caffemodel file is missing intentionally (due to wrong usage of test framework).
Commit 81b3c20 was green - check the difference.

ieliz · 2020-09-04T14:56:02Z

Commit was green, but the test was unfinished. So, green build for this commit is not significant.
There are many changes since then. Log for Linux is OK:

[----------] 1 test from GOTURN
[ RUN      ] GOTURN.memory_usage
Frame: 0
Predicted ROI: [120.525 x 119.661 from (122.097, 46.9778)]
Frame: 1
Predicted ROI: [164.742 x 161.387 from (88.5267, 18.5295)]
Frame: 2
Predicted ROI: [226.065 x 167.955 from (45.9109, 38.3096)]
Frame: 3
Predicted ROI: [242.584 x 181.255 from (101.457, 59.245)]
Frame: 4
Predicted ROI: [294.815 x 228.594 from (112.759, 77.605)]
Frame: 5
Predicted ROI: [297.004 x 230.113 from (109.816, 64.7538)]
Frame: 6
Predicted ROI: [297.747 x 230.933 from (107.465, 71.9642)]
Frame: 7
Predicted ROI: [296.028 x 229.803 from (110.216, 68.2637)]
Frame: 8
Predicted ROI: [294.399 x 228.464 from (108.576, 72.2389)]
Frame: 9
Predicted ROI: [299.703 x 233.589 from (106.475, 66.5782)]
Frame: 10
Predicted ROI: [310.581 x 242.612 from (99.8672, 64.0757)]
Frame: 11
Predicted ROI: [310.716 x 243.187 from (102.846, 64.6556)]
Frame: 12
Predicted ROI: [321.733 x 252.641 from (91.3217, 59.2588)]
Frame: 13
Predicted ROI: [320.79 x 251.715 from (94.1933, 63.3603)]
Frame: 14
Predicted ROI: [324.607 x 255.338 from (90.0974, 59.1685)]
[       OK ] GOTURN.memory_usage (1944 ms)
[----------] 1 test from GOTURN (1944 ms total)

But Windows hosts can not find the file. Is it possible that files are not there?

alalek

Looks like you have broken serialization process.

Can't open ""

alalek · 2020-09-04T15:05:34Z

modules/tracking/test/test_trackers.cpp

+    "%YAML:1.0\n"
+    "---\n"
+    "prototxt: " + model + "\n"
+    "caffemodel: " + weights + "\n";


Avoid that. We don't testing parameters serialization here.

Use existed straightforward approach:

opencv_contrib/modules/tracking/test/test_trackerParametersIO.cpp

Lines 11 to 19 in e502f30

TrackerMedianFlow::Params parameters;

parameters.maxLevel = 10;

parameters.maxMedianLengthOfDisplacementDifference = 11;

parameters.pointsInGrid = 12;

parameters.winSize = Size(6, 5);

parameters.winSizeNCC = Size(41, 40);

parameters.termCriteria.maxCount = 100;

parameters.termCriteria.epsilon = 0.1;

But I implemented usage of the paths via cv::FileNode. So, this is gonna be no used for the test?

Test must be simple and straightforward.

this is gonna be no used for the test

At least not in this test with the name "memory_usage".

Please, check last changes:
I was forced to remove cv::FileNode, but instead used cv::TrackerGOTURN::Params params;, as @alalek said.
Win32/win64 builds are passed, so I think this test is ready.

dkurt

👍 Thanks!

dkurt reviewed Jul 30, 2020

View reviewed changes

modules/tracking/src/gtrTracker.cpp Outdated Show resolved Hide resolved

dkurt reviewed Jul 31, 2020

View reviewed changes

modules/tracking/src/gtrTracker.cpp Outdated Show resolved Hide resolved

dkurt reviewed Aug 10, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Aug 10, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Aug 10, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Aug 10, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Aug 10, 2020

View reviewed changes

modules/tracking/src/gtrTracker.cpp Outdated Show resolved Hide resolved

ieliz and others added 2 commits August 14, 2020 14:39

Merge pull request #2 from opencv/3.4

735511b

3.4

Trying to change branch from master to 3.4

63c3cdf

ieliz changed the base branch from master to 3.4 August 14, 2020 12:48

ieliz changed the base branch from 3.4 to master August 14, 2020 12:50

ieliz force-pushed the goturn branch from 5bc659b to 63c3cdf Compare August 14, 2020 13:41

ieliz changed the base branch from master to 3.4 August 14, 2020 13:41

Removing unnecessary if statement

5ab85c1

ieliz mentioned this pull request Aug 14, 2020

LaSOT-based benchmark for trackers #2516

Open

dkurt reviewed Aug 15, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Aug 15, 2020

View reviewed changes

modules/tracking/src/gtrTracker.cpp Outdated Show resolved Hide resolved

Replacing get_data_path with findDataFile and moving paths in parameters

229653a

dkurt reviewed Aug 20, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

tracking: fix GOTURN model loading

81b3c20

dkurt reviewed Aug 26, 2020

View reviewed changes

modules/tracking/include/opencv2/tracking/tracker.hpp Outdated Show resolved Hide resolved

ieliz added 3 commits September 3, 2020 13:12

Fixing GOTURN memory test: paths to GOTURN files are in cv::FileNode now

8146256

Fix

115a52e

Fix

3707382

dkurt reviewed Sep 3, 2020

View reviewed changes

modules/tracking/include/opencv2/tracking/tracker.hpp Outdated Show resolved Hide resolved

dkurt reviewed Sep 3, 2020

View reviewed changes

modules/tracking/src/gtrTracker.cpp Outdated Show resolved Hide resolved

dkurt reviewed Sep 3, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Sep 3, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Show resolved Hide resolved

Fixing comments

840992d

dkurt reviewed Sep 4, 2020

View reviewed changes

modules/tracking/test/test_trackers.cpp Show resolved Hide resolved

ieliz added 4 commits September 4, 2020 11:03

Deleting trailing space

b066fd8

Trying to solve the problem with test

ca9e474

Removing ground truth reading part in GOTURN test

7c1148c

Removing trailing spaces

743b5e0

Trying to fix win64/win32 issue

e54e573

alalek reviewed Sep 4, 2020

View reviewed changes

ieliz added 2 commits September 4, 2020 18:24

Fixing problem with win32/win64

6303fb9

Trying to solve issue with win32/win64

447cf39

dkurt approved these changes Sep 4, 2020

View reviewed changes

dkurt linked an issue Sep 4, 2020 that may be closed by this pull request

Memory problem with GOTURN tracker #2589

Closed

4 tasks

dkurt self-assigned this Sep 4, 2020

alalek merged commit d706072 into opencv:3.4 Sep 5, 2020

alalek mentioned this pull request Sep 5, 2020

Merge 3.4 #2658

Merged

ieliz deleted the goturn branch September 7, 2020 06:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bounding box fixes for GOTURN tracker #2620

Bounding box fixes for GOTURN tracker #2620

ieliz commented Jul 30, 2020 •

edited

dkurt commented Jul 30, 2020

ieliz commented Jul 30, 2020

dkurt commented Jul 31, 2020

mshabunin commented Jul 31, 2020

ieliz commented Jul 31, 2020 •

edited

mshabunin commented Jul 31, 2020

ieliz commented Aug 3, 2020

dkurt commented Aug 3, 2020

ieliz commented Aug 3, 2020

dkurt commented Aug 4, 2020

ieliz commented Aug 11, 2020 •

edited

ieliz commented Aug 14, 2020

ieliz commented Aug 14, 2020

ieliz commented Sep 4, 2020

alalek commented Sep 4, 2020

ieliz commented Sep 4, 2020

alalek left a comment

alalek Sep 4, 2020

ieliz Sep 4, 2020

alalek Sep 4, 2020

ieliz Sep 4, 2020

dkurt left a comment

	TrackerMedianFlow::Params parameters;

	parameters.maxLevel = 10;
	parameters.maxMedianLengthOfDisplacementDifference = 11;
	parameters.pointsInGrid = 12;
	parameters.winSize = Size(6, 5);
	parameters.winSizeNCC = Size(41, 40);
	parameters.termCriteria.maxCount = 100;
	parameters.termCriteria.epsilon = 0.1;

Bounding box fixes for GOTURN tracker #2620

Bounding box fixes for GOTURN tracker #2620

Conversation

ieliz commented Jul 30, 2020 • edited

Pull Request Readiness Checklist

dkurt commented Jul 30, 2020

ieliz commented Jul 30, 2020

dkurt commented Jul 31, 2020

mshabunin commented Jul 31, 2020

ieliz commented Jul 31, 2020 • edited

mshabunin commented Jul 31, 2020

ieliz commented Aug 3, 2020

dkurt commented Aug 3, 2020

ieliz commented Aug 3, 2020

dkurt commented Aug 4, 2020

ieliz commented Aug 11, 2020 • edited

ieliz commented Aug 14, 2020

ieliz commented Aug 14, 2020

ieliz commented Sep 4, 2020

alalek commented Sep 4, 2020

ieliz commented Sep 4, 2020

alalek left a comment

Choose a reason for hiding this comment

alalek Sep 4, 2020

Choose a reason for hiding this comment

ieliz Sep 4, 2020

Choose a reason for hiding this comment

alalek Sep 4, 2020

Choose a reason for hiding this comment

ieliz Sep 4, 2020

Choose a reason for hiding this comment

dkurt left a comment

Choose a reason for hiding this comment

ieliz commented Jul 30, 2020 •

edited

ieliz commented Jul 31, 2020 •

edited

ieliz commented Aug 11, 2020 •

edited