G-API: Introduce cv::MediaFrame, a host type for cv::GFrame #18415

dmatveev · 2020-09-25T03:47:59Z

This PR brings a new data structure cv::MediaFrame, which is a host-type for a graph-based GFrame. The idea of cv::MediaFrame is to handle various media formats (also tied to an external memory) under the same uniform hood.

There's only a cv::MediaFrame class itself + tests in this PR, no integration with the runtime part of G-API done yet (to be pushed and merged separately).

Documentation is by now internal, but will be published as Doxygen with the upcoming PRs.

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom,Custom Win,Custom Mac
build_gapi_standalone:Linux x64=ade-0.1.1f
build_gapi_standalone:Win64=ade-0.1.1f
build_gapi_standalone:Mac=ade-0.1.1f
build_gapi_standalone:Linux x64 Debug=ade-0.1.1f

Xbuild_image:Custom=centos:7
Xbuildworker:Custom=linux-1
build_gapi_standalone:Custom=ade-0.1.1f

build_image:Custom=ubuntu-openvino-2020.4.0:16.04
build_image:Custom Win=openvino-2020.4.0
build_image:Custom Mac=openvino-2020.4.0

test_modules:Custom=gapi
test_modules:Custom Win=gapi
test_modules:Custom Mac=gapi

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

dmatveev · 2020-09-25T03:49:29Z

modules/gapi/src/api/media.cpp

+}
+
+cv::MediaFrame::View::~View() {
+    GAPI_Assert(m_cb != nullptr);


Oh this may be dangerous, as GAPI_Assert throws exceptions and ~View could be called in context of an unwinding stack.

dmatveev · 2020-09-25T08:56:40Z

@mshabunin can the below issue be related to #18397 ?

[----------] 2 tests from TestAgeGenderIE
[ RUN      ] TestAgeGenderIE.InferBasicTensor
unknown file: Failure
C++ exception with description "OpenCV(4.5.0-pre) /build/precommit_custom_mac/opencv/modules/ts/src/ts.cpp:1052: error: (-2:Unspecified error) OpenCV tests: Can't find required data file: intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml in function 'findData'
" thrown in the test body.
[  FAILED  ] TestAgeGenderIE.InferBasicTensor (2 ms)
[ RUN      ] TestAgeGenderIE.InferBasicImage
unknown file: Failure
C++ exception with description "OpenCV(4.5.0-pre) /build/precommit_custom_mac/opencv/modules/ts/src/ts.cpp:1052: error: (-2:Unspecified error) OpenCV tests: Can't find required data file: intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml in function 'findData'
" thrown in the test body.
[  FAILED  ] TestAgeGenderIE.InferBasicImage (0 ms)
[----------] 2 tests from TestAgeGenderIE (2 ms total)

[----------] 2 tests from ROIList
[ RUN      ] ROIList.TestInfer
unknown file: Failure
C++ exception with description "OpenCV(4.5.0-pre) /build/precommit_custom_mac/opencv/modules/ts/src/ts.cpp:1052: error: (-2:Unspecified error) OpenCV tests: Can't find required data file: intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml in function 'findData'
" thrown in SetUp().
[  FAILED  ] ROIList.TestInfer (0 ms)
[ RUN      ] ROIList.TestInfer2
unknown file: Failure
C++ exception with description "OpenCV(4.5.0-pre) /build/precommit_custom_mac/opencv/modules/ts/src/ts.cpp:1052: error: (-2:Unspecified error) OpenCV tests: Can't find required data file: intel/age-gender-recognition-retail-0013/FP32/age-gender-recognition-retail-0013.xml in function 'findData'
" thrown in SetUp().
[  FAILED  ] ROIList.TestInfer2 (1 ms)
[----------] 2 tests from ROIList (1 ms total)

https://pullrequest.opencv.org/buildbot/builders/precommit_custom_mac/builds/1162/steps/test_gapi/logs/stdio

mshabunin · 2020-09-25T09:15:12Z

@dmatveev , yes the location of OMZ have been changed. @alalek , is it possible to change environment variables OPENCV_DNN_TEST_DATA_PATH or OPENCV_OPEN_MODEL_ZOO_DATA_PATH in configurations with OpenVINO? It should be set one level lower than 2020.3.0, so that the directory $OPENCV_OPEN_MODEL_ZOO_DATA_PATH/intel/age-gender-recognition-retail-0013/FP32/ should exist.

alalek · 2020-09-25T12:35:19Z

Fixed (missing test data on build nodes after #18393 )

rgarnov · 2020-09-25T15:21:47Z

modules/gapi/include/opencv2/gapi/media.hpp

+
+class GAPI_EXPORTS MediaFrame::IAdapter {
+public:
+    virtual ~IAdapter() = 0;


This = 0 looks misleading since ~IAdapter is implemented. So it simply can be = default

This is never misleading as it was quite common in pre-Cxx11 era.

My main concern about = default is that when a media.hpp is included in the user application, its destructor will be generated by the user's compiler (different from ours)? I don't trust much to this :)

So my point here is that = 0 usually says that user must implement this method in a derived class, but actually it turns out that there is an implementation in a base class and user can utilize this base class implementation, that's what I meant by misleading. You can ask your compiler to generate a destructor by putting = default in a destructor definition at .cpp file

modules/gapi/include/opencv2/gapi/media.hpp

rgarnov · 2020-09-25T15:36:58Z

modules/gapi/src/api/media.cpp

+    std::unique_ptr<IAdapter> adapter;
+};
+
+cv::MediaFrame::MediaFrame() {


You can simply write = default here

Same concern as above

But that's a .cpp, you can ask your compiler to generate this constructor here by

cv::MediaFrame::MediaFrame() = default;

dmatveev · 2020-09-29T09:41:45Z

@rgarnov @AsyaPronina please have a look again. Need to merge this earlier to get the integration working.

dmatveev · 2020-09-29T11:12:36Z

Windows build failed due to PCH generation problem (out of disk?)

dmatveev · 2020-09-29T16:18:53Z

@alalek can you please proceed with merge?

alalek · 2020-09-29T19:19:14Z

modules/gapi/src/api/media.cpp

+//
+// Copyright (C) 2020 Intel Corporation
+
+#include "precomp.hpp"


relative paths should be used (opencv_world build with PCH is broken - precomp.hpp from another module may be used)

Will fix in a follow-up MR, thanks

Wait.. We've been never using relative paths for precomp.hpp:

grep -Rn "precomp.h" . ./api/ft_render.cpp:7:#include "precomp.hpp" ./api/garray.cpp:8:#include "precomp.hpp" ./api/gcall.cpp:8:#include "precomp.hpp" ./api/gcomputation.cpp:8:#include "precomp.hpp" ./api/gframe.cpp:8:#include "precomp.hpp" ./api/gframe.cpp~:8:#include "precomp.hpp" ./api/ginfer.cpp:8:#include "precomp.hpp" ./api/gkernel.cpp:8:#include "precomp.hpp" ./api/gmat.cpp:8:#include "precomp.hpp" ./api/gnode.cpp:8:#include "precomp.hpp" ./api/gopaque.cpp:8:#include "precomp.hpp" ./api/gorigin.cpp:8:#include "precomp.hpp" ./api/gproto.cpp:8:#include "precomp.hpp" ./api/gscalar.cpp:8:#include "precomp.hpp" ./api/kernels_core.cpp:8:#include "precomp.hpp" ./api/kernels_imgproc.cpp:8:#include "precomp.hpp" ./api/kernels_nnparsers.cpp:8:#include "precomp.hpp" ./api/kernels_video.cpp:8:#include "precomp.hpp" ./api/operators.cpp:8:#include "precomp.hpp" ./api/render.cpp:1:#include "precomp.hpp" ./backends/common/gcompoundbackend.cpp:8:#include "precomp.hpp" ./backends/common/gcompoundkernel.cpp:8:#include "precomp.hpp" ./backends/cpu/gcpubackend.cpp:8:#include "precomp.hpp" ./backends/cpu/gcpucore.cpp:8:#include "precomp.hpp" ./backends/cpu/gcpuimgproc.cpp:8:#include "precomp.hpp" ./backends/cpu/gcpukernel.cpp:8:#include "precomp.hpp" ./backends/cpu/gcpuvideo.cpp:8:#include "precomp.hpp" ./backends/fluid/gfluidbackend.cpp:8:#include "precomp.hpp" ./backends/fluid/gfluidbuffer.cpp:8:#include "precomp.hpp" ./backends/fluid/gfluidcore.cpp:9:#include "precomp.hpp" ./backends/fluid/gfluidimgproc.cpp:9:#include "precomp.hpp" ./backends/ie/giebackend.cpp:7:#include "precomp.hpp" ./backends/ocl/goclbackend.cpp:8:#include "precomp.hpp" ./backends/ocl/goclcore.cpp:8:#include "precomp.hpp" ./backends/ocl/goclimgproc.cpp:8:#include "precomp.hpp" ./backends/openvx/govxbackend.cpp~:9:#include "precomp.hpp" ./backends/plaidml/gplaidmlbackend.cpp:10:#include "precomp.hpp" ./backends/plaidml/gplaidmlcore.cpp:8:#include "precomp.hpp" ./backends/render/grenderocvbackend.cpp:7:#include "precomp.hpp" ./compiler/gcompiled.cpp:8:#include "precomp.hpp" ./compiler/gcompiler.cpp:8:#include "precomp.hpp" ./compiler/gislandmodel.cpp:8:#include "precomp.hpp" ./compiler/gmodel.cpp:8:#include "precomp.hpp" ./compiler/gmodelbuilder.cpp:14:#include "precomp.hpp" ./compiler/gstreaming.cpp:8:#include "precomp.hpp" ./compiler/passes/dump_dot.cpp:8:#include "precomp.hpp" ./compiler/passes/exec.cpp:8:#include "precomp.hpp" ./compiler/passes/helpers.cpp:8:#include "precomp.hpp" ./compiler/passes/intrin.cpp~:8:#include "precomp.hpp" ./compiler/passes/islands.cpp:8:#include "precomp.hpp" ./compiler/passes/kernels.cpp:8:#include "precomp.hpp" ./compiler/passes/meta.cpp:8:#include "precomp.hpp" ./compiler/passes/streaming.cpp:7:#include "precomp.hpp" ./compiler/passes/transformations.cpp:7:#include "precomp.hpp" ./executor/gexecutor.cpp:8:#include "precomp.hpp" ./executor/gstreamingexecutor.cpp:7:#include "precomp.hpp"

Is this something new we should follow now?

dmatveev · 2020-09-30T03:44:46Z

@alalek thanks!

* G-API: Introduce cv::MediaFrame, a host type for cv::GFrame * G-API: RMat -- address review comments

dmatveev added the category: g-api / gapi label Sep 25, 2020

dmatveev assigned AsyaPronina and rgarnov Sep 25, 2020

dmatveev commented Sep 25, 2020

View reviewed changes

dmatveev added this to the 4.5.0 milestone Sep 25, 2020

G-API: Introduce cv::MediaFrame, a host type for cv::GFrame

cd4103d

rgarnov requested changes Sep 25, 2020

View reviewed changes

G-API: RMat -- address review comments

0b22a4c

dmatveev force-pushed the dm/gframe_01_new_host_type branch from e0bda12 to 0b22a4c Compare September 29, 2020 04:22

rgarnov approved these changes Sep 29, 2020

View reviewed changes

alalek reviewed Sep 29, 2020

View reviewed changes

alalek merged commit 43d306f into opencv:master Sep 29, 2020

a-sajjad72 pushed a commit to a-sajjad72/opencv that referenced this pull request Mar 30, 2023

Merge pull request opencv#18415 from dmatveev:dm/gframe_01_new_host_type

e59012a

* G-API: Introduce cv::MediaFrame, a host type for cv::GFrame * G-API: RMat -- address review comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

G-API: Introduce cv::MediaFrame, a host type for cv::GFrame #18415

G-API: Introduce cv::MediaFrame, a host type for cv::GFrame #18415

dmatveev commented Sep 25, 2020

dmatveev Sep 25, 2020

dmatveev commented Sep 25, 2020 •

edited

mshabunin commented Sep 25, 2020

alalek commented Sep 25, 2020

rgarnov Sep 25, 2020

dmatveev Sep 29, 2020

rgarnov Sep 29, 2020

rgarnov Sep 25, 2020

dmatveev Sep 29, 2020

rgarnov Sep 29, 2020 •

edited

dmatveev commented Sep 29, 2020

dmatveev commented Sep 29, 2020

dmatveev commented Sep 29, 2020

alalek Sep 29, 2020 •

edited

dmatveev Sep 30, 2020

dmatveev Sep 30, 2020

dmatveev commented Sep 30, 2020

G-API: Introduce cv::MediaFrame, a host type for cv::GFrame #18415

G-API: Introduce cv::MediaFrame, a host type for cv::GFrame #18415

Conversation

dmatveev commented Sep 25, 2020

Choose a reason for hiding this comment

dmatveev commented Sep 25, 2020 • edited

mshabunin commented Sep 25, 2020

alalek commented Sep 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgarnov Sep 29, 2020 • edited

Choose a reason for hiding this comment

dmatveev commented Sep 29, 2020

dmatveev commented Sep 29, 2020

dmatveev commented Sep 29, 2020

alalek Sep 29, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmatveev commented Sep 30, 2020

dmatveev commented Sep 25, 2020 •

edited

rgarnov Sep 29, 2020 •

edited

alalek Sep 29, 2020 •

edited