Initial media support for G-API background subtraction demo #3535

TolyaTalamanov · 2022-08-24T16:31:37Z

Overview

Switched demo from GMat to GFrame for input image. (Background img is still GMat).
Add basic onevpl support. (See --use_onevpl, --vpl_params)

The next steps will be:

Fully support onevpl functionality
Use GFrame for background img.

dmatveev

Is there any performance numbers? Did you test it with GPU?

dmatveev · 2022-08-25T12:17:31Z

demos/background_subtraction_demo/cpp_gapi/main.cpp

+cv::gapi::wip::onevpl::CfgParam createFromString(const std::string &line) {
+    using namespace cv::gapi::wip;
+
+    if (line.empty()) {
+        throw std::runtime_error("Cannot parse CfgParam from emply line");
+    }
+
+    std::string::size_type name_endline_pos = line.find(':');
+    if (name_endline_pos == std::string::npos) {
+        throw std::runtime_error("Cannot parse CfgParam from: " + line +
+                                 "\nExpected separator \":\"");
+    }
+
+    std::string name = line.substr(0, name_endline_pos);
+    std::string value = line.substr(name_endline_pos + 1);
+
+    return cv::gapi::wip::onevpl::CfgParam::create(name, value,
+                                                   /* vpp params strongly optional */
+                                                   name.find("vpp.") == std::string::npos);
+}
+
+static std::vector<cv::gapi::wip::onevpl::CfgParam> parseVPLParams(const std::string& cfg_params) {
+    std::vector<cv::gapi::wip::onevpl::CfgParam> source_cfgs;
+    std::stringstream params_list(cfg_params);
+    std::string line;
+    while (std::getline(params_list, line, ';')) {
+        source_cfgs.push_back(createFromString(line));
+    }
+    return source_cfgs;
+}


I believe user apps shouldn't reinvent this bycicle every time.. We should ship some default implementation for this, please add it to the todo list :)

From my opinion it should be default args-list parsing routine (with find-first-one-separator string ",;\n\r") which provides a std::list<std::string> and then transforming this list into cfgParams by using createFromString in loop on them (or CfgParam::create<string> if works)

Let it be as part of this issue: opencv/opencv#22506

dmatveev · 2022-08-25T12:19:04Z

demos/background_subtraction_demo/cpp_gapi/src/custom_kernels.cpp

@@ -169,7 +169,7 @@ custom::MaskRCNNBGReplacer::MaskRCNNBGReplacer(const std::string& model_path) :
    }
 }

-cv::GMat custom::MaskRCNNBGReplacer::replace(cv::GMat in, const cv::Size& in_size, cv::GMat background) {
+cv::GMat custom::MaskRCNNBGReplacer::replace(cv::GFrame in, cv::GMat bgr, const cv::Size& in_size, cv::GMat background) {


why do you need BGR here as an explicit parameter?

Can BGR() be called right here, in this function?

In the BGR GMat used somewhere else in the pipeline?

(If yes, we need to consider auto-optimizing this)

BGR GMat might be used as a background in case blur option.

Could you elaborate What do you mean by auto-optimizing

TolyaTalamanov · 2022-09-04T17:25:37Z

Is there any performance numbers? Did you test it with GPU?

That's complicated, I've only managed to run vpl with inference on CPU, might be some OpenVINO bug.

TolyaTalamanov · 2022-09-04T17:28:01Z

@eaidova Could you quickly review from OMZ perspective? I guess I forgot update readme.md since I added new option to demo, is anything else missing?

eaidova · 2022-09-05T03:13:33Z

@eaidova Could you quickly review from OMZ perspective? I guess I forgot update readme.md since I added new option to demo, is anything else missing?

@TolyaTalamanov From omz point of view, you need update readme, also it will be nice if you provide test for demo work with new keys.
Does introduction of onevpl requires additional flags for opencv building? If yes, please also review and update https://github.com/opencv/opencv/wiki/BuildOpenCV4OpenVINO

demos/background_subtraction_demo/cpp_gapi/background_subtraction_demo_gapi.hpp

demos/background_subtraction_demo/cpp_gapi/main.cpp

TolyaTalamanov · 2022-09-07T08:27:29Z

@eaidova Could you quickly review from OMZ perspective? I guess I forgot update readme.md since I added new option to demo, is anything else missing?

@TolyaTalamanov From omz point of view, you need update readme, also it will be nice if you provide test for demo work with new keys. Does introduction of onevpl requires additional flags for opencv building? If yes, please also review and update https://github.com/opencv/opencv/wiki/BuildOpenCV4OpenVINO

Yes, it actually requires additional flags for building, but it relates only to G-API module. I'm not sure that I should reflect it in BuildOpenCV4OpenVINO.
Information about build is placed there: https://github.com/opencv/opencv/wiki/Graph-API#building-with-onevpl-toolkit-support. Does it make sense to put this link into demo readme.md at least?

demos/background_subtraction_demo/cpp_gapi/main.cpp

Wovchena · 2022-09-08T12:17:44Z

.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl
fails with

[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ WARN:0@1.573] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\demux\async_mfp_demux_data_provider.cpp (448) cv::gapi::wip::onevpl::MFPAsyncDemuxDataProvider::MFPAsyncDemuxDataProvider [0000026D5915E9A0] couldn't find video stream with supported params, expected codecs: MFX_CODEC_VC1, MFX_CODEC_AV1, MFX_CODEC_VP9, MFX_CODEC_AVC, MFX_CODEC_MPEG2, MFX_CODEC_HEVC, MFX_CODEC_JPEG
[ WARN:0@1.573] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\data_provider_dispatcher.cpp (60) cv::gapi::wip::onevpl::DataProviderDispatcher::create Cannot find suitable data provider
[ ERROR ] Unsupported source or configuration parameters

Running without --use_onevpl works.
2. .\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC"
fails with

[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ ERROR ] OpenCV(4.6.0-dev) C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cpp:232: error: (-215:Assertion failed) false && "GSource mfx_impl_description->ApiVersion.Major >= VPL_NEW_API_MAJOR_VERSION" " - is not implemented" in function 'cv::gapi::wip::onevpl::GSource::Priv::Priv'

It looks, gapi version of demo supports only maskrcnn as --at. It that is correct, could you remove --at completely?

sivanov-work · 2022-09-08T12:37:19Z

Wovchena

.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC"
fails with

about fails:
There are two versions of VPL library implementations: the old one called MFX and the new one: VPL
Both are selected in VPL dispatcher library - that's usually shipped with Windwos & device driver

MFX is bacward compatible with Intel Media ... Library and supports vast devices variety, but VPL implementation supports only new Intel GPU/CPU devices.
So, when VPL Source starts - it tries to use the version that provided by VPL dispatcher and on your test machine it seems VPL implementation for new CPU/GPU devices.
GAPI-VPL was developed and tested for MFX implementation (because no hardware assets were available that days) and it produce that kind of error, which must be warning actually. SO the fix is: using cfg params like COdecID:

mfxImplDescription.ApiVersion.Version:1.12
to force VPL dispatcher use MFX implementation (but this params was not tested yet)

or change VPL Source code to ignore this and reuse MFX forcibly (@TolyaTalamanov we discussed how to change condition in this piece of code)

Wovchena · 2022-09-08T12:49:17Z

I installed oneVPL via installation manager following https://github.com/opencv/opencv/wiki/Graph-API#building-with-onevpl-toolkit-support.

Added mfxImplDescription.ApiVersion.Version:1.12, so the full command is
.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC;mfxImplDescription.ApiVersion.Version:1.12"
it still gives

[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ ERROR ] OpenCV(4.6.0-dev) C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cpp:232: error: (-215:Assertion failed) false && "GSource mfx_impl_description->ApiVersion.Major >= VPL_NEW_API_MAJOR_VERSION" " - is not implemented" in function 'cv::gapi::wip::onevpl::GSource::Priv::Priv'

sivanov-work · 2022-09-08T12:56:53Z

@Wovchena
I seems that vpl implementation version is new (not tested in gapi) and configuration params for specific version doesn't work properly.

So we need 2-rows changes in C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cp which is quick fix
or find a bug in mfxImplDescription.ApiVersion.Version encoding (as far I remember it uses special encoding in version digits representation by VPL) - which is the right way but will take more additional time for fixing that

sivanov-work · 2022-09-08T13:10:47Z

@Wovchena about 1 and 2
In case of 'intel.avi' file, as I mentioned before, we must not use CodecID because mediafile contains containerized (AVI). But it seems, that video format in avi-container is not supported by VPL implementation, because
expected codecs: MFX_CODEC_VC1, MFX_CODEC_AV1, MFX_CODEC_VP9, MFX_CODEC_AVC, MFX_CODEC_MPEG2, MFX_CODEC_HEVC, MFX_CODEC_JPEG as mentioned on oneVPL enumerations documentations.
And demultiplexing stage fails, right before running a stage for loading windows VPL library

The use-case 2 skip this demultiplexing routing and tries to load and apply VPL library but fails because VPL library version is new and non-tested, right before extract frames from video file. The last operation would fail too, because you set CodedID for containerized video which is incorrect here

So, the right facility is using command line from your first version (without specfified CodecID) and make sure that video format inside AVI is supported by VPL library codec.
But it would fail since we had assert on library version check in VPL Source

Little comment: For checking what video format exactly inside AVI file - you can use DEBUG level in opencv library by:
set OPENCV_LOG_LEVEL=Debug and launch example again.
It will print actual video format in container, as far I remember, like it is parsed by Windows.
But according to opencv policy DEBUG level may be used in debug build of library. Otherwise it ignored.

Wovchena · 2022-09-08T13:11:10Z

I removed the assertion from C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cpp. Now I have

.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC;mfxImplDescription.ApiVersion.Version:1.12"
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ WARN:0@1.752] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\engine\decode\decode_engine_legacy.cpp (220) cv::gapi::wip::onevpl::VPLLegacyDecodeEngine::prepare_session_param cannot decode header from provider: 0000029A0F279830. Make sure data source is valid and/or "mfxImplDescription.mfxDecoderDescription.decoder.CodecID" has correct value in case of demultiplexed raw input
[ ERROR ] Error decode header, error: MFX_ERR_MORE_DATA

I guess the codec I selected doesn't match with the provided video. So I tried

.\intel64\Release\background_subtraction_demo_gapi.exe -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\store-aisle-detection.mp4"
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ ERROR ] Cannot determine surface size from frame: FrameRateExtN: 30
FrameRateExtD: 1
AspectRatioW: 1
AspectRatioH: 1
CropX: 0
CropY: 0
CropW: 0
CropH: 0
ChannelId: 0
BitDepthLuma: 0
BitDepthChroma: 0
Shift: 0
FourCC: 0
Width: 0
Height: 0
BufferSize: 0
PicStruct: 0
ChromaFormat: 0

It still doesn't run. How do I fix it?

sivanov-work · 2022-09-08T13:19:33Z

@Wovchena

.\intel64\Release\background_subtraction_demo_gapi.exe -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\store-aisle-detection.mp4"

looks correct.
May I ask you to share code when you removed assert in source?

+I suppose it is used CPU version for decode/encode. We need to force GPU device by set MFX acceleration param = MFX_ACCEL_TYPE_D3D11 (not sure i've spelled it correctly) - to choose default GPU device.
Or pass D3Device pointer when you create VPLSource

Wovchena · 2022-09-08T13:39:33Z

Diff: https://github.com/opencv/opencv/compare/4.x...Wovchena:onevpl-remove-version-assertion?expand=1
It looks, the correct option is --onevpl_params "mfxImplDescription.AccelerationMode:MFX_ACCEL_MODE_VIA_D3D11", but it gives long list of warnings and [ ERROR ] OpenCV(4.6.0-dev) C:\Users\vzlobin\r\opencv\modules\core\src\arithm.cpp:214: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and type), nor 'array op scalar', nor 'scalar op array' in function 'cv::binary_op'. I asked @TolyaTalamanov

Wovchena · 2022-09-08T14:15:51Z

@sivanov-work, do I understand it correctly that the demo is required to be run with "mfxImplDescription.AccelerationMode:MFX_ACCEL_MODE_VIA_D3D11"? If so, @TolyaTalamanov, please hard code adding this value in demo, and mention in flag description, that it is added automatically

sivanov-work · 2022-09-09T06:58:24Z

@Wovchena
As I was told by oneVPL team there was several VPL implementation also, which were selected by VPL dispatcher lib during loading substitution: CPU implementation, GPU implementation and maybe something others. They also told that CPU implementation is deprecated and is not developed and no shipped by default in later build. So, I suppose they may broke this CPU version for the latest hardware (your case) implementation. It's hard to say...
Anyway
{code}
[ ERROR ] Cannot determine surface size from frame: FrameRateExtN: 30
FrameRateExtD: 1
AspectRatioW: 1
AspectRatioH: 1
CropX: 0
CropY: 0
CropW: 0
CropH: 0
ChannelId: 0
BitDepthLuma: 0
BitDepthChroma: 0
Shift: 0
FourCC: 0
Width: 0
Height: 0
BufferSize: 0
PicStruct: 0
ChromaFormat: 0
{code}
is related to MFX API call in result we nothing can do from our side: just ask oneVPL team about that

TolyaTalamanov · 2022-09-12T16:32:21Z

.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl
fails with
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ WARN:0@1.573] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\demux\async_mfp_demux_data_provider.cpp (448) cv::gapi::wip::onevpl::MFPAsyncDemuxDataProvider::MFPAsyncDemuxDataProvider [0000026D5915E9A0] couldn't find video stream with supported params, expected codecs: MFX_CODEC_VC1, MFX_CODEC_AV1, MFX_CODEC_VP9, MFX_CODEC_AVC, MFX_CODEC_MPEG2, MFX_CODEC_HEVC, MFX_CODEC_JPEG
[ WARN:0@1.573] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\data_provider_dispatcher.cpp (60) cv::gapi::wip::onevpl::DataProviderDispatcher::create Cannot find suitable data provider
[ ERROR ] Unsupported source or configuration parameters
Running without --use_onevpl works. 2. .\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC" fails with
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ ERROR ] OpenCV(4.6.0-dev) C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cpp:232: error: (-215:Assertion failed) false && "GSource mfx_impl_description->ApiVersion.Major >= VPL_NEW_API_MAJOR_VERSION" " - is not implemented" in function 'cv::gapi::wip::onevpl::GSource::Priv::Priv'
It looks, gapi version of demo supports only maskrcnn as --at. It that is correct, could you remove --at completely?

Demo supports background matting models, but they aren't part of OMZ because of the license issues.

TolyaTalamanov · 2022-09-12T16:33:22Z

@sivanov-work, do I understand it correctly that the demo is required to be run with "mfxImplDescription.AccelerationMode:MFX_ACCEL_MODE_VIA_D3D11"? If so, @TolyaTalamanov, please hard code adding this value in demo, and mention in flag description, that it is added automatically

This PR is bringing only basic CPU implementation, not sure that D3D11 is really needed to be set as default

TolyaTalamanov · 2022-09-12T16:53:03Z

I removed the assertion from C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cpp. Now I have

.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC;mfxImplDescription.ApiVersion.Version:1.12"
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ WARN:0@1.752] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\engine\decode\decode_engine_legacy.cpp (220) cv::gapi::wip::onevpl::VPLLegacyDecodeEngine::prepare_session_param cannot decode header from provider: 0000029A0F279830. Make sure data source is valid and/or "mfxImplDescription.mfxDecoderDescription.decoder.CodecID" has correct value in case of demultiplexed raw input
[ ERROR ] Error decode header, error: MFX_ERR_MORE_DATA

I guess the codec I selected doesn't match with the provided video. So I tried

.\intel64\Release\background_subtraction_demo_gapi.exe -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\store-aisle-detection.mp4"
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ ERROR ] Cannot determine surface size from frame: FrameRateExtN: 30
FrameRateExtD: 1
AspectRatioW: 1
AspectRatioH: 1
CropX: 0
CropY: 0
CropW: 0
CropH: 0
ChannelId: 0
BitDepthLuma: 0
BitDepthChroma: 0
Shift: 0
FourCC: 0
Width: 0
Height: 0
BufferSize: 0
PicStruct: 0
ChromaFormat: 0

It still doesn't run. How do I fix it?

I think you shouldn't have just removed assertion because execution should fall to else branch.

Wovchena · 2022-09-12T17:27:51Z

Please add a note somewhere about background matting models that aren't in OMZ so I wouldn't forget next time

TolyaTalamanov · 2022-09-13T06:35:58Z

I removed the assertion from C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\source_priv.cpp. Now I have

.\intel64\Release\background_subtraction_demo_gapi.exe -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\intel1.avi" -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl --onevpl_params "mfxImplDescription.mfxDecoderDescription.decoder.CodecID:MFX_CODEC_HEVC;mfxImplDescription.ApiVersion.Version:1.12"
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ WARN:0@1.752] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\engine\decode\decode_engine_legacy.cpp (220) cv::gapi::wip::onevpl::VPLLegacyDecodeEngine::prepare_session_param cannot decode header from provider: 0000029A0F279830. Make sure data source is valid and/or "mfxImplDescription.mfxDecoderDescription.decoder.CodecID" has correct value in case of demultiplexed raw input
[ ERROR ] Error decode header, error: MFX_ERR_MORE_DATA

I guess the codec I selected doesn't match with the provided video. So I tried

.\intel64\Release\background_subtraction_demo_gapi.exe -m C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml -at maskrcnn --use_onevpl -i "C:\Users\vzlobin\OneDrive - Intel Corporation\a\videos\store-aisle-detection.mp4"
[ INFO ] OpenVINO
[ INFO ]        version: 2022.1.0
[ INFO ]        build: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ INFO ] The background matting model C:\Users\vzlobin\Downloads\d\intel\instance-segmentation-person-0007\FP16\instance-segmentation-person-0007.xml is loaded to CPU device.
[ ERROR ] Cannot determine surface size from frame: FrameRateExtN: 30
FrameRateExtD: 1
AspectRatioW: 1
AspectRatioH: 1
CropX: 0
CropY: 0
CropW: 0
CropH: 0
ChannelId: 0
BitDepthLuma: 0
BitDepthChroma: 0
Shift: 0
FourCC: 0
Width: 0
Height: 0
BufferSize: 0
PicStruct: 0
ChromaFormat: 0

It still doesn't run. How do I fix it?

opencv/opencv#22507 - this makes it work without providing codec.

TolyaTalamanov · 2022-09-13T16:27:57Z

Diff: https://github.com/opencv/opencv/compare/4.x...Wovchena:onevpl-remove-version-assertion?expand=1 It looks, the correct option is --onevpl_params "mfxImplDescription.AccelerationMode:MFX_ACCEL_MODE_VIA_D3D11", but it gives long list of warnings and [ ERROR ] OpenCV(4.6.0-dev) C:\Users\vzlobin\r\opencv\modules\core\src\arithm.cpp:214: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and type), nor 'array op scalar', nor 'scalar op array' in function 'cv::binary_op'. I asked @TolyaTalamanov

Must be fixed with the latest commit

TolyaTalamanov · 2022-09-13T16:28:27Z

Please add a note somewhere about background matting models that aren't in OMZ so I wouldn't forget next time

Done

demos/background_subtraction_demo/cpp_gapi/main.cpp

TolyaTalamanov · 2022-09-14T06:35:07Z

@dmatveev @Wovchena Folks, could you have a look one more time, please?

Wovchena

Please, do something about warning spam like [ WARN:2@3.889] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\accelerators\surface\surface_pool.cpp (66) cv::gapi::wip::onevpl::CachedPool::find_free cannot get free surface from pool, size: 21. That doesn't seem right that correctly configured demo produces that.

demos/background_subtraction_demo/cpp_gapi/README.md

Wovchena · 2022-09-14T12:47:59Z

demos/background_subtraction_demo/cpp_gapi/README.md

@@ -83,7 +102,7 @@ Options:
    -target_bgr                Optional. Background onto which to composite the output (by default to green field).
    -u                         Optional. List of monitors to show initially.
    -use_onevpl                Optional. Use onevpl video decoding.
-    -onevpl_params             Optional. Parameters for onevpl video decoding. Format: <prop name>:<value>;<prop name>:<value> Semicolon separated list of oneVPL mfxVariants which is used for configuring source (see `MFXSetConfigFilterProperty` by https://spec.oneapi.io/versions/latest/elements/oneVPL/source/index.html)
+    -onevpl_params             Optional. Parameters for onevpl video decoding. OneVPL source can be fine-grained by providing configuration parameters. Format: <prop name>:<value>;<prop name>:<value> Several important configuration parameters: -mfxImplDescription.mfxDecoderDescription.decoder.CodecID values: https://spec.oneapi.io/onevpl/2.7.0/API_ref/VPL_enums.html?highlight=mfx_codec_hevc#codecformatfourcc-mfxImplDescription.AccelerationMode values: https://spec.oneapi.io/onevpl/2.7.0/API_ref/VPL_disp_api_enum.html?highlight=d3d11#mfxaccelerationmode(see `MFXSetConfigFilterProperty` by https://spec.oneapi.io/versions/latest/elements/oneVPL/source/index.html)


What is OneVPL source can be fine-grained by providing configuration parameters. sentence for? To me it's the same as Parameters for onevpl video decoding.

Committed your suggestion

demos/background_subtraction_demo/cpp_gapi/README.md

demos/background_subtraction_demo/cpp_gapi/main.cpp

TolyaTalamanov · 2022-10-01T14:38:33Z

Please, do something about warning spam like [ WARN:2@3.889] global C:\Users\vzlobin\r\opencv\modules\gapi\src\streaming\onevpl\accelerators\surface\surface_pool.cpp (66) cv::gapi::wip::onevpl::CachedPool::find_free cannot get free surface from pool, size: 21. That doesn't seem right that correctly configured demo produces that.

Yeaaa, I think the only fix for that is configure OPENCV_LOG_LEVEL=Error @SergeyIvanov87

@Wovchena This warning might be fixed by increasing onevpl frame pool size. I've added new flag -onevpl_pool_size and reflected about this in README.md.

It's actually produce another one:

[ WARN:2@2.472] global C:\workspace\development\opencv\modules\gapi\src\streaming\onevpl\engine\decode\decode_engine_legacy.cpp (125) cv::gapi::wip::onevpl::VPLLegacyDecodeEngine::{ctor}::<lambda_715ce2d6fa20067ff98a837885731f85>::operator () decode pending ops count: 1, sync id: 0000000000000000, status: MFX_WRN_VIDEO_PARAM_CHANGED

But it doesn't spam console so much. I guess this is also fixable by providing some configuration option.

@sivanov-work Could you advise us please?

* Initial onevpl suppport * Switch to GFrame

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

TolyaTalamanov · 2022-10-01T14:57:40Z

@Wovchena Could you go through it one more time, please? I guess the majority of issues have been addressed.
Can we merge it?

demos/background_subtraction_demo/cpp_gapi/background_subtraction_demo_gapi.hpp

demos/background_subtraction_demo/cpp_gapi/README.md

demos/background_subtraction_demo/cpp_gapi/background_subtraction_demo_gapi.hpp

demos/background_subtraction_demo/cpp_gapi/README.md

…ion_demo_gapi.hpp Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

…ion_demo_gapi.hpp Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

TolyaTalamanov · 2022-10-04T07:52:56Z

🍾

dmatveev approved these changes Aug 25, 2022

View reviewed changes

eaidova mentioned this pull request Aug 30, 2022

Using oneVPL and QSV/Intel GPU decoder in images_capture #3538

Open

eaidova requested a review from Wovchena September 5, 2022 03:19

Wovchena requested changes Sep 5, 2022

View reviewed changes

demos/background_subtraction_demo/cpp_gapi/background_subtraction_demo_gapi.hpp Show resolved Hide resolved

demos/background_subtraction_demo/cpp_gapi/main.cpp Outdated Show resolved Hide resolved

TolyaTalamanov commented Sep 8, 2022

View reviewed changes

demos/background_subtraction_demo/cpp_gapi/main.cpp Outdated Show resolved Hide resolved

TolyaTalamanov mentioned this pull request Sep 13, 2022

G-API: Replace MFX major version assertion to warning opencv/opencv#22507

Merged

6 tasks

TolyaTalamanov commented Sep 14, 2022

View reviewed changes

demos/background_subtraction_demo/cpp_gapi/main.cpp Show resolved Hide resolved

Wovchena requested changes Sep 14, 2022

View reviewed changes

TolyaTalamanov and others added 14 commits October 1, 2022 15:53

Switch demo to GFrame input

ab0b4c4

Refactor MediaCommonCapSrc

0d65526

Initial media support for G-API bgr subtraction

c4829cd

* Initial onevpl suppport * Switch to GFrame

Add flags to readme and help print

31e592b

Fix background_subtraction_demo_gapi documentation

c264197

Round up frame_size by 16 for use_onevpl case.

b4a7379

Update readme

c7b8ae4

Replace "note:" in readme

ede3d29

Update demos/background_subtraction_demo/cpp_gapi/main.cpp

ca3eab4

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Fix warning and add onevpl_pool_size flag

c779aac

Update demos/background_subtraction_demo/cpp_gapi/README.md

236835c

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Update demos/background_subtraction_demo/cpp_gapi/README.md

2ce7886

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Update demos/background_subtraction_demo/cpp_gapi/main.cpp

7165e90

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Fix comments to review

1284917

TolyaTalamanov force-pushed the at/add-onevpl-support-background-subtraction-gapi branch from 3dbee89 to 1284917 Compare October 1, 2022 14:53

Wovchena requested changes Oct 3, 2022

View reviewed changes

TolyaTalamanov and others added 9 commits October 3, 2022 10:45

Update demos/background_subtraction_demo/cpp_gapi/background_subtract…

2bca2c3

…ion_demo_gapi.hpp Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Update demos/background_subtraction_demo/cpp_gapi/README.md

d1554f9

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Update demos/background_subtraction_demo/cpp_gapi/background_subtract…

d85470d

…ion_demo_gapi.hpp Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Update demos/background_subtraction_demo/cpp_gapi/README.md

1e71604

Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>

Update background_subtraction_demo_gapi.hpp

2c2db8a

Update background_subtraction_demo_gapi.hpp

23801ac

Update background_subtraction_demo_gapi.hpp

526756b

Update README.md

91fdcf2

Fix flag message

0ff90e0

Wovchena approved these changes Oct 4, 2022

View reviewed changes

Wovchena merged commit e3f46cc into openvinotoolkit:master Oct 4, 2022

Initial media support for G-API background subtraction demo #3535

Initial media support for G-API background subtraction demo #3535

Conversation

TolyaTalamanov commented Aug 24, 2022 • edited

Overview

dmatveev left a comment

Choose a reason for hiding this comment

dmatveev Aug 25, 2022

Choose a reason for hiding this comment

sivanov-work Sep 8, 2022 • edited

Choose a reason for hiding this comment

TolyaTalamanov Sep 12, 2022

Choose a reason for hiding this comment

dmatveev Aug 25, 2022

Choose a reason for hiding this comment

TolyaTalamanov Aug 25, 2022

Choose a reason for hiding this comment

TolyaTalamanov Sep 4, 2022

Choose a reason for hiding this comment

TolyaTalamanov commented Sep 4, 2022 • edited

TolyaTalamanov commented Sep 4, 2022

eaidova commented Sep 5, 2022

TolyaTalamanov commented Sep 7, 2022

Wovchena commented Sep 8, 2022

sivanov-work commented Sep 8, 2022

Wovchena commented Sep 8, 2022

sivanov-work commented Sep 8, 2022

sivanov-work commented Sep 8, 2022

Wovchena commented Sep 8, 2022

sivanov-work commented Sep 8, 2022

Wovchena commented Sep 8, 2022

Wovchena commented Sep 8, 2022

sivanov-work commented Sep 9, 2022 • edited

TolyaTalamanov commented Sep 12, 2022

TolyaTalamanov commented Sep 12, 2022

TolyaTalamanov commented Sep 12, 2022

Wovchena commented Sep 12, 2022

TolyaTalamanov commented Sep 13, 2022

TolyaTalamanov commented Sep 13, 2022

TolyaTalamanov commented Sep 13, 2022

TolyaTalamanov commented Sep 14, 2022

Wovchena left a comment

Choose a reason for hiding this comment

Wovchena Sep 14, 2022

Choose a reason for hiding this comment

TolyaTalamanov Oct 1, 2022 • edited

Choose a reason for hiding this comment

TolyaTalamanov commented Oct 1, 2022 • edited

TolyaTalamanov commented Oct 1, 2022

TolyaTalamanov commented Oct 4, 2022

TolyaTalamanov commented Aug 24, 2022 •

edited

sivanov-work Sep 8, 2022 •

edited

TolyaTalamanov commented Sep 4, 2022 •

edited

sivanov-work commented Sep 9, 2022 •

edited

TolyaTalamanov Oct 1, 2022 •

edited

TolyaTalamanov commented Oct 1, 2022 •

edited