Pass through mode for VIDEO->SYSTEM with MFX_FOURCC_A2RGB10 format #1654

sunxinpeng · 2019-09-20T06:46:21Z

When I support the feature "convert P010 to 10bit RGBA" in ffmpeg qsv, I find that if downloading the output surface from driver in MSDK, it will use MSDK VPP path(although I just want to get the surface without any vpp conversion). In this step, MFXVideoVPP_Init will fail because of the format of input context is MFX_FOURCC_A2RGB10(this format is supported for output only). So if MSDK can provide a method which can directly download the a2r10g10b10 surface without passing the vpp path?

dmitryermilov · 2019-09-25T08:02:08Z

It was fixed by #1664.
Could you confirm?

sunxinpeng · 2019-09-25T08:31:04Z

It was fixed by #1664.
Could you confirm?
I have tested with the newest code. The path with IO pattern of "sys_to_sys"(sample_vpp) has passed. But hwdownload(video_to_sys) still failed because A2RGB10 is supported as output only which cause vpp init failed in msdk.

dmitryermilov · 2019-09-25T08:37:24Z

@AntonGrishin , can you please take a look?

AntonGrishin · 2019-09-30T12:21:31Z

Hi @sunxinpeng,
I tested with sample_vpp and all iopattern`s works fine for me:

./sample_vpp -o outa2rgb10.yuv -i stream_352x288_24_p010.yuv  \
-sw 352 -sh 288 -iopattern d3d_to_sys -lib hw -sbitshift 1 -scc p010 \
-dcc a2rgb10 -n 10

AntonGrishin · 2019-10-16T05:45:29Z

Hi @sunxinpeng,
Can you please provide any updates?

sunxinpeng · 2019-10-16T06:17:54Z

Hi @AntonGrishin
Sorry for the delay reply. Maybe you misunderstood my question. When I do vpp in ffmpeg qsv, the hwdownload stage need MSDK download the surface from the driver. I track the code and I find that it use MFXVideoVPP to do this task, which mean that the input format and the output format are both A2RGB10 for MFXVideoVPP_Init at the hwdownload stage. But MSDK limits the input format of
MFXVideoVPP_Init to not be A2RGB10, which will cause failure when hwdownload. So if there is another API path to do this hwdownload of if MSDK can support download A2RGB10 surface using MFXVideoVPP?

AntonGrishin · 2019-10-30T08:27:41Z

Hi @sunxinpeng,

As I understand, ffmpeg called MFXVideoVPP_Init with wrong formats (sample_vpp works fine), so It think this is not MSDK problem, but ffmpeg.

Regarding to API, maybe @dmitryermilov can answer you question.

fulinjie · 2020-06-18T02:52:16Z

@AntonGrishin @dmitryermilov , is it possible to add identical logic like copy pass thru for P010?
#1027
#1268

or allow A2RGB10 as an input for vpp if we only needs it for copy pass through?
or any other suggestions?

(Since X2RGB10 format is accepted in FFmpeg, we'd like to add full support for CSC in QSV now)

dmitryermilov · 2020-06-18T22:07:52Z

Hi @fulinjie ,

I can't restore full context. Could you remind me/explain more? For which exactly combination of input and output FOURCCs/memory types you'd like to have a passthru mode in VPP? AFAIR https://github.com/Intel-Media-SDK/MediaSDK/pull/1268/files introduced passthru mode (simple copy) if input==output. But it seems you're requesting something different?

fulinjie · 2020-06-19T08:01:37Z

Fourcc:
In == Out (MFX_FOURCC_A2RGB10)

IOPattern:
MFX_IOPATTERN_IN_VIDEO_MEMORY | MFX_IOPATTERN_OUT_SYSTEM_MEMORY

FFmpeg currently calls hwdownload to dump the data to system memory, which is a VPP procedure inside MSDK actually.

The story is A2RGB10 is not allowed as an input fourcc for VPP: https://github.com/Intel-Media-SDK/MediaSDK/blob/master/_studio/mfx_lib/vpp/src/mfx_vpp_utils.cpp#L1378
And A2RGB10 is not a supported input fourcc for the ddi caps:
https://github.com/Intel-Media-SDK/MediaSDK/blob/master/_studio/shared/src/mfx_vpp_vaapi.cpp#L472

So my quesion is:
FFmpeg now calls msdk to use VPP procedure to dump A2RGB10 data, is it possible to loose the restricts and allow A2RGB10 as an input while calling MFXVideoVPP_Init()?
since we only need it to be copied pass through.

fulinjie · 2020-06-19T08:09:47Z

For verifying, you may simply try this:
intel-media-ci/ffmpeg#219

dmitryermilov · 2020-06-19T09:08:41Z

I got your point. Thanks for explanation. It's definitely technically feasible. @FurongZhang , please enable it.

fixes: Intel-Media-SDK#1654

AntonGrishin · 2020-06-24T11:57:58Z

Hi @fulinjie,
I verified #2182 with sample_vpp:

./sample_vpp -i a2rgb10.yuv  -sw 352 -sh 288 -iopattern d3d_to_sys -lib hw -scc a2rgb10 -dcc a2rgb10 -n 10 -o cpthru.yuv

Can you please try with ffmpeg?

fulinjie · 2020-06-24T13:05:25Z

Hi @fulinjie,
I verified #2182 with sample_vpp:
./sample_vpp -i a2rgb10.yuv  -sw 352 -sh 288 -iopattern d3d_to_sys -lib hw -scc a2rgb10 -dcc a2rgb10 -n 10 -o cpthru.yuv
Can you please try with ffmpeg?

Hi @AntonGrishin, thanks for the quick response!
Had a quick check, however meet a synchronize error:

VPP PIPELINE:
MFX_EXTBUFF_VPP_RSHIFT_IN
RESIZE
CSC_A2RGB10
MFX_EXTBUFF_VPP_SCALING
VPP PIPELINE:
MFX_EXTBUFF_VPP_RSHIFT_IN
RESIZE
CSC_A2RGB10
MFX_EXTBUFF_VPP_SCALING
VPP PIPELINE:
MFX_EXTBUFF_VPP_RSHIFT_IN
RESIZE
CSC_A2RGB10
MFX_EXTBUFF_VPP_SCALING
VPP PIPELINE:
MFX_EXTBUFF_VPP_RSHIFT_IN
RESIZE
CSC_A2RGB10
MFX_EXTBUFF_VPP_SCALING
[Parsed_scale_qsv_0 @ 0x5627c37e6f40] w:64 h:64 -> w:64 h:64
VPP PIPELINE:
CSC_NV12
RESIZE
VPP PIPELINE:
CSC_NV12
RESIZE
VPP PIPELINE:
CSC_NV12
RESIZE
VPP PIPELINE:
CSC_NV12
RESIZE
[AVHWFramesContext @ 0x5627c37370c0] Error synchronizing the operation: -16
[hwdownload @ 0x5627c37e47c0] Failed to download frame: -1313558101.
Error while filtering: Unknown error occurred
Failed to inject frame into filter network: Unknown error occurred
Error while processing the decoded data for stream #0:0

I'll look into this later in FFmpeg.

fixes: Intel-Media-SDK#1654

fulinjie · 2020-06-29T05:51:11Z

Hi @AntonGrishin,

Confirmed that it seems to be something inside MSDK.
MFX_ERR_LOCK_MEMORY returned while calling mfxDefaultAllocatorVAAPI::SetFrameData() in Sync operation.

We'd better add support for A2RGB10 in SetFrameData() as well, otherwise it'll return MFX_ERR_LOCK_MEMORY by default.
https://github.com/Intel-Media-SDK/MediaSDK/blob/master/_studio/shared/src/libmfx_allocator_vaapi.cpp#L448

You may want to try or debug with ffmpeg just in case there is somewhere else needs the support too.
https://github.com/fulinjie/ffmpeg/tree/pr-qsv_csc_10bit

AntonGrishin · 2020-07-03T11:16:09Z

Hi @fulinjie,
As I understood, we have this support:

MediaSDK/_studio/shared/src/libmfx_allocator_vaapi.cpp

Line 518 in 2ea6923

if (mfx_fourcc == MFX_FOURCC_A2RGB10)

fulinjie · 2020-07-10T02:54:39Z

Hi @fulinjie,
As I understood, we have this support:

MediaSDK/_studio/shared/src/libmfx_allocator_vaapi.cpp

Line 518 in 2ea6923

if (mfx_fourcc == MFX_FOURCC_A2RGB10)

case VA_FOURCC_A2R10G10B10:

Let me be more clear:
FFmpeg uses VA_FOURCC_X2R10G10B10 instead of VA_FOURCC_A2R10G10B10, hence we'd meet this issue.

fulinjie · 2020-07-10T03:23:27Z

@AntonGrishin Another thing is,

Why msdk needs to check each address of pDst->Data.R, pDst->Data.G, pDst->Data.B?
Note that A2RGB10 doesn't have a "byte adddress" for R or G plane, since they are packed 10 bits.

https://github.com/Intel-Media-SDK/MediaSDK/blob/master/_studio/shared/src/libmfx_core.cpp#L1497
case MFX_FOURCC_A2RGB10:
{
mfxU8* ptrSrc = std::min({pSrc->Data.R, pSrc->Data.G, pSrc->Data.B});
mfxU8* ptrDst = std::min({pDst->Data.R, pDst->Data.G, pDst->Data.B});
...
}

IMHO pDst->Data.A2RGB10 would be the exact field applications/users want.

fulinjie · 2020-07-10T03:25:50Z

Verified that after these two concerns fixed(work around for now), the CSC to X2RGB10 works.

fulinjie · 2020-07-10T08:42:28Z

Hi @dmitryermilov , this issue is partially fixed by a786fde.

We still need @AntonGrishin's help for:

Add VA_FOURCC_X2R10G10B10 support in mfxStatus mfxDefaultAllocatorVAAPI::SetFrameData;
Check whether it's possible to modify "ptrDst = std::min({pDst->Data.R, pDst->Data.G, pDst->Data.B});" inside MSDK.

dmitryermilov · 2020-07-10T08:45:23Z

Ohhh. Okay. You was confused my you last comment so merged the change. it was too early.

fulinjie · 2020-07-16T02:52:40Z

Hi @AntonGrishin, any further updates?

AntonGrishin · 2020-07-16T11:48:10Z

Hi @fulinjie,

Basically, I`m not sure, that we can map VA_FOURCC_X2R10G10B10 to MFX_FOURCC_A2RGB10 because they are not fully the same (AFAIK, the first one is without alpha channel). Why FFmpeg use such mapping?
I think that we can modify this condition because R-G-B pointers are definitely the same

fulinjie · 2020-07-20T09:34:39Z

Hi @AntonGrishin ,

X2R10G10B10 could be taken as A2RGB10. This is required by FFmpeg community to avoid the false impression that we has alpha plane data supported in HW for now. The reason for adding this mapping is to address the concern from community,
Yes, please. And I'd prefer to use mfxA2RGB10 structure.

If input use only a2rgb10 pointer and we get min pointer, this leads to nullptr error. Since a2rgb10(x2rgb10) packed 10 bit format, no need to check all pointers before copy, we only need to copy B pointer (that is in union with a2rgb10 pointer). cmd to check: ./ffmpeg -hwaccel qsv -v verbose -c:v hevc_qsv -i \ ../p010.h265 -vf scale_qsv=format=x2rgb10,hwdownload,format=x2rgb10 \ -vframes 1 out.yuv -y fixes: Intel-Media-SDK#1654

AntonGrishin · 2020-08-06T07:25:11Z

Hi @fulinjie,

Sorry, I was on vacation for the last two weeks. Can you please try #2268?
I checked with PR & cmd, and it`s worked for me.

If input use only a2rgb10 pointer and we get min pointer, this leads to nullptr error. Since a2rgb10(x2rgb10) packed 10 bit format, no need to check all pointers before copy, we only need to copy B pointer (that is in union with a2rgb10 pointer). cmd to check: ./ffmpeg -hwaccel qsv -v verbose -c:v hevc_qsv -i \ ../p010.h265 -vf scale_qsv=format=x2rgb10,hwdownload,format=x2rgb10 \ -vframes 1 out.yuv -y fixes: Intel-Media-SDK#1654

fulinjie · 2020-08-06T14:08:41Z

Hi @AntonGrishin, commented in the pr, thx.

FFmpeg uses x2rgb10 format, which matches to VA X2R10G10B10 and MFX a2rgb10 formats cmd to check: ./ffmpeg -hwaccel qsv -v verbose -c:v hevc_qsv -i \ ../p010.h265 -vf scale_qsv=format=x2rgb10,hwdownload,format=x2rgb10 \ -vframes 1 out.yuv -y fixes: Intel-Media-SDK#1654

daleksan · 2020-10-15T10:35:00Z

@xhaihao could you cover this thread on behalf of @fulinjie ?

FurongZhang · 2021-01-04T01:43:34Z

I am following up this issue and double confirming with validation. After confirmation, I will close this issue if test results are good.

daleksan · 2021-06-09T22:55:04Z

Hi @FurongZhang do you have any updates?

dmitryermilov closed this as completed Sep 25, 2019

dmitryermilov reopened this Sep 25, 2019

dmitryermilov assigned AntonGrishin Sep 25, 2019

dmitryermilov assigned FurongZhang Jun 19, 2020

dmitryermilov added the vpp label Jun 19, 2020

dmitryermilov changed the title ~~MSDK vpp can't download the driver surface with format "MFX_FOUCC_A2RGB10"~~ Pass through mode for VIDEO->SYSTEM with MFX_FOURCC_A2RGB10 format Jun 19, 2020

dmitryermilov added the feature request label Jun 23, 2020

dmitryermilov unassigned FurongZhang Jun 23, 2020

AntonGrishin pushed a commit to AntonGrishin/MediaSDK that referenced this issue Jun 24, 2020

vpp: allow a2rgb10 input for copy pass thru

fcbdbd1

fixes: Intel-Media-SDK#1654

AntonGrishin pushed a commit to AntonGrishin/MediaSDK that referenced this issue Jun 25, 2020

vpp: allow a2rgb10 input for copy pass thru

20058c0

fixes: Intel-Media-SDK#1654

dmitryermilov closed this as completed in a786fde Jul 10, 2020

dmitryermilov reopened this Jul 10, 2020

AntonGrishin mentioned this issue Aug 6, 2020

Fix passthru copy when input X2RGB10 #2268

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pass through mode for VIDEO->SYSTEM with MFX_FOURCC_A2RGB10 format #1654

Pass through mode for VIDEO->SYSTEM with MFX_FOURCC_A2RGB10 format #1654

sunxinpeng commented Sep 20, 2019

dmitryermilov commented Sep 25, 2019

sunxinpeng commented Sep 25, 2019

dmitryermilov commented Sep 25, 2019

AntonGrishin commented Sep 30, 2019

AntonGrishin commented Oct 16, 2019

sunxinpeng commented Oct 16, 2019

AntonGrishin commented Oct 30, 2019

fulinjie commented Jun 18, 2020

dmitryermilov commented Jun 18, 2020

fulinjie commented Jun 19, 2020

fulinjie commented Jun 19, 2020

dmitryermilov commented Jun 19, 2020

AntonGrishin commented Jun 24, 2020 •

edited

fulinjie commented Jun 24, 2020

fulinjie commented Jun 29, 2020

AntonGrishin commented Jul 3, 2020

fulinjie commented Jul 10, 2020

fulinjie commented Jul 10, 2020

fulinjie commented Jul 10, 2020

fulinjie commented Jul 10, 2020

dmitryermilov commented Jul 10, 2020

fulinjie commented Jul 16, 2020

AntonGrishin commented Jul 16, 2020

fulinjie commented Jul 20, 2020

AntonGrishin commented Aug 6, 2020

fulinjie commented Aug 6, 2020

daleksan commented Oct 15, 2020

FurongZhang commented Jan 4, 2021

daleksan commented Jun 9, 2021

Pass through mode for VIDEO->SYSTEM with MFX_FOURCC_A2RGB10 format #1654

Pass through mode for VIDEO->SYSTEM with MFX_FOURCC_A2RGB10 format #1654

Comments

sunxinpeng commented Sep 20, 2019

dmitryermilov commented Sep 25, 2019

sunxinpeng commented Sep 25, 2019

dmitryermilov commented Sep 25, 2019

AntonGrishin commented Sep 30, 2019

AntonGrishin commented Oct 16, 2019

sunxinpeng commented Oct 16, 2019

AntonGrishin commented Oct 30, 2019

fulinjie commented Jun 18, 2020

dmitryermilov commented Jun 18, 2020

fulinjie commented Jun 19, 2020

fulinjie commented Jun 19, 2020

dmitryermilov commented Jun 19, 2020

AntonGrishin commented Jun 24, 2020 • edited

fulinjie commented Jun 24, 2020

fulinjie commented Jun 29, 2020

AntonGrishin commented Jul 3, 2020

fulinjie commented Jul 10, 2020

fulinjie commented Jul 10, 2020

fulinjie commented Jul 10, 2020

fulinjie commented Jul 10, 2020

dmitryermilov commented Jul 10, 2020

fulinjie commented Jul 16, 2020

AntonGrishin commented Jul 16, 2020

fulinjie commented Jul 20, 2020

AntonGrishin commented Aug 6, 2020

fulinjie commented Aug 6, 2020

daleksan commented Oct 15, 2020

FurongZhang commented Jan 4, 2021

daleksan commented Jun 9, 2021

AntonGrishin commented Jun 24, 2020 •

edited