vpp_qsv hardware context issue

[akhilxavi]

On my ADL device running 6.8.0.11 kernel, I am trying to decode, scale and format conversion using hevc_qsv, vpp_qsv, vpp_scale respectively.
below command work as expected on my device

ffmpeg -hwaccel qsv -c:v hevc_qsv -i /media/pcie/latency-test/complex-video/nv12-5000.h265 -vf 'vpp_qsv=framerate=60,scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx' -f null -

But when I code after linking the filters
source -> vpp_qsv -> scale_qsv -> hwdownload -> format -> sink, I am seeing below error,

[vpp_qsv @ 0x5c5eb9dc18c0] No hw context provided.
[vpp_qsv @ 0x5c5eb9dc18c0] Failed to configure output pad on vpp_qsv

How to specify hw context for vpp in code? How the vpp sink pad is expected to be configured after linking?
I configures the in and out pads as below.

/*
 * The buffer source output must be connected to the input pad of
 * the first filter described by filters_descr; since the first
 * filter input label is not specified, it is set to "in" by
 * default.
 */
outputs->name       = av_strdup("in");
outputs->filter_ctx = buffersrc_ctx;
outputs->pad_idx    = 0;
outputs->next       = NULL;

/*
 * The buffer sink input must be connected to the output pad of
 * the last filter described by filters_descr; since the last
 * filter output label is not specified, it is set to "out" by
 * default.
 */
inputs->name       = av_strdup("out");
inputs->filter_ctx = buffersink_ctx;
inputs->pad_idx    = 0;
inputs->next       = NULL;

Thank you in advance.

[feiwan1]

Which version of onevpl/vpl-gpu-rt did you use? Could you try the master code?

[feiwan1]

BTW, if possible, please use kernel 6.9.0+. See more details in #322

[akhilxavi]

@feiwan1 I was trying the filter chain "vpp_qsv=framerate=60,scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx" but it failed with vpp_qsv issues. Now I modified the filter chain to:

filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx)

"scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx" and the filter source is feed with AV_PIX_FMT_QSV from the decoder itself. It works with the same code which I set hw and frame context for each members in the filter chain.

    if(avfilter_graph_parse_ptr(filter_graph, args, &inputs, &outputs, NULL) <0)
             av_log(NULL, AV_LOG_ERROR, "failed to parse filter graph description\n");

    for(unsigned int i = 0; i < filter_graph->nb_filters; i++){
            if(!(filter_graph->filters[i]->hw_device_ctx = avctx->hw_device_ctx))
                    av_log(NULL, AV_LOG_ERROR, "hw device ctx to filters failed\n");
        }

seems like the issue is only with "vpp_qsv" component. Does "vpp_qsv" need any extra initialization than the above one?

My kernel version is: 6.8.0-31-generic
Vaapi version: 1.22.0 (libva 2.22.0.pre1)
libmfx: 1.2.10
intel iHD drivers - 24.2.1
firmwares:
DMC firmware i915/adlp_dmc.bin (v2.20)
GuC firmware i915/tgl_guc_70.bin version 70.20.0
HuC firmware i915/tgl_huc.bin version 7.9.3

[feiwan1]

seems like the issue is only with "vpp_qsv" component. Does "vpp_qsv" need any extra initialization than the above one?

No, I guess. vpp_qsv and scale_qsv shares same configuration and initialization.

[xhaihao]

But when I code after linking the filters
source -> vpp_qsv -> scale_qsv -> hwdownload -> format -> sink, I am seeing below error,

Does the source support hw frames (AV_PIX_FMT_QSV) ? If not, you should specify avctx->hw_device_ctx for vpp_qsv filter in your code.

[xhaihao]

Now I modified the filter chain to:

filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx)

"scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx" and the filter source is feed with AV_PIX_FMT_QSV from the decoder itself. It works with the same code which I set hw and frame context for each members in the filter chain.

The filter chain: decoder(qsv)->vpp_qsv->hwdownload->output(vuyx) should work too.

[akhilxavi]

But when I code after linking the filters
source -> vpp_qsv -> scale_qsv -> hwdownload -> format -> sink, I am seeing below error,

Does the source support hw frames (AV_PIX_FMT_QSV) ? If not, you should specify avctx->hw_device_ctx for vpp_qsv filter in your code.

Yes my Alder lake decoder is supporting hw frames. By outputting AV_PIX_FMT_QSV from decoder, below configuration works with hw context and and frame context in my code.

filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx)

If I introduce vpp_qsv into my code filter chain and specifying avctx->hw_device_ctx through the above for loop, I am not seeing any frames output from sink.

vpp_qsv -> scale_qsv -> hwdownload -> format -> sink

Below script works from cmd line though.
ffmpeg -hwaccel qsv -c:v hevc_qsv -i /media/pcie/latency-test/complex-video/nv12-5000.h265 -vf 'vpp_qsv=framerate=60,scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx' -f null -

Not sure what is missing in code which is specific to vpp_qsv.

[xhaihao]

If I introduce vpp_qsv into my code filter chain and specifying avctx->hw_device_ctx through the above for loop, I am not seeing any frames output from sink.

vpp_qsv -> scale_qsv -> hwdownload -> format -> sink

Does vpp_qsv->hwdownload->format->sink work ?

[akhilxavi]

If I introduce vpp_qsv into my code filter chain and specifying avctx->hw_device_ctx through the above for loop, I am not seeing any frames output from sink.
vpp_qsv -> scale_qsv -> hwdownload -> format -> sink

Does vpp_qsv->hwdownload->format->sink work ?

with above filter,
when configuring input to filter as QSV and output as vuyx, seeing below error

libva info: va_openDriver() returns 0
[hwdownload @ 0x59108a444fc0] Invalid output format vuyx for hwframe download.
[Parsed_hwdownload_1 @ 0x59108a44b800] Failed to configure output pad on Parsed_hwdownload_1
failed to configure filter graph

when configuring input to filter as codeccontext pix format and output as vuyx, seeing below error (same with vpp_qsv -> scale_qsv -> hwdownload -> format -> sink)
[AVHWFramesContext @ 0x61446b6369c0] Unsupported pixel format
[Parsed_vpp_qsv_0 @ 0x61446b634e80] Error creating frames_ctx for output pad.
[Parsed_vpp_qsv_0 @ 0x61446b634e80] Failed to configure output pad on Parsed_vpp_qsv_0
failed to configure filter graph
[hevc_qsv @ 0x61446ad36140] Invalid pkt_timebase, passing timestamps as-is.
[in @ 0x61446b629a00] Changing video frame properties on the fly is not supported by all filters.
[in @ 0x61446b629a00] filter context - w: 3840 h: 2160 fmt: 0 csp: unknown range: unknown, incoming frame - w: 3840 h: 2160 fmt: 114 csp: unknown range: tv pts_time: 0
[hwdownload @ 0x61446b635080] Input frames must have hardware context.
[hwdownload @ 0x61446b635080] Input frames must have hardware context.

[xhaihao]

scale_qsv and vpp_qsv have the same code for initialization, it is weird scale_qsv can work but vpp_qsv doesn't work. What's your FFmpeg version ?

[akhilxavi]

scale_qsv and vpp_qsv have the same code for initialization, it is weird scale_qsv can work but vpp_qsv doesn't work. What's your FFmpeg version ?

I build it from source:

ffmpeg version N-114973-gce33a5f16d Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13 (Ubuntu 13.2.0-23ubuntu4)

[xhaihao]

If I introduce vpp_qsv into my code filter chain and specifying avctx->hw_device_ctx through the above for loop, I am not seeing any frames output from sink.
vpp_qsv -> scale_qsv -> hwdownload -> format -> sink

Does vpp_qsv->hwdownload->format->sink work ?

with above filter, when configuring input to filter as QSV and output as vuyx, seeing below error

libva info: va_openDriver() returns 0 [hwdownload @ 0x59108a444fc0] Invalid output format vuyx for hwframe download. [Parsed_hwdownload_1 @ 0x59108a44b800] Failed to configure output pad on Parsed_hwdownload_1 failed to configure filter graph

Now I modified the filter chain to:

filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx)

"scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx" and the filter source is feed with AV_PIX_FMT_QSV from the decoder itself. It works with the same code which I set hw and frame context for each members in the filter chain.

Seems your test clip is not a YUV444 8bit stream, you set the output sw format of scale_qsv to vuyx when testing filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx), you should change the output sw format of vpp_qsv to vuyx too when trying filter chain: decoder (qsv)->vpp_qsv->hwdownload->output(vuyx)
"vpp_qsv=format=vuyx,hwdownload,format=vuyx"

[akhilxavi]

If I introduce vpp_qsv into my code filter chain and specifying avctx->hw_device_ctx through the above for loop, I am not seeing any frames output from sink.
vpp_qsv -> scale_qsv -> hwdownload -> format -> sink

Does vpp_qsv->hwdownload->format->sink work ?

with above filter, when configuring input to filter as QSV and output as vuyx, seeing below error
libva info: va_openDriver() returns 0 [hwdownload @ 0x59108a444fc0] Invalid output format vuyx for hwframe download. [Parsed_hwdownload_1 @ 0x59108a44b800] Failed to configure output pad on Parsed_hwdownload_1 failed to configure filter graph

Now I modified the filter chain to:
filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx)
"scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx" and the filter source is feed with AV_PIX_FMT_QSV from the decoder itself. It works with the same code which I set hw and frame context for each members in the filter chain.

Seems your test clip is not a YUV444 8bit stream, you set the output sw format of scale_qsv to vuyx when testing filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx), you should change the output sw format of vpp_qsv to vuyx too when trying filter chain: decoder (qsv)->vpp_qsv->hwdownload->output(vuyx) "vpp_qsv=format=vuyx,hwdownload,format=vuyx"

If I introduce vpp_qsv into my code filter chain and specifying avctx->hw_device_ctx through the above for loop, I am not seeing any frames output from sink.
vpp_qsv -> scale_qsv -> hwdownload -> format -> sink

Does vpp_qsv->hwdownload->format->sink work ?

with above filter, when configuring input to filter as QSV and output as vuyx, seeing below error
libva info: va_openDriver() returns 0 [hwdownload @ 0x59108a444fc0] Invalid output format vuyx for hwframe download. [Parsed_hwdownload_1 @ 0x59108a44b800] Failed to configure output pad on Parsed_hwdownload_1 failed to configure filter graph

Now I modified the filter chain to:
filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx)
"scale_qsv=w=1920:h=1080:format=vuyx,hwdownload,format=vuyx" and the filter source is feed with AV_PIX_FMT_QSV from the decoder itself. It works with the same code which I set hw and frame context for each members in the filter chain.

Seems your test clip is not a YUV444 8bit stream, you set the output sw format of scale_qsv to vuyx when testing filter chain: decoder(qsv)->scale_qsv->hwdownload->output(vuyx), you should change the output sw format of vpp_qsv to vuyx too when trying filter chain: decoder (qsv)->vpp_qsv->hwdownload->output(vuyx) "vpp_qsv=format=vuyx,hwdownload,format=vuyx"

with format set for vpp_qsv the code starts working. The decoder output is in hw_frame format.

Thank you for the help.

does the decoder should always output hw_frames to feed to filter source? My decoder is always outputting frames in hw_frames format, is that because I set AVHWFramesContext -> format to AV_PIX_FMT_QSV?

[xhaihao]

does the decoder should always output hw_frames to feed to filter source? My decoder is always outputting frames in hw_frames format, is that because I set AVHWFramesContext -> format to AV_PIX_FMT_QSV?

hevc_qsv should be able to output frames in system memory if you neither set avctx->hw_device_ctx nor set avctx->hw_frames_ctx in your code. You may run the command below to verify it:

ffmpeg -c:v hevc_qsv -i input.mp4 -vf "scale=w=1920:h=1080" -f null -