VAAPI reencoding 10-Bit HEVC produces artifacts on Coffee Lake

[Sonic-Y3k]

Hi,

decoding a 10-Bit HEVC Stream with VAAPI causes artifacts while using a coffee lake processor.

uname -a:
Linux abc 4.16.2-2-ARCH #1 SMP PREEMPT Tue Apr 17 19:37:34 UTC 2018 x86_64 GNU/Linux

vainfo:

libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Coffee Lake - 2.1.0
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileH264MultiviewHigh : VAEntrypointVLD
VAProfileH264MultiviewHigh : VAEntrypointEncSlice
VAProfileH264StereoHigh : VAEntrypointVLD
VAProfileH264StereoHigh : VAEntrypointEncSlice
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointEncSlice
VAProfileVP9Profile2 : VAEntrypointVLD

steps to reproduce:

1. Download Big Buck Bunny
2. Convert file to 10-Bit HEVC with ffmpeg -i bbb_sunflower_2160p_30fps_normal.mp4 -c:v libx265 -pix_fmt yuv420p10le -crf 18 -c:a copy -t 00:01:00.000 -f matroska testfile.mkv
3. Reencode testfile with hardware decoding using ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i testfile.mkv -c:v libx264 -c:a copy -f matroska -t 00:01:00.000 testfile_output.mkv (note: encoder doesn't matter, applies to all of them)

If you are executing the same commands on Kaby Lake GT2, no artifacts are visible in the output!