[Bug]: Encoding AV1 with color range "Full" outputs wrong colors

[akiirui]

Describe the bug

Encoding AV1 with color range "Full" outputs file has wrong color.

To Reproduce
Steps to reproduce the behavior:

1. Open OBS
2. Set Output - "Video Encoder" to AMD HW AV1
3. Set Advanced - "Color Range" to limited. Equivalent to AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE=AMF_VIDEO_CONVERTER_COLOR_PROFILE_709.
4. Record a video with range limited
5. Set Advanced - "Color Range" to full. Equivalent to AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE=AMF_VIDEO_CONVERTER_COLOR_PROFILE_FULL_709.
6. Record a video with range full
7. Compare two videos

Setup (please complete the following information):

• OS: Windows 11
• Driver Version: 23.5.2
• GPU: RX 7900 XTX
• Which component has the issue: Encoder

Expected behavior

For the different color range "limited" and "full" the colors should be nearly identical when viewed with the naked eye.

Screenshots

Screenshot of limited:

Screenshot of full:

The original background image:

Additional context

mediainfo outputs of limited:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L5.2
Codec ID                                 : V_AV1
Duration                                 : 5 s 434 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Original height                          : 1 082 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

mediainfo outputs of full:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L5.2
Codec ID                                 : V_AV1
Duration                                 : 5 s 400 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Original height                          : 1 082 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Full
colour_range_Original                    : Limited
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

example videos.zip

And in AMF_Video_Encode_AV1_API.md - Name: AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE:

Description: Color profile of the compressed output stream. SDR - Setting this parameter (COLOR_PROFILE) can fully describe a surface for SDR use case. HDR – For HDR use case the TRANSFER_CHARACTERISTIC, COLOR_PRIMARIES, and NOMINAL_RANGE parameters describe the surface. Determines the optional VUI parameter “matrix_coefficients”.

But NOMINAL_RANGE isn't available in VideoEncoderAV1.h

Original issue: obsproject/obs-studio#9121

[MikhailAMD]

Could you please provide more details why you consider the result as an error?
What I see is in "full" image black color is more deep compare to "limited" case. See details around thee dots in the text at left top corner.
This is expected as for limited case color is compressed into limited range 16-235, while full is 0-255.
MediaInfo seems showing the correct flags.
More importantly, I believe, in OBS color conversion to YUV (where range is applied) is done in OBS by shaders, so AMF encoder just sets flags in VUI header.

[akiirui]

Please see "original background image", the full color range screenshot is blacker than original.

In a properly implemented color space, the player correctly displays limited and full as near-consistent colors.

And it differs from other implementations (AMF H264/H265 or SVT-AV1/AOM AV1).

mediainfo outputs of AOM-limited:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L4.1
Codec ID                                 : V_AV1
Duration                                 : 5 s 367 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

mediainfo outputs of AOM-full:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L4.1
Codec ID                                 : V_AV1
Duration                                 : 5 s 384 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Full
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

Screenshot of limited from MPV:

Screenshot of full from MPV:

[akiirui]

And here have some test charts:

Video: test_charts.zip

Full:

Limited:

[flaeri]

More importantly, I believe, in OBS color conversion to YUV (where range is applied) is done in OBS by shaders, so AMF encoder just sets flags in VUI header.

I believe the issue lies with the tags/flags. When parsing the bitstream, the color range shows:
seq_header_obu\color_config():
color_range 0 (should be 1 for full range)

Both FFmpeg, MPV and VLC reads the file as being partial range color (incorrect), hence the values being mishandeled, resulting in the full/partial mismatch.

[MikhailAMD]

I was confused by MediaInfo. It says "Full". Yes, both clips have color_range = 0. I've opened internal ticket. Thanks.