AMD HW AV1 cannot use P010 colour format with QVBR, HQVBR, HQCBR rate control, OBS does not provide any useful errors why recording fails

[empZealoth]

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

30.0.2

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/tSO2db1BOnxNJl4Q

OBS Studio Crash Log URL

No response

Expected Behavior

Succesfully record HDR footage using AMD HW AV1 by switching to P010 colour format and Rec. 2100 (PQ) colour space as well as switching capture colour space to Rec. 2100 (PQ)

Current Behavior

Switching to P010 colour space causes recording to fail with weird/irrelevant error

I already verified both my AMD drivers and OBS are up-to-date

The log line is also pretty cryptic:
12:23:05.451: [fallback-amf-av1] amf_av1_create_fallback: AMFComponent::Init failed: AMF_INVALID_ARG

Steps to Reproduce

1. I'm using W11 and 7800XT GPU and these settings:
   
   
   
2. In my inexperienced/limited testing it's enough to switch to P010 colour format to get the error in my setup
3. Start Recording
4. Error -> recording cancelled

...

Anything else we should know?

Current settings work perfectly when I'm using AMD HW AV1 with NV12 and Rec.709, so at least some part of my hardware encoder is clearly working

[SuslikV]

has related thread on the forum:
https://obsproject.com/forum/threads/amd-hw-av1-hdr-nonsensical-errors.172990/

[lextra2]

QVBR, HQVBR & HQCBR all set EnablePreAnalysis=true
But PreAnalysis only works with NV12

Use Rate Control CQP if you want to record in HDR
Your title is misleading

[MattyFresh68]

Can confirm at least on HEVC that 10 bit works when PreAnalysis is disabled

[empZealoth]

QVBR, HQVBR & HQCBR all set EnablePreAnalysis=true But PreAnalysis only works with NV12

Use Rate Control CQP if you want to record in HDR Your title is misleading

Thanks, that worked, in a sense of "I could record P010 successfully after switching to CQP"

How is it misleading?
My recording was working fine in NV12, switching to P010 breaks it with a nonsensical error that tells you nothing. Googling also returned zero useful information

Obviously OBS can't do anything about the fact that HQVBR requires preanalysis and that only works with NV12, but it sure can provide a more meaningful error

Basically your message minus the snark would explain everything perfectly

[lextra2]

How is it misleading?

Because your title implies P010 doesn't work with AMF AV1
But in reality, it only doesn't work with your selected Rate Control method, and the ones I've listed.
You could have tested them, each.
Just something to keep in mind the next time you report an issue.
Don't take it personal.

And yes, I agree that the error could be more informative.

[Fenrirthviti]

@lextra2 Please keep comments on-topic, and don't berate users for not having knowledge you think they should posses.

As for this issue, I agree that if this is a known universal fail condition we should have some kind of protection or warning against it. @empZealoth Can you adjust the title to reflect the error is with P010 and specific rate control methods so this can be better tracked?

[RytoEX]

Obviously OBS can't do anything about the fact that HQVBR requires preanalysis and that only works with NV12, but it sure can provide a more meaningful error

As for this issue, I agree that if this is a known universal fail condition we should have some kind of protection or warning against it.

I'm not sure that we can reliably know that QVBR, HQVBR & HQCBR will forever set EnablePreAnalysis=true. That does seem to be true now, but I think that option is set internally by AMF. I'm not sure that AMF provides any extended error messages or if it only returns the error code (in this case, AMF_INVALID_ARG for "invalid argument" or "memory type or size are invalid").

We can probably add detection on our end for this (check color format and rate control method?), but we'll have to be mindful if AMF changes what it does or supports.

[empZealoth]

Can you not check for EnablePreAnalysis=true directly? Technically you should be able to enable it with other rate control methods manually too, right?

I changed the title. If it still needs to be changed just say what needs to be improved

[lextra2]

@RytoEX
You could check for the selected Color Format and selected Rate Control, and then give the user an error if incompatibility arises.

Also, AMD is probably working on making PreAnalysis work with P010

[RytoEX]

Can you not check for EnablePreAnalysis=true directly? Technically you should be able to enable it with other rate control methods manually too, right?

I'm not sure the property itself is initialized in AMF until we try to call Init, since I don't think we are setting it ourselves. I'm actually not clear from the AMF docs if these rate control methods forcefully enable PreAnalysis, or if you must enable it for them to work. From the AMF docs:

QVBR, HQVBR and HQCBR are only supported if PreAnalysis is enabled.

You could check for the selected Color Format and selected Rate Control, and then give the user an error if incompatibility arises.

Also, AMD is probably working on making PreAnalysis work with P010

Yes, I mentioned this as a possibility above, with the caveat that we will have to keep on top of what changes AMD continues to make:

We can probably add detection on our end for this (check color format and rate control method?), but we'll have to be mindful if AMF changes what it does or supports.