AMF H264 encoder hangs OBS on shutdown

[flaeri]

Description

Log: https://obsproject.com/logs/ifdwIjjLYlmHCr9S

Current behavior:
If the encoder is set to use the preset "Quality", and you trigger a recording, the encoder seems to deadlock/hang on shutdown, leaving the OBS process hanging (seemingly forever).

Expected Behavior

I expect OBS to shutdown cleanly when using AMF, without having to kill it in task manager

Reproduction Steps

1. Create a new profile
2. Skip the auto config wizard
3. Go into settings - > video, ensure both canvas and output is set to 1080p, 60fps
4. Go into settings -> output, and change the mode to "advanced"
5. Go to the recording tab, and change the encoder to H264 (AMD)
6. Change the "quality preset" to "quality"
7. hit ok, and initiate a recording
8. After recording for 10 seconds, stop the recording.
9. Exit OBS by clicking the x in the top right.
10. Open up task manager (ctrl+shift+esc) -> details, and observe how OBS64.exe is still open.

System Information

Win10 20h2
OBS 27.0.1

• CPU: AMD Ryzen 5 4500u APU
• GPU: AMD Ryzen 5 4500u APU
• RAM: 16gb

Checklist:

• I have read the contribution guidelines.
• I can reproduce the issue with the exact reproduction steps I have provided.
• The issue appears on all of my systems that can run this software.

Additional information

Stack trace during hang:
https://hastebin.com/imuzoyahaq
(check line 612)

Previously I was running older AMD display drivers (20.11.2), and they did not exhibit this issue.
Once I updated to more recent drivers (21.6.1) this issue becomes apparent.

I've attempted to mimic OBS' encoder settings in ffmpeg 4.4, and it does not display similar behavior on the recent drivers.
Changing the quality preset back to "balanced" destroys the encoder on shutdown successfully.

@Pablomx2 brought this issue to my attention, and we've spent some time working it out.
Very similar (I think identical) behavior is observed both on my AMD 4500U APU, and their RX 6700 XT.

[evieveevee]

I have confirmed this hang on my system.

[Pablomx2]

Can confirm as well.
Only fix so far is to use StreamFX FFMPEG

[flaeri]

well, the fix would just be to not use the "quality" preset.
Couple of other things worth noting, that cause the same behavior.

• Increasing the Profile (5.2)
• Increasing the resolution (this has worse outcome, not cleanly stopping the recording, leaving the encoder incapable of starting new ones). Seems to happen at 1440p60 and above.

[flaeri]

https://www.amd.com/en/support/kb/release-notes/rn-rad-win-21-10-2

New driver, and it seems to be working super well for me. No crashes or hangs on shutdown. h264 issue seem resolved. I can run 1440p i444. Quality preset seems to be working, as well as motion boost etc. Give it a try.

[Pablomx2]

Newest Optional Driver 21.10.2 doesn't work with 6700 XT GPU. OBS fails to start the encoder and has issues with the Pre-Pass Mode. Here are the logs
2021-10-11 14-00-47.txt
2021-10-11 13-59-03.txt
2021-10-11 13-58-39.txt
2021-10-11 13-57-37.txt
.

[flaeri]

with obsproject/obs-studio#6508 being merged, I think it makes sense to just close this issue, as it wont be relevant any more fairly soon.