NVENC: Recording never stops with B-Frame Reference not set to Disabled. (Workaround inside!)

[TheRealDadbeard]

Description

I saw your optimized settings on your blog and wanted to try them. Unfortunately I can't stop the recording as it will just be stuck on "Stopping Recording". OBS does not freeze or crash and I have to force close it via Task Manager.

This happens if B-Frame Reference Mode is set to your suggested setting. If I set it to Default then the recording closes fine. Setting it to use every B-Frame will not even start the recording and just throw an error.

Log from a recording and then stopping it https://pastebin.com/Jn3z6MWr

System Information

• Software Version: OBS 25.0.8 StreamFX 0.8.0
• Operating System: Windows
• Kernel Version: 2004
• CPU: 3700x
• GPU: 2070RTX
• RAM: 16

[Xaymar]

Both of these are known bugs caused by FFmpegs use of NVENC. If you encounter these, your only option is to Disable B-Frame references, which hurts quality.

[TheRealDadbeard]

Thanks for the info.

So it works without issues when I set it to default. Is default in this context disabled? Or something else?

Any workaround or do we have to wait for it to be fixed by ffmpeg devs?

[Xaymar]

The value of 'Default' depends on the preset chosen, but usually is 'Disabled' for the majority of them. There's no known workaround other than luck, and since it works fine for streaming (most of the time), waiting for the FFmpeg devs to fix it is probably the only way. While I could apply a workaround, probably, it would likely not do much - similar to the ProRes postprocessing it would likely cost extra performance.

[TheRealDadbeard]

Oh so while streaming it can be fine. Yeah gonna try this later and see if it works out if not then I'll just use OBS ones until this is maybe fixed in the future.

[TheRealDadbeard]

Okay can confirm it works without issues while streaming.

[Xaymar]

Further information, the "Each" mode is only supported for HEVC according to the NVENC manual.

[Xaymar]

Good news, it's actually fixed in the recent ffmpeg versions. Just need to wait for OBS to update their FFmpeg version.

[TheRealDadbeard]

Oh thats good news, let's hope the next OBS release has the new ffmpeg version in it.

[MythionVR]

I'm reporting in with the same issue and glad to hear that there's a fix in the works.

A side note and not sure if it's completely relevant, when choosing the Encoder I don't have the same option as your settings.
I see "H.264/AVC NVidia NVENC" minus the "(via ffmpeg)" part in the drop down menu. However it includes all the same settings further in the settings guide. Is that the option I should be picking or am I actually missing a dependancy?

[Xaymar]

@MythionVR The latest version has the (via FFmpeg) name in it.

[MythionVR]

This is what I currently have listed in my Encoder options. H.264/AVC NVidia NVENC gives me the same options, but there's no option for FFmpeg despite having StreamFX. Should I create a new thread?

edit I updated to 0.9.0, I thought 0.8.0 was the lastest, my apologies! It's now working as intended.

[Xaymar]

0.8.0 is the latest stable, 0.9.0 is currently in alpha.

[Xaymar]

Closing this for tracking purposes. FFmpeg has fixed this bug for several months now, the OBS Project just needs to get their version of FFmpeg updated accordingly.

[Xaymar]

Fixed by OBS Studio 26.1 and newer, which now uses FFmpeg 4.2.4 + some patches.