[WIP] obs-nvenc on Linux

[openglfreak]

Description

Adds an OpenGL version of jim-nvenc to obs-ffmpeg.

Motivation and Context

Why is this change required?

The fallback NVENC encoder that is used on Linux copies rendered frames back to the system RAM, copies them again into an FFMPEG AVFrame, and makes FFMPEG upload them back to the GPU. This takes a lot of CPU time especially on older or lower-end CPUs, like my Intel i3-4370. Encoding the textures directly without copying them to system RAM solves this issue.

How Has This Been Tested?

[WIP] I'm testing on Arch Linux using my Intel i3-4370 + Nvidia GTX 1060 6GB. CPU usage is decreased from around 25-30% to around 11%. The resulting video file looks correct.

Types of changes

• Performance enhancement (non-breaking change which improves efficiency)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[tt2468]

How's this going? Many Linux OBS users have been anticipating this kind of addition

[WizardCM]

Hi there, any updates on this PR? Thanks!

[openglfreak]

Hi there, any updates on this PR? Thanks!

Hi, I didn't have interest in working on this more because solving this properly is much more involved than I anticipated. I have a branch in my fork with more WIP work that anyone else is free to pick up.

I have a bit of free time over christmas where I might work on this but no promises.

[openglfreak]

Wait I actually didn't test if it compiles lmao

Update: Works On My Machine

[openglfreak]

I am especially interested in Windows testers, because this PR also does some things to the Windows side. For example, it should have ARGB support now, but that is completely untested and might just be completely broken for all I know, so that would need to be tested and either fixed or disabled again before this PR can be considered for merging. Also testing Intel QSV and other encoders might be a good idea to make sure I didn't break any of them.

[openglfreak]

second obs-qsv11 commit adapted from #4931

[ogmkp]

I applied this PR to OBS 27.2.0-beta1 on Debian 11 making something wrong is possible because I'm a user:
With NV12 I got 15% of cpu usage
With RGBA I got 45% of cpu usage.

Edit: I forgot to choose the "New NVENC" for jim-nvenc, sorry my fault.
With NV12, I still get 15% of cpu usage and 50% of NVENC usage
On RGBA, I get 10% of CPU usage and 40% of NVENC usage. Thats using only 2% more CPU and 20% more NVENC usage than no encoding amazing.

[kkartaltepe]

I rebased the commits for encode_texture2 to play with another encoder. And at least for vaapi its pretty much what we want to pass textures around as well. For linux we might have also considered exporting the dmabuf and passing that to the encoder but in my tests exporting can be quite slow. And on platforms like intel we need a graphics context to blit so we would end up just re-importing in the encoder anyway, its nicer to send the textures and let us add an gs_* to export on the encoder side if needed.

One small change id recommend struct encoder_texture { probably just having a array of 4 textures as thats the most planes the kernel supports so we should never go beyond that. (maybe +1 to be a null just to be safe, but the planes must be known from the format anyway due to how we create nv12/p010 textures).

Thanks a bunch for this change... despite it not landing yet.

[Sid127]

Out of curiosity, is this still being worked on?

[jp9000]

It is actually. I've finally started working on this just recently in fact

[Sid127]

I applied the commits best I could on top of master. It compiles, but doesn't launch, errors out with ./obs: symbol lookup error: ../../obs-plugins/64bit/obs-ffmpeg.so: undefined symbol: load_nvenc_lib (built portable mode)

Commits have been applied on top of 34e3d64

Patch can be found here, because github won't let me attach the patch file directly.

[Sid127]

Made some more progress, fixed the linkage errors. Compiles, launches, non-ffmpeg nvenc is selectable, but obs segfaults when trying to start a recording. Updated patchset is here, and the backtrace from the segfault is here.

I'm struggling to move forward, my lack of in-depth knowledge about all this is finally showing up. Hoping the patchset is a good base for getting things working sooner than later

[Bleuzen]

Hi @jp9000 / @Lain-B , hope you're doing fine :)
Don't want to stress, just curious how it is going. In case you have no time for this anymore, can you publish your code so far?
Thanks.

[Lain-B]

yea, I was working on it, but ended up delaying it again 😅 ...I'm sorry about that. It requires merging the texture sharing PR so I had to go through those changes first. I just need to get through that PR and then I'll be able to apply it to these changes.

[tari3x]

I'd love to test this on linux and windows but can't compile openglfreak:linux-jim-nvenc. I can compile origin:master though, is there a more recent rebase I could try?

[Sid127]

@Lain-B could you mention which PR it is so we can track progress and help with testing wherever applicable?

[Fenrirthviti]

@Lain-B could you mention which PR it is so we can track progress and help with testing wherever applicable?

#8312

[ogmkp]

I'd like to see this improvement on linux version :)