Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Visuals Stop Rendering When Source is Repeatedly Refreshed #356

Closed
trevorrl opened this issue Feb 28, 2022 · 18 comments · Fixed by obsproject/obs-studio#6867
Closed

Visuals Stop Rendering When Source is Repeatedly Refreshed #356

trevorrl opened this issue Feb 28, 2022 · 18 comments · Fixed by obsproject/obs-studio#6867

Comments

@trevorrl
Copy link

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

27.2.1

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/qeEA-zfJeNJauEFY

OBS Studio Crash Log URL

No response

Expected Behavior

For the source not to stop rendering visually after being refreshed.

Current Behavior

After several refreshes, the source stops visually rendering.

Steps to Reproduce

  1. Add a browser source, for repro use this URL: trev.zone/alerts
  2. Close properties
  3. re-open properties
  4. check the box for custom fps or control audio
  5. close properties
  6. repeat 2-5 until source stops rendering (on my machine it takes 4-5 times)

Anything else we should know?

In my testing, this bug does not happen on every single URL. "trev.zone/alerts" is my own personal website and is the first URL I discovered the bug on. However, after further testing, I am able to reproduce on some other URLs as well (specifically Streamlabs widget URLs).

@WizardCM
Copy link
Member

Hmm, confirmed with your example URL (thanks for providing that!). This only seems to happen with hardware acceleration enabled.

@trevorrl
Copy link
Author

not sure if relevant, hopefully not derailing investigation:

  • /alerts is the only route on trev.zone that has this issue
  • the only difference afaik between that route and every other is that it: has CSS animations, and uses the socket.io websocket module

@WizardCM
Copy link
Member

WizardCM commented Mar 1, 2022

This is interesting. The first time the bug is triggered, it's like the page/renderer dies entirely:

image

image

After the bug is triggered once, if you then tweak the properties again, it recovers on the CEF side, but not in the OBS renderer:

image

image

Deleting and recreating the source does not fix rendering (essentially it breaks rendering for all browser sources).

This is confirmed with CEF error logging enabled:

[0301/204449.536:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=-1073741819
[0301/204453.936:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=-1073741819
[0301/204456.431:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=-1073741819

With additional logging, what actually seems to be happening is that each time we trigger a properties update in this situation, we're actually crashing the GPU process. This is after the first properties update:

Updated Browser Source Properties
[0301/204810.837:VERBOSE1:render_view_host_impl.cc(678)] RenderViewHostImpl::ClosePage() IsRenderViewLive() = 1, SuddenTerminationAllowed() = 0
[0301/204810.850:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=12, frame_id=1]
[0301/204810.890:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=-1073741819
[0301/204810.890:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 1 time(s)
[0301/204811.188:WARNING:gpu_process_host.cc(1007)] Reinitialized the GPU process after a crash. The reported initialization time was 252 ms
[0301/204811.197:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=12, frame_id=1]

Obviously you repeat 2-5 times, which crashes the GPU process 3 times, and it just.. doesn't relaunch properly.

[0301/204943.236:ERROR:gpu_process_host.cc(979)] GPU process exited unexpectedly: exit_code=-1073741819
[0301/204943.237:WARNING:gpu_process_host.cc(1292)] The GPU process has crashed 3 time(s)
[0301/204943.549:WARNING:gpu_process_host.cc(1007)] Reinitialized the GPU process after a crash. The reported initialization time was 0 ms

(notice the 0ms, it should be closer to 200ms)

The reason this doesn't happen with other browser URLs, is because other URLs don't crash the process:

Updated Browser Source Properties
[0301/205114.621:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=12, frame_id=1]
[0301/205114.860:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=12, frame_id=1]
Updated Browser Source Properties
[0301/205118.304:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=13, frame_id=1]
[0301/205118.539:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=13, frame_id=1]
Updated Browser Source Properties
[0301/205121.370:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=14, frame_id=1]
[0301/205121.605:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=14, frame_id=1]
Updated Browser Source Properties
[0301/205124.170:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=15, frame_id=1]
[0301/205124.405:VERBOSE1:media_stream_manager.cc(727)] RFAOSF::Core() [process_id=15, frame_id=1]

@paul-lrr
Copy link

paul-lrr commented Mar 4, 2022

As another point of data, I am also experiencing the same issue. After doing a few tests, it seems like it might be related to socket.io? I'm using locally served html documents, so I can't link to an example, but a totally static html page reloads without any problems. When I add socket.io to the header, it will consistently stop loading after 4 or 5 refreshes. I have no idea how socket.io could be causing a GPU crash, but it seems to be related, at least

@trevorrl
Copy link
Author

trevorrl commented Mar 4, 2022

As another point of data, I am also experiencing the same issue. After doing a few tests, it seems like it might be related to socket.io? I'm using locally served html documents, so I can't link to an example, but a totally static html page reloads without any problems. When I add socket.io to the header, it will consistently stop loading after 4 or 5 refreshes. I have no idea how socket.io could be causing a GPU crash, but it seems to be related, at least

Interesting, I had this theory as well since the /alerts route on trev.zone is the only page with this issue, and is also the only page on the site that uses socket.io. Additionally, streamlabs.com is fine whereas our widget URLs are not (which also use socket.io).

Don't necessarily want to mislead the investigation but signs do seem to point in that direction 🤔

@lindenkron
Copy link

This is unrelated to socket.io.

I've had this happen with a variety of sites, and on a rather frequent basis at this point. Have it happen with Raider.io Widgets (https://raider.io/widgets/) (scroll down the page).

Even occurred once while being in Studio mode, and adding a VDO.Ninja browser source to an empty scene.

Results are basically: All browser docks flicker white for a second, then restores themselves. Any browser source you have will either try to restore itself if possible - if not stay broken. Players such as twitch players for example do not restore themselves.

Unfortunately, this has resulted in me 4 times in the last 10 days having to kill OBS; to start it up again while being live. I am fortunate enough that we rebroadcast through 2PC setup (for this exact reason) so damages have been minimal.

Considering VDO doesn't do anything but display a WebRTC feed, and having reached out to Raider.io staff and having them disable all GPU accelerated items they could get their hands on (and it still happening) this is a rather peculiar bug.

@Fenrirthviti
Copy link
Member

Your issue sounds entirely unrelated to this, and doesn't match the behavior being described at all.

@WizardCM
Copy link
Member

@lindenkron Please provide an actual URL we can test with rather than "scroll down".

@lindenkron
Copy link

lindenkron commented Mar 22, 2022

Your issue sounds entirely unrelated to this, and doesn't match the behavior being described at all.

This thread was mentioned to me by Matt when describing the issues I was having. If it's unrelated to this, then I apologize.

@lindenkron Please provide an actual URL we can test with rather than "scroll down".

I've had the issue to a higher/lesser extend with all of their widgets. And I've only had it occur once with a VDO link.
However, this is one of the links I had the highest crash rate with:
https://raider.io/guilds/eu/twisting-nether/Method/raid-encounters/heroic/sepulcher-of-the-first-ones/anduin-wrynn?embed=1&embedmode=roster

Step by step:

  • Add browser dock (for visual presentation)
  • Add browser source (1920x1080)
  • Add url above
  • Press ok
  • Open browser source
  • Change url to random number
  • Press ok

Repeat steps until your Browser Dock flashes white (which I believe Matt said indicates the gpu renderer crashed?) Do this enough times, and all browser docks will disappear.

Short video of the outcome on an empty Scene Collection:
https://streamable.com/9g1rex

@elvissteinjr
Copy link

I have an url to add to the pile: From https://galactic.ink/labs/JS1k/BreathingGalaxies.html navigating to https://galactic.ink/ (top left link). Only happens with hardware acceleration and indeed logs to the GPU process crashing, same exit code.

As a bit of a background, I'm actually trying to make use of OBS' CEF fork for my own application (I get the same behavior in OBS). The affected URL here is part of the tests listed in the cefclient test application. I don't exactly need this page to work, but thought it might help at narrowing down the exact cause here.

Don't really know anything about the CEF's internals, so this is about all I can say.

@lindenkron
Copy link

I've managed to crash the GPU renderer without touching any sources. Basically just switching between scene collections. It appears it can be triggered simply by loading a browser source/browser dock.

@WizardCM
Copy link
Member

WizardCM commented Apr 9, 2022

I've managed to crash the GPU renderer without touching any sources. Basically just switching between scene collections. It appears it can be triggered simply by loading a browser source/browser dock.

The whole reason for the crash in the original ticket is when a browser source is unloading. So yes, switching scene collections will trigger it too. OBS has to unload all the sources before it switches to the new collection.

@lindenkron
Copy link

lindenkron commented Apr 25, 2022

I doubt unloading is a necessity. Had all browser sources crash/flicker twice during tonight's stream - and second time I did not touch anything prior to it happening. It appears it can happen at any given time for any or no reason.

@pkviet
Copy link
Member

pkviet commented Jun 7, 2022

The bug is fixed with an upgrade to cef 5060 (chromium 103). We'll provide a test build to confirm.

@walker-WSH
Copy link

This issue can be reproduced almost 100% on my PC.
I can verify it when you provide the test version.

@lindenkron
Copy link

lindenkron commented Jun 15, 2022

So far I've not been able to replicate any of the issues from CEF 95 in the CEF 103 test version.
Spent approximately 30 mins trying to break it in the usual ways I knew of, in a setup including all my regular settings, plugins & browser docks - in a heavy Scene Collection. 👍

@WizardCM WizardCM added this to In progress in CEF 5060 Upgrade Jun 15, 2022
@lindenkron
Copy link

lindenkron commented Jul 10, 2022

Unfortunately, having tested CEF 103 in production - while the regular ways we used to force the issue, appear gone - other methods still crash the GPU render.

I'm unable to find a specific method for reproducing - but having used a CEF 103 in live production, I still ran into all browser sources crashing multiple times, and a strange 'freeze speedup' issue.

CEF103 GPU render crash
https://clips.twitch.tv/TangentialDreamyHerbsGivePLZ-GPnpSTcz1Wiym6cf
VDO.ninja links recup on their own, as does the https link we get info from. However the Twitch video player needs hard refreshing to come back.

CEF103 freeze speedup issue
https://clips.twitch.tv/CoyTrustworthyJackalM4xHeh-yidLxVUXDfiF9WnO
I believe this is also a chromium issue since I've never experienced anything like this on CEF75 using the same setup. It however is a preferred issue since it doesn't cause a crash and thus does not require OBS restart after third time.

Has anyone else experienced similar issues? I know my vast amount of browser sources is not common, but hoping there's others out there with similar experiences that might help get the issue solved.

@lindenkron
Copy link

For anyone who is having issues with this. R1CH dug out this argument that I tested and confirmed working:
--disable-gpu-process-crash-limit

Add it to your OBS executable path, and you will not have to reboot OBS after third crash. You still have to refresh the browser sources after they crash, but it saves you a boot/potential stream going down.

Your path should look something like this:
"C:\Program Files\obs-studio\bin\64bit\obs64.exe" --disable-gpu-process-crash-limit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging a pull request may close this issue.

8 participants