Video decoding falling behind can cause large memory consumption

[MiraiSubject]

Describe the bug:
The tournament showcase screen I recently PR'ed ppy/osu#8859 is leaking memory. I first encountered this issue when livestreaming a showcase myself using this screen.

So I decided to take a look in the debugger and checking all the scenes in lazer's tournament mode. Apparently only the showcase screen showed symptoms of memory leaking. In the graph shown you can see it linearly increasing and then flattening. The flattening is me switching to the other scenes and keeping it on for a few seconds. After that I switched back to the showcase screen and the memory usage started increasing again.

Screenshots or videos showing encountered issue:

osu!lazer version:
2020.429.0 and ppy/osu@3bf58b7

Logs:
network.log
performance.log
runtime.log

[peppy]

Have you tried without any custom videos you have loaded?

[MiraiSubject]

Okay so I've tested the following:

• No video -> no memory leaks
• 1080p h264 video -> no memory leaks
• 2160p h264 video -> memory leaks

I've included the specific video file that is causing the leak over here

[peppy]

Can you check/include log files? I'd suspect something may show up there, or something in Ctrl-f2 or Ctrl-f3 view that gives an idea of what is going wrong.

[MiraiSubject]

After some further testing I decided to drop this video in on the other tournament scenes, namely, the gameplay, mappool and seeding scene. All of these started to behave the same way as the showcase scene.

In the CTRL+F2 menu the Garbage Collection Gen0 keeps increasing with 1 every 1-2 seconds. All the other values stayed the same or fluctuated across the same values. I have included the performance and runtime logs with these menus open.

This happened with both of the videos, but the logs are with the 2160p video in place.

It now seems like the leak is also there with just the 1080p video, but it's less noticeable due to the memory consumption being lower.

performance.log
runtime.log

[bdach]

Yup, very reproducible with the attached video (to the point that I'd warn to not try to reproduce this on a low-memory system, as this fills up memory quite quickly, so better to watch out to not stall the entire system on a swap). After prodding this with dotmemory it seems that the decoding loop is doing a lot of allocations on the large object heap, therefore I assume introducing memory pressure via fragmentation:

Looking at the source VideoDecoder is allocating a potentially large buffer on every readPacket call, which certainly has the potential to make large enough allocations to end up on the LOH (sometimes to the tune of 2MB - way over the minimum limit of 85kB).

Weirdly enough even setting

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;

doesn't really do anything. From these messages that show in console:

[runtime] 2020-05-04 17:49:02 [verbose]: Video too far out of sync (2452.4500000000003), seeking to 4958.551299999999
[runtime] 2020-05-04 17:49:02 [verbose]: Video too far out of sync (2469.1333333333337), seeking to 4980.4449
[runtime] 2020-05-04 17:49:06 [verbose]: Video too far out of sync (5572.2333333333345), seeking to 1602.6105666666635

I assume the reason for this is that due to the time it takes to decode, the decoding loop falls behind so far that the compaction doesn't even manage to keep up with the rate of allocations anyway.

In fact, as long as this branch of decoding doesn't execute (you can emulate that at all times by artificially setting lenience_before_seek to an arbitrarily large value) the memory usage stays constant, but the video plays painfully slowly. The instant it falls out of sync the memory usage balloons out of control.

Unfortunately the requested buffer size does not seem to be constant, so at current the best solution I can see is to keep a single buffer instance which is enlarged whenever necessary and manually keep byte count of what data is valid for a given packet to avoid double-reading stale buffer data, but even that doesn't stop the memory usage from uncontrollably increasing when video is too far out of sync.

[peppy]

Going to move this to framework and add to may milestone (seems like something we should get fixed asap)

[huoyaoyuan]

I detected memory leak when trying to refactor video decoding half a year ago, with 1080p video in visual test.