Tuning: Memory Usage

[naanlizard]

Hello,

We run OME on several heterogeneous hosts, from cheap VPSs with ~ 4-8gb ram to dedicated servers with 128gb+

On the low-memory hosts, we seem to run out of memory even under light load. See below graph

Note that that is the edge server, origin is much lower memory usage, under a few hundred megabytes. Which is very good :)

Server.xml.j2 Ansible template

EdgeConfig.xml.j2 Ansible template

Some questions based on the above

1. What are the primary factors that affect memory usage for OME edge? e.g. viewers connected, streams being played, playback method, DVR
2. What tuning options do we have to reduce memory usage? I'm going to try massively reducing DVR retention first, but perhaps there are better ways?
3. What are the effects on OME if we set a RAM limit for its container in docker? Will it crash, or start dropping data from e.g. DVR, if possible?

[naanlizard]

Relating to the recent jemalloc fix - could that help with memory usage? Unfortunately, my low level programming is not very strong

@getroot

[getroot]

In my experience, jemalloc seems slightly better than glibc, but that's mostly about memory allocation/deallocation timing. It doesn't actually reduce memory usage.
HLS/LLHLS uses the most memory, because all segments are kept in memory. When DVR is enabled, segments beyond SegmentCount are written to disk, but everything within SegmentCount is still held in memory. So if you have a lot of streams like in your environment, memory usage will be high. The same applies to Edge, since OME's Edge isn't an HTTP proxy; it packages HLS on its own.
Running out of memory is undefined behavior. It can cause crashes or the server to freeze, and that's not something OME can control.
I've been busy lately and haven't had time to read your post properly yet. This is just a quick explanation for now. I'll take a closer look when I get a chance and see if there's anything we can improve.

[naanlizard]

Understood, thank you.

I’ve been experimenting and running tests in the meantime. I’ll compile what I’ve found in a few days and share it