Memory full leads to crash

[x1p]

After about 16 hours of playtime, waterfall crashed
Config:
Shared host server
1536 mb ram
about 50 unique people joining a day

Over time the memory fills to the max and eventually (not immediately) it leads to a crash.

Tested with BungeeCord but this doesn't seem to crash. It fills its mem to the max and nothing happens. I can imagine that Waterfall uses more Heap mem than BC. Since it's a shared host, I cannot see the overhead memory. But if I want to convince the hosting, I need some evidence what is the problem here. They didn't support waterfall in the first place. Most people use BC. Now I would love to run Waterfall. But this bug only makes the hosting company only more afraid of Waterfall.

Log is just truncated. Ends without warning. Probably an out of memory message which does not get logged. Tested on a test server with less memory to speed up the proces.
https://pastebin.com/kV4nnWVF

Plugins:
BungeeControlBridge, CommandControl, Litebans, Luckperms, Slashserver, Staffchat

[electronicboy]

We can't really provide any form of support around memory leaks without any heap dumps, etc; Your Xmx is 512M, what is the memory limit in the panel?
given that it looks like the OOM killer is what killed it, it becomes more complex, as this means that we didn't run the heap out of memory, but either the host machine ran out of memory and killed a process to carry on, or your waterfall instance is somehow leaking off-heap memory (can you replicate this without plugins, for example)

[x1p]

The startup command is java -Xms128M -Xmx512M -jar Waterfall.jar
I presume the host is reserving some over head mem. I can't know how much.
I can ask for there support to see what their dump says. I don't have crash dumps. It's not made by waterfall that I can see.
I can try without plugins. I already minimized it. But I will do some tests without any.

[x1p]

Same result with the bare minimum of litebans, luckperms and slashserver.

If I give him more memory, it fills to the max. Even 1024 mb gets full with only 2 players. The memory use only goes up, and never down.

It seems to halt at approx 2,5 gig. But that's almost insane for a bungee

[x1p]

Tested without any plugins. Same result. After numerous rejoins, it crashes.

/entrypoint.sh: line 15: 65 Killed java -Xms128M -Xmx512M -jar Waterfall.jar
(Daemon) ---------- This server has crashed! ----------
(Daemon) The server has run out of memory. Please consider upgrading your server package or removing plugins/addons if applicable.
(Daemon) Exit Code: 137

Any suggestions how to determine the possible source are always welcome. Otherwise I have no option left than to return to BungeeCord.

[x1p]

Noticed furthermore that over the day, BC can lower the amount of memory if not needed. Where as Waterfall can't seem to unload memory and can only use more memory.

I did tests with the same plugins or even none.
No java flags were used.

[Janmm14]

I suggest you to start waterfall testwise with -Dio.netty.leakDetectionLevel=advanced
You bungee WILL run slower with that. Log will then likely contain errors starting like

ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected.

With this, debugging can go on.

[x1p]

I will contact the hosting to use the flag and give the feedback the results. Will take some time...

[x1p]

I'm not getting any clues.

/entrypoint.sh: line 15: 65 Killed java -Xms128M -Xmx1024M -jar Waterfall.jar -Dio.netty.leakDetectionLevel=advanced
(Daemon) ---------- This server has crashed! ----------
(Daemon) The server has run out of memory. Please consider upgrading your server package or removing plugins/addons if applicable.
(Daemon) Exit Code: 137
(Daemon) Server marked as OFF
(Daemon) Server detected as crashed! Attempting to reboot server now.
(Daemon) Server marked as STARTING

I'm getting 1024mb full with only 2 players. Where Bungecord is running for about 12 hours with several people joining, and only takes about 900mb and does not rise enormously like Waterfall does is my case.

[electronicboy]

All we know from that is that the process was forcefully killed by the system as it hit the memory limit, shamefully, without any form of useful debug information, there is pretty much nothing for us to see here

[x1p]

Between BC and Waterfall there are no differences in config setting. So I can only conclude that the hosting changed some settings how they manage the memory. I don't think I can get this solved (soon). For my production server, I returned to Bungeecord. For this experiment with Waterfall has not worked out for me.

[x1p]

Some more details about the memory behavior in addition to our conversation.

After about 12 hours, players come and go, BC stabilizes about 1000 mb of ram. Under the same conditions WF rises quickly with every connecting player and stabilizes around 2500 mb if run without a limit. WF consumes more mem and seems not to lower the memory needs.
With a limit, this rise in memory continues until pterodactyl kills the process. Hosting claims to have some overhead mem for every docker container. I assume WF needs more reserved memory than allocated.

I find it curious that WF allocates more memory at every connection and can't lower its memory or reuse this memory.
If WF needs more reserved memory, or should use a lower limit than the hard docker limit, than I'm interested in any details to provide my hosting with information.

[Janmm14]

this looks like there is a bug in waterfall maybe, but it seems that the setup you have (shared hosting) cannot give you the flexibility of providing in-depth diagnosis. starting with the startup flag and going on with eventual furher requests to create a jvm heap dump.

I guess you might want to stick to bungeecord until there is someone else experiencing your problem, but running his wf on bare metal or a vserver.

[x1p]

Update here. In collaboration with the hosting provider we did a test running a container with 512 MB ram with 512 MB on top of that reserved for overhead. It still crashed. This proves to me that the memory of WF is very thirsty or has insufficient control over the amount of memory although a leak could not be proven.

[astei]

Java does not cooperate that well with Docker. We have experienced a lot of OOM issues on our network and we managed to fix our issues with our proxy by allocating twice the proxy heap uses in overhead.

The primary reason is because aside from JVM overhead you need to also have room for direct buffers, which can get big.

One startup option I would recommend is -Dio.netty.allocator.maxOrder=9. By default Netty pools up to 16MiB buffers and Minecraft only sends packets 2MiB in size, maximum. This option will bring it down to 4MiB and should provide some breathing room if you don't want to increase your memory allocations.