eheap_alloc: Cannot allocate 4454408120 bytes of memory (of type "old_heap").

[hamano]

I have get to crash frequently after upgrading ejabberd 13.12.
I check the crashdump, It's seems trying to allocate the huge memory in XML parsing.
I think it's problematic that VM crashing by XML input from the external.
Is there any way to limit the memory usage in XML parsing?

this is crashdump information:
Slogan: eheap_alloc: Cannot allocate 4454408120 bytes of memory (of type "old_heap").
Node name: 'ejabberd@localhost'
Crashdump created on: Tue Apr 22 18:56:37 2014
System version: Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:6:6] [async-threads:0] [kernel-poll:true]
Compiled: Sun Jan 27 18:19:34 2013
Taints: asn1rt_nif,crypto,stringprep,p1_sha,p1_yaml
Memory allocated: 7858 MB
Memory maximum:
Atoms: 19707
Processes: 5389
ETS tables: 90
Timers: 835
Funs: 2911

This is largest process information:
Initial Call: proc_lib:init_p/5
Last scheduled in for: xml:'-attrs_to_list/1-lc$^0/1-0-'/1
Registered Name: proc_lib:init_p/5
Status: Garbing (limited info)
Started: Tue Apr 22 18:47:36 2014
Parent:
Message Queue Len: 3475994
Reductions: 138774257
Program counter: 0x00007f297f505438 (xml:'-attrs_to_list/1-lc$^0/1-0-'/1 + 8)
Continuation pointer: 0x0000000000000000 (invalid)

Thank you.

[zinid]

No. XML parsing is not guilty here. Without looking into the full dump it's hard to say the exact reason. More likely there is a DoS attack or just some FSM queue overload.

[hamano]

Thank you for your reply.
Ya, I don't know which process try allocating 4G memory.
It seems XML parsing process consumed 3.5G memory, then unknown process tried 4G memory allocation.

Is there a solution?

[zinid]

I can tell nothing by the PNGs.

[zinid]

Also, it's NOT xml parsing process. It's XML serialization. You have a slow sender (either C2S or S2S). This is a pretty typical situation, but you don't wan't to listen to me. You already know the answer and keep trying to get the confirmation. Better set max_fsm_queue option to 1000 or something like that and make sure you have Erlang R16Bx.

[hamano]

Already set max_fsm_queue: 1000 by default and already tried Erlang R16B.
Please reopen due to issue has not been solved at all.
ejabberd should limit the memory usage before the VM crash.

[zinid]

It is impossible to do due to many factor not depending on ejabberd anyway. The issue is to common anyway.

[hamano]

I understand that difficult to solve this issue.
Speaking clearly, this is a DoS vulnerability in ejabberd.
The process crash by external input is jsut a bug, but VM crash is vulnerability.
Memory usage of a process should be limited in Erlang application.
Please recognize the issue for enhance of ejabberd, I do not say solve issue immediately to you.

[zinid]

I clearly realize the issue for the last 8 years or so, thanks for yet another reminder.

[hamano]

Why did you close the issue while you realize the issue.
Please keep open until issue is solved.

[zinid]

I think we have 100+ issues like that in Jira.

[hamano]

Nothing in github.
please reopen this.

[zinid]

Now you should provide something more than PNGs or I close the ticket.
Because nobody here will handle such things except me.

[hamano]

OK, I will debug when I get a chance.
Basically, I try to fix the issue by myself.
I posted the issue for shared information.
Thank you.