Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eheap_alloc: Cannot allocate 4454408120 bytes of memory (of type "old_heap"). #175

Closed
hamano opened this issue Apr 23, 2014 · 15 comments

Comments

@hamano
Copy link
Contributor

commented Apr 23, 2014

I have get to crash frequently after upgrading ejabberd 13.12.
I check the crashdump, It's seems trying to allocate the huge memory in XML parsing.
I think it's problematic that VM crashing by XML input from the external.
Is there any way to limit the memory usage in XML parsing?

this is crashdump information:
Slogan: eheap_alloc: Cannot allocate 4454408120 bytes of memory (of type "old_heap").
Node name: 'ejabberd@localhost'
Crashdump created on: Tue Apr 22 18:56:37 2014
System version: Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:6:6] [async-threads:0] [kernel-poll:true]
Compiled: Sun Jan 27 18:19:34 2013
Taints: asn1rt_nif,crypto,stringprep,p1_sha,p1_yaml
Memory allocated: 7858 MB
Memory maximum:
Atoms: 19707
Processes: 5389
ETS tables: 90
Timers: 835
Funs: 2911

This is largest process information:
Initial Call: proc_lib:init_p/5
Last scheduled in for: xml:'-attrs_to_list/1-lc$^0/1-0-'/1
Registered Name: proc_lib:init_p/5
Status: Garbing (limited info)
Started: Tue Apr 22 18:47:36 2014
Parent:
Message Queue Len: 3475994
Reductions: 138774257
Program counter: 0x00007f297f505438 (xml:'-attrs_to_list/1-lc$^0/1-0-'/1 + 8)
Continuation pointer: 0x0000000000000000 (invalid)

Thank you.

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 23, 2014

No. XML parsing is not guilty here. Without looking into the full dump it's hard to say the exact reason. More likely there is a DoS attack or just some FSM queue overload.

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 23, 2014

Thank you for your reply.
Ya, I don't know which process try allocating 4G memory.
It seems XML parsing process consumed 3.5G memory, then unknown process tried 4G memory allocation.
crashdump
crashdump2

Is there a solution?

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 25, 2014

I can tell nothing by the PNGs.

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 25, 2014

Also, it's NOT xml parsing process. It's XML serialization. You have a slow sender (either C2S or S2S). This is a pretty typical situation, but you don't wan't to listen to me. You already know the answer and keep trying to get the confirmation. Better set max_fsm_queue option to 1000 or something like that and make sure you have Erlang R16Bx.

@zinid zinid closed this Apr 25, 2014

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2014

Already set max_fsm_queue: 1000 by default and already tried Erlang R16B.
Please reopen due to issue has not been solved at all.
ejabberd should limit the memory usage before the VM crash.

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 25, 2014

It is impossible to do due to many factor not depending on ejabberd anyway. The issue is to common anyway.

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2014

I understand that difficult to solve this issue.
Speaking clearly, this is a DoS vulnerability in ejabberd.
The process crash by external input is jsut a bug, but VM crash is vulnerability.
Memory usage of a process should be limited in Erlang application.
Please recognize the issue for enhance of ejabberd, I do not say solve issue immediately to you.

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 25, 2014

I clearly realize the issue for the last 8 years or so, thanks for yet another reminder.

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2014

Why did you close the issue while you realize the issue.
Please keep open until issue is solved.

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 25, 2014

I think we have 100+ issues like that in Jira.

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2014

Nothing in github.
please reopen this.

@zinid zinid reopened this Apr 25, 2014

@zinid

This comment has been minimized.

Copy link
Member

commented Apr 25, 2014

Now you should provide something more than PNGs or I close the ticket.
Because nobody here will handle such things except me.

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2014

OK, I will debug when I get a chance.
Basically, I try to fix the issue by myself.
I posted the issue for shared information.
Thank you.

@hamano

This comment has been minimized.

Copy link
Contributor Author

commented Jun 4, 2014

This issue is no longer happened.
Probably, This crash caused by recieving ton of offline message.
So, This issue was resolved by #200.
I don't care close the issue.
Thank you.

@badlop badlop closed this Jul 11, 2014

@lock

This comment has been minimized.

Copy link

commented Jun 12, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Jun 12, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
3 participants
You can’t perform that action at this time.