eheap_alloc: Cannot allocate 1318267840 bytes of memory (of type "old_heap").

[userlocalhost]

This is a client script named sender.rb cuases RabbitMQ crashes.

#!/usr/bin/env ruby

require 'bunny'

raise 'invalid message size' unless ARGV[0] =~ /^[0-9]+$/

c = Bunny.new(:host => 'mq1', :user => 'admin', :password => 'passwd', :automatically_recover => false)
c.start

ch = c.create_channel
q = ch.queue('test-queue')

puts "(started) #{Time.now}"

begin
  loop { q.publish('a' * ARGV[0].to_i) }
rescue Bunny::NetworkFailure => _
  puts "(aborted) #{Time.now}"
rescue Exception => _
  ch.close
  c.close
end

When a client publishes a lot of messages that has small content (less than 200 bytes), RabbitMQ crashes pretty soon such like that.
But this problem has never ocurred at the case of large size messages (more than 4096 bytes) at least for 1 hour or more.

hiroyasu-ohyama@localhost:~$ ruby sender.rb 10
(started) 2016-03-10 14:12:47 +0900
E, [2016-03-10T14:13:09.663983 #11809] ERROR -- #<Bunny::Session:0x7faef3143da0 admin@mq1:5672, vhost=/, addresses=[mq1:5672]>: Got an exception when sending data: Broken pipe (Errno::EPIPE)
...
(aborted) 2016-03-10 14:13:09 +0900
hiroyasu-ohyama@localhost:~$ ruby sender.rb 50
(started) 2016-03-10 14:19:22 +0900
E, [2016-03-10T14:19:39.050927 #11838] ERROR -- #<Bunny::Session:0x7fb7b11ffde0 admin@mq1:5672, vhost=/, addresses=[mq1:5672]>: Got an exception when sending data: Broken pipe (Errno::EPIPE)
...
(aborted) 2016-03-10 14:19:39 +0900
hiroyasu-ohyama@localhost:~$ ruby sender.rb 100
(started) 2016-03-10 14:20:17 +0900
E, [2016-03-10T14:20:39.005741 #11839] ERROR -- #<Bunny::Session:0x7fe2291afa28 admin@mq1:5672, vhost=/, addresses=[mq1:5672]>: Got an exception when receiving data: Connection reset by peer (Errno::ECONNRESET)
...
(aborted) 2016-03-10 14:20:39 +0900
hiroyasu-ohyama@localhost:~$ ruby sender.rb 200
(started) 2016-03-10 14:21:30 +0900
E, [2016-03-10T14:21:47.995257 #11840] ERROR -- #<Bunny::Session:0x7fd09114ba18 admin@mq1:5672, vhost=/, addresses=[mq1:5672]>: Got an exception when sending data: Broken pipe (Errno::EPIPE)
...
(aborted) 2016-03-10 14:21:47 +0900
hiroyasu-ohyama@localhost:~$

Here is the system information where RabbitMQ runs.

### Hardware environment
* CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
* RAM: 4GB

### Software environment
* OS      : Ubuntu 14.04.4 LTS
* Erlang  : R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:64] [kernel-poll:true]
* RabbitMQ: v3.6.1-1

Following is the message which is dumped to STDERR when RabbitMQ is crashed.

Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 1318267840 bytes of memory (of type "old_heap").
Aborted (core dumped)

And here is the erl_crash.dump.

• https://gist.github.com/userlocalhost2000/53f0cafcf32ca4beb742

I'm not sure either RabbitMQ or Erlang causes this problem.

Thank you.

[michaelklishin]

RabbitMQ does not allocate memory, the runtime does.

[michaelklishin]

Lazy queues will avoid keeping messages in RAM until they need to be delivered to a consumer.

[lemenkov]

I guess this is related to PR #1302.

[divick]

@michaelklishin I see that this issue has been closed but it could be helpful, if you could mention that if runtime is unable to allocate memory then what could be the reason for that. I am running rabbitmq via its official docker image (alpine variant 3.7.7) on aws and even though there is quite some memory available to it, but still the rabbitmq got killed. What could be the reason for this and how to go about figuring it out?
Would appreciate any help.

[michaelklishin]

@divick please post your questions to rabbitmq-users.

RabbitMQ does not allocate memory directly and the core and tier 1 plugins do not have any native extensions that could do that. "old_heap" in the message suggests a single Erlang process (very likely a queue) used the vast majority of the heap.
See Reasoning About Memory Use to learn how to verify that hypothesis. With "binary" a more likely reason is memory fragmentation.

rabbitmq-users archives with the same question.