Cannot allocate memory on elkstack::single

[jujugrrr]

Hi,
Some tests were failing while trying to integrate ELKstack::single recipe. The node was not able to converge due to a lack of memory preventing to install a simple nginx package.

    Recipe: nginx::package
         * package[nginx] action install[2014-09-30T08:25:11+00:00] INFO: Processing package[nginx] action install (nginx::package line 40)

           Error executing action `install` on resource 'package[nginx]'

           Errno::ENOMEM
           -------------
           Cannot allocate memory - fork(2)

           Resource Declaration:
           ---------------------

           # In /tmp/kitchen/cache/cookbooks/nginx/recipes/package.rb

            40: package node['nginx']['package_name'] do
            41:   options package_install_opts

The stack is integrated like that:

include_recipe 'java'
include_recipe 'elkstack::single'

Tests were done on a Rackspace performance1-2 flavor instance running only the recipe above. It works well after the second chef-run so I suspect we do something consuming too much memory during the first run?

A work around for now has been to reduce ES memory usage :

+es_mem = (node['memory']['total'].to_i * 0.2).floor / 1024
+default['elasticsearch']['allocated_memory'] = "#{es_mem}m"

[martinb3]

20% (400mb) of performance1-2 heap for Elasticsearch is very, very small. Could you mention which cloud image you used and/or point to the recipe you were using (if you did it in a wrapper)? This may be specific to a particular cloud image.

Note that the defaults are 20% logstash heap, 40% elasticsearch heap, so that still leaves 40% of the box free. I wonder what else is running that is consuming 40% of the system memory.

[jujugrrr]

I've changed the workaround so it's applied to test-kitchen only, I've sent you more details in private (irc)

[jujugrrr]

I wonder what else is running that is consuming 40% of the system memory.

I don't know, I've tried to monitor memory while the chef-run was happening but I've not spotted anything except chef processes.By example the total memory used was 1500MB/2000MB after ES and logstash finished to start, but still the chef-run stopped before the end due to a lack of memory, I've not been able to see what was filling the last 500MB.

[martinb3]

I'm not sure what to do next. I don't like the workaround since we really should be using more than 500mb on a 2gb server, but perhaps 2gb of RAM is too small for the stack if chef-client eats so much memory it won't converge.