Error on first run - Nginx memory allocation

[therebelrobot]

Ubuntu 14.04 - digitalocean

After I installed Oracle Java 8, Cassandra, Lua5.1, and Kong, I run sudo kong start and I get the following error:

[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Proxy port.........8000
       Admin API port.....8001
       Database...........cassandra keepalive=60000 port=9042 timeout=1000 hosts=127.0.0.1 keyspace=kong
[INFO] Database not initialized. Running migrations...
[OK] Migrated up to: 2015-01-12-175310_init_schema
[OK] Migrated up to: 2015-04-24-154530_plugins
[WARN] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n"
nginx: [alert] mmap(MAP_ANON|MAP_SHARED, 536870912) failed (12: Cannot allocate memory)
[ERR] Could not start Kong

Any ideas how to get around that?

[montanaflynn]

You don't have enough memory. I've run into this problem before on a small 512mb digital ocean droplet and was able to lower the lua shared dict memory which solved my issue. Keep in mind Cassandra also allocates ~350mb for the JVM so you'll be cutting it pretty close if you're running both Kong and Cassandra on the same machine. Here's what you can try:

Open up /etc/kong/kong.yml

Change the line lua_shared_dict cache 512m; to lua_shared_dict cache 32m;

[nijikokun]

You should also raise the shared memory (default is 32MB on the 5$ DO Droplet):

Place an entry in /etc/sysctl.conf like:

kernel.shmmax=134217728
kernel.shmall=134217728

Which will be processed at your next reboot and set it to 128MB (128*1024*1024).

[therebelrobot]

Kk. Will try a larger droplet, and if I still run into issues, I'll try the kong.yml and the sysctl.conf fixes. Thanks!

[montanaflynn]

A larger droplet will definitely work. I just tested a brand new DO 512mb droplet, followed these instructions and set lua_shared_dict cache 32m; which resulted in Kong starting up.

@nijikokun why do you recommend upping the shared memory?

[nijikokun]

@montanaflynn to ensure everything will run smoothly throughout its lifetime.

[montanaflynn]

@nijikokun all droplet sizes (and AWS instances types) have a default of 33554432 for the shared memory when running Ubuntu 14.04, I think upping it would potentially make this issue worse by allocating more memory to something other than Kong or Cassandra. It's not something I would say is necessary to have everything run smoothly.

[sonicaghi]

we should put these things in the doc somewhere. or when we'll add DO.

[nijikokun]

@nijikokun all droplet sizes (and AWS instances types) have a default of 33554432 for the shared memory when running Ubuntu 14.04, I think upping it would potentially make this issue worse by allocating more memory to something other than Kong or Cassandra. It's not something I would say is necessary to have everything run smoothly.

At this scale it wouldn't really have much of an effect, but you should really manage your shared memory, especially if you're planning on scaling something properly, especially when dealing with databases.

https://git.kernel.org/cgit/linux/kernel/git/mhocko/mm.git/commit/include/uapi/linux/shm.h?id=060028bac94bf60a65415d1d55a359c3a17d5c31

[montanaflynn]

At this scale it wouldn't really have much of an effect, but you should really manage your shared memory, especially if you're planning on scaling something properly, especially when dealing with databases.

Sure, my response was specific to this issue in regards to running Kong with low memory constraints.

@therebelrobot I can assume this has been solved?

I think it's "solved" for @therebelrobot but let's keep the issue open until @thefosk or @thibaultcha can comment on lowering lua_shared_dict cache and any implications that may have, it might be something that should be lowered by default or as an option in the top of your kong.yml so people can run Kong on a small DO droplet.

[nijikokun]

I think it's "solved" for @therebelrobot but let's keep the issue open until @thefosk or @thibaultcha can comment on lowering lua_shared_dict cache and any implications that may have, it might be something that should be lowered by default or as an option in the top of your kong.yml so people can run Kong on a small DO droplet.

Agreed, this is why I had removed my comment, it should be set lower, perhaps 128 or 64. 512m seems extremely high.

Sure, my response was specific to this issue in regards to running Kong with low memory constraints.

I thought you meant in general my apologies.

[montanaflynn]

@augusto we will have recommendations for common setups ranging from this type of usage (development, one machine) all the way to clusters large enough to handle billions of requests a day (enterprise, multi-datacenter).

Setting system-level performance tweaks inside premade images and finding optimal performance in benchmarks will decrease server costs for everyone so it's good to have these discussions on

[therebelrobot]

I can verify this is "solved" for me, I'm fine leaving this open for discussion 👍

[subnetmarco]

I was thinking to lower the shared memory from 512 to 256 or possibly 128MB, which should still be plenty of memory for Kong to function properly.

[montanaflynn]

That would be good for a default but I think it should also be an option at the top of kong.yml like the ports so people can change it depending on their specific setup, right now there's a big scary comment above it which is essentially saying "don't change this".

[subnetmarco]

@montanaflynn are you referring to this https://github.com/Mashape/kong/blob/master/kong.yml#L14-L15 ?

[nijikokun]

I think it would be good as an option as well.

Comment about don't change this:

@thefosk http://getkong.org/docs/0.2.0-2/configuration/#nginx

[subnetmarco]

Oh okay, so regarding the ports those can be changed without changing the nginx configuration using these properties:

• http://getkong.org/docs/0.2.0-2/configuration/#proxy_port
• http://getkong.org/docs/0.2.0-2/configuration/#admin_api_port

[nijikokun]

@thefosk yes, and it would be nice to make such an option for this value as well:

https://github.com/Mashape/kong/blob/master/kong.yml#L91

[subnetmarco]

I have added a new property called memory_cache_size in the configuration, with the default value (MB) being 128 and the minimum value allowable set to 32.

[nimboya]

The first option worked thanks. Using an Ubuntu 14.04 thank @montanaflynn