Arango DB not picking the memory mappings from sysctl vm parameters

[SharadUnifyed]

my environment running ArangoDB

I'm using the latest ArangoDB of the respective release series:
[ ] 3.3
Mode:
[ ] Single-Server

Storage-Engine:

• rocksdb

On this operating system:

• DCOS on
  • own infrastructure
• Linux
  • Ubuntu .deb

Hi Team,
We are evaluating ArangoDB for one of our requirements. We are facing an issue related to memory mappings. We have set the syscntl vm parameter -vm.max_map_count=1024000 but the same is not getting reflected while running the ArangoDB instance.

We are setting syscntl parameters as below
vm.max_map_count=1024000
fs.file-max=500000

Please find the ArangoDB logs as below

2018-06-14T09:41:34Z [32753] INFO detected operating system: Linux version 4.4.0-1052-aws (buildd@lgw01-amd64-031) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018
2018-06-14T09:41:34Z [32753] INFO using storage engine rocksdb
2018-06-14T09:41:34Z [32753] INFO {cluster} Starting up with role SINGLE
2018-06-14T09:41:34Z [32753] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2018-06-14T09:41:34Z [32753] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2018-06-14T09:41:34Z [32753] INFO using endpoint 'http+ssl://10.150.1.9:8530' for ssl-encrypted requests
2018-06-14T09:41:34Z [32753] INFO using endpoint 'http+tcp://10.150.1.9:8529' for non-encrypted requests
2018-06-14T09:41:35Z [32753] INFO Please note that a new bugfix version '3.3.10' is available
2018-06-14T09:41:35Z [32753] INFO ArangoDB (version 3.3.3 enterprise [linux]) is ready for business. Have fun!
2018-06-18T08:56:39Z [18243] ERROR {memory} process is near the maximum number of memory mappings. current: 127319, maximum: 128000. it may be sensible to increase the maximum number of mappings per process
2018-06-18T09:08:29Z [18243] ERROR {memory} process is near the maximum number of memory mappings. current: 127360, maximum: 128000. it may be sensible to increase the maximum number of mappings per process
2018-06-18T09:55:13Z [18243] ERROR {communication} unable to perform ssl handshake: Connection reset by peer : 104
2018-06-18T09:57:50Z [18243] ERROR {communication} unable to perform ssl handshake: Connection reset by peer : 104
2018-06-18T10:05:10Z [18243] ERROR {communication} unable to perform ssl handshake: malloc failure : 117850177
2018-06-18T10:05:11Z [18243] ERROR {communication} unable to perform ssl handshake: malloc failure : 117850177
2018-06-18T10:05:11Z [18243] ERROR {communication} unable to perform ssl handshake: malloc failure : 117850177
2018-06-18T10:05:11Z [18243] ERROR {communication} unable to perform ssl handshake: malloc failure : 117850177
2018-06-18T10:05:11Z [18243] ERROR {communication} unable to perform ssl handshake: malloc failure : 117850177
2018-06-18T10:05:20Z [18243] ERROR {communication} unable to perform ssl handshake: malloc failure : 336183361
2018-06-18T10:05:23Z [18243] ERROR {communication} unable to perform ssl handshake: Connection reset by peer : 104
2018-06-18T10:05:23Z [18243] ERROR {threads} scheduler loop caught exception: out of memory (exception location: /var/lib/jenkins/workspace/RELEASE__BuildPackages/lib/Basics/StringBuffer.h:263)

Please find the list of all sysctl vm parameters as below

vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256 256 32 1
vm.max_map_count = 1024000
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 2
vm.overcommit_ratio = 125
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.zone_reclaim_mode = 0

Please help resolve the issue.

[jsteemann]

ArangoDB will simply read the value of /proc/sys/vm/max_map_count on start.

ArangoDB will only pick up updated sysctl parameters after a restart of ArangoDB. It will not help if sysctl parameters are changed after ArangoDB was already started.

Apart from that, have the sysctl parameter changes been made permanent, or will they change back to the original values after a restart of the system?

[SharadUnifyed]

Thanks for the quick response Jan.
ArangoDB instance is restarted post sysctl parameter update. Also, sysctl parameter changes been made permanent.

[jsteemann]

Ok.
I just double-checked the code. The value is definitely read from /proc/sys/vm/max_map_count at arangod start. If it reads 128,000, then the arangod process will have seen a value of 128000 in that file.
Which by the way seems like an adjusted value already, because the default value on most Linux kernels is around 65535 (or slightly lower).
Do sudo sysctl -a | grep max_map and cat /proc/sys/vm/max_map_count also show 128000 on the same server arangod is running on?
If so, can you check in /etc/sysctl.d if there are any further places in your configuration that override the max_map_count value?

[SharadUnifyed]

Both "sudo sysctl -a | grep max_map" and "cat /proc/sys/vm/max_map_count" show 1024000 not 128000.

[jsteemann]

Ok, the code in ArangoDB 3.3.3 (and all other versions) only reads from file /proc/sys/vm/max_map_count:

    std::string value =
        basics::FileUtils::slurp("/proc/sys/vm/max_map_count");

    maxMappings = basics::StringUtils::uint64(value);

And it works fine whatever value I try.
So I guess there must be something wrong in the setup, so that arangod has either not been restarted or is running on a different server or something else.

[jsteemann]

If you think the new (raised) value is in effect, you may also turn off arangod reading the max_map_count value by passing the startup option --server.check-max-memory-mappings false to arangod (or putting the option into arangod's conf file).

[maxkernbach]

Hi @SharadUnifyed,

how are things going? Did you find an error in your setup or does the problem still persist?

[OmarAyo]

Hi @SharadUnifyed

Just sending a quick followup - Could you please send us a feedback.
Are we all set with this ticket or any further help is required here ?

Thanks

[OmarAyo]

Hi @SharadUnifyed

We haven't heard from you recently, I am going ahead and close this ticket.

Note: tickets can be reopened at anytime by adding a new comment

Best,