Elasticsearch giving error for 'max file descriptors' for deployment using docker-compose

[shaileshvaidya]

I am trying to deploy using docker-compose following instructions from https://github.com/chaoss/grimoirelab#using-docker-compose

It gives me error as,

elasticsearch_1  | [2020-10-10T22:21:24,030][INFO ][o.e.b.BootstrapChecks    ] [VKgWLoB] bound or publishing to a non-loopback address, enforcing bootstrap checks
elasticsearch_1  | ERROR: [1] bootstrap checks failed
elasticsearch_1  | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
elasticsearch_1  | [2020-10-10T22:21:24,047][INFO ][o.e.n.Node               ] [VKgWLoB] stopping ...
elasticsearch_1  | [2020-10-10T22:21:24,123][INFO ][o.e.n.Node               ] [VKgWLoB] stopped
elasticsearch_1  | [2020-10-10T22:21:24,123][INFO ][o.e.n.Node               ] [VKgWLoB] closing ...

Setup Details -

[root@ip-172-31-26-234 docker-compose]# git --version
git version 2.23.3
[root@ip-172-31-26-234 docker-compose]# docker --version
Docker version 19.03.6-ce, build 369ce74
[root@ip-172-31-26-234 docker-compose]# docker-compose --version
docker-compose version 1.27.4, build 40524192
[root@ip-172-31-26-234 docker-compose]# cat /etc/*release*
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)
cpe:2.3:o:amazon:amazon_linux:2
[root@ip-172-31-26-234 docker-compose]#

For troubleshooting I tried using docker run option in same cloned repository and it worked.

[root@ip-172-31-26-234 grimoirelab]# docker run -p 172.31.26.234:5601:5601 -v $(pwd)/default-grimoirelab-settings/projects.json:/projects.json -v $(pwd)/default-grimoirelab-settings/setup.cfg:/setup.cfg -t grimoirelab/full
Starting container: b0c329ecf740
Starting Elasticsearch
[....] Starting Elasticsearch Server:/etc/init.d/elasticsearch: line 118: ulimit: open files: cannot modify limit: Operation not permitted
. ok
Waiting for Elasticsearch to start...
tcp        0      0 0.0.0.0:9200            0.0.0.0:*               LISTEN      -
Elasticsearch started
Starting MariaDB
[ ok ] Starting MariaDB database server: mysqld.
Waiting for MariaDB to start...
tcp6       0      0 :::3306                 :::*                    LISTEN      -
MariaDB started
Starting Kibiter
Waiting for Kibiter to start...
...Kibiter started
Starting SirMordred to build a GrimoireLab dashboard
This will usually take a while...
Collection for git: starting...
Loading blacklist...
0/0 blacklist entries loaded
Loading unique identities...
Collection for git: finished after 00:00:03 hours
53/53 unique identities loaded

[StingRayZA]

As per https://github.com/chaoss/grimoirelab/tree/master/docker-compose#linux you may need to set vm.max_map_count=262144 first...

[vchrombie]

Hi @shaileshvaidya
Thanks for your interest in GrimoireLab.

I checked the logs you have provided.

elasticsearch_1  | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]

This is caused because of the low virtual memory for elasticsearch. As @StingRayZA suggested, you need to increase it before running the docker-compose solution. link

$ sysctl -w vm.max_map_count=262144

Can you check it and lets us know how it goes?

[shaileshvaidya]

Hello @vchrombie and @StingRayZA Thanks for suggestions. Sorry for delayed reply. I have updated vm.max_map_count value in my previous setup. To reconfirm I recreated setup again.

Got again same error,

elasticsearch_1  | [2020-10-24T21:28:58,876][INFO ][o.e.n.Node               ] [bSxYbFt] starting ...
elasticsearch_1  | [2020-10-24T21:28:59,165][INFO ][o.e.t.TransportService   ] [bSxYbFt] publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch_1  | [2020-10-24T21:28:59,184][INFO ][o.e.b.BootstrapChecks    ] [bSxYbFt] bound or publishing to a non-loopback address, enforcing bootstrap checks
elasticsearch_1  | ERROR: [1] bootstrap checks failed
elasticsearch_1  | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
elasticsearch_1  | [2020-10-24T21:28:59,216][INFO ][o.e.n.Node               ] [bSxYbFt] stopping ...
elasticsearch_1  | [2020-10-24T21:28:59,266][INFO ][o.e.n.Node               ] [bSxYbFt] stopped
elasticsearch_1  | [2020-10-24T21:28:59,267][INFO ][o.e.n.Node               ] [bSxYbFt] closing ...
elasticsearch_1  | [2020-10-24T21:28:59,286][INFO ][o.e.n.Node               ] [bSxYbFt] closed

vm.max_map_count is configured as expected.

[root@ip-172-31-24-150 docker-compose]# sysctl vm.max_map_count
vm.max_map_count = 262144
[root@ip-172-31-24-150 docker-compose]#

On investigation found that error is not due to memory allocation setting but due to "max file descriptors" configuration . There are recommended settings as per https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html

Modified docker-compose.yml as https://github.com/shaileshvaidya/grimoirelab/blob/master/docker-compose/docker-compose.yml#L31

After that I am able to start all services and Dashboard come up properly.

I am using AWS instance and software version's are as below,

[root@ip-172-31-24-150 ~]# docker --version
Docker version 19.03.6-ce, build 369ce74
[root@ip-172-31-24-150 ~]# docker-compose --version
docker-compose version 1.27.4, build 40524192
[root@ip-172-31-24-150 ~]#
[root@ip-172-31-24-150 ~]# docker images
REPOSITORY                                          TAG                  IMAGE ID            CREATED             SIZE
grimoirelab/hatstall                                latest               5a8f35a8f1c5        5 weeks ago         875MB
bitergia/mordred                                    latest               db35730b8a86        2 months ago        1.03GB
bitergia/kibiter                                    community-v6.8.6-3   f510594a2996        4 months ago        1.14GB
docker.elastic.co/elasticsearch/elasticsearch-oss   6.8.6                0575a0ebbd06        10 months ago       747MB
mariadb                                             10.0                 d1bde56970c6        17 months ago       353MB
[root@ip-172-31-24-150 ~]#

[vchrombie]

Thanks for the clarifications and instructions @shaileshvaidya.
Closing the issue since it seems to be resolved.