Race during initial setup while updating kibana_system user

[antoineco]

Problem description

Lately I've been encountering on several occasions what looks like a race during the stack's initial setup.

Right after waiting for Elasticsearch to become responsive, we initialize users by:

• updating the password of pre-existing (built-in) users, such as kibana_system.
• creating our own users with a corresponding role, such as logstash_internal.

Once in a full moon, the setup logic doesn't find any kibana_system user, and skips its initialization entirely, which prevents Kibana from starting.

It seems like there is a delay between the startup of the Elasticsearch API server and the moment built-in users are actually created, and we should account for it as reliably as possible (no sleep 10...).

Workaround

1. Remove the setup volume with docker volume rm docker-elk_setup.
2. Run docker-compose up setup again.

Extra information

Stack configuration

default

Docker setup

$ docker version

20.10.20

$ docker compose version

2.12.2

Container logs

docker-elk-setup-1  | -------- Wed Nov 16 21:06:04 UTC 2022 --------
docker-elk-setup-1  | [+] Waiting for availability of Elasticsearch. This can take several minutes.
docker-elk-setup-1  |    ⠿ Elasticsearch is running
docker-elk-setup-1  | [+] Role 'logstash_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1  | [+] User 'kibana_system'
docker-elk-setup-1  | [x]   No role defined, skipping creation
docker-elk-setup-1  | [+] User 'logstash_internal'
docker-elk-setup-1  |    ⠿ User does not exist, creating