error registering kibana privileges with elasticsearch for kibana-.kibana - POST /_security/privilege hangs

[dlukyanov]

Elasticsearch Version

8.9.0

Installed Plugins

No response

Java Version

bundled

OS Version

Linux b3865cfa1be4 5.15.49-linuxkit-pr #1 SMP Thu May 25 07:17:40 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

initial problem:
following the instruction to run elastic+kibana in docker
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_prepare_the_environment

however after startup kibana reports a lot of errors like:
error registering kibana privileges with elasticsearch for kibana-.kibana: request timed out

after debugging i minimized the steps to reproduce to following

Steps to Reproduce

create file docker-compose.yaml:

version: '3'
name: elastic

services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.0
    volumes:
      - ./data_elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    environment:
      - cluster.routing.allocation.disk.threshold_enabled=false
      - node.name=elasticsearch
      - cluster.name=docker-cluster
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=elastic

create the folder data_elasticsearch with just one file body-502.json.gz (extract it)

it's a real content kibana sends to elastic

now after starting elasticsearch try to call following command from elascticsearch container/terminal:

curl -v -X POST -u elastic:elastic http://localhost:9200/_security/privilege -H "Expect:" -H "Content-Type: application/json" --data @$HOME/data/body-502.json

as a result curl just hangs

I believe it's a source of the issue error registering kibana privileges with elasticsearch for kibana-.kibana

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[slobodanadamovic]

@dlukyanov I haven't been able to reproduce this issue. I highly suspect that the problem is with a slow hardware resources.
With docker configured to use 1 CPU and 4GB of RAM, the request on my local machine was the slowest and took ~16s to execute. Downloading Elasticsearch 8.9.0 and running the command locally takes around 2s to execute.

The default Kibana's request timeout is 30s. So the request to configure application privileges must be taking longer than 30s on your machine?

[dlukyanov]

i have 4gb memory and 4cpu.
tried to set cpu to 1 - the same result - curl just waiting for response without success... no timeout. nothing...
the same request with es3.17.10 returns result in ~15 seconds on the same docker configuration.

with kibana timeout it's clear. but the problem that i can't get response even in minutes with curl.

[dlukyanov]

tested it without docker - response is coming in 2 seconds.

[dlukyanov]

after setting org.elasticsearch=debug inside docker i see the following in logs during POST /_security/privilege :

2023-08-21T15:54:05,455 DEBUG [o.e.i.e.I.EngineMergeScheduler] [.security-7][0]  merge segment [_75] done: took [25s], [0.6 MB], [140 docs], [0s stopped], [0s throttled]
2023-08-21T15:56:54,337 DEBUG [o.e.i.e.I.EngineMergeScheduler] [.security-7][0]  merge segment [_7f] done: took [27.4s], [0.1 MB], [10 docs], [0s stopped], [0s throttled]
2023-08-21T16:00:56,879 DEBUG [o.e.i.e.I.EngineMergeScheduler] [.security-7][0]  merge segment [_7p] done: took [46.2s], [0.1 MB], [19 docs], [0s stopped], [0s throttled]
2023-08-21T16:05:26,847 DEBUG [o.e.i.e.I.EngineMergeScheduler] [.security-7][0]  merge segment [_7z] done: took [29.7s], [0.3 MB], [143 docs], [0s stopped], [0s throttled]

so, probably it's doing something but veeery slow when inside the docker...

[dlukyanov]

interesting. removing external drive mapping for /usr/share/elasticsearch/data solved the issue.
something is slowing down the file access on my system. (antivirus?)

we can close this issue. sorry for disturbing.

[dlukyanov]

Just to summarize an issue. I was running docker on windows machine in hyper-v mode.
It's a well known issue that volume binding in this mode is very slow.
I switched to WSL2-based docker engine and elastic is running well now.

[mtovmassian]

I encountered the same error while connecting to Kibana for the first time (leading to a 504 Gateway Timeout).
And as noticed by @slobodanadamovic it was a hardware limitation issue.
My initial configuration was based on 2 vCPUs. I could make it work by increasing it to 4 vCPUs.