New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No index information when host configured in elasticsearch_hosts is unreachable (but elastic cluster is not red) #3993

Closed
fadenb opened this Issue Jul 12, 2017 · 0 comments

Comments

Projects
None yet
2 participants
@fadenb

fadenb commented Jul 12, 2017

Expected Behavior

I configured elasticsearch_hosts with 8 elastic hosts:

elasticsearch_hosts = http://backend01-int:9200,http://backend02-int:9200,http://backend03-int:9200,http://backend04-int:9200,http://backend05-int:9200,http://backend06-int:9200,http://backend07-int:9200,http://backend08-int:9200

When 1 elastic node went offline (stoped elasticsearch for test purposes) graylog was unable to display the "indices" page and displayed a warning that I should check that the elastic cluster is reachable.

As the cluster was only in yellow state (and shortly after green) I expected graylog to be able to still display index information. During the whole time graylog was able to output logs and search them.

Current Behavior

An error message is displayed telling me to ensure the elastic cluster is reachable.

Possible Solution

As advised by Joschi on irc:

  • Configure only 1 elastic node per elasticsearch_hosts
  • Enable elasticsearch_discovery_enabled
  • Graylog will show connection refused message in the console but work fine
    • Log now shows a pool of available servers
io.searchbox.client.AbstractJestClient - Setting server pool to a list of 6 servers: [http://10.20.30.4:9200,http://10.20.30.8:9200,http://10.20.30.5:9200,http://10.20.30.3:9200,http://10.20.30.1:9200,http://10.20.30.2:9200]

Steps to Reproduce (for bugs)

  1. Configure multiple elastic nodes in elasticsearch_hosts
  2. Use graylog as you would usually do
  3. Stop 1 elastic server
  4. Try to open https://graylog.some.tld/system/index_sets/<some_id>
  5. See error message (graylog console: https://gist.githubusercontent.com/fadenb/6aa5b7f4bbfdcdf99e4c03a29b6e2b61/raw/b954406c496c8d14b86c358879e9b89181cece9e/graylog.log )

Context

Your Environment

  • Graylog Version: Graylog 2.3.0-rc.1+8077842
  • Elasticsearch Version: 5.4.2 build[929b078/2017-06-15T02:29:28.122Z]
  • MongoDB Version: 3.2.9
  • Operating System: NixOs 17.09

@joschi joschi added the to-verify label Jul 12, 2017

@joschi joschi self-assigned this Jul 12, 2017

@joschi joschi added this to the 2.3.0 milestone Jul 13, 2017

@joschi joschi added bug elasticsearch and removed to-verify labels Jul 13, 2017

joschi added a commit that referenced this issue Jul 14, 2017

Add retry handling for failed Elasticsearch requests
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993

@wafflebot wafflebot bot added the in progress label Jul 14, 2017

@bernd bernd closed this in #4002 Jul 17, 2017

@wafflebot wafflebot bot removed the in progress label Jul 17, 2017

bernd added a commit that referenced this issue Jul 17, 2017

Add retry handling for failed Elasticsearch requests (#4002)
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993

joschi added a commit that referenced this issue Jul 17, 2017

Add retry handling for failed Elasticsearch requests
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993

(cherry picked from commit 5ea9428)

bernd added a commit that referenced this issue Jul 17, 2017

Add retry handling for failed Elasticsearch requests (#4012)
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993

(cherry picked from commit 5ea9428)

mariussturm added a commit that referenced this issue Jul 18, 2017

Add retry handling for failed Elasticsearch requests (#4002)
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment