Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No index information when host configured in elasticsearch_hosts is unreachable (but elastic cluster is not red) #3993

Closed
fadenb opened this issue Jul 12, 2017 · 0 comments
Assignees
Milestone

Comments

@fadenb
Copy link

@fadenb fadenb commented Jul 12, 2017

Expected Behavior

I configured elasticsearch_hosts with 8 elastic hosts:

elasticsearch_hosts = http://backend01-int:9200,http://backend02-int:9200,http://backend03-int:9200,http://backend04-int:9200,http://backend05-int:9200,http://backend06-int:9200,http://backend07-int:9200,http://backend08-int:9200

When 1 elastic node went offline (stoped elasticsearch for test purposes) graylog was unable to display the "indices" page and displayed a warning that I should check that the elastic cluster is reachable.

As the cluster was only in yellow state (and shortly after green) I expected graylog to be able to still display index information. During the whole time graylog was able to output logs and search them.

Current Behavior

An error message is displayed telling me to ensure the elastic cluster is reachable.

Possible Solution

As advised by Joschi on irc:

  • Configure only 1 elastic node per elasticsearch_hosts
  • Enable elasticsearch_discovery_enabled
  • Graylog will show connection refused message in the console but work fine
    • Log now shows a pool of available servers
io.searchbox.client.AbstractJestClient - Setting server pool to a list of 6 servers: [http://10.20.30.4:9200,http://10.20.30.8:9200,http://10.20.30.5:9200,http://10.20.30.3:9200,http://10.20.30.1:9200,http://10.20.30.2:9200]

Steps to Reproduce (for bugs)

  1. Configure multiple elastic nodes in elasticsearch_hosts
  2. Use graylog as you would usually do
  3. Stop 1 elastic server
  4. Try to open https://graylog.some.tld/system/index_sets/<some_id>
  5. See error message (graylog console: https://gist.githubusercontent.com/fadenb/6aa5b7f4bbfdcdf99e4c03a29b6e2b61/raw/b954406c496c8d14b86c358879e9b89181cece9e/graylog.log )

Context

Your Environment

  • Graylog Version: Graylog 2.3.0-rc.1+8077842
  • Elasticsearch Version: 5.4.2 build[929b078/2017-06-15T02:29:28.122Z]
  • MongoDB Version: 3.2.9
  • Operating System: NixOs 17.09
@joschi joschi added the to-verify label Jul 12, 2017
@joschi joschi self-assigned this Jul 12, 2017
@joschi joschi added this to the 2.3.0 milestone Jul 13, 2017
@joschi joschi added bug elasticsearch and removed to-verify labels Jul 13, 2017
joschi pushed a commit that referenced this issue Jul 14, 2017
Jochen Schalanda
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993
@ghost ghost added the in progress label Jul 14, 2017
@bernd bernd closed this in #4002 Jul 17, 2017
@ghost ghost removed the in progress label Jul 17, 2017
bernd added a commit that referenced this issue Jul 17, 2017
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993
joschi added a commit that referenced this issue Jul 17, 2017
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993

(cherry picked from commit 5ea9428)
bernd added a commit that referenced this issue Jul 17, 2017
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993

(cherry picked from commit 5ea9428)
mariussturm added a commit that referenced this issue Jul 18, 2017
Jest now retries failed Elasticsearch requests a configurable number of times (default: 2)
until it gives up.

Unfortunately, the retry implementations in Apache Http Client via `HttpRequestRetryHandler`
couldn't be used because they don't allow changing the request (or at least the request URI)
between retries which is necessary to use all configured servers in the server pool.

Fixes #3993
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants