New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

getFileDescriptorStats() not working with ES 5.3 #4119

Closed
maxirus opened this Issue Sep 1, 2017 · 1 comment

Comments

Projects
None yet
3 participants
@maxirus

maxirus commented Sep 1, 2017

org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats() is incompatible with ES 5.3. Specifically, the header host and option full_id=true are not supported. Additionally, the response back is in text, NOT json as expected. Must pass format=json with no headers to receive a JSON response. See https://www.elastic.co/guide/en/elasticsearch/reference/5.3/cat-nodes.html

Here is the error I see in the logs:

org.graylog2.indexer.ElasticsearchException: Unable to read Elasticsearch node information
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Cluster.catNodes(Cluster.java:119) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats(Cluster.java:124) ~[graylog.jar:?]
	at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:56) ~[graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.io.IOException: Request GET http://localhost:9200/_cat/nodes?h=name%2Chost%2CfileDescriptorMax&full_id=true HTTP/1.1 yielded text/plain;charset=UTF-8, should be json: HTTP/1.1 200 OK
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:224) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
	... 12 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('T' (code 84)): Expected space separating root-level values
 at [Source: 2Tq4m8h 128000
; line: 1, column: 3]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportMissingRootWS(ParserMinimalBase.java:503) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._verifyRootSpace(ReaderBasedJsonParser.java:1668) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parsePosNumber(ReaderBasedJsonParser.java:1311) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:746) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:53) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:211) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
	... 12 more

My setup is as follows:

Host A:

  • CentOS 7.3.1611
  • Docker v1.13.1
  • Docker Container(s):
    • MongoDB v3.4.7

Host B:

AWS ElasticSearch Service v5.3

More debug info:

[root@Host_B /]# curl -XGET http://localhost:9200/_cat/nodes?format=json
[{"ip":"x.x.x.x","heap.percent":"8","ram.percent":"98","cpu":"0","load_1m":"0.01","load_5m":"0.02","load_15m":"0.05","node.role":"mdi","master":"*","name":"2Tq4m8h"}]
[root@Host_B /]# curl -XGET http://localhost:9200/_cat/nodes?h=name%2Chost%2CfileDescriptorMax&full_id=true
2Tq4m8h 128000

server.conf (removed comments)

[root@3da50b4bf137 /]# cat /opt/graylog-server/conf/server.conf 

is_master = true
node_id_file = /opt/graylog-server/node-id
password_secret = <password_secret_here>
root_password_sha2 = <SHA256_Hash_here>
plugin_dir = /opt/graylog-server/plugin
rest_listen_uri = http://0.0.0.0:9000/api/
rest_transport_uri = http://<Host_B_ip>:9000/api/
web_listen_uri = http://0.0.0.0:9000/
web_endpoint_uri = http://<Host_B_ip>:9000/
elasticsearch_hosts = http://localhost:9200
elasticsearch_discovery_enabled = false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /opt/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://<Host_A>/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
rules_file = /opt/graylog-server/rules.drl
content_packs_dir = /opt/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Let me know if there is any more info I can provide.

final JsonNode nodes = catNodes("name", "host", "fileDescriptorMax");

@jalogisch jalogisch added the to-verify label Sep 5, 2017

@joschi

This comment has been minimized.

Contributor

joschi commented Sep 6, 2017

@maxirus The cat API works perfectly fine with Elasticsearch 5.3.x as being used right now, so I guess it's aws-es-proxy or the AWS Elasticsearch Service which isn't 100% compatible with Elasticsearch 5.x.

Example with Elasticsearch 5.3.0:

$ curl http://localhost:9200/?pretty
{
  "name" : "NztkI3W",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "xZGewbmZRwScOZAtqmNXQA",
  "version" : {
    "number" : "5.3.0",
    "build_hash" : "3adb13b",
    "build_date" : "2017-03-23T03:31:50.652Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}

# Use HTTP "Accept" header to indicate that the response should be JSON - this is what Graylog 2.3.1 does.
$ curl -H 'Accept: application/json' http://localhost:9200/_cat/nodes?pretty
[
  {
    "ip" : "127.0.0.1",
    "heap.percent" : "14",
    "ram.percent" : "98",
    "cpu" : "14",
    "load_1m" : "2.66",
    "load_5m" : null,
    "load_15m" : null,
    "node.role" : "mdi",
    "master" : "*",
    "name" : "NztkI3W"
  }
]

# Use "format" query parameter to indicate that the response should be JSON 
$ curl 'http://localhost:9200/_cat/nodes?format=json&pretty'
[
  {
    "ip" : "127.0.0.1",
    "heap.percent" : "15",
    "ram.percent" : "99",
    "cpu" : "5",
    "load_1m" : "2.01",
    "load_5m" : null,
    "load_15m" : null,
    "node.role" : "mdi",
    "master" : "*",
    "name" : "NztkI3W"
  }
]

# Use "full_id" query parameter to receive the full node ID
$ curl 'http://localhost:9200/_cat/nodes?format=json&h=id,name&full_id=true&pretty'
[
  {
    "id" : "NztkI3WkSIKbwJzvP00V4w",
    "name" : "NztkI3W"
  }
]

This being said, we'll try to make it easier by using the format query parameter additionally to the "Accept" HTTP header in the next version of Graylog.

joschi added a commit that referenced this issue Sep 6, 2017

Add "format" query parameter to Elasticsearch cat request
While using the "Accept" HTTP request header should be perfectly sufficient,
some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem
to support it.

This change set adds the alternative "format" query parameter to the cat request
in order to satisfy these clones.

Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor

Closes #4119

@joschi joschi self-assigned this Sep 6, 2017

@wafflebot wafflebot bot added the in progress label Sep 6, 2017

@joschi joschi added this to the 2.3.2 milestone Sep 6, 2017

joschi added a commit that referenced this issue Sep 19, 2017

Add "format" query parameter to Elasticsearch cat request
While using the "Accept" HTTP request header should be perfectly sufficient,
some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem
to support it.

This change set adds the alternative "format" query parameter to the cat request
in order to satisfy these clones.

Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor

Closes #4119

@bernd bernd closed this in #4127 Sep 19, 2017

@wafflebot wafflebot bot removed the in progress label Sep 19, 2017

bernd added a commit that referenced this issue Sep 19, 2017

Add "format" query parameter to Elasticsearch cat request (#4127)
* Add "format" query parameter to Elasticsearch cat request

While using the "Accept" HTTP request header should be perfectly sufficient,
some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem
to support it.

This change set adds the alternative "format" query parameter to the cat request
in order to satisfy these clones.

Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor

Closes #4119

* Use IP address as fallback for hostname in file descriptor stats

If the hostname is empty or cannot be identified, use the IP address
of the Elasticsearch node as fallback.

* Add request timeout for Cluster#clusterHealth()

* Add integration test for Cluster class

* Migrate to new Elasticsearch integration tests

* Hopefully fix ClusterIT

@joschi joschi modified the milestones: 2.3.2, 2.4.0 Oct 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment