getFileDescriptorStats() not working with ES 5.3 #4119

maxirus · 2017-09-01T20:51:28Z

org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats() is incompatible with ES 5.3. Specifically, the header host and option full_id=true are not supported. Additionally, the response back is in text, NOT json as expected. Must pass format=json with no headers to receive a JSON response. See https://www.elastic.co/guide/en/elasticsearch/reference/5.3/cat-nodes.html

Here is the error I see in the logs:

org.graylog2.indexer.ElasticsearchException: Unable to read Elasticsearch node information
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Cluster.catNodes(Cluster.java:119) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats(Cluster.java:124) ~[graylog.jar:?]
	at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:56) ~[graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.io.IOException: Request GET http://localhost:9200/_cat/nodes?h=name%2Chost%2CfileDescriptorMax&full_id=true HTTP/1.1 yielded text/plain;charset=UTF-8, should be json: HTTP/1.1 200 OK
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:224) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
	... 12 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('T' (code 84)): Expected space separating root-level values
 at [Source: 2Tq4m8h 128000
; line: 1, column: 3]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportMissingRootWS(ParserMinimalBase.java:503) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._verifyRootSpace(ReaderBasedJsonParser.java:1668) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parsePosNumber(ReaderBasedJsonParser.java:1311) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:746) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:53) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:211) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
	... 12 more

My setup is as follows:

Host A:

CentOS 7.3.1611
Docker v1.13.1
Docker Container(s):
- MongoDB v3.4.7

Host B:

CentOS 7.3.1611
Docker v1.13.1
Docker Container(s):
- Graylog v2.3 & aws-es-proxy v0.3 (https://github.com/abutaha/aws-es-proxy/releases/download/v0.3/aws-es-proxy-0.3-linux-amd64)

AWS ElasticSearch Service v5.3

More debug info:

[root@Host_B /]# curl -XGET http://localhost:9200/_cat/nodes?format=json
[{"ip":"x.x.x.x","heap.percent":"8","ram.percent":"98","cpu":"0","load_1m":"0.01","load_5m":"0.02","load_15m":"0.05","node.role":"mdi","master":"*","name":"2Tq4m8h"}]

[root@Host_B /]# curl -XGET http://localhost:9200/_cat/nodes?h=name%2Chost%2CfileDescriptorMax&full_id=true
2Tq4m8h 128000

server.conf (removed comments)

[root@3da50b4bf137 /]# cat /opt/graylog-server/conf/server.conf 

is_master = true
node_id_file = /opt/graylog-server/node-id
password_secret = <password_secret_here>
root_password_sha2 = <SHA256_Hash_here>
plugin_dir = /opt/graylog-server/plugin
rest_listen_uri = http://0.0.0.0:9000/api/
rest_transport_uri = http://<Host_B_ip>:9000/api/
web_listen_uri = http://0.0.0.0:9000/
web_endpoint_uri = http://<Host_B_ip>:9000/
elasticsearch_hosts = http://localhost:9200
elasticsearch_discovery_enabled = false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /opt/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://<Host_A>/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
rules_file = /opt/graylog-server/rules.drl
content_packs_dir = /opt/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Let me know if there is any more info I can provide.

graylog2-server/graylog2-server/src/main/java/org/graylog2/indexer/cluster/Cluster.java

Line 124 in 6d98f00

final JsonNode nodes = catNodes("name", "host", "fileDescriptorMax");

The text was updated successfully, but these errors were encountered:

joschi · 2017-09-06T13:53:49Z

@maxirus The cat API works perfectly fine with Elasticsearch 5.3.x as being used right now, so I guess it's aws-es-proxy or the AWS Elasticsearch Service which isn't 100% compatible with Elasticsearch 5.x.

Example with Elasticsearch 5.3.0:

$ curl http://localhost:9200/?pretty
{
  "name" : "NztkI3W",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "xZGewbmZRwScOZAtqmNXQA",
  "version" : {
    "number" : "5.3.0",
    "build_hash" : "3adb13b",
    "build_date" : "2017-03-23T03:31:50.652Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}

# Use HTTP "Accept" header to indicate that the response should be JSON - this is what Graylog 2.3.1 does.
$ curl -H 'Accept: application/json' http://localhost:9200/_cat/nodes?pretty
[
  {
    "ip" : "127.0.0.1",
    "heap.percent" : "14",
    "ram.percent" : "98",
    "cpu" : "14",
    "load_1m" : "2.66",
    "load_5m" : null,
    "load_15m" : null,
    "node.role" : "mdi",
    "master" : "*",
    "name" : "NztkI3W"
  }
]

# Use "format" query parameter to indicate that the response should be JSON 
$ curl 'http://localhost:9200/_cat/nodes?format=json&pretty'
[
  {
    "ip" : "127.0.0.1",
    "heap.percent" : "15",
    "ram.percent" : "99",
    "cpu" : "5",
    "load_1m" : "2.01",
    "load_5m" : null,
    "load_15m" : null,
    "node.role" : "mdi",
    "master" : "*",
    "name" : "NztkI3W"
  }
]

# Use "full_id" query parameter to receive the full node ID
$ curl 'http://localhost:9200/_cat/nodes?format=json&h=id,name&full_id=true&pretty'
[
  {
    "id" : "NztkI3WkSIKbwJzvP00V4w",
    "name" : "NztkI3W"
  }
]

This being said, we'll try to make it easier by using the format query parameter additionally to the "Accept" HTTP header in the next version of Graylog.

While using the "Accept" HTTP request header should be perfectly sufficient, some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem to support it. This change set adds the alternative "format" query parameter to the cat request in order to satisfy these clones. Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor Closes #4119

* Add "format" query parameter to Elasticsearch cat request While using the "Accept" HTTP request header should be perfectly sufficient, some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem to support it. This change set adds the alternative "format" query parameter to the cat request in order to satisfy these clones. Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor Closes #4119 * Use IP address as fallback for hostname in file descriptor stats If the hostname is empty or cannot be identified, use the IP address of the Elasticsearch node as fallback. * Add request timeout for Cluster#clusterHealth() * Add integration test for Cluster class * Migrate to new Elasticsearch integration tests * Hopefully fix ClusterIT

jalogisch added the to-verify label Sep 5, 2017

joschi added elasticsearch improvement low hanging fruit papercut triaged and removed to-verify labels Sep 6, 2017

joschi self-assigned this Sep 6, 2017

joschi mentioned this issue Sep 6, 2017

Add "format" query parameter to Elasticsearch cat request #4127

Merged

ghost added the in progress label Sep 6, 2017

joschi added this to the 2.3.2 milestone Sep 6, 2017

bernd closed this as completed in #4127 Sep 19, 2017

ghost removed the in progress label Sep 19, 2017

joschi modified the milestones: 2.3.2, 2.4.0 Oct 19, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getFileDescriptorStats() not working with ES 5.3 #4119

getFileDescriptorStats() not working with ES 5.3 #4119

maxirus commented Sep 1, 2017 •

edited

joschi commented Sep 6, 2017 •

edited

getFileDescriptorStats() not working with ES 5.3 #4119

getFileDescriptorStats() not working with ES 5.3 #4119

Comments

maxirus commented Sep 1, 2017 • edited

joschi commented Sep 6, 2017 • edited

maxirus commented Sep 1, 2017 •

edited

joschi commented Sep 6, 2017 •

edited