Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "format" query parameter to Elasticsearch cat request #4127

Merged
merged 6 commits into from Sep 19, 2017
Merged

Conversation

@joschi
Copy link
Contributor

@joschi joschi commented Sep 6, 2017

While using the "Accept" HTTP request header should be perfectly sufficient, some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem to support it.

This change set adds the alternative "format" query parameter to the cat request in order to satisfy these clones.

Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor

Additionally to adding an integration test for the Cluster class, this PR also addresses two other minor issues:

  • Add missing timeout in the Cluster#clusterHealth() method
  • Use IP address as fallback if the hostname of an Elasticsearch node can't be identified

Closes #4119

@joschi joschi added this to the 2.4.0 milestone Sep 6, 2017
@joschi joschi force-pushed the issue-4119 branch from 3a018e9 to 5c86d4f Sep 6, 2017
@bernd
Copy link
Member

@bernd bernd commented Sep 18, 2017

@joschi The failed graylog-project/pr test is expected or an issue?

@joschi
Copy link
Contributor Author

@joschi joschi commented Sep 18, 2017

@bernd The build works on Travis CI and locally on my machine, so I guess it's (again) some flakiness in our Jenkins CI instance:

$ mvn clean install -Dskip.web.build=true
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Graylog Parent POM
[INFO] Graylog Project Parent POM
[INFO] AutoValue JavaBean extension
[INFO] Graylog
[INFO] integration-tests
[INFO] Graylog Plugin Parent POM
[INFO] Graylog Plugin with Web support Parent POM
[INFO] graylog-plugin-archetype
[Lots of output…]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Graylog Parent POM ................................. SUCCESS [  0.300 s]
[INFO] Graylog Project Parent POM ......................... SUCCESS [  0.252 s]
[INFO] AutoValue JavaBean extension ....................... SUCCESS [  2.391 s]
[INFO] Graylog ............................................ SUCCESS [03:08 min]
[INFO] integration-tests .................................. SUCCESS [  0.606 s]
[INFO] Graylog Plugin Parent POM .......................... SUCCESS [  0.009 s]
[INFO] Graylog Plugin with Web support Parent POM ......... SUCCESS [  0.005 s]
[INFO] graylog-plugin-archetype ........................... SUCCESS [  0.457 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:13 min
[INFO] Finished at: 2017-09-18T10:01:58+02:00
[INFO] Final Memory: 105M/1828M
[INFO] ------------------------------------------------------------------------

Possible root cause on our Jenkins CI instance:

2017-09-18 09:39:10,599 WARN : org.elasticsearch.threadpool - [Blue Shield] failed to run org.elasticsearch.index.shard.IndexShard$EngineRefresher@659091f
org.elasticsearch.index.engine.EngineClosedException: CurrentState[CLOSED] Closed
	at org.elasticsearch.index.shard.IndexShard.engine(IndexShard.java:1462) ~[elasticsearch-2.4.4.jar:2.4.4]
	at org.elasticsearch.index.shard.IndexShard$EngineRefresher.run(IndexShard.java:1338) ~[elasticsearch-2.4.4.jar:2.4.4]
	at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:668) [elasticsearch-2.4.4.jar:2.4.4]
@bernd bernd self-assigned this Sep 18, 2017
@bernd
Copy link
Member

@bernd bernd commented Sep 18, 2017

@joschi Running mvn clean package assembly:single in graylog-project fails for me with a similar error:

ests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.084 sec - in org.graylog2.migrations.V20161215163900_MoveIndexSetDefaultConfigTest

Results :

Failed tests: 
  ClusterTest.deflectorHealth:124->lambda$deflectorHealth$1:124 expected:<"[green]"> but was:<"[red]">

Tests run: 1530, Failures: 1, Errors: 0, Skipped: 3

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Graylog Parent POM ................................. SUCCESS [  0.389 s]
[INFO] Graylog Project Parent POM ......................... SUCCESS [  0.039 s]
[INFO] AutoValue JavaBean extension ....................... SUCCESS [  1.589 s]
[INFO] Graylog ............................................ FAILURE [04:43 min]
[INFO] Graylog Plugin Parent POM .......................... SKIPPED
[INFO] Graylog Usage Statistics Plugin .................... SKIPPED
[INFO] Graylog Plugin with Web support Parent POM ......... SKIPPED
[INFO] graylog-plugin-map-widget .......................... SKIPPED
[INFO] Graylog Pipeline Plugin Parent ..................... SKIPPED
[INFO] graylog-plugin-pipeline-processor .................. SKIPPED
[INFO] graylog-plugin-collector ........................... SKIPPED
[INFO] graylog-plugin-enterprise-integration .............. SKIPPED
[INFO] graylog-plugin-beats ............................... SKIPPED
[INFO] graylog-plugin-aws ................................. SKIPPED
[INFO] graylog-plugin-netflow ............................. SKIPPED
[INFO] graylog-plugin-threatintel ......................... SKIPPED
[INFO] graylog-plugin-license-parent ...................... SKIPPED
[INFO] graylog-plugin-license-api ......................... SKIPPED
[INFO] graylog-plugin-archive ............................. SKIPPED
[INFO] graylog-plugin-auditlog ............................ SKIPPED
[INFO] graylog-plugin-license ............................. SKIPPED
[INFO] graylog-plugin-license-test ........................ SKIPPED
[INFO] graylog-project .................................... SKIPPED
[INFO] runner ............................................. SKIPPED
[INFO] graylog-enterprise ................................. SKIPPED
[INFO] integration-tests .................................. SKIPPED
[INFO] graylog-plugin-archetype ........................... SKIPPED
[INFO] graylog-plugin-license-generator-cli ............... SKIPPED
[INFO] graylog-plugin-license-example ..................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:46 min
[INFO] Finished at: 2017-09-18T19:57:03+02:00
[INFO] Final Memory: 68M/2506M
[INFO] ------------------------------------------------------------------------
Jochen Schalanda added 5 commits Sep 6, 2017
Jochen Schalanda
While using the "Accept" HTTP request header should be perfectly sufficient,
some Elasticsearch "clones" (such as the AWS Elasticsearch Service) don't seem
to support it.

This change set adds the alternative "format" query parameter to the cat request
in order to satisfy these clones.

Also see: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cat.html#_response_as_text_json_smile_yaml_or_cbor

Closes #4119
Jochen Schalanda
If the hostname is empty or cannot be identified, use the IP address
of the Elasticsearch node as fallback.
Jochen Schalanda
Jochen Schalanda
Jochen Schalanda
@joschi joschi force-pushed the issue-4119 branch from 5c86d4f to 59cb04d Sep 19, 2017
Jochen Schalanda
@joschi joschi force-pushed the issue-4119 branch from 5272070 to 90fdaed Sep 19, 2017
@joschi
Copy link
Contributor Author

@joschi joschi commented Sep 19, 2017

@bernd The current iteration seems to work (after migrating to the new Elasticsearch integration tests).

@bernd
bernd approved these changes Sep 19, 2017
Copy link
Member

@bernd bernd left a comment

LGTM 👍

@bernd bernd merged commit 122f00c into master Sep 19, 2017
5 checks passed
5 checks passed
ci-web-linter Jenkins build graylog-pr-linter-check 1922 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
graylog-project/pr Jenkins build graylog-project-pr-snapshot 468 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details
@bernd bernd deleted the issue-4119 branch Sep 19, 2017
@bernd bernd added the backport label Sep 19, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.