Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow indexing when cluster state is red but write index state is healthy #2477

Merged
merged 3 commits into from Jul 14, 2016

Conversation

@bernd
Copy link
Member

@bernd bernd commented Jul 13, 2016

The cluster state can turn red even though the current write index is green. This might happen when running rotation/retention during an ES maintenance window where not all nodes are present.

For index related tasks we are now checking the state of the current write index (deflector) instead of the state for all Graylog managed indices.

The logs and UI are still showing the red cluster state to make sure the admin will be notified.

Refs #2429
Fixes #2371

…yellow

The cluster state can turn red even though the current write index is
green. This might happen when running rotation/retention during an ES
maintenance window where not all nodes are present.

For index related tasks we are now checking the state of the current
write index (deflector) instead of the state for all Graylog managed
indices.

The logs and UI are still showing the red cluster state to make sure the
admin will be notified.

Refs #2429
Fixes #2371
@bernd bernd added this to the 2.1.0 milestone Jul 13, 2016
if (isConnected() && isHealthy()) {
LOG.debug("Cluster is healthy again, unblocking waiting threads.");
if (isConnected() && isDeflectorHealthy()) {
LOG.debug("Cluster and write index is healthy again, unblocking waiting threads.");

This comment has been minimized.

@joschi

joschi Jul 13, 2016
Contributor

I think this should be "or".

I would remove the part about the cluster completely: "Write-active index is healthy again, unblocking waiting threads."

@@ -176,12 +217,18 @@ public void waitForConnectedAndHealthy(long timeout, TimeUnit unit) throws Inter
scheduledFuture.cancel(true); // Make sure to cancel the task to avoid task leaks!

if (!waitSuccess) {
throw new TimeoutException("Elasticsearch cluster didn't get healthy within timeout");
throw new TimeoutException("Elasticsearch cluster or write index didn't get healthy within timeout");

This comment has been minimized.

@joschi

joschi Jul 13, 2016
Contributor

I would remove the part about the cluster completely: "Write-active index didn't get healthy within timeout"

@joschi joschi self-assigned this Jul 13, 2016
@joschi
Copy link
Contributor

@joschi joschi commented Jul 13, 2016

LGTM except for those 2 wordings. 👍

@joschi joschi merged commit 6fbf9e9 into master Jul 14, 2016
4 checks passed
4 checks passed
@garybot2
ci-server-integration Jenkins build graylog2-server-integration-pr 1085 has succeeded
Details
@garybot2
ci-web-linter Jenkins build graylog-pr-linter-check 571 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@joschi joschi deleted the issue-2429 branch Jul 14, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants