New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow indexing when cluster state is red but write index state is healthy #2477

Merged
merged 3 commits into from Jul 14, 2016

Conversation

Projects
None yet
2 participants
@bernd
Member

bernd commented Jul 13, 2016

The cluster state can turn red even though the current write index is green. This might happen when running rotation/retention during an ES maintenance window where not all nodes are present.

For index related tasks we are now checking the state of the current write index (deflector) instead of the state for all Graylog managed indices.

The logs and UI are still showing the red cluster state to make sure the admin will be notified.

Refs #2429
Fixes #2371

Allow indexing when cluster state is red but write index state green/…
…yellow

The cluster state can turn red even though the current write index is
green. This might happen when running rotation/retention during an ES
maintenance window where not all nodes are present.

For index related tasks we are now checking the state of the current
write index (deflector) instead of the state for all Graylog managed
indices.

The logs and UI are still showing the red cluster state to make sure the
admin will be notified.

Refs #2429
Fixes #2371

@bernd bernd added this to the 2.1.0 milestone Jul 13, 2016

if (isConnected() && isHealthy()) {
LOG.debug("Cluster is healthy again, unblocking waiting threads.");
if (isConnected() && isDeflectorHealthy()) {
LOG.debug("Cluster and write index is healthy again, unblocking waiting threads.");

This comment has been minimized.

@joschi

joschi Jul 13, 2016

Contributor

I think this should be "or".

I would remove the part about the cluster completely: "Write-active index is healthy again, unblocking waiting threads."

@@ -176,12 +217,18 @@ public void waitForConnectedAndHealthy(long timeout, TimeUnit unit) throws Inter
scheduledFuture.cancel(true); // Make sure to cancel the task to avoid task leaks!
if (!waitSuccess) {
throw new TimeoutException("Elasticsearch cluster didn't get healthy within timeout");
throw new TimeoutException("Elasticsearch cluster or write index didn't get healthy within timeout");

This comment has been minimized.

@joschi

joschi Jul 13, 2016

Contributor

I would remove the part about the cluster completely: "Write-active index didn't get healthy within timeout"

@joschi joschi self-assigned this Jul 13, 2016

@joschi

This comment has been minimized.

Contributor

joschi commented Jul 13, 2016

LGTM except for those 2 wordings. 👍

@joschi joschi merged commit 6fbf9e9 into master Jul 14, 2016

4 checks passed

ci-server-integration Jenkins build graylog2-server-integration-pr 1085 has succeeded
Details
ci-web-linter Jenkins build graylog-pr-linter-check 571 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@joschi joschi deleted the issue-2429 branch Jul 14, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment