New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow indexing even if ElasticSearch cluster health status is 'red', but current index (deflector) is 'green' #2371

Closed
gitfrederic opened this Issue Jun 17, 2016 · 0 comments

Comments

Projects
None yet
4 participants
@gitfrederic
Contributor

gitfrederic commented Jun 17, 2016

Today, Graylog 2.0.2 will pause indexing if the cluster health status is 'red'. However (if I'm not mistaken), ElasticSearch will still permit indexing to occur if the index being written to isn't 'red'.

This is a feature request to add a configuration property to Graylog Server to enable indexing if the ElasticSearch index pointed-to by the deflector is 'green' (or even 'yellow') -- even if the cluster itself is reporting a health status of 'red' (which could mean an old index, or even an index not managed by Graylog is 'red').

See:

https://github.com/Graylog2/graylog2-server/blob/2.0.2/graylog2-server/src/main/java/org/graylog2/outputs/BlockingBatchedESOutput.java .

and:

https://github.com/Graylog2/graylog2-server/blob/2.0.2/graylog2-server/src/main/java/org/graylog2/indexer/cluster/Cluster.java

Thanks!

@joschi joschi added the feature label Jun 20, 2016

@joschi joschi assigned joschi and unassigned joschi Jun 20, 2016

@joschi joschi added this to the 2.x milestone Jun 20, 2016

@kroepke kroepke self-assigned this Jul 1, 2016

@kroepke kroepke modified the milestones: 2.1.0, 2.x Jul 4, 2016

@bernd bernd self-assigned this Jul 5, 2016

bernd added a commit that referenced this issue Jul 13, 2016

Allow indexing when cluster state is red but write index state green/…
…yellow

The cluster state can turn red even though the current write index is
green. This might happen when running rotation/retention during an ES
maintenance window where not all nodes are present.

For index related tasks we are now checking the state of the current
write index (deflector) instead of the state for all Graylog managed
indices.

The logs and UI are still showing the red cluster state to make sure the
admin will be notified.

Refs #2429
Fixes #2371

@joschi joschi closed this in #2477 Jul 14, 2016

joschi added a commit that referenced this issue Jul 14, 2016

Allow indexing when cluster health state is RED but write-active inde…
…x is healthy (#2477)

The cluster state can turn red even though the current write index is
green. This might happen when running rotation/retention during an ES
maintenance window where not all nodes are present.

For index related tasks we are now checking the state of the current
write index (deflector) instead of the state for all Graylog managed
indices.

The logs and UI are still showing the red cluster state to make sure the
admin will be notified.

Refs #2429
Fixes #2371
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment