Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition between flushing and closing index #3909

Closed
kroepke opened this issue Jun 13, 2017 · 1 comment
Closed

Race condition between flushing and closing index #3909

kroepke opened this issue Jun 13, 2017 · 1 comment
Assignees
Milestone

Comments

@kroepke
Copy link
Member

@kroepke kroepke commented Jun 13, 2017

The effects are probably harmless (because I guess closing an index will internally flush any changes to it anyway), but the error looks dangerous:

2017-06-13 14:52:55,746 ERROR: org.graylog2.system.jobs.SystemJobManager - Unhandled error while running SystemJob <267dc710-5037-11e7-a3f9-26623c50cacc> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
org.graylog2.indexer.ElasticsearchException: Couldn't flush index graylog_0

closed
	at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:101) ~[classes/:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[classes/:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:70) ~[classes/:?]
	at org.graylog2.indexer.indices.Indices.flush(Indices.java:503) ~[classes/:?]
	at org.graylog2.indexer.SetIndexReadOnlyJob.execute(SetIndexReadOnlyJob.java:73) ~[classes/:?]
	at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:51) ~[classes/:?]
	at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [classes/:?]
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [metrics-core-3.2.2.jar:3.2.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
@kroepke kroepke added this to the 2.3.0 milestone Jun 13, 2017
@bernd bernd added the triaged label Jun 19, 2017
@bernd bernd self-assigned this Jun 22, 2017
@bernd
Copy link
Member

@bernd bernd commented Jun 22, 2017

This happens when you rotate the active write index and after that close the previous write index. The SetIndexReadOnlyAndCalculateRangeJob only runs after 30 seconds and if the previous index gets closed during that time, we see the error. I will prepare a PR to check if the index is closed before running the SetIndexReadOnlyAndCalculateRangeJob.

bernd added a commit that referenced this issue Jun 22, 2017
Also add that check to other jobs that could fail if an index is already
closed.

Fixes #3909
@bernd bernd added the in progress label Jun 22, 2017
@joschi joschi closed this in #3931 Jun 27, 2017
joschi added a commit that referenced this issue Jun 27, 2017
…3931)

Also add that check to other jobs that could fail if an index is already
closed.

Fixes #3909
@ghost ghost removed the in progress label Jun 27, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants