Race condition between flushing and closing index

[kroepke]

The effects are probably harmless (because I guess closing an index will internally flush any changes to it anyway), but the error looks dangerous:

2017-06-13 14:52:55,746 ERROR: org.graylog2.system.jobs.SystemJobManager - Unhandled error while running SystemJob <267dc710-5037-11e7-a3f9-26623c50cacc> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
org.graylog2.indexer.ElasticsearchException: Couldn't flush index graylog_0

closed
	at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:101) ~[classes/:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[classes/:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:70) ~[classes/:?]
	at org.graylog2.indexer.indices.Indices.flush(Indices.java:503) ~[classes/:?]
	at org.graylog2.indexer.SetIndexReadOnlyJob.execute(SetIndexReadOnlyJob.java:73) ~[classes/:?]
	at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:51) ~[classes/:?]
	at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [classes/:?]
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [metrics-core-3.2.2.jar:3.2.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]

[bernd]

This happens when you rotate the active write index and after that close the previous write index. The SetIndexReadOnlyAndCalculateRangeJob only runs after 30 seconds and if the previous index gets closed during that time, we see the error. I will prepare a PR to check if the index is closed before running the SetIndexReadOnlyAndCalculateRangeJob.