Race condition during index delete

[patrickmann]

HS-2287071296

Race condition during index deletion may cause errors in event queries.

Expected Behavior

I should be able to delete an index via the Graylog API without triggering any errors due to race conditions.

Current Behavior

There is a brief lag between deleting an index in OS and in MongoDB.
Event query execution may thus include the index, even though it is already deleted in OS.

Possible Solution

• Flip the order of events: first delete the index in Mongo, then in OS. This may run into problems, if another thread decides to rebuild the meta-data.
• use ignore_unavailable option. We usually do this, so we should do it here too.

Steps to Reproduce (for bugs)

Repro is tricky, since it is a short timing window. But if indices are regularly deleted, and event queries are using those indices at the same time, you will eventually see an OS exception index_not_found_exception, reason=no_such_index

Your Environment

• Graylog Version: 5.2.2