Primary shard stuck initialising on dynamic index creation

[robertsmarty]

Hi,

I have a 5 node 2.1.0 elasticsearch cluster receiving about 150k of firewall logs per minute from 3 load balanced logstash servers. The index template is configured for 5 shards and 1 replica and each shard contains about 20GB of logs per day. It all works great until the clock ticks over to the next day. The new index is dynamically created however at least one of the primary shards is stuck initiallising and doesn't recovery. At this point I have to stop the logs coming in. Delete the index and start the logs again.

Any ideas?

Cheers,

Marty

[clintongormley]

Hi @robertsmarty

Please can you upload your logs around midnight. Also, do you have any shard allocation settings?

[robertsmarty]

Sorry to waste your time with this. I'll investigate this further myself as I think it's a load issue and I need to do some tuning.