[Security Solution] Detections fail when an index is missing a primary shard

[BenB196]

Describe the bug:

If there is an index that is used in a detection rule, and that index is missing a primary shard (cluster in red state), the detection rules that use that index will fail with the error:

Bulk Indexing of signals failed: index: "<index_with_primary_shard_missing>" reason: "[<node_with_missing_shard>][127.0.0.1:9300][indices:data/read/search[phase/query]]" type: "no_shard_available_action_exception" name: "<detection_rule_name>" id: "ac117e42-a91f-11eb-a252-7f35a8822039" rule id: "5e552599-ddec-4e14-bad1-28aa42404388" signals index: ".siem-signals-security"

This results in the detection rule(s) not running, and therefore not detecting things.

Kibana/Elasticsearch Stack version:

Elasticsearch: 7.13.1
Kibana: 7.13.1

Server OS version:

OS: Docker/ECK

Browser and Browser OS versions:

N/A

Elastic Endpoint version:

N/A

Original install method (e.g. download page, yum, from source, etc.):

ECK

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Detections

Steps to reproduce:

1. Create a detection rule that uses an index pattern. ex: filebeat-*
2. Create two indices, filebeat-1, filebeat-2
3. Start detection rule
4. Cause a primary shard to fail on filebeat-1
5. Detection rule will fail

Current behavior:

Detection rule fails when a primary shard is missing. This results in no detections occurring at all.

Expected behavior:

Detection rule should still work, but should instead show as warning. Detections should still trigger from the available indices.

Screenshots (if relevant):

N/A

Errors in browser console (if relevant):

N/A

Provide logs and/or server output (if relevant):

N/A

Any additional context (logs, chat logs, magical formulas, etc.):

I know that a red cluster state is bad, but sometimes it happens, and I would expect a security solution to best effort maintain monitoring of the environment while the red state is resolved.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)