Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Failed rule execution shows 0 for index duration even though alerts were written #155672

Open
Tracked by #165878
spong opened this issue Apr 24, 2023 · 5 comments
Labels
bug Fixes for quality problems that affect the customer experience consider-next Feature:Rule Monitoring Security Solution Detection Rule Monitoring impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@spong
Copy link
Member

spong commented Apr 24, 2023

Summary

First identified in 8.8/main (in testing #155384), the Rule Execution Results table can show 0 for Index Duration even though alerts were written. This is problematic as we don't currently show how many alerts were created per execution, so users must use the Index Duration column to determine if alerts were created.

Steps to recreate

Additionally, in recreating this issue it looks like Index Duration can sometimes be non-zero even when alerts weren't created (i.e. the inverse of the above):

Index Duration shows non 0 value, however no alerts were indexed:

Resulting documents

Rule Execution Results API response:

{
    "total": 5,
    "events": [
        {
            "execution_uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f",
            "timestamp": "2023-04-24T18:29:29.391Z",
            "duration_ms": 3439,
            "status": "success",
            "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
            "num_active_alerts": 0,
            "num_new_alerts": 0,
            "num_recovered_alerts": 0,
            "num_triggered_actions": 0,
            "num_succeeded_actions": 0,
            "num_errored_actions": 0,
            "total_search_duration_ms": 0,
            "es_search_duration_ms": 2676,
            "schedule_delay_ms": 3074,
            "timed_out": false,
            "indexing_duration_ms": 0,
            "search_duration_ms": 0,
            "gap_duration_s": 739,
            "security_status": "failed",
            "security_message": "12 minutes (739115ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances"
        },
        {
            "execution_uuid": "33dad50c-5bb6-4294-8f4a-adf1655d912b",
            "timestamp": "2023-04-24T18:15:20.269Z",
            "duration_ms": 113,
            "status": "success",
            "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
            "num_active_alerts": 0,
            "num_new_alerts": 0,
            "num_recovered_alerts": 0,
            "num_triggered_actions": 1,
            "num_succeeded_actions": 1,
            "num_errored_actions": 0,
            "total_search_duration_ms": 0,
            "es_search_duration_ms": 0,
            "schedule_delay_ms": 3001,
            "timed_out": false,
            "indexing_duration_ms": 0,
            "search_duration_ms": 1,
            "gap_duration_s": 0,
            "security_status": "succeeded",
            "security_message": "Rule execution completed successfully"
        },
        {
            "execution_uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1",
            "timestamp": "2023-04-24T18:14:17.251Z",
            "duration_ms": 634,
            "status": "success",
            "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
            "num_active_alerts": 0,
            "num_new_alerts": 0,
            "num_recovered_alerts": 0,
            "num_triggered_actions": 1,
            "num_succeeded_actions": 1,
            "num_errored_actions": 0,
            "total_search_duration_ms": 0,
            "es_search_duration_ms": 532,
            "schedule_delay_ms": 3061,
            "timed_out": false,
            "indexing_duration_ms": 2,
            "search_duration_ms": 535,
            "gap_duration_s": 0,
            "security_status": "succeeded",
            "security_message": "Rule execution completed successfully"
        },
        {
            "execution_uuid": "8f79a93b-37b4-49b5-b7fa-92d66978bf57",
            "timestamp": "2023-04-24T18:13:44.261Z",
            "duration_ms": 3144,
            "status": "success",
            "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
            "num_active_alerts": 0,
            "num_new_alerts": 0,
            "num_recovered_alerts": 0,
            "num_triggered_actions": 1,
            "num_succeeded_actions": 1,
            "num_errored_actions": 0,
            "total_search_duration_ms": 0,
            "es_search_duration_ms": 2195,
            "schedule_delay_ms": 2981,
            "timed_out": false,
            "indexing_duration_ms": 0,
            "search_duration_ms": 0,
            "gap_duration_s": 756,
            "security_status": "failed",
            "security_message": "13 minutes (756149ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances"
        },
        {
            "execution_uuid": "a329aeeb-13bc-4d04-849f-d233056d4fb1",
            "timestamp": "2023-04-24T17:57:38.114Z",
            "duration_ms": 1807,
            "status": "success",
            "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
            "num_active_alerts": 0,
            "num_new_alerts": 0,
            "num_recovered_alerts": 0,
            "num_triggered_actions": 42,
            "num_succeeded_actions": 42,
            "num_errored_actions": 0,
            "total_search_duration_ms": 0,
            "es_search_duration_ms": 561,
            "schedule_delay_ms": 568,
            "timed_out": false,
            "indexing_duration_ms": 951,
            "search_duration_ms": 570,
            "gap_duration_s": 0,
            "security_status": "succeeded",
            "security_message": "Rule execution completed successfully"
        }
    ]
}

Event Log docs for execution 4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f (gap failure, but alerts generated)

{
  "took": 444,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 9,
      "relation": "eq"
    },
    "max_score": 6.178817,
    "hits": [
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "EE6HtIcBSfLKi4FVK1hq",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:29.391Z",
          "event": {
            "provider": "alerting",
            "action": "execute-start",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:29:29.391Z"
          },
          "kibana": {
            "alert": {
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f"
                }
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "task": {
              "scheduled": "2023-04-24T18:29:26.317Z",
              "schedule_delay": 3074000000
            },
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem"
          },
          "message": "rule execution start: \"7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac\"",
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "EU6HtIcBSfLKi4FVK1hq",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:29.460Z",
          "event": {
            "provider": "securitySolution.ruleExecution",
            "kind": "event",
            "action": "status-change",
            "sequence": 0,
            "severity": 20
          },
          "message": "",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "uuid": "e2f7a1b0-ae75-412a-978f-795118790742",
            "name": "Testing #155384",
            "category": "siem.queryRule"
          },
          "log": {
            "level": "info"
          },
          "kibana": {
            "alert": {
              "rule": {
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f",
                  "status": "running",
                  "status_order": 15
                },
                "revision": 8
              }
            },
            "space_ids": [
              "default"
            ],
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac"
              }
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "Ek6HtIcBSfLKi4FVK1hq",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:29.470Z",
          "event": {
            "provider": "securitySolution.ruleExecution",
            "kind": "metric",
            "action": "execution-metrics",
            "sequence": 1,
            "severity": 10
          },
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "uuid": "e2f7a1b0-ae75-412a-978f-795118790742",
            "name": "Testing #155384",
            "category": "siem.queryRule"
          },
          "log": {
            "level": "debug"
          },
          "kibana": {
            "alert": {
              "rule": {
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f",
                  "metrics": {
                    "execution_gap_duration_s": 739
                  }
                },
                "revision": 8
              }
            },
            "space_ids": [
              "default"
            ],
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac"
              }
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "E06HtIcBSfLKi4FVK1hq",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:29.471Z",
          "event": {
            "provider": "securitySolution.ruleExecution",
            "kind": "event",
            "action": "status-change",
            "sequence": 2,
            "severity": 40
          },
          "message": "12 minutes (739115ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "uuid": "e2f7a1b0-ae75-412a-978f-795118790742",
            "name": "Testing #155384",
            "category": "siem.queryRule"
          },
          "log": {
            "level": "error"
          },
          "kibana": {
            "alert": {
              "rule": {
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f",
                  "status": "failed",
                  "status_order": 30
                },
                "revision": 8
              }
            },
            "space_ids": [
              "default"
            ],
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac"
              }
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "FE6HtIcBSfLKi4FVN1gm",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:32.802Z",
          "event": {
            "provider": "alerting",
            "action": "new-instance",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:29:32.802Z",
            "duration": "0"
          },
          "kibana": {
            "alert": {
              "flapping": false,
              "maintenance_window_ids": [],
              "uuid": "246eabb0-6a01-4f80-8023-c0aff523efa9",
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f"
                }
              }
            },
            "alerting": {
              "instance_id": "579f890a392b6d897ccd40bb324e162f0a08a2d8eb4bb3a4668038f8e74378a1",
              "action_group_id": "default"
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384' created new alert: '579f890a392b6d897ccd40bb324e162f0a08a2d8eb4bb3a4668038f8e74378a1'",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "FU6HtIcBSfLKi4FVN1gm",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:32.802Z",
          "event": {
            "provider": "alerting",
            "action": "new-instance",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:29:32.802Z",
            "duration": "0"
          },
          "kibana": {
            "alert": {
              "flapping": false,
              "maintenance_window_ids": [],
              "uuid": "66a4236c-1c12-4095-abfe-dd4b2095c614",
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f"
                }
              }
            },
            "alerting": {
              "instance_id": "9b64f45baa19fa2175dad69ba2724d2466eef74a9d87676b295fec780b121350",
              "action_group_id": "default"
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384' created new alert: '9b64f45baa19fa2175dad69ba2724d2466eef74a9d87676b295fec780b121350'",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "Fk6HtIcBSfLKi4FVN1gm",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:32.802Z",
          "event": {
            "provider": "alerting",
            "action": "active-instance",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:29:32.802Z",
            "duration": "0"
          },
          "kibana": {
            "alert": {
              "flapping": false,
              "maintenance_window_ids": [],
              "uuid": "246eabb0-6a01-4f80-8023-c0aff523efa9",
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f"
                }
              }
            },
            "alerting": {
              "instance_id": "579f890a392b6d897ccd40bb324e162f0a08a2d8eb4bb3a4668038f8e74378a1",
              "action_group_id": "default"
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384' active alert: '579f890a392b6d897ccd40bb324e162f0a08a2d8eb4bb3a4668038f8e74378a1' in actionGroup: 'default'",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "F06HtIcBSfLKi4FVN1gm",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:32.802Z",
          "event": {
            "provider": "alerting",
            "action": "active-instance",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:29:32.802Z",
            "duration": "0"
          },
          "kibana": {
            "alert": {
              "flapping": false,
              "maintenance_window_ids": [],
              "uuid": "66a4236c-1c12-4095-abfe-dd4b2095c614",
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f"
                }
              }
            },
            "alerting": {
              "instance_id": "9b64f45baa19fa2175dad69ba2724d2466eef74a9d87676b295fec780b121350",
              "action_group_id": "default"
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384' active alert: '9b64f45baa19fa2175dad69ba2724d2466eef74a9d87676b295fec780b121350' in actionGroup: 'default'",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "GE6HtIcBSfLKi4FVN1gm",
        "_score": 6.178817,
        "_source": {
          "@timestamp": "2023-04-24T18:29:32.830Z",
          "event": {
            "provider": "alerting",
            "action": "execute",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:29:29.391Z",
            "outcome": "success",
            "end": "2023-04-24T18:29:32.830Z",
            "duration": "3439000000"
          },
          "kibana": {
            "alert": {
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "4c6fee0e-3c8a-4e3b-b9d1-c7a04869918f",
                  "metrics": {
                    "number_of_triggered_actions": 0,
                    "number_of_generated_actions": 0,
                    "alert_counts": {
                      "active": 2,
                      "new": 2,
                      "recovered": 0
                    },
                    "number_of_searches": 10,
                    "es_search_duration_ms": 2676,
                    "total_search_duration_ms": 2697,
                    "claim_to_start_duration_ms": 71,
                    "total_run_duration_ms": 3510,
                    "prepare_rule_duration_ms": 18,
                    "rule_type_run_duration_ms": 3356,
                    "process_alerts_duration_ms": 1,
                    "trigger_actions_duration_ms": 1,
                    "process_rule_duration_ms": 19
                  }
                }
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "task": {
              "scheduled": "2023-04-24T18:29:26.317Z",
              "schedule_delay": 3074000000
            },
            "alerting": {
              "outcome": "success",
              "status": "ok"
            },
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
          "ecs": {
            "version": "1.8.0"
          }
        }
      }
    ]
  }
}

Event Log docs for execution a18e1a6-a143-4bf8-8983-94d49e132ff1 (successful execution w/ 2 index duration but no alerts)

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 8,
      "relation": "eq"
    },
    "max_score": 6.292004,
    "hits": [
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "4U55tIcBSfLKi4FVPVe1",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:17.251Z",
          "event": {
            "provider": "alerting",
            "action": "execute-start",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:14:17.251Z"
          },
          "kibana": {
            "alert": {
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1"
                }
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "task": {
              "scheduled": "2023-04-24T18:14:14.190Z",
              "schedule_delay": 3061000000
            },
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem"
          },
          "message": "rule execution start: \"7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac\"",
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "4k55tIcBSfLKi4FVQVeg",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:17.287Z",
          "event": {
            "provider": "securitySolution.ruleExecution",
            "kind": "event",
            "action": "status-change",
            "sequence": 0,
            "severity": 20
          },
          "message": "",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "uuid": "e2f7a1b0-ae75-412a-978f-795118790742",
            "name": "Testing #155384",
            "category": "siem.queryRule"
          },
          "log": {
            "level": "info"
          },
          "kibana": {
            "alert": {
              "rule": {
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1",
                  "status": "running",
                  "status_order": 15
                },
                "revision": 2
              }
            },
            "space_ids": [
              "default"
            ],
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac"
              }
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "4055tIcBSfLKi4FVQVeg",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:17.834Z",
          "event": {
            "provider": "securitySolution.ruleExecution",
            "kind": "metric",
            "action": "execution-metrics",
            "sequence": 1,
            "severity": 10
          },
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "uuid": "e2f7a1b0-ae75-412a-978f-795118790742",
            "name": "Testing #155384",
            "category": "siem.queryRule"
          },
          "log": {
            "level": "debug"
          },
          "kibana": {
            "alert": {
              "rule": {
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1",
                  "metrics": {
                    "total_search_duration_ms": 535,
                    "total_indexing_duration_ms": 2,
                    "total_enrichment_duration_ms": 0
                  }
                },
                "revision": 2
              }
            },
            "space_ids": [
              "default"
            ],
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac"
              }
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "5E55tIcBSfLKi4FVQVeg",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:17.834Z",
          "event": {
            "provider": "securitySolution.ruleExecution",
            "kind": "event",
            "action": "status-change",
            "sequence": 2,
            "severity": 20
          },
          "message": "Rule execution completed successfully",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "uuid": "e2f7a1b0-ae75-412a-978f-795118790742",
            "name": "Testing #155384",
            "category": "siem.queryRule"
          },
          "log": {
            "level": "info"
          },
          "kibana": {
            "alert": {
              "rule": {
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1",
                  "status": "succeeded",
                  "status_order": 0
                },
                "revision": 2
              }
            },
            "space_ids": [
              "default"
            ],
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac"
              }
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "5U55tIcBSfLKi4FVQVeg",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:17.869Z",
          "event": {
            "provider": "alerting",
            "action": "execute-action",
            "kind": "alert",
            "category": [
              "siem"
            ]
          },
          "kibana": {
            "alert": {
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1"
                }
              }
            },
            "alerting": {
              "summary": {
                "new": {
                  "count": 0
                },
                "ongoing": {
                  "count": 0
                },
                "recovered": {
                  "count": 0
                }
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              },
              {
                "type": "action",
                "id": "c3189330-e095-11ed-8910-bf63b87615dd",
                "type_id": ".slack"
              }
            ],
            "space_ids": [
              "default"
            ],
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "alert: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384' instanceId: 'undefined' scheduled actionGroup: 'undefined' action: .slack:c3189330-e095-11ed-8910-bf63b87615dd",
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "5k55tIcBSfLKi4FVQVeg",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:17.885Z",
          "event": {
            "provider": "alerting",
            "action": "execute",
            "kind": "alert",
            "category": [
              "siem"
            ],
            "start": "2023-04-24T18:14:17.251Z",
            "outcome": "success",
            "end": "2023-04-24T18:14:17.885Z",
            "duration": "634000000"
          },
          "kibana": {
            "alert": {
              "rule": {
                "rule_type_id": "siem.queryRule",
                "consumer": "siem",
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1",
                  "metrics": {
                    "number_of_triggered_actions": 1,
                    "number_of_generated_actions": 1,
                    "alert_counts": {
                      "active": 0,
                      "new": 0,
                      "recovered": 0
                    },
                    "number_of_searches": 2,
                    "es_search_duration_ms": 532,
                    "total_search_duration_ms": 536,
                    "claim_to_start_duration_ms": 67,
                    "total_run_duration_ms": 701,
                    "prepare_rule_duration_ms": 16,
                    "rule_type_run_duration_ms": 552,
                    "process_alerts_duration_ms": 1,
                    "trigger_actions_duration_ms": 34,
                    "process_rule_duration_ms": 9
                  }
                }
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "task": {
              "scheduled": "2023-04-24T18:14:14.190Z",
              "schedule_delay": 3061000000
            },
            "alerting": {
              "outcome": "success",
              "status": "ok"
            },
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "rule": {
            "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
            "license": "basic",
            "category": "siem.queryRule",
            "ruleset": "siem",
            "name": "Testing #155384"
          },
          "message": "rule executed: siem.queryRule:7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac: 'Testing #155384'",
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "5055tIcBSfLKi4FVTVdY",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:20.293Z",
          "event": {
            "provider": "actions",
            "action": "execute-start",
            "kind": "action",
            "start": "2023-04-24T18:14:20.293Z"
          },
          "kibana": {
            "alert": {
              "rule": {
                "consumer": "siem",
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1"
                },
                "rule_type_id": "siem.queryRule"
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "action",
                "id": "c3189330-e095-11ed-8910-bf63b87615dd",
                "type_id": ".slack"
              },
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "task": {
              "scheduled": "2023-04-24T18:14:17.857Z",
              "schedule_delay": 2436000000
            },
            "action": {
              "name": "Message Garrett",
              "id": "c3189330-e095-11ed-8910-bf63b87615dd",
              "execution": {
                "uuid": "fcc9b89d-e66e-4fd1-8814-5adacf617f12",
                "source": "alert"
              }
            },
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "action started: .slack:c3189330-e095-11ed-8910-bf63b87615dd: Message Garrett",
          "ecs": {
            "version": "1.8.0"
          }
        }
      },
      {
        "_index": ".kibana-event-log-8.8.0-000001",
        "_id": "6E55tIcBSfLKi4FVTVdY",
        "_score": 6.292004,
        "_source": {
          "@timestamp": "2023-04-24T18:14:20.571Z",
          "event": {
            "provider": "actions",
            "action": "execute",
            "kind": "action",
            "start": "2023-04-24T18:14:20.293Z",
            "end": "2023-04-24T18:14:20.570Z",
            "duration": "277000000",
            "outcome": "success"
          },
          "kibana": {
            "alert": {
              "rule": {
                "consumer": "siem",
                "execution": {
                  "uuid": "a18e1a68-a143-4bf8-8983-94d49e132ff1"
                },
                "rule_type_id": "siem.queryRule"
              }
            },
            "saved_objects": [
              {
                "rel": "primary",
                "type": "action",
                "id": "c3189330-e095-11ed-8910-bf63b87615dd",
                "type_id": ".slack"
              },
              {
                "rel": "primary",
                "type": "alert",
                "id": "7ef56a30-e2c9-11ed-b0fe-ffb46d4ec8ac",
                "type_id": "siem.queryRule"
              }
            ],
            "space_ids": [
              "default"
            ],
            "task": {
              "scheduled": "2023-04-24T18:14:17.857Z",
              "schedule_delay": 2436000000
            },
            "action": {
              "name": "Message Garrett",
              "id": "c3189330-e095-11ed-8910-bf63b87615dd",
              "execution": {
                "uuid": "fcc9b89d-e66e-4fd1-8814-5adacf617f12",
                "source": "alert"
              }
            },
            "server_uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
            "version": "8.8.0"
          },
          "message": "action executed: .slack:c3189330-e095-11ed-8910-bf63b87615dd: Message Garrett",
          "ecs": {
            "version": "1.8.0"
          }
        }
      }
    ]
  }
}

@spong spong added bug Fixes for quality problems that affect the customer experience triage_needed Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Monitoring Security Solution Detection Rule Monitoring Team:Detection Rule Management Security Detection Rule Management Team labels Apr 24, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@spong
Copy link
Member Author

spong commented Apr 24, 2023

Note: Now that we're passing our execution status through the alerting framework (#147035), we can probably just move to implementing a solution using the find api instead of further tweaking the big 'ole agg we originally had to use to create these composite results:

e.g. the event.action:execute rollup event:

@banderror
Copy link
Contributor

banderror commented May 5, 2023

@spong We don't have a history of rule execution metrics in the rule object. We need to work on #135209 before we could get rid of this aggregation.

@banderror banderror added impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team:Detection Alerts Security Detection Alerts Area Team and removed triage_needed labels May 5, 2023
@banderror
Copy link
Contributor

@marshallmain This is a cross-area thing, but I think it is more on the rule executors' side. I added this bug to the Alerts area's backlog.

By the way, do we have (should we create) a Feature: label for referring to general (non-rule-type-specific) execution logic?

@yctercero yctercero added Team:Detection Engine Security Solution Detection Engine Area and removed Team:Detection Alerts Security Detection Alerts Area Team labels May 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience consider-next Feature:Rule Monitoring Security Solution Detection Rule Monitoring impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Development

No branches or pull requests

4 participants