[ResponseOps] searchsource references deleted from es query rule after API key changed

[pmuellr]

stack version: 8.5.3

User is seeing error when a rule is running; at some point, the rule was updated (we don't know how, still gathering info), and on the next run produced the following message:

Error: Error injecting reference into rule params for rule id <id> - 
           Could not find reference for kibanaSavedObjectMeta.searchSourceJSON.index
    at loadRule (/x-pack/plugins/alerting/server/task_runner/rule_loader.js:59:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at TaskRunner.prepareToRun (/x-pack/plugins/alerting/server/task_runner/task_runner.js:494:12)
    at TaskRunnerTimer.runWithTimer (/x-pack/plugins/alerting/server/task_runner/task_runner_timer.js:57:20)
    at TaskRunner.run (/x-pack/plugins/alerting/server/task_runner/task_runner.js:611:30)
    at TaskManagerRunner.run (/x-pack/plugins/task_manager/server/task_running/task_runner.js:299:22)

Below are the relevant bits of the rule SO - notice that there are references to the references in the params (indexRefName), however the actual references attribute is empty.

relevant bits from rule SO

{
  "alert": {
    "params": {
      "excludeHitsFromPreviousRun": true,
      "searchConfiguration": {
        "filter": [
          {
            "$state": {
              "store": "appState"
            },
            "meta": {
              "alias": null,
              "disabled": false,
              "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index",
              "key": "...redacted...",
              "negate": false,
              "params": {
                "query": "...redacted..."
              },
              "type": "phrase"
            },
            "query": {
              "match_phrase": {
                "...redacted...": "...redacted..."
              }
            }
          },
          {
            "$state": {
              "store": "appState"
            },
            "meta": {
              "alias": null,
              "disabled": false,
              "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index",
              "key": "...redacted...",
              "negate": false,
              "params": {
                "query": "...redacted..."
              },
              "type": "phrase"
            },
            "query": {
              "match_phrase": {
                "...redacted...": "...redacted..."
              }
            }
          }
        ],
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index",
        "query": {
          "language": "kuery",
          "query": ""
        }
      }
    }
  },
  "references": []
}

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

A clue from the user who saw this was that it seemed to happen after they performed an "update API key" operation.

So, did a repro of that on 8.6 - user saw it on 8.5.3.

• browsed an index in Discover that has a time field (most do)
• created an alert from Discover - that way I knew it was going to use searchSource and not an explicit index
• let the rule run a few times
• use the "update API key" action in the "..." menu beside the rule
• blammo!

"Blammo!" was some toasts saying the API key update failed, and then one saying it could load a rule, or rules (not sure, the toasts disappear and I'm not sure how to see them again). The screen also went blank with a spinner in the main section. Reloading the rule ends up in the same state, this time with a toast "Unable to load rule:" with no further info - obviously the rule name/id and error would be useful to provide here.

Looking at the Kibana logs, here's what I'm seeing; the last message is repeated for every rule run:

Mar 10, 2023 @ 19:45:31.414     Rule schedule interval (1s) for ".es-query" rule type with ID "0dc61a50-bf7c-11ed-b5c4-27d1f241d08e" is less
    than the minimum value (1m). Running rules at this interval may impact alerting performance. Set 
    "xpack.alerting.rules.minimumScheduleInterval.enforce" to true to prevent such changes.

Mar 10, 2023 @ 19:46:14.257     Rule schedule interval (1s) for ".es-query" rule type with ID "undefined" is less 
    than the minimum value (1m). Running rules at this interval may impact alerting performance. Set 
    "xpack.alerting.rules.minimumScheduleInterval.enforce" to true to prevent such changes.

Mar 10, 2023 @ 19:46:18.603		Executing Rule default:.es-query:0dc61a50-bf7c-11ed-b5c4-27d1f241d08e has
   resulted in Error: Error injecting reference into rule params for rule id 0dc61a50-bf7c-11ed-b5c4-27d1f241d08e -
   Could not find reference for kibanaSavedObjectMeta.searchSourceJSON.index - Error: Error injecting reference
   into rule params for rule id 0dc61a50-bf7c-11ed-b5c4-27d1f241d08e - Could not find reference for 
   kibanaSavedObjectMeta.searchSourceJSON.index
    at loadRule (/usr/share/kibana/x-pack/plugins/alerting/server/task_runner/rule_loader.js:52:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at TaskRunner.prepareToRun (/usr/share/kibana/x-pack/plugins/alerting/server/task_runner/task_runner.js:409:12)
    at TaskRunnerTimer.runWithTimer (/usr/share/kibana/x-pack/plugins/alerting/server/task_runner/task_runner_timer.js:49:20)
    at TaskRunner.run (/usr/share/kibana/x-pack/plugins/alerting/server/task_runner/task_runner.js:515:30)
    at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)

Looks similar to the user's scenario, so guessing this is a good repro for the user.

Note also that I set the rule interval to 1s (as one does), and the second time it complained about that 1s interval, it logged the rule ID as undefined. Not sure if that's a different problem or another symptom of the root problem here.

[pmuellr]

To "fix" the problem this issue causes - can't view the Rules list in the space where this happened - you can delete the rule via Dev Tools

DELETE kbn:/api/alerting/rule/<rule-id>

I was a little worried we'd have to manually delete the task document as well, but the DELETE route seems to ignore many types of semantic errors and delete documents - it appears it ended up deleting the rule AND task document fine.

I would be a little surprised if it would be possible to "fix" this without deleting it, but I haven't tried. I think the idea would be to do an update of the rule, adding the searchSource back. DELETEing the rule will at least get the other rules in the space to show in the Rules list ...

[pmuellr]

I was curious if ANY reference would suffer this problem - for example actions (which have references to the connectors).

Nope! When I update the API key for a rule with actions, everything appears fine. So this appears to be something specific regarding the references for the searchSource.

[essodjolo]

Hi @pmuellr,

Just a FYI: I have a user who had this after they attempted to snooze multiple rules using a bulk action in de rules overview page.

[damianpfister]

@pmuellr - you were correct that the rules need to be deleted first before fixing the issue, even post upgrade to 8.8.0 of the stack.