[Discover][Alerts] Fix Discover results when alert excludes matches from previous runs

[jughosta]

• Closes [Discover][Alerts] Exclude matches from previous runs  #148282

Summary

In case if user creates a rule and enables "Exclude matches from previous runs", Discover link will now include a time filter to filter previous results out.

For testing:

• Open Discover with an index which has documents before and after current time (e.g. a freshly installed Kibana Sample Data Logs)
• Create a new rule
  • Enable/disable "Exclude matches from previous runs" switch
  • Define an index connector with a link

 {
  "rule_id": "{{rule.id}}",
  "rule_name": "{{rule.name}}",
  "alert_id": "{{alert.id}}",
  "context_message": "{{context.message}}",
  "link": "{{context.link}}"
}

• Now navigate to Discover, create a data view for the connector index
• Copy locator links from the appearing alerts and open Discover with them in another tab

If "Exclude matches from previous runs" was enabled, then an additional filter will show up on Discover page for the locator link.

Check that Discover total count is the same as the one mentioned in context_message field.

[jughosta]

/ci

[jughosta]

/ci

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[pmuellr]

LGTM; works as expected

I'm wondering though whether we should enable this "behavior" in other ways. For instance, when I turn "exclude matches" off, but obviously still have a "time window" (FOR THE LAST ), shouldn't we use a filter that would only show the docs within that window? Currently that link doesn't filter on time ...

This will be very useful as-is though, as a lot of folks seem to use exclude previous matches but then are confused by the generated (or non-generated) alerts. I think this will help!

[jughosta]

Hi @pmuellr,

Thanks for the review! Yes, indeed, it makes sense to show results only for the alert's time window. We are already doing it by passing timeRange.from and timeRange.to as params to Discover locator.

kibana/x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.ts

Line 213 in 1e08ba0

timeRange: { from: dateStart, to: dateEnd },

This sets the correct time range for the time picker control in Discover UI which automatically translates into a time range filter for ES request. Or it's not working for you?

For this PR I initially was going to replace dateStart with latestTimestamp in Discover locator params to narrow down the search (if "exclude matches" is on) but it would include results for latestTimestamp too as Discover uses gte. So, I changed the approach to add an extra filter with gt instead to fetch results within dateStart and dateEnd but only after latestTimestamp.

[pmuellr]

This sets the correct time range for the time picker control in Discover UI which automatically translates into a time range filter for ES request. Or it's not working for you?

Let me retry that - I saw a big difference in the number of docs, more than I was expecting. But it's also possible I was testing with a long window and short intervals so there WERE lots of docs.

[pmuellr]

Let me retry that - I saw a big difference in the number of docs, more than I was expecting. But it's also possible I was testing with a long window and short intervals so there WERE lots of docs.

Heh, ya, that was just me. I think the difference ended up confusing me, but I wasn't actually looking at the date values:

with exclude matches:

without exclude matches:

Both show the time window of the rule, which makes sense. When I saw the second one, I figured it actually extended out more to the left, basically listing ALL the matching docs. But, it's just showing the exact window of matching docs :-)

This PR certainly improves things!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: a884311

Failed CI Steps

• FTR Configs #22

Metrics [docs]

✅ unchanged

History

• 💛 Build #192943 was flaky 1e08ba0
• 💛 Build #192912 was flaky 6044f6d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jughosta

[davismcphee]

Tested locally and it works as expected, great work on this! And thanks a bunch for the testing instructions, they made it much easier for me to test.

I actually think using the GT filter in addition to the time filter is preferable since it shows the expected number of docs by default, but also allows users to disable the filter to see all matches if they want to. Thanks for fixing this and LGTM 👍

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.12

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation