[Meta] MVP for Alerting in Discover

[lukeelmers]

With the introduction of the alerting service, we want to introduce support for configuring alerts based on a specific view in Discover. Based on discussions with alerting, app, & arch teams, these are the steps we've identified to get us there:

• [ ] Add support for ES DSL "search alert" in Alerting service
  • First-class support for ES DSL is not needed for MVP; we can simply use the callCluster service in the executor
• Expose tools from data plugin to convert a query to raw DSL for use in alert
• Create alert executor to define an alert based on ES DSL
• Build alert creation UI for Discover

Concepts

Clicking on create alert opens a flyout. The flyout takes into account the search bar query and filters. Whatever is in blue is configurable:

Architecture

---

Related meta issue for alerting on Dashboards: #71560

cc @AlonaNadler @shaunmcgough @arisonl @rayafratkina @stacey-gammon @elastic/kibana-alerting-services @elastic/kibana-app @elastic/kibana-app-arch

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[mikecote]

In regards to the first step (alerting service w/ ES DSL #61313). Is the expectation out of that to be able to run ES DSL queries against Elasticsearch or to build on top of our ES DSL alert type to support different types of queries? or maybe both? :)

[lukeelmers]

@mikecote I think the assumption for an MVP is simply to be able to provide the DSL as an input to the alerting service to run against ES. Extending on top of that would be something to consider as a future step.

[lukeelmers]

As @mikecote pointed out offline, there's already the callCluster service exposed to alert executors, which means we aren't necessarily blocked on first-class ES DSL support for this MVP... we can instead use callCluster with whatever DSL we want.

I've updated the list above to reflect this.

[ppisljar]

A lot of things happened over the last 6 months so I think we should revisit the approach/ architecture around alerting in discover.

Discover is using SearchSource (high level search service) to make the queries. I think it would be best to have a search alert executor which can work with SearchSource (as that's available on the server as well). This hopefully makes implementation simpler, more error prone and future proof.

[alexfrancoeur]

I'm sure the team is aware of this new alert type, but I wanted to link to the PR for the upcoming "Elasticsearch query" alert. #88528. I'm guessing this bypasses SearchSource, but may be helpful to review the implementation when we're building out the search alert in Discover

[shaunmcgough]

With the new search alert created, we are planning on moving alerting forward in Discover. @timductive @timroes I'm going to bring this up on our call next week, and we should have a plan forward. From a product perspective, alerting in Discover is one of the most requested features from our users.

[shaunmcgough]

Alerting in Discover was talked about again on our Discover pre-planning meeting for 7.14 on April 22nd, 2021

A few things were decided;

1. We must get alerting team buy in from product, tech, and team.
2. We must get tech, team, and product buy in for prioritization of alerting in Discover. There is a general thought that alerting in Discover is not prioritized and this lack of prioritization is holding the issue up, and causing confusion.
3. A new alert type is not enough. We will need a Search Source alerter for this
4. @gmmorris and @timroes have spoken and know about the blockers for this.
5. There is a sense that the Elastic alerting effort in general has been focused on RAC work
6. There are two issues holding us up:
   6a. Blocking from starting: Access to Data Services is REQUIRED - We could build this partially ourselves (in Discover team) (Expose data service in the alert execution method #87990)
   6b. Saved object references (we cannot build this simply) This is a larger effort the alerting team needs do. (Allow usage of saved object references in alerts #87992)

This is the current status of Alerting in Discover. Please get in touch @timroes @timductive or @shaunmcgough for more detailed info from a tech, team, or product perspective respectively.

[AlonaNadler]

Thanks for the summary @shaunmcgough

A new alert type is not enough. We will need _search source for this
I'm not sure I understand this bullet, what does that mean?

[shaunmcgough]

Hey @AlonaNadler, @timroes can elaborate. My understanding is we'll need to use this for our alert.

[mikecote]

@gmmorris and @timroes have spoken and know about the blockers for this.

@gmmorris, can you provide an update on what was discussed?

[timroes]

@AlonaNadler (I rephrased the point you were asking about a bit). The current Alert is using Elasticsearch queries directly. We use in Discover (and most other Analytics Apps) the higher level abstraction of Search Source to communicate with ES. We'd need an alert executor using Search Source not raw Elasticsearch DSL, since Search Source handles a lot of "higher level" logic for us (legacy support for scripted fields, runtime field support, filtering fields, etc. etc.).

Peter, Mika and I also discussed that before the Elasticsearch DSL alerter was created, that this won't be enough for our use-case and we'll still follow our original plan of having a search source specific alerter that we use in Discover (and potentially other apps using search source), that we'll need to build in context of the Discover Alert, but App Services will take ownership afterwards of.

@mikecote Not sure if you've already got an answer offline to this: but we just discussed that the above mentioned 2 issues have a priority for us, to be able to implement this, and Gidi suggested, that we can build the data access potentially just ourselves. Haven't discussed much more technical things. Since you already commented on the data service issue, I'd like to continue the discussion around it there.

[timroes]

Closing in favor of #117532