[discuss] Passing data from one action to subsequent actions

[mikecote]

As mentioned here: #45023 (comment). Folks will want a way to get a url to a generated ticket to use in a subsequent action. Eg, generate a GH issue, then post a slack message with the url to that GH issue.

[elasticmachine]

Pinging @elastic/kibana-stack-services (Team:Stack Services)

[mikecote]

@pmuellr was mentioning we could leverage expressions to narrate the flow of the actions and their dependencies.

[pmuellr]

Changing this to discuss as we don't have any requirements for this from anyone, yet. But believe we will eventually.

As Mike mentions ^^^, one of my thoughts on "Kibana expressions as alerts" is that an expression could both perform the "alert executor" role AND also perform the relevant actions. In fact, it may be hard / unwieldy to have a notion of just a Kibana expression as an alert, without the actions, because then we won't be able to identify the "context variables" that alert would be setting. If we added the ability to actually invoke connectors within the expression, then we get out of the business of having to have "context variables" (as we use them today) associated with such alerts. This implies that we can also invoke connectors via expressions, which seems like a pretty reasonable thing to do, except for addressing them via their UUID which would be painful - perhaps taking a name, or even git-style "first 7 chars of the SHA" kind of short forms.

[chrisronline]

@pmuellr Do you think this is still valuable? I'm wondering if we've heard requests like this recently. I'm wondering how much time we should spend researching what it would take to do this, as I think we'd want to maintain some parallelism of actions but perhaps allow some opt-in sequential-ness and support passing in the results from the previous action execution - maybe we can start grouping actions into a single action and a grouped action can handle something like this?

[pmuellr]

My current thinking is that we'd leverage Kibana expressions to do this. Somehow expose connectors in an expression context, such that you could execute them in the expression, and then capture the results. Which could then be used in a subsequent connector execution in the same expression, which used that result.

I think I was originally hopeful that we could somehow just "pipeline" actions together, but it's not clear how you'd tell one action to use a particular piece of data from the output of another action. Especially if there were > 2 actions "chained together". And likely it might need some formatting or such.

So, I've been thinking more of a "low code" story like Kibana expressions might work for this. Allow some simple chaining, along with data shaping with existing kibana expression functions.

The fundamental blocker for this right now is that we don't support any output for connectors, beyond pass/fail and an error message.

And I'm thinking this would be a new type of connector anyway, which would mean we could keep the existing semantics, and still run everything "virtually simultaneously". These are just special connectors that allow finer grained access to other connectors, with their own control flow.

So, I think the requirement is still valid - but how we get there is kinda unknown.

[jvalente-salemstate]

With the feature sets elastic has added since this was suggested, I think this merits another evaluation.

My primary use case is within Elastic Security and this would bring SOAR functionality within elastic and is a value-add when comparing security platforms. This would also enhance the experience those using one of the existing SOAR connectors as the output from the previous actions can be included.

This, and workflows even without other connectors, is already technically possible if one gets creative with Logstash and the write to index or webhook connector. Ingest the triggering event and throughout a pipeline transform events or make other calls with the http filter plugin and so forth. Drop the configuration in centralized pipeline management and all set.

ES ingest pipelines already have a UI in Kibana that could be a base for doing it all within there. They're missing some of the filters that'd make it work in Logstash and http plugins might help.

Or maybe something entirely different but at least two options that are already almost all the way there exist within Elastic's products.