[Security Solution] Provide a workflow for adding runtime fields to an alerts index for unmapped fields

[andrew-goldstein]

Unmapped fields in an alerts index, (e.g. .siem-signals-default) are not displayed in the Detection alerts table, as shown in the screenshot below, until users take one of the following actions:

• manually add a runtime field to the alerts index (for the unmapped field), as illustrated by the runtime fields documentation
• manually add a mapping to the alerts index (for the unmapped field) as illustrated by the mapping documentation, and re-index the alert data

In the screenshot above, the user.effective.group.id field displays the numeric value 0 in the Alert details view, but appears as a - in the alerts table, because there is no mapping for this field in the alerts index.

Since adding a runtime field doesn't require re-indexing the existing alert data, it's the preferable action for users to take when compared with manually adding a mapping to the alerts index (e.g. .siem-signals-default).

The purpose of this enhancement request is to provide a UI workflow that assists the user in adding a runtime field to the alerts index, which would enable those fields to be displayed in the Detections alerts table. The workflow could, for example, automatically suggest the correct data type for the new runtime field by allowing the user to select which source index contains the correct definition of the field.

Kibana/Elasticsearch Stack version:

7.12.0

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[andrew-goldstein]

@MikePaquette, @dontcallmesherryli this is the enhancement request that we discussed today with @stephmilovic @rylnd and @XavierM

EDIT: Per our discussion, I also created the following bug

[marshallmain]

As an alternative to adding each new field to the mapping permanently, perhaps timeline could add runtime fields for unmapped columns in the search request itself? https://www.elastic.co/guide/en/elasticsearch/reference/master/runtime-search-request.html

If timeline can detect which fields are unmapped and attempt to add them into the query as runtime fields before loading the table then we can provide this enhancement without the additional user effort of adding the field to the mapping and also avoid leaving extra runtime fields in the mapping indefinitely.

[andrew-goldstein]

As an alternative to adding each new field to the mapping permanently, perhaps timeline could add runtime fields for unmapped columns in the search request itself? https://www.elastic.co/guide/en/elasticsearch/reference/master/runtime-search-request.html

If timeline can detect which fields are unmapped and attempt to add them into the query as runtime fields before loading the table then we can provide this enhancement without the additional user effort of adding the field to the mapping and also avoid leaving extra runtime fields in the mapping indefinitely.

Thanks for suggesting this approach @marshallmain! We'll definitely consider it when we discuss the design and implementation.

[maederm]

I believe until this recent change Security Solution didn't expect a non-conflicting data mapping on the indices that were queried.
I'm a bit worried that with changing the mapping of .siem-signals-* at some point we could lose alerts because of mapping conflicts (how would Security Solution store an alert of a different index with a mapping conflict?).
So I support the proposal of @marshallmain which doesn't require to make changes on the mapping.

[andrew-goldstein]

I'm closing this now-obsolete issue, because adding unmapped fields will now work as expected (the value of the unmapped field will be displayed), with this fix from @XavierM in this PR: #99130