[Request] [Draft] Document impact of using logsDB for security users

[approksiu]

[Work in progress]

Description

In 8.17 the new index mode will be available for Elastic users. @yctercero trying to figure out if in 9.0 it will be on by default. This mode will be on by default for Serverless users (without the ability to change it?).

We need to provide information to the security users about it's impact:

Alerts

As part of the logsDB changes, certain field types will be stored in ES differently than their original format. During alert creation, we copy the original event over into the alert in order to keep reference to the original data. However, we now need to be more specific, perhaps, about what we mean by "original". We will copy over the fields as given to us by ES, which may now be in a different format than the original source for:

• dotted field names
• Array of objects versus object with nested array
• Duplicate values

@approksiu can you confirm if we make any kind of assurances about maintaining original copies of source events? We'll need to update any language that makes those kind of assurances to note that this is not the case when logsDB is enabled.

• we don't seem to be mentioning anything about it, except warning users against using _source in the Alert ECS schema doc.

Affected rule types would be:

• Custom query
• EQL non sequence
• Non aggregate rule types. There are some rule types where we already do not maintain copies of the original source (Threshold, ML, EQL sequence).

Runtime fields

Runtime fields that reference source may be affected. The following are examples of accessing _source that work:

"source": """  emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """ 
"source": """  emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value );  """
"source": """  emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """
"source": """  emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """

The following will not work with synthetic source (logsDB enabled):

"source": """  emit(params._source['agent.name'] + "_____" + doc['agent.name'].value );  """

More details on this found in this issue and doc'd here.

Actions

No anticipated issues, but there may be an edge case we haven't thought of. Per @vitaliidm 's doc:

Properties of alert are accessible in action template when synthetic source is enabled.
The Resulting alert is constructed from a synthetic source, which means any dot notation fields will be converted to a nested object. But the template works for both cases.
Moreover, when the original event is a nested object(when the synthetic source is disabled) it will also be stored in the resulting alert as a nested object. So, there should not be any impact on functionality.

There could however happen to be a mistmatch between how users expect field values to be formatted based on what we outlined in the "Alerts" section above. If users ship out their alerts using actions and then have logic outside our system that is based on fields being in the format of the original source, their flows may be broken. In particular, affected fields would be array of objects (see description of differences here).

Background & resources

• PRs:
• Issues/metas:
• Point of contact: @yctercero
• Test environments:
• Details: https://docs.google.com/document/d/1lNb2V_HEot90W913iJTyRv4l90WCiWdtslexUMn35IE/edit#heading=h.m5d9irssane6

Which documentation set does this change impact?

ESS

ESS release

8.17

Serverless release

• will be default behavior for GA

Feature differences

no differences

API docs impact

No impact to our APIs.

Prerequisites, privileges, feature flags

Should make reference to ES team docs for more info on logsdb - https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-source-field.html#synthetic-source

[approksiu]

@paulewing could you please add information about the impact on the security analyst workflows?

[approksiu]

@caitlinbetz https://www.elastic.co/guide/en/security/8.15/add-exceptions.html#ex-nested-conditions this workflow might be affected by logsdb. Could you please check with your team what needs to be documented?

[yctercero]

@marshallmain @YulNaumenko @vitaliidm would you mind reviewing the above to ensure accuracy? Let me know if I missed anything or any information is now out of date.

[approksiu]

Here's the draft document cc @yctercero @nastasha-solomon @vitaliidm @jmikell821

I think it makes sense to add this as a section at the end of this page: https://www.elastic.co/guide/en/security/current/detection-engine-overview.html.
The other option is to make a separate page just to document this impact, and reference it from the end of this page: https://www.elastic.co/guide/en/security/current/detection-engine-overview.html.

[gaurav-elastic]

Here's the draft document cc @yctercero @nastasha-solomon @vitaliidm @jmikell821

I think it makes sense to add this as a section at the end of this page: https://www.elastic.co/guide/en/security/current/detection-engine-overview.html. The other option is to make a separate page just to document this impact, and reference it from the end of this page: https://www.elastic.co/guide/en/security/current/detection-engine-overview.html.

Let's go with Option 2 : make a separate page just to document this impact, and reference it from the end of this page