[Security Solution] Alerts table does not show threat enrichments values

[e40pud]

Describe the bug:

This issue was discovered while fixing skipped cypress test #162818

Here is the link to the slack discussion which describes all the details and root causes https://elastic.slack.com/archives/C056TQ5J81Y/p1692115257121509

Steps to reproduce:

1. Create threat indicator rule which generates alerts with threat enrichments
2. Open Rule details page
3. Scroll down to Alerts Table
4. Add threat.enrichments.* fields

Current behavior:

Alerts table does not show threat.enrichments.* fields values and instead displays -

Expected behavior:

Alerts table should show threat.enrichments.* fields values if those exist

Screenshots (if relevant):

Screen.Recording.2023-08-16.at.10.53.04.mov

In this recording you can see that alerts table does not show any of threat.enrichments.* fields values. Second part of the recording shows that values exist by adding the same fields in timeline.

cc @michaelolo24

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[michaelolo24]

@e40pud thanks for raising this issue. I believe this was fixed by this PR here: #165040. If everything looks good to you on main can you close? Thanks!

[e40pud]

Hey @michaelolo24 I just checked latest changes from main branch and I still see the issue.

Here in screenshot, I opened alert's details flyout for the first alert in the table and there is data for the threat.enrichments.matched.field inside the flyout but table does not reflect that and continue showing - instead.

[michaelolo24]

Okay, we'll take a look thanks for double checking @e40pud !

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[yctercero]

Just a note - when this is addressed, please unskip cypress test - #162818

[michaelolo24]

This is an existing issue with our tables in kibana (both our security tables and Discover (fyi @davismcphee)).

Currently for nested fields, they are returned as:

"threat.enrichments": [
  {
      "indicator.provider": [
        "some-provider"
      ]
  }
]

while the tables expect them in the shape:

threat.enrichments.indicator.provider: ["some-provider"]

In the timeline application, we flatten all of the fields to render them properly in the table, so as a workaround, I would ask the user to use timeline, but this is something we should resolve at the kibana level with a common renderer for nested fields.

@davismcphee are you aware of this issue and would you be interested in using a common nested field cell renderer?

[davismcphee]

@michaelolo24 Thanks for the ping. I'm aware that nested field handling in general isn't great, but I wasn't aware of this specific issue. I tested Discover with a nested field and object field to compare, and I agree it seems broken:

The nested field itself displays as expected, but the inner fields are blank. Although I'm not actually sure what should be expected here for inner fields. Do you know how Timeline would handle it?

We also flatten fields in Discover using a shared flattenHit utility, so maybe there's something we could do at that level to address it. But also a common renderer (maybe a field formatter?) is an interesting idea too.

cc @elastic/kibana-data-discovery for additional thoughts.

[michaelolo24]

@michaelolo24 Thanks for the ping. I'm aware that nested field handling in general isn't great, but I wasn't aware of this specific issue. I tested Discover with a nested field and object field to compare, and I agree it seems broken:

The nested field itself displays as expected, but the inner fields are blank. Although I'm not actually sure what should be expected here for inner fields. Do you know how Timeline would handle it?

We also flatten fields in Discover using a shared flattenHit utility, so maybe there's something we could do at that level to address it. But also a common renderer (maybe a field formatter?) is an interesting idea too.

cc @elastic/kibana-data-discovery for additional thoughts.

This is our implementation, and I haven't dug into it to confirm all the details of it's expected functionality, but using a common util makes the most sense in my mind. Fyi @XavierM

[davismcphee]

After discussing it with my team, it seems this is expected behaviour in 7.12+ according to the "Differences in how nested fields are displayed" section of this blog post. With that said, if this is expected I'm not sure why the leaf fields still appear in our field list if they can't be used. I'll try to figure out why they show up and if it's a bug that they do, and follow up here.

[michaelolo24]

After discussing it with my team, it seems this is expected behaviour in 7.12+ according to the "Differences in how nested fields are displayed" section of this blog post. With that said, if this is expected I'm not sure why the leaf fields still appear in our field list if they can't be used. I'll try to figure out why they show up and if it's a bug that they do, and follow up here.

Sorry for the delayed response @davismcphee, got bogged down with other review work a bit. Yea, it seems like we have the same issue here, but I'm also curious as to how the fields appear not just in the fields list bust also in the flyout fields table. I know we do some transformations on the security side to help that work, but I'm not sure if the same happens in discover?

[yctercero]

@michaelolo24 @davismcphee sorry to jump in here, but any updates? If this is expected behavior are there any changes we need to make on our end?

[damianpfister]

Appears this is still an issue in 8.14.1 of the stack, so keen to understand if there has been any further discussion around ways around this?

[LANopop]

any chance to get this fixed? thanks