crowdstrike: add userinfo enrichment support and map fields to ECS

[efd6]

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Make the following change:

diff --git a/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml b/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml
index b795fcdeb..6e1f17f7a 100644
--- a/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml
+++ b/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml
@@ -7,3 +7,4 @@ services:
       - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}
       - AWS_PROFILE=${AWS_PROFILE}
       - AWS_REGION=${AWS_REGION:-us-east-1}
+      - TF_VAR_eventbridge_role_arn=arn:aws:iam::144492464627:role/eb-scheduler-role-20231101165501426500000001

Then:

$ export AWS_DEFAULT_PROFILE=elastic-siem
$ aws-mfa --profile=elastic-siem
$ eval $(grep ^aws ~/.aws/credentials | gsed -r 's/^(aws[^ ]+) = (.*)$/export \U\1\E=\2/g')

(gsed is GNU sed so if on macos you will need to install that, is on linux s/gsed/sed/ in the command above)

Then test with elastic-package as normal.

Related issues

• For Crowdstrike FDR hostname/username enrichment #2816 and Crowdstrike FDR Mapping Enhancement #4040
• Close Crowdstrike FDR user enrichment #8685

Screenshots

[efd6]

aid changed to hit an event that has both host and user metadata keys.

[efd6]

timestamp changed to bring it to the front of the queue and become the sample event.

[efd6]

125 events, 5 aidmaster records and 1 userinfo record.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-12-19T11:21:12.300+0000

• Duration: 25 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	35
Skipped	0
Total	35

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (15/15)	💚
Classes	100.0% (15/15)	💚
Methods	95.918% (94/98)	👎 -4.082
Lines	87.718% (3628/4136)	👎 -10.346
Conditionals	100.0% (0/0)	💚

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[kcreddy]

Can ignore_failure be removed from this processor, since you already have a conditional null check?

[efd6]

Fair enough. Was following the pattern of the appends above.

[kcreddy]

Is this also a good candidate for related.hosts? Not sure, just checking.

[efd6]

This makes sense. Will do that.

[kcreddy]

LGTM 👍🏼

[elasticmachine]

Package crowdstrike - 1.28.0 containing this change is available at https://epr.elastic.co/search?package=crowdstrike

[elasticmachine]

Package crowdstrike - 1.28.0 containing this change is available at https://epr.elastic.co/search?package=crowdstrike

[mbudge]

After upgrading 2 days ago, many ProcessRollup2 events are missing the username within the last 24 hours. It looks like the user enrichment isn't working as expected.

Here's some metrics which show how the user enrichment is working over the last 24 hours.

event.action:ProcessRollup2 and user.id: S-1-5-21* and not user.name:* = 1,678,329 hits

event.action:ProcessRollup2 and user.id: S-1-5-21* and user.name:* = 292,066 hits

Here's some other events which could improve the enrichment process. Crowdstrike previously told us to use the UserIdentity event for username enrichment.

UserIdentity
UserLogon
UserLogoff
UserInformationEtw

UserIdentity
Description
Platforms: Linux, Forensics
Platforms: Windows, macOS
The UserIdentity event is generated when a user logs in to a host. It conveys important security-related characteristics associated with a user to the CrowdStrike cloud, such as the user name. It’s normally generated once per security principal, and is thus not on its own a sign of a suspicious activity.

UserLogon
Description
Platforms: Linux, Windows, macOS
This event is generated when a user logs on to a host.

UserLogoff
Description
Platforms: Linux, ChromeOS
Platforms: Windows, macOS
This event is generated when a user logs off from a host.

UserInformationEtw
Description
Platforms: Windows
An event that indicates the password of a user was changed or set and other user information taken from UserLogonEtw.

[mbudge]

Just a reminder this doesn't work.

We have 20 million logs in the last 24 hours with where crowdstrike.AuthenticationId:*

17 million logs have no user.name which means enrichment isn't working for the majority.

Another view is ProcessRollup2

crowdstrike.AuthenticationId:* and not user.name:* = 14,092,401 hits where enrichment isn't working

crowdstrike.AuthenticationId:* and user.name:* = 1,178,348 hits

Crowdstrike told us to use the UserIdentity event for enrichment.

[tsqrd]

To add on to @mbudge, that UserInfo dataset only exists if you have Falcon Discover. Additionally the current implementation only "works" for Windows. Seems like it would be feasible to build the cache based on UserLogon event types, which include the Host/AID-->SID/UID-->UserName mapping required for the enrichment to work properly.

[Rockso]

@mbudge @tsqrd I took a try at at improving the user enrichment by modifying the elastic-agent processor that comes in the fdr config to utilize UserIdentity and UserLogon events. I opted to build the userinfo (useridentity_Win/useridentity_Lin) caches with the aid_SID/UID (windows/Linux) with the reported username and then match it up to the aid_SID/UID to other events if those two values exist, but not overwriting events that match but already have a UserName field. Feel free to tweak as necessary. So far it's been working for me using the default ingest pipelines.

fdr.enrich.processor.txt

[mbudge]

@mbudge @tsqrd I took a try at at improving the user enrichment by modifying the elastic-agent processor that comes in the fdr config to utilize UserIdentity and UserLogon events. I opted to build the userinfo (useridentity_Win/useridentity_Lin) caches with the aid_SID/UID (windows/Linux) with the reported username and then match it up to the aid_SID/UID to other events if those two values exist, but not overwriting events that match but already have a UserName field. Feel free to tweak as necessary. So far it's been working for me using the default ingest pipelines.

fdr.enrich.processor.txt

I'm testing the processor now. Another improvement for UserIdentity event caching could be crowdstrike.info.host.MachineDomain as well as host.name, where MachineDomain != "none". Obvs this would cache the user.name the first time it's seen on the domain.

[Rockso]

I did consider adding the domain name information but didn't think it was overly useful in my use cases as you can differentiate a local user from a domain user based on the SID length. But yes it would be useful for others.

Also, I'm noticing that when UserID/UserName mapping is not in the cache the enrichment script doesn't add the hostname field. I Need to build some logic around that.... Or reorder it.... Still investigating but I did notice this issue come up with missing host.name enrichment for certain conditions.