crowdstrike: add support for host info enrichment #8474

efd6 · 2023-11-13T09:44:01Z

Proposed commit message

See title.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

The fdr datastream offers both s3 and logfile inputs. Limitations in elastic-package prevent us from being able to have two system test infrastructure deployments, so only the ~~logfile~~ s3 input is tested.

How to test this PR locally

Make the following change:

diff --git a/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml b/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml
index b795fcdeb..6e1f17f7a 100644
--- a/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml
+++ b/packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/env.yml
@@ -7,3 +7,4 @@ services:
       - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}
       - AWS_PROFILE=${AWS_PROFILE}
       - AWS_REGION=${AWS_REGION:-us-east-1}
+      - TF_VAR_eventbridge_role_arn=arn:aws:iam::144492464627:role/eb-scheduler-role-20231101165501426500000001

Then:

$ export AWS_DEFAULT_PROFILE=elastic-siem
$ aws-mfa --profile=elastic-siem
$ eval $(grep ^aws ~/.aws/credentials | gsed -r 's/^(aws[^ ]+) = (.*)$/export \U\1\E=\2/g')

(gsed is GNU sed so if on macos you will need to install that, is on linux s/gsed/sed/ in the command above)

Then test with elastic-package as normal.

For convenience (because the AWS setup was not trivial), the TF graph is here.

Related issues

Closes Crowdstrike FDR hostname/username enrichment #2816

Screenshots

elasticmachine · 2023-11-13T09:47:28Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-28T19:34:05.461+0000
Duration: 20 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	34
Skipped	0
Total	34

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-11-13T10:11:53Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`15/15`)	💚
Classes	100.0% (`15/15`)	💚
Methods	95.918% (`94/98`)	👎 -4.082
Lines	88.261% (`3609/4089`)	👎 -11.739
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-11-13T21:29:11Z

/test

elasticmachine · 2023-11-15T01:33:35Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kcreddy · 2023-11-16T10:32:36Z

packages/crowdstrike/data_stream/fdr/manifest.yml

+        required: true
+        show_user: true
+        title: Enrich Host Metadata
+        description: Uses data in aidmaster to add host information to events. The aidmaster blob must string "aidmaster" in its path and the FDR Notification Parsing Script must sort events so that aidmaster events appear first in the stream.


Suggested change

description: Uses data in aidmaster to add host information to events. The aidmaster blob must string "aidmaster" in its path and the FDR Notification Parsing Script must sort events so that aidmaster events appear first in the stream.

description: Uses data in aidmaster to add host information to events. The aidmaster blob must contain string "aidmaster" in its path and the FDR Notification Parsing Script must sort events so that aidmaster events appear first in the stream.

kcreddy · 2023-11-16T10:33:32Z

packages/crowdstrike/data_stream/fdr/manifest.yml

+        required: true
+        show_user: true
+        title: Host Metadata TTL
+        description: The period of time that host metadata is considered valid for.


Can you also add valid time units?

kcreddy · 2023-11-16T10:37:14Z

packages/crowdstrike/data_stream/fdr/agent/stream/aws-s3.yml.hbs

+    target: crowdstrike
+- if:
+    contains:
+      log.file.path: aidmaster


Is the field log.file.path going to exist for S3 input in this case?

kcreddy · 2023-11-16T10:38:23Z

packages/crowdstrike/data_stream/fdr/manifest.yml

+        required: true
+        show_user: true
+        title: Host Metadata TTL
+        description: The period of time that host metadata is considered valid for.


time units here as well

kcreddy

LGTM 👍🏼

andrewkroh

Have you manually tested this using the aws-s3 input?

The fdr datastream offers both s3 and logfile inputs. Limitations in elastic-package prevent us from being able to have two system test infrastructure deployments, so only the logfile input is tested.

If we have to sacrifice the logfile testing to get aws-s3 system tests then let's do it. The aws-s3 input is what most users are running with.

andrewkroh · 2023-11-21T00:52:20Z

packages/crowdstrike/data_stream/fdr/agent/stream/stream.yml.hbs

  - {{path}}
 {{/each}}
+scan:
+  sort: filename
+  order: asc


This will be a weak ordering guarantee because the scan can start concurrent readers. You could set harvester_limit: 1 to control concurrency. I don't think the log file input is used in production by anyone so this matters very little.

Thanks, I was worried about that. We can't limit though; making the limit 1 results in a deadlock.

packages/crowdstrike/data_stream/fdr/manifest.yml

packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/main.tf

Error: error running package system tests: could not complete test run: could not add data stream config to policy: could not add package to policy; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Package policy is invalid: inputs.aws-s3.streams.crowdstrike.fdr.vars.fdr_parsing_script: Invalid YAML format"}

andrewkroh

LGTM. Thanks for getting the system testing setup for the aws-s3 input.

I left some minor suggestions related to the test config.

andrewkroh · 2023-11-28T13:32:19Z

packages/crowdstrike/data_stream/fdr/_dev/test/system/test-default-config.yml

+    secret_access_key: "{{AWS_SECRET_ACCESS_KEY}}"
+    session_token: "{{AWS_SESSION_TOKEN}}"
+    queue_url: "{{TF_OUTPUT_queue_url}}"
+    preserve_original_event: true


I think we should be able to know the expected number of events so can we add the

assert: hit_count: N

andrewkroh · 2023-11-28T13:36:43Z

packages/crowdstrike/data_stream/fdr/_dev/test/system/test-default-config.yml

@@ -0,0 +1,9 @@
+input: aws-s3
+wait_for_data_timeout: 20m


This wait period seems quite high. Once Terraform completes its provisioning an SQS notification should be sent after about 1 minute. I think the default of 10m should be fine.

The 20m value appears to be commonly used, but agreed. Will change.

elasticmachine · 2023-11-28T20:35:34Z

Package crowdstrike - 1.26.0 containing this change is available at https://epr.elastic.co/search?package=crowdstrike

efd6 added enhancement New feature or request Team:Security-External Integrations Integration:crowdstrike CrowdStrike labels Nov 13, 2023

efd6 self-assigned this Nov 13, 2023

efd6 force-pushed the 2816-crowdstrike branch from 9042287 to 0457f4b Compare November 13, 2023 09:44

efd6 force-pushed the 2816-crowdstrike branch 5 times, most recently from 2751001 to db9ce4a Compare November 15, 2023 01:00

efd6 marked this pull request as ready for review November 15, 2023 01:33

efd6 requested a review from a team as a code owner November 15, 2023 01:33

kcreddy reviewed Nov 16, 2023

View reviewed changes

efd6 requested a review from kcreddy November 16, 2023 19:48

kcreddy approved these changes Nov 17, 2023

View reviewed changes

andrewkroh reviewed Nov 21, 2023

View reviewed changes

efd6 force-pushed the 2816-crowdstrike branch 3 times, most recently from e5a1d0d to affaee3 Compare November 23, 2023 00:30

andrewkroh reviewed Nov 27, 2023

View reviewed changes

packages/crowdstrike/data_stream/fdr/_dev/deploy/tf/main.tf Outdated Show resolved Hide resolved

efd6 added 6 commits November 28, 2023 08:47

crowdstrike: add support for host info enrichment from logfile input

b09287a

crowdstrike: add support for host info enrichment from s3 input

5b16b1a

address pr comments

b8a6943

address pr comment

8837ab0

add tf for aws infrastructure

421df55

fix terraform queue definition

867de0a

work around absence of optional chaining support

8770ae4

efd6 force-pushed the 2816-crowdstrike branch from affaee3 to 8770ae4 Compare November 27, 2023 22:48

use andrew's tf definitions

c305dc4

efd6 force-pushed the 2816-crowdstrike branch from 65571d8 to c305dc4 Compare November 27, 2023 23:18

andrewkroh and others added 5 commits November 27, 2023 19:22

Make role_arn configurable for use with elastic-siem account

9a125a8

reduce code and fix field type

ab317e5

fixup fields and generate docs

6525cf0

remove local testing var

036d08b

efd6 requested a review from andrewkroh November 28, 2023 03:43

clarify local testing comment

c027048

efd6 force-pushed the 2816-crowdstrike branch from b621c88 to c027048 Compare November 28, 2023 03:52

andrewkroh approved these changes Nov 28, 2023

View reviewed changes

address pr comments

118b147

efd6 merged commit 6f5b9b7 into elastic:main Nov 28, 2023
4 checks passed

efd6 mentioned this pull request Nov 28, 2023

Crowdstrike FDR hostname/username enrichment #2816

Closed

efd6 mentioned this pull request Dec 11, 2023

Crowdstrike FDR user enrichment #8685

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crowdstrike: add support for host info enrichment #8474

crowdstrike: add support for host info enrichment #8474

efd6 commented Nov 13, 2023 •

edited

Loading

elasticmachine commented Nov 13, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Nov 13, 2023 •

edited

Loading

efd6 commented Nov 13, 2023

elasticmachine commented Nov 15, 2023

kcreddy Nov 16, 2023

kcreddy Nov 16, 2023

kcreddy Nov 16, 2023

efd6 Nov 16, 2023

kcreddy Nov 16, 2023

kcreddy left a comment

andrewkroh left a comment

andrewkroh Nov 21, 2023

efd6 Nov 21, 2023

andrewkroh left a comment

andrewkroh Nov 28, 2023

andrewkroh Nov 28, 2023 •

edited

Loading

efd6 Nov 28, 2023

elasticmachine commented Nov 28, 2023

	description: Uses data in aidmaster to add host information to events. The aidmaster blob must string "aidmaster" in its path and the FDR Notification Parsing Script must sort events so that aidmaster events appear first in the stream.
	description: Uses data in aidmaster to add host information to events. The aidmaster blob must contain string "aidmaster" in its path and the FDR Notification Parsing Script must sort events so that aidmaster events appear first in the stream.

crowdstrike: add support for host info enrichment #8474

crowdstrike: add support for host info enrichment #8474

Conversation

efd6 commented Nov 13, 2023 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Nov 13, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Nov 13, 2023 • edited Loading

🌐 Coverage report

efd6 commented Nov 13, 2023

elasticmachine commented Nov 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Nov 28, 2023

efd6 commented Nov 13, 2023 •

edited

Loading

elasticmachine commented Nov 13, 2023 •

edited

Loading

elasticmachine commented Nov 13, 2023 •

edited

Loading

andrewkroh Nov 28, 2023 •

edited

Loading