[CrowdStrike] Add support of REST API for Alert and Host

[mohitjha-elastic]

Type of change

• Enhancement

What does this PR do?

1. Add Support of Alert and Host Data Stream using REST API.
2. Add Support of Scheduled Report Notification events in Falcon Data Stream.
3. Convert event.id to string in Mobile Detection Summary Pipeline.
4. Remove some empty spaces from files.
5. Added conditions to avoid the error while renaming the custom fields to cloud.* ECS fields in the CSPM Events.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

All changes

• Change follows the contributing guidelines
• Supported versions of the monitoring target are documented
• Supported operating systems are documented (if applicable)
• Integration or System tests exist
• Documentation exists
• Fields follow ECS and naming conventions
• At least a manual test with ES / Kibana / Agent has been performed.
• Required Kibana version set to: ^8.11.0

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/crowdstrike directory.
Run the following command to run tests.
elastic-package test -v

Automated Test

verbose_crowdstrike.txt

Related issues

• Closes [Crowdstrike] CSPM Events pipeline failure during account id rename processor #7171

Screenshots

[jamiehynds]

This datastream also supports events via Event Stream? If that's the case we should make this more descriptive as it currently sounds like it's limited to alerts.

[mohitjha-elastic]

This PR contains support for Alert and Host endpoint only.
Waiting for the Customer's feedback on the Event Stream API Data Collection then we can plan to add the support in this PR or maybe in a separate PR.

[jamiehynds]

Does a customer need to ensure the Event Stream API is enabled also? Do they have to contact CrowdStrike support in order to enable the API?

Worth noting that a user who tested the pre-release of this package had to grant “read” scope to “alerts” and “hosts” via Crowdstrike in order to get the data. Are these permissions required on our end? If so, we should be documenting it.

[mohitjha-elastic]

Yes. Customers need to add the read scopes for the endpoints using the CrowdStrike instance. As this PR only supports Alert and Host endpoint hence added details of the required scopes for the same inside the Setup section.

[jamiehynds]

We should adjust this if we no longer require the Falcon SIEM Connector. While users can still use it, this update provides the ability to ingest data directly via the API, avoiding the need for SIEM Connector. If possible, we should also include language to explain API vs SIEM Connector to avoid any confusion.

[mohitjha-elastic]

Currently, we have not removed the support of Falcon SIEM connector. Users will have two options -

1. Collect logs via API
2. Collect logs via Falcon SIEM connector

Let me know if you want me to remove the support of Falcon SIEM connector.

Agrees with you that we can include language to explain API vs SIEM connector to avoid the confusion. Will update that.

[jamiehynds]

@mohitjha-elastic definitely don't remove the SIEM Connector support, as that's what user rely on today. But definitely some language to make it clear that we support both SIEM Connector and API, and the SIEM Connector is not required if you're ingest directly via the API.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

/test

[efd6]

Would it be reasonable to pretty format this JSON to reduce diff noise in the future? This line is almost 10kB long.

I'm wondering if an option in stream to allow minified JSON to be returned when the body is structured for readability.

[efd6]

Separate change entries?

Suggested change

	- description: Add the Input Support of REST API (Alert, Host, and Event Streams) and update the minimum kibana version to 8.10.1.
	type: enhancement
	link: https://github.com/elastic/integrations/pull/8790
	- description: Add support for Alert, Host, and Event Streams API endpoints.
	type: enhancement
	link: https://github.com/elastic/integrations/pull/8790
	- description: Update minimum kibana version to 8.10.1.
	type: enhancement
	link: https://github.com/elastic/integrations/pull/8790

[efd6]

Consistent indentation helps readability (untested changes)

Suggested change

	(
	state.with(
	(
	!state.want_more ?
	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=0&limit="
	+ string(state.batch_size) + "&filter=timestamp:>\"" +
	(
	has(state.cursor) && has(state.cursor.last_timestamp) && state.cursor.last_timestamp != null ?
	state.cursor.last_timestamp + "\""
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339) + "\""
	)
	)
	:
	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=" + string(state.offset) +
	"&limit=" + string(state.batch_size) + "&filter=timestamp:>\"" +
	(
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	state.cursor.first_timestamp + "\""
	:
	"\""
	)
	)
	).do_request().as(resp, bytes(resp.Body).decode_json().as(body, {
	"resources": has(body.resources) && body.resources.size() > 0 ? body.resources : "",
	"want_more": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total),
	"offset": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total) ?
	int(state.offset) + int(body.resources.size())
	:
	0,
	"url": state.url,
	"batch_size": state.batch_size,
	}))
	).as(state, state.with(
	has (state.resources) && state.resources != "" ?
	post_request(
	state.url + "/alerts/entities/alerts/v1",
	"application/json",
	{"ids": state.resources }.encode_json()
	)
	.do_request().as(resp, bytes(resp.Body).decode_json().as(inner_body, {
	"events": inner_body.resources.map(e, {
	"message": e.encode_json(),
	}),
	"cursor": {
	"last_timestamp": (
	has(inner_body.resources) && inner_body.resources.size() > 0 ?
	inner_body.resources.map(e, e.timestamp).max()
	:
	(
	has(state.cursor) && has(state.cursor.last_timestamp) ?
	state.cursor.last_timestamp
	:
	null
	)
	),
	"first_timestamp": (
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	( state.want_more ? state.cursor.first_timestamp : state.cursor.last_timestamp )
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339)
	),
	},
	}))
	:
	{}
	)
	)
	)
	(
	state.with(
	(
	!state.want_more ?
	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=0&limit=" + string(state.batch_size) + "&filter=timestamp:>\"" + (
	has(state.cursor) && has(state.cursor.last_timestamp) && state.cursor.last_timestamp != null ?
	state.cursor.last_timestamp + "\""
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339) + "\""
	))
	:
	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=" + string(state.offset) + "&limit=" + string(state.batch_size) + "&filter=timestamp:>\"" + (
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	state.cursor.first_timestamp + "\""
	:
	"\""
	))
	).do_request().as(resp, bytes(resp.Body).decode_json().as(body, {
	"resources": has(body.resources) && body.resources.size() > 0 ? body.resources : "",
	"want_more": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total),
	"offset": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total) ?
	int(state.offset) + int(body.resources.size())
	:
	0,
	"url": state.url,
	"batch_size": state.batch_size,
	}))
	).as(state, state.with(
	!has(state.resources) || state.resources == "" ? {} :
	post_request(
	state.url + "/alerts/entities/alerts/v1",
	"application/json",
	{"ids": state.resources}.encode_json()
	).do_request().as(resp, bytes(resp.Body).decode_json().as(inner_body, {
	"events": inner_body.resources.map(e, {
	"message": e.encode_json(),
	}),
	"cursor": {
	"last_timestamp": (
	has(inner_body.resources) && inner_body.resources.size() > 0 ?
	inner_body.resources.map(e, e.timestamp).max()
	: has(state.cursor) && has(state.cursor.last_timestamp) ?
	state.cursor.last_timestamp
	:
	null
	),
	"first_timestamp": (
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	(state.want_more ? state.cursor.first_timestamp : state.cursor.last_timestamp)
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339)
	),
	},
	}))
	)
	)
	)

[efd6]

Final new line

[efd6]

Suggested change

	(
	state.with(
	(
	!state.want_more ?
	request("GET", state.url + "/devices/queries/devices/v1?sort=modified_timestamp.asc&offset=0&limit=" + string(state.batch_size) + "&filter=modified_timestamp:>\"" +
	(
	has(state.cursor) && has(state.cursor.last_timestamp) && state.cursor.last_timestamp != null ?
	state.cursor.last_timestamp + "\""
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339) + "\""
	)
	)
	:
	request("GET", state.url + "/devices/queries/devices/v1?sort=modified_timestamp.asc&offset=" + string(state.offset) +
	"&limit=" + string(state.batch_size) + "&filter=modified_timestamp:>\"" +
	(
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	state.cursor.first_timestamp + "\""
	:
	"\""
	)
	)
	).do_request().as(resp, bytes(resp.Body).decode_json().as(body, {
	"resources": has(body.resources) && body.resources.size() > 0 ? body.resources : "",
	"want_more": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total),
	"offset": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total) ?
	int(state.offset) + int(body.resources.size())
	:
	0,
	"url": state.url,
	"batch_size": state.batch_size,
	}))
	).as(state, state.with(
	has (state.resources) && state.resources != "" ?
	post_request(
	state.url + "/devices/entities/devices/v2",
	"application/json",
	{"ids": state.resources }.encode_json()
	)
	.do_request().as(resp, bytes(resp.Body).decode_json().as(inner_body, {
	"events": inner_body.resources.map(e, {
	"message": e.encode_json(),
	}),
	"cursor": {
	"last_timestamp": (
	has(inner_body.resources) && inner_body.resources.size() > 0 ?
	inner_body.resources.map(e, e.modified_timestamp).max()
	:
	(
	has(state.cursor) && has(state.cursor.last_timestamp) ?
	state.cursor.last_timestamp
	:
	null
	)
	),
	"first_timestamp": (
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	( state.want_more ? state.cursor.first_timestamp : state.cursor.last_timestamp )
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339)
	),
	},
	}))
	:
	{}
	)
	)
	)
	(
	state.with(
	(
	!state.want_more ?
	request("GET", state.url + "/devices/queries/devices/v1?sort=modified_timestamp.asc&offset=0&limit=" + string(state.batch_size) + "&filter=modified_timestamp:>\"" + (
	has(state.cursor) && has(state.cursor.last_timestamp) && state.cursor.last_timestamp != null ?
	state.cursor.last_timestamp + "\""
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339) + "\""
	))
	:
	request("GET", state.url + "/devices/queries/devices/v1?sort=modified_timestamp.asc&offset=" + string(state.offset) + "&limit=" + string(state.batch_size) + "&filter=modified_timestamp:>\"" + (
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	state.cursor.first_timestamp + "\""
	:
	"\""
	))
	).do_request().as(resp, bytes(resp.Body).decode_json().as(body, {
	"resources": has(body.resources) && body.resources.size() > 0 ? body.resources : "",
	"want_more": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total),
	"offset": ((int(state.offset) + body.resources.size()) < body.meta.pagination.total) ?
	int(state.offset) + int(body.resources.size())
	:
	0,
	"url": state.url,
	"batch_size": state.batch_size,
	}))
	).as(state, state.with(
	!has(state.resources) || state.resources == "" ? {} :
	post_request(
	state.url + "/devices/entities/devices/v2",
	"application/json",
	{"ids": state.resources }.encode_json()
	).do_request().as(resp, bytes(resp.Body).decode_json().as(inner_body, {
	"events": inner_body.resources.map(e, {
	"message": e.encode_json(),
	}),
	"cursor": {
	"last_timestamp": (
	has(inner_body.resources) && inner_body.resources.size() > 0 ?
	inner_body.resources.map(e, e.modified_timestamp).max()
	: has(state.cursor) && has(state.cursor.last_timestamp) ?
	state.cursor.last_timestamp
	:
	null
	),
	"first_timestamp": (
	has(state.cursor) && has(state.cursor.first_timestamp) && state.cursor.first_timestamp != null ?
	( state.want_more ? state.cursor.first_timestamp : state.cursor.last_timestamp )
	:
	(now - duration(state.initial_interval)).format(time_layout.RFC3339)
	),
	},
	}))
	)
	)
	)

[efd6]

Please add a redact field. docs.

[kcreddy]

Add if: ctx.event?.original == null

[kcreddy]

Can you remove ignore_failure: true wherever its not needed? We shouldn't be hiding any errors. Applies to rest of the datastreams as well.

[kcreddy]

This should be split into user.domain\\user.name

[kcreddy]

Can you apply geoip processor on this IP?

[kcreddy]

Add if: ctx.event?.original == null

[jamiehynds]

Can we adjust the introduction to read: The CrowdStrike Falcon integration allows you to easily connect your CrowdStrike Falcon platform to Elastic for seamless onboarding of alerts and telemetry from CrowdStrike Falcon and Falcon Data Replicator. Elastic Security can leverage this data for security analytics including correlation, visualisation and incident response.

[efd6]

Sorry I missed some issues with the CEL code in the previous review (nits only). I think there is an issue with the user name splitting condition.

[efd6]

Suggested change

	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=0&limit=" + string(state.batch_size) + "&filter=timestamp:>\"" + (
	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=0&limit=" + string(state.batch_size) + '&filter=timestamp:>"' + (

[efd6]

Suggested change

	state.cursor.last_timestamp + "\""
	state.cursor.last_timestamp + '"'

[efd6]

Suggested change

	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=" + string(state.offset) + "&limit=" + string(state.batch_size) + "&filter=timestamp:>\"" + (
	request("GET", state.url + "/alerts/queries/alerts/v1?sort=timestamp|asc&offset=" + string(state.offset) + "&limit=" + string(state.batch_size) + '&filter=timestamp:>"' + (

[efd6]

Suggested change

	state.cursor.first_timestamp + "\""
	state.cursor.first_timestamp + '"'

[efd6]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[jamiehynds]

@mohitjha-elastic Falcon Data Replicator (FDR) is a separate product from CrowdStrike and it doesn't rely on the SIEM Connector. Can we ensure FDR has it's own section. It's currently under our SIEM Connector section, which implies that SIEM Connector is required.

[jamiehynds]

@mohitjha-elastic is it possible to also populate event.category:malware for alerts that relate to malware?

[mohitjha-elastic]

We are not getting any specific fields from which we can categorize it if it's related to malware.

[jamiehynds]

map to file.extension?

[mohitjha-elastic]

I guess we can not because looking at the responses it's not giving file extensions everytime.

[jamiehynds]

Got it,thanks @mohitjha-elastic. I was going on the assumption that the filetype is always an extension.

[jamiehynds]

map to process.start?

[jamiehynds]

map to process.end?

[jamiehynds]

@mohitjha-elastic map to threat.tactic.id? https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-tactic-id

[jamiehynds]

@mohitjha-elastic map to https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-tactic-name ?

[jamiehynds]

@mohitjha-elastic map to https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-technique-name ?

[jamiehynds]

@mohitjha-elastic map to https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-technique-id ?

[jamiehynds]

@tomsonpl we're about to add support for CrowdStrike unified alerts, which pull alerts directly from their API and provides more information than the DetectionSummary events. Can you take a quick look at the mappings in this sample log and assess if any tweaks are required in order to populate Analyzer?

[kcreddy]

LGTM once merge conflicts are fixed. Lets wait for Jamie's review comments to be answered.

[jamiehynds]

@kcreddy LGTM to me - ok to merge.

[kcreddy]

@mohitjha-elastic we can merge once you remove remove merge conflicts and after Jamie's suggestions are implemented.

[mohitjha-elastic]

@mohitjha-elastic we can merge once you remove remove merge conflicts and after Jamie's suggestions are implemented.

@kcreddy Removed the merge conflicts and implemented the changed suggested by Jamie. You can take a final look then we are good to go.

[kcreddy]

@mohitjha-elastic Just finished with final review, apologise for the delay.
I verified all of Jamie's comments are either included or answered in the comments.

[kcreddy]

Suggested change

	- description: Update minimum kibana version to 8.11.0.
	- description: Update minimum kibana version to 8.12.0.

[mohitjha-elastic]

@kcreddy Ah, missed that.
As this change is coming from the previous PR so I guess we should remove this changelog description from this PR.

[kcreddy]

The tactic.id and technique.id doesn't seem to be prefixed with CS as per https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-tactic-id. May I know where these values are derived from?

[kcreddy]

When assigning threat.tactic.id, we could remove CS prefix to match it with ECS standards https://attack.mitre.org/tactics/. Same for techniques

[kcreddy]

Suggested change

The [CrowdStrike](https://www.crowdstrike.com/) Falcon integration allows you to easily connect your CrowdStrike Falcon platform to Elastic for seamless onboarding of alerts and telemetry from CrowdStrike Falcon and Falcon Data Replicator. Elastic Security can leverage this data for security analytics including correlation, visualisation and incident response. It provides support using three different modes for integrating CrowdStrike to the Elastic:

The [CrowdStrike](https://www.crowdstrike.com/) Falcon integration allows you to easily connect your CrowdStrike Falcon platform to Elastic for seamless onboarding of alerts and telemetry from CrowdStrike Falcon and Falcon Data Replicator. Elastic Security can leverage this data for security analytics including correlation, visualization and incident response. It provides support using three different modes for integrating CrowdStrike to the Elastic:

Just to have conformity with our docs.

[kcreddy]

LGTM 👍🏼
Will merge once CI finishes successfully.

[kcreddy]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 00e2159

History

• 💚 Build #8634 succeeded 684478a
• 💚 Build #8451 succeeded 08e658b
• 💔 Build #8092 failed ac2680e

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
97.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package crowdstrike - 1.31.0 containing this change is available at https://epr.elastic.co/search?package=crowdstrike