[CrowdStrike]: A empty index was created and missing log field @timestamp

[hardyqiu]

Integration Name

CrowdStrike [crowdstrike]

Dataset Name

dataset is alright

Integration Version

3.14.0

Agent Version

8.19.6

Agent Output Type

logstash

Elasticsearch Version

8.19

OS Version and Architecture

Ubuntu

Software/API Version

No response

Error Message

I upgrade CrowdStrike integrations from 2.8.0 to 3.14.0 yesterday, after that, in Security -> detection prompt a warning message for all rule.

The warning message is The following indices are missing the timestamp field "@timestamp": ["logs-crowdstrike_lookup.dest_aidmaster-1"]

And this index "logs-crowdstrike_lookup.dest_aidmaster-1" is empty without any data inside, the index created time aligns with when I upgraded the integration.
So far, this warning message hasn't affected my ELK function, but I would like to know how to resolve this issue.
Much appreciated.

Event Original

No response

What did you do?

1

What did you see?

1

What did you expect to see?

1

Anything else?

No response

[hardyqiu]

added some screenshot for the mentioned index.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[hardyqiu]

As I found "logs-crowdstrike_lookup.dest_aidmaster-1" is the destination index in Transform: logs-crowdstrike.latest_aidmaster-default-0.2.0, but this Transform job is not working because the source index doesn't exist.
I not sure if due to some mis-config in CrowdStrike integrations.

[kcreddy]

@hardyqiu, can you confirm which datasets you enabled currently in Crowdstrike integration?
We support 5 datasets right now.

• falcon
• fdr
• alert
• host
• vulnerability

Just wanted to check if you are consuming fdr dataset to understand the extent of the problem.

[kcreddy]

2 concerns discussed in the issue.

1. Warning in the detection rule

The warning message is: The following indices are missing the timestamp field "@timestamp": ["logs-crowdstrike_lookup.dest_aidmaster-1"]

I can confirm this happens even with fdr data stream is enabled.
Steps to Reproduce (uses elastic-package):

1. Run elastic-package test system --generate -v --defer-cleanup 120m --data-streams=fdr
2. Verify data is being ingested and transform logs-crowdstrike.latest_aidmaster-default-0.2.0 is running healthy.
3. Create a custom detection rule.

• In Kibana → Security → Rules → Create new rule → Custom query.
• Set the index pattern to something that matches the index above (e.g. logs-*) Query: *.
• Save and enable.

4. Navigate to Kibana → Security → Rules
5. The Last Response column indicates Warning with message The following indices are missing the timestamp field "@timestamp": ["logs-crowdstrike_lookup.dest_aidmaster-1"]

2. Transform issue

An error occurred loading the index data. Please ensure your query syntax is valid.
Bad Request: [[status_exception] Source indices have been deleted or closed.]: Source indices have been deleted or closed.

This only occurs when the fdr data stream doesn't contain any documents. This is only expected when users don't enable the fdr data stream and thus doesn't receive any fdr documents.
It just has an ugly preview error in the Transforms UI. It is not a hard bug.

The transform is still Healthy:

[kcreddy]

@navnit-elastic, can you look at the issue #1. Warning in the detection rule ? Is @timestamp deliberately omitted by design from the destination indices?

[navnit-elastic]

Thank you @hardyqiu, @kcreddy for raising this. Correct, the @timestamp field is omitted intentionally from the transform destination index (logs-crowdstrike_lookup.dest_aidmaster-1) to enable consistent host metadata enrichment in FDR events.

I'm looking into whether or not we can silent this warning in Kibana.

[hardyqiu]

@hardyqiu, can you confirm which datasets you enabled currently in Crowdstrike integration? We support 5 datasets right now.

• falcon
• fdr
• alert
• host
• vulnerability

Just wanted to check if you are consuming fdr dataset to understand the extent of the problem.

@kcreddy Much appreciated for your attention. Yes, we did enable fdr in CrowdStrike integration.