[integrations][CrowdStrike] - Fixed Windows NT timestamp parsing issue and IDP Log pipeline field naming issue #7548

ShourieG · 2023-08-26T07:01:25Z

Type of change

Bug

What does this PR do?

This fixes the Windows NT timestamp parsing issues by introducing a conditional painless script that converts the NT timestamp to unix epoch timestamp. The PR also fixes a small field naming issue in the IDP Log pipeline, where target_field: event.end was mistakenly labelled as target_field: event.start.

Note

The logic for this PR is derived from the PR: #5168

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [CrowdStrike] - Windows NT timestamp parsing issue #7547

Screenshots

elasticmachine · 2023-08-26T07:02:45Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-08-26T07:18:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-28T06:40:08.188+0000
Duration: 16 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28
Skipped	0
Total	28

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-08-26T07:18:13Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`15/15`)	💚
Classes	100.0% (`15/15`)	💚
Methods	95.876% (`93/97`)	👍 20.876
Lines	88.18% (`3581/4061`)	👎 -11.82
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-08-26T07:18:13Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`15/15`)	💚 3.35
Classes	100.0% (`15/15`)	💚 3.35
Methods	95.876% (`93/97`)	👍 3.998
Lines	88.157% (`3573/4053`)	👎 -0.162
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-08-27T21:27:32Z

.../crowdstrike/data_stream/falcon/_dev/test/pipeline/test-falcon-ipd-summary.log-expected.json

@@ -28,12 +28,13 @@
                "category": [
                    "malware"
                ],
-                "created": "+4223582-11-13T00:53:20.000Z",
+                "created": "2023-03-01T05:50:56.000Z",
+                "end": "1992-03-21T19:15:00.000Z",


This doesn't look right.

@efd6 This is what the source log has. I did not change anything here. You can check it here.

Both StartTime & EndTime have the exact value and if you plug these values here https://www.epochconverter.com/ldap, the result is the exactly the same.

Oh, I absolutely agree that this is what the input says, but I don't understand why the input has an event that occurred in 1992.

I'm not sure either, maybe @P1llus has some idea since he worked on adding these events and logs.

efd6 · 2023-08-27T21:28:13Z

.../crowdstrike/data_stream/falcon/_dev/test/pipeline/test-falcon-ipd-summary.log-expected.json

                "kind": "alert",
                "original": "{\n\t\"metadata\": {\n\t\t\"customerIDString\": \"12312312312312321\",\n\t\t\"offset\": 2662765,\n\t\t\"eventType\": \"IdpDetectionSummaryEvent\",\n\t\t\"eventCreationTime\": 1686848064000,\n\t\t\"version\": \"1.0\"\n\t},\n\t\"event\": {\n\t\t\"ContextTimeStamp\": 133221234560000000,\n\t\t\"DetectId\": \"12345678901234567890123456789012:ind:12345678901234567890123456789012:12345678-1234-1234-1234-123456789000\",\n\t\t\"DetectName\": \"Unusual login to an endpoint\",\n\t\t\"DetectDescription\": \"A user logged in to a machine for the first time\",\n\t\t\"FalconHostLink\": \"https://falcon.crowdstrike.com/identity-protection/detections/12345678901234567890123456789012:ind:12345678901234567890123456789012:12345678-1234-1234-1234-123456789000?cid=12345678901234567890123456789012\",\n\t\t\"StartTime\": 123456789000000000,\n\t\t\"EndTime\": 123456789000000000,\n\t\t\"Severity\": 7,\n\t\t\"Tactic\": \"Initial Access\",\n\t\t\"Technique\": \"Valid Accounts\",\n\t\t\"Objective\": \"Gain Access\",\n\t\t\"SourceAccountDomain\": \"DOMAIN.COM\",\n\t\t\"SourceAccountName\": \"johnb\",\n\t\t\"SourceAccountObjectSid\": \"S-1-3-44-55555555-666666666-7777777777-88888\",\n\t\t\"SourceEndpointAccountObjectGuid\": \"12345678-1234-1234-1234-123456789000\",\n\t\t\"SourceEndpointAccountObjectSid\": \"S-1-3-44-55555555-666666666-7777777777-88888\",\n\t\t\"SourceEndpointHostName\": \"pc01.domain.com\",\n\t\t\"SourceEndpointIpAddress\": \"81.2.69.144\",\n\t\t\"SourceEndpointSensorId\": \"12345678901234567890123456789012\",\n\t\t\"PrecedingActivityTimeStamp\": 133154452345780000,\n\t\t\"MostRecentActivityTimeStamp\": 133313215755670000,\n\t\t\"ActivityId\": \"12345678-1234-1234-1234-123456789000\",\n\t\t\"PatternId\": 51135\n\t}\n}",
                "reference": "https://falcon.crowdstrike.com/identity-protection/detections/12345678901234567890123456789012:ind:12345678901234567890123456789012:12345678-1234-1234-1234-123456789000?cid=12345678901234567890123456789012",
                "severity": 7,
-                "start": "+3914159-11-26T20:00:00.000Z",
+                "start": "1992-03-21T19:15:00.000Z",


Same response as above.

packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml

packages/crowdstrike/changelog.yml

ShourieG · 2023-08-28T04:23:57Z

@efd6 I've resolved all the comments except for why the source log timestamp is the way it is atm. For that we need to wait for @P1llus. If you feel that this is not a blocker for the approval then could you approve the PR if everything looks ok ?

packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml

…r of 100-nanosecond intervals that have elapsed since January 1, 1601 (UTC). The high bit of this value is set to 1 to indicate that it's a Windows NT timestamp. The condition longValue > 0x0100000000000000L checks if the high bit of the 64-bit value is set, which indicates that it's a Windows NT timestamp. If this condition is true, it means the timestamp is a Windows NT timestamp and needs to be converted to a Unix timestamp. The conversion involves dividing the value by 10000000 to convert from 100-nanosecond intervals to seconds and then subtracting the offset 11644473600L to account for the difference between Windows NT and Unix epoch times.

ShourieG · 2023-08-28T06:13:28Z

@efd6 I've made the suggested updates

elasticmachine · 2023-08-29T07:28:13Z

Package crowdstrike - 1.18.1 containing this change is available at https://epr.elastic.co/search?package=crowdstrike

…e and IDP Log pipeline field naming issue (#7548) * fixed windows NT timestamp issue and idp logs pipeline issues * updated changelog * updated comments and implemented PR sugestions * In Windows NT, the timestamp is a 64-bit value representing the number of 100-nanosecond intervals that have elapsed since January 1, 1601 (UTC). The high bit of this value is set to 1 to indicate that it's a Windows NT timestamp. The condition longValue > 0x0100000000000000L checks if the high bit of the 64-bit value is set, which indicates that it's a Windows NT timestamp. If this condition is true, it means the timestamp is a Windows NT timestamp and needs to be converted to a Unix timestamp. The conversion involves dividing the value by 10000000 to convert from 100-nanosecond intervals to seconds and then subtracting the offset 11644473600L to account for the difference between Windows NT and Unix epoch times. * made PR suggested changes * updated comments

fixed windows NT timestamp issue and idp logs pipeline issues

8351d32

ShourieG requested a review from a team as a code owner August 26, 2023 07:01

updated changelog

0b26a27

ShourieG self-assigned this Aug 26, 2023

ShourieG added the Team:Security-External Integrations label Aug 26, 2023

ShourieG added bugfix 8.10 candidate integration Label used for meta issues tracking each integration labels Aug 26, 2023

ShourieG requested review from efd6 and P1llus August 26, 2023 07:03

efd6 reviewed Aug 27, 2023

View reviewed changes

updated comments and implemented PR sugestions

996e3ae

efd6 reviewed Aug 28, 2023

View reviewed changes

packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

efd6 reviewed Aug 28, 2023

View reviewed changes

packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

ShourieG added 2 commits August 28, 2023 11:39

made PR suggested changes

2733a4a

updated comments

311196f

efd6 approved these changes Aug 28, 2023

View reviewed changes

ShourieG merged commit 7a6bf69 into elastic:main Aug 29, 2023
4 checks passed

ShourieG deleted the bugfix/crowdstrike_nt_timestamp branch August 29, 2023 07:05

This was referenced Aug 29, 2023

The [date] processor allows dates beyond the Gregorian calendar. elastic/go-elasticsearch#714

Closed

The [date] processor allows dates beyond the Gregorian calendar elastic/elasticsearch#98982

Open

taylor-swanson mentioned this pull request Sep 8, 2023

[crowdstrike] Convert Win32 timestamps to unix millisecond timestamp #7734

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[integrations][CrowdStrike] - Fixed Windows NT timestamp parsing issue and IDP Log pipeline field naming issue #7548

[integrations][CrowdStrike] - Fixed Windows NT timestamp parsing issue and IDP Log pipeline field naming issue #7548

ShourieG commented Aug 26, 2023

elasticmachine commented Aug 26, 2023

elasticmachine commented Aug 26, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Aug 26, 2023 •

edited

Loading

elasticmachine commented Aug 26, 2023

efd6 Aug 27, 2023

ShourieG Aug 28, 2023

efd6 Aug 28, 2023

ShourieG Aug 28, 2023

efd6 Aug 27, 2023

ShourieG Aug 28, 2023

ShourieG commented Aug 28, 2023

ShourieG commented Aug 28, 2023

elasticmachine commented Aug 29, 2023

[integrations][CrowdStrike] - Fixed Windows NT timestamp parsing issue and IDP Log pipeline field naming issue #7548

[integrations][CrowdStrike] - Fixed Windows NT timestamp parsing issue and IDP Log pipeline field naming issue #7548

Conversation

ShourieG commented Aug 26, 2023

Type of change

What does this PR do?

Note

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Aug 26, 2023

elasticmachine commented Aug 26, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Aug 26, 2023 • edited Loading

🌐 Coverage report

elasticmachine commented Aug 26, 2023

🌐 Coverage report

efd6 Aug 27, 2023

Choose a reason for hiding this comment

ShourieG Aug 28, 2023

Choose a reason for hiding this comment

efd6 Aug 28, 2023

Choose a reason for hiding this comment

ShourieG Aug 28, 2023

Choose a reason for hiding this comment

efd6 Aug 27, 2023

Choose a reason for hiding this comment

ShourieG Aug 28, 2023

Choose a reason for hiding this comment

ShourieG commented Aug 28, 2023

ShourieG commented Aug 28, 2023

elasticmachine commented Aug 29, 2023

elasticmachine commented Aug 26, 2023 •

edited

Loading

elasticmachine commented Aug 26, 2023 •

edited

Loading