Parsing rules fix filter 9 (demisto#28478)

* Added filter to MicrosoftCloudAppSecurity

* Added release notes to MicrosoftCloudAppSecurity

* Added release notes to MicrosoftCloudAppSecurity

* Added filter to MicrosoftDHCP parsing rules

* Added release note to MicrosoftDHCP

* Modified microsoftadfs parsing rule

* Added release note for microsoftadfs

* Updated MicrosoftDefenderAdvancedThreatProtection parsing rule

* Added release note for microsoftdefenderadvancedthreatprotection

* Updated parsing rule for MicrosoftCloudAppSecurity

* Modified the filter for DHCP parsing rule, and modified the readme.

* Modified microsoft cloud app security readme.

* Modified microsoft dhcp readme.

* Modified microsoft defender advanced threat protection readme.

* Modified microsoft defender advanced threat protection parsing rule.

* Updated MicrosoftWindowsEvents parsing rules

* Added release note for microsoftwindowsevents

* Added note to microsoft windows events

* Added note to microsoft windows events

* Bump pack from version MicrosoftDefenderAdvancedThreatProtection to 1.15.35.

* Bump pack from version MicrosoftCloudAppSecurity to 2.1.36.

---------

Co-authored-by: Content Bot <bot@demisto.com>

Packs/MicrosoftADFS/ParsingRules/MicrosoftADFSParsingRules/MicrosoftADFSParsingRules.xif

@@ -1,5 +1,5 @@
 [INGEST:vendor="microsoft", product="windows", target_dataset="microsoft_adfs_raw", no_hit=drop]
-config case_sensitive=false 
-| filter provider_name="AD FS Auditing" OR provider_name="AD FS"
-| alter _product="adfs"
-| alter _time = coalesce(time_created,_insert_time);
+config case_sensitive = false
+| filter provider_name = "AD FS Auditing" OR provider_name="AD FS"
+| alter _product = "adfs"
+| alter _time = if(to_string(time_created) ~= "\d{2}:\d{2}:\d{2}.*", time_created, _insert_time);

Packs/MicrosoftADFS/ReleaseNotes/1_0_15.md

@@ -0,0 +1,3 @@
+#### Parsing Rules
+##### Microsoft ADFS Collection Parsing Rule
+- Added a filter in the parsing rule to enhance its logic.

Packs/MicrosoftADFS/pack_metadata.json

@@ -2,7 +2,7 @@
     "name": "Microsoft AD FS Collection",
     "description": "Microsoft Active Directory Federation Services",
     "support": "xsoar",
-    "currentVersion": "1.0.14",
+    "currentVersion": "1.0.15",
     "author": "Cortex XSOAR",
     "url": "https://www.paloaltonetworks.com/cortex",
     "email": "",

Packs/MicrosoftCloudAppSecurity/ParsingRules/MicrosoftCloudAppSecurity/MicrosoftCloudAppSecurity.xif

@@ -1,2 +1,3 @@
 [INGEST:vendor="Microsoft", product="defender_cloud_apps", target_dataset="microsoft_defender_cloud_apps_raw", no_hit=keep]
-alter _time = to_timestamp(to_integer(timestamp), "millis");
+filter to_string(timestamp) ~= "\d{13,}"
+| alter _time = to_timestamp(to_integer(timestamp), "millis");

Packs/MicrosoftCloudAppSecurity/README.md

@@ -1,2 +1,4 @@
 ## License Information
 Available for E3 (EMS), E5, and standalone licenses.
+
+Note: Time parsing is supported only when the field "timestamp" is in epoch milliseconds format (for example 1674208800123).

Packs/MicrosoftCloudAppSecurity/ReleaseNotes/2_1_36.md

@@ -0,0 +1,3 @@
+#### Parsing Rules
+##### Microsoft Cloud App Security Parsing Rule
+- Added a filter in the parsing rule to enhance its logic.

Packs/MicrosoftCloudAppSecurity/pack_metadata.json

@@ -2,7 +2,7 @@
     "name": "Microsoft Defender for Cloud Apps",
     "description": "Microsoft Cloud App Security Integration, a Cloud Access Security Broker that supports various deployment modes",
     "support": "xsoar",
-    "currentVersion": "2.1.35",
+    "currentVersion": "2.1.36",
     "author": "Cortex XSOAR",
     "url": "https://www.paloaltonetworks.com/cortex",
     "email": "",

Packs/MicrosoftDHCP/ParsingRules/MicrosoftDHCP/MicrosoftDHCP.xif

@@ -1,4 +1,6 @@
 [INGEST:vendor="microsoft", product="DHCP", target_dataset="microsoft_dhcp_raw", no_hit=keep]
-alter tmp_time = concat(date, " ", time," ", `timezone`)
-|alter _time=parse_timestamp("%D %X %Ez", tmp_time)
-|fields -tmp_time;
+// Support only date time of format: MM/dd/yy hh:mm:ss [+|-]nn:nn. For example: "01/10/21 10:00:00 +03:00".
+filter date ~= "\d{2}\/\d{2}\/\d{2}" and time ~= "\d{2}:\d{2}:\d{2}.*" and `timezone` ~= "[\+|-]\d{2}\:?\d{2}"
+| alter tmp_time = concat(date, " ", time," ", `timezone`)
+| alter _time = parse_timestamp("%D %X %Ez", tmp_time)
+| fields -tmp_time;

Packs/MicrosoftDHCP/README.md

@@ -9,15 +9,18 @@ This pack includes Cortex XSIAM content.
 3. Select the **General** tab.
 4. Select the **Enable DHCP audit logging** checkbox.
 5. Click **OK**.
-
+6. 
+Note:
+Time parsing is supported only when the below fields have the mentioned formats:
+- date - MM/dd/yy (01/10/21)
+- time - hh:mm:ss (10:00:00)
+- timezone - [+|-]nn:nn (+03:00)
 
 ## Collect Events from Vendor
 
 In order to use the collector, use the [XDRC (XDR Collector)](#xdrc-xdr-collector) option.
 
 
-
-
 ### XDRC (XDR Collector)
 
 To create or configure the Filebeat collector, use the information described [here](https://docs.paloaltonetworks.com/cortex/cortex-xdr/cortex-xdr-pro-admin/cortex-xdr-collectors/xdr-collector-datasets#id7f0fcd4d-b019-4959-a43a-40b03db8a8b2) and [here](https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR/Cortex-XDR-Pro-Administrator-Guide/Add-an-XDR-Collector-Profile-for-Windows).

Packs/MicrosoftDHCP/ReleaseNotes/1_0_1.md

@@ -0,0 +1,3 @@
+#### Parsing Rules
+##### Microsoft DHCP
+- Added a filter in the parsing rule to enhance its logic.

Packs/MicrosoftDHCP/pack_metadata.json

@@ -2,7 +2,7 @@
     "name": "Microsoft DHCP",
     "description": "Dynamic Host Configuration Protocol (DHCP) is a client/server protocol that automatically provides an Internet Protocol (IP) host with its IP address and other related configuration information such as the subnet mask and default gateway.",
     "support": "xsoar",
-    "currentVersion": "1.0.0",
+    "currentVersion": "1.0.1",
     "author": "Cortex XSOAR",
     "url": "https://www.paloaltonetworks.com/cortex",
     "email": "",

Packs/MicrosoftDefenderAdvancedThreatProtection/ParsingRules/MicrosoftDefenderAdvancedThreatProtection/MicrosoftDefenderAdvancedThreatProtection.xif

@@ -1,5 +1,7 @@
 [INGEST:vendor="Microsoft 365", product="Defender", target_dataset="microsoft_365_defender_raw", no_hit = keep]
-alter tmp_splitStr = split(lastEventTime , ".")
+// Support only date time of format: yyyy-MM-ddThh:mm:%E3S. For example: "2021-07-01T10:00:00.667Z".
+filter to_string(lastEventTime) ~= "\d{4,}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d*Z"
+| alter tmp_splitStr = split(to_string(lastEventTime) , ".")
 | alter tmp_epoch_s = to_epoch(parse_timestamp("%Y-%m-%dT%H:%M:%S", arrayindex(tmp_splitStr ,0)), "millis")
 | alter tmp_epoch_ms = to_integer(arrayindex(regextract(arrayindex(tmp_splitStr, 1),"^(\d{3}).*"),0))
 | alter tmp_cc = add(tmp_epoch_s , tmp_epoch_ms)

Packs/MicrosoftDefenderAdvancedThreatProtection/README.md

@@ -42,6 +42,12 @@ To find the appropriate Endpoint URI, use the following resources:
 - For Worldwide or Geo Proximity URLs, see [Supported Microsoft 365 Defender APIs](https://learn.microsoft.com/en-us/microsoft-365/security/defender/api-supported?view=o365-worldwide#endpoint-uris)
 - For Us Government & DoD URLs, see [Microsoft Defender for Endpoint for US Government customers](https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/gov?view=o365-worldwide#api)
 
+## Log ingestion
+
+Note: In order to parse the timestamp correctly, make sure that the timestamp field is in UTC time zone (timestamp ends with "Z").
+The supported time format is yyyy-MM-ddThh:mm:%E3S (2021-12-08 10:00:00.123Z). The relevant field is "lastEventTime".
+
+
 ## Licence information
 
 Available for E3, E5, and standalone licenses.

Packs/MicrosoftDefenderAdvancedThreatProtection/ReleaseNotes/1_15_35.md

@@ -0,0 +1,3 @@
+#### Parsing Rules
+##### Microsoft Defender Advanced Threat Protection Parsing Rule
+- Added a filter in the parsing rule to enhance its logic.

Packs/MicrosoftDefenderAdvancedThreatProtection/pack_metadata.json

@@ -2,7 +2,7 @@
     "name": "Microsoft Defender for Endpoint",
     "description": "Microsoft Defender for Endpoint (previously Microsoft Defender Advanced Threat Protection (ATP)) is a unified platform for preventative protection, post-breach detection, automated investigation, and response.",
     "support": "xsoar",
-    "currentVersion": "1.15.34",
+    "currentVersion": "1.15.35",
     "author": "Cortex XSOAR",
     "url": "https://www.paloaltonetworks.com/cortex",
     "email": "",

Packs/MicrosoftWindowsEvents/ParsingRules/MicrosoftWindowsEvents/MicrosoftWindowsEvents.xif

@@ -1,4 +1,4 @@
 [INGEST:vendor="microsoft", product="windows", target_dataset="microsoft_windows_raw", no_hit=drop]
 config case_sensitive=false 
-| filter provider_name!="AD FS Auditing" OR provider_name!="AD FS"
-| alter _time = coalesce(time_created,_insert_time);
+| filter provider_name != "AD FS Auditing" OR provider_name != "AD FS"
+| alter _time = if(to_string(time_created) ~= "\d{2}:\d{2}:\d{2}.*", time_created, _insert_time);

Packs/MicrosoftWindowsEvents/ReleaseNotes/1_0_7.md

@@ -0,0 +1,4 @@
+#### Parsing Rules
+##### Microsoft Windows Events Parsing Rule
+- Added a filter in the parsing rule to enhance its logic.
+- Please notice "Microsoft Logs Base" pack is deprecated, if you have it installed, please remove it and use the "Microsoft Windows Event Logs" pack instead.

Packs/MicrosoftWindowsEvents/pack_metadata.json

@@ -2,7 +2,7 @@
     "name": "Microsoft Windows Event Logs",
     "description": "The Windows event log is a detailed record of system, security and application notifications stored by the Windows operating system.",
     "support": "xsoar",
-    "currentVersion": "1.0.6",
+    "currentVersion": "1.0.7",
     "author": "Cortex XSOAR",
     "url": "https://www.paloaltonetworks.com/cortex",
     "email": "",